Have you had a meal or dish that you wish you could recreate at home? I know I have. A few years ago I was in Seattle and I went to Molly Moon’s ice cream shop where I had the best ice cream of my life. It was honey and lavender in a waffle cone and I have never had an ice cream that good again. I half-joke with my family that I think about that ice cream at least once a week. I recently received an ice cream maker as a housewarming present and I have made it my own personal mission to recreate this ice cream. So far I haven’t been able to get the recipe right, but there is hope for my adventure. MIT researchers have discovered that artificial intelligence (AI) could help people recreate their favorite dishes by analyzing food photos.
Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) believe that AI could analyze food photos and help people learn recipes and understand their eating habits. A team from MIT and Qatar Computing Research Institute (QCRI) trained an artificial intelligence system called Pic2Recipe to view a photo of food and predict the ingredients as well as suggest similar recipes.
Food is mostly neglected in computer vision because their large-scale datasets that are needed to make the food predictions are not available. Food photos from social media like Instagram or Twitter could provide a look into a person’s dietary and health habits.
Social media and the internet have spurred research growth in classifying food data, but most of this research has used smaller data sets, which can lead to major gaps in food labeling.
In 2014, Swiss researchers created the Food-101 data set. They used this dataset to develop an algorithm that can recognize pictures of food with 50% accuracy. Future developments of this research have only improved to 80% accuracy, which suggests that the size of data set may be a limiting factor.
Larger data sets have been limited in their generalizations of populations. For example, a database from City University in Hong Kong has over 110,000 images with 65,000 recipes complete with ingredient lists and instructions, but that data is only Chinese cuisine.
The CSAIL team’s project goal is to build off this work but majorly expand the scope. They combined recipe sites like All Recipes and Food.com to develop Recipe1M. This database has over a million recipes that are annotated with information about ingredients in many dishes. The researchers used this data to train an AI network to find patterns and connections between food images and the corresponding ingredients and recipes.
Pic2Recipe can identify ingredients like flour, eggs and butter from only a photo and then give the user a few suggested recipes that it found in the database.
“You can imagine people using this to track their daily nutrition, or to photograph their meal at a restaurant and know what's needed to cook it at home later," said Christoph Trattner, an assistant professor at MODUL University, Vienna, in the New Media Technology Department who was not involved in the paper.
The AI system did particularly well with desserts like cookies or muffins (this bodes well for my ice cream adventure). It has difficulty determining and recognizing ingredients for more eclectic foods like sushi or smoothies.
The system was also stumped when it detected similar recipes for the same dishes, like lasagna. The team needed to make sure that the system isn’t penalized for similar recipes when trying to separate the different ones. One way to solve this problem was to see if the ingredients in each are generally similar before comparing the recipes.
The team hopes to improve the system so it can understand food in more detail. This could range from being able to infer how a food is prepared, to distinguishing variations of foods.
Researchers are interested in developing a system into a “dinner aide” that could figure out what a person should cook given their dietary preference and what the person has in their fridge.
This system could help people figure out what is in their food when the nutritional information isn’t readily available. For example, if a person knows what ingredients are in a dish but not the exact amount, the user can take a photo of the dish, enter the ingredients and run the system to find a similar recipe with the known quantities.
A paper on this research will be presented at the Computer Vision and Pattern Recognition conference in Honolulu.