Imagine a robot sous chef predicting the next ingredient in a recipe, or robot vacuums that can predict which room needs the most vacuuming. Researchers from the University of Bonn have created a computer system that moves closer to these robotic assistants becoming part of our lives. The new computer system can accurately estimate the timing and duration of an activity up to five minutes into the future.
Prof. Jürgen Gall (right) and Yazan Abu Farha from the Institute of Computer Science at the University of Bonn. Source: Barbara Frommann/Uni Bonn
The software trained by watching a typical sequence of actions in videos for four hours — in this case, 40 videos of people preparing different salads. Each video was around six minutes long and had at least twenty different actions.
The researchers then asked the software to follow a new video, acting as an assistant. They told it what would happen for the first 20-30 percent of the video, and then asked it to predict the following actions.
"Accuracy was over 40 percent for short forecast periods, but then dropped the more the algorithm had to look into the future,” said Prof. Dr. Jürgen Gall, lead researcher of the study.
When the computer was asked to look beyond three minutes into the future, the success of the program dropped to 15 percent.
The computer's performance becomes worse when it is not told what happened in the first part of the video. The researchers hope to improve this aspect as they continue to develop the program.
The paper on this research will be presented at the Conference on Computer Vision and Pattern Recognition in Salt Lake City.
