When analyzing headcam videos of people performing tasks, it is important for researchers to be able to characterize where a person is focusing during the task. This has been difficult in the past because this kind of technology didn’t exist - until now.
The AI-based computational tool can predict the way a user will act (Source: The University of Tokyo)
Researchers from the University of Tokyo have developed a computational tool that learns from past headcam footage and predicts where the user’s focus is going next. This is the next step to creating video-linked technology that can predict a user’s actions and provide them with guidance on their next step. For example this could be useful when cooking from recipes.
Gaze-predicting technologies currently rely on visual saliency, which uses distinctions like color, intensity and contrast to predict where the user is looking. Visual saliency becomes inadequate when shown complex-task videos, where users are likely to change their attention in a unpredictable way.
The team took visual saliency and built upon it to create a more accurate predictive computational tool. They combined visual saliency with a gaze prediction tool. The gaze prediction tool is an artificial intelligence (AI) tool that studies the sequence of actions from existing footage and applies the patterns it found to predict the user’s gaze when shown new footage.
"Our new approach involves the construction of first a 'saliency map' for each frame of footage, then an 'attention map' based on where the user was previously looking and on motion of the user's head, and finally the combination of both of these into a 'gaze map,'" researcher Yoichi Sato says, "Our results showed that this new tool outperformed earlier alternatives in terms of predicting where the gaze of the headcam user was actually directed."
According to lead author Yifei Huang, "Tools for evaluating so-called egocentric videos of this kind could even be applied in a medical context, such as assessing where a surgeon is focusing and offering guidance on the most appropriate steps to be taken next in an operation."
The paper on this new tool will be published in the proceedings of the European Conference on Computer Vision.
