The Defense Advanced Research Projects Agency (DARPA) has just awarded $2.1 million to Colorado State University (CSU) researchers to develop gesture technology that would allow for humans to interact with computers in a different way than ever before.
Typical computer interactions include typing, clicking and scrolling, but DARPA wants to create a level of communication that would make computers smart enough to reliably recognize non-verbal cues from humans in the most natural way possible.
The CSU project, titled “Communication Through Gestures, Expression and Shared Perception”, is being led by Professor of Computer Science Bruce Draper. It falls under DARPA’s “Communicating with Computers” funding program.
“Current human-computer interfaces are still severely limited,” says Draper, who is also joined by CSU researchers from the computer science and mathematics departments. “First, they provide essentially one-way communication; users tell the computer what to do. This was fine when computers were crude tools, but more and more, computers are becoming our partners and assistants in complex tasks. Communication with computers needs to become a two-way dialogue.”
To achieve its goal, the team has proposed creating a library of elementary composable ideas (ECIs), which can be described as little packets of information recognizable to computers. Each ECI contains information about a gesture or facial expression, derived from human users, as well as a syntactical element that constrains how the information can be read.
The researchers have set up a Microsoft Kinect interface to test out different methods. A human subject sits down at a table with blocks, pictures and other stimuli. The researchers try to communicate with and record the person’s natural gestures for concepts like “stop,” or, “huh?”
“We don’t want to say what gestures you should use,” Draper explains. “We want people to come in and tell us what gestures are natural. Then, we take those gestures and say, ‘OK, if that’s a natural gesture, how do we recognize it in real time, and what are its semantics? What roles does it play in the conversation? When do you use it? When do you not use it?’”
According to the project proposal, the work could someday allow people to communicate more easily with computers in noisy settings, or when a person is deaf or hard of hearing, or speaks another language and could revolutionize computer-human interaction by getting the computer to recognize non-verbal cues such as gestures, body language and facial expressions.