Acquired Electronics360

Emerging Display Technologies

New Virtual Reality Technology Uses Mouth Gestures to Operate

06 October 2017

Binghamton University Professor of Computer Science Lijun Yin and his team created a new framework that interprets mouth gestures as a medium for interaction within virtual reality in real-time. Source: Binghamton UniversityBinghamton University Professor of Computer Science Lijun Yin and his team created a new framework that interprets mouth gestures as a medium for interaction within virtual reality in real-time. Source: Binghamton University

Researchers from Binghamton University State University of New York have developed a new technology that allows users to interact in virtual reality using only mouth gestures.

The increase of affordable virtual reality head-mounted displays provides users with realistic immersive visual experiences. But head-mounted displays occlude the upper half of a user’s face, which prevents facial action recognition from the whole face. In order to combat this, Binghamton University Professor of Computer Science Lijuin Yin and his team created a new framework that interprets mouth gestures as a medium for interaction within virtual reality in real-time.

Yin’s team tested this application on a group of graduate students. When a user put on the head-mounted display, a simple game was displayed. The objective of the game was to guide the avatar around a forest and eat as many cakes as possible. Players had to select their movement direction using head rotation and move using mouth gestures. They could only eat a cake by smiling. The system could describe and classify the mouth movements and it achieved high correct recognition rates. The system has been demonstrated and validated through a real-time virtual reality application.

"We hope to make this applicable to more than one person, maybe two. Think Skype interviews and communication," said Yin. "Imagine if it felt like you were in the same geometric space, face to face, and the computer program can efficiently depict your facial expressions and replicate them so it looks real."

The tech is still in the prototype phase, but Yin believes that this technology is applicable for many fields.

"The virtual world isn't only for entertainment. For instance, healthcare uses VR to help disabled patients," said Yin. "Medical professionals or even military personnel can go through training exercises that may not be possible to experience in real life. This technology allows the experience to be more realistic."

A paper on this research was presented at the IEEE International Conference on Multimedia and Expo.

To contact the author of this article, email Siobhan.Treacy@ieeeglobalspec.com


Powered by CR4, the Engineering Community

Discussion – 0 comments

By posting a comment you confirm that you have read and accept our Posting Rules and Terms of Use.
Engineering Newsletter Signup
Get the Engineering360
Stay up to date on:
Features the top stories, latest news, charts, insights and more on the end-to-end electronics value chain.
Advertisement
Weekly Newsletter
Get news, research, and analysis
on the Electronics industry in your
inbox every week - for FREE
Sign up for our FREE eNewsletter
Advertisement

CALENDAR OF EVENTS

Date Event Location
30 Nov-01 Dec 2017 Helsinki, Finland
23-27 Apr 2018 Oklahoma City, Oklahoma
18-22 Jun 2018 Honolulu, Hawaii
Find Free Electronics Datasheets
Advertisement