Mobile devices and smartwatches have pretty small screens, making it difficult for many to interact with them. Computer scientists and electrical engineers from the University of Washington have come up with a solution—sonar technology that lets you interact with your tiny-screen devices by writing or gesturing on any nearby surface, such as a tabletop, paper or even in mid-air.
The technology, called FingerIO, tracks fine-grained finger movements by turning your smart device into an active sonar system using its own microphones and speakers. The signals bounce off the finger, and the "echoes" are recorded by the device's microphones and used to calculate the finger's location in space.
And because sound waves travel through fabric and do not require line-of-sight to work, users can even interact with their device when it’s in his or her pocket or under the sleeve of a shirt, like a smartwatch.
The team opted for four acoustic signals because sound waves travel slower than radio waves used in radar; therefore less processing bandwidth is needed, which simplifies the system. According to Shyam Gollakota, University of Washington (UW) Assistant Professor of Computer Science and Engineering, it also reduces costs because every device has a speaker and microphones, eliminating the need for special hardware.
Throughout the course of their research, the UW engineers demonstrated that FingerIO can accurately track two-dimensional finger movements to within 8 mm, which is enough accuracy to interact with today's mobile devices.
"You can't type very easily onto a smartwatch display, so we wanted to transform a desk or any area around a device into an input surface," said Rajalakshmi Nandakumar, a UW doctoral student in computer science and engineering. "I don't need to instrument my fingers with any other sensors—I just use my finger to write something on a desk or any other surface, and the device can track it with high resolution."
With FingerIO, the team suggests that a user could adjust the volume, press buttons or scroll through menus on a device without ever touching it—or even type in a search command or send a text in the air rather than typing on a tiny screen. But sonar echoes are weak and typically not accurate enough to track finger motion at a high resolution. Errors of a few centimeters make it impossible to differentiate between writing individual letters or subtle hand gestures. In order to make the sonar echoes stronger and more accurate, the UW researchers employed a type of signal used in wireless communication called Orthogonal Frequency Division Multiplexing. The signal made it possible to achieve high-resolution finger tracking using sound.
To test the system, the researchers created a FingerIO prototype app for Android devices and downloaded it to a Samsung Galaxy S4 smartphone and a smartwatch customized with two microphones, needed to track finger motion in two dimensions. Test subjects drew shapes: stars, squiggles and figure eights on a touchpad next to a smartphone or smartwatch running FingerIO. When the researchers compared the touchpad tracings to the shapes created by FingerIO's tracking, they found that the average difference between the drawings and the FingerIO tracings was 0.8 centimeters for the smartphone and 1.2 centimeters for the smartwatch.
"Given that your finger is already a centimeter thick, that's sufficient to accurately interact with the devices," said Vikram Iyer, co-author of the research study and electrical engineering graduate student.
Next the team will demonstrate how FingerIO can be used to track multiple fingers moving at the same time, and extend its tracking abilities into 3-D by adding additional microphones to the devices.