Carnegie Mellon University researchers are using laser vibrometry — a method similar to one once used by the KGB for eavesdropping — to monitor vibrations and movements of objects, enabling smart devices to be aware of human activity. Source: Carnegie Mellon University
Two teams from Carnegie Mellon University have developed two new systems that enable smart devices to be aware of the environment around them. If smart devices don’t have environmental awareness, smart devices seem stupid. The two new systems allow smart devices to analyze either sound or vibrations to register the world around them.
"A smart speaker sitting on a kitchen countertop cannot figure out if it is in a kitchen, let alone know what a person is doing in a kitchen," said Chris Harrison, assistant professor in CMU's Human-Computer Interaction Institute (HCII). "But if these devices understood what was happening around them, they could be much more helpful."
The first new system developed uses common sensors and microphones to register the sound in the surrounding environment. The team named this system Ubicoustics. Ubicoustics uses the microphones from smart speakers, smartphones, smartwatches and other smart devices to register and classify the sounds associated with an environment.
"The main idea here is to leverage the professional sound-effect libraries typically used in the entertainment industry," said Gierad Laput, a Ph.D. student in HCII, "They are clean, properly labeled, well-segmented and diverse. Plus, we can transform and project them into hundreds of different variations, creating volumes of data perfect for training deep-learning models. This system could be deployed to an existing device as a software update and work immediately.”
Ubicoustics could be trained to work in any environment for which a training set of sounds exists. The system is based on an existing model that can label sounds. The new system is trained using variations of common sounds derived from libraries of professional sound effects. Hundreds of variations are possible. Once trained, Ubicoustics recognizes sounds and places them in the right context to allow a smart device to recognize where it is. Under testing, the team found that their system had around 80% accuracy. The team said that with additional research and development, that accuracy could improve even more.
The second method to enable environmental awareness is called Vibrosight. Vibrosight allows smart devices to register its surroundings based on vibrations detected with laser vibrometry.
"The cool thing about vibration is that it is a byproduct of most human activity," Ph.D. student Yang Zhang said. "The other cool thing is that vibrations are localized to a surface," he added.
Pretty much anything makes a vibration. Walking down a hall, pouring a cup of coffee, turning on a light, all create vibrations. Since, as Zhang points out, vibrations are specific to a surface, they don’t interfere with each other. This allows individual vibrations to be detected by a device without getting muddled by other vibrations.
Vibrosight uses a special sensor, low powered laser and a motorized steerable mirror. Overall the system costs $80. Researchers applied reflective tags, like the ones runners use at night, on the objects that the user wants to monitor. Vibrosight had 92% accuracy at identifying devices and 98% accuracy at finding if the device is on or off. Vibrosight can also detect movement and when the device is blocked by a person or non-tagged object.
Both systems were presented at the Association for Computing Machinery's User Interface Software and Technology Symposium in Berlin.
