A University of Glasgow-led team has put forth a study on how they set out to harness cutting-edge sensing technology to read lips. New “smart” hearing aids would combine conventional audio amplification with a second device to collect additional data for improved performance.
An international team of engineers and computing scientists developed the technology, which pairs radio-frequency (RF) sensing with artificial intelligence for the first time, along with traditional hearing aid technology, to identify lip movements.
Previous technological attempts used cameras to read lips, which raised obvious privacy issues and did not work with masks. Source: Peakstock/Adobe
To develop the system, the researchers asked volunteers to repeat the five vowel sounds (A, E, I, O and U) first while unmasked and then while wearing a surgical mask. As the volunteers repeated the vowel sounds, a dedicated radar sensor and a Wi-Fi transmitter scanned their faces using RF signals. Researchers also scanned their faces while their lips remained still.
It could also help tackle the “cocktail party effect”, a common shortcoming of traditional hearing aids as the broad spectrum of amplification can make it difficult for users to focus on specific sounds, like conversation with one specific person in a crowd.
Previous similar attempts used cameras to read lips, which raised obvious privacy issues and did not work with masks. This method preserves privacy by collecting only RF data, with no accompanying video footage.
“Around five percent of the world’s population – about 430 million people – have some kind of hearing impairment," says Dr. Qammer Abbasi of the University of Glasgow’s James Watt School of Engineering in a university release. Dr. Abbasi is a co-researcher in the study published in Nature Communications.
“Hearing aids have provided transformative benefits for many hearing-impaired people. A new generation of technology which collects a wide spectrum of data to augment and enhance the amplification of sound could be another major step in improving hearing-impaired people’s quality of life.
“With this research, we have shown that radio-frequency signals can be used to accurately read vowel sounds on people’s lips, even when their mouths are covered. While the results of lip-reading with radar signals are slightly more accurate, the Wi-Fi signals also demonstrated impressive accuracy,” Dr. Abbasi continues.
