Industrial Electronics

Video: Using a camera and shadows to give robots a sense of touch

09 February 2021

Researchers at Cornell University have created a low-cost method for soft, deformable robots that can detect a range of physical interactions using shadow movements and cameras.

The interactions range from pats to punches to hugs and instead of relying on actual touch, a USB camera located inside the robot captures the shadow movements of hand gestures on the robot’s “skin” and classifies them with machine learning.

“Touch is such an important mode of communication for most organisms, but it has been virtually absent from human-robot interaction,” said Guy Hoffman, associate professor and Mills Family Faculty Fellow in the Sibley School of Mechanical and Aerospace Engineering. “One of the reasons is that full-body touch used to require a massive number of sensors and was therefore not practical to implement. This research offers a low-cost alternative.”

This type of robot would need to communicate with humans in extreme conditions and environments such as a robot leading someone down a noisy, smoke-filed corridor by detecting the pressure of the person’s hand. Rather than installing many contact sensors, the team decided to use sight, thereby avoiding the complexity of wiring.

The camera inside the robot can infer how the person is touching it and what the person’s intent is by looking at shadow images. Currently, most robots, especially social robots, are not able to detect touch gestures.

How they did it

The prototype robot includes a soft inflatable bladder of nylon skin stretched around the skeleton and mounted to a mobile base. The USB camera is under the robot’s skin and connects to a laptop. Then a neural-network-based algorithm records data to distinguish between six tough gestures with an accuracy of 87.5% to 96%, depending on the lighting.

The robot can be programmed to respond to touches and gestures and the skin could be used as an interactive screen. Future versions of the robot could be trained to recognize a wider vocabulary of interactions depending on the task assigned.

The ShadowSense technology could be used in other materials as well, such as balloons, turning them into touch-sensitive devices.

“While the technology has certain limitations, for example requiring a line of sight from the camera to the robot’s skin, these constraints could actually spark a new approach to social robot design that would support a visual touch sensor like the one we proposed,” Hoffman said. “In the future, we would like to experiment with using optical devices such as lenses and mirrors to enable additional form factors.”

Researchers said the robot can detect what a person is doing without actual touch, giving it a physical filter and protection as well as providing psychological comfort, while simultaneously understanding a person’s movements and moods.

The full research can be found in the journal Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies.

To contact the author of this article, email

Powered by CR4, the Engineering Community

Discussion – 0 comments

By posting a comment you confirm that you have read and accept our Posting Rules and Terms of Use.
Engineering Newsletter Signup
Get the Engineering360
Stay up to date on:
Features the top stories, latest news, charts, insights and more on the end-to-end electronics value chain.
Weekly Newsletter
Get news, research, and analysis
on the Electronics industry in your
inbox every week - for FREE
Sign up for our FREE eNewsletter
Find Free Electronics Datasheets