Industrial Electronics

Video: Robot arm can move objects it has never seen before

19 March 2019

MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) has developed a new system that allows robots to do pick-and-place tasks — from hanging mugs in a kitchen to putting shoes on shelves — without having ever seen the object it is interacting with.

Factory robots can pick up the same object over and over again, but generally have trouble understanding a wide range of shapes and sizes. Moreover, these robots must identify the object they are going to pick up before it can take action. Yet, robots are becoming more sophisticated in what they can pick up or what they can do in the industrial and consumer space.

MIT used an approach that enabled robots to manipulate objects using keypoints from the two most common approaches to picking up objects, which are “pose-based” systems that estimate an object’s position and orientation, and general grasping algorithms that are strongly geometry-based. However, these methods have problems because pose estimators often do not work with objects of significantly different shapes, while grasping approaches have no notion of pose and cannot place objects with much subtlety.

MIT’s keypoint affordance manipulation (KPAM) system instead detects a collection of coordinates on an object. These coordinates provide all the information the robot needs to determine what to do with the object and can handle variations of any objects.

“Understanding just a little bit more about the object — the location of a few key points — is enough to enable a wide range of useful manipulation tasks,” said Russ Tedrake, MIT professor. “And this particular representation works magically well with today’s state-of-the-art machine learning perception and planning algorithms.”

The next step is to get the system to perform tasks with even greater generalizations, like unloading the dishwasher or wiping down the counters of a kitchen. MIT said KPAM could also be incorporated into larger manipulation systems used in factories or other environments.

To contact the author of this article, email

Powered by CR4, the Engineering Community

Discussion – 0 comments

By posting a comment you confirm that you have read and accept our Posting Rules and Terms of Use.
Engineering Newsletter Signup
Get the GlobalSpec
Stay up to date on:
Features the top stories, latest news, charts, insights and more on the end-to-end electronics value chain.
Weekly Newsletter
Get news, research, and analysis
on the Electronics industry in your
inbox every week - for FREE
Sign up for our FREE eNewsletter
Find Free Electronics Datasheets