Home / Blog / Getting robots to understand what they see …

Getting robots to understand what they see …

Lucas Manuelli uses the DON system and Kuka robot to grasp a cup. Credit: MIT

It is so easy for us to look around, see what is out there  and make sense out of it. This happens every single moment, and we are able to make sense also of things we are seeing for the very first time (well, 99.99% of the time!).

Our brain is also very good in creating abstractions and use those abstractions to identify new “things” that would fit that abstraction (by the way, this is why we look at clouds and “see” a face, a dog, a hammer: the brain find in the abstraction of that cloud a match with the abstraction of a face….).

This is not so for computers. They can very well distinguish two objects because one is one mm larger than the other or the color hue is a tad different (something our brain would not notice -that’s why we play “find the 7 differences…”).
Not so anymore! A team of researchers at CSAIL, MIT, has equipped a robot with a computer vision system with the capability to self learn objects characteristics, create abstractions and use them to identify new objects.

The trick is done by … DON: Dense Object Nets. When looking at an object DON creates a dense representation made of points in a tri-dimensional space. These points create a model that will be used to abstract some visual characteristics. As you can see in the clip below, DON is able to form the abstract concept of “shoe” and of the tongue that each shoe has. You can place very different kinds of shoes but since DON has got the meaning of “shoe” it will recognise any kind of shoe, look for its tongue and pick it up if asked to do so. Since the model abstraction is tri-dimensional the robot can recognise a shoe even if it is upside down or laying on a side.

This is a very important evolution, on the path of creating systems that can become autonomous, self teaching. There are many further steps needed but at least this one has been taken.

About Roberto Saracco

Roberto Saracco fell in love with technology and its implications long time ago. His background is in math and computer science. Until April 2017 he led the EIT Digital Italian Node and then was head of the Industrial Doctoral School of EIT Digital up to September 2018. Previously, up to December 2011 he was the Director of the Telecom Italia Future Centre in Venice, looking at the interplay of technology evolution, economics and society. At the turn of the century he led a World Bank-Infodev project to stimulate entrepreneurship in Latin America. He is a senior member of IEEE where he leads the Industry Advisory Board within the Future Directions Committee and co-chairs the Digital Reality Initiative. He teaches a Master course on Technology Forecasting and Market impact at the University of Trento. He has published over 100 papers in journals and magazines and 14 books.