Home / Blog / Watching in the brain … what you are watching

Watching in the brain … what you are watching

Watching in near-real-time what the brain sees. Visual information generated by a video (a) is processed in a cascade from the retina through the thalamus (LGN area) to several levels of the visual cortex (b), detected from fMRI activity patterns (c) and recorded. A powerful deep-learning technique (d) then models this detected cortical visual processing. This model can then be used to reverse this process, reconstructing the original videos. Credit: Haiguang Wen et al./Cerebral Cortex

What seemed pure science fiction just 5 years ago is now becoming a matter of perfecting technology to make it better: seeing what you are seeing by looking at your brain activity.

Researchers at Purdue University have demonstrated “mind reading” decoding technology.  Three volunteers’ brains were monitored using fMRI -functional Magnetic Resonance Imaging- as they were shown over 900 video clips for a total of 11.5 hours. The resulting data were analysed by a program that was able to extract the “images” seen by the volunteers (look at the video). The program used Convolutional Neural Networks, CNN, and was able to identify scenes, persons and faces. These “recognitions” were used to recreate the images that were seen.

Obviously what has been recreated is pretty different from what was actually seen, in terms of definition but it was still possible to match the various scenes, like observing two persons moving, identifying a face and so on.

Now, this is an amazing results. It proves that it is possible to “read the mind”. It doesn’t matter that it requires a gigantic sensor (the fMRI machine) nor that the reading in sketchy. From now on the goal is to make it better, just yesterday we were wondering is such a thing would ever be possible. No more. we know that it is possible.


About Roberto Saracco

Roberto Saracco fell in love with technology and its implications long time ago. His background is in math and computer science. Until April 2017 he led the EIT Digital Italian Node and then was head of the Industrial Doctoral School of EIT Digital up to September 2018. Previously, up to December 2011 he was the Director of the Telecom Italia Future Centre in Venice, looking at the interplay of technology evolution, economics and society. At the turn of the century he led a World Bank-Infodev project to stimulate entrepreneurship in Latin America. He is a senior member of IEEE where he leads the Industry Advisory Board within the Future Directions Committee and co-chairs the Digital Reality Initiative. He teaches a Master course on Technology Forecasting and Market impact at the University of Trento. He has published over 100 papers in journals and magazines and 14 books.