Home / Blog / Moving your fingers in the cyberspace

Moving your fingers in the cyberspace

Next generation VR finger tracker. Image credit: Sony

Sony researchers were due to present a paper at CHI 2020,  scheduled at the end of April in Honolulu, Hawaii, but because of the epidemic the conference was cancelled. Their paper, however, is available. And it is a paper worth reading.

The title, “Evaluation of Machine Learning Techniques for hand pose estimation on handheld device with proximity sensors” says quite a bit and if you have no time to dig into the paper you may want to take a look at the video explaining their research.

Basically they want to be able to sense the movement of a hand and its fingers to replicate this in the cyberspace. Now picking up with accuracy the movements of fingers is a complex task.  Our human hand (including fingers) have 27 degrees of freedom. meaning that any of 27 parameters can change independently one from the other. Having sensors to detect all the possible positions is not only complex, it is expensive and results in a bulky apparatus.

Here comes the solution proposed by the researchers: train a software, using machine learning, to get the feeling on how a human hand can move (this has been done by tracking, optically, the movements of the hand and finger with cameras detecting reference points placed on the various moving parts of the hand) and then have the software make sense from the signals generated by proximity sensors to recreate the actual movements.

The proximity sensors have been embedded in a sort of bar sticking out from a base, used as reference plane (see the figure).
As shown in the video this setting allows the software to mimic with high fidelity the movements of the hand making it possible to replicate them in the cyberspace. You see objects int he cyberspace using VR goggles and by moving your hand (the real one) around the stick you can control the virtual hand with the precision needed to pick up virtual objects as if you were actually using your hand.

The result is really impressive. It opens the door to a more seamless interaction with the cyberspace that will probably be used in games on a future Playstation but that can also find application in the remote control of Robots.

It is another step towards the fading boundary between the physical and the virtual reality taking us into the new world of digital reality.

About Roberto Saracco

Roberto Saracco fell in love with technology and its implications long time ago. His background is in math and computer science. Until April 2017 he led the EIT Digital Italian Node and then was head of the Industrial Doctoral School of EIT Digital up to September 2018. Previously, up to December 2011 he was the Director of the Telecom Italia Future Centre in Venice, looking at the interplay of technology evolution, economics and society. At the turn of the century he led a World Bank-Infodev project to stimulate entrepreneurship in Latin America. He is a senior member of IEEE where he leads the Industry Advisory Board within the Future Directions Committee and co-chairs the Digital Reality Initiative. He teaches a Master course on Technology Forecasting and Market impact at the University of Trento. He has published over 100 papers in journals and magazines and 14 books.