Sony researchers were due to present a paper at CHI 2020, scheduled at the end of April in Honolulu, Hawaii, but because of the epidemic the conference was cancelled. Their paper, however, is available. And it is a paper worth reading.
The title, “Evaluation of Machine Learning Techniques for hand pose estimation on handheld device with proximity sensors” says quite a bit and if you have no time to dig into the paper you may want to take a look at the video explaining their research.
Basically they want to be able to sense the movement of a hand and its fingers to replicate this in the cyberspace. Now picking up with accuracy the movements of fingers is a complex task. Our human hand (including fingers) have 27 degrees of freedom. meaning that any of 27 parameters can change independently one from the other. Having sensors to detect all the possible positions is not only complex, it is expensive and results in a bulky apparatus.
Here comes the solution proposed by the researchers: train a software, using machine learning, to get the feeling on how a human hand can move (this has been done by tracking, optically, the movements of the hand and finger with cameras detecting reference points placed on the various moving parts of the hand) and then have the software make sense from the signals generated by proximity sensors to recreate the actual movements.
The proximity sensors have been embedded in a sort of bar sticking out from a base, used as reference plane (see the figure).
As shown in the video this setting allows the software to mimic with high fidelity the movements of the hand making it possible to replicate them in the cyberspace. You see objects int he cyberspace using VR goggles and by moving your hand (the real one) around the stick you can control the virtual hand with the precision needed to pick up virtual objects as if you were actually using your hand.
The result is really impressive. It opens the door to a more seamless interaction with the cyberspace that will probably be used in games on a future Playstation but that can also find application in the remote control of Robots.
It is another step towards the fading boundary between the physical and the virtual reality taking us into the new world of digital reality.