Current software libraries for AR, the ones programmers are using to speed up their work, support the creation of a layer where one can insert artefact, digital entities, to overlay of real objects captured by, as an example, the camera of your smartphone.
You point your smartphone to a building and you see words becoming overlaid on it providing indication on the kind of shops you can find there, point to a restaurant menu in Japanese and your smartphone overlays the English translation, In many situations this works just fine. In others… no!
To transform the overlay of artefacts into actually placing them in the real world you need (the software needs) to understand the real world in 3D. Like in the photos above, the software needs to understand that there is a couch, a 3D object that can hide part of the cat. Indeed, if you look at the two photos, the first one feels strange, looks like the cat is flat, may be jumping from the couch. On the contrary the second one feel natural, the cat is partly hidden behind the couch.
This is now possible thanks to a new software released by Google as part if their ARCore API supporting augmented reality, called ARCore Depth API . You can see a few animations in the clip below, explaining the technology.
In order to be able to place an artefact in a way that makes sense, i.e. takes into account the physical objects in the real world, the software has to identify the objects and create a 3D map on their position. This is quite complex. I haven’t had the opportunity of playing with the new software so, as an example, I do not know if the software is able to detect object transparency, like a glass vase and alters the artefact behind the vase to take into account its optical properties, nor I know if the software can understand that you cannot squeeze a cat between a couch and the wall if the space between the two is just 2cm. I suspect that real life situations make the interplay of artefacts with the existing objects lay-out quite complex (like managing shades and reflection that lead to a change in the appearance of the artefact.
Nevertheless, this is a quite significant evolution bringing AR a bit closer to be perceived as a seamless part of reality.