It is now several decades that video games have acquainted us with synthetic reality. Imaginary landscapes and ambient are recreated by computer graphics along with sounds that match the objects and the ongoing actions providing a “credible” reality. Actually, the first synthetic environment screamed “fake” at all times and in spite of the amazing progress in computer graphics (partly due to software and a lot to huge increase in processing power provided by GPUs) these synthetic media don’t fool us into believing it is the real thing.
Something that has happened in these last years is the construction of the synthetic reality/media starting from real world data. As an example, the MS Flight Simulator 2020 (watch the clip) provides amazingly nice scenery (but they still don’t look “real”). These are created in the Azure cloud by AI software making use of some 2 PetaBytes of photographies that have been taken from satellite, covering the whole planet. The AI software stitches the photos together and modifies the result based on the point of view (and the altitude of the aircraft). As I said, the result is impressive, at some points your brain may be fooled into believing it is real, but most of the time it will look somehow fake (it might be difficult to describe why it looks fake, but it does…). However, and this is the important piece of information, the rendering might look fake but the mirroring of the real world is accurate since it starts from real data.
Similarly, look at the image in this post: it is a snapshot of the rendering of a suburb of Seattle created by Amazon to test the behaviour of their autonomous robot to deliver groceries at home. The rendering has been made to deliver a 1 cm accuracy (so that all possible obstacles on the path of the robot can be simulated and tested). The robot itself, shown in the photo, is a digital replica of the real robot and engineers at Amazon study its behaviour to fine tune it. Actually, they are using the digital twin of the physical robot, the same one that will be used to monitor the actual robot as it delivers goods.
The FTI’s report foresees an increase in synthetic media in this decade, with an increased use of real data for rendering. The evolution will continue in making the rendering more “real” (with attention given to shadows – a crucial hint used by our brain in decoding the world) but the most important part of the evolution will be in the use of real world data, sometimes harvested through entities digital twins.
The main technologies that will make possible the improvement of synthetic media are:
- speech syntheses, already well advanced, will make use of frequencies and intonation of a specific person to provide a perfect copy of that voice (further increasing the deepfake issues);
- custom voice modulation based on the emotional content of the discourse, using AI
- deep behaviour and predictive machine vision (also fostered by advances in autonomous driving) where the software is capable to predict the next behaviour of the entities in an ambient, including how a person would react to a specific situation (notice that this can be used in the reverse, that is to generate a situation leading to a specific -desired- behaviour, a marketeers/advertisers dream)
- generative algorithms for voice and movement, allowing their transfer from one person in a video to another in a different video (deepfake again!). The Israeli start up D-ID applies this technology to deliver several impressive services (watch the second clip visualising their living portraits)
- automatic generation of synthetic environment based on real world data (as in the afore mentioned MS Flight Simulator)
- simulation of human behaviour through a synthetic character. Samsung, as an example, is proposing the creation of artificial humans you can interact with (look at NEON). This can be used as a companion, as a character in a synthetic media. It could also be used to simulate an interaction: first you “embed” the characteristics of a person into the artificial character (as they can be derived by AI software analysing the behaviour of that person through video clips, posts, messages…) and then you can simulate various interactions to understand how that person would be likely to react letting you practice and finely tuning the approach. It could also be used as a personal digital twin, automatically created by AI that you can insert in a synthetic ambient and study how it (you) would react to a variety of situations. It looks like science fiction and yet it is the bread and butter of this decade’s research.
The whole area of synthetic media is fraught with social issues that go beyond the deepfake problem.