Artificial intelligence is becoming a powerful tool that can be applied to a variety of context. In image processing it is used for a better rendering of the data captured by the digital sensor but it can also be used to work on an image that has been captured with older technologies. I found this example created by researchers at the Tong University particularly interesting.
If you are in hurry, just look at the two video clips below. The first one is the digital representation of the original clip created by the Lumiere brothers, the famous “arrival of a train at the Ciotat station” (I am no movie expert but it seems that the Lumiere brothers had a special interest for filming trains…). The second one is a “re-mastering” of the clip using artificial intelligence to create a movie at 4k resolution. Impressive!
The researchers have applied a technology called Depth Aware Video Frame Interpolation -DAVFI. Basically, using artificial intelligence the software looks into a frame (a photogram) and extract the “meaning” of the objects included in the frame. This of course requires the capability to single out the individual objects (something trivial for us but that proved extremely difficult for computers and still now requires quite a lot of processing -think of Captcha when you are asked to identify cars in a set of images…) first and then to understand what these objects behaviour could be.
Then the software analyses the following frame and look at the differences. At that point it can work on creating additional resolution in each frame making the image crisper and can create additional frames making the vision smoother. Look at the clips. There are people and of course there is the train. In creating new frames the software needs to know where are the persons and where is the train (or the cart) because when looking at the difference in consecutive frames it will detect movement in all objects. To render this movement in a natural way the software needs to know that people move differently than trains or carts and therefore should create a movement that fits the natural behaviour of each object. If you look at just the second clip, the artificial one, it looks like a capturing of a real scene, yet it is a complete artefact. An artefact that looks better than the real thing!
The original of the clip:
The clip transformed using Artificial Intelligence: