Home / Blog / When the artefact is better than the original …

When the artefact is better than the original …

Depth Aware Video Frame Interpolation is a technology that using artificial intelligence singles out the various components of an image by evaluating consecutive frames of a video and creating a depth matrix. Image credit: Wenbo Bao et Al, Tong University

Artificial intelligence is becoming a powerful tool that can be applied to a variety of context. In image processing it is used for a better rendering of the data captured by the digital sensor but it can also be used to work on an image that has been captured with older technologies. I found this example created by researchers at the Tong University particularly interesting.

If you are in hurry, just look at the two video clips below. The first one is the digital representation of the original clip created by the Lumiere brothers, the famous “arrival of a train at the Ciotat station” (I am no movie expert but it seems that the Lumiere brothers had a special interest for filming trains…). The second one is a “re-mastering” of the clip using artificial intelligence to create a movie at 4k resolution.  Impressive!

The researchers have applied a technology called Depth Aware Video Frame Interpolation -DAVFI. Basically, using artificial intelligence the software looks into a frame (a photogram) and extract the “meaning” of the objects included in the frame. This of course requires the capability to single out the individual objects (something trivial for us but that proved extremely difficult for computers and still now requires quite a lot of processing -think of Captcha when you are asked to identify cars in a set of images…) first and then to understand what these objects behaviour could be.

Then the software analyses the following frame and look at the differences. At that point it can work on creating additional resolution in each frame making the image crisper and can create additional frames making the vision smoother. Look at the clips. There are people and of course there is the train. In creating new frames the software needs to know where are the persons and where is the train (or the cart) because when looking at the difference in consecutive frames it will detect movement in all objects. To render this movement in a natural way the software needs to know that people move differently than trains or carts and therefore should create a movement that fits the natural behaviour of each object. If you look at just the second clip, the artificial one, it looks like a capturing of a real scene, yet it is a complete artefact. An artefact that looks better than the real thing!

The original of the clip:

The clip transformed using Artificial Intelligence:





About Roberto Saracco

Roberto Saracco fell in love with technology and its implications long time ago. His background is in math and computer science. Until April 2017 he led the EIT Digital Italian Node and then was head of the Industrial Doctoral School of EIT Digital up to September 2018. Previously, up to December 2011 he was the Director of the Telecom Italia Future Centre in Venice, looking at the interplay of technology evolution, economics and society. At the turn of the century he led a World Bank-Infodev project to stimulate entrepreneurship in Latin America. He is a senior member of IEEE where he leads the New Initiative Committee and co-chairs the Digital Reality Initiative. He is a member of the IEEE in 2050 Ad Hoc Committee. He teaches a Master course on Technology Forecasting and Market impact at the University of Trento. He has published over 100 papers in journals and magazines and 14 books.