Articles Technology Blog News Company
What is super-resolution?
Super-resolution (also spelled as super resolution and superresolution) is a term for a set of methods of upscaling video or images. Terms such as "upscale", "upsize", "up-convert" and "uprez" also describe increase of resolution in either image processing or video editing. Most super-resolution techniques are based on the same idea: using information from several different images to create one upsized image. Algorithms try to extract details from every image in a sequence to reconstruct other frames. This multiframe approach differs significantly from sophisticated image (single frame) upsizing methods which try to synthesize artificial details.

"Super-resolution" is not a marketing buzzword, it's a mathematical term used by scientists. First work on this topic was published in 1984 [1] and the term "Super-resolution" itself appeared at around 1990 [2].

Methods usually discussed in scientific literature try to reproduce process of losing quality when shooting video with low-res cameras and then solve inverse problem of finding video which being downsized with that process gives us known low-res material. This is an ill-posed inverse problem which doesn't have straightforward solution and usually requires some additional regularization (applying some artificial constraints) and huge CPU time to check an awful lot of variants. Modern practical methods are usually simpler but still effective.

Super-resolution (SR) works effectively when several low resolution images contain slightly different perspectives of the same object. Then total information about the object exceeds information from any single frame. The best case is when an object moves in the video. Motion detection and tracking are then employed to benefit upscaling. If an object doesn't move at all and is identical in all frames, no extra information can be collected. If it moves or transforms too fast then it looks very different in different frames and it's too hard to use information from one frame in reconstructing the other.

Super-resolution is used now in two tasks: extracting single frames from video (or a set of images/photos) with high quality, or upsizing the whole video. SR methods are usually based on two important algorithms: high quality spatial (in-frame) upscaling, and motion compensation for finding corresponding areas in neighbor frames. Many practical implementations of super-resolution software upscale original material two times. If we need to upsize it four times, we usually apply SR twice (this can be done internally in implementation).

Let's consider a real example of upsizing video 4 times with super-resolution algorithm. This example was made using Video Enhancer. First, we have original low-resolution sequential frames of video (or consider them a sequence of slightly different low quality photos):

low res video frames

The same frames zoomed 4 times:

low res video frames zoomed 4x

Step 1. Upsize all frames 2 times using good quality interpolation, in this case Lanczos3 method (note: our current implementation doesn't use Lancsoz anymore, it uses a similar method but with a different kernel):

frames after Lanczos3 resize, zoomed 2x

Step 2. Using sub-pixel accurate motion compensation, find similar areas in neighbor frames and intelligently merge frames to combine information (and in some cases reduce noise):

frames after first pass of super-resolution, zoomed 2x

Now we have our video upsized 2 times using super-resolution. We repeat this process to get 4x enlargement.

Step 3. Spatial interpolation again:

frames after SR and Lanczos3

Finally, Step 4. Combine neighbor frames to get more details using motion compensation:

frames after 2 passes of super-resolution

Now let's compare it with what we originally had:

original low res frames zoomed 4x
Quite amazing, isn't it? But it's no magic, only a bit of math. ;)

One may ask: was that motion compensated merging really required? Weren't spatial interpolated frames good enough? The answer will become obvious if you look at comparison of different upsize methods. Shortly, spatial interpolation methods don't even come closer to a good super-resolution in terms of details and objective metrics.

Note that super-resolution cannot always provide excellent results. Read in the Infognition knowledge base when super-resolution doesn't work.

Papers:

1. R. Y. Tsai and T. S. Huang, "Multiframe image restoration and registration," in Advances in Computer Vision and Image Processing, vol. 1, chapter 7, pp. 317-339, JAI Press, Greenwich, Conn, USA, 1984.

2. M. Irani and S. Peleg. 1991, "Super Resolution From Image Sequences" ICPR, 2:115--120, June 1990.

Try this yourself with Video Enhancer

Get Super Resolution plugin for VirtualDub
Get Super Resolution plugin for Adobe After Effects and Premier Pro
Get Super Resolution plugin for AviSynth
OFX plugin coming soon.

Other articles...