Articles Technology Blog News Forum Company
When super resolution doesn't work
If you've already read how super-resolution works, you know that it uses subpixel accurate motion compensation to find similar areas in neighbor frames in order to intelligenlty merge them combining the details. There are cases when this can't be done successfully or cannot bring new details.

Stills

First of all, if an object doesn't move or change at all, if it's identical in several frames, then there's no additional information to extract. The quality will be as of simple spatial upsize.

Very fast motion or change

Opposite extreme is very fast motion or change. If an object moves too fast, it's difficult to track its motion, and often it becomes blurred due to two reasons. One reason is motion blur - when camera's exposure is set so that object moves a notable distance during shooting of one frame. The other reason is compression. If the video is compressed with a method that uses delta-frames (like MPEG1-2-4, WMV and many others), intense motion creates a lot of differences between frames so the codec quantizes data stronger to fit into required bitrate, so more details are lost on the way.

Just too strong compression

If your video is compressed to a low bitrate, in many cases this is very bad for super-resolution. There are two types of lossy video codecs: those that use delta-frames and those that use only key-frames.

If the video is compressed heavily by a key-frame-only codec like MJPEG (DV is also such codec, but DV video always has high enough bitrate), then each frame is compressed independently and a lot of details are lost in each frame. This usually leads to an artefact called "blocking" (visible squares in the picture), you can see it in strongly compressed JPEG images. When an object moves in the video, the blocks don't move, so the object is changing a lot during this motion making it impossible to accurately track its motion and get some details. What you can get from other frames is their blocking artefacts, making picture even worse.

If the video is heavily compressed by a codec with delta-frames (most common codecs), then the codec performs motion compensation, gets the difference between frames and quantizes it losing all details and leaving only major differences. When we try to apply super-resolution, it performs similar motion compensation to use differences between frames and it finds almost no details, only gross changes that cannot be used to improve the frame. One practical conclusion is that there's no sense in trying to use Video Enhancer on video shot with mobile phones.

Other articles...