blog tags:

About:

I'm Dmitry Popov,
lead developer and director of Infognition.

Known in the interwebs as Dee Mon since 1997. You could see me as thedeemon on reddit or LiveJournal.

RSS
Articles Technology Blog News Forum Company
Blog
One-pass deshaker
November 25, 2015

This autumn, by request of our client, we developed a video deshaking method that works in a single pass, in streaming mode. I know there is a very nice Deshaker filter for VirtualDub but it works in two passes: first it collects motion data, then plans for whole movie and in the second pass performs actual changes to the video. When your video processing pipeline is quite heavy, loaded with many filters or something computationally expensive, doing the work twice or saving it into a huge lossless file and then processing twice with Deshaker can be extremely time-consuming. So, there might be a need for something quicker, working in one pass.

Here are a couple of examples showing our own deshaker in action (before/after) :

The second clip is cut from a recent earthquake movie, our deshaker is able to fight the earthquakes rather well. ;) When deshaking applied, we can see how that girl actively shakes the boy to simulate an earthquake. This is something not so apparent in the original footage.

This is how the deshaker thinks about it:

From motion analysis it takes the main motion vector and uses it to model global motion, drawn in blue (only vertical motion is shown here). By looking at motion of up to 50 frames (using a sliding window) it chooses a trajectory (green) which mostly consists of straight lines, as if the camera mostly stays still or moves steadily without acceleration. Then it shifts each frame by the difference between new and original trajectories. Which means it's only good for translational motion as in examples above, it does not currently fix any in-frame distortions nor rotation. It also zooms a bit (so you don't see black borders near the edges), and the zoom factor is constant, because as Deshaker for VirtualDub demonstrated, variable zoom factor can be really sickening.

Currently this one-pass deshaker is a part of our SDK (among with super resolution resizer, auto-brightness and film dirt cleaner filters) that we offer to other developers. Internally it uses motion information collected by either super resolution or dirt cleaner filters. We're certainly going to include the deshaker in Video Enhancer 2.0 coming soon.

There is also a possibility to release it as a VirtualDub plugin or a plugin for some other hosts. Please let me know if you'd like to see it as a plugin.