More efficient deshake filter

[ubitux]

An interesting de-shake filter article:  http://googleresearch.blogspot.in/2012/05/video-stabilization-on-youtube.html

Our deshake filter isn't very efficient with the provided sample (at least with the default settings)

clive -f best 'http://www.youtube.com/watch?v=627MqC6E5Yo'

ffplay 'Sam and Cocoa shaky original.webm' \
    -vf 'split[a][b]; [a]pad=iw*2:ih[src]; [b]deshake[filt]; [src][filt]overlay=w'

It would be nice to improve the filter to handle such sample.

[cehoyos]

Please provide the sample, your url is 403 here.

[jbvsmo]

Sam and Cocoa shaky original (bitrate reduced to fit 2.5MB)

[cehoyos]

I don't know much about the deshake filter but I wonder if a high bitrate sample wouldn't make more sense - please consider uploading the larger sample to  http://www1.datafilehost.com/ and post the link here.

[ubitux]

Original uploaded here:  http://lucy.pkh.me/samples/Sam%20and%20Cocoa%20shaky%20original.webm

[cehoyos]

Works better here with "-vf deshake=rx=48:ry=64"

[cehoyos]

Works even better with "-cpuflags -sse2 -vf deshake=rx=63:ry=64"

[jbvsmo]

Replying to cehoyos:

Works even better with "-cpuflags -sse2 -vf deshake=rx=63:ry=64"

But it is not even close from what the proposed link shows. Also, this configuration makes it very slow to be used on realtime.

A good deshake filter should add the possibility to crop the image instead of filling it with surrounding pixels.

Try adding "crop=600:400" after the deshake filter to see a better result. This will crop in the middle while the deshake filter could manage to crop in the right places.

---
BTW, in the ffmpeg-devel list, there's a proposal to include a binding for vid.stab ( https://github.com/georgmartius/vid.stab and  http://public.hronopik.de/vid.stab/) library.

The results seems to be superior. Why can't the library be included directly in ffmpeg?

[cehoyos]

Replying to jbvsmo:

Replying to cehoyos:

Works even better with "-cpuflags -sse2 -vf deshake=rx=63:ry=64"

But it is not even close from what the proposed link shows.

Sorry, which link do you mean?

Also, this configuration makes it very slow to be used on realtime.

A good deshake filter should add the possibility to crop the image instead of filling it with surrounding pixels.

Try adding "crop=600:400" after the deshake filter to see a better result.

Of course, I don't understand the above argument...

This will crop in the middle while the deshake filter could manage to crop in the right places.

---
BTW, in the ffmpeg-devel list, there's a proposal to include a binding for vid.stab ( https://github.com/georgmartius/vid.stab and  http://public.hronopik.de/vid.stab/) library.

The results seems to be superior. Why can't the library be included directly in ffmpeg?

I'd like to know the answer as well, please ask the developer!

[jbvsmo]

Replying to cehoyos:

Sorry, which link do you mean?

The article in the ticket description:

 http://googleresearch.blogspot.in/2012/05/video-stabilization-on-youtube.html

Of course, I don't understand the above argument...

Using "-cpuflags -sse2 -vf 'deshake=rx=63:ry=64, crop=600:400'"
The crop filter will be in the middle of the video, but it may not be the optimal place to crop. The deshake filter could choose it better. That's what I'm saying

[cehoyos]

Replying to jbvsmo:

Using "-cpuflags -sse2 -vf 'deshake=rx=63:ry=64, crop=600:400'"
The crop filter will be in the middle of the video, but it may not be the optimal place to crop. The deshake filter could choose it better.

That is likely true, thank you for explaining.

[flexman]

We are working on adding video stabilization using vid.stab library ( http://public.hronopik.de/vid.stab/). The sample video can still not dealt with very well because it has massive rolling shutter which is not compensated by vid.stab yet, but I am working on it.

[cehoyos]

What's wrong with improving the existing filter?

[flexman]

Well its architecture is too simple. A single pass cannot work without the possibility for the filter to look ahead for a few frames (not supported by the AVFilter API).
Deshake only uses compensates translations no rotations, and many more small features are missing which vid.stab offers, e.g. contrast based selection of relevant parts of the frame.

[cehoyos]

An external library is simply no solution and I don't understand how the API does not work for the internal filter but works for an external one. (and why shouldn't deshake buffer some frames?)

[flexman]

Why is an external library no solution? ffmpeg relies on many external libs!
The external lib is a 2-pass version. (I also have a single pass one for transcode at the moment, but it suffers from the same problem that you cannot look into the future and thus the corrections are worse).
Buffering is not problem, but you have to produce frames for every frame you read so you cannot wait for say 10 frames before you output anything.