[FFmpeg-devel] tkhd transformation matrix in mov is ignored except for width/height and scaling

[Daniel G. Taylor]

On Sat, 2009-10-31 at 09:53 -0700, Samuel Gendler wrote:
> I've been trying for months to get transcoding an mov via ffmpeg to
> correctly rotate a video that includes a transformation matrix in the tkhd
> node (iphone video shot in portrait, basically).  I could see that various
> patches about the matrix were discussed on this list, and I couldn't figure
> out why nothing seemed to be functioning even after said patches were
> applied. The main patch is here:
> 
> http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2008-July/049941.html
> 
> and it was corrected by this patch:
> 
> http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2009-June/071622.html
> 
> In desperation, I finally dug into the source and tried to figure out what i
> was doing wrong.  What was almost immediately apparent is that those patches
> deal only with width and height of the track as well as scaling the width
> and height.  In fact, the values in the matrix are only stored locally in
> variables on the stack in the static function that reads the tkhd node. The
> width and height are computed from the matrix and stored in a location
> accessible from outside the mov_read_tkhd() function, but the rest of the
> matrix is completely inaccessible once the function returns.
> 
> I'll admit to being fairly surprised that this is the case, since the iphone
> is an increasingly popular platform for recording video, and the default
> orientation for the phone in a users hand (vertical - portrait mode) will
> result in a video that appears rotated counterclockwise by 90 degrees when
> played via libavformat.  More importantly, any video transcoded via
> libavformat will result in a video that is rotated AND which no longer has
> the transformation matrix values to compensate stored in the tkhd, so that
> even quicktime and the iphone play the video sideways.  This is such a
> strange state of affairs so long after the release of the iphone, that I
> have to wonder if I'm reading things incorrectly.
> 
> Incidentally, a description of the matrix transformation process is here:
> 
> http://developer.apple.com/mac/library/documentation/QuickTime/RM/MovieBasics/MTEditing/K-Chapter/11MatrixFunctions.html
> 
> I'm an experienced developer, but I know next to nothing about av codecs and
> such.  I could attempt to hack in a fix, but I don't even know where to
> begin.  Storing the matrix somewhere that is accessible outside
> mov_read_tkhd() is simple enough.  Running each pixel through the transform
> is also pretty easy.  But I've got no clue where to place that code. Could
> someone maybe point me in the correct direction for where to find code that
> places each pixel in the outbound stream?
> 
> My alternative is a lame hack of the code in libavformat/movenc.c which
> writes a default identity matrix into the matrix.  I can copy a rotation
> matrix into there instead based on a commandline param, but I am loathe to
> do that.
> 
> Thanks in advance for your help.

I believe the issue is related to not having a rotation filter in FFmpeg
at the moment (correct me if I'm mistaken) so even if we have the data
stored in the mov context the best that can be done is writing the same
values to the final output.

I take a slightly different approach here and use a little script I
wrote to try and guess the rotation angle using the inverse sin/cos and
a little Python. You aren't the first to have issues with this stuff so
today I released it here:

http://github.com/danielgtaylor/qtrotate/

It's the same script I use at work to automagically detect rotation and
then I insert the MPlayer rotate filter into our processing chain so
that we can process rotated videos from customers (and we get a few from
iPhones and digicams). It's far from perfect but so far it has worked
nicely.

A lot of work has been done on libavfilter, so hopefully in the near
future that can be linked to the mov stuff in libavformat to
automatically get rotation (or ignore all that and just copy values
over, I'm not really an expert on what the best approach is there).

Take care,
-- 
Daniel G. Taylor
http://programmer-art.org/