[FFmpeg-devel] [PATCH] [ISSUE 798] VP3/Theora: prevent stack corruption

[Alex Converse]

Hi,

The attached patch fixes the crash associated with issue 798. (Though the
real issue appears to be with the demuxer).

Basically a custom coding method doesn't get get set for one index value, so
when that index occurs a garbage value is used. This garbage value
propagates into current_frame_type through the compatible compatible_frame
table. current_frame_type is then used to index a write into last_dc causing
stack corruption.

The behavior for an unset custom coding method is not specified by the
theora specification, so I chose to copy libtheora's behavior and assign it
to MODE_INTER_NO_MV.

Regards,
Alex Converse
-------------- next part --------------
A non-text attachment was scrubbed...
Name: issue798-crash.diff
Type: text/x-diff
Size: 489 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20090129/ed54f89d/attachment.diff>