[FFmpeg-devel] [PATCH] avcodec/rv10: check size of s->mb_width * s->mb_height

[Andreas Cadhalpun]

Hi,

encoding large frames with rv10 currently results in an assertion, because the 
number of macro blocks doesn't fit into 12 bits.

Attached patch avoids this by just not trying to write the macro block number in 
that case. The decoder can figure it out on it's own:
     /* if multiple packets per frame are sent, the position at which
      * to display the macroblocks is coded here */

     mb_xy = s->mb_x + s->mb_y * s->mb_width;
     if (show_bits(&s->gb, 12) == 0 || (mb_xy && mb_xy < s->mb_num)) {
         s->mb_x  = get_bits(&s->gb, 6); /* mb_x */
         s->mb_y  = get_bits(&s->gb, 6); /* mb_y */
         mb_count = get_bits(&s->gb, 12);
     } else {
         s->mb_x  = 0;
         s->mb_y  = 0;
         mb_count = s->mb_width * s->mb_height;
     }

The other option would be to abort encoding with an error, but 
ff_rv20_encode_picture_header currently returns void.

Best regards,
Andreas
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-avcodec-rv10-check-size-of-s-mb_width-s-mb_height.patch
Type: text/x-diff
Size: 1192 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20150302/91111a9e/attachment.bin>