[Libav-user] Possible bug in msmpeg4.c

[Tom Vercauteren]

On Thu, Feb 9, 2012 at 19:12, Tom Vercauteren <tom.vercauteren at gmail.com> wrote:
> Hi Folks,
>
> First of all, thanks for the nice piece of software that has been
> serving me quite nicely. I have read that bug reports should be made
> with the bug tracker but I first wanted to confirm an issue that I am
> encountering. I am indeed not really familiar with the ffmpeg code.
>
> I am calling ffmpeg from c++ code to encode MSMPEG4V3 video from
> grabbed images. My program segfaulted within get_rl_index. This was
> caused because of the a previous issue in ff_msmpeg4_encode_block.
>
> ---------------
> void ff_msmpeg4_encode_block(MpegEncContext * s, DCTELEM * block, int n)
> {
>    int level, run, last, i, j, last_index;
>    int last_non_zero, sign, slevel;
>    int code, run_diff, dc_pred_dir;
>    const RLTable *rl;
>    const uint8_t *scantable;
>
>    if (s->mb_intra) {
>        msmpeg4_encode_dc(s, block[0], n, &dc_pred_dir);
>        i = 1;
>        if (n < 4) {
>               rl = &rl_table[s->rl_table_index];
> ----------------
> The problem arises here as s->rl_table_index is equal to -1. I
> attempted a very quick and dirty workaround without actually trying to
> understand the code. It is shown below. It seems to actually provide
> usable results but, as I said, I have no clue whether this really
> makes sense.
>
> ----------------
> void ff_msmpeg4_encode_block(MpegEncContext * s, DCTELEM * block, int n)
> {
>    int level, run, last, i, j, last_index;
>    int last_non_zero, sign, slevel;
>    int code, run_diff, dc_pred_dir;
>    const RLTable *rl;
>    const uint8_t *scantable;
>
>    if (s->mb_intra) {
>        msmpeg4_encode_dc(s, block[0], n, &dc_pred_dir);
>        i = 1;
>        if (n < 4) {
>            if( !( s->rl_table_index>0 && s->rl_table_index<NB_RL_TABLES ) ){
>               rl = &rl_table[1];
>            }
>            else {
>               rl = &rl_table[s->rl_table_index];
>            }
> ----------------
>
> Can this problem come from my bad usage of ffmpeg or is this an actual
> bug that I should file? Also, would someone suggest a better
> workaround? I know that this is not a self-contained test case but I
> hope that it provides sufficient information to help.

For the record, I have been able to repoduce the bug with the ffmpeg
executable. I have therefore opened a bug report  that contains the
test case:
https://ffmpeg.org/trac/ffmpeg/ticket/990

Tom