[FFmpeg-devel] [HACK] fix CAVS decoder crashes

[Stefan Gehrer]

On 12/12/2010 05:04 PM, Reimar D?ffinger wrote:
> Hello,
> I have the suspicion this decoder needs heavy fuzzing testing.
> Anyway, trying to play http://samples.mplayerhq.hu/AVS/AVSFileFormat/AVSFileFormat.es
> results in crashes which below hack "fixes".
> Index: libavcodec/cavs.h
> ===================================================================
> --- libavcodec/cavs.h   (revision 25928)
> +++ libavcodec/cavs.h   (working copy)
> @@ -242,6 +242,7 @@
>   extern const cavs_vector ff_cavs_dir_mv;
>
>   static inline void modify_pred(const int_fast8_t *mod_table, int *mode) {
> +    if (*mode<  0) *mode = 0;
>       *mode = mod_table[*mode];
>       if(*mode<  0) {
>           av_log(NULL, AV_LOG_ERROR, "Illegal intra prediction mode\n");

Not sure how the mode can be negative here (also for the change at the
bottom), I will have a look soon.

> Index: libavcodec/cavsdec.c
> ===================================================================
> --- libavcodec/cavsdec.c        (revision 25928)
> +++ libavcodec/cavsdec.c        (working copy)
> @@ -122,7 +122,7 @@
>
>       for(i=0;i<65;i++) {
>           level_code = get_ue_code(gb,r->golomb_order);
> -        if(level_code>= ESCAPE_CODE) {
> +        if(level_code>= ESCAPE_CODE || level_code<  0) {
>               run = ((level_code - ESCAPE_CODE)>>  1) + 1;
>               esc_code = get_ue_code(gb,esc_golomb_order);
>               level = esc_code + (run>  r->max_run ? 1 : r->level_add[run]);

get_ue_golomb() from golomb.h and get_ue_code() should probably be fixed
to return unsigned values, the negative values don't have any meaning.

> @@ -234,7 +234,7 @@
>       for(block=0;block<4;block++) {
>           d = h->cy + h->luma_scan[block];
>           ff_cavs_load_intra_pred_luma(h, top,&left, block);
> -        h->intra_pred_l[h->pred_mode_Y[ff_cavs_scan3x3[block]]]
> +        h->intra_pred_l[FFMAX(h->pred_mode_Y[ff_cavs_scan3x3[block]], 0)]
>               (d, top, left, h->l_stride);
>           if(h->cbp&  (1<<block))
>               decode_residual_block(h,gb,ff_cavs_intra_dec,1,h->qp,d,h->l_stride);


Stefan