[Ffmpeg-devel] [PATCH] VC1 init fail and crash

[Reimar Döffinger]

Hi,
attached patches fix two minor problems:
1) vc1 (not wmv3) decoder init fails due to a missing case in h263dec
2) decoder crashes since EMU_EDGE is set for the AVCodecContext but not
for MpegEncContext. Feel free to move this initialization someplace
else...

Greetings,
Reimar D?ffinger

P.S. I did test with the HDTV Alexander trailer, I frames decode fine,
everything else is as expected a mess.
-------------- next part --------------
Index: libavcodec/h263dec.c
===================================================================
--- libavcodec/h263dec.c	(revision 5535)
+++ libavcodec/h263dec.c	(working copy)
@@ -85,6 +85,7 @@
         s->h263_pred = 1;
         s->msmpeg4_version=5;
         break;
+    case CODEC_ID_VC1:
     case CODEC_ID_WMV3:
         s->h263_msmpeg4 = 1;
         s->h263_pred = 1;
-------------- next part --------------
Index: libavcodec/vc1.c
===================================================================
--- libavcodec/vc1.c	(revision 5535)
+++ libavcodec/vc1.c	(working copy)
@@ -2529,6 +2529,7 @@
     }
     avctx->has_b_frames= !!(avctx->max_b_frames);
 
+    s->flags |= CODEC_FLAG_EMU_EDGE;
     s->mb_width = (avctx->coded_width+15)>>4;
     s->mb_height = (avctx->coded_height+15)>>4;