[Ffmpeg-devel] Re: mjpeg with trellis quant segaults ... to be expected?

[Karl H. Beckers]

Hi,
dunno if trellis quant with mjpeg makes any sense, but at any rate,
ffmpeg should treat this gracefully. The error occurs in mpegvideo.c at
line 5920 and the following patch seems to be the safe (if possibly not
the most resource conservative) approach for avoiding it.

mjpeg.c doesn't seem to set intra_ac_vlc_length on the MpegEncContext.

Cheers,

Karl.

svn diff mpegvideo.c
Index: mpegvideo.c
===================================================================
--- mpegvideo.c (Revision 6791)
+++ mpegvideo.c (Arbeitskopie)
@@ -5916,8 +5916,13 @@
             level+=64;
             if((level&(~127)) == 0){
                 for(j=survivor_count-1; j>=0; j--){
+                   int score;
                     int run= i - survivor[j];
-                    int score= distoration +
length[UNI_AC_ENC_INDEX(run, level)]*lambda;
+
+//av_log(NULL, AV_LOG_INFO, "here distoration %i lambda %i run %i level
%i\n\tindex %i\n", distoration, lambda, run, level,
UNI_AC_ENC_INDEX(run, level));
+                   if (!length) break;
+                    score= distoration + length[UNI_AC_ENC_INDEX(run,
level)]*lambda;
+//av_log(NULL, AV_LOG_INFO, "here2\n");
                     score += score_tab[i-run];

                     if(score < best_score){