[Ffmpeg-devel] h.264 decoder segfault

[Benoit Fouet]

Benoit Fouet wrote:
> Hi,
>
> Michael Niedermayer wrote:
>   
>> the linenumber in decode_residual() :)))
>>   
>>     
> granted :)
>
>   
>> which is definitly not in bitstream.h but h264.c
>> i do know its crashing in get_vlc*() which is called by decode_residual()
>> but i dont know in which, there are several
>> one of the vlc tables either is damaged, or more likely a index into one
>> is out of range
>>
>> maybe a "#define always_inline" before #include "bitstream.h" with -O0
>> would help
>>
>> or the good old add a printf() before each get_vlc*() in decode_residual()
>>   
>>     
> here is the one that causes the SEGFAULT:
> first line is line 4957
> if(total_coeff == max_coeff)
>         zeros_left=0;
>     else{
>         if(n == CHROMA_DC_BLOCK_INDEX)
>             zeros_left= get_vlc2(gb, chroma_dc_total_zeros_vlc[
> total_coeff-1 ].table, CHROMA_DC_TOTAL_ZEROS_VLC_BITS, 1);
>         else
>             zeros_left= get_vlc2(gb, total_zeros_vlc[ total_coeff-1
> ].table, TOTAL_ZEROS_VLC_BITS, 1);
>     }
>
>
> the second one causes the crash...
> when it crashes, gb is correct (same as previous calls)
> total_zeros_vlc[ total_coeff-1 ].table = NULL
>
> Hope that helps more...
>   
just for test purpose, i tried the patch attached, and it avoids the crash:

Ben

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test_patch_h264.diff
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20070123/6dd45508/attachment.txt>