[FFmpeg-trac] #6379(avcodec:new): vaapi_encode_check_config invalid free

[FFmpeg]

#6379: vaapi_encode_check_config invalid free
----------------------------------+----------------------------------
             Reporter:  serafean  |                     Type:  defect
               Status:  new       |                 Priority:  normal
            Component:  avcodec   |                  Version:  3.2.4
             Keywords:            |               Blocked By:
             Blocking:            |  Reproduced by developer:  0
Analyzed by developer:  0         |
----------------------------------+----------------------------------
 Invalid free in vaapi_encode_check_config ( vaapi_encode_config_attributes
 in 3.3 branch - manually checked source code )
 How to reproduce:

 {{{
 % MALLOC_CHECK_=2 ffmpeg -loglevel debug -hwaccel vaapi -vaapi_device
 /dev/dri/renderD128 -i Elephants_Dream_HD.avi -vf format=nv12,hwupload
 -map 0:0 -map 0:1 -y -f matroska -bf 0 -c:v h264_vaapi ~/test.mkv
 3.2.4
 built on Gentoo
 }}}

 {{{
 (gdb) bt
 #0  0x00007f0fbeb1eeb8 in __GI_raise (sig=sig at entry=6) at
 ../sysdeps/unix/sysv/linux/raise.c:54
 #1  0x00007f0fbeb2044a in __GI_abort () at abort.c:89
 #2  0x00007f0fbeb63890 in malloc_printerr (action=<optimized out>,
 str=0x7f0fbec55c27 "free(): invalid pointer", ptr=<optimized out>,
     ar_ptr=<optimized out>) at malloc.c:5008
 #3  0x00007f0fc01e88b2 in vaapi_encode_check_config (avctx=0x55e6562a3620)
 at src/libavcodec/vaapi_encode.c:1024
 #4  ff_vaapi_encode_init (avctx=0x55e6562a3620, type=<optimized out>) at
 src/libavcodec/vaapi_encode.c:1076
 #5  0x00007f0fc06353f0 in avcodec_open2 (avctx=0x55e6562a3620,
 codec=0x7f0fc1043cc0 <ff_h264_vaapi_encoder>, options=0x55e65624f888)
     at src/libavcodec/utils.c:1608
 #6  0x000055e655ab58ca in init_output_stream (error_len=1024,
 error=0x7ffde9fa30e0 "", ost=0x55e65624f740) at src/ffmpeg.c:3024
 #7  transcode_init () at src/ffmpeg.c:3482
 #8  0x000055e655a98352 in transcode () at src/ffmpeg.c:4358
 #9  main (argc=23, argv=0x7ffde9fa3a48) at src/ffmpeg.c:4592
 }}}

 The issue is that every "goto fail" tries to free both "profiles" and
 "entrypoints", when entrypoints might not even be allocated yet.

--
Ticket URL: <https://trac.ffmpeg.org/ticket/6379>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker