[FFmpeg-trac] #10221(avcodec:new): Null pointer when nvenc gets aborted

[FFmpeg]

#10221: Null pointer when nvenc gets aborted
-------------------------------------+-------------------------------------
             Reporter:  Daniel       |                     Type:  defect
  Stankewitz                         |
               Status:  new          |                 Priority:  normal
            Component:  avcodec      |                  Version:  git-
                                     |  master
             Keywords:               |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 **Summary of the bug:**

 libavcodec crashes if nvenc gets aborted with i.e. "No capable devices
 found" or any other error.

 **How to reproduce:**

 With any NVIDIA GPU that is not AV1 capable (because that'll create the
 error "No capable devices found") with this example.

 {{{
 const AVCodec* codec = avcodec_find_encoder_by_name("av1_nvenc");
 AVCodecContext* codecContext = avcodec_alloc_context3(codec);
 codecContext->width = 320;
 codecContext->height = 240;
 codecContext->time_base = { 1, 25 };
 codecContext->pix_fmt = AV_PIX_FMT_YUV420P;
 int res = avcodec_open2(codecContext, codec, NULL);
 avcodec_free_context(&codecContext);

 ffmpeg version: a02e45a1f3cc6f07c7437c1e225dea2683f843cf
 }}}

 In ff_nvenc_encode_init the initialization order is:

 {{{
 if ((ret = nvenc_load_libraries(avctx)) < 0)
     return ret;

 if ((ret = nvenc_setup_device(avctx)) < 0)
     return ret;

 if ((ret = nvenc_setup_encoder(avctx)) < 0)
     return ret;

 if ((ret = nvenc_setup_surfaces(avctx)) < 0)
     return ret;
 }}}

 ... but in ff_nvenc_encode_close() it tries to work with fifo buffers that
 are still null because they'll get initialized in nvenc_setup_surfaces()
 later (after nvenc_setup_device()).
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/10221>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker