[FFmpeg-trac] #10221(avcodec:new): Null pointer when nvenc gets aborted

[FFmpeg]

#10221: Null pointer when nvenc gets aborted
-------------------------------------+-------------------------------------
             Reporter:  Daniel       |                    Owner:  (none)
  Stankewitz                         |
                 Type:  defect       |                   Status:  new
             Priority:  normal       |                Component:  avcodec
              Version:  git-master   |               Resolution:
             Keywords:               |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
Description changed by Daniel Stankewitz:

Old description:

> **Summary of the bug:**
>
> libavcodec crashes if nvenc gets aborted with i.e. "No capable devices
> found" or any other error.
>
> **How to reproduce:**
>
> With any NVIDIA GPU that is not AV1 capable (because that'll create the
> error "No capable devices found") with this example.
>
> {{{
> const AVCodec* codec = avcodec_find_encoder_by_name("av1_nvenc");
> AVCodecContext* codecContext = avcodec_alloc_context3(codec);
> codecContext->width = 320;
> codecContext->height = 240;
> codecContext->time_base = { 1, 25 };
> codecContext->pix_fmt = AV_PIX_FMT_YUV420P;
> int res = avcodec_open2(codecContext, codec, NULL);
> avcodec_free_context(&codecContext);
>
> ffmpeg version: a02e45a1f3cc6f07c7437c1e225dea2683f843cf
> }}}
>
> In ff_nvenc_encode_init the initialization order is:
>
> {{{
> if ((ret = nvenc_load_libraries(avctx)) < 0)
>     return ret;
>
> if ((ret = nvenc_setup_device(avctx)) < 0)
>     return ret;
>
> if ((ret = nvenc_setup_encoder(avctx)) < 0)
>     return ret;
>
> if ((ret = nvenc_setup_surfaces(avctx)) < 0)
>     return ret;
> }}}
>
> ... but in ff_nvenc_encode_close() it tries to work with fifo buffers
> that are still null because they'll get initialized in
> nvenc_setup_surfaces() later (after nvenc_setup_device()).

New description:

 **Summary of the bug:**

 libavcodec crashes if nvenc gets aborted with i.e. "No capable devices
 found" or any other error.

 **How to reproduce:**

 With any NVIDIA GPU that is not AV1 capable (because that'll create the
 error "No capable devices found") with this example.

 {{{
 % ffmpeg.exe -v trace -f lavfi -i smptebars=size=1920x1080 -t 5 -c:v
 av1_nvenc -b:v 500k -c:a none -pix_fmt yuv420p test.mp4
 }}}

 or

 {{{
 const AVCodec* codec = avcodec_find_encoder_by_name("av1_nvenc");
 AVCodecContext* codecContext = avcodec_alloc_context3(codec);
 codecContext->width = 320;
 codecContext->height = 240;
 codecContext->time_base = { 1, 25 };
 codecContext->pix_fmt = AV_PIX_FMT_YUV420P;
 int res = avcodec_open2(codecContext, codec, NULL);
 avcodec_free_context(&codecContext);

 ffmpeg version: a02e45a1f3cc6f07c7437c1e225dea2683f843cf
 }}}

 In ff_nvenc_encode_init the initialization order is:

 {{{
 if ((ret = nvenc_load_libraries(avctx)) < 0)
     return ret;

 if ((ret = nvenc_setup_device(avctx)) < 0)
     return ret;

 if ((ret = nvenc_setup_encoder(avctx)) < 0)
     return ret;

 if ((ret = nvenc_setup_surfaces(avctx)) < 0)
     return ret;
 }}}

 ... but in ff_nvenc_encode_close() it tries to work with fifo buffers that
 are still null because they'll get initialized in nvenc_setup_surfaces()
 later (after nvenc_setup_device()).

--
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/10221#comment:1>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker