[FFmpeg-devel] [PATCH] DXVA2: Fix crash releasing IDirect3D9Surface's with av_buffer_default_free instead of Release

[Hendrik Leppkes]

On Fri, Sep 30, 2016 at 7:48 AM, Min <yosoymin at gmail.com> wrote:
> diff --git a/libavutil/hwcontext_dxva2.c b/libavutil/hwcontext_dxva2.c
> index e79254b..17d8eb5 100644
> --- a/libavutil/hwcontext_dxva2.c
> +++ b/libavutil/hwcontext_dxva2.c
> @@ -101,6 +101,11 @@ static void dxva2_frames_uninit(AVHWFramesContext *ctx)
>      }
>  }
>
> +void dxva2_pool_free(void *opaque, uint8_t *data)
> +{
> +    // No need to free surfaces here, they will be Released later
> +}
> +
>  static AVBufferRef *dxva2_pool_alloc(void *opaque, int size)
>  {
>      AVHWFramesContext      *ctx = (AVHWFramesContext*)opaque;
> @@ -110,7 +115,7 @@ static AVBufferRef *dxva2_pool_alloc(void *opaque, int
> size)
>      if (s->nb_surfaces_used < hwctx->nb_surfaces) {
>          s->nb_surfaces_used++;
>          return
> av_buffer_create((uint8_t*)s->surfaces_internal[s->nb_surfaces_used - 1],
> -                                sizeof(*hwctx->surfaces), NULL, 0, 0);
> +                                sizeof(*hwctx->surfaces), dxva2_pool_free,
> 0, 0);
>      }
>
>      return NULL;

This is not correct and will leak memory. The buffer created here is
not a "surface", its plain memory to hold information about a surface,
and needs to be free'ed properly when its no longer used.
On top of all that, DXVA2 usage through ffmpeg.c, for example, also
doesn't crash here.

So, how exactly does one reproduce the problem you're trying to fix?

- Hendrik