[FFmpeg-devel] [PATCH] vdpau: Fix buffer overflow with old hwaccel_context API

[wm4]

On Fri,  8 Sep 2017 21:08:13 +0200
Błażej Szczygieł <spaz16 at wp.pl> wrote:

> VDPAUHWContext struct is used internally, so allocate required amount
> of memory. Also move hwctx->reset as it was prior to
> 7e4ba776a2240d40124d5540ea6b2118fa2fe26a to make sure that buffer
> overflow doesn't happen if application allocates only
> sizeof(AVVDPAUContext) bytes of memory.
> ---
>  libavcodec/vdpau.c | 5 ++---
>  1 file changed, 2 insertions(+), 3 deletions(-)
> 
> diff --git a/libavcodec/vdpau.c b/libavcodec/vdpau.c
> index 9c7804a287..8d1748347f 100644
> --- a/libavcodec/vdpau.c
> +++ b/libavcodec/vdpau.c
> @@ -142,14 +142,13 @@ int ff_vdpau_common_init(AVCodecContext *avctx, VdpDecoderProfile profile,
>          return AVERROR(ENOSYS);
>  
>      if (hwctx) {
> -        hwctx->reset            = 0;
> -
>          if (hwctx->context.decoder != VDP_INVALID_HANDLE) {
>              vdctx->decoder = hwctx->context.decoder;
>              vdctx->render  = hwctx->context.render;
>              vdctx->device  = VDP_INVALID_HANDLE;
>              return 0; /* Decoder created by user */
>          }
> +        hwctx->reset            = 0;
>  
>          vdctx->device           = hwctx->device;
>          vdctx->get_proc_address = hwctx->get_proc_address;
> @@ -797,7 +796,7 @@ do {                                       \
>  
>  AVVDPAUContext *av_vdpau_alloc_context(void)
>  {
> -    return av_mallocz(sizeof(AVVDPAUContext));
> +    return av_mallocz(sizeof(VDPAUHWContext));
>  }
>  
>  int av_vdpau_bind_context(AVCodecContext *avctx, VdpDevice device,

I realize now this is with the old-old API. av_vdpau_alloc_context() is
really not supposed to be used. You're supposed to call
av_vdpau_bind_context(), which allocates the context for you. So I
don't think applying this patch makes much sense or helps anyone. We
should just deprecate all the old crappy API instead.