[FFmpeg-devel] [PATCH] avcodec/vc1dec: Re-order init to avoid initting hwaccel too early

[Michael Niedermayer]

On Sat, Aug 08, 2015 at 03:03:27PM -0700, Philip Langdale wrote:
> At least for vdpau, the hwaccel init code tries to check the video
> profile and ensure that there is a matching vdpau profile available.
> 
> If it can't find a match, it will fail to initialise.
> 
> In the case of wmv3/vc1, I observed initialisation to fail all the
> time. It turns out that this is due to the hwaccel being initialised
> very early in the codec init, before the profile has been extracted
> and set.
> 
> Conceptually, it's a simple fix to reorder the init code, but it gets
> messy really fast because ff_get_format(), which is what implicitly
> trigger hwaccel init, is called multiple times through various shared
> init calls from h263, etc. It's incredibly hard to prove to my own
> satisfaction that it's safe to move the vc1 specific init code
> ahead of this generic code, but all the vc1 fate tests pass, and I've
> visually inspected a couple of samples and things seem correct.
> 
> Signed-off-by: Philip Langdale <philipl at overt.org>
> ---
>  libavcodec/vc1dec.c | 47 ++++++++++++++++++++++++-----------------------
>  1 file changed, 24 insertions(+), 23 deletions(-)
> 
> diff --git a/libavcodec/vc1dec.c b/libavcodec/vc1dec.c
> index abaa0b4..12257b8 100644
> --- a/libavcodec/vc1dec.c
> +++ b/libavcodec/vc1dec.c
> @@ -428,31 +428,8 @@ static av_cold int vc1_decode_init(AVCodecContext *avctx)
>  
>      if (!avctx->extradata_size || !avctx->extradata)
>          return -1;
> -    if (!CONFIG_GRAY || !(avctx->flags & AV_CODEC_FLAG_GRAY))
> -        avctx->pix_fmt = ff_get_format(avctx, avctx->codec->pix_fmts);
> -    else {
> -        avctx->pix_fmt = AV_PIX_FMT_GRAY8;
> -        if (avctx->color_range == AVCOL_RANGE_UNSPECIFIED)
> -            avctx->color_range = AVCOL_RANGE_MPEG;
> -    }
>      v->s.avctx = avctx;
>  
> -    if ((ret = ff_vc1_init_common(v)) < 0)
> -        return ret;
> -    // ensure static VLC tables are initialized
> -    if ((ret = ff_msmpeg4_decode_init(avctx)) < 0)
> -        return ret;
> -    if ((ret = ff_vc1_decode_init_alloc_tables(v)) < 0)
> -        return ret;
> -    // Hack to ensure the above functions will be called
> -    // again once we know all necessary settings.
> -    // That this is necessary might indicate a bug.
> -    ff_vc1_decode_end(avctx);
> -
> -    ff_blockdsp_init(&s->bdsp, avctx);
> -    ff_h264chroma_init(&v->h264chroma, 8);
> -    ff_qpeldsp_init(&s->qdsp);
> -
>      if (avctx->codec_id == AV_CODEC_ID_WMV3 || avctx->codec_id == AV_CODEC_ID_WMV3IMAGE) {
>          int count = 0;
>  
> @@ -532,6 +509,30 @@ static av_cold int vc1_decode_init(AVCodecContext *avctx)
>      if (v->profile == PROFILE_ADVANCED)
>          avctx->level = v->level;
>  
> +    if (!CONFIG_GRAY || !(avctx->flags & AV_CODEC_FLAG_GRAY))
> +        avctx->pix_fmt = ff_get_format(avctx, avctx->codec->pix_fmts);
> +    else {
> +        avctx->pix_fmt = AV_PIX_FMT_GRAY8;
> +        if (avctx->color_range == AVCOL_RANGE_UNSPECIFIED)
> +            avctx->color_range = AVCOL_RANGE_MPEG;
> +    }
> +
> +    if ((ret = ff_vc1_init_common(v)) < 0)
> +        return ret;
> +    // ensure static VLC tables are initialized
> +    if ((ret = ff_msmpeg4_decode_init(avctx)) < 0)
> +        return ret;
> +    if ((ret = ff_vc1_decode_init_alloc_tables(v)) < 0)
> +        return ret;
> +    // Hack to ensure the above functions will be called
> +    // again once we know all necessary settings.
> +    // That this is necessary might indicate a bug.

> +    ff_vc1_decode_end(avctx);

this will dealocate sprite_output_frame and crash when its used later

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If you think the mosad wants you dead since a long time then you are either
wrong or dead since a long time.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20150809/6cc756e5/attachment.sig>