[FFmpeg-devel] [PATCH 1/2] pixdesc: Improve scoring for opaque/unknown pixel formats

[Michael Niedermayer]

On Thu, Jul 06, 2017 at 10:59:24PM +0100, Mark Thompson wrote:
> Hardware pixel formats do not tell you anything about their actual
> contents, but should still score higher than formats with completely
> unknown properties, which in turn should score higher than invalid
> formats.
> 
> Do not return an AVERROR code as a score.
> 
> Fixes a hang in libavfilter where format negotiation gets stuck in a
> loop because AV_PIX_FMT_NONE scores more highly than all other
> possibilities.
> ---
> The hang in libavfilter happens when trying to choose a pixfmt for output from a list of software formats when a hardware format is the input (the hwdownload filter does this).  The matching begins with AV_PIX_FMT_NONE as an invalid value and compares against each possibility in turn, but unfortunately it scores -1 when considered as a conversion from an opaque hardware format, higher than the AVERROR(EINVAL) (== -22 on Linux) scored for all of the real formats.  Hence the format selection code chooses AV_PIX_FMT_NONE and thinks it is making forward progress, but actually hasn't and therefore gets stuck in an infinite loop.
> 
> 
>  libavutil/pixdesc.c | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/libavutil/pixdesc.c b/libavutil/pixdesc.c
> index 46a7eff06d..35b63f43c6 100644
> --- a/libavutil/pixdesc.c
> +++ b/libavutil/pixdesc.c
> @@ -2512,7 +2512,11 @@ static int get_pix_fmt_score(enum AVPixelFormat dst_pix_fmt,
>      int score = INT_MAX - 1;
>  
>      if (dst_pix_fmt >= AV_PIX_FMT_NB || dst_pix_fmt <= AV_PIX_FMT_NONE)
> -        return ~0;
> +        return -2;
> +

> +    if ((src_desc->flags & AV_PIX_FMT_FLAG_HWACCEL) ||
> +        (dst_desc->flags & AV_PIX_FMT_FLAG_HWACCEL))
> +        return 0;

this breaks ffmpegs choose_pixel_fmt()
src_desc being NULL


>  
>      /* compute loss */
>      *lossp = loss = 0;

shouldnt this be set on a 0 return ?

i also wonder if src and dst being identical should score different
than if not if either is AV_PIX_FMT_FLAG_HWACCEL

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If you fake or manipulate statistics in a paper in physics you will never
get a job again.
If you fake or manipulate statistics in a paper in medicin you will get
a job for life at the pharma industry.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20170707/26521678/attachment.sig>