[FFmpeg-cvslog] avcodec/libx264: leave full range flag unchanged if unknown

[Jan Ekström]

ffmpeg | branch: master | Jan Ekström <jeebjp at gmail.com> | Tue Aug 17 22:57:36 2021 +0300| [7ca71b79f2b3256a0eef1a099b857ac9e4017e36] | committer: Jan Ekström

avcodec/libx264: leave full range flag unchanged if unknown

By default the x264 full range flag is set to -1. By not setting
it to something else, we can let libx264 handle the RGB case.
Additionally, change the preference order to user-specified range
first, and then any fall-back logic left for the YUVJ pix_fmts.

Fixes the capture part of #9374

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=7ca71b79f2b3256a0eef1a099b857ac9e4017e36
---

 libavcodec/libx264.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/libavcodec/libx264.c b/libavcodec/libx264.c
index 9afaf19547..d48e142e41 100644
--- a/libavcodec/libx264.c
+++ b/libavcodec/libx264.c
@@ -857,10 +857,12 @@ static av_cold int X264_init(AVCodecContext *avctx)
 
     x4->params.i_slice_count  = avctx->slices;
 
-    x4->params.vui.b_fullrange = avctx->pix_fmt == AV_PIX_FMT_YUVJ420P ||
-                                 avctx->pix_fmt == AV_PIX_FMT_YUVJ422P ||
-                                 avctx->pix_fmt == AV_PIX_FMT_YUVJ444P ||
-                                 avctx->color_range == AVCOL_RANGE_JPEG;
+    if (avctx->color_range != AVCOL_RANGE_UNSPECIFIED)
+        x4->params.vui.b_fullrange = avctx->color_range == AVCOL_RANGE_JPEG;
+    else if (avctx->pix_fmt == AV_PIX_FMT_YUVJ420P ||
+             avctx->pix_fmt == AV_PIX_FMT_YUVJ422P ||
+             avctx->pix_fmt == AV_PIX_FMT_YUVJ444P)
+        x4->params.vui.b_fullrange = 1;
 
     if (avctx->colorspace != AVCOL_SPC_UNSPECIFIED)
         x4->params.vui.i_colmatrix = avctx->colorspace;