[FFmpeg-devel] [PATCH]lavc/libx265: Support gray encoding

[Carl Eugen Hoyos]

Hi!

Attached patch implements gray encoding with libx265, does not fix or 
workaround ticket #5536.

I am also attaching the patch I just sent to x265-devel that allows to 
test the feature.

Please comment, Carl Eugen
-------------- next part --------------
# HG changeset patch
# User Carl Eugen Hoyos <cehoyos at ag.or.at>
# Date 1463045462 -7200
# Node ID 10763a24dfb97725b291a35787c207eb9931e1a0
# Parent  a5362b9533f6a5b77740b4e8f97dba2555b6f929
encoder/sao: Do not assume three planes.

Fixes a crash when encoding gray input.

diff -r a5362b9533f6 -r 10763a24dfb9 source/encoder/sao.cpp
--- a/source/encoder/sao.cpp	Wed May 04 21:08:09 2016 +0000
+++ b/source/encoder/sao.cpp	Thu May 12 11:31:02 2016 +0200
@@ -1265,7 +1265,7 @@
             if (!allowMerge[mergeIdx])
                 continue;
 
-            for (int plane = 0; plane < 3; plane++)
+            for (int plane = 0; plane < planes; plane++)
             {
                 int64_t estDist = 0;
                 SaoCtuParam* mergeSrcParam = &(saoParam->ctuParam[plane][addrMerge[mergeIdx]]);
exporting patch:
<fdopen>
-------------- next part --------------
diff --git a/libavcodec/libx265.c b/libavcodec/libx265.c
index 11088b2..f8dd741 100644
--- a/libavcodec/libx265.c
+++ b/libavcodec/libx265.c
@@ -165,6 +165,9 @@ static av_cold int libx265_encode_init(AVCodecContext *avctx)
     case AV_PIX_FMT_YUV444P12:
         ctx->params->internalCsp = X265_CSP_I444;
         break;
+    case AV_PIX_FMT_GRAY8:
+        ctx->params->internalCsp = X265_CSP_I400;
+        break;
     }
 
     if (ctx->crf >= 0) {
@@ -325,6 +328,7 @@ static const enum AVPixelFormat x265_csp_eight[] = {
     AV_PIX_FMT_YUV422P,
     AV_PIX_FMT_YUV444P,
     AV_PIX_FMT_GBRP,
+    AV_PIX_FMT_GRAY8,
     AV_PIX_FMT_NONE
 };
 
@@ -337,6 +341,7 @@ static const enum AVPixelFormat x265_csp_ten[] = {
     AV_PIX_FMT_YUV422P10,
     AV_PIX_FMT_YUV444P10,
     AV_PIX_FMT_GBRP10,
+    AV_PIX_FMT_GRAY8,
     AV_PIX_FMT_NONE
 };
 
@@ -353,6 +358,7 @@ static const enum AVPixelFormat x265_csp_twelve[] = {
     AV_PIX_FMT_YUV422P12,
     AV_PIX_FMT_YUV444P12,
     AV_PIX_FMT_GBRP12,
+    AV_PIX_FMT_GRAY8,
     AV_PIX_FMT_NONE
 };