[FFmpeg-devel] Defect: MediaCodec Encoder - Pixel ColorSpace of many Adreno GPUs not supported

[Ben Temple]

My apologies for using the incorrect channel. I mis-read the new ticket
description that states that *patches* must come through the mailing list,
not new tickets.

I've created a ticket here: https://trac.ffmpeg.org/ticket/10096 where we
can continue the discussion.

As a quick side-note, using -px_fmt nv12 does resolve the issue on a quick
test that I ran on a phone that was failing. I'll respond further on the
ticket so as to not add noise to this mailing list. Thank you for your
response!

On Wed, Dec 7, 2022 at 9:16 AM Zhao Zhili <quinkblack at foxmail.com> wrote:

> > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Ben
> Temple
> > Sent: 2022年12月7日 6:53
> > To: ffmpeg-devel at ffmpeg.org; Zhao Zhili <zhilizhao at tencent.com>
> > Subject: [FFmpeg-devel] Defect: MediaCodec Encoder - Pixel ColorSpace of
> many Adreno GPUs not supported
> >
> > MediaCodec Encoder: Pixel ColorSpace of many Adreno GPUs is not supported
> > (<= Adreno 650 in our testing)What you were trying to accomplish?
> >
> > I am trying to transcode an MP4 file using the MediaCodec encoder support
> > that was recently added by Zhao Zhili <zhilizhao at tencent.com> here
> > <
> https://urldefense.com/v3/__https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0ff18a7d6d496f89d9e007ddd1cad7116baf5c7c__;!!N1d2nkBXpQ!K7sceiwL4SR1PIFjy4TY99mSRBvCQZi78fR4-CH4X5H9Q_KXbtmpW22WH9y-eagaazoZSyHLeITHcSseZdyLWTgl$
> >
> > (
> >
> https://urldefense.com/v3/__https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0ff18a7d6d496f89d9e007ddd1cad7116baf5c7c__;!!N1d2nkBXpQ!K7sceiwL4SR1PIFjy4TY99mSRBvCQZi78fR4-CH4X5H9Q_KXbtmpW22WH9y-eagaazoZSyHLeITHcSseZdyLWTgl$
> > )
> > The problem you encountered:
> >
> > On multiple phones containing an Adreno GPU, the encoding process
> > immediately fails because the color space expected by the GPU is
> > unsupported.
> >
> > The exact error output: “does not support color format 19”
> >
> ...
> >
> > The exact command line you were using:
> >
> > -hwaccel mediacodec -hwaccel_output_format mediacodec -i
> > "/data/user/0/com.test.ffmpeg_kit_android_demo/cache/test_input.mp4"
> -c:v
> > h264_mediacodec  -ndk_codec 1  -y
> > "/data/user/0/com.test.ffmpeg_kit_android_demo/cache/test_output.mp4"
> > -loglevel trace
>
> Thank you for your valuable test!
>
> There is an unexpected behavior in the test. The test was conducted with
> ffmpeg-kit, not ffmpeg cmdline.
> With FFmpeg cmdline and -hwcontext mediacodec, the encoder should have
> "-pix_fmt mediacodec" by
> default, and have an AVMediaCodecDeviceContext (with Android 8.0 or
> higher). The encoder doesn't
> setup properly with ffmpeg-kit.
>
> pix_fmt mediacodec should be supported by most of Android devices. It do
> have its limitation, e.g., it needs
> a Surface object or require Android 8.0, and it doesn't work with avfilter.
>
> The above issue can be fixed on the ffmpeg-kit's side. Now comes the dark
> corner of Mediacodec. It should
> work without pix_fmt mediacodec. Can you try specify the encoder pix_fmt
> as nv12, that's
>
> -hwaccel mediacodec -hwaccel_output_format mediacodec -i test_input.mp4
> -c:v h264_mediacodec -ndk_codec 1 -pix_fmt nv12 -y test_output.mp4
>
> I guess these devices work with nv12.
>
> There is an old story "MediaCodec and device compatibility"
>
>
> https://urldefense.com/v3/__https://groups.google.com/g/android-platform/c/p_MoSk0JPNM__;!!N1d2nkBXpQ!K7sceiwL4SR1PIFjy4TY99mSRBvCQZi78fR4-CH4X5H9Q_KXbtmpW22WH9y-eagaazoZSyHLeITHcSseZZeyTDJ9$
>
> Ten years on, the situation doesn't improved much:
> 1. There is an ugly API to get color format supported by a codec.
> 2. It's only available in Java.
> 3. Different codec component have different color format.
>
> Yes, we can use a bunch of JNI to get the supported color format. It must
> be done before create
> AVCodecContext. But different mediacodec component have different color
> format, user can select
> mediacodec by name (see the following patch), we have only one
> AVCodec.pix_fmts.
>
> avcodec/mediacodecenc: add option to select codec by name
>
> https://urldefense.com/v3/__https://patchwork.ffmpeg.org/project/ffmpeg/patch/tencent_729F52ABE34E4770972DB89AF840549A8208@qq.com/__;!!N1d2nkBXpQ!K7sceiwL4SR1PIFjy4TY99mSRBvCQZi78fR4-CH4X5H9Q_KXbtmpW22WH9y-eagaazoZSyHLeITHcSseZTsxLMxB$
>
> We can add query the supported color format and add some hints when
> mediacodec configure failed.
> Or we can add a helper function for user to query the color format. It
> will be a bunch of JNI code, limited
> by the ugly MediaCodecInfo.CodecCapabilities API. It's easy to be done
> with Java. TBH, it's much easier with
> trial and error, try nv12 then yuv420.
>
> We can do something with FFCodec.init_static_data, but we need a strategy
> first. I don't want to go with a long
> blacklist/whitelist.
>
> ** We only need one pixel format which supported by all devices,
> COLOR_FormatSurface doesn't count, and
> COLOR_FormatYUV420Flexible doesn't count. **
>
> Maybe Google can fix these issues with another 10 year.
>
> > Console output:
> >
> > See attached console_output.log
> > Input Files:
> >
> > See attached test_input.mp4
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
>
> https://urldefense.com/v3/__https://ffmpeg.org/mailman/listinfo/ffmpeg-devel__;!!N1d2nkBXpQ!K7sceiwL4SR1PIFjy4TY99mSRBvCQZi78fR4-CH4X5H9Q_KXbtmpW22WH9y-eagaazoZSyHLeITHcSseZR7beCWm$
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>