[FFmpeg-devel] [PATCH 2/2] lavc/videotoolboxenc: enable low-latency mode for HEVC

[Richard Kern]

> On Sep 1, 2023, at 12:07 AM, mypopy at gmail.com wrote:
> 
> On Fri, Sep 1, 2023 at 11:50 AM "zhilizhao(赵志立)" <quinkblack at foxmail.com> wrote:
>> 
>> 
>> 
>>>> On Sep 1, 2023, at 09:13, mypopy at gmail.com wrote:
>>> 
>>> On Fri, Sep 1, 2023 at 2:13 AM Zhao Zhili <quinkblack at foxmail.com> wrote:
>>>> 
>>>> 
>>>>> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Jun Zhao
>>>>> Sent: 2023年8月31日 22:50
>>>>> To: ffmpeg-devel at ffmpeg.org
>>>>> Cc: Jun Zhao <barryjzhao at tencent.com>; Jun Zhao <mypopydev at gmail.com>
>>>>> Subject: [FFmpeg-devel] [PATCH 2/2] lavc/videotoolboxenc: enable low-latency mode for HEVC
>>>>> 
>>>>> Enable the HEVC encoder that supports low-latency mode.
>>>>> 
>>>>> Signed-off-by: Jun Zhao <barryjzhao at tencent.com>
>>>>> ---
>>>>> libavcodec/videotoolboxenc.c | 3 ++-
>>>>> 1 file changed, 2 insertions(+), 1 deletion(-)
>>>>> 
>>>>> diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
>>>>> index 8e493c4f7a..bfc03787a0 100644
>>>>> --- a/libavcodec/videotoolboxenc.c
>>>>> +++ b/libavcodec/videotoolboxenc.c
>>>>> @@ -1568,7 +1568,8 @@ static int vtenc_configure_encoder(AVCodecContext *avctx)
>>>>> #endif
>>>>> 
>>>>>    // low-latency mode: eliminate frame reordering, follow a one-in-one-out encoding mode
>>>>> -    if ((avctx->flags & AV_CODEC_FLAG_LOW_DELAY) && avctx->codec_id == AV_CODEC_ID_H264) {
>>>>> +    if ((avctx->flags & AV_CODEC_FLAG_LOW_DELAY) &&
>>>>> +        (avctx->codec_id == AV_CODEC_ID_H264 || avctx->codec_id == AV_CODEC_ID_HEVC)) {
>>>> 
>>>> How about just remove the codec_id check?
>>>> Is there any unwelcome side effects for encoders other than H264/HEVC?
>>> 
>>> 1. One reason is that there are now other encoders available, such as
>>> prores_videotoolbox.
>>> 
>>> 2. It may be more appropriate to use a strictly qualified check
>>> instead of an open condition.
>> 
>> Both HEVC and ProRes failed with low_latency on my machine (with or without -allow_sw 1)
>> 
>> [hevc_videotoolbox @ 0x61900002a380] Error: cannot create compression session: -12902
>> [prores_videotoolbox @ 0x61900002a380] Error: cannot create compression session: -12902
>> 
>>        kVTParameterErr                                                 = -12902,
>> 
> I think you missed the other options, low delay in VT encoder need to
> co-work with some other options
> 
>> 1. FFmpeg users should have an idea whether or not to set AV_CODEC_FLAG_LOW_DELAY.
>> AV_CODEC_FLAG_LOW_DELAY doesn’t make sense for ProRes, but if user wants to try, let he/she
>> do.
>> 
>> 2. I can’t find API to query the capability. Only VideoToolBox framework itself know how
>> to handle the argument. There isn’t much protect to provide for users.
>> 
> Yes, now FFmpeg VT encoder didn't used the capability query, I think
> need to used the some API VTCopySupportedPropertyDictionaryForEncoder
> for capability query, this my next step
That’s a good idea. We don’t have to worry about codec checks or unexpected failures if the HEVC encoder on some devices doesn’t support it. Ideally we can print out a warn message and ignore the low-delay flag. 

>> Even with H.264, the results is unexpected: the input framerate is 25 but the output
>> framerate is 5, it dropped 80 percent frames silently.
>> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".