[FFmpeg-devel] [PATCH] avcodec/videotoolboxenc: always release supported_props

[Gnattu OC]

> On Aug 15, 2024, at 19:59, Zhao Zhili <quinkblack at foxmail.com> wrote:
> 
> 
> 
>> On Aug 10, 2024, at 14:06, gnattu via ffmpeg-devel <ffmpeg-devel at ffmpeg.org> wrote:
>> 
>> In vtenc_populate_extradata, supported_props should always be released
>> to avoid memory leak.
>> 
>> Regression from cd2f8a22e94700c68b1de7968df11e8bebfd315b
>> 
>> Signed-off-by: gnattu <gnattuoc at me.com>
>> ---
>> libavcodec/videotoolboxenc.c | 13 ++++++++++---
>> 1 file changed, 10 insertions(+), 3 deletions(-)
>> 
>> diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
>> index 78ef474d7ae..6cc45db4a96 100644
>> --- a/libavcodec/videotoolboxenc.c
>> +++ b/libavcodec/videotoolboxenc.c
>> @@ -2787,9 +2787,16 @@ pe_cleanup:
>> 
>>    if (status) {
>>        vtenc_reset(vtctx);
>> -    } else if (vtctx->session) {
>> -        CFRelease(vtctx->session);
>> -        vtctx->session = NULL;
>> +    } else {
>> +        if (vtctx->session) {
>> +            CFRelease(vtctx->session);
>> +            vtctx->session = NULL;
>> +        }
>> +
>> +        if (vtctx->supported_props) {
>> +            CFRelease(vtctx->supported_props);
>> +            vtctx->supported_props = NULL;
>> +        }
>>    }
> 
> Could you elaborate on how supported_props is leaked? Isn’t it cleaned by
> vtenc_close -> vtenc_reset?

In the function `vtenc_populate_extradata`, a call is made to `vtenc_create_encoder` to create a `VTCompressionSession`. During this process, a `supported_props` dictionary is created. However, if the status is 0, only the compression session is released, while `supported_props` remains allocated. 

Later, in `vtenc_configure_encoder`, `vtenc_create_encoder` is called again after `vtenc_populate_extradata` returns. This call replaces the reference to `supported_props` with a new dictionary, causing the reference to the old one (created in `vtenc_populate_extradata`) to be lost. As a result, when `vtenc_close -> vtenc_reset` is called, it will only release the most recent `supported_props`, leading to a memory leak as all previous allocations are not properly released.

> 
>> 
>>    vtctx->frame_ct_out = 0;
>> -- 
>> 2.39.3 (Apple Git-146)
>> 
>> _______________________________________________
>> ffmpeg-devel mailing list
>> ffmpeg-devel at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>> 
>> To unsubscribe, visit link above, or email
>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org <mailto:ffmpeg-devel at ffmpeg.org>
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org <mailto:ffmpeg-devel-request at ffmpeg.org> with subject "unsubscribe".