[FFmpeg-devel] [PATCH v9 3/5] avcodec/libjxl: add Jpeg XL encoding via libjxl

[Andreas Rheinhardt]

Leo Izen:
> On 3/23/22 11:08, Andreas Rheinhardt wrote:
>>> +            temp = av_realloc(ctx->buffer, ctx->buffer_size);
>>> +            if (!temp) {
>>> +                av_freep(&ctx->buffer);
>> If you free this, you will be in a scenario where ctx->buffer is NULL,
>> yet ctx->buffer_size is > 0. This is inconsistent and might lead to
>> crashs in JxlEncoderProcessOutput. So don't free this and only set
>> buffer_size after the reallocation succeeded.
> Does it matter what ctx->buffer_size is if ctx->buffer is NULL?
> On 3/23/22 11:08, Andreas Rheinhardt wrote:
>> Unnecessary: av_freep() resets ctx->buffer and resetting buffer_size is
>> unnecessary.
> 
> Here, you claim that resetting buffer_size is unnecessary but above you
> insist it is necessary. I'm confused here.
> 

buffer_size is not used after libjxl_encode_close() at all, as this is
the codec's close callback; libjxl_encode_frame() meanwhile may be
called again after freeing and if it reaches JxlEncoderProcessOutput(),
the latter would be called with next_out == NULL and available != 0.
JxlEncoderProcessOutput() might check for this (and error out) or it may
just crash.

- Andreas