[FFmpeg-devel] [PATCH 3/4] proresenc_kostya: realloc if buffer too small

[Michael Niedermayer]

On Mon, Aug 11, 2014 at 10:06:09PM +0000, Christophe Gisquet wrote:
> The buffer allocation may be incorrect (e.g. with an alpha plane),
> and currently causes the buffer to be set to NULL by init_put_bits,
> later on causing crashing.
> 
> So, detect that situation, and if detected, reallocate the buffer
> and ask a sample if it happens.
> 
> Fixes ticket #2760
> ---
>  libavcodec/proresenc_kostya.c | 24 ++++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
> 
> diff --git a/libavcodec/proresenc_kostya.c b/libavcodec/proresenc_kostya.c
> index a70ae3c..2cd33ce 100644
> --- a/libavcodec/proresenc_kostya.c
> +++ b/libavcodec/proresenc_kostya.c
> @@ -1023,6 +1023,30 @@ static int encode_frame(AVCodecContext *avctx, AVPacket *pkt,
>                  bytestream_put_byte(&buf, slice_hdr_size << 3);
>                  slice_hdr = buf;
>                  buf += slice_hdr_size - 1;
> +                if (pkt_size <= buf - orig_buf) {

this isnt sufficient, there could be 1 byte space left, then the
reallocate wouldnt run and encode_slice() would run into the extra
padding and fail

I think either enough space should be allocated to begin with (like
your patch 4) then the reallocation shouldnt be needed

or the buffer could be allocated to an average size and reallocated
when the encoder gets close to its end
in which case reallocation also would not need a warning
as it would be a normal operation

that is unless iam missing something

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

No great genius has ever existed without some touch of madness. -- Aristotle
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20140812/7f8aa855/attachment.asc>