[FFmpeg-devel] [PATCH] fixed luma quantizing when q >= MAX_STORED_Q

[Derek Buitenhuis]

On 28/12/2018 17:04, Alex Mogurenko wrote:
> problem occurs in slice quant estimation and slice encoding.
> 
> if slice quant >= MAX_STORED_Q we dont use pre-calculated quant matrices
> but generate new:
> 
> qmat = ctx->custom_q;
> 
>         qmat_chroma = ctx->custom_q;
> 
>         for (i = 0; i < 64; i++) {
> 
>             qmat[i] = ctx->quant_mat[i] * quant;
> 
>             qmat_chroma[i] = ctx->quant_chroma_mat[i] * quant;
> 
>         }
> 
> 
> as you see both qmat and qmat_chroma both point to same ctx -> custom_q
> 
> as result they will contain chroma qunatizers as
> 
> 
> qmat_chroma[i] = ctx->quant_chroma_mat[i] * quant;
> 
> 
> last in loop, after all we pass qmat/qmat_chroma to function where we
> estimate encoded slice size or encode slice:
> 
> estimate_slice_plane / encode_slice_plane
> 
> 
> I will make new patch with detailed description
> 

Thanks, this makes sense, I missed that the fact it was setting it in the loop.

- Derek