[FFmpeg-trac] #1851(swscale:new): libswscale does incorrect range conversion for RGB->YUV and RGB->RGB

[FFmpeg]

#1851: libswscale does incorrect range conversion for RGB->YUV and RGB->RGB
---------------------------------+--------------------------------------
             Reporter:  gjdfgh   |                     Type:  defect
               Status:  new      |                 Priority:  important
            Component:  swscale  |                  Version:  git-master
             Keywords:           |               Blocked By:
             Blocking:           |  Reproduced by developer:  0
Analyzed by developer:  0        |
---------------------------------+--------------------------------------
 libswscale seems to do completely confused range conversions for RGB->YUV.
 RGB->RGB randomly ignores the source range (at least in the unscaled
 case), and always ignores the destination range. YUV->RGB ignores
 destination range as well. The only cases that appear to be working as
 expected are conversions with YUV->YUV.

 Here's the output of a test program, that scales a solid image filled with
 gray=200, using various colorspace and range parameters. "200" is the
 input value, and the number after that the retrieved luma value or that of
 a RGB component. "no-scale" means it has been scaled with same source and
 destination width/height. The test program is attached as swscale.c to
 this ticket.

 {{{
 no-scale rgb-lim rgb-lim  200 -> 200
 no-scale rgb-lim rgb-ful  200 -> 200
 no-scale rgb-lim yuv-lim  200 -> 188
 no-scale rgb-lim yuv-ful  200 -> 200
 no-scale rgb-ful rgb-lim  200 -> 200
 no-scale rgb-ful rgb-ful  200 -> 200
 no-scale rgb-ful yuv-lim  200 -> 177
 no-scale rgb-ful yuv-ful  200 -> 188
 no-scale yuv-lim rgb-lim  200 -> 214
 no-scale yuv-lim rgb-ful  200 -> 214
 no-scale yuv-lim yuv-lim  200 -> 200
 no-scale yuv-lim yuv-ful  200 -> 214
 no-scale yuv-ful rgb-lim  200 -> 200
 no-scale yuv-ful rgb-ful  200 -> 200
 no-scale yuv-ful yuv-lim  200 -> 188
 no-scale yuv-ful yuv-ful  200 -> 200

 scale    rgb-lim rgb-lim  200 -> 200
 scale    rgb-lim rgb-ful  200 -> 200
 scale    rgb-lim yuv-lim  200 -> 188
 scale    rgb-lim yuv-ful  200 -> 200
 scale    rgb-ful rgb-lim  200 -> 188
 scale    rgb-ful rgb-ful  200 -> 188
 scale    rgb-ful yuv-lim  200 -> 177
 scale    rgb-ful yuv-ful  200 -> 188
 scale    yuv-lim rgb-lim  200 -> 214
 scale    yuv-lim rgb-ful  200 -> 214
 scale    yuv-lim yuv-lim  200 -> 200
 scale    yuv-lim yuv-ful  200 -> 214
 scale    yuv-ful rgb-lim  200 -> 200
 scale    yuv-ful rgb-ful  200 -> 200
 scale    yuv-ful yuv-lim  200 -> 188
 scale    yuv-ful yuv-ful  200 -> 200
 }}}

 YUV->RGB ignores destination ranges, and always assumes full destination
 range.

 RGB->YUV looks completely messed up. Maybe someone flipped a bit
 somewhere?

 Scaled RGB->RGB ignores the destination range, and always seems to assume
 limitted range (so that rgb-lim->* does no conversion). Unscaled RGB->RGB
 ignores ranges completely, at least when the same pixel format is used.

 If there's some logic behind this, I don't get it, and it's not
 documented.

 If libswscale simply doesn't support limitted RGB range (which would lower
 its usefulness), at least the RGB->RGB case must be fixed.

-- 
Ticket URL: <https://ffmpeg.org/trac/ffmpeg/ticket/1851>
FFmpeg <http://ffmpeg.org>
FFmpeg issue tracker