libswscale does incorrect range conversion for RGB->YUV and RGB->RGB

[gjdfgh]

libswscale seems to do completely confused range conversions for RGB->YUV. RGB->RGB randomly ignores the source range (at least in the unscaled case), and always ignores the destination range. YUV->RGB ignores destination range as well. The only cases that appear to be working as expected are conversions with YUV->YUV.

Here's the output of a test program, that scales a solid image filled with gray=200, using various colorspace and range parameters. "200" is the input value, and the number after that the retrieved luma value or that of a RGB component. "no-scale" means it has been scaled with same source and destination width/height. The test program is attached as swscale.c to this ticket.

no-scale rgb-lim rgb-lim  200 -> 200
no-scale rgb-lim rgb-ful  200 -> 200
no-scale rgb-lim yuv-lim  200 -> 188
no-scale rgb-lim yuv-ful  200 -> 200
no-scale rgb-ful rgb-lim  200 -> 200
no-scale rgb-ful rgb-ful  200 -> 200
no-scale rgb-ful yuv-lim  200 -> 177
no-scale rgb-ful yuv-ful  200 -> 188
no-scale yuv-lim rgb-lim  200 -> 214
no-scale yuv-lim rgb-ful  200 -> 214
no-scale yuv-lim yuv-lim  200 -> 200
no-scale yuv-lim yuv-ful  200 -> 214
no-scale yuv-ful rgb-lim  200 -> 200
no-scale yuv-ful rgb-ful  200 -> 200
no-scale yuv-ful yuv-lim  200 -> 188
no-scale yuv-ful yuv-ful  200 -> 200

scale    rgb-lim rgb-lim  200 -> 200
scale    rgb-lim rgb-ful  200 -> 200
scale    rgb-lim yuv-lim  200 -> 188
scale    rgb-lim yuv-ful  200 -> 200
scale    rgb-ful rgb-lim  200 -> 188
scale    rgb-ful rgb-ful  200 -> 188
scale    rgb-ful yuv-lim  200 -> 177
scale    rgb-ful yuv-ful  200 -> 188
scale    yuv-lim rgb-lim  200 -> 214
scale    yuv-lim rgb-ful  200 -> 214
scale    yuv-lim yuv-lim  200 -> 200
scale    yuv-lim yuv-ful  200 -> 214
scale    yuv-ful rgb-lim  200 -> 200
scale    yuv-ful rgb-ful  200 -> 200
scale    yuv-ful yuv-lim  200 -> 188
scale    yuv-ful yuv-ful  200 -> 200

YUV->RGB ignores destination ranges, and always assumes full destination range.

RGB->YUV looks completely messed up. Maybe someone flipped a bit somewhere?

Scaled RGB->RGB ignores the destination range, and always seems to assume limitted range (so that rgb-lim->* does no conversion). Unscaled RGB->RGB ignores ranges completely, at least when the same pixel format is used.

If there's some logic behind this, I don't get it, and it's not documented.

If libswscale simply doesn't support limitted RGB range (which would lower its usefulness), at least the RGB->RGB case must be fixed.

[gjdfgh]

Test program

[michael]

There exists just one 8bit RGB range and that is 0..255, the difference in ranges is specific to YUV. I think thats the only source of errors in the table so iam closing this, but if theres another problem that i missed, please reopen.
Also ive changed sws so it ignores the range for RGB instead of doing something quite random if such combination is used

[gjdfgh]

Limited RGB does seem to exist: for RGB output to TVs and projectors.

Having no limited RGB support is probably better to breaking completely unrelated conversions, though.