Banding/discoloration in darker gradients when converting to rgb24

[y3kcjd5]

Summary: I've discovered that when I'm converting videos to the rgb24 format (in attempting to use the libx264rgb codec) I get banding and discoloration in darker areas with (color) gradients. To eliminate the possibility that it was the codec (as opposed to the format filter) causing the problem, I did some tests converting the video to .png images. Specifically, I found that:
yuv420p10le(source video) -- looks nice
yuv420p10le -> rgb24 -- banding/discoloration
yuv420p10le -> rgb48be -- looks nice
yuv420p10le -> rgb48be -> rgb24 -- banding/discoloration
yuv420p10le -> yuv420p -> rgb24 -- banding but no discoloration

I assume that the banding in the yuv420p example is simply a result of the reduced colorspace, but I don't see why any of the other conversions should result in any visible changes.
I'm using one of Zeranoe's windows builds (win7 64bit) so I don't know if this happens on Linux. Log file output for the yuv420p10le->rgb24 example follows:

ffmpeg started on 2012-07-28 at 10:56:16
Report written to "ffmpeg-20120728-105616.log"
Command line:
ffmpeg -i vtst0.mp4 -map 0:0 -vsync 0 -report -filter:v "format=rgb24" "tst%03d.png"
ffmpeg version N-42704-g85761ef Copyright (c) 2000-2012 the FFmpeg developers
  built on Jul 20 2012 20:39:19 with gcc 4.7.1 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-runtime-cpudetect --enable-avisynth --enable-bzlib --enable-frei0r --enable-libass --enable-libcelt --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-libnut --enable-libopenjpeg --enable-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libutvideo --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      51. 65.100 / 51. 65.100
  libavcodec     54. 42.100 / 54. 42.100
  libavformat    54. 18.100 / 54. 18.100
  libavdevice    54.  1.100 / 54.  1.100
  libavfilter     3.  2.100 /  3.  2.100
  libswscale      2.  1.100 /  2.  1.100
  libswresample   0. 15.100 /  0. 15.100
  libpostproc    52.  0.100 / 52.  0.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000002e2ec0] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000002e2ec0] ISO: File Type Major Brand: isom
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000002e2ec0] File position before avformat_find_stream_info() is 7074306
[h264 @ 00000000002f40a0] no picture 
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000002e2ec0] All info found
rfps: 23.750000 0.011875
rfps: 23.833333 0.004757
rfps: 23.916667 0.000859
rfps: 24.000000 0.000180
rfps: 24.083333 0.002721
rfps: 24.166667 0.008482
rfps: 24.250000 0.017461
rfps: 47.666667 0.019029
rfps: 47.750000 0.009623
rfps: 47.833333 0.003436
rfps: 47.916667 0.000469
rfps: 48.000000 0.000722
rfps: 48.083333 0.004194
rfps: 48.166667 0.010885
rfps: 23.976024 0.000046
rfps: 47.952048 0.000183
[mov,mp4,m4a,3gp,3g2,mj2 @ 00000000002e2ec0] File position after avformat_find_stream_info() is 1243950
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'vtst0.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf54.18.100
  Duration: 00:00:10.09, start: 0.000000, bitrate: 5607 kb/s
    Stream #0:0(und), 41, 1/1000: Video: h264 (High 10) (avc1 / 0x31637661), yuv420p10le, 1920x1080, 5604 kb/s, 23.98 fps, 23.98 tbr, 1k tbn, 47.95 tbc
    Metadata:
      handler_name    : VideoHandler
[buffer @ 0000000004ba0f20] Setting entry with key 'video_size' to value '1920x1080'
[buffer @ 0000000004ba0f20] Setting entry with key 'pix_fmt' to value '72'
[buffer @ 0000000004ba0f20] Setting entry with key 'time_base' to value '1/1000'
[buffer @ 0000000004ba0f20] Setting entry with key 'pixel_aspect' to value '0/1'
[buffer @ 0000000004ba0f20] Setting entry with key 'sws_param' to value 'flags=2'
[buffer @ 0000000004ba0f20] Setting entry with key 'frame_rate' to value '24000/1001'
[graph 0 input from stream 0:0 @ 0000000004d73980] w:1920 h:1080 pixfmt:yuv420p10le tb:1/1000 fr:24000/1001 sar:0/1 sws_param:flags=2
[Parsed_format_0 @ 0000000004d738e0] auto-inserting filter 'auto-inserted scaler 0' between the filter 'graph 0 input from stream 0:0' and the filter 'Parsed_format_0'
[auto-inserted scaler 0 @ 0000000004d73b20] w:1920 h:1080 fmt:yuv420p10le sar:0/1 -> w:1920 h:1080 fmt:rgb24 sar:0/1 flags:0x4
[h264 @ 00000000002f40a0] detected 2 logical cores
Output #0, image2, to 'tst%03d.png':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf54.18.100
    Stream #0:0(und), 0, 1/90000: Video: png, rgb24, 1920x1080, q=2-31, 200 kb/s, 90k tbn, 23.98 tbc
    Metadata:
      handler_name    : VideoHandler
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> png)
Press [q] to stop, [?] for help
[h264 @ 00000000003aa100] no picture 
DTS -42, next:-62146 st:0 invalid dropping
[h264 @ 0000000004cdbf00] no picture 
frame=    3 fps=0.0 q=0.0 size=       0kB time=00:00:00.12 bitrate=   0.0kbits/s    
frame=    5 fps=3.1 q=0.0 size=       0kB time=00:00:00.20 bitrate=   0.0kbits/s    
frame=    7 fps=3.0 q=0.0 size=       0kB time=00:00:00.29 bitrate=   0.0kbits/s    

...
 
frame=  237 fps=2.6 q=0.0 size=       0kB time=00:00:09.88 bitrate=   0.0kbits/s    
No more inputs to read from, finishing.
[output stream 0:0 @ 0000000004d73a00] EOF on sink link output stream 0:0:default.
frame=  242 fps=2.6 q=0.0 Lsize=       0kB time=00:00:10.26 bitrate=   0.0kbits/s    

video:276218kB audio:0kB subtitle:0 global headers:0kB muxing overhead -100.000000%
[AVIOContext @ 000000000216fd80] Statistics: 4336059 bytes read, 2 seeks

[cehoyos]

Please provide a sample.

[y3kcjd5]

yuv420p10le->rgb24 frame #150 (banding/discoloration)

[y3kcjd5]

OK, seeing as how I can't get the other samples to fit:
Source video clip vtst0.mp4:  http://www.datafilehost.com/download-fb229dc6.html
yuv420p10le->rgb48be frame 150 (looks like source):  http://www.datafilehost.com/download-bc7b949d.html
yuv420p10le->yuv420p->rgb24 frame 150 (banding no discoloration):  http://www.datafilehost.com/download-8a131612.html

[cehoyos]

Just to make sure:
Is the image I attached, rgb48.png ok, but the result of ffmpeg -i rgb48.png -pix_fmt rgb24 out24.png shows the banding?

[y3kcjd5]

That is correct.

[fftest800]

I have the same problem!
More info:  https://ffmpeg.org/trac/ffmpeg/ticket/1611

[cehoyos]

Replying to fftest800:

I have the same problem!

More info:  https://ffmpeg.org/trac/ffmpeg/ticket/1611

I don't see a conversion from higher bpp to lower bpp of the same type of colourspace in that ticket, why do you think it is a duplicate?

[11rcombs]

Found this issue while working on some screenshot comparisons; figured I'd post 'em as another reproduction:
rgb48 = frame extracted from H.264 High 10 video with -ss and -vframes 1
rgb24 = same, but with -filter:v "format=rgb24"
VLCSnap = snapshot of same frame using VLC (probably irrelevant)
Preview = rgb48 frame downsampled to rgb24 using OSX's Preview.app (to show that it's possible to downsample this image without banding)
rgb48 vs rgb24:  http://screenshotcomparison.com/comparison/20164
VLCSnap vs rgb24:  http://screenshotcomparison.com/comparison/20165
rgb24 vs Preview:  http://screenshotcomparison.com/comparison/20166
rgb48 vs Preview:  http://screenshotcomparison.com/comparison/20167