[Libav-user] HEVC 2/3 of the frame is green

Ben Harper rogojin at gmail.com
Tue Apr 4 12:17:30 EEST 2023 

Did you get this working?

In my brief experience with h264 codecs, sometimes I've had to merge NAL
units together and then send it to the encoder with a single av_write_frame
call. I described it here:
https://www.terminateandstayresident.com/2022-07-16/h264-nalus-to-avcodec

On Fri, Mar 31, 2023 at 1:52 PM Michael Wood <mornwood23 at gmail.com> wrote:

> Hi Srinivas,
>
> Thanks for your reply.
>
> The live555 library is outputting NAL units, which should be sent to the
> decoder. I am not sure if i'm supposed to merge multiple nal units with the
> same timestamp before sending it to the decoder or is libav smart enough to
> figure out that those are fragments and should be decoded together ?
>
> Best,
> Milos
>
> On Thu, Mar 30, 2023 at 4:17 PM srinivas gudumasu <srinu.try at gmail.com>
> wrote:
>
>> Hi,
>>
>> Refer https://www.rfc-editor.org/rfc/rfc7798.html clause 4.4.3 (Fragmentation
>> Unit).
>>
>> When you see the PayloadHdr type as 49 it's a Fragmentation Unit RTP
>> packet. The FU header has two fields S and E. S bit is set for the first Fragmentation
>> Unit of a NAL Unit and E bit is set for the last Fragmentation Unit of a
>> NAL Unit.
>>
>>  0                   1                   2                   3
>>     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>    |    PayloadHdr (Type=49)       |   FU header   | DONL (cond)   |
>>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-|
>>    | DONL (cond)   |                                               |
>>    |-+-+-+-+-+-+-+-+                                               |
>>    |                         FU payload                            |
>>    |                                                               |
>>    |                               +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>    |                               :...OPTIONAL RTP padding        |
>>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>
>>
>>    +---------------+
>>    |0|1|2|3|4|5|6|7|
>>    +-+-+-+-+-+-+-+-+
>>    |S|E|  FuType   |
>>    +---------------+
>>
>>    Figure 10: The Structure of FU Header
>>
>> Another way of detecting the last packet of a frame is using a Marker bit. M bit is set to 1, for the last packet of the access unit, carried in the current RTP stream.
>>
>>   0                   1                   2                   3
>>     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>    |V=2|P|X|  CC   |M|     PT      |       sequence number         |
>>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>    |                           timestamp                           |
>>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>    |           synchronization source (SSRC) identifier            |
>>    +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
>>    |            contributing source (CSRC) identifiers             |
>>    |                             ....                              |
>>    +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>>
>>
>> Thanks and Regards,
>>
>> Srinivas G
>>
>>
>> On Wed, Mar 29, 2023 at 6:28 AM Michael Wood <mornwood23 at gmail.com>
>> wrote:
>>
>>>
>>> On Tue, Mar 28, 2023 at 7:53 PM Michael Wood <mornwood23 at gmail.com>
>>> wrote:
>>>
>>>> Yeah that is correct,
>>>>
>>>> I'm still unable to fix it, i'm trying to figure out what could cause
>>>> the frame to not be properly decoded.
>>>>
>>>> Best,
>>>> Milos
>>>>
>>>> On Sat, Mar 25, 2023 at 2:48 PM Strahinja Radman <dr.strashni at gmail.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On 25. 3. 2023., at 13:26, Michael Wood <mornwood23 at gmail.com> wrote:
>>>>>
>>>>> Hi Strahinja,
>>>>>
>>>>> Thanks for the help,
>>>>>
>>>>> I'm wondering if I need to update extradata (VPS,SPS,PPS) after
>>>>> initial setup of the AVCodecContext from the initial value coming from RTP ?
>>>>>
>>>>> Best,
>>>>> Milos
>>>>>
>>>>> On Sat, Mar 25, 2023 at 11:25 AM Strahinja Radman <
>>>>> dr.strashni at gmail.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Sat, 25 Mar 2023 at 11:09, Michael Wood <mornwood23 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Mar 24, 2023 at 1:40 PM Strahinja Radman <
>>>>>>> dr.strashni at gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 24. 3. 2023., at 10:59, Michael Wood <mornwood23 at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Mar 24, 2023 at 10:28 AM Strahinja Radman <
>>>>>>>> dr.strashni at gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 23. 3. 2023., at 20:19, Michael Wood <mornwood23 at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 23, 2023 at 8:06 PM Strahinja Radman <
>>>>>>>>> dr.strashni at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, 23 Mar 2023 at 20:05, Michael Wood <mornwood23 at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 23, 2023 at 6:20 PM Michael Wood <mornwood23 at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm having a weird issue with the decoding of H265 4K stream where
>>>>>>>>> 1/3 of the frame is rendered normally and the other 2/3 is in green (
>>>>>>>>> https://ibb.co/C8yFygm). This only happens if i try to decode 4K
>>>>>>>>> stream, for non 4K streams everything works as expected.
>>>>>>>>>
>>>>>>>>> I'm using live555 library to get stream from RTSP server after
>>>>>>>>> which packets are sent to the ffmpeg to be decoded and later rendered
>>>>>>>>>
>>>>>>>>> DESCRIBE response contains all required VPS,SPS,PPS fields that
>>>>>>>>> are passed as extra data to the codec context.
>>>>>>>>>
>>>>>>>>> This is what is returned from DESCRIBE:
>>>>>>>>>
>>>>>>>>> v=0
>>>>>>>>> o=- 1679598602413481 1679598602413481 IN IP4 192.168.1.212
>>>>>>>>> s=Media Presentation
>>>>>>>>> e=NONE
>>>>>>>>> b=AS:5050
>>>>>>>>> t=0 0
>>>>>>>>> a=control:rtsp://
>>>>>>>>> 192.168.1.212/Streaming/Channels/101/?transportmode=unicast&profile=Profile_1
>>>>>>>>> m=video 0 RTP/AVP 96
>>>>>>>>> c=IN IP4 0.0.0.0
>>>>>>>>> b=AS:5000
>>>>>>>>> a=recvonly
>>>>>>>>> a=x-dimensions:3840,2160
>>>>>>>>> a=control:rtsp://
>>>>>>>>> 192.168.1.212/Streaming/Channels/101/trackID=1?transportmode=unicast&profile=Profile_1
>>>>>>>>> a=rtpmap:96 H265/90000
>>>>>>>>> a=fmtp:96 sprop-vps=QAEMA///AWAAAAMAgAAAAwAAAwCWAAC94Eg=;
>>>>>>>>> sprop-sps=QgEDAWAAAAMAgAAAAwAAAwCWAACgAeAgAhx/jb33am7uS/YC3AQEBBAAAD6AAALuByHe5RAAmRgACsPgAEyMAAVh8EACZGAAKw+AATIwABWHwg==;
>>>>>>>>> sprop-pps=RAHBcrCcCgFGZA==
>>>>>>>>>
>>>>>>>>> a=Media_header:MEDIAINFO=494D4B48010300000400050000000000000000000000000081000000000000000000000000000000;
>>>>>>>>> a=appversion:1.0
>>>>>>>>>
>>>>>>>>> When trying to decode packets i'm getting this output from hevc,
>>>>>>>>> the ffplay/VLC works normally with the incoming stream.
>>>>>>>>>
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 32(VPS), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 33(SPS), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 34(PPS), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] Decoding VPS
>>>>>>>>> [hevc @ 0x14ac15600] Main profile bitstream
>>>>>>>>> [hevc @ 0x14ac15600] Decoding SPS
>>>>>>>>> [hevc @ 0x14ac15600] Main profile bitstream
>>>>>>>>> [hevc @ 0x14ac15600] Decoding VUI
>>>>>>>>> [hevc @ 0x14ac15600] Decoding PPS
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 32(VPS), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] Decoding VPS
>>>>>>>>> [hevc @ 0x14ac15600] Main profile bitstream
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 33(SPS), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] Decoding SPS
>>>>>>>>> [hevc @ 0x14ac15600] Main profile bitstream
>>>>>>>>> [hevc @ 0x14ac15600] Decoding VUI
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 34(PPS), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] Decoding PPS
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id:
>>>>>>>>> 0, temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] Format yuvj420p chosen by get_format().
>>>>>>>>> [hevc @ 0x14ac15600] Output frame with POC 0.
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id:
>>>>>>>>> 0, temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id:
>>>>>>>>> 0, temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 32(VPS), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] Decoding VPS
>>>>>>>>> [hevc @ 0x14ac15600] Main profile bitstream
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 33(SPS), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] Decoding SPS
>>>>>>>>> [hevc @ 0x14ac15600] Main profile bitstream
>>>>>>>>> [hevc @ 0x14ac15600] Decoding VUI
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 34(PPS), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] Decoding PPS
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id:
>>>>>>>>> 0, temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] Output frame with POC 0.
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id:
>>>>>>>>> 0, temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 19(IDR_W_RADL), nuh_layer_id:
>>>>>>>>> 0, temporal_id: 0
>>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 2
>>>>>>>>> [hevc @ 0x14ac15600] Output frame with POC 1.
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 2
>>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 2
>>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 2
>>>>>>>>> [hevc @ 0x14ac15600] Output frame with POC 2.
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 2
>>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>>> [hevc @ 0x14ac15600] nal_unit_type: 1(TRAIL_R), nuh_layer_id: 0,
>>>>>>>>> temporal_id: 2
>>>>>>>>> [hevc @ 0x14ac15600] First slice in a frame missing.
>>>>>>>>>
>>>>>>>>> If I write the stream to the file and play that file either
>>>>>>>>> through ffplay or VLC the frames are rendered normally.
>>>>>>>>>
>>>>>>>>> Any suggestion would be welcomed.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Milos
>>>>>>>>>
>>>>>>>>>  Hey Strahinja,
>>>>>>>>>
>>>>>>>>> Thanks for the answer,
>>>>>>>>>
>>>>>>>>> Can you provide more info on how to do it ?
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Milos
>>>>>>>>> _______________________________________________
>>>>>>>>> Libav-user mailing list
>>>>>>>>> Libav-user at ffmpeg.org
>>>>>>>>> https://ffmpeg.org/mailman/listinfo/libav-user
>>>>>>>>>
>>>>>>>>> To unsubscribe, visit link above, or email
>>>>>>>>> libav-user-request at ffmpeg.org with subject "unsubscribe".
>>>>>>>>> Hey Milos,
>>>>>>>>>
>>>>>>>>> In AVFrame structure, check width, height and stride variables.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Strahinja Radman
>>>>>>>>> _______________________________________________
>>>>>>>>> Libav-user mailing list
>>>>>>>>> Libav-user at ffmpeg.org
>>>>>>>>> https://ffmpeg.org/mailman/listinfo/libav-user
>>>>>>>>>
>>>>>>>>> To unsubscribe, visit link above, or email
>>>>>>>>> libav-user-request at ffmpeg.org with subject "unsubscribe".
>>>>>>>>>
>>>>>>>>> Hey Strahinja,
>>>>>>>>>
>>>>>>>>> It seems that both width & height are set correctly. I can't
>>>>>>>>> find stride, i don't see any reference to it in the AVFrame header.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Milos
>>>>>>>>> _______________________________________________
>>>>>>>>> Libav-user mailing list
>>>>>>>>> Libav-user at ffmpeg.org
>>>>>>>>> https://ffmpeg.org/mailman/listinfo/libav-user
>>>>>>>>>
>>>>>>>>> To unsubscribe, visit link above, or email
>>>>>>>>> libav-user-request at ffmpeg.org with subject "unsubscribe”.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hey Milos,
>>>>>>>>>
>>>>>>>>> What does a parameter linesize say?
>>>>>>>>>
>>>>>>>>> AVFrame->linesize[0]
>>>>>>>>>
>>>>>>>>> Kind regards,
>>>>>>>>> Strahinja
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Hi Strahinja,
>>>>>>>>
>>>>>>>> The linesize looks good and corresponds to frame size
>>>>>>>>
>>>>>>>> frame->linesize[0] = 3840
>>>>>>>> frame->linesize[1] = 1920
>>>>>>>> frame->linesize[2] = 1920
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Milos
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hey Milos,
>>>>>>>>
>>>>>>>> Can you use this method to save one frame to the PGM file and check
>>>>>>>> its contents right after you receive an AVFrame
>>>>>>>>
>>>>>>>> static void pgm_save(unsigned char* buf, int wrap, int xsize, int
>>>>>>>> ysize, const char* filename)
>>>>>>>>  {
>>>>>>>> FILE* f;
>>>>>>>> int i;
>>>>>>>>         f = fopen(filename, "w");
>>>>>>>> fprintf(f, "P5\n%d %d\n%d\n", xsize, ysize, 255);
>>>>>>>> for (i = 0; i < ysize; i++)
>>>>>>>> fwrite(buf + i * wrap, 1, xsize, f);
>>>>>>>> fclose(f);
>>>>>>>> }
>>>>>>>>
>>>>>>>> buf is frame->data[0]
>>>>>>>> wrap is frame->linesize[0]
>>>>>>>> xsize is width and ysize is height
>>>>>>>>
>>>>>>>> Kind regards,
>>>>>>>> Strahinja
>>>>>>>>
>>>>>>>>
>>>>>>> Hi Strahinja,
>>>>>>>
>>>>>>> I'm getting the same result with saved picture, here is the link to
>>>>>>> the output (
>>>>>>> https://drive.google.com/file/d/1zEABl32ehVu-gnPdqjWciAb7im18VZIf/view?usp=sharing
>>>>>>> )
>>>>>>>
>>>>>>> Best,
>>>>>>> Milos
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Hey Milos,
>>>>>>
>>>>>> One thing to check is if live555 is configured correctly. Check if
>>>>>> every decoder field is set properly and contains expected values. Sometimes
>>>>>> stream properties can change during the decoding process and that can
>>>>>> affect the output.
>>>>>>
>>>>>> Kind regards,
>>>>>> Strahinja
>>>>>>
>>>>>>> --
>>>>>>
>>>>>> Regards
>>>>>> Strahinja Radman
>>>>>>
>>>>>
>>>>> Hey Milos,
>>>>>
>>>>> That shouldnt be necessary, that information should be used while
>>>>> configuring the decoder context.
>>>>>
>>>>> Kind regards,
>>>>> Strahinja
>>>>>
>>>>>
>>>
>>> Update:
>>>
>>> So I've managed to figure out what is happening, the NAL units I'm
>>> receiving from the live555 library are actually fragments of a single
>>> picture. Technically I need to combine those fragments into a single packet
>>> and send it to ffmpeg for decoding.
>>>
>>> Any suggestions on how to recognize which fragments belong to a single
>>> packet ?
>>>
>>> Best,
>>> Milos
>>> _______________________________________________
>>> Libav-user mailing list
>>> Libav-user at ffmpeg.org
>>> https://ffmpeg.org/mailman/listinfo/libav-user
>>>
>>> To unsubscribe, visit link above, or email
>>> libav-user-request at ffmpeg.org with subject "unsubscribe".
>>>
>>
>>
>> --
>> Thanks and Regards,
>> Srinivas G
>> _______________________________________________
>> Libav-user mailing list
>> Libav-user at ffmpeg.org
>> https://ffmpeg.org/mailman/listinfo/libav-user
>>
>> To unsubscribe, visit link above, or email
>> libav-user-request at ffmpeg.org with subject "unsubscribe".
>>
> _______________________________________________
> Libav-user mailing list
> Libav-user at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/libav-user
>
> To unsubscribe, visit link above, or email
> libav-user-request at ffmpeg.org with subject "unsubscribe".
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ffmpeg.org/pipermail/libav-user/attachments/20230404/d970f00a/attachment.htm>

More information about the Libav-user mailing list