<div dir="ltr"><div>Carl.  </div><div>Thank you for your interest.</div><div>Sorry for the length of this note.</div><div><br></div><div>Short answer...</div><div>My encoder gives me timestamps but I don't know how to make multi-packet frames that include the metadata.</div><div><br></div><div>Long answer...</div>My encoder gives me a timestamp on every frame (int64_t in microseconds). The hardware is pretty good and the timestamps are very nearly always separated to 1/framerate. In my case the framerate is 30fps so the frames come out of the encoder about every 30ms (lets just say for easy math for now).<div><br></div><div>Of course, I could use those timestamps for my pts/dts. Indeed, I have tried.<div><br></div><div>But here is the root of my problem. The key frames are handed to me as four NAL packets (SPS, PPS, SEI, key_frame) all at the same time and all with the same timestamp. That makes sense. The metadata meant for the decoder (SPS, PPS and SEI) should to be associated with the leading key_frame and they should all share the same timestamp. <b>My problem is I don't know how to make a multi-packet 'AVPacket' to submit as a single frame to the 'av_write_frame' call. </b> Instead, I submit every packet as a frame with no timestamp. This seems to work. Something inside the 'av_write_frame' call sees the metadata and squirrels it away for use by the decoder. Ultimately, I end up with a perfectly good HLS file(s). But unfortunately, that HLS file(s) has 'made up' timing.</div><div><br></div><div>If I use the same timestamp for all four pieces of the key_frame, then I end up submitting packets as frames with the following timestamps (in this case, I am generating a keyframe every 30 frames and my framerate is 30fps):</div><div><br></div><div>first SPS (pts/dts = 0ms)   // notice these all have the same timestamp</div><div>first PPS (pts/dts = 0ms)   //  .... and this one ...</div><div>first SEI (pts/dts = 0ms)    //          ... and this one ...</div><div>first keyframe (pts/dts = 0ms) //                      ... and this one</div><div>1st delta frame (pts/dts = 30ms)</div><div>2nd delta frame (pts/dts = 60ms)</div><div>3rd delta frame (pts/dts = 90ms)</div><div>...</div><div>29th delta frame (timestamp=970ms)</div><div><div>second SPS (pts/dts = 1000ms)   // notice these all have the same timestamp</div><div>second PPS (pts/dts = 1000ms)   //  .... and this one ...</div><div>second SEI (pts/dts = 1000ms)   //          ... and this one ...</div><div>second keyframe (pts/dts = 1000ms) //                    ... and this one</div><div>1st delta frame (pts/dts = 1030ms)</div><div>2nd delta frame (pts/dts = 1060ms)</div><div>3rd delta frame (pts/dts = 1090ms)</div><div>...</div><div>29th delta frame (timestamp=1970ms)</div></div><div>...  and_so_on ...</div><div><br></div><div>As you can see, I am telling the libraries that my frame timestamps are (in order):</div><div><br></div><div>0, 0, 0, 0, 30, 60, 90 ... 970, 1000, 1000, 1000, 1000, 1030, 1060, 1090... etc.</div><div><br></div><div>Those are <b>not</b> constantly increasing times stamps. FFMPEG complains.</div><div><br></div><div>So, for now, I am marking every thing as unknown (AV_NOPTS_VALUE) which produces an HLS file(s).</div><div><br></div><div>If I knew how to submit a multi-packet AVPacket to av_write_frame and whatever was eating that packet saw the leading metadata and used it, then I would be golden.</div><div><br></div><div>In general, am I going about this the right way?</div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 20, 2017 at 12:44 PM, Carl Eugen Hoyos <span dir="ltr"><<a href="mailto:ceffmpeg@gmail.com" target="_blank">ceffmpeg@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">2017-11-20 17:50 GMT+01:00 Tyler Brooks <<a href="mailto:tylerjbrooks@gmail.com">tylerjbrooks@gmail.com</a>>:<br>
> I am using a Hi35xx camera processor from HiSilicon. It is an Arm9 with a<br>
> video pipeline bolted on the side. At one end of the pipeline is the CMOS<br>
> sensor. At the other end is a H264 encoder. When I turn on the pipeline,<br>
> the encoder outputs H264 NAL packets like this:<br>
><br>
>Â Â Â frame0: <SPS>,<PPS>,<SEI>,<key frame><br>
>Â Â Â frame1: <delta frame><br>
>Â Â Â frame2: <delta frame><br>
>Â Â Â ...<br>
>Â Â Â frameN: <delta frame><br>
>Â Â Â frameN+1: <SPS>,<PPS>,<SEI><key frame><br>
>Â Â Â frameN+2: <delta frame><br>
>Â Â Â frameN+3: <delta frame><br>
>Â Â Â ...<br>
>Â Â Â etc.<br>
><br>
> I am turning that into HLS clips by doing the following (pseudo code for<br>
> clarity) :<br>
><br>
>Â Â Â av_register_all();<br>
>Â Â Â avformat_network_init();<br>
><br>
>Â Â Â avformat_alloc_output_<wbr>context2(&ctx_out, NULL, "hls", "./foo.m3u8");<br>
><br>
>Â Â Â strm_out = avformat_new_stream(ctx_out, NULL);<br>
><br>
>Â Â Â codec_out = strm_out->codecpar;<br>
>Â Â Â codec_out->codec_id = AV_CODEC_ID_H264;<br>
>Â Â Â codec_out->codec_type = AVMEDIA_TYPE_VIDEO;<br>
>Â Â Â codec_out->width = encoder_width;<br>
>Â Â Â codec_out->height = encoder_height;<br>
>Â Â Â codec_out->bit_rate = encoder_bitrate;<br>
>Â Â Â codec_out->codec_tag = 0;<br>
><br>
>Â Â Â avformat_write_header(ctx_out, NULL);<br>
><br>
>Â Â Â while(get_packet_from_<wbr>pipeline_encoder(&encoder_<wbr>packet)) {<br>
>Â Â Â Â AVPacket pkt;<br>
>Â Â Â Â av_init_packet(&pkt);<br>
>Â Â Â Â pkt.stream_index = 0;<br>
><br>
>Â Â Â Â pkt.dts = AV_NOPTS_VALUE;<br>
>Â Â Â Â pkt.pts = AV_NOPTS_VALUE;<br>
>Â Â Â Â pkt.duration = (1000000/FRAMERATE);Â Â // frame rate in microseconds<br>
><br>
>Â Â Â Â pkt.data = encoder_packet.data;<br>
>Â Â Â Â pkt.size = encoder_packet.size;<br>
><br>
>Â Â Â Â if (is_keyframe(&encoder_packet)) {<br>
>Â Â Â Â Â pkt.flags |= AV_PKT_FLAG_KEY;<br>
>Â Â Â Â }<br>
><br>
>Â Â Â Â av_write_frame(ctx_out, &pkt);<br>
>Â Â Â }<br>
><br>
>Â Â Â av_write_trailer(ctx_out);<br>
>Â Â Â avformat_free_context(ctx_out)<wbr>;<br>
><br>
> This seems to work fine except that the resulting HLS frame rate is not<br>
> right. Of course, this happens because I am not setting the pts/dts stuff<br>
> correctly and ffmpeg lets me know that. So I have two quetions:<br>
><br>
>Â 1. Am I going about this right?<br>
>Â 2. How can I set the pts/dts stuff correctly?<br>
><br>
> The encoder is giving me packets and I am submitting them as frames.<br>
<br>
> Those `<SPS>, <PPS> and <SEI>` packets are really out of band<br>
> data and don't really have a timestamp.<br>
<br>
</div></div>If the encoder does not tell you the timestamps of the encoded<br>
frames, how are you - unrelated to FFmpeg ! - supposed to<br>
write files, no matter the format?<br>
<br>
Carl Eugen<br>
______________________________<wbr>_________________<br>
Libav-user mailing list<br>
<a href="mailto:Libav-user@ffmpeg.org">Libav-user@ffmpeg.org</a><br>
<a href="http://ffmpeg.org/mailman/listinfo/libav-user" rel="noreferrer" target="_blank">http://ffmpeg.org/mailman/<wbr>listinfo/libav-user</a><br>
</blockquote></div><br></div>