[Libav-user] Encoding images to h264 and rtp output: SDP file without sprop-parameter-sets does not play

[Rasmus]

Alright, I seem to have solved my problem. The RTP port number must be
even, of which I cannot find clear documentation, but it seems the
live555 library used by VLC modifies the port number to be even
(https://github.com/rgaufman/live555/blob/master/liveMedia/MediaSession.cpp#L696).
If you then send to an odd port, nothing is listening there, or it
might get used for RTCP, not RTP.

So takeaway: Send your RTP packets to an even-numbered port.

---

Also, the line

out_codec_ctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;

Can be added (only when this flag is set will the AVCodecContext's
extradata be filled by libavcodec during X264_init() (in libx264.c)),
but does not have to. The call to avcodec_parameters_from_context()
should come after avcodec_open2(), because the extradata gets filled
during avcodec_open2(), if global header is enabled. Saving the SDP
file with av_sdp_create() then writes sprop-parameter-sets as well.

But this is indeed not necessary, I assume since the same info (SPS
and PPS) can also be sent inside the stream, so the SDP file does not
have to contain this information.


On Mon, 13 Sept 2021 at 15:57, Rasmus <rasmusdiederichsen at gmail.com> wrote:
>
> tl;dr: I try to encode acquired camera frames to h264, send via RTP
> and play this back on another device. SDP file generated by ffmpeg for
> a sample video has info which my own SDP file misses. My SDP file
> plays in ffplay, but not VLC, while both play ffmpeg's SDP file. I am
> suspecting missing sprop-parameter-sets in my SDP file.
>
> Ultimately I want to play this back in VLC.
>
> ---
>
> I am writing code that encodes images to h264 and outputs to an RTP
> server (or client? anyway the part that is listening). I generate an
> SDP file for this.
> - ffplay plays the stream without problem
> - mplayer shows a green box embedded in a larger black box, but I read
> somewhere it only supports mpegts over RTP, so not sure
> - VLC does not play the SDP file.
>
> Now when instead I use some random video and have ffmpeg output an SDP
> file like so
>
> ffmpeg     -re     -i some.mp4     -an     -c:v copy -f rtp -sdp_file
> video.sdp     "rtp://127.0.0.1:5004"
>
> I can see that the generated SDP file – which plays in both ffplay and
> VLC – includes the base64 encoded sprop-parameter-sets field, and
> removing this causes the stream to not play.
>
> > cat video.sdp
> v=0
> o=- 0 0 IN IP4 127.0.0.1
> s=No Name
> c=IN IP4 127.0.0.1
> t=0 0
> a=tool:libavformat 58.76.100
> m=video 5004 RTP/AVP 96
> b=AS:1034
> a=rtpmap:96 H264/90000
> a=fmtp:96 packetization-mode=1;
> sprop-parameter-sets=Z2QANKzZQDAA7fiMBagICAoAAAMAAgAAAwDwHjBjLA==,aOvjyyLA;
> profile-level-id=640034
>
> My own SDP file on the other hand, does not contain this information,
> and VLC hangs for 10s and then stops trying with "no data received".
>
> > cat test.sdp
> v=0
> o=- 0 0 IN IP4 127.0.0.1
> s=No Name
> c=IN IP4 127.0.0.1
> t=0 0
> a=tool:libavformat 58.76.100
> m=video 44499 RTP/AVP 96
> b=AS:2000
> a=rtpmap:96 H264/90000
> a=fmtp:96 packetization-mode=1
>
> So my theory is that my custom code must somehow add this SPS
> information to the SDP file. But despite hours of searching, I could
> not find a structured way to set the extradata field on the AVStream's
> AVCodeParams. The code I'm using is roughly this (I'm sure there's
> unrelated errors in there):
>
> // variables
> std::vector<std::uint8_t> imgbuf;
> AVFormatContext *ofmt_ctx = nullptr;
> AVCodec *out_codec = nullptr;
> AVStream *out_stream = nullptr;
> AVCodecContext *out_codec_ctx = nullptr;
> SwsContext *swsctx = nullptr;
> cv::Mat canvas_;
> unsigned int height_;
> unsigned int width_;
> unsigned int fps_;
> AVFrame *frame_ = nullptr;
>
> AVOutputFormat *format = av_guess_format("rtp", nullptr, nullptr);
> const auto url = std::string("rtp://127.0.0.1:5001");
> avformat_alloc_output_context2(ofmt_ctx, format, format->name, url.c_str());
>
> out_codec = avcodec_find_encoder(AV_CODEC_ID_H264);
> stream = avformat_new_stream(ofmt_ctx, out_codec);
> out_codec_ctx = avcodec_alloc_context3(out_codec);
>
> // not sure if these do anything, did not help in my case
> out_codec_ctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
> ofmt_ctx->flags |= AV_CODEC_FLAG_GLOBAL_HEADER;
>
> // then, for each incoming image:
> while (receive_image) {
>   static bool first_time = true;
>   if (first_time) {
>     // discover necessary params such as image dimensions from the first
>     // received image
>     first_time = false;
>     height_ = image.rows;
>     width_ = image.cols;
>
>     codec_ctx->codec_tag = 0;
>     codec_ctx->bit_rate = 2e6;
>     // does nothing, unfortunately
>     codec_ctx->thread_count = 1;
>     codec_ctx->codec_id = AV_CODEC_ID_H264;
>     codec_ctx->codec_type = AVMEDIA_TYPE_VIDEO;
>     codec_ctx->width = width_;
>     codec_ctx->height = height_;
>     codec_ctx->gop_size = 6;
>     codec_ctx->pix_fmt = AV_PIX_FMT_YUV420P;
>     codec_ctx->framerate = fps_;
>     codec_ctx->time_base = av_inv_q(fps_);
>
>     avcodec_parameters_from_context(stream, out_codec_ctx);
>
>     // this stuff is empty: is that the problem?
>     stream->codecpar->extradata = codec_ctx->extradata;
>     stream->codecpar->extradata_size = codec_ctx->extradata_size;
>
>     AVDictionary *codec_options = nullptr;
>     av_dict_set(&codec_options, "profile", "high", 0);
>     av_dict_set(&codec_options, "preset", "ultrafast", 0);
>     av_dict_set(&codec_options, "tune", "zerolatency", 0);
>
>     // open video encoder
>     avcodec_open2(codec_ctx, codec, &codec_options);
>
>     stream->time_base.num = 1;
>     stream->time_base.den = fps_;
>     avio_open(&(ofmt_ctx->pb), ofmt_ctx->filename, AVIO_FLAG_WRITE);
>
>     /* Write a file for VLC */
>     char buf[200000];
>     AVFormatContext *ac[] = {ofmt_ctx};
>     av_sdp_create(ac, 1, buf, 20000);
>     printf("sdp:\n%s\n", buf);
>     FILE *fsdp = fopen("test.sdp", "w");
>     fprintf(fsdp, "%s", buf);
>     fclose(fsdp);
>
>     swsctx = sws_getContext(width_, height_, AV_PIX_FMT_BGR24, width_, height_,
>                             out_codec_ctx->pix_fmt, SWS_BICUBIC, nullptr,
>                             nullptr, nullptr);
>   }
>
>   if (!frame_) {
>     frame_ = av_frame_alloc();
>
>     std::uint8_t *framebuf = new uint8_t[av_image_get_buffer_size(
>         codec_ctx->pix_fmt, width_, height_, 1)];
>     av_image_fill_arrays(frame_->data, frame_->linesize, framebuf,
>                          codec_ctx->pix_fmt, width, height, 1);
>     frame_->width = width_;
>     frame_->height = height_;
>     frame_->format = static_cast<int>(codec_ctx->pix_fmt);
>     success = avformat_write_header(ofmt_ctx, nullptr);
>   }
>   if (imgbuf.empty()) {
>     imgbuf.resize(height_ * width_ * 3 + 16);
>     canvas_ = cv::Mat(height_, width_, CV_8UC3, imgbuf.data(), width_ * 3);
>   } else {
>     image.copyTo(canvas_);
>   }
>
>   const int stride[] = {static_cast<int>(image.step[0])};
>
>   sws_scale(swsctx, &canvas_.data, stride, 0, canvas_.rows, frame_->data,
>             frame_->linesize);
>   frame_->pts += av_rescale_q(1, out_codec_ctx->time_base, stream->time_base);
>
>   AVPacket pkt = {0};
>   avcodec_send_frame(out_codec_ctx, frame_);
>   avcodec_receive_packet(out_codec_ctx, &pkt);
>   av_interleaved_write_frame(ofmt_ctx, &pkt);
> }
>
> Can anyone offer some advice here?
>
> Best,
> Rasmus