[FFmpeg-user] A53 Captions Garbled If Frame Duplication Occurs

[Shane Warren]

Devin,

Thanks for the tips, I tried using the fps filter (and omitting -v) but this stream I'm testing on still gets garbled captions using either method. I have tried hardware encoding (h264_nvenc) and software encoding (libx264) both produce garbled captions using the fps filter.

If I use an old ffmpeg from before that checkin (which I noted earlier), both the fps filter and -v produce valid captions on output. Anything past that checkin will not produce valid captions for this file.

Here is my software encoding command line:

ffmpeg -y -threads 2 -nostats  -loglevel verbose  -probesize 5M -filter_threads 4  -fflags +genpts -fflags discardcorrupt  -i "/tmp/sample-low-framerate.ts"  -map "0:v:0" -map "0:a:0"  -af "aresample=async=10000,volume=1.00" -c:a:0 aac -threads 2 -ac:a:0 2 -ar:a:0 48000 -b:a:0 192k  -flush_packets 0 -vf "fps=fps=29.97"  -c:v libx264 -b:v 5500k -minrate:v 5500k -maxrate:v 5500k -bufsize:v 11000k -a53cc 1  -profile:v high -level:v 4.2 -bf:v 0 -g:v 15  -f mpegts  /tmp/subtitle-encode-test.ts

I made a short 30 second capture of the transport stream that causes this issue:

https://drive.google.com/file/d/1IJwqxG92LH4fDyoZp3qhBL50PErKxIYN/view?usp=drive_link

Here is the transcoded output where the captions are messed up:

https://drive.google.com/file/d/1cbpK5NVBHY73OI9rtbO9rvjeIWV_y0tI/view?usp=drive_link

Hopefully someone can identify what I'm doing wrong, I'm leaning towards a bug that has existing for a while though.

-----Original Message-----
From: ffmpeg-user <ffmpeg-user-bounces at ffmpeg.org> On Behalf Of Devin Heitmueller
Sent: Tuesday, May 7, 2024 10:21 AM
To: FFmpeg user questions <ffmpeg-user at ffmpeg.org>
Subject: Re: [FFmpeg-user] A53 Captions Garbled If Frame Duplication Occurs

This email originated from outside Innovative Systems. Do not click links or open attachments unless you recognize the sender and know the content is safe.


Hi Shane,

On Fri, May 3, 2024 at 5:48 PM Shane Warren <shanew at innovsys.com> wrote:
>
> I ingest some multicast video that has a53 captions embedded in h264 video, the video claims to have a framerate of 29.97 but it must not quite be that because ffmpeg duplicates frames when transcoding this stream. The output stream has a53 captions, but they appear to be somewhat garbled, like some letters are duplicated, sometimes.
>
> I decided to git bisect since I knew this worked in a much older ffmpeg binary I had. I finally found the commit that broke this:
>
> a11ab647304307d0340fd052e04cf638b6bef309 : fftools/ffmpeg: share the
> code encoding a single frame between video and audio
>
> Call do_video_stats() for every video packet produced by the encoder,
> rather than for every frame sent to the encoder.
>
> ----------------------------------------------------------------------
> --
>
> The key part that was missed in this commit was calling this directly after avcodec_send_frame:
>
> // Make sure Closed Captions will not be duplicated
> av_frame_remove_side_data(in_picture, AV_FRAME_DATA_A53_CC);
>
> I fixed this in that commit, and captions worked again. I tried to fix it the same way in a the current ffmpeg source by changing:
>
> ffmpeg_enc.c encode_frame:
>
>     ret = avcodec_send_frame(enc, frame);
>     if (ret < 0 && !(ret == AVERROR_EOF && !frame)) {
>         av_log(ost, AV_LOG_ERROR, "Error submitting %s frame to the encoder\n",
>                type_desc);
>         return ret;
>     }
>
>     +if (frame != NULL)
>     +{
>     +   av_frame_remove_side_data(frame, AV_FRAME_DATA_A53_CC);
>     +}
>
> But that didn't fix the problem, I suspect this is needed when a frame is duplicated, but I can't figure out the place to put that.
>
> Anyone have any ideas what I need to change to get captions working in the current ffmpeg source?
> _______________________________________________
> ffmpeg-user mailing list
> ffmpeg-user at ffmpeg.org
> https://ffmp/
> eg.org%2Fmailman%2Flistinfo%2Fffmpeg-user&data=05%7C02%7Cshanew%40inno
> vsys.com%7C4b7d6324d2b141517be908dc6ea95eff%7C7a48ce45ee974a95ac183390
> 878a179b%7C0%7C0%7C638506920940580581%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi
> MC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7
> C&sdata=9664azc5fc%2BwKMnBGFjJVNFk9gM0Tx0u2mwLwho%2FZX0%3D&reserved=0
>
> To unsubscribe, visit link above, or email
> ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".

This was discussed last May in the following thread:

https://ffmpeg.org/pipermail/ffmpeg-devel/2023-June/310534.html

The easiest thing for you to do is use the "fps" filter (i.e. -vf
fps=60000/1001) rather than using "-r".  That makes use of the ccfifo framework to ensure that the data is properly preserved.  The hack to do it with "-r" actually doesn't work correctly in a number of cases (e.g. reducing the framerate), and in fact even with upconverting to
59.94 the data doesn't actually conform to the spec, although it works in some decoders (specifically you should have every frame with a cc_count of 10, when in fact you have it alternating between 20 and being absent).

Devin

--
Devin Heitmueller, Senior Software Engineer LTN Global Communications
o: +1 (301) 363-1001
w: https://ltnglobal.com/  e: devin.heitmueller at ltnglobal.com _______________________________________________
ffmpeg-user mailing list
ffmpeg-user at ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email ffmpeg-user-request at ffmpeg.org with subject "unsubscribe".