[FFmpeg-devel] Fix for ticket 1917

[James Almer]

Commit 160a27c introduced the latm AVOption to the libfdk-aac encoder which makes it encapsulate 
the packets in LATM format before sending them to the muxer.
The LATM muxer doesn't expect this, instead expecting frames with either ADTS headers (Which 
it doesn't support and promptly aborts the muxing upon detection), or the frame headers stored 
inside the AVCodecContext extradata.
The ffmpeg aac encoder always stores the header inside the extradata, libvo-aacenc does it only
when ffmpeg is run with the global_header flag (ADTS headers otherwise), and before commit 160a27c 
libfdk-aac would do the same as libvo-aacenc.

The latm AVOption makes libfdk-aac send frames with no extradata and no ADTS header, which means 
that as soon as the LATM muxer passes the ADTS header check it will try to access the non existent 
extradata, generating a segfault.

I can think of two ways to fix this. The first is to undo commit 160a27c and go back to use the 
global_header flag just like it's done with libvo-aacenc, and let the muxer do the encapsulation.
The second is to make the muxer first check for ADTS headers and then check if there's any 
extradata available. If there isn't, then the frames are already encapsulated and it should write 
the raw packet just like it does when it detects a remuxing of aac_latm.

The attached patch is a way to implement the second option above. 
Which of the two, if any, would be the best solution for this?

Regards.
-------------- next part --------------
diff --git a/libavformat/latmenc.c b/libavformat/latmenc.c
index 233eab8..7e6ad9c 100644
--- a/libavformat/latmenc.c
+++ b/libavformat/latmenc.c
@@ -149,13 +149,14 @@ static int latm_write_packet(AVFormatContext *s, AVPacket *pkt)
     int i, len;
     uint8_t loas_header[] = "\x56\xe0\x00";
 
-    if (s->streams[0]->codec->codec_id == AV_CODEC_ID_AAC_LATM)
-        return ff_raw_write_packet(s, pkt);
-
     if (pkt->size > 2 && pkt->data[0] == 0xff && (pkt->data[1] >> 4) == 0xf) {
         av_log(s, AV_LOG_ERROR, "ADTS header detected - ADTS will not be incorrectly muxed into LATM\n");
         return AVERROR_INVALIDDATA;
     }
+
+    if (s->streams[0]->codec->codec_id == AV_CODEC_ID_AAC_LATM || !s->streams[0]->codec->extradata)
+        return ff_raw_write_packet(s, pkt);
+
     if (pkt->size > 0x1fff)
         goto too_large;