[FFmpeg-devel] [RFC/PATCH]Avoid huge allocations from asf demuxer

[Carl Eugen Hoyos]

Hi!

Using the command line from ticket #1888, asf_read_frame_header() first fills 
asf->packet_obj_size with 0xda327c93, this value is later passed to 
av_new_packet() but av_malloc() refuses to alloc anything.
Next value for asf->packet_obj_size is 0x7ef0fec0, av_malloc allocates 
2129723088 bytes.
Attached patch avoids this and fixes ticket #1888, I don't know enough about 
the asf demuxer to judge if this is an acceptable solution.

Please comment, Carl Eugen
-------------- next part --------------
diff --git a/libavformat/asfdec.c b/libavformat/asfdec.c
index 233b6ca..38f46a6 100644
--- a/libavformat/asfdec.c
+++ b/libavformat/asfdec.c
@@ -1011,6 +1011,7 @@ static int asf_read_frame_header(AVFormatContext *s, AVIOContext *pb)
         asf->packet_obj_size = avio_rl32(pb);
         if (asf->packet_obj_size >= (1 << 24) || asf->packet_obj_size <= 0) {
             av_log(s, AV_LOG_ERROR, "packet_obj_size invalid\n");
+            asf->packet_obj_size = 0;
             return AVERROR_INVALIDDATA;
         }
         asf->packet_frag_timestamp = avio_rl32(pb); // timestamp