[FFmpeg-cvslog] avformat/rmdec: Actually return value < 0 on read_header failure

[Andreas Rheinhardt]

ffmpeg | branch: master | Andreas Rheinhardt <andreas.rheinhardt at gmail.com> | Mon Jul 20 21:14:15 2020 +0200| [4cc43d30c387fda30e34b7075670dfb28e868acb] | committer: Andreas Rheinhardt

avformat/rmdec: Actually return value < 0 on read_header failure

The RealMedia demuxer's read_header function initially initializes ret,
the variable designated for the return variable to -1. Afterwards, chunks
of the file are parsed in a loop until an error happens or until the actual
frame data is encountered. If the first function whose return
value is put into ret doesn't fail, then ret contains a value >= 0
(actually == 0) and this is what will be returned if an error is
encountered afterwards.

This is a regression since 35bbc1955a58ba74552c50d9161084644f00bbd3.
Before that, ret had never been overwritten with a nonnegative value.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at gmail.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=4cc43d30c387fda30e34b7075670dfb28e868acb
---

 libavformat/rmdec.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/libavformat/rmdec.c b/libavformat/rmdec.c
index a36e693ab2..6851b7e1f4 100644
--- a/libavformat/rmdec.c
+++ b/libavformat/rmdec.c
@@ -538,7 +538,7 @@ static int rm_read_header(AVFormatContext *s)
     unsigned int data_off = 0, indx_off = 0;
     char buf[128], mime[128];
     int flags = 0;
-    int ret = -1;
+    int ret;
     unsigned size, v;
     int64_t codec_pos;
 
@@ -554,6 +554,7 @@ static int rm_read_header(AVFormatContext *s)
     avio_skip(pb, tag_size - 8);
 
     for(;;) {
+        ret = AVERROR_INVALIDDATA;
         if (avio_feof(pb))
             goto fail;
         tag = avio_rl32(pb);
@@ -619,8 +620,9 @@ static int rm_read_header(AVFormatContext *s)
                 avio_seek(pb, codec_pos + size, SEEK_SET);
             } else {
                 avio_skip(pb, -4);
-                if (ff_rm_read_mdpr_codecdata(s, s->pb, st, st->priv_data,
-                                              size, mime) < 0)
+                ret = ff_rm_read_mdpr_codecdata(s, s->pb, st, st->priv_data,
+                                                size, mime);
+                if (ret < 0)
                     goto fail;
             }