[FFmpeg-devel] [PATCH 3/4] h264_metadata: Fix handling of level 1b

[Andreas Rheinhardt]

The earlier logic had two errors:
1. If level 1b for a non-High profile had been guessed, level 1b was not always written to the output, because constraint_set3_flag wasn't set.
2. If a level different from 1b that shares its level_idc with 1b was
intended to be set, constraint_set3_flag wasn't cleared so that the
output could mistakenly be read as 1b (in case none of the High profiles
is in use).

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at googlemail.com>
---
 libavcodec/h264_metadata_bsf.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/libavcodec/h264_metadata_bsf.c b/libavcodec/h264_metadata_bsf.c
index 521bc36b7e..7fe213e8cd 100644
--- a/libavcodec/h264_metadata_bsf.c
+++ b/libavcodec/h264_metadata_bsf.c
@@ -244,7 +244,12 @@ static int h264_metadata_update_sps(AVBSFContext *bsf,
                                        width, height,
                                        sps->vui.max_dec_frame_buffering);
             if (desc) {
-                level_idc = desc->level_idc;
+                if ((desc->level_idc == 11) && desc->constraint_set3_flag)
+                    // This ensures that for level 1b the correct level
+                    // will be inferred below.
+                    level_idc = 9;
+                else
+                    level_idc = desc->level_idc;
             } else {
                 av_log(bsf, AV_LOG_WARNING, "Stream does not appear to "
                        "conform to any level: using level 6.2.\n");
@@ -264,6 +269,15 @@ static int h264_metadata_update_sps(AVBSFContext *bsf,
                 sps->level_idc = 9;
             }
         } else {
+            // If the earlier level was 1b or the new level
+            // is in danger of being mistaken for 1b,
+            // we clear constraint_set3_flag.
+            if ((level_idc        == 11 ||
+                 sps->level_idc   == 11) &&
+                (sps->profile_idc == 66 ||
+                 sps->profile_idc == 77 ||
+                 sps->profile_idc == 88))
+                sps->constraint_set3_flag = 0;
             sps->level_idc = level_idc;
         }
     }
-- 
2.19.0