[FFmpeg-devel] [PATCH, v2] lavc/qsvenc: replace assert with error return

[Linjie Fu]

bs->FrameType is not set in MSDK in some cases (mjpeg encode for example),
and assert on a value coming from an external library is not proper.

Add default type check for bs->FrameType, and return invalid data error in function
ff_qsv_encode to avoid using uninitialized value.

Fix #7593.

Signed-off-by: Linjie Fu <linjie.fu at intel.com>
---
[v2]: Add default bs->FrameType check.

 libavcodec/qsvenc.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
index 7f4592f878..917344b60c 100644
--- a/libavcodec/qsvenc.c
+++ b/libavcodec/qsvenc.c
@@ -1337,8 +1337,10 @@ int ff_qsv_encode(AVCodecContext *avctx, QSVEncContext *q,
             pict_type = AV_PICTURE_TYPE_P;
         else if (bs->FrameType & MFX_FRAMETYPE_B || bs->FrameType & MFX_FRAMETYPE_xB)
             pict_type = AV_PICTURE_TYPE_B;
+        else if (bs->FrameType == MFX_FRAMETYPE_UNKNOWN)
+            pict_type = AV_PICTURE_TYPE_NONE;
         else
-            av_assert0(!"Uninitialized pict_type!");
+            return AVERROR_INVALIDDATA;
 
 #if FF_API_CODED_FRAME
 FF_DISABLE_DEPRECATION_WARNINGS
-- 
2.17.1