Hey guys,<div><br></div><div>I'm doing some simply video analysis in which I walk through frames of a video and do some calculations on the pixel data. I started noticing some issues with frame timing so I tried to compare my results with a simple thumbnail extraction script using ffmpeg.exe.</div>
<div><br></div><div><b>The thumbnails from my code did not visually line up with the thumbnails generated by ffmpeg.exe.</b></div><div><br><div>For example, without frame skipping, frame #225 in my code visually lines up to #250 from ffmpeg.exe. When skipping every 10 frames (in my "real" video analysis"), #280 visually lines up to #250 from ffmpeg.exe! I'd like to think I'm just calculating my frame number wrong, but even the total frames generated by both are different.</div>
<div><br></div><div>I recently upgraded ffmpeg from 0.8.3 to 0.11.1 and I don't remember these issues before. I'm going to check 0.8.3 now.</div><div><br></div><div><b>More info</b></div><div><div>- Link to video: <a href="https://dl.dropbox.com/u/28441949/sample%20video.MP4">https://dl.dropbox.com/u/28441949/sample%20video.MP4</a></div>
<div>- Total frames reported using avStream->nb_frames: <span style="background-color:rgb(255,255,0)"><b>480</b></span></div><div><span style>- Total frames by iterating through </span>av_read_frame(): <span style="background-color:rgb(255,255,0)"><b>480</b></span></div>
<div><div>- Total frames if I save each frame as a jpg (<font face="courier new, monospace">ffmpeg.exe -i input.mp4 -vframes 1 -f image2 –sameq -vcodec mjpeg frame-%05d.jpg)</font>: <span style="background-color:rgb(255,255,0)"><b>510</b></span></div>
<div>- If I turn AV_LOG_DEBUG on, I see several messages like this: <font face="courier new, monospace">[h264 @ 06e420c0] Frame num gap</font></div><br class="Apple-interchange-newline"></div><div><span style="background-color:rgb(255,255,0)"><div>
<div><font face="courier new, monospace">int SimplifiedScan(const char* inputfile)</font></div><div><font face="courier new, monospace">{</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>AVFormatContext *pFormatCtx;</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>AVStream<span class="Apple-tab-span" style="white-space:pre"> </span>*pStream;</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>AVFrame *pFrameYUV;</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>AVCodecContext *pCodecCtx;</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>AVCodec<span class="Apple-tab-span" style="white-space:pre"> </span>*pCodec;</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>AVPacket packet;</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>int numBytes;</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>uint8_t *buffer;</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>int<span class="Apple-tab-span" style="white-space:pre"> </span>width;</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>int<span class="Apple-tab-span" style="white-space:pre"> </span>height;</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>int<span class="Apple-tab-span" style="white-space:pre"> </span>av_errno;</font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font face="courier new, monospace"> </font></span></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>int isFrameFinished = 0;</font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font face="courier new, monospace"> </font></span></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>int frameLocation = 0;</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>av_register_all();</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font face="courier new, monospace"> </font></span></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Open video file</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>pFormatCtx = avformat_alloc_context();</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>av_errno = avformat_open_input(&pFormatCtx, inputfile, NULL, NULL);</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>if (av_errno < 0)</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>{</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>return ErrorOpeningFile;</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>}</font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font face="courier new, monospace"> </font></span></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Retrieve stream information</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>av_errno = avformat_find_stream_info(pFormatCtx, NULL);</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>if (av_errno < 0)</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>{</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>return ErrorFindingStreamInformation;</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>}</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Find the first video stream</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>int videoStreamIndex = -1;</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>for(unsigned int i=0; i<pFormatCtx->nb_streams; i++)</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>{</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO)</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>{</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>videoStreamIndex = i;</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>break;</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>}</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>}</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>if(videoStreamIndex == -1)</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>{</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>pLog->Write("AVHelper::GetAVStream(): Couldn't find a video stream");</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>return ErrorFindingVideoStream;</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>}</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>pStream = pFormatCtx->streams[videoStreamIndex];</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Get a pointer to the codec context for the video stream</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>pCodecCtx = pStream->codec;</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Find the decoder for the video stream</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>pCodec=avcodec_find_decoder(pCodecCtx->codec_id);</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>if(pCodec==NULL)</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>{</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>return UnsupportedCodec;</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>}</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Inform the codec that we can handle truncated bitstreams -- i.e.,</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// bitstreams where frame boundaries can fall in the middle of packets</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>if(pCodec->capabilities & CODEC_CAP_TRUNCATED)</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>pCodecCtx->flags|=CODEC_FLAG_TRUNCATED;</font></div><div><font face="courier new, monospace"><br></font></div><div>
<font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Open codec</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>av_errno = avcodec_open2(pCodecCtx, pCodec, NULL);</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>if (av_errno<0)</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>{</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>return ErrorOpeningCodec;</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>}</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>width = pCodecCtx->width;</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>height = pCodecCtx->height;</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Allocate an AVFrame structure</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>pFrameYUV=avcodec_alloc_frame();</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>if(pFrameYUV==NULL)</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>{</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>return NULL;</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>}</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Determine required buffer size and allocate buffer</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>numBytes=avpicture_get_size(PIX_FMT_YUV420P, width, height);</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>buffer=(uint8_t *)av_malloc(numBytes*sizeof(uint8_t));</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Assign appropriate parts of buffer to image planes in pFrameYUV</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>avpicture_fill((AVPicture *)pFrameYUV, buffer, PIX_FMT_YUV420P, width, height);</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font face="courier new, monospace"> </font></span></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>int frameSkip = 1;</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font face="courier new, monospace"> </font></span></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>while (av_read_frame(pFormatCtx, &packet) >= 0) </font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>{</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Is this a packet from the video stream?</font></div><div>
<font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>if (packet.stream_index == videoStreamIndex && frameLocation++ % frameSkip == 0)</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>{</font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font face="courier new, monospace"> </font></span></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Decode video frame</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>avcodec_decode_video2(pCodecCtx, pFrameYUV, &isFrameFinished, &packet);</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">#if DEBUG</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>WriteJPEG(pCodecCtx, pFrameYUV, frameLocation);</font></div>
<div><font face="courier new, monospace">#endif</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Did we get a video frame?</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>if (isFrameFinished) {</font></div><div><span class="Apple-tab-span" style="white-space:pre"><font face="courier new, monospace"> </font></span></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// done scanning frame</font></div><div><font face="courier new, monospace">#if DEBUG</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>pLog->Write("Frame finished %d", frameLocation);</font></div>
<div><font face="courier new, monospace">#endif</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>}</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>else</font></div>
<div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>pLog->Write("Frame *NOT* finished %d", frameLocation);</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>} // whether we were supposed to decode the frame</font></div>
<div><span class="Apple-tab-span" style="white-space:pre"><font face="courier new, monospace"> </font></span></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>av_free_packet(&packet);</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>} // for each available frame</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>//delete [] buffer;</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>av_free(buffer);</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Free the YUV frame</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>av_free(pFrameYUV);</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Close the codec</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>avcodec_close(pCodecCtx);</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>// Close the video file</font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>av_close_input_file(pFormatCtx);</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"><span class="Apple-tab-span" style="white-space:pre"> </span>return 0;</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">}</font></div></div><div><br></div></span></div>
<div><br></div><div><br></div><div><br></div><div><b>Ffmpeg version 0.11.1: </b></div><div><div><font face="courier new, monospace">ffmpeg version 0.11.1 Copyright (c) 2000-2012 the FFmpeg developers</font></div>
<div><font face="courier new, monospace"> built on Jun 12 2012 21:10:06 with gcc 4.6.3</font></div><div><font face="courier new, monospace"> configuration: --enable-version3 --disable-w32threads --enable-runtime-cpudete</font></div>
<div><font face="courier new, monospace">ct --enable-avisynth --enable-bzlib --enable-frei0r --enable-libass --enable-lib</font></div><div><font face="courier new, monospace">celt --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libfreetype</font></div>
<div><font face="courier new, monospace">--enable-libgsm --enable-libmp3lame --enable-libnut --enable-libopenjpeg --enabl</font></div><div><font face="courier new, monospace">e-librtmp --enable-libschroedinger --enable-libspeex --enable-libtheora --enable</font></div>
<div><font face="courier new, monospace">-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enabl</font></div><div><font face="courier new, monospace">e-zlib</font></div><div><font face="courier new, monospace"> libavutil 51. 54.100 / 51. 54.100</font></div>
<div><font face="courier new, monospace"> libavcodec 54. 23.100 / 54. 23.100</font></div><div><font face="courier new, monospace"> libavformat 54. 6.100 / 54. 6.100</font></div><div><font face="courier new, monospace"> libavdevice 54. 0.100 / 54. 0.100</font></div>
<div><font face="courier new, monospace"> libavfilter 2. 77.100 / 2. 77.100</font></div><div><font face="courier new, monospace"> libswscale 2. 1.100 / 2. 1.100</font></div><div><font face="courier new, monospace"> libswresample 0. 15.100 / 0. 15.100</font></div>
<div><br></div></div><div><br></div><div>Any thoughts? Thanks guys!</div><div><br></div><div>Simon</div><div><br></div>
</div></div>