<div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">Hello All,</span><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">First of all this is my first post to the mailing list so hello all =)</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Recently I have been working with fmpeg in order to render an mp4 using still images via rgba values and a user selected audio file. I have set up the video portion which encodes perfectly. I have used the <a href="https://www.ffmpeg.org/doxygen/trunk/group__lavc__decoding.html#ga834bb1b062fbcc2de4cf7fb93f154a3e" target="_blank" style="color:rgb(70,101,162);font-weight:bold;font-family:Roboto,sans-serif;font-size:14px;line-height:19px">avcodec_decode_audio4</a> function to pull frames from a user selected audio file. Once I get the frame the samples are then stored in <span style="font-family:Menlo;font-size:11px">AV_SAMPLE_FMT_FLTP, so at this point I either have to manually cast the values as a short or use the sampler functionality. I did the first as a proof of concept but in order to make the conversion as flexible as possible I opted for the sampler route. I set my sampler as follows:</span></div>
<div style="font-family:arial,sans-serif;font-size:13px"><span style="font-family:Menlo;font-size:11px"><br></span></div><div><p style="font-family:Menlo;font-size:11px;margin:0px;color:rgb(0,132,0)"><span style="color:rgb(0,0,0)"> </span>/* set options */</p>
<p style="font-family:Menlo;font-size:11px;margin:0px"> <span style="color:rgb(49,89,93)">av_opt_set_int</span> (<span style="color:rgb(79,129,135)">swr_ctx</span>, <span style="color:rgb(209,47,27)">"in_channel_count"</span>, frameIn-><span style="color:rgb(79,129,135)">channels</span>, <span style="color:rgb(39,42,216)">0</span>);</p>
<p style="font-family:Menlo;font-size:11px;margin:0px"> <span style="color:rgb(49,89,93)">av_opt_set_int</span> (<span style="color:rgb(79,129,135)">swr_ctx</span>, <span style="color:rgb(209,47,27)">"in_sample_rate"</span>, frameIn-><span style="color:rgb(79,129,135)">sample_rate</span>, <span style="color:rgb(39,42,216)">0</span>);</p>
<p style="font-family:Menlo;font-size:11px;margin:0px"> <span style="color:rgb(49,89,93)">av_opt_set_sample_fmt</span>(<span style="color:rgb(79,129,135)">swr_ctx</span>, <span style="color:rgb(209,47,27)">"in_sample_fmt"</span>, (<span style="color:rgb(79,129,135)">AVSampleFormat</span>)frameIn-><span style="color:rgb(79,129,135)">format</span>, <span style="color:rgb(39,42,216)">0</span>);</p>
<p style="font-family:Menlo;font-size:11px;margin:0px"> <span style="color:rgb(49,89,93)">av_opt_set_int</span> (<span style="color:rgb(79,129,135)">swr_ctx</span>, <span style="color:rgb(209,47,27)">"out_channel_count"</span>, codecOut-><span style="color:rgb(79,129,135)">channels</span>, <span style="color:rgb(39,42,216)">0</span>);</p>
<p style="font-family:Menlo;font-size:11px;margin:0px"> <span style="color:rgb(49,89,93)">av_opt_set_int</span> (<span style="color:rgb(79,129,135)">swr_ctx</span>, <span style="color:rgb(209,47,27)">"out_sample_rate"</span>, codecOut-><span style="color:rgb(79,129,135)">sample_rate</span>, <span style="color:rgb(39,42,216)">0</span>);</p>
<p style="font-family:Menlo;font-size:11px;margin:0px"> <span style="color:rgb(49,89,93)">av_opt_set_sample_fmt</span>(<span style="color:rgb(79,129,135)">swr_ctx</span>, <span style="color:rgb(209,47,27)">"out_sample_fmt"</span>, codecOut-><span style="color:rgb(79,129,135)">sample_fmt</span>, <span style="color:rgb(39,42,216)">0</span>);</p>
<p style="font-family:Menlo;font-size:11px;margin:0px"><br></p><p style="font-family:Menlo;font-size:11px;margin:0px">my target audio profile is as follows:</p><p style="font-family:Menlo;font-size:11px;margin:0px"><br></p>
<p style="font-family:Menlo;font-size:11px;margin:0px"> (*codec)-><span style="color:rgb(79,129,135)">sample_fmts</span>[<span style="color:rgb(39,42,216)">0</span>] : <span style="color:rgb(49,89,93)">AV_SAMPLE_FMT_FLTP</span>;</p>
<p style="font-family:Menlo;font-size:11px;margin:0px"> c-><span style="color:rgb(79,129,135)">bit_rate</span> = <span style="color:rgb(39,42,216)">64000</span>;</p><p style="font-family:Menlo;font-size:11px;margin:0px">
c-><span style="color:rgb(79,129,135)">sample_rate</span> = <span style="color:rgb(39,42,216)">22050</span>;</p><p style="font-family:Menlo;font-size:11px;margin:0px"> c-><span style="color:rgb(79,129,135)">channels</span> = <span style="color:rgb(39,42,216)">2</span>;</p>
<p style="font-family:Menlo;font-size:11px;margin:0px">
</p><p style="font-family:Menlo;font-size:11px;margin:0px;color:rgb(120,73,42)"><span style="color:rgb(0,0,0)"> c-></span><span style="color:rgb(79,129,135)">channel_layout</span><span style="color:rgb(0,0,0)"> = </span>AV_CH_LAYOUT_STEREO<span style="color:rgb(0,0,0)">;</span></p>
<p style="font-family:Menlo;font-size:11px;margin:0px;color:rgb(120,73,42)"><span style="color:rgb(0,0,0)"><br></span></p><p style="margin:0px"><font color="#000000" face="Menlo"><span style="font-size:11px">I has a hell of a time debugging random black box crashes in the swr_convert file and it turned out to be that I was passing the uint8 ** by value instead of reference so when there was more than one more channel it would crash, but changing my parameter from ->data to &->data[0] solved that. So just in case that might help anyone having that particular problem... </span></font></p>
<p style="margin:0px"><font color="#000000" face="Menlo"><span style="font-size:11px"><br></span></font></p><p style="margin:0px"><font color="#000000" face="Menlo"><span style="font-size:11px"><br></span></font></p><p style="font-family:Menlo;font-size:11px;margin:0px;color:rgb(120,73,42)">
<span style="color:rgb(0,0,0)">..Anyways, Now the sampler works great for a 1 channel file at the same sample rate. When I change the input file to a 2 channel or change the sample rate i still hear the audio but it is very disjointed as if there are gaps in the samples during the encoding process. </span></p>
<p style="font-family:Menlo;font-size:11px;margin:0px;color:rgb(120,73,42)"><span style="color:rgb(0,0,0)"><br></span></p><p style="font-family:Menlo;font-size:11px;margin:0px;color:rgb(120,73,42)"><span style="color:rgb(0,0,0)">Here is the function, where the get_audio_frame() function uses the decode 4 function:</span></p>
<p style="font-family:Menlo;font-size:11px;margin:0px;color:rgb(120,73,42)"><span style="color:rgb(0,0,0)"><br></span></p><p style="margin:0px;font-size:11px;font-family:Menlo"><span style="color:rgb(187,44,162)">static</span> <span style="color:rgb(187,44,162)">int</span> write_audio_frame_flip(<span style="color:rgb(79,129,135)">AVFormatContext</span> *oc, <span style="color:rgb(79,129,135)">OutputStream</span> *ost)</p>
<p style="margin:0px;font-size:11px;font-family:Menlo">{</p><p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(79,129,135)"><span style="color:rgb(0,0,0)"> </span>AVCodecContext<span style="color:rgb(0,0,0)"> *c;</span></p>
<p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(0,132,0)"><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(79,129,135)">AVPacket</span><span style="color:rgb(0,0,0)"> pkt = { </span><span style="color:rgb(39,42,216)">0</span><span style="color:rgb(0,0,0)"> }; </span>// data and size must be 0;</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(79,129,135)">AVFrame</span> *frame;</p><p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(187,44,162)">int</span> ret;</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(187,44,162)">int</span> got_packet;</p><p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(187,44,162)">int</span> dst_nb_samples;</p>
<p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px"> </p><p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(49,89,93)"><span style="color:rgb(0,0,0)"> </span>av_init_packet<span style="color:rgb(0,0,0)">(&pkt);</span></p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> c = ost-><span style="color:rgb(79,129,135)">st</span>-><span style="color:rgb(79,129,135)">codec</span>;</p><p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px">
</p><p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(0,132,0)"><span style="color:rgb(0,0,0)"> </span>//Get decoded audio frame</p><p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(49,89,93)">
<span style="color:rgb(0,0,0)"> frame = </span>get_audio_frame_flip<span style="color:rgb(0,0,0)">(ost);</span></p><p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px"> </p><p style="margin:0px;font-size:11px;font-family:Menlo">
<span style="color:rgb(187,44,162)">if</span> (frame)</p><p style="margin:0px;font-size:11px;font-family:Menlo"> {</p><p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(49,89,93)">SetSampler</span>(c, frame);</p>
<p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px"> </p><p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(0,132,0)"><span style="color:rgb(0,0,0)"> </span>// convert samples from native format to destination codec format, using the resampler</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(187,44,162)">if</span> (<span style="color:rgb(79,129,135)">swr_ctx</span>)</p><p style="margin:0px;font-size:11px;font-family:Menlo"> {</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(187,44,162)">if</span> (ret < <span style="color:rgb(39,42,216)">0</span>) {</p><p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(209,47,27)">
<span style="color:rgb(0,0,0)"> </span><span style="color:rgb(61,29,129)">fprintf</span><span style="color:rgb(0,0,0)">(</span><span style="color:rgb(120,73,42)">stderr</span><span style="color:rgb(0,0,0)">, </span>"Could not allocate destination samples\n"<span style="color:rgb(0,0,0)">);</span></p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(187,44,162)">return</span> <span style="color:rgb(39,42,216)">0</span>;</p><p style="margin:0px;font-size:11px;font-family:Menlo">
}</p><p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px"> </p><p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(209,47,27)"><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(120,73,42)">LOGD</span><span style="color:rgb(0,0,0)">(</span>"Scaling frame->sample_rate=%d frame->nb_samples=%d c->sample_rate=%d"<span style="color:rgb(0,0,0)">, frame-></span><span style="color:rgb(79,129,135)">sample_rate</span><span style="color:rgb(0,0,0)">, frame-></span><span style="color:rgb(79,129,135)">nb_samples</span><span style="color:rgb(0,0,0)">, c-></span><span style="color:rgb(79,129,135)">sample_rate</span><span style="color:rgb(0,0,0)">);</span></p>
<p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px"> </p><p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(0,132,0)"><span style="color:rgb(0,0,0)"> </span>// compute destination number of samples</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> dst_nb_samples = <span style="color:rgb(49,89,93)">av_rescale_rnd</span>(<span style="color:rgb(49,89,93)">swr_get_delay</span>(<span style="color:rgb(79,129,135)">swr_ctx</span>, frame-><span style="color:rgb(79,129,135)">sample_rate</span>) + frame-><span style="color:rgb(79,129,135)">nb_samples</span>,</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> c-><span style="color:rgb(79,129,135)">sample_rate</span>, frame-><span style="color:rgb(79,129,135)">sample_rate</span>, <span style="color:rgb(49,89,93)">AV_ROUND_UP</span>);</p>
<p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px"> </p><p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(0,132,0)"><span style="color:rgb(0,0,0)"> </span>//convert the samples</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> ret = <span style="color:rgb(49,89,93)">swr_convert</span>(<span style="color:rgb(79,129,135)">swr_ctx</span>, &ost-><span style="color:rgb(79,129,135)">tmp_frame</span>-><span style="color:rgb(79,129,135)">data</span>[<span style="color:rgb(39,42,216)">0</span>], dst_nb_samples, (<span style="color:rgb(187,44,162)">const</span> <span style="color:rgb(112,61,170)">uint8_t</span> **)&frame-><span style="color:rgb(79,129,135)">data</span>[<span style="color:rgb(39,42,216)">0</span>], dst_nb_samples);</p>
<p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px"> </p><p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(187,44,162)">if</span> (ret < <span style="color:rgb(39,42,216)">0</span>) {</p>
<p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(209,47,27)"><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(120,73,42)">LOGD</span><span style="color:rgb(0,0,0)">(</span>"Error while converting\n"<span style="color:rgb(0,0,0)">);</span></p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(187,44,162)">return</span> <span style="color:rgb(39,42,216)">0</span>;</p><p style="margin:0px;font-size:11px;font-family:Menlo">
}</p><p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px"> </p><p style="margin:0px;font-size:11px;font-family:Menlo"> frame = ost-><span style="color:rgb(79,129,135)">tmp_frame</span>;</p>
<p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px"> </p><p style="margin:0px;font-size:11px;font-family:Menlo"> } <span style="color:rgb(187,44,162)">else</span> {</p><p style="margin:0px;font-size:11px;font-family:Menlo">
dst_nb_samples = frame-><span style="color:rgb(79,129,135)">nb_samples</span>;</p><p style="margin:0px;font-size:11px;font-family:Menlo"> }</p><p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px">
</p><p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(79,129,135)"><span style="color:rgb(0,0,0)"> frame-></span>pts<span style="color:rgb(0,0,0)"> = </span><span style="color:rgb(49,89,93)">av_rescale_q</span><span style="color:rgb(0,0,0)">(</span>samples_count<span style="color:rgb(0,0,0)">, (</span>AVRational<span style="color:rgb(0,0,0)">){</span><span style="color:rgb(39,42,216)">1</span><span style="color:rgb(0,0,0)">, c-></span>sample_rate<span style="color:rgb(0,0,0)">}, c-></span>time_base<span style="color:rgb(0,0,0)">);</span></p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(79,129,135)">samples_count</span> += dst_nb_samples;</p><p style="margin:0px;font-size:11px;font-family:Menlo"> }</p><p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px">
</p><p style="margin:0px;font-size:11px;font-family:Menlo"> ret = <span style="color:rgb(49,89,93)">avcodec_encode_audio2</span>(c, &pkt, frame, &got_packet);</p><p style="margin:0px;font-size:11px;font-family:Menlo">
<span style="color:rgb(187,44,162)">if</span> (ret < <span style="color:rgb(39,42,216)">0</span>) {</p><p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(209,47,27)"><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(61,29,129)">fprintf</span><span style="color:rgb(0,0,0)">(</span><span style="color:rgb(120,73,42)">stderr</span><span style="color:rgb(0,0,0)">, </span>"Error encoding audio frame: %s\n"<span style="color:rgb(0,0,0)">, </span><span style="color:rgb(120,73,42)">av_err2str</span><span style="color:rgb(0,0,0)">(ret));</span></p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(187,44,162)">return</span> <span style="color:rgb(39,42,216)">0</span>;</p><p style="margin:0px;font-size:11px;font-family:Menlo"> }</p>
<p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px"> </p><p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(187,44,162)">if</span> (got_packet) {</p><p style="margin:0px;font-size:11px;font-family:Menlo">
ret = <span style="color:rgb(49,89,93)">write_frame</span>(oc, &c-><span style="color:rgb(79,129,135)">time_base</span>, ost-><span style="color:rgb(79,129,135)">st</span>, &pkt);</p><p style="margin:0px;font-size:11px;font-family:Menlo">
<span style="color:rgb(187,44,162)">if</span> (ret < <span style="color:rgb(39,42,216)">0</span>) {</p><p style="margin:0px;font-size:11px;font-family:Menlo;color:rgb(209,47,27)"><span style="color:rgb(0,0,0)"> </span><span style="color:rgb(61,29,129)">fprintf</span><span style="color:rgb(0,0,0)">(</span><span style="color:rgb(120,73,42)">stderr</span><span style="color:rgb(0,0,0)">, </span>"Error while writing audio frame: %s\n"<span style="color:rgb(0,0,0)">,</span></p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(120,73,42)">av_err2str</span>(ret));</p><p style="margin:0px;font-size:11px;font-family:Menlo"> <span style="color:rgb(187,44,162)">return</span> <span style="color:rgb(39,42,216)">0</span>;</p>
<p style="margin:0px;font-size:11px;font-family:Menlo"> }</p><p style="margin:0px;font-size:11px;font-family:Menlo"> }</p><p style="margin:0px;font-size:11px;font-family:Menlo;min-height:13px"> </p><p style="margin:0px;font-size:11px;font-family:Menlo">
<span style="color:rgb(187,44,162)">return</span> (frame || got_packet) ? <span style="color:rgb(39,42,216)">0</span> : <span style="color:rgb(39,42,216)">1</span>;</p><p style="font-family:Menlo;font-size:11px;margin:0px;color:rgb(120,73,42)">
</p><p style="margin:0px;font-size:11px;font-family:Menlo">}</p></div><div><br></div><div><br></div><div>Thank you for any insight into what I might be doing wrong, this is all very new to me and I am sure that I am misunderstanding some call or concept. </div>
<div><br></div><div>Ryan</div></div>