<div dir="ltr"><div>I have been attempting to use the ffmpeg libraries to take as input a RTMP stream consisting of a video stream and an audio stream, and decode the respective streams in order to push the contents out to a website via a websocket. The video portion of this task is complete and works fine without issue. Converting frames of h.264 video to single motion jpeg images and pushing those on without a problem. I am attempting to do something similar with audio, takes frames of aac audio and passing those out through a websocket to be played back in chunks, this portion of the task is proving to be a lot more difficult than I thought it would be. </div><div><br></div><div>The way I am testing this is simply learning the way to decode an encode audio (if necessary) to a file before trying to implement it practically via websockets and even this is proving more difficult than I expected, I found some examples via google, but I can't get any of these to work: </div><div>  </div><div><br></div><div><div>int PullAudioFrames( char input[], char output[] )</div><div>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">     </span>const int MAX_AUDIO_FRAME_SIZE = 192000;</div><div>    bool bRet = false;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">        </span>bool bFileOpened = false;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">      </span>bool bAudioCodecOpen = false;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>bool abort = false;</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">   </span>int nVideoStream = -1;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>int nAudioStream = -1;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>AVDictionary *optionsDict = NULL;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">      </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">       </span>//-- Register all formats and codecs and network tools</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>avcodec_register_all();</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">        </span>av_register_all();<span class="gmail-Apple-tab-span" style="white-space:pre">                                                                                                                                    </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">       </span>avformat_network_init();</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">       </span>//--</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>AVCodecContext *pIn_AudioCodecCtx = NULL;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">      </span>AVCodec *pIn_AudioCodec = NULL;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">        </span>struct SwrContext *au_convert_ctx;</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">    </span>AVFormatContext *pFormatCtx = avformat_alloc_context();</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">        </span>if(avformat_open_input(&pFormatCtx, input, NULL, &optionsDict) == 0 )<span class="gmail-Apple-tab-span" style="white-space:pre">                                 </span>//Open Stream<span class="gmail-Apple-tab-span" style="white-space:pre">                 </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">       </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">              </span>bFileOpened = true;<span class="gmail-Apple-tab-span" style="white-space:pre">   </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">               </span>if(avformat_find_stream_info(pFormatCtx, &optionsDict) >= 0 )<span class="gmail-Apple-tab-span" style="white-space:pre">                                                  </span>//Get Stream information (video, audio, subtitle)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">              </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                      </span>for(int i=0; i<(int)pFormatCtx->nb_streams; i++)<span class="gmail-Apple-tab-span" style="white-space:pre">                                                        </span>//Find the video stream</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                              </span>if(pFormatCtx->streams[i]->codecpar->codec_type==AVMEDIA_TYPE_AUDIO)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                  </span>nAudioStream=i;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">        </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                               </span>if( nAudioStream >= 0 ) </div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                   </span>break;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                 </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                      </span>bool bCleanUp = false;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                 </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                       </span>if( nAudioStream >= 0 )</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                     </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                              </span>pIn_AudioCodec = avcodec_find_decoder( pFormatCtx->streams[nAudioStream]->codecpar->codec_id );</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                               </span>if(pIn_AudioCodec  != NULL)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                           </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                      </span>pIn_AudioCodecCtx = avcodec_alloc_context3(pIn_AudioCodec);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">            </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                       </span>// Fill the codecCtx with the parameters of the codec used in the read file.</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                   </span>if( avcodec_parameters_to_context(pIn_AudioCodecCtx, pFormatCtx->streams[nAudioStream]->codecpar) == 0  )</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                       </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                              </span>if( avcodec_open2(pIn_AudioCodecCtx, pIn_AudioCodec, NULL) >= 0 )<span class="gmail-Apple-tab-span" style="white-space:pre">          </span>//Open Audio codec<span class="gmail-Apple-tab-span" style="white-space:pre">    </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                       </span>bAudioCodecOpen = true;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                </span>else</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                   </span>bCleanUp = true;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                       </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                      </span>else</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                           </span>bCleanUp = true;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                               </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                      </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                      </span>if( bCleanUp ) </div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                       </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                              </span>avcodec_close( pIn_AudioCodecCtx );</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                            </span>avcodec_free_context( &pIn_AudioCodecCtx );</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                        </span>}</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                     </span>if( bAudioCodecOpen )</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                  </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                              </span>au_convert_ctx = swr_alloc();</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                          </span>au_convert_ctx=swr_alloc_set_opts(au_convert_ctx, AV_CH_LAYOUT_STEREO, AV_SAMPLE_FMT_S16, 44100, av_get_default_channel_layout(pIn_AudioCodecCtx->channels), pIn_AudioCodecCtx->sample_fmt , pIn_AudioCodecCtx->sample_rate,0, NULL);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                         </span>swr_init(au_convert_ctx);</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                             </span>AVFrame *pFrame=av_frame_alloc();</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                              </span>AVPacket packet;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                               </span>av_init_packet(&packet);</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                          </span>while( !abort ) </div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                              </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                      </span>av_packet_unref(&packet);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                  </span>av_frame_unref(pFrame);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                        </span>int result = av_read_frame( pFormatCtx, &packet);<span class="gmail-Apple-tab-span" style="white-space:pre">                                                 </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                       </span>if( result < 0 )</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                    </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                              </span>if( result == AVERROR(EAGAIN) )<span class="gmail-Apple-tab-span" style="white-space:pre">                                                                                               </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                       </span>continue;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                              </span>else if( result == AVERROR(EPIPE) )</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                    </span>printf( "[\"Error\", {\"message\" : \"Unable to read Frame pipe error. [%d] [%d]\"}]", result, AVERROR(result) );<span class="gmail-Apple-tab-span" style="white-space:pre">                             </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                               </span>else if (result == AVERROR_EOF)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                        </span>printf( "[\"Error\", {\"message\" : \"Unable to read Frame end of File. [%d] [%d]\"}]", result, AVERROR(result) );</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                             </span>else</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                   </span>printf( "[\"Error\", {\"message\" : \"Unable to read Frame. [%d] [%d]\"}] ", result, AVERROR(result) );</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                </span>abort = true;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                  </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                      </span>else</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                   </span>{<span class="gmail-Apple-tab-span" style="white-space:pre">     </span> </div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                             </span>if(packet.stream_index==nAudioStream)<span class="gmail-Apple-tab-span" style="white-space:pre"> </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                               </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                      </span>int frameFinished=0;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                   </span>int nResult = avcodec_send_packet(pIn_AudioCodecCtx, &packet);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                     </span>if( nResult == 0 )</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                             </span>nResult = avcodec_receive_frame( pIn_AudioCodecCtx, pFrame );</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                 </span>if( nResult == 0 )</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                     </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                              </span>uint8_t *out_buffer = (uint8_t *)av_malloc(MAX_AUDIO_FRAME_SIZE*2);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                            </span>int data_size = av_samples_get_buffer_size(NULL, pIn_AudioCodecCtx->channels, pFrame->nb_samples, pIn_AudioCodecCtx->sample_fmt, 1);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                          </span>swr_convert(au_convert_ctx,&out_buffer, MAX_AUDIO_FRAME_SIZE,(const uint8_t **)pFrame->data , pFrame->nb_samples);</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                          </span>FILE *fOut;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                            </span>if( fopen_s( &fOut, output, "a+" ) == 0 )</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                                </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                                      </span>fwrite(out_buffer, 1, data_size, fOut);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                                        </span>fclose(fOut);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                          </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                                      </span>}<span class="gmail-Apple-tab-span" style="white-space:pre">     </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                               </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                      </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                                      </span>av_packet_unref(&packet);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                          </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                              </span>av_frame_free(&pFrame);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                    </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                      </span>else </div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                         </span>printf( "Unable to open Codec in %s", input );<span class="gmail-Apple-tab-span" style="white-space:pre">              </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">               </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">              </span>else</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                   </span>printf( "No stream information found in %s", input );<span class="gmail-Apple-tab-span" style="white-space:pre">               </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">       </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">      </span>else</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">           </span>printf("Unable to open live video stream %s", input );</div><div><br></div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">     </span>if( bAudioCodecOpen )</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">  </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">              </span>avcodec_close(pIn_AudioCodecCtx);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">              </span>avcodec_free_context(&pIn_AudioCodecCtx);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">          </span>swr_free(&au_convert_ctx);</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">         </span>bAudioCodecOpen  = false;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">     </span>}</div><div><br></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">     </span>if(bFileOpened)</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">        </span>{</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">              </span>if( pFormatCtx )</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">                       </span>avformat_close_input(&pFormatCtx);<span class="gmail-Apple-tab-span" style="white-space:pre">        </span></div><div><span class="gmail-Apple-tab-span" style="white-space:pre">               </span>bFileOpened = false;</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">   </span>}</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">      </span>return 0;</div><div>}</div></div><div><br></div><div><br></div><div>The resulting file is not playable and passing it back through CLI ffmpeg.exe doesn't give a whole lot of useful information why that is. Any help, or point in the right direction would be greatly appreciated. </div><div><br></div><div><br></div></div>