[FFmpeg-devel] [PATCH 03/11] decklink: Introduce support for capture of multiple audio streams
Marton Balint
cus at passwd.hu
Fri Jan 12 20:41:22 EET 2018
On Mon, 8 Jan 2018, Devin Heitmueller wrote:
> Add support for the ability to capture all audio pairs available
> to the capture hardware. Each pair is exposed as a different audio
> stream, which matches up with the most common use cases for the
> broadcast space (i.e. where there is one stereo pair per audio
> language).
>
> To support the existing use case where multi-channel audio can be
> captured (i.e. 7.1), we introduced a new configuration option, which
> defaults to the existing behavior.
>
> Updated to reflect comments from Carl Eugen Hoyos <ceffmpeg at gmail.com>,
> Aaron Levinson <alevinsn at levland.net>, and Matthias Hunstock
> <atze at fem.tu-ilmenau.de>.
>
> Signed-off-by: Devin Heitmueller <dheitmueller at ltnglobal.com>
> ---
> doc/indevs.texi | 8 ++-
> libavdevice/decklink_common.cpp | 12 ++++
> libavdevice/decklink_common.h | 8 ++-
> libavdevice/decklink_common_c.h | 6 ++
> libavdevice/decklink_dec.cpp | 136 +++++++++++++++++++++++++++++++---------
> libavdevice/decklink_dec_c.c | 3 +
> 6 files changed, 142 insertions(+), 31 deletions(-)
>
> diff --git a/doc/indevs.texi b/doc/indevs.texi
> index 56066bf..4760d70 100644
> --- a/doc/indevs.texi
> +++ b/doc/indevs.texi
> @@ -278,9 +278,15 @@ For SD sources, ffmpeg needs to be compiled with @code{--enable-libzvbi}. For
> HD sources, on older (pre-4K) DeckLink card models you have to capture in 10
> bit mode.
>
> + at item audio_mode
> +Defines whether to capture a bundle of audio channels (the number of which is determined
> +by the channels argument), or whether to capture a number of audio pairs (the number of
> +which is determined by the maximum number of pairs supported by the card). Must be
> + at samp{bundled} or @samp{pairs}. Defaults to @samp{bundled}.
IMHO it makes more sense to always respect the requested number of capture
channels. If you want to capture all channels, you can define a special
constant (e.g.: all) which maps to -1 as the number of channels, and
modify it according to the determined number of available maximum
channels.
> +
> @item channels
> Defines number of audio channels to capture. Must be @samp{2}, @samp{8} or @samp{16}.
> -Defaults to @samp{2}.
> +Defaults to @samp{2}. This parameter is ignored if audio_mode is set to pairs.
>
> @item duplex_mode
> Sets the decklink device duplex mode. Must be @samp{unset}, @samp{half} or @samp{full}.
> diff --git a/libavdevice/decklink_common.cpp b/libavdevice/decklink_common.cpp
> index c432189..e7daa63 100644
> --- a/libavdevice/decklink_common.cpp
> +++ b/libavdevice/decklink_common.cpp
> @@ -446,6 +446,7 @@ int ff_decklink_init_device(AVFormatContext *avctx, const char* name)
> struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data;
> struct decklink_ctx *ctx = (struct decklink_ctx *)cctx->ctx;
> IDeckLink *dl = NULL;
> + int64_t maxAudioChannels;
> IDeckLinkIterator *iter = CreateDeckLinkIteratorInstance();
> if (!iter) {
> av_log(avctx, AV_LOG_ERROR, "Could not create DeckLink iterator\n");
> @@ -479,5 +480,16 @@ int ff_decklink_init_device(AVFormatContext *avctx, const char* name)
> return AVERROR_EXTERNAL;
> }
>
> + if (ctx->attr->GetInt(BMDDeckLinkMaximumAudioChannels, &maxAudioChannels) != S_OK) {
> + av_log(avctx, AV_LOG_WARNING, "Could not determine number of audio channels\n");
> + ctx->max_audio_channels = 0;
I think you can return failure here.
> + } else {
> + ctx->max_audio_channels = maxAudioChannels;
> + }
> + if (ctx->max_audio_channels > DECKLINK_MAX_AUDIO_CHANNELS) {
> + av_log(avctx, AV_LOG_WARNING, "Decklink card reported support for more channels than ffmpeg supports\n");
> + ctx->max_audio_channels = DECKLINK_MAX_AUDIO_CHANNELS;
> + }
> +
> return 0;
> }
> diff --git a/libavdevice/decklink_common.h b/libavdevice/decklink_common.h
> index 143bbb9..b262780 100644
> --- a/libavdevice/decklink_common.h
> +++ b/libavdevice/decklink_common.h
> @@ -37,6 +37,10 @@
> #define DECKLINK_BOOL bool
> #endif
>
> +/* Maximum number of channels possible across variants of Blackmagic cards.
> + Actual number for any particular model of card may be lower */
This is actually the maximum number of channels FFMPEG supports.
> +#define DECKLINK_MAX_AUDIO_CHANNELS 32
> +
> class decklink_output_callback;
> class decklink_input_callback;
>
> @@ -71,6 +75,7 @@ struct decklink_ctx {
> int bmd_height;
> int bmd_field_dominance;
> int supports_vanc;
> + int max_audio_channels;
>
> /* Capture buffer queue */
> AVPacketQueue queue;
> @@ -85,7 +90,8 @@ struct decklink_ctx {
> int64_t last_pts;
> unsigned long frameCount;
> unsigned int dropped;
> - AVStream *audio_st;
> + AVStream *audio_st[DECKLINK_MAX_AUDIO_CHANNELS];
> + int num_audio_streams;
> AVStream *video_st;
> AVStream *teletext_st;
> uint16_t cdp_sequence_num;
> diff --git a/libavdevice/decklink_common_c.h b/libavdevice/decklink_common_c.h
> index 368ac25..3a21bae 100644
> --- a/libavdevice/decklink_common_c.h
> +++ b/libavdevice/decklink_common_c.h
> @@ -30,6 +30,11 @@ typedef enum DecklinkPtsSource {
> PTS_SRC_WALLCLOCK = 4,
> } DecklinkPtsSource;
>
> +typedef enum DecklinkAudioMode {
> + AUDIO_MODE_BUNDLED = 0,
> + AUDIO_MODE_PAIRS = 1,
> +} DecklinkAudioMode;
> +
> struct decklink_cctx {
> const AVClass *cclass;
>
> @@ -42,6 +47,7 @@ struct decklink_cctx {
> double preroll;
> int v210;
> int audio_channels;
> + int audio_mode;
> int audio_depth;
> int duplex_mode;
> DecklinkPtsSource audio_pts_source;
> diff --git a/libavdevice/decklink_dec.cpp b/libavdevice/decklink_dec.cpp
> index 94dae26..6c1ff82 100644
> --- a/libavdevice/decklink_dec.cpp
> +++ b/libavdevice/decklink_dec.cpp
> @@ -627,9 +627,56 @@ static int64_t get_pkt_pts(IDeckLinkVideoInputFrame *videoFrame,
> return pts;
> }
>
> +static int setup_audio(AVFormatContext *avctx)
> +{
> + struct decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data;
> + struct decklink_ctx *ctx = (struct decklink_ctx *)cctx->ctx;
> + AVStream *st;
> + int ret = 0;
> +
> + if (cctx->audio_mode == AUDIO_MODE_BUNDLED) {
> + st = avformat_new_stream(avctx, NULL);
> + if (!st) {
> + av_log(avctx, AV_LOG_ERROR, "Cannot add stream\n");
> + ret = AVERROR(ENOMEM);
> + goto error;
> + }
> + st->codecpar->codec_type = AVMEDIA_TYPE_AUDIO;
> + st->codecpar->codec_id = ctx->audio_depth == 32 ? AV_CODEC_ID_PCM_S32LE : AV_CODEC_ID_PCM_S16LE;
> + st->codecpar->sample_rate = bmdAudioSampleRate48kHz;
> + st->codecpar->channels = cctx->audio_channels;
> + st->codecpar->channel_layout = AV_CH_LAYOUT_STEREO;
> + avpriv_set_pts_info(st, 64, 1, 1000000); /* 64 bits pts in us */
> + ctx->audio_st[0] = st;
> + ctx->num_audio_streams++;
> + } else {
> + for (int i = 0; i < ctx->max_audio_channels / 2; i++) {
> + st = avformat_new_stream(avctx, NULL);
> + if (!st) {
> + av_log(avctx, AV_LOG_ERROR, "Cannot add stream %d\n", i);
> + ret = AVERROR(ENOMEM);
> + goto error;
> + }
> + st->codecpar->codec_type = AVMEDIA_TYPE_AUDIO;
> + st->codecpar->codec_id = ctx->audio_depth == 32 ? AV_CODEC_ID_PCM_S32LE : AV_CODEC_ID_PCM_S16LE;
> + st->codecpar->sample_rate = bmdAudioSampleRate48kHz;
> + st->codecpar->channels = 2;
> + st->codecpar->channel_layout = AV_CH_LAYOUT_STEREO;
> + avpriv_set_pts_info(st, 64, 1, 1000000); /* 64 bits pts in us */
> + ctx->audio_st[i] = st;
> + ctx->num_audio_streams++;
> + }
> + cctx->audio_channels = ctx->max_audio_channels;
> + }
> +
> +error:
> + return ret;
> +}
> +
> HRESULT decklink_input_callback::VideoInputFrameArrived(
> IDeckLinkVideoInputFrame *videoFrame, IDeckLinkAudioInputPacket *audioFrame)
> {
> + decklink_cctx *cctx = (struct decklink_cctx *)avctx->priv_data;
> void *frameBytes;
> void *audioFrameBytes;
> BMDTimeValue frameTime;
> @@ -777,24 +824,57 @@ HRESULT decklink_input_callback::VideoInputFrameArrived(
>
> // Handle Audio Frame
> if (audioFrame) {
> - AVPacket pkt;
> - BMDTimeValue audio_pts;
> - av_init_packet(&pkt);
> -
> - //hack among hacks
> - pkt.size = audioFrame->GetSampleFrameCount() * ctx->audio_st->codecpar->channels * (ctx->audio_depth / 8);
> audioFrame->GetBytes(&audioFrameBytes);
> - audioFrame->GetPacketTime(&audio_pts, ctx->audio_st->time_base.den);
> - pkt.pts = get_pkt_pts(videoFrame, audioFrame, wallclock, ctx->audio_pts_source, ctx->audio_st->time_base, &initial_audio_pts);
> - pkt.dts = pkt.pts;
>
> - //fprintf(stderr,"Audio Frame size %d ts %d\n", pkt.size, pkt.pts);
> - pkt.flags |= AV_PKT_FLAG_KEY;
> - pkt.stream_index = ctx->audio_st->index;
> - pkt.data = (uint8_t *)audioFrameBytes;
> + if (cctx->audio_mode == AUDIO_MODE_BUNDLED) {
> + AVPacket pkt;
> + BMDTimeValue audio_pts;
> + av_init_packet(&pkt);
>
> - if (avpacket_queue_put(&ctx->queue, &pkt) < 0) {
> - ++ctx->dropped;
> + //hack among hacks
> + pkt.size = audioFrame->GetSampleFrameCount() * ctx->audio_st[0]->codecpar->channels * (ctx->audio_depth / 8);
> + audioFrame->GetBytes(&audioFrameBytes);
> + audioFrame->GetPacketTime(&audio_pts, ctx->audio_st[0]->time_base.den);
> + pkt.pts = get_pkt_pts(videoFrame, audioFrame, wallclock, ctx->audio_pts_source, ctx->audio_st[0]->time_base, &initial_audio_pts);
> + pkt.dts = pkt.pts;
> +
> + pkt.flags |= AV_PKT_FLAG_KEY;
> + pkt.stream_index = ctx->audio_st[0]->index;
> + pkt.data = (uint8_t *)audioFrameBytes;
> +
> + if (avpacket_queue_put(&ctx->queue, &pkt) < 0) {
> + ++ctx->dropped;
> + }
> + } else {
> + /* Need to deinterleave audio */
> + int audio_offset = 0;
> + int audio_stride = cctx->audio_channels * ctx->audio_depth / 8;
> + for (int i = 0; i < ctx->num_audio_streams; i++) {
> + int sample_size = ctx->audio_st[i]->codecpar->channels *
> + ctx->audio_st[i]->codecpar->bits_per_coded_sample / 8;
> + AVPacket pkt;
> + int ret = av_new_packet(&pkt, audioFrame->GetSampleFrameCount() * sample_size);
> + if (ret != 0)
> + continue;
> +
> + pkt.pts = get_pkt_pts(videoFrame, audioFrame, wallclock, ctx->audio_pts_source,
> + ctx->audio_st[i]->time_base, &initial_audio_pts);
> + pkt.dts = pkt.pts;
> + pkt.flags |= AV_PKT_FLAG_KEY;
> + pkt.stream_index = ctx->audio_st[i]->index;
> +
> + uint8_t *audio_in = ((uint8_t *) audioFrameBytes) + audio_offset;
> + for (int x = 0; x < pkt.size; x += sample_size) {
> + memcpy(&pkt.data[x], audio_in, sample_size);
> + audio_in += audio_stride;
> + }
> +
> + if (avpacket_queue_put(&ctx->queue, &pkt) < 0)
> + ++ctx->dropped;
> +
> + av_packet_unref(&pkt);
> + audio_offset += sample_size;
> + }
> }
> }
>
> @@ -999,18 +1079,7 @@ av_cold int ff_decklink_read_header(AVFormatContext *avctx)
> #endif
>
> /* Setup streams. */
> - st = avformat_new_stream(avctx, NULL);
> - if (!st) {
> - av_log(avctx, AV_LOG_ERROR, "Cannot add stream\n");
> - ret = AVERROR(ENOMEM);
> - goto error;
> - }
> - st->codecpar->codec_type = AVMEDIA_TYPE_AUDIO;
> - st->codecpar->codec_id = cctx->audio_depth == 32 ? AV_CODEC_ID_PCM_S32LE : AV_CODEC_ID_PCM_S16LE;
> - st->codecpar->sample_rate = bmdAudioSampleRate48kHz;
> - st->codecpar->channels = cctx->audio_channels;
> - avpriv_set_pts_info(st, 64, 1, 1000000); /* 64 bits pts in us */
> - ctx->audio_st=st;
> + setup_audio(avctx);
>
> st = avformat_new_stream(avctx, NULL);
> if (!st) {
> @@ -1096,8 +1165,17 @@ av_cold int ff_decklink_read_header(AVFormatContext *avctx)
> ctx->teletext_st = st;
> }
>
> - av_log(avctx, AV_LOG_VERBOSE, "Using %d input audio channels\n", ctx->audio_st->codecpar->channels);
> - result = ctx->dli->EnableAudioInput(bmdAudioSampleRate48kHz, cctx->audio_depth == 32 ? bmdAudioSampleType32bitInteger : bmdAudioSampleType16bitInteger, ctx->audio_st->codecpar->channels);
> + if (cctx->audio_mode == AUDIO_MODE_BUNDLED) {
> + av_log(avctx, AV_LOG_VERBOSE, "Using %d input audio channels\n", ctx->audio_st[0]->codecpar->channels);
> + result = ctx->dli->EnableAudioInput(bmdAudioSampleRate48kHz,
> + ctx->audio_depth == 32 ? bmdAudioSampleType32bitInteger : bmdAudioSampleType16bitInteger,
> + ctx->audio_st[0]->codecpar->channels);
> + } else {
> + av_log(avctx, AV_LOG_VERBOSE, "Using %d input audio channels\n", ctx->max_audio_channels);
> + result = ctx->dli->EnableAudioInput(bmdAudioSampleRate48kHz,
> + ctx->audio_depth == 32 ? bmdAudioSampleType32bitInteger : bmdAudioSampleType16bitInteger,
> + ctx->max_audio_channels);
> + }
>
> if (result != S_OK) {
> av_log(avctx, AV_LOG_ERROR, "Cannot enable audio input\n");
> diff --git a/libavdevice/decklink_dec_c.c b/libavdevice/decklink_dec_c.c
> index 1c6d826..fe8e9fd 100644
> --- a/libavdevice/decklink_dec_c.c
> +++ b/libavdevice/decklink_dec_c.c
> @@ -44,6 +44,9 @@ static const AVOption options[] = {
> { "standard", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = 0x7fff9fffeLL}, 0, 0, DEC, "teletext_lines"},
> { "all", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = 0x7ffffffffLL}, 0, 0, DEC, "teletext_lines"},
> { "channels", "number of audio channels", OFFSET(audio_channels), AV_OPT_TYPE_INT , { .i64 = 2 }, 2, 16, DEC },
> + { "audio_mode", "audio mode", OFFSET(audio_mode), AV_OPT_TYPE_INT, { .i64 = AUDIO_MODE_BUNDLED}, 0, 1, DEC, "audio_mode"},
> + { "bundled", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = AUDIO_MODE_BUNDLED}, 0, 0, DEC, "audio_mode"},
> + { "pairs", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = AUDIO_MODE_PAIRS}, 0, 0, DEC, "audio_mode"},
> { "duplex_mode", "duplex mode", OFFSET(duplex_mode), AV_OPT_TYPE_INT, { .i64 = 0}, 0, 2, DEC, "duplex_mode"},
> { "unset", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = 0}, 0, 0, DEC, "duplex_mode"},
> { "half", NULL, 0, AV_OPT_TYPE_CONST, { .i64 = 1}, 0, 0, DEC, "duplex_mode"},
Regards,
Marton
More information about the ffmpeg-devel
mailing list