[FFmpeg-devel] Using FFMPEG to encode multiple outputs in 'parallel' (doesn't work?)

[Shalin Mehta]

I am posting this question in development community because the feature
that I am looking for is not supported as per my understanding. I would
like to know what ffmpeg developers think about implementing this. I hope
this is the right place for such question.

So this is how it goes:

Quoting from the ffmpeg's wiki for creating multiple outputs.
https://trac.ffmpeg.org/wiki/Creating%20multiple%20outputs
*Parallel encoding*:
Outputting and re encoding multiple times in the same FFmpeg process will
typically slow down to the "slowest encoder" in your list. Some encoders
(like libx264) perform their encoding "threaded and in the background" so
they will effectively allow for parallel encodings.

Unfortunately, my results are not correlating with what is mentioned above.
By that I mean, I am not seeing encodings being performed in parallel on
multiple outputs even when using libx264.

To simplify the problem and to understand it easily, here's what I am doing
for strictly benchmarking purposes.

Option1: Run 3 instances of ffmpeg in parallel as 3 separate processes.
ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null -
ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null -
ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null -

Option2: run all three encodes in single ffmpeg process
ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null - -c:v libx264 -b:v 12M
-f null - -c:v libx264 -b:v 12M -f null -

I am executing this workload on 32 core platform on google cloud.
Option1 gives me nearly 100% cpu utilization. In case of Option 2 cpu
utilization doesn't go beyond 50% and also fps is about 70% of the option1.

Then why don't I just use Option1?
Because Option2 decodes the input only once. I need to generate many
streams (up to 25 streams) from the same input file which could be 4K or
even bigger. So I want to avoid the overhead of decoding the same input so
many times in Option1.
So basically, I want to make Option2 work but I am not understanding why
it's not running all the encodes in parallel to give me better fps and
maximum cpu utilization.

After taking a quick look at the code, it doesn't seem like ffmpeg is
creating multiple threads to run multiple x264 encode sessions in parallel.
I know x264 library itself will create multiple threads but that's for a
single output. What I am looking for is output level parallelism.
Can some one explain if this is possible?
Or may be, I can use some different kind of command line to achieve such
output level parallelism?
If this is not currently supported in ffmpeg what would be your estimate to
implement some thing like this given that it is even possible to begin with.

Shalin