[Libav-user] How to stream incoming chunked data to libavformat reliably?

[Jack Bruienne]

I'm developing a library in which you can push in a stream of bytes to a 
decoder for a specific codec (right now I'm testing with OGG), and it 
will send back partial audio packets as it becomes available, plus a 
final stream of decoded audio once the user indicates the stream is 
finished. This data could come in at any time with any size, and I keep 
a buffer of the unprocessed data until it's consumed. I want to be able 
to support partial frames/streaming as well, so I need to have the data 
processing in the background before the finish method is called. The 
library is written in Java and is interfaced with in Lua, but it uses a 
native wrapper to FFmpeg in C (https://github.com/Manevolent/ffmpeg4j). 
For example, here's an example of how a user could pipe data from a file 
and then wait for the result:

local file = io.open("audio.ogg", "rb") local bus = 
peripheral.wrap("back") -- gets the decoder object repeat local data = 
file:read(16384) if data == nil then break end bus.input(1, data) -- 
send 16384 bytes to the decoder in slot 1 and process it in the 
background until #data < 16384 file:close() bus.finish(1) -- tell the 
decoder that the data is finished local event, side, slot, ok, data = 
coroutine.yield("asicraft.result") -- wait for processing to finish & 
get the result -- do things with data

I'm currently using the Java library's wrapper around an `InputStream` 
to send the data. I have my own `InputStream` which holds the buffered 
input, and the library will call its `read` method through an 
implementation of libavformat `AVIOContext`'s `read_packet` function. If 
the buffer runs out, the read method returns -1, and I used some 
mixins/reflection on the Java library to make it return 
`AVERROR(EAGAIN)` to libavformat. This worked for the first packet sent 
to the stream, but despite always sending `EAGAIN` I was still getting 
EOFs simply from returning less data than the demuxer expected. To solve 
this, I forced the `eof_reached` flag to be reset to 0 each time data is 
added to the stream, but because a partial amount of data was read 
during the last decoding period, it ends up skipping that frame.

I've considered putting the audio processing on a separate thread, and 
having the `read_packet` function wait for more input before returning 
(which luckily Java makes pretty easy). But this would mean that I would 
need a new thread for every single input process, which theoretically 
could be opened and then left running forever, and I want to avoid 
ending up with thread exhaustion (this needs to be able to run on public 
servers where users can trigger the decoder at any time). I also tried 
throwing an exception to attempt to unwind the reader back across 
`av_frame_read`, but this failed spectacularly as I expected.

What's the most optimal way to decode/demux a data stream in chunks that 
may arrive at any time, without losing any frames in the process? And, 
more specifically for my current method, is there any way to figure out 
where in the data stream the last frame was read by the demuxer? If I 
can know that, I can just rewind the data stream to that point and 
continue the decoder.

If anyone wants to look at my current code, it's available at 
https://github.com/MCJack123/ASICraft/tree/ffmpeg-decode/common/src/main/java/cc/craftospc/ASICraft/algorithms/OGGDecodeAlgorithm.java. 
I don't expect anyone to want to poke through it, but it's there if 
someone does. There's also some related code in 
`util/FFmpegInputFixed.java` and `mixin/FFmpegSourceStreamInject.java` 
for patching a few things - mainly, switching it to use `EAGAIN` instead 
of EOF.

I would appreciate any help in getting this working, as it drives a 
major part of my project, and I don't want to have to drop it or limit 
the features because I can't get libav to cooperate with my input scheme.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://ffmpeg.org/pipermail/libav-user/attachments/20230616/18f80625/attachment.htm>