[FFmpeg-trac] #6066(avformat:new): Handling HTTP 500 errors for input files

[FFmpeg]

#6066: Handling HTTP 500 errors for input files
-------------------------------------+-------------------------------------
             Reporter:  dprestegard  |                     Type:
               Status:  new          |  enhancement
            Component:  avformat     |                 Priority:  normal
             Keywords:               |                  Version:
             Blocking:               |  unspecified
Analyzed by developer:  0            |               Blocked By:
                                     |  Reproduced by developer:  0
-------------------------------------+-------------------------------------
 Summary of the bug:
 Sometimes when working with cloud object storage systems like AWS S3 it's
 normal to experience some 500 errors as S3 only guarantees 99.9%
 availability. As it currently operates, ffmpeg appears to interpret a 500
 error as the end of the input file. When working with large source files
 (100+ GB) this can lead to ffmpeg transcodes truncating.

 It would be ideal if ffmpeg could handle a 500 error gracefully by
 retrying the request up to a certain number of tries.

 I have tried setting the -reconnect 1 flag, but this had no effect.

 Simply re-trying the transcode always succeeds, but this is wasteful.

 When the error occurs, the following will show up in the transcode report:
 {{{
 [https @ 0x55e3cde7e140] request: GET
 /foo.mov?AWSAccessKeyId=bar&Expires=baz&Signature=qux HTTP/1.1
 User-Agent: Lavf/57.56.100
 Accept: */*
 Range: bytes=44435210752-
 Connection: close
 Host: foo.s3.amazonaws.com
 Icy-MetaData: 1

 [https @ 0x5636385ed140] HTTP error 500 Internal Server Error
 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x5636385ec7c0] stream 0, offset 0xe65761200:
 partial file
 https://foo.s3.amazonaws.com/foo.mov?AWSAccessKeyId=bar&Expires=baz&Signature=qux:
 Invalid data found when processing input
 }}}

 I experienced this with multiple versions, including the latest master
 from a few weeks ago.

 How to reproduce:
 This is difficult to reproduce, since S3 usually works fine but only
 occasionally returns 500 errors for a request. In my testing encoding
 feature length content this occurs approximately 10-15% of the time. In
 other words, if I encode 10 movies in series, 1 or 2 of them will be
 truncated

 {{{
 ffmpeg -y -report -reconnect 1 -i
 "https://foo.s3.amazonaws.com/foo.mov?AWSAccessKeyId=bar&Expires=baz&Signature=qux"
 -an -pix_fmt yuv420p -c:v libx264 -b:v 8M -preset superfast -tune film
 -fastfirstpass 0 -pass 1 -force_key_frames "expr:gte(t,n_forced*4)"
 -x264opts vbv-maxrate=8800:vbv-bufsize=16000 10test.mp4
 }}}

--
Ticket URL: <https://trac.ffmpeg.org/ticket/6066>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker