[FFmpeg-trac] #7894(avformat:new): SChannel TLS implementation bug

[FFmpeg]

#7894: SChannel TLS implementation bug
-------------------------------------+-------------------------------------
             Reporter:  Adion        |                     Type:  defect
               Status:  new          |                 Priority:  normal
            Component:  avformat     |                  Version:  git-
             Keywords:  schannel     |  master
  tls                                |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 There's a bug in tls_schannel.c that prevents rtmps to work correctly when
 using this tls implementation. (It hangs in an infinite loop in tcp_read)
 It is present in ffmpeg v4.1.3 and at this time is also still present in
 master.

 The problem is in tls_read, when only a subset of the data returned by the
 server is requested.
 First data is read from the server into c->enc_buf.
 Next the data is decoded, but data is only encoded until enough data is
 decoded to fulfill the read request.
 In the case of SECBUFFER_EXTRA, there may still be undecoded data in
 enc_buf when the loop stops and the data is returned.

 On the next call to tls_read, there is not enough data in the decoded
 buffer, so it requests more data from the tcp layer, but this never comes
 as the data is already there, just not decoded yet.

 I fixed it by decoding all the data before returning. Another way would be
 to check if there is still undecoded data in the encode data on the next
 request and decode it at that time.

 I changed
 {{{
 while (c->enc_buf_offset > 0 && sspi_ret == SEC_E_OK && c->dec_buf_offset
 < len) {
 }}}
 to
 {{{
 while (c->enc_buf_offset > 0 && sspi_ret == SEC_E_OK) {
 }}}

 Related to this, I found it odd that by default there is no timeout
 specified, is this normal?

--
Ticket URL: <https://trac.ffmpeg.org/ticket/7894>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker