[FFmpeg-trac] #10371(avformat:new): hls_enc_key does not work as advertised

[FFmpeg]

#10371: hls_enc_key does not work as advertised
-------------------------------------+-------------------------------------
             Reporter:  Mark         |                     Type:  defect
  Karpeles                           |
               Status:  new          |                 Priority:  normal
            Component:  avformat     |                  Version:  git-
                                     |  master
             Keywords:  hls encrypt  |               Blocked By:
             Blocking:               |  Reproduced by developer:  0
Analyzed by developer:  0            |
-------------------------------------+-------------------------------------
 Summary of the bug: hls_enc_key is not properly handled as hex data but is
 instead truncated and not used as binary data as it should

 How to reproduce:
 {{{
 % ffmpeg -i input -f hls -hls_enc_key 42424242424242424242424242424242 ...
 output
 ffmpeg version 5.1.3
 }}}

 master.key will contain the actual string "4242424242424242" instead of
 containing "BBBBBBBBBBBBBBBB" (B has ascii value 0x42).

 ----

 The hls_enc_key option has the following description: hex-coded 16 byte
 key to encrypt the segments

 Setting it will change the value of hls->key, however libavformat/hlsenc.c
 on line 773 a simple memcpy() is performed instead of reading the string,
 decoding its hex value and setting key appropriately.

 - this reads data that might not be allocated, since it will always read
 16 bytes even if "key" length is lower than that (unlikely)
 - encryption is weaker because the key can only contain alphanumeric
 characters and will typically only contain 0-9 a-f

 This does not break anything as is because ffmpeg will output the key it
 used in the right file (unless ones ignores the outputted file, generates
 the key file separately and spends hours wondering why nothing works).

 libavformat/hlsenc.c:773 should be modified to decode the passed value as
 hex-encoded data. I do not know if keeping the existing behavior is
 needed.
-- 
Ticket URL: <https://trac.ffmpeg.org/ticket/10371>
FFmpeg <https://ffmpeg.org>
FFmpeg issue tracker