[FFmpeg-devel] [PATCH] libavfilter: temporarily remove DNN framework and vf_sr filter

[Pedro Arthur]

Hi,

I'm surprised with this patch, there wasn't any concern raised in the
patch review process.

2018-07-26 16:26 GMT-03:00 Rostislav Pehlivanov <atomnuker at gmail.com>:
> As discussed recently, the vf_sr filter and the DNN framework have an
> issue: unreproducable weights and questionable license, as well as
> overall unfitting coding style to the rest of the project.
I think I'm not aware of these discussions could you provide a
reference? I also don't understand why the coding style is unfitting
(again no concern was raised).

>
> The vf_sr filter in particular has weights embedded which weight the
> libavfilter binary by a bit and cannot currently be reproduced.
> There's an overall consensus that NN filters should accept external
> weights only, as the nnedi filter currently does.
>
> So, temporarily remove both until the coding style issues have been
> fixed with the framework and the filter has been modified to accept
> external weights.
What are these issues so we can fix them?

>
> Also, there's a discussion by the Debian folks as to whether to treat
> pretrained NNs as non-free[0], hence its not just our project that's
> affected by the questionable license of distributing pretrained NN
> weights.
>
> Due to the weight of the patch (more than 1mb!) I've uploaded it to
> https://0x0.st/sVEH.patch if anyone wants to test it. The change stat
> is printed below.
>
> [0]: https://lwn.net/Articles/760142/
I took the time to read the whole discussion and in my opinion it is
flawed, except for the storage requirement, which Sergey already
worked on a patch to reduce the stored data.


I think before any discussion, it should be clear what is the ffmpeg
policy on adding data, and what is considered data, and it should be
consistent.

I'll try to address the topics in the above discussion.

First what is data? is it expected that all data stored should be
easily reproducible?
I guess not, what is the point in storing data that is easily reproducible?
The entire humanity is built on previous stored knowledge, namely
data, do we require each time one is going to use some form of
knowledge to reproduce it? that is, proof everything one is using?
The answer is no, the whole point in storing data is that you had once
worked hard to proof it works and onwards just use it and believe it
works. That does not mean it is imposible to proof everything.

I think the above fits perfectly with NN weights as data.

The next point is the reproducibility, one should be reasonable, it is
hard to reproduce bit by bit of NN stored data but is totally doable
to achieve similar results following the same training metodology
used.


Then there is the question what is open-source, once again one should
be reasonable.
The NN model is public available, everything is documented, the math
machinery is also widely available and documented.
There is also a repository containing everything one needs to train
the NN and achieve similar results, the trainig data is public also.
The training software is open-source, the user could also use any
machine learning framework of their choice to perform the training
since the model is documented, there is nothing locking one to a
specific software or hardware.
I can't see what is not open.


Does we impose all requiriments imposed for NN weights on all other
data stored in ffmpeg?
I guess not, once more one should be consistent.


If you compare NN weights with quantization tables they are pretty
similar, both can be obtained from a training process over a dataset
so it achieves better results (quality/compression). Are quantization
tables evil? I don't think so.
It seems people is afraid of  NN just because we give it a fancy name,
while it is just tables of data as we always used in our code.

>
> Signed-off-by: Rostislav Pehlivanov <atomnuker at gmail.com>
>
> Rostislav Pehlivanov (1):
>   libavfilter: temporarily remove DNN framework and vf_sr filter
>
>  Changelog                        |     1 -
>  configure                        |     8 -
>  libavfilter/Makefile             |     3 -
>  libavfilter/allfilters.c         |     1 -
>  libavfilter/dnn_backend_native.c |   495 --
>  libavfilter/dnn_backend_native.h |    40 -
>  libavfilter/dnn_backend_tf.c     |   325 -
>  libavfilter/dnn_backend_tf.h     |    40 -
>  libavfilter/dnn_espcn.h          | 12637 -----------------------------
>  libavfilter/dnn_interface.c      |    60 -
>  libavfilter/dnn_interface.h      |    63 -
>  libavfilter/dnn_srcnn.h          |  4957 -----------
>  libavfilter/vf_sr.c              |   354 -
>  13 files changed, 18984 deletions(-)
>  delete mode 100644 libavfilter/dnn_backend_native.c
>  delete mode 100644 libavfilter/dnn_backend_native.h
>  delete mode 100644 libavfilter/dnn_backend_tf.c
>  delete mode 100644 libavfilter/dnn_backend_tf.h
>  delete mode 100644 libavfilter/dnn_espcn.h
>  delete mode 100644 libavfilter/dnn_interface.c
>  delete mode 100644 libavfilter/dnn_interface.h
>  delete mode 100644 libavfilter/dnn_srcnn.h
>  delete mode 100644 libavfilter/vf_sr.c
>
> --
> 2.18.0
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel