[FFmpeg-devel] [PATCH] ffbuild: read library linker objects from a file

[Martin Storsjö]

On Wed, 12 Mar 2025, Gyan Doshi wrote:

> On 2025-03-12 03:12 pm, Martin Storsjö wrote:
>> On Wed, 12 Mar 2025, Gyan Doshi wrote:
>>
>>> On 2025-03-12 01:29 pm, Martin Storsjö wrote:
>>>> On Wed, 12 Mar 2025, Gyan Doshi wrote:
>>>>
>>>>> The linker command can exceed the maximum argument limit on MinGW,
>>>>> especially for libavcodec.
>>>>>
>>>>> The objects list is now stored in a file and passed to the linker.
>>>>> ---
>>>>> ffbuild/library.mak | 4 +++-
>>>>> 1 file changed, 3 insertions(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/ffbuild/library.mak b/ffbuild/library.mak
>>>>> index 793e9d41fa..781b018e00 100644
>>>>> --- a/ffbuild/library.mak
>>>>> +++ b/ffbuild/library.mak
>>>>> @@ -66,8 +66,10 @@ $(SUBDIR)$(SLIBNAME): 
>>>>> $(SUBDIR)$(SLIBNAME_WITH_MAJOR)
>>>>>
>>>>> $(SUBDIR)$(SLIBNAME_WITH_MAJOR): $(OBJS) $(SHLIBOBJS) $(SLIBOBJS) 
>>>>> $(SUBDIR)lib$(NAME).ver
>>>>>     $(SLIB_CREATE_DEF_CMD)
>>>>> -    $$(LD) $(SHFLAGS) $(LDFLAGS) $(LDSOFLAGS) $$(LD_O) $$(filter 
>>>>> %.o,$$^) $(FFEXTRALIBS)
>>>>> +    $(Q)echo $$(filter %.o,$$^) > $$@.objs
>>>>> +    $$(LD) $(SHFLAGS) $(LDFLAGS) $(LDSOFLAGS) $$(LD_O) @$$@.objs 
>>>>> $(FFEXTRALIBS)
>>>>>     $(SLIB_EXTRA_CMD)
>>>>> +    -$(RM) $$@.objs
>>>>
>>>> Don't we need do the same for static libraries too?
>>>>
>>>> On first glance, this looks quite reasonable... However, the limit 
>>>> that is an issue is the length of a command line. The .objs file is 
>>>> written with an "echo" command - doesn't that fundamentally also hit 
>>>> the same limit (just a little bit later, as there are fewer command 
>>>> line flags in this command)?
>>>>
>>>> Or do we assume that make executes it with a shell, where the shell 
>>>> handles "echo" as a shell builtin so that it doesn't actually spawn 
>>>> a subprocess for this? Can you test it, if you could extend the list 
>>>> of object files to the point where the .objs file is clearly over 32 
>>>> KB, and verify that it did include all the files you intended?
>>>
>>> The specific error I got when building a shared build of 7.1.1 (with 
>>> ~90 external libs) was
>>>
>>> /bin/sh: line 1: /mingw64/bin/ccache: Argument list too long
>>>
>>> Got same error without ccache.
>>>
>>> The static build of the same config succeeded without any patching.
>>> The .objs file generated for libavcodec shared build is 29KB.
>>>
>>> How do I inflate the size to above 32K? I've enabled pretty much 
>>> everything I can.
>>
>> The simplest would probably be to add a bunch of empty .c files (with 
>> long file names) in libavcodec and hook them up in the makefile. We 
>> won't notice if they are missed when linking of course, but if we pass 
>> the command line length limit, we should still notice it in one way or 
>> another.
>
> I did something simpler. I just duplicated the arg on the echo line. The 
> build was successful. Each objs file was twice as big, same for the 
> generated DLL.
> Does that answer your queries?

Yes, thanks.

Btw, just to be clear - are you running this in msys2? If you're running 
mingw build tools via WSL, there wouldn't be any issue when running the 
shell commands on the linux side, but it's specifically with the msys2 
make where it's interesting to know which limitations make and the shell 
runs into or avoids.

// Martin