[FFmpeg-devel] [PATCH] ffbuild: read library linker objects from a file

[Gyan Doshi]

On 2025-03-12 05:52 pm, Martin Storsjö wrote:
> On Wed, 12 Mar 2025, Gyan Doshi wrote:
>
>> On 2025-03-12 03:12 pm, Martin Storsjö wrote:
>>> On Wed, 12 Mar 2025, Gyan Doshi wrote:
>>>
>>>> On 2025-03-12 01:29 pm, Martin Storsjö wrote:
>>>>> On Wed, 12 Mar 2025, Gyan Doshi wrote:
>>>>>
>>>>>> The linker command can exceed the maximum argument limit on MinGW,
>>>>>> especially for libavcodec.
>>>>>>
>>>>>> The objects list is now stored in a file and passed to the linker.
>>>>>> ---
>>>>>> ffbuild/library.mak | 4 +++-
>>>>>> 1 file changed, 3 insertions(+), 1 deletion(-)
>>>>>>
>>>>>> diff --git a/ffbuild/library.mak b/ffbuild/library.mak
>>>>>> index 793e9d41fa..781b018e00 100644
>>>>>> --- a/ffbuild/library.mak
>>>>>> +++ b/ffbuild/library.mak
>>>>>> @@ -66,8 +66,10 @@ $(SUBDIR)$(SLIBNAME): 
>>>>>> $(SUBDIR)$(SLIBNAME_WITH_MAJOR)
>>>>>>
>>>>>> $(SUBDIR)$(SLIBNAME_WITH_MAJOR): $(OBJS) $(SHLIBOBJS) $(SLIBOBJS) 
>>>>>> $(SUBDIR)lib$(NAME).ver
>>>>>>     $(SLIB_CREATE_DEF_CMD)
>>>>>> -    $$(LD) $(SHFLAGS) $(LDFLAGS) $(LDSOFLAGS) $$(LD_O) $$(filter 
>>>>>> %.o,$$^) $(FFEXTRALIBS)
>>>>>> +    $(Q)echo $$(filter %.o,$$^) > $$@.objs
>>>>>> +    $$(LD) $(SHFLAGS) $(LDFLAGS) $(LDSOFLAGS) $$(LD_O) @$$@.objs 
>>>>>> $(FFEXTRALIBS)
>>>>>>     $(SLIB_EXTRA_CMD)
>>>>>> +    -$(RM) $$@.objs
>>>>>
>>>>> Don't we need do the same for static libraries too?
>>>>>
>>>>> On first glance, this looks quite reasonable... However, the limit 
>>>>> that is an issue is the length of a command line. The .objs file 
>>>>> is written with an "echo" command - doesn't that fundamentally 
>>>>> also hit the same limit (just a little bit later, as there are 
>>>>> fewer command line flags in this command)?
>>>>>
>>>>> Or do we assume that make executes it with a shell, where the 
>>>>> shell handles "echo" as a shell builtin so that it doesn't 
>>>>> actually spawn a subprocess for this? Can you test it, if you 
>>>>> could extend the list of object files to the point where the .objs 
>>>>> file is clearly over 32 KB, and verify that it did include all the 
>>>>> files you intended?
>>>>
>>>> The specific error I got when building a shared build of 7.1.1 
>>>> (with ~90 external libs) was
>>>>
>>>> /bin/sh: line 1: /mingw64/bin/ccache: Argument list too long
>>>>
>>>> Got same error without ccache.
>>>>
>>>> The static build of the same config succeeded without any patching.
>>>> The .objs file generated for libavcodec shared build is 29KB.
>>>>
>>>> How do I inflate the size to above 32K? I've enabled pretty much 
>>>> everything I can.
>>>
>>> The simplest would probably be to add a bunch of empty .c files 
>>> (with long file names) in libavcodec and hook them up in the 
>>> makefile. We won't notice if they are missed when linking of course, 
>>> but if we pass the command line length limit, we should still notice 
>>> it in one way or another.
>>
>> I did something simpler. I just duplicated the arg on the echo line. 
>> The build was successful. Each objs file was twice as big, same for 
>> the generated DLL.
>> Does that answer your queries?
>
> Yes, thanks.
>
> Btw, just to be clear - are you running this in msys2? If you're 
> running mingw build tools via WSL, there wouldn't be any issue when 
> running the shell commands on the linux side, but it's specifically 
> with the msys2 make where it's interesting to know which limitations 
> make and the shell runs into or avoids.

Yes, via msys2.

This is what `xargs --show-limits` prints in the same shell,

POSIX upper limit on argument length (this system): 25382
POSIX smallest allowable upper limit on argument length (all systems): 4096
Maximum length of command we could actually use: 20812
Size of command buffer we are actually using: 25382

Regards,
Gyan