ulimit and W^X

[redwyre]

Hi, we have a user that is trying to run our game server with ulimit like so:
ulimit -f 8500000; ./EcoServer
However they are finding that the program is creating a file called "doublemapper" which tries to resize to a huge size and SIGXFSZ.

I have suggested they try setting DOTNET_EnableWriteXorExecute=0, but is there a way for them to use W^X with ulimit?

I should note they are on Debian 11 (unsure what point release)

[EgorBo]

cc @janvorli

[mirasrael]

#80580 seems related

[janvorli]

@redwyre .NET does not create the file per se, it uses memfd_create("doublemapper", MFD_CLOEXEC). This creates an anonymous file and unlike regular files, it lives in RAM. I was not aware of the fact that it is subject to the ulimit -f limit. So currently, the only way to make it work with that limit is what you have suggested - to disable W^X.
I wonder if we should add honoring the limit though. The limit effectively limits amount of memory that can be used for allocating executable code. In the case someone sets the limit too low, it could result in OOM errors due to that. But maybe that's acceptable.

@mirasrael the #80580 is unrelated, that one is due to another limit - the number of allowed memory mappings.

[redwyre]

I assume the file in memory is sparse, so I wonder if ulimit is the bad guy here since what it should really care about is what's written to the file - though in this case it should be what's actually committed to memory?

[janvorli]

After creating the memfd, we call ftruncate with 2TB size, but don't write anything to the file. When allocating executable memory, the allocator starts at the beginning of this virtual file and as more and more memory is allocated, the index in the file is moved further and further. The mmap is then called to map pages from that virtual file at the specific index. When we free memory, we zero the range being freed and call munmap on the mapping created by the mmap. The block at the related index can later be reallocated.
Unless there is unloadability in play, the freeing should be quite rare.
It would be interesting to see if the crash happens at the time of ftruncate => initialization time or after the usage of the file goes over the limit.