Question about open files in Server

[rwalkerands]

Please pardon my asking about OpenRDF Sesame Server ... because I also need to know whether the situation is any different in RDF4J.

It seems that Sesame Server keeps repository files open. E.g., if you do a SPARQL query against a Native Store repo, there are eleven files opened for that repo, and they stay open. (To see this: on Linux, run lsof against the web server process.) If you have a "lot" of repositories (say, over 300 ...), and you do queries against them, you eventually run out of open files (hard per-process limit of 4096 open files on Linux). Ouch.

Did I miss a configuration option to close repo files? Does the same behaviour apply in RDF4J Server?

Richard.

[barthanssens]

Hi,

don't know the answer by heart and haven't checked it yet, but it could be still the case (data file, index file etc and IIRC network connections can also consume file connections)

It should be possible for the system administrator to raise the per process limit by changing a kernel setting (and/or raise it in security-related components like apparmor, pam.d)

Or for a running process using e.g. prlimit -n16384 <PID>

An alternative would be to use a different store instead of the native store repo: there is an experimental store based on ElasticSearch, and a very interesting LMDB store will be part of the upcoming v4.0 release.

Best regards

Bart

[rwalkerands]

Thank you for the suggestion to use prlimit, which would have helped, except ... it turns out I can't use it, because the server's kernel version is too old: it doesn't have the necessary underlying prlimit system call.

It looks like I can edit the Tomcat (shell) initialization script and add

ulimit -Hn 16384
ulimit -Sn 16384

instead.

(Various solutions for different Linux distributions documented here: https://blog.jayway.com/2012/02/11/how-to-really-fix-the-too-many-open-files-problem-for-tomcat-in-ubuntu/ and here: https://www.malasuk.com/linux/increase-open-file-limit-tomcat-centos-7/ )

[kenwenzel]

Hi,

as with other databases it has a large performance impact if the index files of the Native Store are closed and reopened. The LMDB store has the advantage that it just uses 2 files - one for values and one for statement indexes. But the main problem of keeping them open will stay the same.
Having said that I would suggest you to share a single repository for your data and separate it by using named graphs.

Best regards
Ken