NativeStore and ID overflow

[kenwenzel]

Hello,

NativeStore internally uses int-based IDs for references to values (literals, blank nodes, IRIs).
Is it possible that the IDs may be subject to overflows?

Thank you and best regards,
Ken

[abrokenjester]

Theoretically, yes, but given that integer max is over 2 billion, and the scalability of the native store in practice tops out about around 500-700 million triples, there is little danger of hitting that overflow. Keep in mind that even very large datasets will (typically) contain a much smaller number of distinct values, as quite a lot of triples will reuse existing values (especially subjects and properties).

[kenwenzel]

Would it be an option to use LevelDB for storing the values or as possible alternative for the NativeStore?

[hmottestad]

If you have experience working with LevelDB and would like to try to improve the NativeStore then it would be an awesome contribution. Another alternative is to use the ExtensibleStore[1] as a base to implement your own store that uses LevelDB as the storage engine. You can take a look at these testes here if you want to see something dead simple to get you started: ExtensibleStoreImplForTests.java and ExtensibleStoreConnectionImplForTests.java and NaiveHashSetDataStructure.java

1: https://github.com/eclipse/rdf4j/tree/main/core/sail/extensible-store/src/main/java/org/eclipse/rdf4j/sail/extensiblestore

[kenwenzel]

Thank you for the pointers. I will take a look. If I find some time then I try to create a first PoC.
How does ExtensibleStore handle transactions? Is there already some logic in place to handle isolation and rollbacks? This is actually what worries me the most when thinking about the LevelDB-based implementation.

[hmottestad]

Handles up to READ COMMITTED for you. Including rollback.

[kenwenzel]

I've added a PR with an LMDB-based SAIL for further discussion: #3413