What is the expected size of triples for the LMDB store in V4? Is it billions?

[AndreaWesterinen]

The current NativeStore is described as appropriate for 100000 to 100M triples.

Is there a similar estimate for the LMDBStore?

Thanks!

[hmottestad]

@kenwenzel have you done any tests?

[kenwenzel]

Yes, but only a very basic one. I've inserted 1 billion quads on a VM with 96 GB RAM and 500 GB HDD RAID. The insertion performance was very stable at 120,000 TPS until the database size reached the RAM size and the insertion rate dropped to 50,000 TPS. The average insertion speed was around 82,000 TPS.

Scans on the database were fast when most of it was cached in RAM but very slow after a system restart. I think this is due to the rather low IOPS of the storage system.

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=fiotest --filename=testfio --bs=4k --iodepth=64 --size=8G --readwrite=randrw --rwmixread=75
fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.16
Starting 1 process
fiotest: Laying out IO file (1 file / 8192MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=7971KiB/s,w=2554KiB/s][r=1992,w=638 IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=69235: Wed Jan 19 08:40:09 2022
  read: IOPS=1017, BW=4070KiB/s (4167kB/s)(6141MiB/1545295msec)
   bw (  KiB/s): min=  376, max= 9344, per=99.99%, avg=4068.60, stdev=616.79, samples=3090
   iops        : min=   94, max= 2336, avg=1017.14, stdev=154.20, samples=3090
  write: IOPS=339, BW=1359KiB/s (1392kB/s)(2051MiB/1545295msec); 0 zone resets
   bw (  KiB/s): min=   64, max= 3160, per=100.00%, avg=1358.65, stdev=220.36, samples=3090
   iops        : min=   16, max=  790, avg=339.65, stdev=55.09, samples=3090
  cpu          : usr=0.94%, sys=3.67%, ctx=1521730, majf=0, minf=9
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
   READ: bw=4070KiB/s (4167kB/s), 4070KiB/s-4070KiB/s (4167kB/s-4167kB/s), io=6141MiB (6440MB), run=1545295-1545295msec
  WRITE: bw=1359KiB/s (1392kB/s), 1359KiB/s-1359KiB/s (1392kB/s-1392kB/s), io=2051MiB (2150MB), run=1545295-1545295msec
Disk stats (read/write):
  sda: ios=1569280/526081, merge=2747/709, ticks=83305963/15024425, in_queue=94126796, util=100.00%

[abrokenjester]

@kenwenzel I realize that's relatively hefty hardware but those numbers look really good to me. It would be interesting to run a few large-dataset benchmarks and see some comparisons with other stores - and not just upload speed but also some query benchmarks.

I'm a little out of the loop on what the current "go-to" benchmarks for this are. Is BSBM (Berlin SPARQL Benchmark) still a thing?

[kenwenzel]

I've used a simple CLI tool that inserts synthetic FOAF data. I think this data represents a good mix between datatype properties and edges that connect nodes throughout the dataset.
I've also added some queries like this one

SELECT DISTINCT ?p ?name (COUNT(?other) AS ?count) WHERE {
        ?p a foaf:Person ; foaf:knows ?other ; rdfs:label ?name
        FILTER (LANG(?name) = "en")
}
GROUP BY ?p ?name
LIMIT 100

but haven't used that yet for any benchmarks. The query was inspired by this discussion on Wikidata.

I also would like to have some comparable benchmarks but I am not sure whether one should use BSBM, LUBM or just a custom benchmark as described above.

[kenwenzel]

This is a list of benchmark reports:
https://www.w3.org/wiki/RdfStoreBenchmarking

Maybe we can run similar benchmarks to the ones of Stardog.

[kenwenzel]

This question is not easy to answer. Storing billions of quads in a single LMDB store will be possible but depends on the available RAM size and the performance of your disks.

I've found an older document about OpenLDAP & LMDB sizing. This explains that more RAM is better. The best is to have enough RAM to accomodate the entire database or at least the database's working set. Another factor is the speed of your disks. You should use SSDs for larger databases.

More up-to-date information about LMDB can be found at: https://www.symas.com/symas-lmdb-tech-info
Especially the SSD-benchmarks may be of interest.

You can expect a footprint of around 120 - 130 bytes per quad when using RDF4J's LMDB store. Therefore you will need 120 - 130 GB storage space per 1 billion quads.