load wikidata on RDF4J

[cbuil]

Dear Community

I want to load Wikidata Truthy on RDF4J, so far I generated a new package following the instructions on the repo, deployed the server and workbench wars on a Tomcat 9 server (the machine has 120GB of ram), I created the native repository on the workbench, loaded that repo on the console and ran the load command. However the load stops, I do not know why. Are there any instructions or recommendations to load Wikidata Truthy on RDF4J?

Thanks!

Carlos

[barthanssens]

Hi Carlos, interesting test case !

Hmz, I assume that's a few billion of triples ? Are you using the JSON or NTriples version ?
Not sure if the native store is the best option here (havent' tested it, but LMDB could be faster)

[cbuil]

Thanks for the response! Actually right now I'm looking at the LMDB documentation (https://rdf4j.org/documentation/programming/lmdb-store/) so I can load Wikidata on it. However I did not see any reference to LMDB in the RDF4J documentation (https://rdf4j.org/documentation/tools/console/), maybe I missed it.

What I'm doing right now is developing a class to create the LMDB store and load into it Wikidata. I hope it will work. Any suggestion?

Thanks!

[cbuil]

btw, any recommendation for indexing the data? spoc,ospc,psoc is enough?

[hmottestad]

If your data doesn't have any names graphs then you are probably fine with those indexes. Other than that your queries will be the deciding factor for which indexes you should have.

[cbuil]

one last question, the following is my code, however when I try to add the indexes through the configuration class I get a compilation error when creating the new Sail repository, I follow the example in https://rdf4j.org/documentation/programming/lmdb-store/
import org.eclipse.rdf4j.repository.sail.SailRepository;
...

                File dataDir = new File(args[0]);
		LmdbStoreConfig config = new LmdbStoreConfig();
		// set triple indexes
		config.setTripleIndexes("spoc,ospc,psoc");

		Repository repo = new SailRepository(new LmdbStore(dataDir));  // here if I add the config class I get a compiling error, opposed to the example in https://rdf4j.org/documentation/programming/lmdb-store/

The dependencies I use in my pom.xml are the following

               <dependency>
			<groupId>org.eclipse.rdf4j</groupId>
			<artifactId>rdf4j-repository-sail</artifactId>
			<version>4.2.2</version>
		</dependency>
		<dependency>
			<groupId>org.eclipse.rdf4j</groupId>
			<artifactId>rdf4j-sail-lmdb</artifactId>
			<version>4.0.2</version>
		</dependency>

Thanks

[kenwenzel]

You are using the wrong version for rdf4j-sail-lmdb. It should be 4.2.2.

You can also initialize DB sizes using:

// set max db sizes to 1 TiB each
config.setValueDBSize(1_099_511_627_776L);
config.setTripleDBSize(1_099_511_627_776L);

This prevents autogrowing and may speed up the loading.

It may be also desirable to use

connection.begin(IsolationLevels.NONE);

and to add intermediate commits after a certain number of triples, e.g. 100000, via

connection.commit()

I've created a small benchmark CLI tool that adds random FOAF-like data to the LMDB store:
https://github.com/kenwenzel/rdf4j/blob/bb9963605c1dd6733188ebe4a5fe957f6b194c5d/core/sail/lmdb/src/test/java/org/eclipse/rdf4j/sail/lmdb/benchmark/BenchmarkToolFoaf.java

Maybe you can use the code as a blueprint.
I am very interested in your results on loading Wikidata Truthy!

[cbuil]

thanks a lot! definitely I will look at this

Carlos

[cbuil]

Dear all,

a quick update:

I'm loading Wikidata truthy by dividing one file with the triples into 1200+ files. Sometimes I find some bad IRI, the error I get is the following:
Partsing error: Illegal percent encoding U+25 at index 98: http://www.jbc.org/cgi/content/full/277/4/2477?maxtoshow=&HITS=10&hits=10&RESULTFORMAT=&author1=Ma% [line 229506] in file x0495.nt.good

When that happens the code throws an exception, which I caught, but the file stops loading. Is there any way to allow inserting line by line the file? I had a look at the code for generating one triple and inserting it, however I needed a bit more of time for identifying what data type the object of a triple is, and adding it correctly to the database.

This is my code https://gist.github.com/cbuil/cf2b9429eeefc5a61cd28fe24d1cfac1

At the beginning of the process loading a file with ~170.000 lines took ~10 seconds, after a day loading each file takes ~400 seconds.

Best

Carlos

[cbuil]

now half wikidata is loaded, so I guess that loading a triple and adding all three indexes is taking more time. I also loaded Wikidata on JenaTDB and happened something similar.

[barthanssens]

You might want to disable some settings that halt the processing when the file contains invalid statements...

See also https://rdf4j.org/javadoc/latest/org/eclipse/rdf4j/rio/helpers/BasicParserSettings.html, the example in 4.1.1 https://rdf4j.org/documentation/programming/rio/#example-iri-syntax-validation and some quick and dirty code on reporting (but not stop parsing on) errors https://github.com/barthanssens/rdf4j-bigfile-validator/blob/main/src/main/java/be/bosa/dt/rbv/Main.java

[kenwenzel]

Can you share your hardware configuration please?
Do you use SSDs?
I even achieved a loading rate of 80.000 to 150.000 TPS up to 1 billion triples (130 GB db) using an HDD RAID.
But I used the synthetic FOAF data...

You should also use separate drives for the source files and the database.

Have you increased the heap memory of the JVM somehow? This should not be required as LMDB uses off-heap memory.

There may be also a problem with the transactions. Connection.begin is only called before the first loop and not within the loop...

[cbuil]

I believe it is an HD, 4TB, 120GB RAM, however if I commit regularly I guess I can resume the loading after clearing swap memory, I guess that will speed up the process.

[kenwenzel]

It won't work well if it isn't an SSD as LMDB is optimized for SSDs.
It is also very important that you change your code and move connection.begin into the loop before loading each file.