JSON-LD statement iterator while parsing

[jjkoehorst]

I have a rather big JSON-LD file with basic context on which I want to apply some modifications before streaming it to a triple store.
However every approach I take it first loads or at least tries to load the entire file.
Is there a way either by fast conversion to ntriples or loading it through a statement iterator to process such a file on the fly?

I tried something like

        try (GraphQueryResult res = QueryResults.parseGraphBackground(inputStream, baseURI, format)) {
            logger.info("Parsing graph statements");
            while (res.hasNext()) {
                Statement statement = res.next();
                statementCounter++;
                System.err.println("Statements: " + statementCounter);
...

but it remains busy before the res.hasNext step...

Any ideas?

[hmottestad]

I think that our JSON-LD implementation needs to parse the entire file before it can return anything. I haven't check though, so I might be wrong.

How big is your file? Maybe you can find a passable solution by increasing the amount of memory you give your program?

[jjkoehorst]

At the moment a few million statements. Maybe there is a way for bulk / speed loading it into an empty instance not sure what I have now is the fastest approach.

[hmottestad]

I would wrap the input stream in a BufferedInputStream.

I'm not sure what you want to load the data into, but I would recommend loading it into the MemoryStore if you intend to make modifications to your data.

Repository repository = new SailRepository(new MemoryStore());

try (RepositoryConnection connection = repository.getConnection()) {
	connection.begin(IsolationLevels.NONE); // Transaction isolation doesn't matter since you're not using the repository from more than one thread. Using NONE makes it faster at loading data.
	connection.add(inputStream, baseURI, format);
	connection.commit();
}

...

repository.shutdown();

Another alternative is to load it into a model:

Model model = Rio.parse(inputStream, baseURI, format);