commons-rdf-rdf4j -> RepositoryDatasetImpl -> streams

[naturzukunft]

hi there, i'm using commons-rdf as abstraction layer to be independent from an DB implementation in the business logik.

In my unittests i've a bit trouble with closing connections. the problem can be found here

commons-rdf Dataset is using streams and it's difficult to close them ?!

Do you have an idea how to solve this?

i currently using a custom commons-rdf-rdf4j which depends on rdf4j 5.0.0-M2 !

I'm trying to release all stuff, but after everything is done it takes 20 seconds before sail is closing the connections.
And i've no idea, how i can force this.

18:50:18.810 [main] INFO  o.l.r.s.r.TestActivityPubStoreSaveAndRead - shutDown
18:50:38.811 [main] WARN  o.e.rdf4j.sail.helpers.AbstractSail - Closing active connection due to shut down; consider setting the org.eclipse.rdf4j.repository.debug system property

[naturzukunft]

I'm using :

repositoryManager = RepositoryProvider.getRepositoryManager(baseDir);
		repositoryManager.init();

and also call repositoryManager.shutDown(); !

[hmottestad]

See line 5.

        final RepositoryConnection conn = getRepositoryConnection();
        Stream<RDF4JQuad> stream = null;
        try {

        >>  final RepositoryResult<Statement> statements = conn.getStatements(subj, pred, obj, includeInferred, contexts); << This is never closed since you override the onClose operation of the stream you create
            
            // NOTE: Iterations.stream should close RepositoryResult as long as
            // our caller closes the stream
            stream = Iterations.stream(statements).map(rdf4jTermFactory::asQuad);
        } finally {
            if (stream == null) {
                // Some exception before we made the stream, close connection
                // here
                conn.close();
            }
        }
        // Make sure the RepositoryConnection is closed
        return stream == null ? null : stream.onClose(conn::close);

[naturzukunft]

You mean that: return stream == null ? null : stream.onClose(conn::close);
i already tried it with return stream;
same behaviour

[hmottestad]

If you just return the stream then you never end up closing the connection, and if you overwrite the onClose method then you never end up closing the repository results.

[hmottestad]

You need to close the result and then the connection from within the onClose method.

[naturzukunft]

i getting closer.
Looks like the problem is located here

Hopefully solved if this PR is accepted.

Now the stream connections looks closed, but there are still three other connections open ;-(

Thanks!

[hmottestad]

You should consider setting the debug system property to true. That should give you some more info about where the connections were created.

[naturzukunft]

current solution:

@Override
   public Stream<RDF4JQuad> stream(final Optional<BlankNodeOrIRI> graphName, final BlankNodeOrIRI subject, final IRI predicate,
           final RDFTerm object) {
       final Resource subj = (Resource) rdf4jTermFactory.asValue(subject);
       final org.eclipse.rdf4j.model.IRI pred = (org.eclipse.rdf4j.model.IRI) rdf4jTermFactory.asValue(predicate);
       final Value obj = rdf4jTermFactory.asValue(object);
       final Resource[] contexts = asContexts(graphName);

       // NOTE: We can't do the usual try..with closing of the
       // RepositoryConnection here as it will have to be closed outside
       // by the user of the returned stream
       final RepositoryConnection conn = getRepositoryConnection();
       Stream<RDF4JQuad> stream = null;
       try {
       	RepositoryResult<Statement> statements = conn.getStatements(subj, pred, obj, includeInferred, contexts);
           // NOTE: Iterations.stream should close RepositoryResult as long as
           // our caller closes the stream
           stream = statements.stream().map(rdf4jTermFactory::asQuad).onClose(() -> {
           	statements.close();
           	conn.close();        	
           	System.out.println("RepositoryDatasetImpl: connection closed! open: " + conn.isOpen() + ", active: " + conn.isActive());
       	});
       } finally {
           if (stream == null) {
               // Some exception before we made the stream, close connection
               // here
               conn.close();
           }
       }
       // Make sure the RepositoryConnection is closed
       return stream;
   }

btw. the same problem is in org.apache.commons.rdfrdf4j.impl.RepositoryGraphImpl.

[naturzukunft]

And it would be great, if rdf4j is providing an implementation for the commons-rdf types. Like Apache jena is doing here

i currently playing with my own copy

[naturzukunft]

Puh, rollback !!
It was in deed not a commons-rdf problem.

the problem has once again been sitting in front of my PC. You have to be careful, a commons-rdf dataset / graph can be bound to a repository. if this is the case, you have to be very careful with streams, because they always have to be used with try...with.