RDF4j in AWS with a EBS volume gets corrupted?

[cgokey]

My scenario looks liked this.

1. I imported 15K RDF documents, which resulted in about 200K triples.
2. I then ran
   DELETE ?s ?p ?o WHERE { ?s ?p ?o }
3. Then attempted to reinsert those same RDF documents, think it 100 at a time, about 150 calls to a lambda, that then inserts the RDF documents.

When I try to delete immediately after, I get this:

{"message":"Error during deletion process","error":"Failed to delete triples: 500: Cannot invoke \"String.hashCode()\" because the return value of \"org.eclipse.rdf4j.model.base.AbstractIRI.stringValue()\" is null","stack":"Error: Failed to delete triples: 500: Cannot invoke \"String.hashCode()\" because the return value of \"org.eclipse.rdf4j.model.base.AbstractIRI.stringValue()\" is null\n    at Runtime.m [as handler] (/var/task/serverless/src/deleteAll/handler.js:25:963)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)"}%

And I get other 500 errors when I try to fetch. I can't seem to repeat this locally, only in AWS where data dir is a EBS volume. This is a single instance accessing the EBS volume, no scaling. I'm wondering any thoughts on what might be happening. I end up having to recreate the db due to whatever being corrupted. If I increase the page size to insert about 1000 at a time, it works and I don't see any errors.

I'm using RDF4j native store. Configuration looks like this:
cat config.ttl

@prefix config: <tag:rdf4j.org,2023:config/> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<#kms> a config:Repository;
  rdfs:label "kms";
  config:rep.id "kms";
  config:rep.impl [
      config:rep.type "openrdf:SailRepository";
      config:sail.impl [
          config:sail.type "openrdf:NativeStore"
        ]
    ] .

I'm using the latest version of RDF4J 5 spun up in a ECS docker container.

Thanks,
Chris

[kenwenzel]

If it is working with larger batches then this could be an indication of some issues with syncing data to disk.
Have you already tried with "forceSync" set to true?

I've found this config https://github.com/nasa/kms/blob/30504da88b43fc08f456e39a79815158267265c0/infrastructure/rdfdb/cdk/docker/scripts/create_repository.sh#L28
where an additional colon is added after forceSync.

[cgokey]

Thanks, I've give forceSync a shot and fix that typo, thank you!
Chris

[cgokey]

That did the trick! Thank you!
Chris

[kenwenzel]

Usually, this should also work without forceSync. You should check your deployment and make sure that the container running RDF4J is not just killed at some point in time. An orderly shutdown of the repository should also close the files and trigger syncing data to disk.

[cgokey]

From what I understand, with a ECS container, it issues a sigterm which should tell tomcat to shutdown gracefully. I'm running RDF4J using the docker container with tomcat, does that have a context listener that will shutdown the repository manager gracefully? I could add more time for in the shutdown, it gives it 30 seconds by default. Locally for testing, I just tried docker stop (which should issue a sigterm) on the container, I don't see any logs saying it received a sigterm, so I'm not sure if it is actually gracefully shutting down rdf4j server.