Move FedXRepository to its own module or merge with SailRepository

[abrokenjester]

The FedXRepository should be promoted to its own module: rdf4j-repository-federated. Alternatively, its functionality (which is a simple extension of SailRepository) could be moved into SailRepository, eliminating the need for a special subclass.

[aschwarte10]

Before starting the discussion here, give me a bit of more time. I think we do require the Repository implementation to move a bit of initialization here, particularly I plan to get rid of the static singletons and move them as instance fields to the repository. I will do write-up and change for this in the next days.

[abrokenjester]

If, hypothetically, we were to try and eliminate the need for a separate FedXRepository(Connection) class, we'd need to either push down the specific things they handle to the Sail level, or alternatively (if it's more generically useful) up to SailRepository.

• The initializeInternal method in FedXRepository looks to me to be mostly concerned with instantiating various management/pooling classes around handling access to the federation members endpoints. I would (naively) guess that this would not be hard to push down to the FedX class.
• The config stuff is probably something I'd like to see pushed up in some form: a more helpful configuration mechanism at the (Sail)Repository level would be beneficial.
• There's three additional public getter methods: getFederationSize, getFederationContext, and getQueryManager. We could conceivably push these down to the FedX Sail as well, but: are these intended to be accessed by end users? If so that might be a good argument for keeping the FedXRepository around.
• The additional functionality in FedXRepositoryConnection mostly revolves around setting additional bindings on the prepared query objects - this is something that can be easily pushed down to the FedX Sail level.

The question becomes: is it worth it to make these changes?

• Point in favor: it removes an additional couple of classes and makes repository initialization more homogenous.
• Point against: if there are functions that end users need to access directly, pushing those down to the Sail would make their coding life more difficult.

I also want to consider existing FedX users: the more we refactor things, the harder we make it for them to migrate to the new version.

[aschwarte10]

Thanks for the write up.

I agree that most things could be easily pushed down to the FedX (SAIL) with just a bit of moving code around. However, this needs to be discussed especially regarding the following points (which you partially already identify in your list above)

• we currently explicitly expose the QueryManager to users. Although this is not yet a well-defined interface for clients in all its methods, there are some really useful concepts here (getting the running queries, aborting a given query, obtaining a query plan, create a query with pre-defined prefixes, ...). If we move this to the SAIL we kind of hide this from the users. I am fine with this for now, but I think such an infrastructure (=> especially seeing the running queries and the possibility to abort) would also be very beneficial for other repository / sail implementations.
• access to the FederationContext currently provides access to certain federation that can be adjusted during runtime. Currently it is possible to define the federation members on demand, i.e. add / remove members. In on of our applications we had a UI to control federation members and used this concepts. This is also not strictly required to be exposed (e.g. you could achieve the same with a shutdown / re-initialize of the federation repository which may even be cleaner). Just wanted to mention the rationals.

Regarding above points it is merely a design decision and I would leave it to you @jeenbroekstra to make a decision. I assume at this point we do not have to care so much for existing FedX users, especially as I made quite a bit of changes to the lifecycle and configuration already (very similar, but not entirely backwards compatible).

One additional point why I actually introduced the RepositoryConnection (and did not do everything inside the SAIL) was that not all information s available in the SAIL, e.g. in the SAIL it is not possible to get hold on the query timeout defined on the query, also it is not possible to get hold of the original query string. My (hacky) workaround to overcome this limitation is to pass those concrete values as "bindings" to the query evaluation, i.e. to the FedX SAIL, where I actually remove them again before the actual evaluation starts.

As a side note: in FedX I do not rely on the wrapping TimeoutIteration, but have a timeout mechanism integrated in the evaluation. This is required to properly handle timeouts with all (remote) data sources. Thus I require the value of the defined timeout.

So to keep all existing functionality in place (without the explicit FedXRepositoryConnection) I would require enhancements to the SAIL infrastructure (=> access to the query timeout and the original query string).

Is above description clear enough to get an understanding of my rationals why it is implemented like this? @jeenbroekstra what's your opinion on the above points?

[abrokenjester]

One additional point why I actually introduced the RepositoryConnection (and did not do everything inside the SAIL) was that not all information s available in the SAIL, e.g. in the SAIL it is not possible to get hold on the query timeout defined on the query, also it is not possible to get hold of the original query string. My (hacky) workaround to overcome this limitation is to pass those concrete values as "bindings" to the query evaluation, i.e. to the FedX SAIL, where I actually remove them again before the actual evaluation starts.

For this reason, and perhaps others as well (such as more advanced query rewriting / drop-in parser replacements) I keep thinking we should look into optionally pushing the prepareQuery operations down to the Sail level, so that a Sail implementation, if it wants to, can manipulate the query parsing and planning at this stage instead of just at evaluation time.

More generally: I think you bring up some very good points. I think the best way forward for now is that we de-prioritize this issue, and do a first release of the FedX code with the Repository implementation as-is - though I'd at least like it marked as experimental in the Javadoc, to give us a bit more leeway if we want to change the signature later.

Refactoring as we discussed here can then be separately planned as part of a wider effort to make some of these features available for the general infrastructure. We'll likely split that out into separate issues and, I suspect, some of them will need to be part of a major release.