How the provider could define a backchannel for an asset

[ndr-brt]

Currently the decision about setting a backchannel during a transfer stands on the consumer side, by adding the -backchannelType suffix in the transferType, but that should be a provider decision.

A proposal could be to have the provider being able to define this in the asset creation, so that in the catalog the backchannel type will be already visible by the consumer.

any thoughts?

[jimmarino]

This was already defined in the DR a while back. Backchannels must always be decided by the provider.

[ndr-brt]

yes, but that's not the current implementation and I think we should track this into an issue, to prevent forgetfulness :)

[jimmarino]

Yes then we should definitely fix it.

[rafaelmag110]

Not only that, but from my understanding of the concept document, the provider would have flexibility in deciding the type of response channel for each asset it offers. This flexibility would be given in the form of additional details for the asset data address, where even a different response channel type could be used.

i.e. the following pseudo json-ld (sorry for the lack of coherence. Its just for explanation purposes)

{
"assetId":"assetX"
"dataAddress":{
        "type":"HttpData"
        "baseUrl":"https://mock.com"
    }
"responseChannelDataAddress":{
        "type": "Kafka"
        "kafkaBroker":"mock.broker.domain"
    }
}

[jimmarino]

Kind of, but the backchannel needs to be specifically advertised as part of the distribution format, not arbitrarily chosen later. The formats can be dynamically generated by introspecting the data address type.

[rafaelmag110]

@ndr-brt @jimmarino
Picking up this topic again as I'd like to follow up on it in the next weeks.
Before I come up with any weird proposal, is there any idea that jumps to mind when thinking about this that I should look into?

[jimmarino]

Weird proposals are sometimes good ;-) I think we had a write-up somewhere about back-channels, and the important thing was the provider is always in control of the back-channel endpoint, which simplifies things greatly.

I also think we need to define a property in the DataAdress which would contain the endpoint information and (optionally) access tokens.

[rafaelmag110]

Given your inputs, here come the weird proposal.

Let's work this backwards, from the generation of an EndpointDataReference containing response channel information to how a provider should define it. It should be easier to understand my reasoning.

Generating EDRs with a response channel

EndpointDataReference objects are created by the DataplaneAuthorizationService in a combination of actions from the PublicEndpointGeneratorService and DataplaneAccessTokenService. In my perspective, and considering the arguments provided in both the initial concept and the decision record that mention response channels can be either HTTP endpoints or a messaging layer, the current implementation is too tied to HTTP endpoints only. This is due to the current solution only providing means to inject an auth token, public endpoint and endpoint type into the response data address. This does not align with what was proposed in the initial concept which mentions that a new responseChannel attribute of type DataAddress should be added to the outer DataAddress (or in another term, the EDR). So, the DataplaneAuthorizationService.createEndpointDataReference() method should be adapted to exihibt the required behavior. Also, I don't believe the endpoint generator should distinguish between generating endpoints for forward or backward channels. Generating an endpoint for a data address of a certain type should always result in the same endpoint. I propose that we harmonize this behavior and remove the PublicEndpointGeneratorService.generateResponseFor() method.

In this direction, the Endpoint record is too oriented to HTTP endpoints only, even if its definition leds us to believe it can be of any type. My proposal is to change the initial parameter of the record from a String to a Map of properties, which will enable different implementations of of generator functions to create richer endpoints (a kafka public endpoint, for example).

Getting response channel information to the DataplaneAuthorizationService.

As it stands, the DataplaneAuthorizationService generates response channel information based on a configurable parameter of the connectors' runtime, and each connector installation can only have one response channel. To allow for multiple response channels to be defined, and for those channels to be owned and defined by the data provider, there needs to be a way for them create and manage these response channels. Requirements I gathered for this are:

• A data provider should be able to create/update/remove response channel information
• A data provider should be able to link a response channel to a data offer

Once again starting backwards from the DataplaneAuthorizationService, the information vehicle that contains all data required for the generation of an EDR is a DataFlowStartMessage, and I saw no objections in proposing that response channel information should be propagated all the way from the TransferProcessManager, hoping on a ride in the TransferProcess when one is initialized until the a dataflow start message is built in the DataFlowController.

Of course response channel details should be persisted and managed somehow, and this is here I faced some doubts.

Alternative 1 - Attaching response channel details to Asset

Response channel details are nothing other than an set of properties of an address, and since the DataPlaneAccessTokenService and the PublicEndpointGeneratorService already know how to handle DataAddresss', it is natural to decide to store this information under this format.

My initial proposal would be to create a new responseChannel attribute in Asset, of type DataAddress. But since this response channel can be an optional attribute, dealing with this setup would be cumbersome as there handling null attributes is not elegant and there is no way to declare an "empty" data address.
Though, this option had the benefit of making the rest of the required changes straightforward, since we could persist response channel information in the asset table and query it the same way for the forward data address.

My other proposal would then be not to create the responseChannel attribute as a DataAddress of Asset, but as a property of it that points to another Asset. Asset should also expose a hasResponseChannel() method to help identify assets that have a response channel and need to be handled differently (i.e. need a specific distribution resolution method) and isResponseChannel() that should help filter these response channel assets from being considered as valid catalog offers.
The only downsides of this proposal are the extra effort required to filter response channel assets from a catalog and the effort required to resolve the actual data address of an assets' corresponding response channel. For example, for the TransferProcessManager to inject the response channel details in the TransferProcess. Here I'd propose adding a new method to the DataAddressResolver interface that given an assetId should resolve the corresponding response channel data address.

Alternative 2 - Specific API for managing ResponseChannel entities

Create a new Entity of type ResponseChannel that is very similar to an asset (in the sense it has an id, might have properties, and has a data address), and create an API to manage it. The API adds a controller, service and store so the effort to implement this is considerable and we have the additional penalty of introducing yet another store to the connector (which will require a connection from the pool leading to increased database usage).
The benefit is that we can inject the ResponseChannelService into the other services that might need it for resolving the response channel data address.
We also don't need to change the DatasetResolver to filter out response channel assets.

Please let me know your thoughts as there is a high chance i'm over complicating this stuff...

[jimmarino]

Thanks for proposing this. The existing EDC architecture mostly accommodates bidirectional channels, so the scope of changes should be limited.

First, please keep the original design of encapsulating the back-channel DataAddress as a property of the single DataAddress associated with the Asset. An Asset should have a 1:1 relationship with the DataAddress, and the backchannel is an addressing characteristic, not a quality of the asset itself.

The second thing is that PublicEndpointGeneratorService was explicitly designed to distinguish between forward and backchannel cases. While some generation types may result in one method, based on experience, this is not always the case.

Please be careful not to conflate an Asset for modelling the physical qualities of how data is transferred. An Asset is a logical representation of data, not the transfer itself. Assets are associated with transfer types (the Dataset format), but they are distinct concepts.

I do agree we need to abstract backchannel generation from the HTTP cases by supporting complex type parameters. Can you reformulate Alternative 1 along the lines above?

[rafaelmag110]

Thanks for the feedback.

Played around with the idea of having the response channel as a property of the single Asset DataAddress. What I tried first was nesting a response channel DataAddress within the outer data address, but failed to do so because serdes was being quite complex (at least for me). Even wondered if this is a practice we want to follow?

Example:

{
"assetId":"assetX"
"dataAddress":{
        "type":"HttpData"
        "baseUrl":"https://mock.com"
        "responseChannel":{
            "type": "Kafka"
            "kafkaBroker":"mock.broker.domain"
        }
    }

}

My other alternative would be to have the responseChannel property as a link to another asset. But then this would require resolving the data address from that asset and carrying it over within the TransferProcess to the DataFlowStartMessage. Don't think this falls under your definition of limited scope of changes.

Nevertheless I also feel the first option is best. In this sense i can see the following required work:

1. The Asset DataAddress should allow the definition of complex endpoint properties. Currently only strings are allowed.
2. The DefaultDistributionResolver should be adapted to only compute distributions that include a responseChannel transfer type, when one is defined in the Assets DataAddress.
3. The DataplaneAuthorizationService should use the responseChannel data address to generate the endpoint and data address for the response channel.

Number 1 should be done first and then we can do 2 and 3 simultaneously.
At this point we achieve the requirement of allowing a provider to define a specific response channel for an Asset (as long as its an HTTP one).

Then, to allow for complex type responseChannels, we can adapt how the DataplaneAuthorizationService builds the responseChannel data address - since now its tailored for HTTP only. To allow for this, the Endpoint class should become more complex and for this I'd propose the usage of a Map<String, Object> properties attribute, that is populated by generator functions.

If this makes sense to you I can create the corresponding issues and start working on them.

[jimmarino]

That sounds about right, and I like the idea of staging changes. One question: why is deserialization difficult? It should be fairly straightforward.

[rafaelmag110]

Most likely i'm making it more complex than it should be. I'll give this another go with a fresh pair of eyes. In case I fail for any reason i'll come here for support.
Many thanks for the discussion!