Infinite retry loop with incorrect AmazonS3-PUSH transfer configuration in v0.11.1

[ppelgmv]

Hello,

We've encountered an issue while upgrading our EDC-based project to version 0.11.1. During testing, we discovered that transfers configured for AmazonS3-PUSH with incorrect information for data destination result in an infinite retry loop.

Specifically, the dataplane gets stuck in the RECEIVED state, and the onProcessReceived method (DataPlaneManagerImpl) is executed indefinitely. This occurs because the method transitions the state to STARTED, and the asynchronous process subsequently fails, triggering the onFailure handler. This, in turn, causes the process to loop back to the onProcessReceived method. Importantly, the stateCount does not increment as the state oscillates between RECEIVED and STARTED.

Note: It's possible that this behavior existed in earlier versions as well. However, we only noticed it after upgrading to version 0.11.1 due to the addition of a log line within the onProcessReceived method, which highlighted the repeated execution.

We would like to inquire whether this behavior is intended.

Could you please provide some insights on this issue?

Thanks in advance for your help.

[ndr-brt]

Could you please provide some insights on this issue?

We could only after you'll provide some insights to us :) e.g. how to reproduce could be a good starting point

[ppelgmv]

These are the steps to reproduce:

1. Provider connector setup:

   • Create an asset with data hosted on a MinIO.
   • Define a policy and a contract definition for the dataset.
   • Dataset is now in the catalog.

2. Consumer connector setup:

   • Use another connector (the consumer) to access the provider's catalog.
   • Negotiate a contract for the dataset's offering. The contract negotiation should complete successfully.

3. Initiate transfer with incorrect S3 configuration:

   • Using the contract agreement, initiate an Amazon S3-Push transfer.
   • Introduce incorrect dataDestination information in the transfer request (to simulate an error caused by a user entering data incorrectly). E.g.

{
	"@context": {
		"@vocab": "https://w3id.org/edc/v0.0.1/ns/"
	},
	"@type": "TransferRequestDto",
	"connectorId": "provider",
	"counterPartyAddress": "http://localhost:19194/protocol",
	"contractId": "{{contract-agreement-id}}",
	"protocol": "dataspace-protocol-http",
	"transferType": "AmazonS3-PUSH",
	"dataDestination": {
		"type": "AmazonS3",
		"region": "fake-data",
		"bucketName": "fake-data",
		"keyPrefix": "fake-data",
		"folderName": "fake-data",
		"accessKeyId": "fake-data",
		"endpointOverride": "fake-data",
		"secretAccessKey": "fake-data"
	}
}

4. Observe transfer state on consumer side:

   • Check the transfer's state on the consumer side. It should remain stuck in the STARTED state indefinitely.

5. Observe provider behavior:

   • On the provider side, observe the dataplane. The dataplane get into a infinit loop between RECEIVED and STARTED states.

Note: If the dataDestination values are correct, the transfer completes successfully.

Please let me know if you require any further information.

Thank you!

[ndr-brt]

I think this is related to #4860

[correiaafonso12]

Hey everyone.
I'll post some code references to help highlighting what is causing the infinite retry loop.

The RetryProcess verifies if the retries were exhausted using the stateCount property of the underlying StatefulEntity

// RetryProcess, lines 88 - 90
protected boolean retriesExhausted(E entity) {
    return entity.getStateCount() > configuration.getRetryLimit();
}

The stateCount is incremented everytime the StatefulEntity transitions to the current state, and resets to 1 if it changes to another state.

// StatefulEntity, lines 94 - 99
protected void transitionTo(int targetState) {
    stateCount = state == targetState ? stateCount + 1 : 1;
    state = targetState;
    updateStateTimestamp();
    setModified();
}

During the processing of the RECEIVED state, the DataFlow transitions to STARTED before performing the data transfer (which is wrapped in a RetryProcess). On failure, the DataFlow transitions back to RECEIVED, so that it can be processed again.

// DataPlaneManagerImpl, lines 244 - 281
private boolean processReceived(DataFlow dataFlow) {
    var request = dataFlow.toRequest();
    var transferService = transferServiceRegistry.resolveTransferService(request);

    if (transferService == null) {
        dataFlow.transitToFailed("No transferService available for DataFlow " + dataFlow.getId());
        update(dataFlow);
        return true;
    }

    dataFlow.transitionToStarted(runtimeId); // This transition ensures that the DataFlow's stateCount will always be 1
    monitor.info("UPDATE dataflow %s. RuntimeId %s, UpdatedAt %s".formatted(dataFlow.getId(), dataFlow.getRuntimeId(), dataFlow.getUpdatedAt()));
    update(dataFlow);

    return entityRetryProcessFactory.doAsyncProcess(dataFlow, () -> transferService.transfer(request))
            .entityRetrieve(id -> store.findByIdAndLease(id).orElse(f -> null))
            .onSuccess((f, r) -> {
                if (f.getState() != STARTED.code()) {
                    return;
                }

                if (r.succeeded()) {
                    f.transitToCompleted();
                } else {
                    f.transitToFailed(r.getFailureDetail());
                }
                update(f);
            })
            .onFailure((f, t) -> {
                f.transitToReceived(); // processReceived() will be executed again, but the stateCount won't increment
                update(f);
            })
            .onRetryExhausted((f, t) -> {
                f.transitToFailed(t.getMessage());
                update(f);
            })
            .execute("start data flow");
}

The constant oscillation between RECEIVED and STARTED doesn't allow the stateCount to increment, and therefore the retry mechanism will always assume that it's on the first attempt. Transitioning to STARTED was added due to this.

NOTE: These references are from v0.11.1! The latest developments in main are using the new RetryProcessor from commit eee070b onwards. Nevertheless, if the onFailure() handler is executed, the same issue will still occur.

[ndr-brt]

@correiaafonso12 thanks for analyzing this.
@ppelgmv could you please open an issue about this?

[ppelgmv]

I've created the issue: #4874
Thank you @correiaafonso12, I've added that information to the issue