Skip to content

Commit 542d326

Browse files
jannotti-ionosGlaucio Jannotti
jannotti-ionos
and
Glaucio Jannotti
authored
Multiple bucket regions (#89)
* chore: refactoring data-plane packages * feat: support to multiple regions * feat: support to multiple regions * feat: support to multiple regions * chore: refactoring provision packages * feat: disabling extension if token not provided * fix: validating if transfer dataSource not have name * feat: removing unused tags (accessKey and secretKey) from dataDestination * feat: disabling extension if token not provided * feat: refactoring validators * feat: deployment scripts review * feat: migrating to edc 0.10.1 * feat: migrating to edc 0.10.1 * feat: README review * feat: disabling extension if token not provided --------- Co-authored-by: Glaucio Jannotti <glaucio.jannotti@dengun.com>
1 parent 3981d9c commit 542d326

File tree

30 files changed

+579
-412
lines changed

30 files changed

+579
-412
lines changed

‎README.md‎

Lines changed: 120 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ The extension has the following dependencies:
4949
| `org.eclipse.edc:data-plane-selector-core` | Main features of the data plane selector |
5050

5151
### Configurations
52-
It is required to configure an `Access key` and a `Secret Access Key` from the IONOS S3 storage service.
52+
It is required to configure an `Authentication Token` [docs](https://docs.ionos.com/cloud/set-up-ionos-cloud/management/token-management) and a `S3 Access key` [docs](https://docs.ionos.com/cloud/storage-and-backup/ionos-object-storage/concepts/key-management) to use the extension.
5353

5454
The credentials can be found/configured in one of the following:
5555
- Vault;
@@ -59,17 +59,15 @@ The credentials can be found/configured in one of the following:
5959

6060
It is required to configure those parameters:
6161

62-
| Parameter name | Description | Mandatory |
63-
|-------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|
64-
| `edc.ionos.access.key` | IONOS Access Key Id to access S3 | Yes if the context is accessing file |
65-
| `edc.ionos.secret.access.key` | IONOS Secret Access Key to access S3 | Yes if the context is accessing file |
66-
| `edc.ionos.token` | IONOS token to allow S3 provisioning | Yes if the context is provisioning access for others |
67-
| `edc.ionos.endpoint.region` | IONOS S3 endpoint region. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/endpoints) for further information. | No, the default value is "de" |
68-
| `edc.ionos.max.files` | Maximum number of files retrieved by list files function. | No, the default value is 5,000 files |
69-
| `edc.ionos.key.validation.attempts` | Maximum number of attemps to validate a temporary key after its creation. | No, the default values is 10 attempts |
70-
| `edc.ionos.key.validation.delay` | Time to wait (in milisseconds) before each key validation attempt. In each new attempt the delay is multiplied by the attempt number. | No, the default value is 3,000 (3 seconds) |
71-
72-
To create the token please take a look at the following [documentation](./ionos_token.md).
62+
| Parameter name | Description | Mandatory |
63+
|-------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|
64+
| `edc.ionos.access.key` | IONOS Access Key Id to access S3 | Yes if the context is accessing file |
65+
| `edc.ionos.secret.access.key` | IONOS Secret Access Key to access S3 | Yes if the context is accessing file |
66+
| `edc.ionos.token` | IONOS Token to allow S3 provisioning | Yes if the context is provisioning access for others |
67+
| `edc.ionos.endpoint.region` | IONOS S3 endpoint default region. It will be used if a region is not defined in the dataAddress or dataDestination. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/endpoints) for further information. | No, the default value is "de" |
68+
| `edc.ionos.max.files` | Maximum number of files copied by S3 bucket folder. | No, the default value is 5,000 files |
69+
| `edc.ionos.key.validation.attempts` | Maximum number of attemps to validate a temporary key after its creation. | No, the default values is 10 attempts |
70+
| `edc.ionos.key.validation.delay` | Time to wait (in milisseconds) before each key validation attempt. In each new attempt the delay is multiplied by the attempt number. | No, the default value is 3,000 (3 seconds) |
7371

7472
## Building and Running
7573

@@ -93,9 +91,115 @@ cd launchers/prod/connector-persistence
9391
java -Dedc.fs.config=resources/config.properties -jar build/libs/dataspace-connector.jar
9492
```
9593

96-
## Examples
97-
In order to see working examples go to [edc-ionos-samples](https://github.com/Digital-Ecosystems/edc-ionos-samples).
98-
99-
## Deploying to IONOS Kubernetes
94+
## Deploying
10095
Check the [deployment readme](https://github.com/ionos-cloud/edc-ionos-s3/tree/main/deployment/README.md) to see how to deploy the Connector locally or to an external Kubernetes cluster.
10196

97+
## Usage
98+
99+
### DataAddress
100+
To create an asset using an IONOS S3 Bucket as its data address, use the following format:
101+
102+
To share a file:
103+
```
104+
{
105+
"@context": {
106+
"@vocab": "https://w3id.org/edc/v0.0.1/ns/")
107+
},
108+
"@id": "asset-671",
109+
"properties": {
110+
"name": "Test Asset"
111+
},
112+
"dataAddress": {
113+
"type": "IonosS3",
114+
"region": "de",
115+
"bucketName": "providerBucket",
116+
"blobName": "device1-data.csv"
117+
}
118+
}
119+
```
120+
121+
To share a folder:
122+
```
123+
{
124+
"@context": {
125+
"@vocab": "https://w3id.org/edc/v0.0.1/ns/")
126+
},
127+
"@id": "asset-671",
128+
"properties": {
129+
"name": "Test Asset"
130+
},
131+
"dataAddress": {
132+
"type": "IonosS3",
133+
"region": "de",
134+
"bucketName": "providerBucket",
135+
"blobName": "folder/",
136+
filter.includes: "*.csv"
137+
}
138+
}
139+
```
140+
141+
| Tag name | Description | Mandatory |
142+
|------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------|
143+
| dataAddress.type | This extension uses the `IonosS3` designation | Yes |
144+
| dataAddress.region | S3 Bucket region used to retrieve the S3 API endpoint. [Possible values](https://docs.ionos.com/cloud/storage-and-backup/ionos-object-storage/endpoints) | No. If not send the configuration `edc.ionos.endpoint.region` will be used |
145+
| dataAddress.bucketName | Name of the S3 Bucket used to store the asset data | Yes |
146+
| dataAddress.blobName | Path to a file or folder on the source S3 Bucket | Yes |
147+
| filter.includes | Regular expression to filter the files or folders to be copied from the blobName | No |
148+
| filter.excludes | Regular expression to filter the files or folders to be NOT be copied from the blobName | No |
149+
150+
### DataDestination
151+
To start a transfer using an IONOS S3 Bucket as its data destination, use the following formats:
152+
153+
To transfer to the root folder:
154+
```
155+
{
156+
"@context":{
157+
"edc":"https://w3id.org/edc/v0.0.1/ns/"
158+
},
159+
"connectorId":"provider",
160+
"counterPartyAddress":"http://localhost:8282/protocol",
161+
"contractId":"3186afb5-7b10-4665-b07b-233f5665eb98",
162+
"protocol":"dataspace-protocol-http",
163+
"transferType": "IonosS3-PUSH",
164+
"dataDestination":{
165+
"type":"IonosS3",
166+
"keyName":"4fc5ecaf-6630-4ce5-aacb-f42778a6a65b",
167+
"region": "de",
168+
"bucketName":"consumerBucket"
169+
}
170+
}
171+
```
172+
173+
To transfer to a folder:
174+
```
175+
{
176+
"@context":{
177+
"edc":"https://w3id.org/edc/v0.0.1/ns/"
178+
},
179+
"connectorId":"provider",
180+
"counterPartyAddress":"http://localhost:8282/protocol",
181+
"contractId":"3186afb5-7b10-4665-b07b-233f5665eb98",
182+
"protocol":"dataspace-protocol-http",
183+
"transferType": "IonosS3-PUSH",
184+
"dataDestination":{
185+
"type":"IonosS3",
186+
"keyName":"4fc5ecaf-6630-4ce5-aacb-f42778a6a65b",
187+
"region": "de",
188+
"bucketName":"consumerBucket",
189+
"path": "subFolder/"
190+
}
191+
}
192+
```
193+
194+
Note: Only the PUSH transfer type is supported.
195+
196+
| Tag name | Description | Mandatory |
197+
|------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------|
198+
| transferType | This extension uses the `IonosS3-PUSH` designation | Yes |
199+
| dataAddress.type | This extension uses the `IonosS3` designation | Yes |
200+
| dataAddress.keyName | Key name used to store the temporary S3 Keys during the transfer. Need to be an unique value. | Yes |
201+
| dataAddress.region | S3 Bucket region used to retrieve the S3 API endpoint. [Possible values](https://docs.ionos.com/cloud/storage-and-backup/ionos-object-storage/endpoints) | No. If not send, the configuration `edc.ionos.endpoint.region` will be used |
202+
| dataAddress.bucketName | Name of the destination S3 Bucket, to receive the transferred data | Yes |
203+
| dataAddress.path | Path of a folder, on the destination S3 Bucket, to receive the transferred data | No |
204+
205+
Note: the scope of this repo is NOT to explain the complete flows (and payloads) of the EDC Connector. If you want to know more please take a look at the [Eclipse EDC Samples](https://github.com/eclipse-edc/Samples).

‎extensions/core-ionos-s3/build.gradle.kts‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ val edcGroup: String by project
77
val edcVersion: String by project
88
val metaModelVersion: String by project
99
val minIOVersion: String by project
10+
val commonsCollectionsVersion: String by project
1011
val extensionsGroup: String by project
1112
val extensionsVersion: String by project
1213

@@ -19,7 +20,10 @@ dependencies {
1920
api("${edcGroup}:runtime-metamodel:${metaModelVersion}")
2021

2122
implementation("${edcGroup}:transfer-spi:${edcVersion}")
23+
implementation("${edcGroup}:validator-spi:${edcVersion}")
24+
2225
implementation("io.minio:minio:${minIOVersion}")
26+
implementation("org.apache.commons:commons-collections4:${commonsCollectionsVersion}")
2327
}
2428

2529
java {

‎extensions/core-ionos-s3/src/main/java/com/ionos/edc/extension/s3/IonosS3CoreExtension.java‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
/*
2+
* Copyright (c) 2022 IONOS
3+
*
4+
* This program and the accompanying materials are made available under the
5+
* terms of the Apache License, Version 2.0 which is available at
6+
* https://www.apache.org/licenses/LICENSE-2.0
7+
*
8+
* SPDX-License-Identifier: Apache-2.0
9+
*
10+
* Contributors:
11+
* IONOS
12+
*
13+
*/
14+
15+
package com.ionos.edc.extension.s3;
16+
17+
import com.ionos.edc.extension.s3.connector.S3Connector;
18+
import com.ionos.edc.extension.s3.connector.S3ConnectorImpl;
19+
import com.ionos.edc.extension.s3.schema.IonosBucketSchema;
20+
import com.ionos.edc.extension.s3.validators.IonosDataAddressValidator;
21+
import com.ionos.edc.extension.s3.validators.IonosDataDestinationValidator;
22+
import org.eclipse.edc.runtime.metamodel.annotation.Extension;
23+
import org.eclipse.edc.runtime.metamodel.annotation.Inject;
24+
import org.eclipse.edc.runtime.metamodel.annotation.Provides;
25+
import org.eclipse.edc.spi.security.Vault;
26+
import org.eclipse.edc.spi.system.ServiceExtension;
27+
import org.eclipse.edc.spi.system.ServiceExtensionContext;
28+
import org.eclipse.edc.validator.spi.DataAddressValidatorRegistry;
29+
30+
import static com.ionos.edc.extension.s3.schema.IonosSettingsSchema.IONOS_ACCESS_KEY;
31+
import static com.ionos.edc.extension.s3.schema.IonosSettingsSchema.IONOS_MAX_FILES;
32+
import static com.ionos.edc.extension.s3.schema.IonosSettingsSchema.IONOS_MAX_FILES_DEFAULT;
33+
import static com.ionos.edc.extension.s3.schema.IonosSettingsSchema.IONOS_REGION;
34+
import static com.ionos.edc.extension.s3.schema.IonosSettingsSchema.IONOS_REGION_DEFAULT;
35+
import static com.ionos.edc.extension.s3.schema.IonosSettingsSchema.IONOS_SECRET_KEY;
36+
import static com.ionos.edc.extension.s3.schema.IonosSettingsSchema.IONOS_TOKEN;
37+
38+
@Provides(S3Connector.class)
39+
@Extension(value = IonosS3CoreExtension.NAME)
40+
public class IonosS3CoreExtension implements ServiceExtension {
41+
42+
public static final String NAME = "IonosS3";
43+
44+
@Inject
45+
private Vault vault;
46+
47+
@Inject
48+
private DataAddressValidatorRegistry dataAddressValidatorRegistry;
49+
50+
@Override
51+
public String name() {
52+
return NAME;
53+
}
54+
55+
@Override
56+
public void initialize(ServiceExtensionContext context) {
57+
var monitor = context.getMonitor();
58+
var contextMonitor = monitor.withPrefix("IonosS3CoreExtension");
59+
60+
contextMonitor.debug("Loading configurations");
61+
62+
var accessKey = vault.resolveSecret(IONOS_ACCESS_KEY);
63+
var secretKey = vault.resolveSecret(IONOS_SECRET_KEY);
64+
var region = vault.resolveSecret(IONOS_REGION);
65+
var token = vault.resolveSecret(IONOS_TOKEN);
66+
67+
if (accessKey == null || secretKey == null || region == null || token == null) {
68+
contextMonitor.warning("Couldn't connect or the vault didn't return values, falling back to ConfigMap Configuration");
69+
accessKey = context.getSetting(IONOS_ACCESS_KEY, null);
70+
secretKey = context.getSetting(IONOS_SECRET_KEY, null);
71+
region = context.getSetting(IONOS_REGION, IONOS_REGION_DEFAULT);
72+
token = context.getSetting(IONOS_TOKEN, null);
73+
}
74+
75+
var maxFiles = Integer.valueOf(context.getSetting(IONOS_MAX_FILES, IONOS_MAX_FILES_DEFAULT));
76+
77+
if (accessKey == null || secretKey == null || token == null) {
78+
contextMonitor.warning("IONOS token and S3 key are not set, disabling IONOS S3 Connector");
79+
} else {
80+
contextMonitor.debug("Initializing S3 Connector");
81+
var s3Connector = new S3ConnectorImpl(region, accessKey, secretKey, token, maxFiles);
82+
context.registerService(S3Connector.class, s3Connector);
83+
}
84+
85+
contextMonitor.debug("Registering validators");
86+
dataAddressValidatorRegistry.registerSourceValidator(IonosBucketSchema.TYPE, new IonosDataAddressValidator());
87+
dataAddressValidatorRegistry.registerDestinationValidator(IonosBucketSchema.TYPE, new IonosDataDestinationValidator());
88+
89+
contextMonitor.debug("Core extension initialized !");
90+
}
91+
}

‎extensions/core-ionos-s3/src/main/java/com/ionos/edc/extension/s3/S3CoreExtension.java‎

Lines changed: 0 additions & 72 deletions
This file was deleted.

0 commit comments

Comments
 (0)