Skip to content

Commit 0366e9c

Browse files
PauloLory-ionosjannotti-glaucioGlaucio Jannotti
authored
Version 2.3.0 (#3)
* feat: s3 endpoint regions validation (Digital-Ecosystems#80) * fix: regions endpoint signature changes (Digital-Ecosystems#82) Co-authored-by: Glaucio Jannotti <glaucio.jannotti@dengun.com> --------- Co-authored-by: Glaucio Jannotti <111659831+jannotti-glaucio@users.noreply.github.com> Co-authored-by: Glaucio Jannotti <glaucio.jannotti@dengun.com>
1 parent 6f982f5 commit 0366e9c

File tree

32 files changed

+203
-237
lines changed

32 files changed

+203
-237
lines changed

‎README.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -59,15 +59,15 @@ The credentials can be found/configured in one of the following:
5959

6060
It is required to configure those parameters:
6161

62-
| Parameter name | Description | Mandatory |
63-
|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|
64-
| `edc.ionos.access.key` | IONOS Access Key Id to access S3 | Yes if the context is accessing file |
65-
| `edc.ionos.secret.access.key` | IONOS Secret Access Key to access S3 | Yes if the context is accessing file |
66-
| `edc.ionos.token` | IONOS token to allow S3 provisioning | Yes if the context is provisioning access for others |
67-
| `edc.ionos.endpoint` | IONOS S3 endpoint address. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/endpoints) for further information. | Yes, if the context is accessing file | No, the default value is |
68-
| `edc.ionos.max.files` | Maximum number of files retrieved by list files function. | No, the default value is 5,000 files |
69-
| `edc.ionos.key.validation.attempts` | Maximum number of attemps to validate a temporary key after its creation. | No, the default values is 10 attempts |
70-
| `edc.ionos.key.validation.delay` | Time to wait (in milisseconds) before each key validation attempt. In each new attempt the delay is multiplied by the attempt number. | No, the default value is 3,000 (3 seconds) |
62+
| Parameter name | Description | Mandatory |
63+
|-------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------|
64+
| `edc.ionos.access.key` | IONOS Access Key Id to access S3 | Yes if the context is accessing file |
65+
| `edc.ionos.secret.access.key` | IONOS Secret Access Key to access S3 | Yes if the context is accessing file |
66+
| `edc.ionos.token` | IONOS token to allow S3 provisioning | Yes if the context is provisioning access for others |
67+
| `edc.ionos.endpoint.region` | IONOS S3 endpoint region. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/endpoints) for further information. | No, the default value is de |
68+
| `edc.ionos.max.files` | Maximum number of files retrieved by list files function. | No, the default value is 5,000 files |
69+
| `edc.ionos.key.validation.attempts` | Maximum number of attemps to validate a temporary key after its creation. | No, the default values is 10 attempts |
70+
| `edc.ionos.key.validation.delay` | Time to wait (in milisseconds) before each key validation attempt. In each new attempt the delay is multiplied by the attempt number. | No, the default value is 3,000 (3 seconds) |
7171

7272
To create the token please take a look at the following [documentation](./ionos_token.md).
7373

‎assets.md‎

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -9,23 +9,23 @@ The asset registration aims to specify which file/folder we want to share. We ca
99
### Requirements
1010

1111

12-
| Parameter | Description | Mandatory |
13-
|------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
14-
| `storage` | IONOS S3 endpoint address. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/s3-endpoints) for further information. | yes |
15-
| `bucketName` | IONOS S3 bucket name. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/concepts/buckets) for further information. | yes |
16-
| `blobName` | File name or path to folder | yes |
17-
| `filterIncludes` | `filterIncludes` use regular expression that will be used to select the file name pattern from the asset's blobName that will be copied during the transfer <br/> * do not consider the blobName in the expression, but the path from it. example: blobName = folder1, filterIncludes=file1.csv, the file foloder1/file1.csv will be copied | no |
18-
| `filterExcludes` | `filterExcludes` use regular expression that will be used to select the file name pattern from the asset's blobName that will NOT be copied during the transfer <br/> | no |
12+
| Parameter | Description | Mandatory |
13+
|------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|
14+
| `region` | IONOS S3 endpoint region. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/s3-endpoints) for further information. | no, default value = de |
15+
| `bucketName` | IONOS S3 bucket name. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/concepts/buckets) for further information. | yes |
16+
| `blobName` | File name or path to folder | yes |
17+
| `filterIncludes` | `filterIncludes` use regular expression that will be used to select the file name pattern from the asset's blobName that will be copied during the transfer <br/> * do not consider the blobName in the expression, but the path from it. example: blobName = folder1, filterIncludes=file1.csv, the file foloder1/file1.csv will be copied | no |
18+
| `filterExcludes` | `filterExcludes` use regular expression that will be used to select the file name pattern from the asset's blobName that will NOT be copied during the transfer <br/> | no |
1919

2020
Note: if `filterIncludes` and `filterExcludes` parameters are satisfied, the files to be copied will be selected using the `filterIncludes` and after that selected list, the files that have the pattern defined in the `filterExcludes` will be ignored.
2121

2222

2323
## Example
2424

2525
```json
26-
"dataAddress":{
26+
"dataAddress": {
2727
"type": "IonosS3", //from EDC
28-
"storage": "s3-eu-central-1.ionoscloud.com",
28+
"region": "de,
2929
"bucketName": "mybucket",
3030
"blobName": "folder1/",
3131
"filterIncludes": "file1.csv",
@@ -40,19 +40,19 @@ The transfer of assets aims to transfer the files/folders from one connector to
4040
### Requirements
4141

4242

43-
| Parameter | Description | Mandatory |
44-
|--------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
45-
| `storage` | IONOS S3 endpoint address. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/s3-endpoints) for further information. | yes |
46-
| `bucketName` | IONOS S3 bucket name. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/concepts/buckets) for further information. | yes |
47-
| `path` | Path of destination where the file/folder will be placed. </br> *if the path not filled, the file will be placed in the root of the bucket. | no |
43+
| Parameter | Description | Mandatory |
44+
|--------------|----------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|
45+
| `region` | IONOS S3 endpoint region. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/s3-endpoints) for further information. | no, default value = de |
46+
| `bucketName` | IONOS S3 bucket name. Refer to [docs](https://docs.ionos.com/cloud/managed-services/s3-object-storage/concepts/buckets) for further information. | yes |
47+
| `path` | Path of destination where the file/folder will be placed. </br> *if the path not filled, the file will be placed in the root of the bucket. | no |
4848

4949

5050
## Example
5151

5252
```json
53-
"dataDestination":{
53+
"dataDestination": {
5454
"type": "IonosS3", //from EDC
55-
"storage": "s3-eu-central-1.ionoscloud.com",
55+
"region": "de",
5656
"bucketName": "mybucket",
5757
"path": "folder2/",
5858
"keyName": "mykey" //from EDC

‎deployment/README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ export TF_VAR_persistence_type='PostgreSQLaaS' # 'PostgreSQLaaS', 'PostgreSQL' o
4848
export TF_VAR_vaultname='vault' # optional if only 1 connector per cluster
4949
export TF_VAR_s3_access_key='' # S3 access key
5050
export TF_VAR_s3_secret_key='' # S3 secret key
51-
export TF_VAR_s3_endpoint='' # s3 endpoint (e.g. s3-eu-central-1.ionoscloud.com)
51+
export TF_VAR_s3_endpoint_region='' # s3 endpoint region (e.g. de)
5252
export TF_VAR_ionos_token='' # IONOS Cloud token, for further information: https://docs.ionos.com/cloud/managed-services/s3-object-storage/endpoints
5353

5454
# Required only if persistence_type is PostgreSQLaaS
@@ -86,7 +86,7 @@ In case you want to configure this Connector without Hashicorp Vault, you need t
8686

8787
```yaml
8888
ionos:
89-
endpoint: <YOUR-S3-ENDPOINT>
89+
region: <IONOS-S3-ENDPOINT-REGION>
9090
accessKey: <YOUR-KEY>
9191
secretKey: <YOUR-SECRET-KEY>
9292
token: <IONOS-TOKEN>

‎deployment/helm/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ The IONOS S3 Extension can be deployed to a Kubernetes cluster using the Helm ch
3535
# Add secrets to Vault
3636
kubectl exec --namespace edc-ionos-s3 -it vault-0 -- vault kv put secret/edc.ionos.access.key content=
3737
kubectl exec --namespace edc-ionos-s3 -it vault-0 -- vault kv put secret/edc.ionos.secret.key content=
38-
kubectl exec --namespace edc-ionos-s3 -it vault-0 -- vault kv put secret/edc.ionos.endpoint content=
38+
kubectl exec --namespace edc-ionos-s3 -it vault-0 -- vault kv put secret/edc.ionos.endpoint.region content=
3939
kubectl exec --namespace edc-ionos-s3 -it vault-0 -- vault kv put secret/edc.ionos.token content=
4040
```
4141

‎deployment/helm/edc-ionos-s3/templates/configmap.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ data:
1818
edc.participant.id={{ .Values.edc.participant.id }}
1919
edc.ionos.access.key={{ .Values.edc.ionos.accessKey }}
2020
edc.ionos.secret.key={{ .Values.edc.ionos.secretKey }}
21-
edc.ionos.endpoint={{ .Values.edc.ionos.endpoint }}
21+
edc.ionos.endpoint.region={{ .Values.edc.ionos.endpoint.region }}
2222
edc.ionos.token={{ .Values.edc.ionos.token }}
2323
edc.vault.hashicorp.url={{ .Values.edc.vault.hashicorp.url }}
2424
edc.vault.hashicorp.token={{ .Values.edc.vault.hashicorp.token }}

‎deployment/kind/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ Set environment variables
2626
export KUBECONFIG=path/to/kubeconfig
2727
export S3_ACCESS_KEY=''
2828
export S3_SECRET_KEY=''
29-
export S3_ENDPOINT=''
29+
export S3_ENDPOINT_REGION=''
3030
export IONOS_TOKEN=''
3131
```
3232

‎deployment/kind/scripts/deploy.sh‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,8 +34,8 @@ if [ -z `printenv S3_SECRET_KEY` ]; then
3434
echo "Stopping because S3_SECRET_KEY is undefined"
3535
exit 1
3636
fi
37-
if [ -z `printenv S3_ENDPOINT` ]; then
38-
echo "Stopping because S3_ENDPOINT is undefined"
37+
if [ -z `printenv S3_ENDPOINT_REGION` ]; then
38+
echo "Stopping because S3_ENDPOINT_REGION is undefined"
3939
exit 1
4040
fi
4141
if [ -z `printenv IONOS_TOKEN` ]; then
@@ -72,7 +72,7 @@ helm install -n edc-ionos-s3 --wait vault hashicorp/vault \
7272
export TF_VAR_kubeconfig=$KUBECONFIG
7373
export TF_VAR_s3_access_key=$S3_ACCESS_KEY
7474
export TF_VAR_s3_secret_key=$S3_SECRET_KEY
75-
export TF_VAR_s3_endpoint=$S3_ENDPOINT
75+
export TF_VAR_s3_endpoint_region=$S3_ENDPOINT_REGION
7676
export TF_VAR_ionos_token=$IONOS_TOKEN
7777
../terraform/vault-init/vault-init.sh
7878

‎deployment/terraform/deploy-services.sh‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,8 @@ if [[ -z `printenv TF_VAR_s3_secret_key` ]]; then
1919
exit 1
2020
fi
2121

22-
if [[ -z `printenv TF_VAR_s3_endpoint` ]]; then
23-
echo "Stopping because TF_VAR_s3_endpoint is undefined"
22+
if [[ -z `printenv TF_VAR_s3_endpoint_region` ]]; then
23+
echo "Stopping because TF_VAR_s3_endpoint_region is undefined"
2424
exit 1
2525
fi
2626

‎deployment/terraform/ionos-s3-deploy/main.tf‎

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -56,9 +56,6 @@ variable "pg_password" {
5656
default = "postgres"
5757
}
5858

59-
variable "s3_access_key" {}
60-
variable "s3_secret_key" {}
61-
variable "s3_endpoint" {}
6259
variable "ionos_token" {}
6360

6461
variable "vaultname" {
@@ -98,8 +95,8 @@ resource "helm_release" "edc-ionos-s3" {
9895
}
9996

10097
set {
101-
name = "edc.ionos.endpoint"
102-
value = var.s3_endpoint
98+
name = "edc.ionos.endpoint.region"
99+
value = var.s3_endpoint_region
103100
}
104101

105102
set {

‎deployment/terraform/vault-init/vault-init.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,5 +39,5 @@ fi
3939
# Add secrets to Vault
4040
kubectl --kubeconfig=$TF_VAR_kubeconfig exec --namespace $NAMESPACE -it "$TF_VAR_vaultname-0" -- vault kv put secret/edc.ionos.access.key content=$TF_VAR_s3_access_key
4141
kubectl --kubeconfig=$TF_VAR_kubeconfig exec --namespace $NAMESPACE -it "$TF_VAR_vaultname-0" -- vault kv put secret/edc.ionos.secret.key content=$TF_VAR_s3_secret_key
42-
kubectl --kubeconfig=$TF_VAR_kubeconfig exec --namespace $NAMESPACE -it "$TF_VAR_vaultname-0" -- vault kv put secret/edc.ionos.endpoint content=$TF_VAR_s3_endpoint
42+
kubectl --kubeconfig=$TF_VAR_kubeconfig exec --namespace $NAMESPACE -it "$TF_VAR_vaultname-0" -- vault kv put secret/edc.ionos.endpoint.region content=$TF_VAR_s3_endpoint_region
4343
kubectl --kubeconfig=$TF_VAR_kubeconfig exec --namespace $NAMESPACE -it "$TF_VAR_vaultname-0" -- vault kv put secret/edc.ionos.token content=$TF_VAR_ionos_token

0 commit comments

Comments
 (0)