Skip to content

Commit 89f54e9

Browse files
KSDaemonGabeInDevOps
KSDaemon
and
GabeInDevOps
authored
feat(duckdb-driver): Add support for using default credential provider chain for duckdb s3 access (#9679)
* use default credential provider chain if access key and secret key are not provided * some polishment * update docs --------- Co-authored-by: Gabriel Rodriguez <grodriguez0394@gmail.com>
1 parent 273d277 commit 89f54e9

File tree

4 files changed

+69
-16
lines changed

4 files changed

+69
-16
lines changed

‎docs/pages/product/configuration/data-sources/duckdb.mdx

Lines changed: 17 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -59,22 +59,23 @@ deployment][ref-demo-deployment] in Cube Cloud.
5959

6060
## Environment Variables
6161

62-
| Environment Variable | Description | Possible Values | Required |
63-
| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | :------: |
64-
| `CUBEJS_DB_DUCKDB_MEMORY_LIMIT` | The maximum memory limit for DuckDB. Equivalent to `SET memory_limit=<MEMORY_LIMIT>`. Default is 75% of available RAM | A valid memory limit | ❌ | ✅ |
65-
| `CUBEJS_DB_DUCKDB_SCHEMA` | The [default search schema][link-duckdb-configuration-ref] | A valid schema name | ❌ | ✅ |
66-
| `CUBEJS_DB_DUCKDB_MOTHERDUCK_TOKEN` | The service token to use for connections to MotherDuck | A valid [MotherDuck service token][motherduck-docs-svc-token] | ❌ | ✅ |
67-
| `CUBEJS_DB_DUCKDB_DATABASE_PATH` | The database filepath to use for connection to a local database. | A valid duckdb database file path | ❌ | ✅ |
68-
| `CUBEJS_DB_DUCKDB_S3_ACCESS_KEY_ID` | The Access Key ID to use for database connections | A valid Access Key ID | ❌ | ✅ |
69-
| `CUBEJS_DB_DUCKDB_S3_SECRET_ACCESS_KEY` | The Secret Access Key to use for database connections | A valid Secret Access Key | ❌ | ✅ |
70-
| `CUBEJS_DB_DUCKDB_S3_ENDPOINT` | The S3 endpoint | A valid [S3 endpoint][duckdb-docs-s3-import] | ❌ | ✅ |
71-
| `CUBEJS_DB_DUCKDB_S3_REGION` | The [region of the bucket][duckdb-docs-s3-import] | A valid AWS region | ❌ | ✅ |
72-
| `CUBEJS_DB_DUCKDB_S3_USE_SSL` | Use SSL for connection | A boolean | ❌ | ❌ |
73-
| `CUBEJS_DB_DUCKDB_S3_URL_STYLE` | To choose the S3 URL style(vhost or path) | 'vhost' or 'path' | ❌ | ❌ |
74-
| `CUBEJS_DB_DUCKDB_S3_SESSION_TOKEN` | The token for the S3 session | A valid Session Token | ❌ | ✅ |
75-
| `CUBEJS_DB_DUCKDB_EXTENSIONS` | A comma-separated list of DuckDB extensions to install and load | A comma-separated list of DuckDB extensions | ❌ | ✅ |
76-
| `CUBEJS_DB_DUCKDB_COMMUNITY_EXTENSIONS` | A comma-separated list of DuckDB community extensions to install and load | A comma-separated list of DuckDB community extensions | ❌ | ✅ |
77-
| `CUBEJS_CONCURRENCY` | The number of [concurrent queries][ref-data-source-concurrency] to the data source | A valid number | ❌ |
62+
| Environment Variable | Description | Possible Values | Required |
63+
|--------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|:--------:|
64+
| `CUBEJS_DB_DUCKDB_MEMORY_LIMIT` | The maximum memory limit for DuckDB. Equivalent to `SET memory_limit=<MEMORY_LIMIT>`. Default is 75% of available RAM | A valid memory limit | ❌ |
65+
| `CUBEJS_DB_DUCKDB_SCHEMA` | The [default search schema][link-duckdb-configuration-ref] | A valid schema name | ❌ |
66+
| `CUBEJS_DB_DUCKDB_MOTHERDUCK_TOKEN` | The service token to use for connections to MotherDuck | A valid [MotherDuck service token][motherduck-docs-svc-token] | ❌ |
67+
| `CUBEJS_DB_DUCKDB_DATABASE_PATH` | The database filepath to use for connection to a local database. | A valid duckdb database file path | ❌ |
68+
| `CUBEJS_DB_DUCKDB_S3_ACCESS_KEY_ID` | The Access Key ID to use for database connections | A valid Access Key ID | ❌ |
69+
| `CUBEJS_DB_DUCKDB_S3_SECRET_ACCESS_KEY` | The Secret Access Key to use for database connections | A valid Secret Access Key | ❌ |
70+
| `CUBEJS_DB_DUCKDB_S3_ENDPOINT` | The S3 endpoint | A valid [S3 endpoint][duckdb-docs-s3-import] | ❌ |
71+
| `CUBEJS_DB_DUCKDB_S3_REGION` | The [region of the bucket][duckdb-docs-s3-import] | A valid AWS region | ❌ |
72+
| `CUBEJS_DB_DUCKDB_S3_USE_SSL` | Use SSL for connection | A boolean | ❌ |
73+
| `CUBEJS_DB_DUCKDB_S3_URL_STYLE` | To choose the S3 URL style(vhost or path) | `vhost` or `path` | ❌ |
74+
| `CUBEJS_DB_DUCKDB_S3_SESSION_TOKEN` | The token for the S3 session | A valid Session Token | ❌ |
75+
| `CUBEJS_DB_DUCKDB_EXTENSIONS` | A comma-separated list of DuckDB extensions to install and load | A comma-separated list of DuckDB extensions | ❌ |
76+
| `CUBEJS_DB_DUCKDB_COMMUNITY_EXTENSIONS` | A comma-separated list of DuckDB community extensions to install and load | A comma-separated list of DuckDB community extensions | ❌ |
77+
| `CUBEJS_DB_DUCKDB_S3_USE_CREDENTIAL_CHAIN` | A flag to use credentials chain for secrets for S3 connections | `true`, `false`. Defaults to `false` | ❌ |
78+
| `CUBEJS_CONCURRENCY` | The number of [concurrent queries][ref-data-source-concurrency] to the data source | A valid number | ❌ |
7879

7980
[ref-data-source-concurrency]: /product/configuration/concurrency#data-source-concurrency
8081

‎docs/pages/product/configuration/reference/environment-variables.mdx

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -375,6 +375,14 @@ A comma-separated list of DuckDB community extensions to install and load.
375375
| ----------------------------------------------------- | ---------------------- | --------------------- |
376376
| A comma-separated list of DuckDB community extensions | N/A | N/A |
377377

378+
## `CUBEJS_DB_DUCKDB_S3_USE_CREDENTIAL_CHAIN`
379+
380+
A flag to use credentials chain for secrets for S3 connections.
381+
382+
| Possible Values | Default in Development | Default in Production |
383+
| ----------------------------------------------------- | ---------------------- | --------------------- |
384+
| `true`, `false` | `false` | `false` |
385+
378386
## `CUBEJS_DB_ELASTIC_APIKEY_ID`
379387

380388
The [ID of the API key from elastic.co][elastic-docs-api-keys]. Required when

‎packages/cubejs-backend-shared/src/env.ts

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1784,6 +1784,7 @@ const variables: Record<string, (...args: any) => any> = {
17841784
}
17851785
return [];
17861786
},
1787+
17871788
duckdbCommunityExtensions: ({
17881789
dataSource
17891790
}: {
@@ -1797,6 +1798,36 @@ const variables: Record<string, (...args: any) => any> = {
17971798
}
17981799
return [];
17991800
},
1801+
1802+
duckdbS3UseCredentialChain: ({
1803+
dataSource
1804+
}: {
1805+
dataSource: string,
1806+
}) => {
1807+
const val = process.env[
1808+
keyByDataSource('CUBEJS_DB_DUCKDB_S3_USE_CREDENTIAL_CHAIN', dataSource)
1809+
];
1810+
1811+
if (val) {
1812+
if (val.toLocaleLowerCase() === 'true') {
1813+
return true;
1814+
} else if (val.toLowerCase() === 'false') {
1815+
return false;
1816+
} else {
1817+
throw new TypeError(
1818+
`The ${
1819+
keyByDataSource(
1820+
'CUBEJS_DB_DUCKDB_S3_USE_CREDENTIAL_CHAIN',
1821+
dataSource,
1822+
)
1823+
} must be either 'true' or 'false'.`
1824+
);
1825+
}
1826+
} else {
1827+
return false;
1828+
}
1829+
},
1830+
18001831
/** ***************************************************************
18011832
* Presto/Trino Driver *
18021833
**************************************************************** */

‎packages/cubejs-duckdb-driver/src/DuckDBDriver.ts

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ export type DuckDBDriverConfiguration = {
2222
initSql?: string,
2323
motherDuckToken?: string,
2424
schema?: string,
25+
duckdbS3UseCredentialChain?: boolean,
2526
};
2627

2728
type InitPromise = {
@@ -164,6 +165,18 @@ export class DuckDBDriver extends BaseDriver implements DriverInterface {
164165
}
165166
}
166167

168+
const useCredentialChain = this.config.duckdbS3UseCredentialChain || getEnv('duckdbS3UseCredentialChain', this.config);
169+
if (useCredentialChain) {
170+
try {
171+
await execAsync('CREATE SECRET (TYPE S3, PROVIDER \'CREDENTIAL_CHAIN\')');
172+
} catch (e) {
173+
if (this.logger) {
174+
console.error('DuckDB - error on creating S3 credential chain secret', { e });
175+
}
176+
throw e;
177+
}
178+
}
179+
167180
// Install & load extensions if configured in env variable.
168181
const officialExtensions = getEnv('duckdbExtensions', this.config);
169182
await this.installExtensions(officialExtensions, execAsync);

0 commit comments

Comments
 (0)