Skip to content

Make accessKey and secretKey optional and add support to use IRSA Iam role #1716

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

Make accessKey and secretKey optional and add support to use IRSA Iam role #1716

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
0a9223b
Make accessKey and secretKey optional and add support to use IRSA Iam…
Anany-k Jul 21, 2025
0733671
Add support for IRSA role with object store
Anany-k Jul 21, 2025
2c72299
Fix test
Anany-k Jul 21, 2025
bf8b27d
Remove .js extension when importing
Anany-k Jul 21, 2025
ae6b38b
revert
Anany-k Jul 21, 2025
331781d
Merge branch 'gristlabs:main' into main
Anany-k Aug 27, 2025
8d10813
Add an env var GRIST_DOCS_MINIO_AUTH_AUTO_DETECT to when should try l…
Anany-k Aug 27, 2025
157f687
Merge branch 'gristlabs:main' into main
Anany-k Sep 5, 2025
5161271
Merge branch 'gristlabs:main' into main
Anany-k Sep 7, 2025
94c9c8a
Update how minio client is initliazed in the constructor
Anany-k Sep 7, 2025
54b36f8
Fix eslint warning
Anany-k Sep 7, 2025
600374a
Merge branch 'gristlabs:main' into main
Anany-k Sep 29, 2025
350e181
Merge branch 'gristlabs:main' into main
Anany-k Oct 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions app/server/lib/MinIOExternalStorage.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import {IncomingMessage} from 'http';
import * as fse from 'fs-extra';
import * as minio from 'minio';
import * as stream from 'node:stream';
import {IamAwsProvider} from 'minio/dist/main/IamAwsProvider.js';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

imports should be ordered alphabetically. The ".js" presumably isn't needed? Also if the import could be just from 'minio' that would be more consistent than reaching into the specific package layout (minio/dist/main). But sometimes packages are wonky, so, fine if unavoidable.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Not a js/ts expert)
I was not able to import from minio directly, looks like it does not export IamAwsProvider.
Also, when remove .js and just use import {IamAwsProvider} from 'minio/dist/main/IamAwsProvider; I get the below error.

node:internal/modules/cjs/loader:657
      throw e;
      ^

Error: Cannot find module '/grist/node_modules/minio/dist/main/IamAwsProvider'
    at createEsmNotFoundErr (node:internal/modules/cjs/loader:1419:15)
    at finalizeEsmResolution (node:internal/modules/cjs/loader:1408:15)
    at resolveExports (node:internal/modules/cjs/loader:650:14)
    at Function._findPath (node:internal/modules/cjs/loader:717:31)
    at Function._resolveFilename (node:internal/modules/cjs/loader:1369:27)
    at defaultResolveImpl (node:internal/modules/cjs/loader:1025:19)
    at resolveForCJSWithHooks (node:internal/modules/cjs/loader:1030:22)
    at Function._load (node:internal/modules/cjs/loader:1192:37)
    at TracingChannel.traceSync (node:diagnostics_channel:322:14)
    at wrapModuleLoad (node:internal/modules/cjs/loader:237:24)
    at Module.require (node:internal/modules/cjs/loader:1463:12)
    at require (node:internal/modules/helpers:147:16)
    at Object.<anonymous> (/grist/_build/app/server/lib/MinIOExternalStorage.js:39:26)
    at Module._compile (node:internal/modules/cjs/loader:1706:14)
    at Object..js (node:internal/modules/cjs/loader:1839:10)
    at Module.load (node:internal/modules/cjs/loader:1441:32) {
  code: 'MODULE_NOT_FOUND',
  path: '/grist/node_modules/minio'
}

Node.js v22.19.0


// The minio-js v8.0.0 typings are sometimes incorrect. Here are some workarounds.
interface MinIOClient extends
Expand Down Expand Up @@ -45,6 +46,8 @@ type RemoveObjectsResponse = null | undefined | {
* will work with MinIO and other S3-compatible storage.
*/
export class MinIOExternalStorage implements ExternalStorage {
private _s3: MinIOClient;

// Specify bucket to use, and optionally the max number of keys to request
// in any call to listObjectVersions (used for testing)
constructor(
Expand All @@ -53,13 +56,17 @@ export class MinIOExternalStorage implements ExternalStorage {
endPoint: string,
port?: number,
useSSL?: boolean,
accessKey: string,
secretKey: string,
accessKey?: string,
secretKey?: string,
credentialsProvider?: IamAwsProvider,
region: string
},
private _batchSize?: number,
private _s3 = new minio.Client(options) as unknown as MinIOClient
private _batchSize?: number
) {
Copy link

Copilot AI Sep 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing validation to ensure that either credentials (accessKey/secretKey) or authAutoDetect is provided. The code should validate that at least one authentication method is configured to prevent runtime failures.

Suggested change
) {
) {
// Validate that at least one authentication method is configured
const hasAccessKeys = Boolean(options.accessKey && options.secretKey);
if (!hasAccessKeys && !options.authAutoDetect) {
throw new ApiError('MinIOExternalStorage requires either accessKey/secretKey or authAutoDetect to be set for authentication.', 400);
}

Copilot uses AI. Check for mistakes.

if (!options.accessKey || !options.secretKey) {
options.credentialsProvider = new IamAwsProvider({});
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this work? If the id/secret are set, and autodetect is also set, will the id/secret be used?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like if autodetect is set, id/secret will be ignored.
It's due to how minio-js works right now. ref code.

Initially, my thought was to not have authAutoDetect flag.
If id/secret env vars are not set we can by default try and load IRSA creds.

Let me know your thoughts.
Thanks

}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I imagine an administrator having written a typo in the env variables for either the access key and the secret key.

I think they would be confused by having switched implicitly to the IRSA authentication method.

Also it seems that it is not the only alternative to exist to authenticate with MinIO, currently there also exists AssumeRoleProvider (I don't ask to implement it, if people ask for supporting this, then it should be in another future PR).

What do you think of something like the following instead:

  1. introduce an env variable, like GRIST_DOCS_MINIO_AUTH_TYPE (or another meaningful name) that would either equal to credentials or to IRSA;
  2. by default, GRIST_DOCS_MINIO_AUTH_TYPE equals to credentials for the sake of backward compatibility;
  3. if GRIST_DOCS_MINIO_AUTH_TYPE equals to credentials, the env variables GRIST_DOCS_MINIO_ACCESS_KEY and GRIST_DOCS_MINIO_SECRET_KEY are mandatory in configureMinIOExternalStorage.ts. Otherwise these values are not read.

Also you may indeed instantiate directly a CredentialProvider or a IamAwsProvider directly in configureMinIOExternalStorage.ts and pass it as the credentialsProvider property to this class (and remove the accessKey and secretKey from the options here).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it probably should be similar to official sdk that just goes through the credential provider chain without having to explicitly specify what method we wanna use.

I believe the minio client kind of does that here.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. Therefore I think you should probably detect the presence of the env variables that should be set for the IRSA, and if they are missing, then switch back to the classic auth method with credentials.

Please note that I am not an authority on this project, I'll let Grist Labs review your work on their side.

Copy link
Author

@Anany-k Anany-k Aug 27, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a env var GRIST_DOCS_MINIO_AUTH_AUTO_DETECT to try and load the credentials according to credential provider chain if GRIST_DOCS_MINIO_AUTH_AUTO_DETECT: true

this._s3 = new minio.Client(options) as unknown as MinIOClient;
}

public async exists(key: string, snapshotId?: string) {
Expand Down
4 changes: 2 additions & 2 deletions app/server/lib/configureMinIOExternalStorage.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,11 @@ export function checkMinIOExternalStorage() {
const useSSL = settings.flag('useSsl').read({
envVar: ['GRIST_DOCS_MINIO_USE_SSL'],
}).getAsBool();
const accessKey = settings.flag('accessKey').requireString({
const accessKey = settings.flag('accessKey').readString({
envVar: ['GRIST_DOCS_MINIO_ACCESS_KEY'],
censor: true,
});
const secretKey = settings.flag('secretKey').requireString({
const secretKey = settings.flag('secretKey').readString({
envVar: ['GRIST_DOCS_MINIO_SECRET_KEY'],
censor: true,
});
Comment on lines +43 to 50
Copy link

Copilot AI Sep 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since accessKey and secretKey are now optional, there should be validation logic to ensure that when authAutoDetect is false, both accessKey and secretKey are provided. Currently, the function could return undefined values for these required credentials.

Copilot uses AI. Check for mistakes.

Expand Down
27 changes: 21 additions & 6 deletions test/server/lib/MinIOExternalStorage.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,18 +29,33 @@ describe("MinIOExternalStorage", function () {
sandbox.restore();
});

// Extend MinIOExternalStorage to allow injecting a mock client for testing
class TestMinIOExternalStorage extends MinIOExternalStorage {
constructor(
bucket: string,
options: any,
batchSize?: number,
mockClient?: any
) {
super(bucket, options, batchSize);
if (mockClient) {
(this as any)._s3 = mockClient;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this differ from what MinIOExternalStorage already does?

Copy link
Author

@Anany-k Anany-k Sep 7, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modified the MinIOExternalStorage constructor a bit, so we don't need this

}
}
}

describe('upload()', function () {
const filename = "some-filename";
let filestream: fse.ReadStream;
let s3: sinon.SinonStubbedInstance<minio.Client>;
let extStorage: MinIOExternalStorage;
let extStorage: TestMinIOExternalStorage;

beforeEach(function () {
filestream = new stream.Readable() as any;
sandbox.stub(fse, "lstat").resolves({} as any);
sandbox.stub(fse, "createReadStream").withArgs(filename).returns(filestream as any);
s3 = sandbox.createStubInstance(minio.Client);
extStorage = new MinIOExternalStorage(
extStorage = new TestMinIOExternalStorage(
dummyBucket,
dummyOptions,
undefined,
Expand Down Expand Up @@ -96,7 +111,7 @@ describe("MinIOExternalStorage", function () {

s3.listObjects.returns(fakeStream);

const extStorage = new MinIOExternalStorage(dummyBucket, dummyOptions, 42, s3 as any);
const extStorage = new TestMinIOExternalStorage(dummyBucket, dummyOptions, 42, s3 as any);
const result = await extStorage.versions(key);

assert.deepEqual(result, []);
Expand All @@ -122,7 +137,7 @@ describe("MinIOExternalStorage", function () {
]);

s3.listObjects.returns(fakeStream);
const extStorage = new MinIOExternalStorage(dummyBucket, dummyOptions, 42, s3 as any);
const extStorage = new TestMinIOExternalStorage(dummyBucket, dummyOptions, 42, s3 as any);
// when
const result = await extStorage.versions(key);
// then
Expand Down Expand Up @@ -155,7 +170,7 @@ describe("MinIOExternalStorage", function () {
let {fakeStream} = makeFakeStream(objectsFromS3);

s3.listObjects.returns(fakeStream);
const extStorage = new MinIOExternalStorage(dummyBucket, dummyOptions, 42, s3 as any);
const extStorage = new TestMinIOExternalStorage(dummyBucket, dummyOptions, 42, s3 as any);

// when
const result = await extStorage.versions(key);
Expand Down Expand Up @@ -193,7 +208,7 @@ describe("MinIOExternalStorage", function () {
.returns(fakeStream as any)
.callsFake(() => fakeStream.emit('error', error));
s3.listObjects.returns(fakeStream);
const extStorage = new MinIOExternalStorage(dummyBucket, dummyOptions, 42, s3 as any);
const extStorage = new TestMinIOExternalStorage(dummyBucket, dummyOptions, 42, s3 as any);

// when
const result = extStorage.versions(key);
Expand Down
Loading