ES 1.3.1 testing against a Minio S3-compatible storage server - please help

[hrstoyanov]

Hello,
I am trying to test ES 1.3.1 (with AWS Java SDK 2.25.18) against a local Minio server - a highly compatible AWS S3 storage server, widely for local development/testing of S3.

This setup code:

var awsBasicCredentials = AwsBasicCredentials.create(s3configuration.accessKey, s3configuration.secretKey);

            var s3Client = S3Client.builder()
                    .region(isEmptyOrNull(s3configuration.region)? Region.AWS_ISO_GLOBAL: Region.of(s3configuration.region))
                    .endpointOverride(s3configuration.endpoint)
                    .credentialsProvider(StaticCredentialsProvider.create(awsBasicCredentials))
                    .build();

            var buckets = s3Client.listBuckets().buckets();//Validate the S3 connection!
            logger.log(INFO, () -> STR."S3 buckets: \{buckets} ");

            var fileSystem = BlobStoreFileSystem.New(S3Connector.Caching(s3Client));

            return EmbeddedStorageFoundation.New()
                    .setConfiguration(StorageConfiguration.Builder()
                                        .setStorageFileProvider(StorageLiveFileProvider.Builder(fileSystem)
                                                                    .setDirectory(fileSystem.ensureDirectoryPath("storage"))
                                                                    .createFileProvider())
                                        .setChannelCountProvider(StorageChannelCountProvider.New(4))
                     .createConfiguration())
                     .onConnectionFoundation(BinaryHandlersJDK8::registerJDK8TypeHandlers)
                     .onConnectionFoundation(BinaryHandlersJDK17::registerJDK17TypeHandlers)
                     .createEmbeddedStorageManager();

generates the following exeception:

software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 400, Request ID: 17C0DACB248DA3B8, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8)
	at software.amazon.awssdk.protocols.xml@2.25.18/software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
	at software.amazon.awssdk.protocols.xml@2.25.18/software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
	at software.amazon.awssdk.protocols.xml@2.25.18/software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
	at software.amazon.awssdk.protocols.xml@2.25.18/software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)

Am I setting up the embedded store manager correctly? In my code, you will see that I list the available buckets from the Minio server, so I know the S3Client connection is working fine. What am I missing?

In the Minio WEB UI admin console I see these traces:

s3.ListBuckets    200 OK
s3.HeadObject     400 Bad Request

[hrstoyanov]

I tried the example from the S3 section of the documentation:

@Test
 void test1() {

        //Set up the S3 client as per AWS SDK 
        var s3Client = S3Client.builder()
                .region(Region.AWS_ISO_GLOBAL)
                .endpointOverride(URI.create("http://127.0.0.1:9000")) //Minio local API endpoint
                .credentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("yyy", "xxx")))
                .build();

        //Prove that it works!
        var buckets= s3Client.listBuckets().buckets(); 

        //Try to launch emnbedded stoarge manager 
        var blobFileSystem = BlobStoreFileSystem.New( S3Connector.Caching(s3Client));
        try(var embeddedStorage = EmbeddedStorage.start(blobFileSystem.ensureDirectoryPath("storage"))) {
           embeddedStorage.isActive();
        }
}

.. and got the same exception!

@fh-ms @zdenek-jonas - can you please take a look - it will be so nice of developers can test ES apps with Minio locally?

[hrstoyanov]

Spent some time debugging this - the issues is with this S3Connetor.internalDirectoryExists(BlobStorePath directory) which ends up doing this (line185):

       s3Client
                .headObject(HeadObjectRequest
                .builder()
                        .bucket("storage")
                        .key("/")
                .build());

In Minio, this results in 400 Bad head request, and there is no way ES can be told to stop ensuring things this way!

A better (and more compatible) way to ensure that "directory" exists is this:

        s3Client
                .putObject(PutObjectRequest
                .builder()
                   .bucket("storage")
                   .key("/")
                .build(), RequestBody.empty());

I verified it works with Minio, and I am sure it also works with AWS S3. Can you please fix ES 1.3.1?

[fh-ms]

#197