[Question] Support for custom headers when integrating AutoMQ with Apache Polaris

[namnt2307]

Hi everyone
I’m deploying an AutoMQ cluster with table-topic enabled and using Apache Polaris as the REST Iceberg catalog.

According to the Polaris production configuration guide:

If a request contains the specified header, Polaris will use the realm specified in the header. If the realm is not in the list of allowed realms, Polaris will return a 404 Not Found response.

If a request does not contain the specified header, however, by default Polaris will use the first realm in the list as the default realm. In the above example, POLARIS is the default realm and would be used if the Polaris-Realm header is not present in the request.

I know i can set polaris.realm-context.require-header=false to get this work.

But is there currently a way for AutoMQ to set custom headers (such as Polaris-Realm) on catalog requests, or is this not supported yet?

Thanks!

[Gezi-lzq]

Based on my review of the usage described in https://polaris.apache.org/in-dev/unreleased/getting-started/using-polaris/, my understanding is that the discussion revolves around configurations like:

spark.sql.catalog.quickstart_catalog.header.X-Iceberg-Access-Delegation=vended-credentials
spark.sql.catalog.quickstart_catalog.credential=${USER_CLIENT_ID}:${USER_CLIENT_SECRET}

and how to configure these in AutoMQ.

In this case, the catalog configuration prefix in AutoMQ is automq.table.topic.catalog.*, so the equivalent configuration should be:

automq.table.topic.catalog.header=vended-credentials
automq.table.topic.catalog.credential=${USER_CLIENT_ID}:${USER_CLIENT_SECRET}

Based on the current configuration approach, I believe this should also be effective in Polaris. If there are any issues, please feel free to provide feedback.

[namnt2307]

Thanks
I will try that

[aaron-seq]

@namnt2307 @Gezi-lzq

I've investigated the codebase and found that custom header support is already implemented in AutoMQ, please confirm the same

Looking at the test file CatalogFactoryTest.java, AutoMQ already supports custom headers for REST catalogs using the pattern:

automq.table.topic.catalog.header.{header-name}={header-value}

For Apache Polaris specifically, you can configure the Polaris-Realm header like this:

automq.table.topic.catalog.type=rest
automq.table.topic.catalog.uri=http://your-polaris-server:8181
automq.table.topic.catalog.header.Polaris-Realm=your-realm-name

This implementation has been working since it was added and is already tested in the codebase. You can see the test example at:
https://github.com/AutoMQ/automq/blob/main/core/src/test/java/kafka/automq/table/CatalogFactoryTest.java#L58

The configuration gets passed through to the underlying Apache Iceberg REST catalog, which handles the HTTP header injection.

Additional examples:

• For authentication headers: automq.table.topic.catalog.header.Authorization=Bearer your-token
• For custom tenant headers: automq.table.topic.catalog.header.X-Tenant-ID=tenant123
• For any custom header: automq.table.topic.catalog.header.{HeaderName}={HeaderValue}

This should resolve your Apache Polaris integration issue! Let me know if you need any clarification or if you encounter any issues testing this configuration.