feat: Add support for Trino 470 (#705)

* test(integration): Add Trino 469, update latest version

* docs: Update trino version to 469

* feat: Add support for Trino 469

* test(unit): Update trino version used in tests

* chore: regenerate Cargo.nix

It was missed in #704

* chore: Update changelog

* wip: native s3 initial cut

* wip: more versiony stuff

* wip: parse Trino version as u16 version from String

* wip: remove comment and restore newline

* feat: Replace 469 with 470

* test: Trino now requires TLS for S3

* chore: Search and replace Trino productVersions

* chore: Rename trino_version to product_version for consistency

* refactor: Change product_version in the jvm mod to u16 for consistency

* chore: Improve note about native s3 support

* chore: Update changelog

* chore: Update changelog

* chore: Add warning about upcoming requirement for TLS requirement for S3 connections

Although it is technically possible to enable legacy S3 support in Trino 469, we have forced native s3 support since 469 so the warning is written that way.

* refactor: Use ensure! instead of if not

* chore: Bump stackable-operator for S3 region support

* feat: Set the S3 region unless it will be auto-discovered

* test(smoke): Override hive metastore memory limit due to OOMKilled

Ideally this should be done by the hive-operator.

* docs: Update supported versions

* Apply suggestions from code review

Improve error message

* Apply suggestions from code review

Emit warning only if tls is disabled.

* test: Update comments about minio and OpenShift testing

* test: Adjusted comments about minio and OpenShift testing

* chore: Bump stackable-operator and use the updated s3 region struct

* chore: Update changelog

* test: Add smoke_aws test to show how to use Trino with AWS S3 buckets

* chore: Make the operator fail for unsupported versions

This is not ideal as it prevents people trying whatever version they want, but we will defer this to a team decision later.

* chore: Undo resource overrides now that Hive resources have been increased

* Apply suggestions from code review

Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>

* chore: Update changelog

---------

Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de>

Author: NickLarsenNZ

CHANGELOG.md

@@ -11,12 +11,20 @@ All notable changes to this project will be documented in this file.
 - Run a `containerdebug` process in the background of each Trino container to collect debugging information ([#687]).
 - Support configuring JVM arguments ([#677]).
 - Aggregate emitted Kubernetes events on the CustomResources ([#677]).
+- Support for Trino 470 ([#705]).
 
 ### Changed
 
 - Increased the default temporary secret lifetime for coordinators from 1 day to 15 days.
   This is because Trino currently does not offer a HA setup for them, a restart kills all running queries ([#694]).
 - Default to OCI for image metadata and product image selection ([#695]).
+- Explicitly set `fs.native-s3.enabled=true` and `fs.hadoop.enabled=true` in applicable catalog config properties ([#705]).
+  - Trino 470 requires the native S3 implementation to be used.
+- BREAKING: Always set the S3 region ([#705]).
+  - Previously Trino used the hadoop s3 implementation which auto-detected the region from the
+    endpoint if it was not provided, falling back to `us-east-2`.
+  - The default is now `us-east-1`. Please set the region explicitly if you are using a different
+    one.
 
 ### Fixed
 
@@ -28,6 +36,7 @@ All notable changes to this project will be documented in this file.
 [#687]: https://github.com/stackabletech/trino-operator/pull/687
 [#694]: https://github.com/stackabletech/trino-operator/pull/694
 [#695]: https://github.com/stackabletech/trino-operator/pull/695
+[#705]: https://github.com/stackabletech/trino-operator/pull/705
 [#715]: https://github.com/stackabletech/trino-operator/pull/715
 [#717]: https://github.com/stackabletech/trino-operator/pull/717
 

Cargo.lock

@@ -23,10 +23,10 @@ serde = { version = "1.0", features = ["derive"] }
 serde_json = "1.0"
 serde_yaml = "0.9"
 snafu = "0.8"
-stackable-operator = { git = "https://github.com/stackabletech/operator-rs.git", tag = "stackable-operator-0.85.0" }
+stackable-operator = { git = "https://github.com/stackabletech/operator-rs.git", tag = "stackable-operator-0.87.2" }
 stackable-versioned = { git = "https://github.com/stackabletech/operator-rs.git", features = ["k8s"], tag = "stackable-versioned-0.5.0" }
 product-config = { git = "https://github.com/stackabletech/product-config.git", tag = "0.7.0" }
-strum = { version = "0.26", features = ["derive"] }
+strum = { version = "0.27", features = ["derive"] }
 tokio = { version = "1.40", features = ["full"] }
 tracing = "0.1"
 

Cargo.nix

@@ -1578,6 +1578,20 @@ spec:
                                   minimum: 0.0
                                   nullable: true
                                   type: integer
+                                region:
+                                  default:
+                                    name: us-east-1
+                                  description: |-
+                                    Bucket region used for signing headers (sigv4).
+
+                                    This defaults to `us-east-1` which is compatible with other implementations such as Minio.
+
+                                    WARNING: Some products use the Hadoop S3 implementation which falls back to us-east-2.
+                                  properties:
+                                    name:
+                                      default: us-east-1
+                                      type: string
+                                  type: object
                                 tls:
                                   description: Use a TLS connection. If not specified no TLS will be used.
                                   nullable: true
@@ -1795,6 +1809,20 @@ spec:
                                   minimum: 0.0
                                   nullable: true
                                   type: integer
+                                region:
+                                  default:
+                                    name: us-east-1
+                                  description: |-
+                                    Bucket region used for signing headers (sigv4).
+
+                                    This defaults to `us-east-1` which is compatible with other implementations such as Minio.
+
+                                    WARNING: Some products use the Hadoop S3 implementation which falls back to us-east-2.
+                                  properties:
+                                    name:
+                                      default: us-east-1
+                                      type: string
+                                  type: object
                                 tls:
                                   description: Use a TLS connection. If not specified no TLS will be used.
                                   nullable: true
@@ -1929,6 +1957,20 @@ spec:
                                   minimum: 0.0
                                   nullable: true
                                   type: integer
+                                region:
+                                  default:
+                                    name: us-east-1
+                                  description: |-
+                                    Bucket region used for signing headers (sigv4).
+
+                                    This defaults to `us-east-1` which is compatible with other implementations such as Minio.
+
+                                    WARNING: Some products use the Hadoop S3 implementation which falls back to us-east-2.
+                                  properties:
+                                    name:
+                                      default: us-east-1
+                                      type: string
+                                  type: object
                                 tls:
                                   description: Use a TLS connection. If not specified no TLS will be used.
                                   nullable: true

Cargo.toml

@@ -5,7 +5,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     catalogLabelSelector:
       matchLabels:

crate-hashes.json

@@ -5,7 +5,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     catalogLabelSelector:
       matchLabels:

deploy/helm/trino-operator/crds/crds.yaml

@@ -17,7 +17,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     catalogLabelSelector:
       matchLabels:

docs/modules/trino/examples/getting_started/code/trino.yaml

@@ -17,7 +17,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     tls:
       internalSecretClass: trino-internal-tls # <1>

docs/modules/trino/examples/getting_started/code/trino.yaml.j2

@@ -17,7 +17,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     tls:
       serverSecretClass: trino-tls # <1>

docs/modules/trino/examples/usage-guide/trino-insecure.yaml

@@ -17,7 +17,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     tls:
       serverSecretClass: trino-tls # <1>

docs/modules/trino/examples/usage-guide/trino-secure-internal-tls.yaml

@@ -73,7 +73,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     catalogLabelSelector:
       matchLabels:

docs/modules/trino/examples/usage-guide/trino-secure-tls-only.yaml

@@ -2,5 +2,6 @@
 // This is a separate file, since it is used by both the direct Trino documentation, and the overarching
 // Stackable Platform documentation.
 
+- 470
 - 455
 - 451 (LTS)

docs/modules/trino/examples/usage-guide/trino-secure-tls.yaml

@@ -5,7 +5,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     authentication:
       - authenticationClass: simple-trino-users

docs/modules/trino/pages/usage-guide/catalogs/index.adoc

@@ -9,7 +9,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     catalogLabelSelector:
       matchLabels:

docs/modules/trino/partials/supported-versions.adoc

@@ -5,7 +5,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     catalogLabelSelector: {}
   coordinators:

examples/simple-trino-cluster-authentication-opa-authorization-s3.yaml

@@ -7,7 +7,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     catalogLabelSelector:
       matchLabels:

examples/simple-trino-cluster-hive-ha-s3.yaml

@@ -5,7 +5,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     catalogLabelSelector:
       matchLabels:

examples/simple-trino-cluster-resource-limits.yaml

@@ -91,7 +91,7 @@ metadata:
   name: simple-trino
 spec:
   image:
-    productVersion: "455"
+    productVersion: "470"
   clusterConfig:
     authentication:
       - authenticationClass: simple-trino-oidc