Skip to content

Commit c44e974

Browse files
adwk67maltesandernightkr
authored
feat: Airflow Listener integration (#604)
* mount listener volumes for relevant roles, type depending on listener class * add podrefs to write webserver endpoint to configmap * moved listener-class to webserver role * move config map creation out of role-loop * merge conflicts * regenerate nix * assign listener-class to all roles and consider all ports * fixed podref name for workers * added integration test * changelog * added docs * Update rust/operator-binary/src/crd/mod.rs Co-authored-by: Malte Sander <malte.sander.it@gmail.com> * Update rust/operator-binary/src/crd/mod.rs Co-authored-by: Malte Sander <malte.sander.it@gmail.com> * Update rust/operator-binary/src/crd/mod.rs Co-authored-by: Malte Sander <malte.sander.it@gmail.com> * Update rust/operator-binary/src/crd/mod.rs Co-authored-by: Malte Sander <malte.sander.it@gmail.com> * corrected callout comments and regenerate charts * use ephemeral listeners classes; fix test missing dimension * allow config map endpoints to be removed if cluster is stopped * review feedback * check for port lower-bound * reworked to only offer listener for webserver role * reworked docs * Update docs/modules/airflow/pages/usage-guide/listenerclass.adoc Co-authored-by: Malte Sander <malte.sander.it@gmail.com> * Update rust/operator-binary/src/crd/mod.rs Co-authored-by: Malte Sander <malte.sander.it@gmail.com> * added service check to test * correctly flatten WebserverConfig * use pvcs for externally reachable endpoints * use group listener for webservers * replace enum with string; use consistent webserver address for all listener classes * removed unused test account * Update docs/modules/airflow/pages/usage-guide/listenerclass.adoc Co-authored-by: Natalie Klestrup Röijezon <nat@nullable.se> * Update tests/templates/kuttl/opa/41_check-authorization.py Co-authored-by: Natalie Klestrup Röijezon <nat@nullable.se> * Update tests/templates/kuttl/oidc/login.py Co-authored-by: Natalie Klestrup Röijezon <nat@nullable.se> * review feedback: tests * review feedback: remove metrics port and re-work conditions * use custom listeners to stay independent of future changes to standard classes * make custom listener classes namespace-specific * changes as per decision 51 * fixed tests --------- Co-authored-by: Malte Sander <malte.sander.it@gmail.com> Co-authored-by: Natalie Klestrup Röijezon <nat@nullable.se>
1 parent f8a984d commit c44e974

File tree

25 files changed

+691
-218
lines changed

25 files changed

+691
-218
lines changed

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
### Added
66

7+
- Added listener support for Airflow ([#604]).
78
- Adds new telemetry CLI arguments and environment variables ([#613]).
89
- Use `--file-log-max-files` (or `FILE_LOG_MAX_FILES`) to limit the number of log files kept.
910
- Use `--file-log-rotation-period` (or `FILE_LOG_ROTATION_PERIOD`) to configure the frequency of rotation.
@@ -28,6 +29,7 @@
2829

2930
[#600]: https://github.com/stackabletech/airflow-operator/pull/600
3031
[#601]: https://github.com/stackabletech/airflow-operator/pull/601
32+
[#604]: https://github.com/stackabletech/airflow-operator/pull/604
3133
[#607]: https://github.com/stackabletech/airflow-operator/pull/607
3234
[#608]: https://github.com/stackabletech/airflow-operator/pull/608
3335
[#613]: https://github.com/stackabletech/airflow-operator/pull/613

‎deploy/helm/airflow-operator/crds/crds.yaml‎

Lines changed: 8 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -584,23 +584,6 @@ spec:
584584
default: false
585585
description: for internal use only - not for production use.
586586
type: boolean
587-
listenerClass:
588-
default: cluster-internal
589-
description: |-
590-
This field controls which type of Service the Operator creates for this AirflowCluster:
591-
592-
* cluster-internal: Use a ClusterIP service
593-
594-
* external-unstable: Use a NodePort service
595-
596-
* external-stable: Use a LoadBalancer service
597-
598-
This is a temporary solution with the goal to keep yaml manifests forward compatible. In the future, this setting will control which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) will be used to expose the service, and ListenerClass names will stay the same, allowing for a non-breaking change.
599-
enum:
600-
- cluster-internal
601-
- external-unstable
602-
- external-stable
603-
type: string
604587
loadExamples:
605588
default: false
606589
description: Whether to load example DAGs or not; defaults to false. The examples are used in the [getting started guide](https://docs.stackable.tech/home/nightly/airflow/getting_started/).
@@ -1338,6 +1321,10 @@ spec:
13381321
description: Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details.
13391322
nullable: true
13401323
type: string
1324+
listenerClass:
1325+
description: This field controls which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) is used to expose the webserver.
1326+
nullable: true
1327+
type: string
13411328
logging:
13421329
default:
13431330
containers: {}
@@ -1555,6 +1542,10 @@ spec:
15551542
description: Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details.
15561543
nullable: true
15571544
type: string
1545+
listenerClass:
1546+
description: This field controls which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) is used to expose the webserver.
1547+
nullable: true
1548+
type: string
15581549
logging:
15591550
default:
15601551
containers: {}

‎deploy/helm/airflow-operator/templates/roles.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,17 @@ rules:
8484
- customresourcedefinitions
8585
verbs:
8686
- get
87+
- apiGroups:
88+
- listeners.stackable.tech
89+
resources:
90+
- listeners
91+
verbs:
92+
- get
93+
- list
94+
- watch
95+
- patch
96+
- create
97+
- delete
8798
- apiGroups:
8899
- {{ include "operator.name" . }}.stackable.tech
89100
resources:

‎docs/modules/airflow/pages/usage-guide/listenerclass.adoc‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,19 @@
11
= Service exposition with ListenerClasses
22
:description: Configure Airflow service exposure with ListenerClasses: cluster-internal, external-unstable, or external-stable.
33

4-
Airflow offers a web UI and an API, both are exposed by the webserver process under the `webserver` role.
5-
The Operator deploys a service called `<name>-webserver` (where `<name>` is the name of the AirflowCluster) through which Airflow can be reached.
6-
7-
This service can have three different types: `cluster-internal`, `external-unstable` and `external-stable`.
8-
Read more about the types in the xref:concepts:service-exposition.adoc[service exposition] documentation at platform level.
9-
10-
This is how the listener class is configured:
4+
The operator deploys a xref:listener-operator:listener.adoc[Listener] for the Webserver pod.
5+
The listener defaults to only being accessible from within the Kubernetes cluster, but this can be changed by setting `.spec.webservers.config.listenerClass`:
116

127
[source,yaml]
138
----
149
spec:
15-
clusterConfig:
16-
listenerClass: cluster-internal # <1>
10+
webservers:
11+
config:
12+
listenerClass: external-unstable # <1>
13+
schedulers:
14+
...
15+
celeryExecutors:
16+
...
1717
----
18-
<1> The default `cluster-internal` setting.
18+
<1> Specify a ListenerClass, such as `external-stable`, `external-unstable`, or `cluster-internal` (the default setting is `cluster-internal`).
19+
This can be set only for the webservers role.

‎nix/sources.json‎

Lines changed: 3 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)