Fix: airflow.task log level default to INFO (#649)

* set INFO as default for airflow.task logging

* Add documentation

* Add example to docs

* Using handler task to set log level, update comment

* Moving handler to comment

* update Changelog.md

* fixing typo

* Better documentation

Co-authored-by: Siegfried Weber <mail@siegfriedweber.net>

* More robust decision makeing

---------

Co-authored-by: Siegfried Weber <mail@siegfriedweber.net>

Author: Maleware

CHANGELOG.md

@@ -11,6 +11,7 @@
   - Use `--console-log-format` (or `CONSOLE_LOG_FORMAT`) to set the format to `plain` (default) or `json`.
 - Add support for airflow `2.10.5` ([#625]).
 - Add experimental support for airflow `3.0.1` ([#630]).
+- "airflow.task" logger defaults to log level 'INFO' instead of 'NOTSET' ([#649]).
 
 ### Changed
 
@@ -50,6 +51,7 @@
 [#630]: https://github.com/stackabletech/airflow-operator/pull/630
 [#636]: https://github.com/stackabletech/airflow-operator/pull/636
 [#645]: https://github.com/stackabletech/airflow-operator/pull/645
+[#649]: https://github.com/stackabletech/airflow-operator/pull/649
 
 ## [25.3.0] - 2025-03-21
 

docs/modules/airflow/pages/usage-guide/logging.adoc

@@ -3,6 +3,11 @@
 
 The logs can be forwarded to a Vector log aggregator by providing a discovery ConfigMap for the aggregator and by enabling the log agent:
 
+NOTE: The `task` handler is responsible for showing the task logs in the UI.
+Unfortunately, the log level of the `task` handler cannot be specified.
+To avoid that all logs are emitted to the UI, the log level of the `airflow.task` logger is set explicitly to `INFO`.
+You can change the log level as shown below.
+
 [source,yaml]
 ----
 spec:
@@ -26,6 +31,8 @@ spec:
             loggers:
               "airflow.processor":
                 level: INFO
+              "airflow.task":
+                level: DEBUG
   schedulers:
     config:
       logging:

rust/operator-binary/src/product_logging.rs

@@ -114,6 +114,11 @@ for logger_name, logger_config in LOGGING_CONFIG['loggers'].items():
     # otherwise DAGs cannot be loaded anymore.
     if logger_name != 'airflow.task':
         logger_config['propagate'] = True
+    # The default behavior of airflow is to enforce log level 'INFO' on tasks. (https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#logging-level)
+    # TODO: Make task handler log level configurable through CRDs with default 'INFO'.
+    # e.g. LOGGING_CONFIG['handlers']['task']['level'] = {{task_log_level}}
+    if 'handlers' in logger_config and 'task' in logger_config['handlers']:
+        logger_config['level'] = logging.INFO
 
 LOGGING_CONFIG.setdefault('formatters', {{}})
 LOGGING_CONFIG['formatters']['json'] = {{