Dagster logs go to stderr by default - how to get INFO logs in stdout and ERROR logs in stderr?

[vadimguma]

Problem Description

I've noticed that Dagster (deployed via Helm on Kubernetes) sends logs to stderr by default, including INFO level logs. This breaks our standard log aggregation setup where we expect:

• stdout = application output and informational logs
• stderr = errors and warnings for alerting

Current Behavior

When running any Dagster job with logging:

@asset
def example_asset(context):
    context.log.info("Processing data...")      # Goes to stderr ❌
    context.log.warning("Warning message")     # Goes to stderr ✅  
    context.log.error("Error occurred")        # Goes to stderr ✅
    return result

All these logs appear in stderr, I expect to follow standard Unix conventions.

What we've found

I have tried various approaches in values.yaml using pythonLogs configuration, but we're not sure if we're going in the right direction or if there's a simpler/better way.
I've seen discussions about this topic (like #24325 (reply in thread)) but couldn't find a clear production-ready solution.

Attempt 1: Multiple handlers

pythonLogs:
  pythonLogLevel: DEBUG
  managedPythonLoggers:
    - dagster
    - dagster.core
  dagsterHandlerConfig:
    handlers:
      json_stdout:
        class: logging.StreamHandler
        formatter: json
        level: DEBUG
        stream: ext://sys.stdout
      json_stderr:
        class: logging.StreamHandler
        formatter: json
        level: ERROR
        stream: ext://sys.stderr
    formatters:
      json:
        '()': dagster.JsonLogFormatter

Result: DEBUG logs go to stderr - should be in stdout

Attempt 2: Multiple handlers with assumed built-in filters

pythonLogs:
  pythonLogLevel: DEBUG
  managedPythonLoggers:
    - dagster
    - dagster.core
    - pipelines

  dagsterHandlerConfig:
    handlers:
      stdout_handler:
        class: logging.StreamHandler
        level: DEBUG
        stream: ext://sys.stdout
        formatter: json
        filters: [stdout_filter]

      stderr_handler:
        class: logging.StreamHandler
        level: ERROR
        stream: ext://sys.stderr
        formatter: json
        filters: [stderr_filter]

    filters:
      stdout_filter:
        '()': dagster.utils.log_filter.StdoutLogFilter
      stderr_filter:
        '()': dagster.utils.log_filter.StderrLogFilter

    formatters:
      json:
        '()': dagster.JsonLogFormatter

Result: ValueError: Unable to configure filter 'stderr_filter'
ModuleNotFoundError: No module named ‚dagster.utils.log_filter.StderrLogFilter'

Environment

• Dagster version: 1.9.9
• Deployment: Kubernetes via official Dagster Helm chart
• Container runtime: Docker

Questions

1. Is this the intended behavior in Dagster? Should all logs go to stderr by default?
2. Do built-in log filters exist for stdout/stderr separation? What are the correct class paths?
3. How to create custom logging filters for this use case? Where should they be placed in the project structure to be accessible via Helm configuration?
4. What's the recommended approach for production deployments that need stdout/stderr separation?