Description

Having all shared defaults in one file makes it challenging for users who manage multiple teams or projects with dag-factory. Also, #441 proposed to support both defaults.yml and defaults.yaml, while it's not decided yet how to handle multiple defaults files.

This feature request proposes a new way handling defaults, which supports multiple default files, allowing custom defaults to be used, and also allowing using default in a DAG YAML as the default of others. Specifically, a new reserved field __extends__ is added accepting multiple default config. The extended default config can have any custom file name.

# dag_1.yaml

__extends__:
    - default_1.yaml
    - project1/project_default.yaml

default:
  default_args:
    start_date: '2024-01-01'

Chained config extending can be allowed:

# default_1.yaml

__extends__:
    - base_default.yaml

Further more, the extended config can not only be default config with only default specified, but can also be an actual DAG config file, because only default will be applied during extending, and other fields will be ignored in the downstream config.

For the backward compatibility, technically, existing defaults.yml can still be supported. While it can also be confusing supporting both ways. I think it's discussable whether we want to keep a global default config. Personally, I think having both can cover more use cases and preferences, while we need to figure out how to support defaults with both yml and yaml suffix. I also think it's better to rename the reversed file name to _defaults.yml to avoid potential file name conflict.

Use case/motivation

For users who use dag-factory for multiple teams/projects, they have the need to setup different shared defaults per teams or projects. This is not possible with current defaults.yml solution, as it can only be one file.

Related issues

#441
#297

Are you willing to submit a PR?

• Yes, I am willing to submit a PR!