Skip to content

[Feature] raise AirflowFailException in order to avoid retrying dbt test tasks on failure #1537

@okayhooni

Description

@okayhooni

Description

Currently, all dbt tasks generated by Cosmos may inherit the same retries setting through the default_args parameter. However, unlike dbt run tasks, it seems unreasonable for dbt test tasks to be retried upon failure.

If a data test fails, rerunning it will likely result in the same failure, making retries unnecessary and incurring additional time and resource costs.

To optimize execution, how about introducing a parameter like no_retries_on_test_failure: bool = True? When enabled, this would ensure that dbt test tasks raise an AirflowFailException instead of an AirflowException, preventing them from being retried while keeping the existing retry behavior for other dbt tasks.

Use case/motivation

To optimize execution, dbt test tasks should raise an AirflowFailException instead of an AirflowException when they fail. This will prevent them from being retried while keeping other dbt tasks, such as dbt run, unaffected.

Related issues

No response

Are you willing to submit a PR?

  • Yes, I am willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    dbt:testPrimarily related to dbt test command or functionalityenhancementNew feature or requeststaleIssue has not had recent activity or appears to be solved. Stale issues will be automatically closedtriage-neededItems need to be reviewed / assigned to milestone

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions