-
Notifications
You must be signed in to change notification settings - Fork 240
Open
Labels
area:renderingRelated to rendering, like Jinja, Airflow tasks, etcRelated to rendering, like Jinja, Airflow tasks, etcbugSomething isn't workingSomething isn't workingcustomer requestAn Astronomer customer made requested thisAn Astronomer customer made requested thisdbt:testPrimarily related to dbt test command or functionalityPrimarily related to dbt test command or functionalityexecution:localRelated to Local execution environmentRelated to Local execution environmentstaleIssue has not had recent activity or appears to be solved. Stale issues will be automatically closedIssue has not had recent activity or appears to be solved. Stale issues will be automatically closedtriage-neededItems need to be reviewed / assigned to milestoneItems need to be reviewed / assigned to milestone
Description
Astronomer Cosmos Version
1.10.0
dbt-core version
1.9.4
Versions of dbt adapters
dbt-bigquery==1.9.1
LoadMode
DBT_LS
ExecutionMode
LOCAL
InvocationMode
DBT_RUNNER
airflow version
2.10.5+astro.2
Operating System
Debian GNU/Linux 12
If a you think it's an UI issue, what browsers are you seeing the problem on?
No response
Deployment
Astronomer
Deployment details
No response
What happened?
While working with cosmos, I found some behavior of node_converters that seems unexpected to me, and possibly a bug:
- When using AFTER_EACH, if you use a conversion function on Model or Source, it converts the ENTIRE task group, including both the model/source task AND the test task. I would expect only the model / source to be converted, not the tasks because the selection is specified by DbtResourceType and test is not being specified.
- When using a conversion function on resource type Test, it very much does not work as expected:
a. When using any test_behavior (except NONE): It does NOT convert detached tests with multiple parents
b. When using any test_behavior (except NONE): It creates separate tasks for non-detached tests (single parent) and converts those.
c. When using AFTER_EACH: it does NOT convert the non-detached test tasks in the task groups (single parent tests). This is in addition to the behavior mentioned in 2.b. so now each test has its own task AND it's still covered in the task_group.test task.
d When using AFTER_ALL: it does NOT convert the final test task. This is addition to the behavior mentioned in 2.b so now each test has its own task AND it's still covered in the project_test task.
Relevant log output
How to reproduce
- Set test_behavior to AFTER_EACH and add a node_converter for DbtResourceType("model")
- Set a node_coverter for DbtResourceType("test") and play around with the different test behaviors
Anything else :)?
I have a proposed fix to convert the task instead of the "node":
- this is more consistent with node converter depending on the resource type, which seems to align better with individual tasks rather than task groups of two different resource types
- this works better with different test_behaviors
- this doesn't work if someone wants to convert entire task groups, but NOT the detached tests, but that seems like an edge-case
Note: this is a large change to how node_converters work and is almost definitely a breaking change.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Contact Details
No response
Metadata
Metadata
Assignees
Labels
area:renderingRelated to rendering, like Jinja, Airflow tasks, etcRelated to rendering, like Jinja, Airflow tasks, etcbugSomething isn't workingSomething isn't workingcustomer requestAn Astronomer customer made requested thisAn Astronomer customer made requested thisdbt:testPrimarily related to dbt test command or functionalityPrimarily related to dbt test command or functionalityexecution:localRelated to Local execution environmentRelated to Local execution environmentstaleIssue has not had recent activity or appears to be solved. Stale issues will be automatically closedIssue has not had recent activity or appears to be solved. Stale issues will be automatically closedtriage-neededItems need to be reviewed / assigned to milestoneItems need to be reviewed / assigned to milestone