Skip to content

[Bug] Node_conversion on resource_type test is broken #1758

@anyapriya

Description

@anyapriya

Astronomer Cosmos Version

1.10.0

dbt-core version

1.9.4

Versions of dbt adapters

dbt-bigquery==1.9.1

LoadMode

DBT_LS

ExecutionMode

LOCAL

InvocationMode

DBT_RUNNER

airflow version

2.10.5+astro.2

Operating System

Debian GNU/Linux 12

If a you think it's an UI issue, what browsers are you seeing the problem on?

No response

Deployment

Astronomer

Deployment details

No response

What happened?

While working with cosmos, I found some behavior of node_converters that seems unexpected to me, and possibly a bug:

  1. When using AFTER_EACH, if you use a conversion function on Model or Source, it converts the ENTIRE task group, including both the model/source task AND the test task. I would expect only the model / source to be converted, not the tasks because the selection is specified by DbtResourceType and test is not being specified.
  2. When using a conversion function on resource type Test, it very much does not work as expected:
    a. When using any test_behavior (except NONE): It does NOT convert detached tests with multiple parents
    b. When using any test_behavior (except NONE): It creates separate tasks for non-detached tests (single parent) and converts those.
    c. When using AFTER_EACH: it does NOT convert the non-detached test tasks in the task groups (single parent tests). This is in addition to the behavior mentioned in 2.b. so now each test has its own task AND it's still covered in the task_group.test task.
    d When using AFTER_ALL: it does NOT convert the final test task. This is addition to the behavior mentioned in 2.b so now each test has its own task AND it's still covered in the project_test task.

Relevant log output

How to reproduce

  1. Set test_behavior to AFTER_EACH and add a node_converter for DbtResourceType("model")
  2. Set a node_coverter for DbtResourceType("test") and play around with the different test behaviors

Anything else :)?

I have a proposed fix to convert the task instead of the "node":

  • this is more consistent with node converter depending on the resource type, which seems to align better with individual tasks rather than task groups of two different resource types
  • this works better with different test_behaviors
  • this doesn't work if someone wants to convert entire task groups, but NOT the detached tests, but that seems like an edge-case

Note: this is a large change to how node_converters work and is almost definitely a breaking change.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Contact Details

No response

Metadata

Metadata

Assignees

Labels

area:renderingRelated to rendering, like Jinja, Airflow tasks, etcbugSomething isn't workingcustomer requestAn Astronomer customer made requested thisdbt:testPrimarily related to dbt test command or functionalityexecution:localRelated to Local execution environmentstaleIssue has not had recent activity or appears to be solved. Stale issues will be automatically closedtriage-neededItems need to be reviewed / assigned to milestone

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions