Run dbt models independently for a @dbt_assets using AutomationCondition.eager() and default_automation_condition_sensor

[julietteponnet]

In my current workflow, we are using Dagster and DBT integration as follow:

Dagster assets are set up with an AutomationCondition.eager() policy, which automatically materialises upstream assets whenever any dependency is updated (see official doc) and because my dbt assets have an automation condition in your code location, a sensor with the name default_automation_condition_sensor is created for automatically. Here is how it looks:

class CustomDagsterDbtTranslator(DagsterDbtTranslator):
    def get_automation_condition(
        self, dbt_resource_props: Mapping[str, Any]
    ) -> Optional[AutomationCondition]:
        return AutomationCondition.eager()

    @classmethod
    def get_group_name(cls, dbt_resource_props: Mapping[str, Any]) -> Optional[str]:
        return dbt_resource_props["config"]["schema"]

As first step, a custom ingestion sensor triggers extraction of our data into a raw asset, and then with the eager automation condition being met, default_automation_condition_sensor is used to materialises downstream asset (dependencies are defined in the dbt manifest). The issue is that AutomationCondition.eager() kicks off a large dbt job that contains many assets in a single run and I am facing two major issues:

• Whole-Pipeline Re-execution on Failure: If one model in the batch fails, the default behaviour is to treat the entire run as failed, and Dagster will retry by re-running all models in that run. This is inefficient – for example, 20+ models each taking ~3 minutes mean ~60 minutes per run, so a failure at the end triggers another full 60-minute run, wasting time and compute. Related issue: Run independent dbt selects for a @dbt_assets declaration with many models #23665 but their solution is not scalable for me and not using default_automation_condition_sensor
• Lack of Per-Asset Visibility: Because multiple assets materialise in the same run, a single failure marks the whole run as failed. This hampers quick debugging and oversight from the overview panel in the UI. Related issue : Viewing the DBT process as a waterfall #24217

Do you have any suggestions to solve those issues? Any workaround maybe? We have more than hundred of models so creating custom sensors with custom conditions per asset is not feasible