How to get successful, missing, and failed partition counts for an asset programmatically?

[chrishiste]

I need to programmatically retrieve the counts of successful, missing, and failed partitions for an asset in Dagster.

Is there a GraphQL query or context method available to achieve this? Any examples or pointers would be helpful. Thank you!

[chrishiste]

Hi @garethbrickman, sorry to ping but this has been open for a while. Do you have any idea how we could achieve this?

[garethbrickman]

There is this get_status_by_partition method available for the DagsterInstance.

Background about it is in this issue: #14988

[chrishiste]

@garethbrickman I tried something similar before opening the issue but it didn't work.

get_status_by_partition second parameter is asking for a list of partitions but I want all of them in my case.

Is there a way to get all partitions?

The code below produces the following error dagster._check.functions.CheckError: Failure condition: The instance is not available to load partitions. You may be seeing this error when using dynamic partitions with a version of dagster-webserver or dagster-cloud that is older than 1.1.18. The other possibility is that an internal framework error where a dynamic partitions store was not properly threaded down a call stack.

statuses = context.instance.get_status_by_partition(
        AssetKey(["my_asset"]),
        daily_partitions_def.get_partition_keys(),
        daily_partitions_def,
    )

My partitions definition is similar to this

daily_partitions_def = MultiPartitionsDefinition(
    {
        "time_window": DailyPartitionsDefinition(
            start_date=start_date,
            timezone="UTC",
        ),
        "table_name": dynamic_table_partitions,
    }
)

Maybe this is broken for dynamic multi partitions?

[lukasz-matter]

Look at this: #18344 (comment)

There's a selector for partitions - unmaterializedPartitions, failedPartitions, materializedPartitions or materializinPartitions.

[chrishiste]

Last time I checked those didn't work with multi partitions. I built my own auto materialization logic so this isn't an issue anymore but I had to write a lot of code to make it work. I would still like an API or GraphQL example query to get those directly.