How to run multiple partitions of an asset with custom configuration

[bolinzzz]

Hello team,

I'm looking for a way to materialize multiple partitions of an asset using a custom run configuration, without the need to open the launchpad and handle each partition individually.

Is defining a job the only approach to achieve this? Specifically, by targeting the partitioned asset and applying a custom configuration through that job? Here's an example of what I currently have:

from dagster import Definitions, asset, Config, define_asset_job

class MyAssetConfig(Config):
    person_name: str

@asset(partitions_def=DailyPartitionsDefinition(start_date="2023-10-01"))
def asset1(ocnfig: MyAssetConfig):
    return f"hell {config.person_name}"


asset1_job = define_asset_job(name="asset1_job", selection="asset1", config=MyAssetConfig(person_name="Lebron James"))

defs = Definitions(
    assets=[asset1, asset2],
    jobs=[all_assets_job, asset1_job],
)

Any suggestions on how to improve or simplify this are welcome.

[gianfrancodemarco]

Is this possible? Any plans to make the configuration applicable to all selected partitions?

[jamiedemaria]

Hi @gianfrancodemarco - this currently isn't possible. We know this is a missing feature, but don't have a specific timeline of when it would be implemented

[gianfrancodemarco]

Understood. Let me know if help is needed, I could try to contribute.

[hnykda]

Darn, this is currently quite blocking me. Processing each partition independently is extremely costly for me (I have to iterate over a huge file only for a few matches), and having an option to process stuff in bulk is crucial (I have thousands of partitions). I can do it from an asset launchpad, but not via job. So right now I have to go via asset-by-asset manually...