Move materialization history when moving assets from one code location to another

[sryza]

I moved a number of assets from one code location to another. Is it possible to move the asset materialization history as well?

[sryza]

As long as an asset has the same asset key, it's materialization history should be unaffected by what code location it's in. I.e. you shouldn't need to take any additional actions to migrate an asset's materialization history between code locations, if the asset's key doesn't change.

[brandonpeebles]

Hi @sryza — has there been any exploration around maintaining/migrating asset history when changing an asset key slightly? Our use case is that we're adopting Dagster in the middle of a larger data/infrastructure migration, so we pretty frequently come across situations where we're like "hmm we should maybe rename this or change the naming convention".

Dagster handles this pretty easily, but the downside is that we immediately break the asset history and have a lot of orphaned materializations we'd need to remove (or else it's cluttered). It also ends up breaking the history of assets on the Insights feature. It's not the end of the world when it happens since hopefully we'll get to a steady state soon, but until then it makes us more likely to stick with a less ideal naming convention for keys of fear of breaking asset history.

[sryza]

We have an issue to track this but alas don't have a straightforward way to do this right now: #9512.

Manually reporting asset materializations can also be helpful in some cases: https://docs.dagster.io/concepts/assets/external-assets#using-the-python-api.

[alexberndt]

Note that although asset history might remain intact during a migration, you may need to copy files depending on the IO manager each asset is using.

Illustrative Example

For example, we are using the GCSPickleIOManager where each code location writes to a different GCS location:

# old code location
defs = Definitions(
    ...
    resources={
        "io_manager": GCSPickleIOManager(
            gcs=gcs_resource,
            gcs_bucket="dagster-assets-gcs-bucket",
            gcs_prefix="old-code-location",
        )
   }
) 

# new code location
defs = Definitions(
    ...
    resources={
        "io_manager": GCSPickleIOManager(
            gcs=gcs_resource,
            gcs_bucket="dagster-assets-gcs-bucket",
            gcs_prefix="new-code-location",
        )
   }
) 

This requires users to copy gs://dagster-assets-gcs-bucket/old-code-location/<asset-key> to gs://dagster-assets-gcs-bucket/new-code-location/<asset-key> for each moved <asset-key>.