Add a dry run feature

[eric-czech]

Is your feature request related to a problem? Please describe.

It would be helpful to know what steps of a DAG would execute when caching is enabled.

Describe the solution you'd like

For an example like this, being able to visualize DAGs or query cache results prior to execution through any Driver function would be great.

I imagine I'm far from the first person to think about this, but I'm not seeing any prior discussion about it. Are there any foundational design issues that would make this difficult to implement?

[elijahbenizzy]

@eric-czech -- nothing that would make this impossible! It's quite doable. In fact, we have https://hamilton.dagworks.io/en/latest/reference/drivers/Driver/#hamilton.driver.Driver.visualize_execution which should get you most of the way there!

[zilto]

Actually, it is likely to be impossible to view "what nodes will be executed before launching execution when using caching".

In short, the caching feature needs to "walk the graph". You start at input nodes and follow the execution path until the final nodes. At each step, caching checks the code_version of the current node and the data_version of its dependencies.

Before executing the DAG, you don't know the data_version of all nodes (if you did, you wouldn't need to execute the DAG at all!). Before executing the DAG, you're able to make 3 types of claims:

• "Will be retrieved from cache"
• "Might be retrieved from cache"
• "Will need to be executed / will not be retrieved from cache"

Visually:

The thing is, whenever you hit a "maybe will be retrieved from cache", everything downstream of it will necessarily be a "maybe" too. Since what changes between runs are the input nodes (i.e., top-level nodes), you will find that you almost always have visualizations filled with maybes. Though, the viz could be useful is certain specific topologies

[eric-czech]

you don't know the data_version of all nodes

That makes sense, thanks @zilto. What is assumed to be capable of changing data if not code though? Put differently, couldn't you provide a dry-run feature that works under the assumption that all data versions will be the same if all upstream code versions have not changed?

[zilto]

How caching works

Before execution, you are certain of your input data_version and the code_version of all nodes. What you don't know is the data_version of all non-input nodes. To know the data_version, you need to apply the caching logic by traversing the DAG.

In DAG A -> B -> C -> D, to know if D needs to be executed, you need to resolve C.data_version (you already know C.code_version). To know C.data_version, you need to resolve B.data_version. The key point being, you need to resolve things in order.

(The best summary I can provide is the picture in the previous reply)

In short

Put differently, couldn't you provide a dry-run feature that works under the assumption that all data versions will be the same if all upstream code versions have not changed?

As I understand, the goal of the visualization is to tell you "how caching will behave before running the DAG". As explained in a previous reply, it is possible, but almost all your nodes will be "maybe execute".

If this visualization makes different assumptions than the caching algorithm, it will display something that diverges from the actual caching behavior. The output will be probably be more confusing than useful.

Assumptions

dry-run

under the assumption that all data versions will be the same if all upstream code versions have not changed?

If you already know the code_version and data_version of A, B, C, D, there's no need to execute the DAG. A data_version is a hash of a specific Python object. If you know D.data_version = XYZ123, then just cache.get(D.data_version) and retrieve the value.

If you do know that B = 4, then you should use Driver.execute(["D"], overrides={"B": 4}). Caching will properly handle this knowledge. If you were to implement the visualization I explained in a previous reply, you would see less "maybe execute", but still potentially a lot of them.

Idempotency

What is assumed to be capable of changing data if not code though?

The same code can produce different results (non-idempotency): it has a stochastic component (random.choice, ML training), it does I/O and reads values from an external file, you use a different version of an external Python library (hypothetically, numpy.float defaulting to 32 precision in numpy==2.0.0 vs. 64 precision in numpy==3.0.0).

By default Hamilton assumes idempotency of nodes and it's the user's responsibility to tag non-idempotent nodes.