Data lineage tracking (aka CID store)

[pditommaso]

Tentative implementation for addressable data store (very basic POC so far).

Update on 1 Mar 2025 from #5787 by @jorgee

M1 Implementation of CID store for provenance

Changes:

• CID store is specified by workflow.data.store.location
• Workflow Hash is created based on the workflow and parameters description
• workflow, tasks and outputs metadata are stored in <cid.store.location>/.meta
• references to other cid metadata are cid://<workflow_hash|task_hash/output_target_path
• CID NIO Filesystem to access data based on CIS URLs
• nextflow cid command to log, show and get lineage from CID store metadata

Known Limitations:

• Outputs which are not published in absolutePaths or URLs which are not subfolders both the outputDir, we can not infer the relative output target path. They are not currently tracked in the CID store. We could create a hash for the parent directory of the URL or absolute path and use it as relative folder.

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	854a9de
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/68089db9fce4e30008a8fa2c

[pditommaso]

@jorgee apologies, can latest changes be made as PR against this branch? so it will be much simpler do understand what's new for me

[jorgee]

@jorgee apologies, can latest changes be made as PR against this branch? so it will be much simpler do understand what's new for me

I have reverted the changes in this branch and created a new one in PR #5787

[bentsherman]

After playing around with the lineage command, I am skeptical about how much we are overloading this lid pseudo-filesystem. I thought it was just a nice add-on that we could experiment with, but now I think it's just getting in the way.

Currently there are three main uses for lid paths:

1. lid://<hash>[#props]: returns a metadata record or sub-path. This has no practical utility in a Nextflow script, not even for workflow outputs. Now that #outputs is a list, I can't access an output by name (e.g. #outputs.samples), which means I can't use channel.fromPath() to access an LID output in the same way as a samplesheet. So the LID output is no longer a drop-in replacement for samplesheets.

On the command line, it would be simpler to just provide the hash and use jq:

# before
# oops, forgot to escape the #...
nextflow li describe lid://<hash>#params

# after
nextflow li describe <hash> | jq .params

In a web interface like the platform, you'll use a graphical interface to navigate this metadata, so the fragment syntax is not needed there.

2. lid:///?<name>=<value>&...: used by the find command to retrieve a collection of metadata records. This also has no utility in a Nextflow script, because it is unrelated to domain-specific data like #outputs. It is only used by the find command, so the URI syntax is just getting in the way:

# before
# oops, forgot to escape the & ...
nextflow li find lid:///?type=DataOutput&workflowRun=lid://2265a814fd1c205ecc5b629070d759e2

# after
nextflow li find type=DataOutput workflowRun=2265a814fd1c205ecc5b629070d759e2

3. lid://<hash>/<path>: returns a content-addressed file. This is the original use case and the only one that still makes sense as far as I can tell. I think this works perfectly both on the command line and in the Nextflow script/runtime.

Based on this analysis, I think we should ditch (1) and (2) entirely and use lid:// only to refer to files.

Maybe we could use the fragment to refer to a specific output, e.g. lid://<hash>#samples. That would at least restore the original use case of passing a workflow output as input to a downstream pipeline.

[jorgee]

8. Command lineage find 'type=DataOutput' work OK, but equivalent channel.fromPath('lid:///?type=DataOutput') seems not working 🔴

The ? is interpreted as a glob. But setting the option glob: false is also failing. It is not detecting is a global query. I will check it. However, I think the result in this case could be a bit undetermined. what do you expect to have in this case? the DataOutput descriptions or the real target paths?

[jorgee]

Based on previous comments, I have pushed some minor changes:

• WorkflowRun.resolvedConfig to config
• XxxOutputs -> XxxOutput, outputs to output and inputs to input
• Command describe to view and log to list
• TaskRun.scriptChckesum -> TaskRun.script including the resolved task script
• DataOutput -> FileOutput

[bentsherman]

TODOs from our discussion

This PR:

• rename log to list
• rename describe to view

Separate PR(s);

• document lineage command
• add channel.fromLineage() factory to query metadata
• remove URL query syntax
  • use key-value pairs on command line
  • use key-value pairs in channel factory, e.g. channel.fromLineage(foo: 'bar')
• simplify URL fragment syntax
  • use jq on command line
  • use json-path in channel factory (I think splitJson operator already implements json-path)
  • use URL fragment to retrieve workflow output by name e.g. channel.fromPath('lid://<hash>#samples')

[pditommaso]

Ok, I've moved the h2 stuff in the corresponding repo https://github.com/nextflow-io/nf-lineage-h2

[jorgee]

This PR:

• rename log to list
• rename describe to view

This is already changed in the current PR

[pditommaso]

Let's move ahead, then

[pditommaso]

[bentsherman]

@jorgee what is the rationale for the .meta subfolder? It looks like the only thing that is created in the lineage folder. Do you intend to store other things under lineage as well?

[jorgee]

@bentsherman, it was storing the output data and the metadata in the first implementation, but currently this sub-folder is not needed.

[bentsherman]

Okay, not urgent but something to consider in the final cleanup before 25.04