Skip to content

Add version, kind, and spec to lineage schema #6075

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jul 9, 2025

Conversation

jorgee
Copy link
Contributor

@jorgee jorgee commented May 14, 2025

This PR refactors the Lineage data encoder to serialize lineage records using the following structure:

{
  "version": <lineage version>,
  "kind": <lineage record class>,
  "spec": <record description>
}

For instance a FileOutput type is serialized as:

{
  "version": "lineage/v1beta1",
  "kind": "FileOutput",
  "spec": {
    "path": "/home/jorgee/nextflow_tests/provenance-test/results/samples.json",
    "checksum": {
      "value": "30a926dd315d4f4b9d6389803a1324ff",
      "algorithm": "nextflow",
      "mode": "standard"
    },
    "source": "lid://31e092e42c71b9545a9c5fa91689340c",
    "workflowRun": "lid://31e092e42c71b9545a9c5fa91689340c",
    "taskRun": null,
    "size": 201,
    "createdAt": "2025-05-14T10:25:51.563182873+02:00",
    "modifiedAt": "2025-05-14T10:25:51.563182873+02:00",
    "labels": [
      "foo"
    ]
  }
}

jorgee added 2 commits May 14, 2025 09:53
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Copy link

netlify bot commented May 14, 2025

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit 7584275
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/686d69e148599b00085dc74e

@pditommaso
Copy link
Member

This looks nice, likely we needed to update the version to v1beta2 if we go ahead with this. Also in a ideal world, it should be possible to keep backward compatibility with previous spec version, but I can understand that cannot be that simple

Co-authored-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Jorge Ejarque <jorgee@users.noreply.github.com>
@jorgee
Copy link
Contributor Author

jorgee commented May 15, 2025

For these PR, as there aren't changes in the model I think there is no need to update the version. I can make able to read both with spec and without with a small change. However, with changes such as in #6069 a mechanism for version compatibility is required.

@bentsherman
Copy link
Member

It might be enough to simply ignore records that have an incompatible version and warn about them so that the user is aware. At least while the version is in beta. Maybe in the future we could add backwards compatibility, e.g. v1 / v2 data models, but hopefully we can iron out any changes in the beta period

@jorgee jorgee changed the title Refactor lineage JSON serilization with a root element including version, kind and spec properties Refactor lineage JSON serialization with a root element including version, kind and spec properties May 20, 2025
@pditommaso pditommaso force-pushed the master branch 3 times, most recently from b4b321e to 069653d Compare June 4, 2025 18:54
@bentsherman bentsherman changed the title Refactor lineage JSON serialization with a root element including version, kind and spec properties Add version, kind, and spec to lineage schema Jul 7, 2025
@bentsherman bentsherman requested a review from pditommaso July 7, 2025 13:49
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso
Copy link
Member

pditommaso commented Jul 8, 2025

I've added the backward compatibility with the previous lineage format and test. Please review and merge if OK.

Claude prompt for the posterity:

This branch changes the JSON structure of the lineage model, see the changes of LinTypeAdapterFactory comparing with the master branch. Howevever I'd to keep the read compatibility with the previous version. Can you make a patch for it? When done write a unit test to validate it.

…pterFactory.java [ci skip]

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@jorgee
Copy link
Contributor Author

jorgee commented Jul 9, 2025

Changes are fine with me.

@pditommaso pditommaso merged commit 6328613 into master Jul 9, 2025
5 checks passed
@pditommaso pditommaso deleted the lineage-spec-json-model branch July 9, 2025 06:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants