Roadmap for nf-lamin

[Zethson]

https://www.notion.so/laminlabs/Build-nf-lamin-39c62bcd97e0408da0522b13f98ff2c0

[rcannood]

Overall plan:

The nf-lamin plugin needs to track a Nextflow run and upload information about the run to Lamin Hub. We need to use the Lamin API for this and not the lamindb Python package, or else the plugin won't work on platforms like Seqera Cloud.

To do so, the plugin needs to implement Nextflow's TraceObserver and interact with the Lamin API to:

• Look up or create a Transform
• Start a Run
• Track input files as Artifacts
• Track output files as Artifacts
• Upload additional information (e.g. environment, trace, reports...)

Timeline:

• POC experiments (nf-lamin@0.0.1)
  • Which information we can extract from the Nextflow Java API while a workflow is running (Beginning of ✨ nf-lamin plugin v0.0.1 #50)
  • This led me to discover that it's currently not possible to track which external files (e.g. S3, GS, HTTP, etc.) were staged as part of the run. I created an issue at Detect when files are being staged nextflow-io/nextflow#5905 and created two alternative implementations Extend observer with onFileStage nextflow-io/nextflow#5907 and Include remote path in FileHolder nextflow-io/nextflow#5911 which would solve this issue
  • I made proposals for to which Python calls this would translate into (Later commits in ✨ nf-lamin plugin v0.0.1 #50)
  • Generate a lamin-api-client and upload to Maven Central (https://central.sonatype.com/artifact/ai.lamin/lamin-api-client)
  • Create a nf-lamin plugin package that reads in the config, tests the connection to the lamindb instance, and prints out some debug messages about what actions it could be taking
• MVP:
  • Create Transform and Run onFlowCreate, finalise Run onFlowComplete (✨ Track Nextflow runs by creating Transforms and Runs #63)
  • Submit to nf-lamin@0.1.0 to nextflow-io/plugins
• Additional features:
  • Track input files (contigent on Detect when files are being staged nextflow-io/nextflow#5905)
  • Track output files
  • Create and upload reports
  • Create and upload environment information
  • Support for retrieving / querying artifacts from lamin, e.g. lamin://laminlabs/lamindata/tCUkRcaEjTjhtozp0000
  • Create and upload a DAG of how input files, intermediate files and output files are linked based on the Nextflow processes in the run.