Workflow composability with external workflow registries

[ksanderer]

What would you like to be added?

I'm building a TypeScript-based Serverless Workflow runtime for short-lived workflows and need guidance on implementing workflow composability with external workflow sources. Currently, the run.workflow task requires hardcoded namespace, name, and version properties, but I need to fetch workflows from external registries (S3, Git, etc.).

Use Case

Enable workflows to reference and execute other workflows stored in external registries, similar to how container orchestrators pull images from different registries.

Current Challenge: The spec's run.workflow task doesn't provide a clear path for external workflow resolution:

# Current spec - works for local/known workflows
run:
  workflow:
    namespace: payments
    name: validate-credit-card
    version: '1.2.0'

What I'm Trying to Achieve

I need workflows to fetch from configured registries. Here's what I'm working toward:

document:
  dsl: '1.0.0'
  namespace: payments
  name: order-processing
  version: '1.0.0'

use:
  workflowRegistries: # I know there is no such thing in the spec, just an illustration
    acmeProd:
      type: s3
      bucket: acme-workflows-prod
      region: us-east-1
      credentials: ${ $secrets.aws_credentials }
    
    acmeDev:
      type: git
      repository: https://github.com/acme/dev-workflows
      auth: ${ $secrets.github_token }

do:
  # Fetch from S3 registry
  - validatePayment:
      run:
        workflow:
          namespace: 'acmeProd://payments'  # {registry}://{namespace}
          name: 'validate-credit-card'
          version: '1.2.0'
          input: ${ .payment_info }
  
  # Fetch from Git registry
  - processOrder:
      run:
        workflow:
          namespace: 'acmeDev://payments'
          name: 'process-order'
          version: 'latest'
          input: ${ .order }

---

Questions

1. What would be the most spec-compliant way to achieve this?

   • URI-style namespace prefixes (registry://namespace, s3://{registry}/{namespace})?
   • Possible we can extend External resources to support workflows?
   • Extensions to the run.workflow task?

2. Am I missing something? Are there existing patterns in the spec that could solve this that I haven't discovered?

3. Implementation guidance? I would be glad for any guidance or insights on how to implement this feature.

Why This Matters

This would enable:

• Workflow reusability across teams and projects
• Version management of shared workflows
• Secure workflow distribution via private registries
• GitOps workflows where workflows are stored in version control

I'd greatly appreciate any guidance, pointers to existing solutions, or thoughts on how others have approached this challenge!

Community Notes

• Please vote by adding a 👍 reaction to the feature to help us prioritize.
• If you are interested to work on this feature, please leave a comment.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.