Modify Running Workflow/Retry with Revisions

[jrussell9000]

I feel like I'm missing something very obvious here but...

Given that failed workflows can be restarted from individual nodes, is it possible to modify the templates for those nodes without restarting the entire workflow? For example, if a node step in my workflow fails because I'm a terrible coder who makes lots of mistakes, is it possible to modify the code (i.e., template) for the failed node and then try to restart it (versus the entire workflow? I don't see a way to do this in the UI, and if I simply update the a workflow template using the CLI, it doesn't seem to change the failed workflow if I try to restart it. Thoughts???

[static-moonlight]

If you define everything within the workflow itself, this is the behavior I would expect. Then you would need to resubmit it. However, there is a way to kind of support what you want.

So, instead of doing this:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: my-workflow
  namespace: my-namespace
spec:
  entrypoint: my-entrypoint
  [...]
  templates:
    - name: my-entrypoint
      dag:
        tasks:
          - name: do-something
            template: do-something
    - name: do-something
      script:
        image: docker.io/busybox:latest
        command: [sh]
        source: echo "whatever"

... you can define the workflow steps in a separate template, and reference them like this:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: my-templates
  namespace: my-namespace
spec:
  templates:
    - name: do-something
      script:
        image: docker.io/busybox:latest
        command: [sh]
        source: echo "whatever"

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: my-workflow
  namespace: my-namespace
spec:
  entrypoint: my-entrypoint
  [...]
  templates:
    - name: my-entrypoint
      dag:
        tasks:
          - name: do-something
            templateRef:
              name: my-templates
              template: do-something

my-templates acts as a library. If you change whatever do-something is doing and redeploy my-templates, I found that it has immediate effect to existing workflows like my-workflow. Then a retry of a failed do-something should use the updated template definition. This is limited to the internal details of do-something though. If you change its interface, like adding another input parameter, then strange things might happen and you need to resubmit the entire workflow anyway.

[jrussell9000]

Thanks @static-moonlight! This is in fact how I've structured my workflow template, as a dag invoking a series of templateRefs. Updating the master template or any of the member templates seems a bit inconsistent at times - though this might be user error. I'm not sure when I need to update the master template - for example, if I change one of the sub-templates do I need to update the master? What happens if I update a template while it's being used in a running workflow? Still figuring these things out....

[static-moonlight]

What happens if I update a template while it's being used in a running workflow?

As I said: upgrades are effective immediately. I never tested it to that extent, but I would assume that am already-running workflow step (pod) would NOT be effected by reploying the workflow "library", because, the pod was already created by using the referenced template. However, every newly created workflow step should use the upgraded template. So you have to be careful that "master" and "member" can work together.

By now I'm using a shared library of several workflow templates, which are used by multiple workflows throughout multiple namespaces. The upside: it reduced redundancies and I can fix the same problem by modifying just one template. The downside: If I fuck up, I break all workflows at once.

I can't give you a blueprint how to deal with it in detail. It kind of depends on your infrastructure and criticality. I tend to do small incremental upgrades to my library templates and workflow templates and maintain temporary backward compatibility and put it though a testing environment first, to look for problems. I usually check my monitoring metrics as well, looking for changes. If I'm confident about a change, I deploy it in production. Basically: small baby steps: change something, deploy to testing, check, deploy to production. It usually means that I have to break down larger changes into smaller ones, each with it's own deployment schedule.

Not sure how helpful this is ...

[jrussell9000]

Extremely helpful, thank you! At this point, I'm still in the development phase of a new pipeline and tend to be making major changes (versus incremental upgrades) though they're becoming smaller as I gain familiarity with the quirks of Argo Workflows. In the context of this discussion, I'm mostly trying to make my development process more efficient. Thanks again!!