Failure using a loop aggregation as workflow value

[EricBoix]

Technical context

• minikube version: v1.19.0 (running on OSX 10.14.6 a.k.a. Mojave)
• argo version:
  • v3.2.2+8897fff.dirty
  • GitCommit: 8897fff
  • GoVersion: go1.17.2

The difficulty

Consider the following workflow (the corresponding workflow file is attached to this discussion, refer below to the "reproducing steps" section) where the comments document the encountered problem

# Workflow saved as/in sprig-usage-issue.yml file 
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: parameters-
spec:
  entrypoint: main
  volumes:
  - name: workdir
    hostPath:
      path: /data/host
  arguments:
    parameters:
    - name: persistedVolume   
      value: /data/host
    - name: json
      # Seems to be a valid JSON according to some online validators e.g.
      # https://jsonlint.com/ or https://jsononline.net/json-validator
      value: '["one","one\ntwo","one\ntwo\nthree"]'

  templates:
  - name: main
    steps:
    - - name: print-sprig-of-simple-lists
        template: whalesay
        arguments:
          parameters:
          - name: message
            # The whale will duly say
            #            _______ 
            #           < three >
            #            ------- 
            value: "{{=sprig.last( ['one', 'two', 'three'] ) }}" 

    - - name: loop-write-some-result
        template: write-some-result
        arguments:
          parameters:
          - name: some-file-content
            value: "{{item}}"
        withItems: 
          - "one"
          - "two"
          - "three"

    - - name: print-loop-result
        template: whalesay
        arguments:
          parameters:
          - name: message
            # This whale will behave and as expected say
            #         _________________________________________ 
            #        / ["two\none","two\none","two\none\nthree \
            #        \ "]                                      /
            #         ----------------------------------------- 
            value: "{{steps.loop-write-some-result.outputs.parameters.results-values}}"

    - - name: print-sprig-of-loop-result
        # If you try to print the following value with whalesay, you might get
        # the follwing error message
        #    This shouldn't happen at /usr/share/perl/5.18/Text/Wrap.pm line 84.
        # We thus resolve to printing through other means 
        template: print-python
        arguments:
          parameters:
          - name: message
            # But now the (python) printed message goes
            #            {{=sprig.last(steps.loop-write-some-result.outputs.parameters.results-values) }}
            # that is the exact input string without the substitution that
            # expr is supposed (?) to realize, prior to interpreting the result.
            # Note that if this expr substitution is also inhibited when using
            # whalesay, this might explain why whalesay crashes (on that 
            # obscure perl related error): the whalesay print gets confused by
            # its argument that is a mis-formed perl string...
            value: "{{=sprig.last(steps.loop-write-some-result.outputs.parameters.results-values) }}"
            
  - name: write-some-result
    inputs:
      parameters:
      - name: some-file-content
    script:
      image: python:alpine3.6
      command: [python]
      source: |
        import os
        output_filename=os.path.join("{{workflow.parameters.persistedVolume}}",
                                     "results.txt")
        with open(output_filename, "a+") as f:
          f.write("{{inputs.parameters.some-file-content}}" + "\n")
      volumeMounts:
       - name: workdir
         mountPath: /data/host
    outputs:
      parameters:
      - name: results-values
        valueFrom:
          path: "{{workflow.parameters.persistedVolume}}/results.txt"

  - name: whalesay
    inputs:
      parameters:
      - name: message
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["{{inputs.parameters.message}}"]

  # When whalesay fails/crashes we need a plan B print
  - name: print-python
    inputs:
      parameters:
      - name: message
    script:
      image: python:alpine3.6
      command: [python]
      source: |
        print("{{inputs.parameters.message}}")

When triggering that workflow with e.g.

argo submit --watch --log sprig-usage-issue.yml

one gets

STEP                                    TEMPLATE           PODNAME                      DURATION  MESSAGE
 ✔ parameters-lsc9f                     main                                                        
 ├───✔ print-sprig-of-simple-lists      whalesay           parameters-lsc9f-3339508213  6s          
 ├─┬─✔ loop-write-some-result(0:one)    write-some-result  parameters-lsc9f-156400594   11s         
 │ ├─✔ loop-write-some-result(1:two)    write-some-result  parameters-lsc9f-3057607737  10s         
 │ └─✔ loop-write-some-result(2:three)  write-some-result  parameters-lsc9f-4178164564  8s          
 ├───✔ print-loop-result                whalesay           parameters-lsc9f-4287932199  6s          
 └───✔ print-sprig-of-loop-result       print-python       parameters-lsc9f-1274151296  6s     

and the following output

parameters-lsc9f-3339508213:  _______ 
parameters-lsc9f-3339508213: < three >
parameters-lsc9f-3339508213:  ------- 
parameters-lsc9f-3339508213:     \
parameters-lsc9f-3339508213:      \
parameters-lsc9f-3339508213:       \     
parameters-lsc9f-3339508213:                     ##        .            
parameters-lsc9f-3339508213:               ## ## ##       ==            
parameters-lsc9f-3339508213:            ## ## ## ##      ===            
parameters-lsc9f-3339508213:        /""""""""""""""""___/ ===        
parameters-lsc9f-3339508213:   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
parameters-lsc9f-3339508213:        \______ o          __/            
parameters-lsc9f-3339508213:         \    \        __/             
parameters-lsc9f-3339508213:           \____\______/   
parameters-lsc9f-4287932199:  _________________________________________ 
parameters-lsc9f-4287932199: / ["three\ntwo\none","three\ntwo","three" \
parameters-lsc9f-4287932199: \ ]                                       /
parameters-lsc9f-4287932199:  ----------------------------------------- 
parameters-lsc9f-4287932199:     \
parameters-lsc9f-4287932199:      \
parameters-lsc9f-4287932199:       \     
parameters-lsc9f-4287932199:                     ##        .            
parameters-lsc9f-4287932199:               ## ## ##       ==            
parameters-lsc9f-4287932199:            ## ## ## ##      ===            
parameters-lsc9f-4287932199:        /""""""""""""""""___/ ===        
parameters-lsc9f-4287932199:   ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~   
parameters-lsc9f-4287932199:        \______ o          __/            
parameters-lsc9f-4287932199:         \    \        __/             
parameters-lsc9f-4287932199:           \____\______/   
parameters-lsc9f-1274151296: {{=sprig.last(steps.loop-write-some-result.outputs.parameters.results-values) }}

In addition one also gets the result.txt file, that should be located in the directory that was exposed to k8s as a persistent volume (that is your current working directory if you used e.g. minikube mount pwd:/data/host & : refer below to the "Reproducing steps" section").

The question is then: what needs to be changed (and in which way :-) in order to get sprig to be
properly executed on the aggregated result of the loop.

Reproducing steps

• Download
  sprig-usage-issue.txt workflow
  and change back its extension to yml (extension had to be changed for github to accept the document attachment) with
  e.g. mv sprig-usage-issue.txt sprig-usage-issue.yml
• argo lint sprig-usage-issue.yml should now report : ✔ no linting errors found!.
• Trigger your k8s cluster e.g. with minikube start
• Create an exported volume e.g. with minikube mount pwd:/data/host & (note that if you choose a different name for the volume then the arguments.parameters.persistedVolume entry of the sprig-usage-issue.yml workflow will need to be adapted accordingly).