Hard-coded restartPolicy for Workflow Pods

[mikekaram]

Hello,

I'm trying to run a simple Argo workflow on a Kubernetes node that has unstable network connectivity.
Sometimes, I'm getting the following weird behaviour:

1. During step cleanup, the Argo Workflows controller execs into the pod to kill the sidecar container
2. Due to a network disruption, sometimes the exec fails with error: error dialing backend: EOF
3. The Argo Workflows controller restarts the pod, although the restartPolicy is Never. However if the restartPolicy is actually Never, the pod for the step should just fail (regardless of the error) and fail the entire workflow. In this case however the step fails and the pod restarts, which is not expected. Finally retryPolicy is probably irrelevant in this case since if the step was retried a new pod would be created and the same pod would not be restarted again.

Example workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-restart-error-
spec:
  serviceAccountName: argo-workflow
  imagePullSecrets:
  - name: my-secret
  ttlStrategy:
    secondsAfterCompletion: 100
    secondsAfterSuccess: 100
    secondsAfterFailure: 1200
  nodeSelector: {testNode: testNode}
  hostNetwork: true
  dnsPolicy: ClusterFirstWithHostNet
  entrypoint: main
​
  templates:
    - name: main
      container:
        image: "curlimages/curl"
        command: [/bin/sh, -c]
        args: ['trap "exit 0" TERM INT; sleep 120 && echo FINISHED & wait']
      sidecars:
      - name: nginx
        image: nginx:1.13
        command: [nginx, -g, daemon off;]

[maxsxu]

Could you give more details and logs? It would be better if you could show how to reproduce.