Argo workflow not retrying inline script workflow on "pod deleted" even thought retry is configured

[omerlh]

I have the following workflow:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  labels:
    argocd.argoproj.io/instance: <your-argocd-instance>
  name: <your-workflow-template-name>
  namespace: <your-namespace>
spec:
  entrypoint: main
  templates:
    - activeDeadlineSeconds: 600
      container:
        activeDeadlineSeconds: 600
        args:
          - >
            <my logic>
        command:
          - sh
          - '-c'
        image: amazon/aws-cli
        name: main
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 256Mi
        retryStrategy:
          backoff:
            duration: 10s
            factor: '2'
            maxDuration: 10h
          limit: '3'
          retryPolicy: Always
        volumeMounts:
          - mountPath: /mnt/data
            name: '{{inputs.parameters.volumeName}}'
      inputs:
        parameters:
          - name: s3FolderName
          - name: inputFile
          - name: volumeName
      metadata:
        labels:
          step: main
      name: main
      outputs:
        parameters:
          - globalName: uploaded_file_s3Path
            name: s3Path
            valueFrom:
              path: /tmp/s3-path.txt
          - globalName: uploaded_file_s3FullFilePath
            name: s3FullFilePath
            valueFrom:
              path: /tmp/s3-file-path.txt
      serviceAccountName: main

I suspect retry policy works differently when handling inline script. When a pod is deleted the workflow is not retried. We have other workflows which just use container instead of inline script, same retry policy but they do retry when pod is deleted.
I can see this in the logs:

Node not set to be retried after status: Failed
And
Node not set to be retried after status: Error
But we do have retry policy so I don't get why it's not working