Retrying evicted workflows on spot instances

[Zujiry]

Hello community,
we have the problem, that tasks that have a long running time are getting evicted - at least that is our assumption from the error code:
"Error: exit status 143".

Accompanied by a warning in the argo-workflow wait container:
"Non-transient error: context canceled"

We are guessing that is not a python error but rather the instance shutting down.

Now the problem is that we have 2 different cases:

• Testing or actual errors in the code / with the data
• Dubious error 143

And we do not want to retry errors that we need to fix first before running them again. Therefore my question: How can we retry only workflows that have this error? As far as Iwecould find there is no: "OnNonTransientError".