409 Conflict from submit_workflow request

[fel-este]

I'm trying to understand the underlying cause of the following Error Response we receive when submitting workflows:

 Status: 409, 
reason: Conflict, 
body: {"code":6,"message":"Operation cannot be fulfilled on resourcequotas \"argo\": the object has been modified; please apply your changes to the latest version and try again"}

Resource Quota argo has been modified. But I do not understand what this means.

[agilgur5]

It would be helpful to show your full Workflow and request for context.
Especially as a 409 Conflict error is a k8s error (not an Argo one) and a ResourceQuota is a k8s resource (not managed by Argo).

A 409 Conflict means that the resourceVersion and content of the resource changed since it was retrieved. For instance, if you did a get and then an update, the update could get a 409 Conflict if the resource has changed since the get

[fel-este]

I can't show my workflow but I can show my request.

If this helps, I was using the python SDK's submit_workflow: https://github.com/argoproj-labs/argo-client-python/blob/master/docs/WorkflowServiceApi.md#submit_workflow

                submitted = self._argo.submit_workflow(
                    namespace=argo_namespace,
                    body=WorkflowSubmitRequest(
                        resource_kind='WorkflowTemplate',
                        resource_name=workflow_template_name,
                        submit_options=submit_options
                    ),
                    _check_return_type=False,
                )

I am assuming the argo client was what had changed causing the 409. Here is how we set up the argo client.

configuration = argo_workflows.Configuration(
            host=host,
            ssl_ca_cert=cert,
            api_key={
                'BearerToken': token
            }
        )
self._api_client = argo_workflows.ApiClient(configuration)
self._argo = workflow_service_api.WorkflowServiceApi(self._api_client)

I am unsure how to evade this problem in the future. What component needs to be "refreshed" to avoid the 409 conflict?

[agilgur5]

I was using the python SDK's

In case you missed it, I did mention that we recommend Hera instead in your earlier discussion #13507

I am assuming the argo client was what had changed causing the 409.

No, it's an error with a k8s resource, and the error says it's on a ResourceQuota

I can't show my workflow

Hence, I imagine it's something in your Workflow. Without seeing the Workflow, it's very difficult to debug anything.

Although getting this during submit is indeed strange, that would suggest the API is modifying something 🤔 Afaik, it has no such capability to modify ResourceQuotas, nor RBAC to do so.
The only other thing I can guess is that you have a ResourceQuota on kind: Workflow, and k8s is giving this error in an odd way

[fel-este]

Here are what we are specifying resourceQuota limits on:

      resourceQuota:
        hard:
          count/workflows.argoproj.io: ####
          limits.memory: ###
          requests.cpu: "###"
          requests.memory: ####

Do changes on these values cause exception when we submit a fresh/new workflow to the argo client?

[Joibel]

Is it possible the 409 is from a mutating admission controller?