Argo Workflows v3.4: not terminate a running workflow after patching with activeDeadlineSeconds=0 or shutdown=Terminate

[gitfish77]

I am working on a KFP (Kubeflow Pipeline) deployment backed by Argo Workflows v3.4.
A workflow run cannot be terminated either via KFP API or via Argo API or via kubectl directly on the Argo workflow.
I tried the following and got no error (none regarding params, access permission, etc) from the client API calls or the kubectl commands, yet the Argo workflow simply didn't stop and keep running. After the current step is completed, the workflow has Phase=Failed and no further steps is executed.

1] KFP client API: kfpRunClient.TerminateRun
This actually patches activeDeadlineSeconds=0 to the Argo workflow, as I can see from kubectl describe workflow.
2] Argo client API:

	patchParams := map[string]interface{}{
		"spec": map[string]interface{}{
			"activeDeadlineSeconds": 0,
			"shutdown":              "Terminate",
		},
	}
OptionTrue := true
argoWorkflowApi.Patch(ctx, argoRunName, types.MergePatchType, patchJson, metav1.PatchOptions{Force: &OptionTrue})

I tried combination of activeDeadlineSeconds and/or shutdown.
Tried without the Force option.
Tried with activeDeadlineSeconds=1 or activeDeadlineSeconds=10.
3] kubectl command:

kubectl patch workflow workflow_name_1 -n kubeflow -p '{"spec": {"shutdown": "Terminate"}}' --type=merge
kubectl patch workflow workflow_name_1 -n kubeflow -p '{"spec": {"activeDeadlineSeconds": 0}}' --type=merge
kubectl patch workflow workflow_name_1 -n kubeflow -p '{"spec": {"activeDeadlineSeconds": 1}}' --type=merge

Then I checked by kubectl describe workflow and confirmed that the expected patches were already in the workflow run.
From the above output for the workflow I didn't see any event related to the termination request.

I can successfully delete the running workflow, however I worry the automatic archiving of pod logs will be lost.
I don't have the option to upgrade the KFP and Argo Workflows at this time though.

I noticed there is another deployment with Argo Workflows v3.6 and the activeDeadlineSeconds=0 patch works well to terminate the workflow immediately.

Can you pls help on other options that I might try with?
Or should I look into any of the Argo Workflows configuration/configmap?

[gitfish77]

I tried the following workaround to forcefully terminating an Argo run and it worked out well: directly deleting the running pods with the matching workflow run label on the Kubernetes cluster. E.g. workflows.argoproj.io/workflow=ecf9eecb-4e35-...
Side effects: the deleted job pods won't have their pod logs archived by Argo Workflows.

Ended up with a better solution:
We know that each running workflow job pod has a main container. Such container can be terminated with "kill -TERM 1" killing the root process with PID==1 in the container. Then the pod will stop with error and leave the pod logs to be collected and archived by Argo Workflows.