Duplicate SparkApplication submission causes failure

[allen-zhang-akunacapital]

What happened?

• ✋ I have searched the open/closed issues and my issue is not listed.

I have upgraded to 2.1.1 and it looks like the controller initially tries to submit the same application 2 times, which causes error like:

failed: failed to submit spark application: failed  │
│ to run spark-submit: driver pod already exist

another side-effect is the UI svc and ingress don't get created either because they occur after SparkApplication enters submitted state

Reproduction Code

No response

Expected behavior

No response

Actual behavior

No response

Environment & Versions

• Kubernetes Version:
• Spark Operator Version:
• Apache Spark Version:

Additional context

No response

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[nabuskey]

I don't think I've seen this. Do you have another instance of spark operator running? Are you reusing application name and the previous one is not cleaned up?

[allen-zhang-akunacapital]

I don't think I've seen this. Do you have another instance of spark operator running? Are you reusing application name and the previous one is not cleaned up?

No, I made sure the old application and its associated pods are deleted. I reverted back to 2.1.0 and the issue disappeared.