[SPARK-52308][CORE] Fix the bug that the actual configs of the driver are overwritten

[huangxiaopingRD]

What changes were proposed in this pull request?

After Spark Driver is started, it is not allowed to overwrite Driver-related configurations.

Why are the changes needed?

Fix the problem that the driver-related configuration was overwritten by the user by mistake, resulting in the final spark UI display error.

Does this PR introduce any user-facing change?

Yes. The values ​​of driver-related configurations on the Spark UI will show the actual effective values

How was this patch tested?

The manual test is as follows:
Test Scripts：
./bin/spark-submit --conf spark.driver.memory=10g --conf spark.driver.memoryOverhead=2g --queue dev test_pyspark.py

test_pyspark.py

from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext, HiveContext, SparkSession

SPARK_CONF = SparkConf().set('spark.driver.memory','2.5G').set('spark.driver.memoryOverhead','2.5G')

sc = SparkContext(appName="test pyspark", conf=SPARK_CONF)

spark = SparkSession.builder.enableHiveSupport().getOrCreate()

df = spark.sql("select 1")
df.show()
spark.stop()

Spark UI display before repair：

Spark UI display after repair：

BTW, Spark does not support double type values ​​for memory configuration ​​by default. So when we used Spark code to parse the event log, we encountered this problem. This PR is to avoid this kind of problem from the source.

Was this patch authored or co-authored using generative AI tooling?

No