[SPARK-53935][BUILD] SBT assembly should handle META-INF correctly

[pan3793]

What changes were proposed in this pull request?

Fix SBT assembly for all sub-projects to handle META-INF correctly

• META-INF/*.SF (signature files, should be dropped for assembly jars)
• META-INF/services/* (service loader implementation list, should be merged for assembly jars)

Similiar thing was already done in the assembly project.

spark/project/SparkBuild.scala

Lines 1300 to 1310 in 83be7e7

	(assembly / assemblyMergeStrategy) := {
	case m if m.toLowerCase(Locale.ROOT).endsWith("manifest.mf")
	=> MergeStrategy.discard
	case m if m.toLowerCase(Locale.ROOT).matches("meta-inf.*\\.sf$")
	=> MergeStrategy.discard
	case "log4j2.properties" => MergeStrategy.discard
	case m if m.toLowerCase(Locale.ROOT).startsWith("meta-inf/services/")
	=> MergeStrategy.filterDistinctLines
	case "reference.conf" => MergeStrategy.concat
	case _ => MergeStrategy.first
	}

Why are the changes needed?

Issue was detected when I'm trying to integrate Connect JDBC driver with BeeLine in dev mode.

Does this PR introduce any user-facing change?

No, official release uses Maven, which does not get affected.

How was this patch tested?

Pass GHA.

Was this patch authored or co-authored using generative AI tooling?

No.

[pan3793]

cc @dongjoon-hyun @LuciferYang

[dongjoon-hyun]

+1, LGTM. Thank you, @pan3793 .

[LuciferYang]

+1, LGTM

[sarutak]

LGTM. Merged to master.
Thank you @pan3793 @dongjoon-hyun @LuciferYang !