[SPARK-51461] Setup `SparkConnect` Swift package structure and CI to test `build` #4

dongjoon-hyun · 2025-03-11T02:30:12Z

What changes were proposed in this pull request?

This PR aims to setup SparkConnect Swift package structure and CI to test build.

Note that this is a subset of the initial implementation.

Initial Implementation #1

Why are the changes needed?

To setup the initial package structure with CI build test coverage before adding the actual code. Currently, the following two OSs are tested.

MacOS 15
Ubuntu 24.04

According to the standard Swift package structure,

https://www.swift.org/getting-started/library-swiftpm/#bootstrapping

this PR adds the following structure for SparkConnect package. SparkConnectError.swift and BuilderTests.swift is added in order to fill the empty directories.

$ tree .
.
├── dev
│   └── merge_spark_pr.py
├── LICENSE
├── Package.swift
├── README.md
├── Sources
│   └── SparkConnect
│       └── SparkConnectError.swift
└── Tests
    └── SparkConnectTests
        └── BuilderTests.swift

Does this PR introduce any user-facing change?

No. This is not released yet.

How was this patch tested?

Pass the CI.

Was this patch authored or co-authored using generative AI tooling?

No.

…build`

dongjoon-hyun · 2025-03-11T02:35:31Z

.github/.licenserc.yaml

@@ -14,5 +14,6 @@ header:
    - 'NOTICE'
    - '.asf.yaml'
    - '.nojekyll'
+    - 'Package.swift'


This file has the required ASF license header. However, the first line of this file should be swift-tools-version: 6.0. So, I added here inevitably.

dongjoon-hyun · 2025-03-11T02:40:35Z

LICENSE

@@ -199,68 +199,3 @@
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-


The rest of this file is copied from Apache Spark repository and is invalid in this Swift repository.

dongjoon-hyun · 2025-03-11T02:42:15Z

Could you review and help this bootstrapping PR when you have some time, @yaooqinn ?

dongjoon-hyun · 2025-03-11T02:43:18Z

.gitignore

@@ -0,0 +1,66 @@
+# Xcode


The following is a standard Git ignore pattern for Xcode IDE.

yaooqinn · 2025-03-11T02:56:15Z

LICENSE

@@ -199,68 +199,3 @@
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.


Let's also create a NOTICE file, here is an example https://www.apache.org/licenses/example-NOTICE.txt

If we have required third-party notices or licenses, we shall also record those portions

Sure, Thank you, @yaooqinn .

yaooqinn · 2025-03-11T02:59:25Z

README.md

+So far, this library project is tracking the upstream changes like the [Apache Spark](https://spark.apache.org) 4.0.0 RC2 release and [Apache Arrow](https://arrow.apache.org) project's Swift-support.
+
+## Requirement
+- [Apache Spark 4.0.0 RC2 (March 2025)](https://dist.apache.org/repos/dist/dev/spark/v4.0.0-rc2-bin/)


It's better to be https://spark.apache.org/downloads.html or https://spark.apache.org/news/spark-4.0.0-preview2.html

In that case, let me remove this section.

dongjoon-hyun · 2025-03-11T03:05:13Z

Just FYI, this needs at least 4.0.0-rc2 because Spark Connect of 4.0.0-preview2 is insufficient like the following, @yaooqinn .

$ git checkout v4.0.0-rc2
$ git diff v4.0.0-preview2 sql/connect/common/src/main/protobuf/ | wc -l
     672

yaooqinn · 2025-03-11T03:07:46Z

README.md

+
+[![GitHub Actions Build](https://github.com/apache/spark-connect-swift/actions/workflows/build_and_test.yml/badge.svg)](https://github.com/apache/spark-connect-swift/blob/main/.github/workflows/build_and_test.yml)
+
+This is an experimental Swift library to show how to connect to a remote Apache Spark Connect Server and run SQL statements to manipulate remote data.


Given that the project is still in an experimental phase, it would be good to add a DISCLAIMER file

Apache Spark Connect Client for Swift is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Spark PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

However, do we have it in our sister repository, Spark Connect Go?

I'm afraid not, Apache Spark PMC doesn't quite follow the ASF podling incubation process for subprojects

Especially, the IP clearance part for huge donations :)

https://incubator.apache.org/ip-clearance/

yaooqinn · 2025-03-11T03:10:20Z

Just FYI, this needs at least 4.0.0-rc2 because Spark Connect of 4.0.0-preview2 is insufficient like the following, @yaooqinn .
$ git checkout v4.0.0-rc2
$ git diff v4.0.0-preview2 sql/connect/common/src/main/protobuf/ | wc -l
     672

Thank you for the explanation.

yaooqinn

LGTM

dongjoon-hyun · 2025-03-11T03:14:43Z

Thank you. For Incubating status, I got what you pointed. For this word, Experimental, I copied it from Spark Connect Go README. However, I'm planning to release the initial version with Apache Spark 4.1.0 timeframe after the QA. Although it doesn't guarantee the full feature parity with PySpark Connect, I believe we can consider the concrete subset as a product of TLP, instead of Incubating.

I'll reevaluate at Spark 4.1 RC period, and mark it Incubation if it's unstable. Thank you for your thoughtful advice!

dongjoon-hyun · 2025-03-11T03:16:01Z

Thank you for your time again, @yaooqinn ! 🙇🏻

viirya · 2025-03-11T03:16:46Z

README.md

+- [gRPC Swift 2.1 (March 2025)](https://github.com/grpc/grpc-swift/releases/tag/2.1.0)
+- [gRPC Swift Protobuf 1.0 (March 2025)](https://github.com/grpc/grpc-swift-protobuf/releases/tag/1.1.0)
+- [gRPC Swift NIO Transport 1.0 (March 2025)](https://github.com/grpc/grpc-swift-nio-transport/releases/tag/1.0.1)
+- [Apache Arrow Swift](https://github.com/apache/arrow/tree/main/swift)


I don't see Arrow in the dependencies, is it a transitive dependency?

For now, Apache Arrow didn't make a Swift release and didn't compile it successfully on the latest Swift 6.0 version.

So, I borrowed and edited some files from Apache Arrow like Apache Spark did for Apache Hive Thrift Server module.

It will be replaced with the official Apache Arrow Swift artifacts when they start to release~

dongjoon-hyun · 2025-03-11T03:19:16Z

Thank you for review, too, @viirya !

dongjoon-hyun · 2025-03-11T03:21:55Z

Merged to main~

[SPARK-51461] Setup SparkConnect package structure and CI to test `…

99afb3c

…build`

dongjoon-hyun commented Mar 11, 2025

View reviewed changes

.gitignore

@@ -0,0 +1,66 @@

# Xcode

Copy link

Member Author

dongjoon-hyun Mar 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following is a standard Git ignore pattern for Xcode IDE.

yaooqinn reviewed Mar 11, 2025

View reviewed changes

dongjoon-hyun added 2 commits March 10, 2025 20:00

Add NOTICE file

c712ff5

Address comment

8097b4a

yaooqinn reviewed Mar 11, 2025

View reviewed changes

yaooqinn approved these changes Mar 11, 2025

View reviewed changes

viirya reviewed Mar 11, 2025

View reviewed changes

viirya approved these changes Mar 11, 2025

View reviewed changes

dongjoon-hyun closed this in 9edc310 Mar 11, 2025

dongjoon-hyun deleted the SPARK-51461 branch March 11, 2025 03:21

This was referenced Mar 11, 2025

Initial Implementation #1

Closed

[SPARK-51483] Add SparkSession and DataFrame actors #10

Closed

		@@ -199,68 +199,3 @@
		WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.


		[![GitHub Actions Build](https://github.com/apache/spark-connect-swift/actions/workflows/build_and_test.yml/badge.svg)](https://github.com/apache/spark-connect-swift/blob/main/.github/workflows/build_and_test.yml)

		This is an experimental Swift library to show how to connect to a remote Apache Spark Connect Server and run SQL statements to manipulate remote data.

[SPARK-51461] Setup SparkConnect Swift package structure and CI to test build #4

[SPARK-51461] Setup SparkConnect Swift package structure and CI to test build #4

Uh oh!

Conversation

dongjoon-hyun commented Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Mar 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yaooqinn commented Mar 11, 2025

Uh oh!

yaooqinn left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Mar 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Mar 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Mar 11, 2025

Uh oh!

dongjoon-hyun commented Mar 11, 2025

Uh oh!

Uh oh!

[SPARK-51461] Setup `SparkConnect` Swift package structure and CI to test `build` #4

[SPARK-51461] Setup `SparkConnect` Swift package structure and CI to test `build` #4

dongjoon-hyun commented Mar 11, 2025 •

edited

Loading

dongjoon-hyun commented Mar 11, 2025 •

edited

Loading

dongjoon-hyun commented Mar 11, 2025 •

edited

Loading

dongjoon-hyun Mar 11, 2025 •

edited

Loading