[SPARK-51461] Setup SparkConnect Swift package structure and CI to test build
#4
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,5 +14,6 @@ header: | |
| - 'NOTICE' | ||
| - '.asf.yaml' | ||
| - '.nojekyll' | ||
| - 'Package.swift' | ||
|
|
||
| comment: on-failure | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| # Xcode | ||
|
There was a problem hiding this comment. The following is a standard Git ignore pattern for |
||
| # | ||
| # gitignore contributors: remember to update Global/Xcode.gitignore, Objective-C.gitignore & Swift.gitignore | ||
|
|
||
| ## Swift Package Manager | ||
| Package.resolved | ||
| .swiftpm/ | ||
|
|
||
| ## User settings | ||
| xcuserdata/ | ||
|
|
||
| ## Obj-C/Swift specific | ||
| *.hmap | ||
|
|
||
| ## App packaging | ||
| *.ipa | ||
| *.dSYM.zip | ||
| *.dSYM | ||
|
|
||
| ## Playgrounds | ||
| timeline.xctimeline | ||
| playground.xcworkspace | ||
|
|
||
| # Swift Package Manager | ||
| # | ||
| # Add this line if you want to avoid checking in source code from Swift Package Manager dependencies. | ||
| # Packages/ | ||
| # Package.pins | ||
| # Package.resolved | ||
| # *.xcodeproj | ||
| # | ||
| # Xcode automatically generates this directory with a .xcworkspacedata file and xcuserdata | ||
| # hence it is not needed unless you have added a package configuration file to your project | ||
| # .swiftpm | ||
|
|
||
| .build/ | ||
|
|
||
| # CocoaPods | ||
| # | ||
| # We recommend against adding the Pods directory to your .gitignore. However | ||
| # you should judge for yourself, the pros and cons are mentioned at: | ||
| # https://guides.cocoapods.org/using/using-cocoapods.html#should-i-check-the-pods-directory-into-source-control | ||
| # | ||
| # Pods/ | ||
| # | ||
| # Add this line if you want to avoid checking in source code from the Xcode workspace | ||
| # *.xcworkspace | ||
|
|
||
| # Carthage | ||
| # | ||
| # Add this line if you want to avoid checking in source code from Carthage dependencies. | ||
| # Carthage/Checkouts | ||
|
|
||
| Carthage/Build/ | ||
|
|
||
| # fastlane | ||
| # | ||
| # It is recommended to not store the screenshots in the git repo. | ||
| # Instead, use fastlane to re-generate the screenshots whenever they are needed. | ||
| # For more information about the recommended setup visit: | ||
| # https://docs.fastlane.tools/best-practices/source-control/#source-control | ||
|
|
||
| fastlane/report.xml | ||
| fastlane/Preview.html | ||
| fastlane/screenshots/**/*.png | ||
| fastlane/test_output | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -199,68 +199,3 @@ | |
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
|
There was a problem hiding this comment. Let's also create a NOTICE file, here is an example https://www.apache.org/licenses/example-NOTICE.txt If we have required third-party notices or licenses, we shall also record those portions There was a problem hiding this comment. Sure, Thank you, @yaooqinn . |
||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
|
There was a problem hiding this comment. The rest of this file is copied from Apache Spark repository and is invalid in this Swift repository. |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| // swift-tools-version: 6.0 | ||
| // The swift-tools-version declares the minimum version of Swift required to build this package. | ||
| // | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
| // | ||
| import PackageDescription | ||
|
|
||
| let package = Package( | ||
| name: "SparkConnect", | ||
| platforms: [ | ||
| .macOS(.v15), | ||
| .iOS(.v18), | ||
| .watchOS(.v11), | ||
| .tvOS(.v18), | ||
| ], | ||
| products: [ | ||
| .library( | ||
| name: "SparkConnect", | ||
| targets: ["SparkConnect"]) | ||
| ], | ||
| dependencies: [ | ||
| .package(url: "https://github.com/grpc/grpc-swift.git", from: "2.1.0"), | ||
| .package(url: "https://github.com/grpc/grpc-swift-protobuf.git", from: "1.1.0"), | ||
| .package(url: "https://github.com/grpc/grpc-swift-nio-transport.git", from: "1.0.1"), | ||
| .package(url: "https://github.com/scottrhoyt/SwiftyTextTable.git", from: "0.9.0"), | ||
| .package(url: "https://github.com/google/flatbuffers.git", branch: "v24.3.7"), | ||
| ], | ||
| targets: [ | ||
| .target( | ||
| name: "SparkConnect", | ||
| dependencies: [ | ||
| .product(name: "GRPCCore", package: "grpc-swift"), | ||
| .product(name: "GRPCProtobuf", package: "grpc-swift-protobuf"), | ||
| .product(name: "GRPCNIOTransportHTTP2", package: "grpc-swift-nio-transport"), | ||
| .product(name: "SwiftyTextTable", package: "SwiftyTextTable"), | ||
| .product(name: "FlatBuffers", package: "flatbuffers"), | ||
| ] | ||
| ), | ||
| .testTarget( | ||
| name: "SparkConnectTests", | ||
| dependencies: ["SparkConnect"] | ||
| ), | ||
| ] | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,60 @@ | ||
| # Apache Spark Connect Client for Swift | ||
|
|
||
| [](https://github.com/apache/spark-connect-swift/blob/main/.github/workflows/build_and_test.yml) | ||
|
|
||
| This is an experimental Swift library to show how to connect to a remote Apache Spark Connect Server and run SQL statements to manipulate remote data. | ||
|
|
||
|
There was a problem hiding this comment. Given that the project is still in an experimental phase, it would be good to add a DISCLAIMER file There was a problem hiding this comment. However, do we have it in our sister repository, There was a problem hiding this comment. I'm afraid not, Apache Spark PMC doesn't quite follow the ASF podling incubation process for subprojects There was a problem hiding this comment. Especially, the IP clearance part for huge donations :) |
||
| So far, this library project is tracking the upstream changes like the [Apache Spark](https://spark.apache.org) 4.0.0 RC2 release and [Apache Arrow](https://arrow.apache.org) project's Swift-support. | ||
|
|
||
| ## Requirement | ||
| - [Apache Spark 4.0.0 RC2 (March 2025)](https://dist.apache.org/repos/dist/dev/spark/v4.0.0-rc2-bin/) | ||
| - [Swift 6.0 (2024)](https://swift.org) | ||
|
There was a problem hiding this comment. There was a problem hiding this comment. In that case, let me remove this section. |
||
| - [gRPC Swift 2.1 (March 2025)](https://github.com/grpc/grpc-swift/releases/tag/2.1.0) | ||
| - [gRPC Swift Protobuf 1.0 (March 2025)](https://github.com/grpc/grpc-swift-protobuf/releases/tag/1.1.0) | ||
| - [gRPC Swift NIO Transport 1.0 (March 2025)](https://github.com/grpc/grpc-swift-nio-transport/releases/tag/1.0.1) | ||
| - [Apache Arrow Swift](https://github.com/apache/arrow/tree/main/swift) | ||
|
|
||
|
There was a problem hiding this comment. I don't see Arrow in the dependencies, is it a transitive dependency? There was a problem hiding this comment. For now, Apache Arrow didn't make a So, I borrowed and edited some files from Apache Arrow like Apache Spark did for Apache Hive Thrift Server module. There was a problem hiding this comment. It will be replaced with the official Apache Arrow Swift artifacts when they start to release~ |
||
| ## Run `Apache Spark 4.0.0 RC2 Connect Server` | ||
|
|
||
| $ curl -LO https://dist.apache.org/repos/dist/dev/spark/v4.0.0-rc2-bin/spark-4.0.0-bin-hadoop3.tgz | ||
| $ tar xvfz spark-4.0.0-bin-hadoop3.tgz | ||
| $ cd spark-4.0.0-bin-hadoop3 | ||
| $ sbin/start-connect-server.sh | ||
|
|
||
| ## Run tests | ||
|
|
||
| ``` | ||
| $ cd spark-connect-swift | ||
| $ swift test | ||
| ``` | ||
|
|
||
| ## How to use in your apps | ||
|
|
||
| Create a Swift project. | ||
| ``` | ||
| $ mkdir SparkConnectSwiftApp | ||
| $ cd SparkConnectSwiftApp | ||
| $ swift package init --name SparkConnectSwiftApp --type executable | ||
| ``` | ||
|
|
||
| Add `SparkConnect` package to the dependency like the following | ||
| ``` | ||
| $ cat Package.swift | ||
| import PackageDescription | ||
|
|
||
| let package = Package( | ||
| name: "SparkConnectSwiftApp", | ||
| platforms: [ | ||
| .macOS(.v15) | ||
| ], | ||
| dependencies: [ | ||
| .package(url: "https://github.com/apache/spark-connect-swift.git", branch: "main") | ||
| ], | ||
| targets: [ | ||
| .executableTarget( | ||
| name: "SparkConnectSwiftApp", | ||
| dependencies: [.product(name: "SparkConnect", package: "spark-connect-swift")] | ||
| ) | ||
| ] | ||
| ) | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| // | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
| // | ||
|
|
||
| /// A enum for ``SparkConnect`` package errors | ||
| enum SparkConnectError: Error { | ||
| case UnsupportedOperationException | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| // | ||
| // Licensed to the Apache Software Foundation (ASF) under one | ||
| // or more contributor license agreements. See the NOTICE file | ||
| // distributed with this work for additional information | ||
| // regarding copyright ownership. The ASF licenses this file | ||
| // to you under the Apache License, Version 2.0 (the | ||
| // "License"); you may not use this file except in compliance | ||
| // with the License. You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, | ||
| // software distributed under the License is distributed on an | ||
| // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, either express or implied. See the License for the | ||
| // specific language governing permissions and limitations | ||
| // under the License. | ||
| // | ||
|
|
||
| import Testing | ||
|
|
||
| @testable import SparkConnect | ||
|
|
||
| /// A test suite for `SparkSession.Builder` | ||
| @Suite(.serialized) | ||
| struct BuilderTests { | ||
| @Test | ||
| func builderDefault() async throws { | ||
| } | ||
|
|
||
| @Test | ||
| func remote() async throws { | ||
| } | ||
|
|
||
| @Test | ||
| func appName() async throws { | ||
| } | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file has the required ASF license header. However, the first line of this file should be
swift-tools-version: 6.0. So, I added here inevitably.