[SPARK-51483] Add SparkSession and DataFrame actors
#10
Conversation
Sources/SparkConnect/DataFrame.swift
Outdated
| } | ||
|
|
||
| /// Add `Apache Arrow`'s `RecordBatch`s to the intenal array. | ||
| /// - Parameter batches: A ``RecordBatch`` instance. |
There was a problem hiding this comment.
Oops. typo. Let me fix it.
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
| @Test | ||
| func range() async throws { | ||
| let spark = try await SparkSession.builder.getOrCreate() | ||
| #expect(try await spark.range(10).count() == 10) |
There was a problem hiding this comment.
Hmm, so the test already connects to real server to execute a range plan?
There was a problem hiding this comment.
Yes, it works with the real server from now. Please see here.
There was a problem hiding this comment.
Oh, I mean in the current Github Actions, do we already have run Connect Server to run these tests?
There was a problem hiding this comment.
Improving CIs by enabling the Spark Connect server via Docker in GitHub Action CIs.
Ah, I see that it is in later steps. Thanks.
|
Thank you for helping this effort so far, @viirya . This is the last of initial implementation. After this, I'm moving forward to
|
|
According to the review comment, I didn't mention For now, this is supposed to support Apache Spark 4.0.0+ only. |
|
Thank you. For the record, the first MVP (Minimum Viable Product) is focusing on
|
|
Merged to main~ |
What changes were proposed in this pull request?
This PR aims to add
SparkSessionandDataFrameactors.SparkSession.SparkContextis defined as an emptystructjust as a type.SparkSession.Builderis defined to match with the builder pattern.Why are the changes needed?
To allow users to start to use this library. After this PR, we can run the test against the real
Spark Connectservers.Does this PR introduce any user-facing change?
No, this is not released yet.
How was this patch tested?
Pass the CIs.
Was this patch authored or co-authored using generative AI tooling?
No.