[SPARK-51465] Use Apache Arrow Swift 19.0.1
#6
Conversation
|
Could you review this PR to use |
|
Could you review this PR when you have some time, @huaxingao ? |
| let byteOffset = self.arrowData.stride * Int(index) | ||
| let milliseconds = self.arrowData.buffers[1].rawPointer.advanced(by: byteOffset).load( | ||
| as: UInt32.self) | ||
| return Date(timeIntervalSince1970: TimeInterval(milliseconds * 86400)) |
There was a problem hiding this comment.
This confuses slightly, you multiple 86400 = 24 * 60 * 60 which is seconds in a day to milliseconds. IMHO, the name should seconds or (24 * 60 * 60 * milliseconds) / 1000
There was a problem hiding this comment.
Thank you for reivew, @MaxGekk .
Of course, we will report back to the upstream to be sync with them. I guess we need to help Apache Arrow Swift community because they didn't make a release yet . I expect more instances like this. We don't know if we don't try to use this.
For now, this PR is only a stepping stone to use Apache Arrow Swift, @MaxGekk , and make a Spark Connect client framework work for Swift users.
There was a problem hiding this comment.
As mentioned in the PR description, this PR is an import of Apache Arrow code with the minimal compilation fixes.
|
For the changes, do you plan to propose them to Arrow Swift repo? - public enum ArrowTypeId {
+ public enum ArrowTypeId: Sendable {
- public enum Info {
+ public enum Info: Sendable {
Is it also because they cannot be compiled in Swift 6.0? |
|
Thank you for review, @viirya . Yes, it's required when Apache Arrow starts to support Swift 6.0. |
|
Just to give the reviewers the background, |
|
After building the initial working code, we are going to entering QA period by adding more unit test coverages. And, all issues are going to the upstreams because we don't want to keep the clone of Apache Arrow Swift. |
|
Thank you so much all for being interested in this new codebase and taking a look at this to help, @MaxGekk , @viirya and @huaxingao . Merged to main. |
What changes were proposed in this pull request?
This PR aims to use
Apache Arrow Swift19.0.1. This will be replaced as a dependency when Apache Arrow Swift package is released later.This is a part of the initial implementation.
Why are the changes needed?
Apache Arrow 19.0.1 is the latest version.
Although Apache Arrow 19.0.1 has
Swiftsource code,Arrowpackage, we need to change two places to compile inSwift 6.0and we need to excludeArrowCExporter.swiftandArrowCImporter.swiftArrowFlightpackage, we need to update two places to compile inSwift 6and use only three files.Lastly,
swift formatis applied.Does this PR introduce any user-facing change?
No, this is not released yet.
How was this patch tested?
Pass the CI.
Was this patch authored or co-authored using generative AI tooling?
No.