[SPARK-53760][Geo][SQL] Introduce GeometryType and GeographyType #52491

uros-db · 2025-09-30T15:21:13Z

What changes were proposed in this pull request?

Introduce two new logical types to Spark:

GeographyType
GeometryType

This PR also adds appropriate SQL parsing and JSON serialization logic for the new types.

Why are the changes needed?

Kicking off https://issues.apache.org/jira/browse/SPARK-51658 by adding GEOMETRY and GEOGRAPHY types to Spark.

Does this PR introduce any user-facing change?

Yes, two new logical data types are added as Experimental.

How was this patch tested?

Added new tests to:

GeographyTypeSuite
GeometryTypeSuite

Also, added appropriate test cases to:

DataTypeSuite

Was this patch authored or co-authored using generative AI tooling?

No.

uros-db

@mkaravel Please review.

szehon-ho

Thanks @uros-db for kicking it off! left some comments/questions as i read it

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

sql/api/src/main/scala/org/apache/spark/sql/types/CrsMappings.java

sql/api/src/main/scala/org/apache/spark/sql/types/DataType.scala

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala

sql/api/src/main/scala/org/apache/spark/sql/types/GeometryType.scala

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala

sql/api/src/main/scala/org/apache/spark/sql/types/GeometryType.scala

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala

uros-db · 2025-10-07T08:14:40Z

Hi folks, please re-review. @szehon-ho @mkaravel

mkaravel

Generally LGTM. Please take a look at the comments I posted.
Thank you for this PR!

common/utils/src/main/resources/error/error-conditions.json

sql/api/src/main/scala/org/apache/spark/sql/types/SpatialReferenceMapper.java

mkaravel

@uros-db Thank you for addressing the comments!

LGTM.

uros-db · 2025-10-09T17:54:21Z

@cloud-fan @szehon-ho Please review. Also cc @srielau (we have some new error classes here).

szehon-ho

lgtm, be nice to document the expectation of the GeographyType and GeometryType argument (what is expected, not expected and only internal), as its a user facing API: ref: #52491 (comment)

jiayuasu · 2025-10-10T05:33:21Z

Great work. The Apache Sedona team was not aware of or informed by this PR. We are now looking into this.

jiayuasu · 2025-10-10T05:39:57Z

We will review and comment on this PR. This is to ensure it does not create incompatibility with Apache Sedona.

uros-db · 2025-10-10T07:12:21Z

Thank you @jiayuasu. We will hold the merge until you have enough time to review. Please feel free to let me know if you have any questions or concerns.

paleolimbot

Very cool! Just a few notes from the GeoArrow/Parquet compatibility angle where most CRSes are written by spatial tools with a PROJJSON or authority:code CRS.

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala

uros-db · 2025-10-13T13:15:19Z

Hi folks @paleolimbot @jiayuasu, do you have any other concerns about this PR? cc @cloud-fan

sql/api/src/main/scala/org/apache/spark/sql/types/GeometryType.scala

jiayuasu · 2025-10-14T05:10:16Z

Thanks. This PR looks good to us!

uros-db · 2025-10-14T11:29:23Z

Thanks everyone for your reviews!

@cloud-fan This PR is ready for your review now.

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala

sql/api/src/main/scala/org/apache/spark/sql/internal/types/SpatialReferenceSystemMapper.java

uros-db · 2025-10-15T01:53:37Z

@cloud-fan This should be ready for merge.

cloud-fan · 2025-10-15T03:08:57Z

thanks, merging to master!

dongjoon-hyun · 2025-10-15T03:16:23Z

Thank you, @uros-db , @cloud-fan and all.

Initial commit

3985d5a

github-actions bot added the SQL label Sep 30, 2025

uros-db commented Sep 30, 2025

View reviewed changes

uros-db added 2 commits October 1, 2025 04:35

Fix scalastyle

1124ff5

Regenerate golden files

3fa63c8

szehon-ho reviewed Oct 1, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala Show resolved Hide resolved

uros-db added 4 commits October 6, 2025 11:20

Address comments

4be6774

Run scalafmt

fcae2a1

Merge branch 'apache:master' into geo-logical-types

fa62491

Refactor classes

e2dfe8e

uros-db requested a review from szehon-ho October 6, 2025 11:08

Fix SQL keywords

e6ceef9

github-actions bot added the DOCS label Oct 6, 2025

uros-db added 3 commits October 6, 2025 17:30

Fix failing tests

3085dc0

Cleanup comment

a25550d

Fix Java lint

27fb2d2

mkaravel reviewed Oct 9, 2025

View reviewed changes

Address comments

b777086

uros-db requested a review from mkaravel October 9, 2025 17:51

mkaravel approved these changes Oct 9, 2025

View reviewed changes

szehon-ho approved these changes Oct 9, 2025

View reviewed changes

paleolimbot reviewed Oct 10, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala Show resolved Hide resolved

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala Outdated Show resolved Hide resolved

uros-db requested a review from paleolimbot October 10, 2025 17:21

Kontinuation reviewed Oct 14, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/types/GeometryType.scala Outdated Show resolved Hide resolved

Move toSrid

ee1a3eb

Move toSrid

91755bb

cloud-fan reviewed Oct 14, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala Show resolved Hide resolved

cloud-fan reviewed Oct 14, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/catalyst/parser/DataTypeAstBuilder.scala Show resolved Hide resolved

cloud-fan reviewed Oct 14, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/types/GeographyType.scala Show resolved Hide resolved

cloud-fan reviewed Oct 14, 2025

View reviewed changes

sql/api/src/main/scala/org/apache/spark/sql/internal/types/SpatialReferenceSystemMapper.java Show resolved Hide resolved

cloud-fan approved these changes Oct 14, 2025

View reviewed changes

uros-db added 2 commits October 14, 2025 17:07

Move SpatialReferenceSystemMapper

64f2dd3

Merge branch 'apache:master' into geo-logical-types

9c0828d

cloud-fan approved these changes Oct 15, 2025

View reviewed changes

cloud-fan closed this in 8ff9feb Oct 15, 2025

[SPARK-53760][Geo][SQL] Introduce GeometryType and GeographyType #52491

[SPARK-53760][Geo][SQL] Introduce GeometryType and GeographyType #52491

Conversation

uros-db commented Sep 30, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

uros-db left a comment

Choose a reason for hiding this comment

Uh oh!

szehon-ho left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uros-db commented Oct 7, 2025

Uh oh!

mkaravel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mkaravel left a comment

Choose a reason for hiding this comment

Uh oh!

uros-db commented Oct 9, 2025

Uh oh!

szehon-ho left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiayuasu commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiayuasu commented Oct 10, 2025

Uh oh!

uros-db commented Oct 10, 2025

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

uros-db commented Oct 13, 2025

Uh oh!

Uh oh!

jiayuasu commented Oct 14, 2025

Uh oh!

uros-db commented Oct 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uros-db commented Oct 15, 2025

Uh oh!

cloud-fan commented Oct 15, 2025

Uh oh!

dongjoon-hyun commented Oct 15, 2025

Uh oh!

Reviewers

Assignees

Labels

szehon-ho left a comment •

edited

Loading

jiayuasu commented Oct 10, 2025 •

edited

Loading