[SPARK-51987][SQL] DSv2 expressions in column defaults on write #51002
Conversation
| sealed class Metadata private[types] (private[types] val map: Map[String, Any]) | ||
| sealed class Metadata private[types] ( | ||
| private[types] val map: Map[String, Any], | ||
| @transient private[types] val runtimeMap: Map[String, Any]) |
There was a problem hiding this comment.
I am adding a runtime-only map of configs that is not serialized or exposed to the user. It will allow me to store alternative in-memory representations for certain configs. In particular, it would allow me to store SQL as well as the expression itself for default values.
There was a problem hiding this comment.
Below is an example of how it is being used:
val existsDefault = extractExistsDefault(default)
val (sql, expr) = extractCurrentDefault(default)
val newMetadata = new MetadataBuilder()
.withMetadata(f.metadata)
.putString(EXISTS_DEFAULT_COLUMN_METADATA_KEY, existsDefault)
.putExpression(CURRENT_DEFAULT_COLUMN_METADATA_KEY, sql, expr)
.build()
f.copy(metadata = newMetadata)
There was a problem hiding this comment.
This allows me to use the expression directly without generating/parsing SQL for it.
| @@ -120,6 +122,12 @@ sealed class Metadata private[types] (private[types] val map: Map[String, Any]) | |||
| map(key).asInstanceOf[T] | |||
| } | |||
|
|
|||
| private[sql] def getExpression[E](key: String): (String, Option[E]) = { | |||
There was a problem hiding this comment.
| private[sql] def getExpression[E](key: String): (String, Option[E]) = { | |
| private[sql] def getExpression[E <: Expression](key: String): (String, Option[E]) = { |
There was a problem hiding this comment.
Unfortunately, this is in api, so I don't have access to Catalyst Expression.
There was a problem hiding this comment.
ah I see. Shall we make the functions general here? e.g. we add def putAny and def getAny in class Metadata
| analyze(field.name, field.dataType, field.metadata.getString(metadataKey), statementType) | ||
| field.metadata.getExpression[Expression](metadataKey) match { | ||
| case (sql, Some(expr)) => | ||
| analyze(field.name, field.dataType, expr, sql, statementType) |
There was a problem hiding this comment.
in this branch, sql will be only used for error message?
What changes were proposed in this pull request?
This PR allows connectors to expose expression-based defaults on write.
Why are the changes needed?
These changes are needed to avoid the requirement of producing Spark SQL dialect in connectors.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
This PR comes with tests.
Was this patch authored or co-authored using generative AI tooling?
No.