[SPARK-51987][SPARK-52235][SQL][FOLLOW-UP] Add ANSI mode check for new tests #51092
Conversation
|
Test failure look unrelated, re-running |
| "1", | ||
| new V2Cast(LiteralValue(1, IntegerType), IntegerType, DoubleType), | ||
| LiteralValue(1.0, DoubleType)))) | ||
| withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") { |
There was a problem hiding this comment.
to avoid similar issues in the future, can we turn on ANSI for the entire test suite? we can override def sparkConf like https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/TPCDSQuerySuite.scala#L32
There was a problem hiding this comment.
Hm it didnt work, i think this goes to another sparkconf that's not used by this suite? I guess that's why there's a separate before to override the conf, i put it there.
| @@ -43,6 +43,7 @@ class DataSourceV2DataFrameSuite | |||
| before { | |||
| spark.conf.set("spark.sql.catalog.testcat", classOf[InMemoryTableCatalog].getName) | |||
| spark.conf.set("spark.sql.catalog.testcat2", classOf[InMemoryTableCatalog].getName) | |||
| spark.conf.set(SQLConf.ANSI_ENABLED.key, "true") | |||
There was a problem hiding this comment.
This works but we shouldn't set this conf repeatedly for each test case. I still prefer to override def sparkConf to set the conf once for the entire suite.
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2DataFrameSuite.scala
Outdated
Show resolved
Hide resolved
…rceV2DataFrameSuite.scala Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>
cloud-fan
changed the title
|
Thanks! Sorry should have seen its cleared by after() |
|
thanks, merging to master! |
…w tests ### What changes were proposed in this pull request? Fix non-ANSI test breakage as per apache#50959 (review) ### Why are the changes needed? Many V2 Expressions are only converted successfully in ANSI mode, so this test for V2 Expression only makes sense in that mode. ### Does this PR introduce _any_ user-facing change? No, test only ### How was this patch tested? Run test in NON-ANSI mode ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#51092 from szehon-ho/SPARK-52235-follow. Lead-authored-by: Szehon Ho <szehon.apache@gmail.com> Co-authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Wenchen Fan <wenchen@databricks.com>
What changes were proposed in this pull request?
Fix non-ANSI test breakage as per #50959 (review)
Why are the changes needed?
Many V2 Expressions are only converted successfully in ANSI mode, so this test for V2 Expression only makes sense in that mode.
Does this PR introduce any user-facing change?
No, test only
How was this patch tested?
Run test in NON-ANSI mode
Was this patch authored or co-authored using generative AI tooling?
No