Skip to content

Update Delta version to 4.0.0 #242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 12, 2025
Merged

Update Delta version to 4.0.0 #242

merged 1 commit into from
Jun 12, 2025

Conversation

chenjian2664
Copy link
Contributor

No description provided.

@cla-bot cla-bot bot added the cla-signed label Jun 10, 2025
@chenjian2664 chenjian2664 requested a review from ebyhr June 10, 2025 01:00
@ebyhr
Copy link
Member

ebyhr commented Jun 10, 2025

Could you rebase on master to resolve conflicts?

Also, I think we should rename the image spark3-delta from spark4-delta.

@chenjian2664 chenjian2664 force-pushed the jack/delta-4.0.0 branch 3 times, most recently from f19bc12 to c21de7c Compare June 10, 2025 01:12
chenjian2664
chenjian2664 commented Jun 10, 2025
View reviewed changes
@chenjian2664
Copy link
Contributor Author

encountered:

spark               | Error executing query with 6f431bb5-f30f-4411-b18d-2a5306836df1, currentState RUNNING, 
spark               | java.lang.NumberFormatException: For input string: "60s"
spark               |   at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67) ~[?:?]
spark               |   at java.base/java.lang.Long.parseLong(Long.java:711) ~[?:?]
spark               |   at java.base/java.lang.Long.parseLong(Long.java:836) ~[?:?]
spark               |   at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1607) ~[hadoop-client-api-3.4.1.jar:?]
spark               |   at org.apache.hadoop.fs.s3a.S3AUtils.longOption(S3AUtils.java:1024) ~[hadoop-aws-3.3.4.jar:?]
spark               |   at org.apache.hadoop.fs.s3a.S3AFileSystem.initThreadPools(S3AFileSystem.java:719) ~[hadoop-aws-3.3.4.jar:?]
spark               |   at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:498) ~[hadoop-aws-3.3.4.jar:?]
spark               |   at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3615) ~[hadoop-client-api-3.4.1.jar:?]
spark               |   at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:172) ~[hadoop-client-api-3.4.1.jar:?]
spark               |   at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3716) ~[hadoop-client-api-3.4.1.jar:?]
spark               |   at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3667) ~[hadoop-client-api-3.4.1.jar:?]
spark               |   at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557) ~[hadoop-client-api-3.4.1.jar:?]
spark               |   at org.apache.hadoop.fs.Path.getFileSystem(Path.java:366) ~[hadoop-client-api-3.4.1.jar:?]
spark               |   at org.apache.spark.sql.delta.DeltaLog$.apply(DeltaLog.scala:966) ~[delta-spark_2.13-4.0.0.jar:4.0.0]
spark               |   at org.apache.spark.sql.delta.DeltaLog$.forTable(DeltaLog.scala:857) ~[delta-spark_2.13-4.0.0.jar:4.0.0]
spark               |   at org.apache.spark.sql.delta.catalog.DeltaTableV2.$anonfun$deltaLog$1(DeltaTableV2.scala:130) ~[delta-spark_2.13-4.0.0.jar:4.0.0]

Add the configuration resolved.

@chenjian2664
Copy link
Contributor Author

encountering :

           | 
tests               | 2025-06-12 07:20:22 INFO: [92 of 171] io.trino.tests.product.deltalake.TestDeltaLakeCheckpointsCompatibility.testTrinoWriteStatsAsJsonEnabled [date, date '2021-02-03', null, 1.0, 0.0, 2021-02-03] (Groups: profile_specific_tests, delta-lake-oss)
spark               | 	at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1(RuleExecutor.scala:239) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
spark               | 	at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$execute$1$adapted(RuleExecutor.scala:231) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
spark               | 	at scala.collection.immutable.List.foreach(List.scala:334) ~[scala-library-2.13.16.jar:?]
spark               | 	at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:231) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
spark               | 	at org.apache.spark.sql.catalyst.analysis.Analyzer.org$apache$spark$sql$catalyst$analysis$Analyzer$$executeSameContext(Analyzer.scala:290) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
spark               | 	at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$execute$1(Analyzer.scala:286) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
spark               | 	at org.apache.spark.sql.catalyst.analysis.AnalysisContext$.withNewAnalysisContext(Analyzer.scala:234) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
spark               | 	at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:286) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
tests               | 2025-06-12 07:20:22 INFO: FlakyTestRetryAnalyzer not enabled: CONTINUOUS_INTEGRATION environment is not detected or system property 'io.trino.testng.services.FlakyTestRetryAnalyzer.enabled' is not set to 'true' (actual: <not set>)
spark               | 	at org.apache.spark.sql.catalyst.analysis.Analyzer.execute(Analyzer.scala:249) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
tests               | 2025-06-12 07:20:22 INFO: FAILURE     /    io.trino.tests.product.deltalake.TestDeltaLakeCheckpointsCompatibility.testTrinoWriteStatsAsJsonEnabled [date, date '2021-02-03', null, 1.0, 0.0, 2021-02-03] (Groups: profile_specific_tests, delta-lake-oss) took 0.2 seconds
tests               | 2025-06-12 07:20:22 SEVERE: Failure cause:
spark               | 	at org.apache.spark.sql.catalyst.rules.RuleExecutor.$anonfun$executeAndTrack$1(RuleExecutor.scala:201) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
tests               | io.trino.tempto.query.QueryExecutionException: java.sql.SQLException: org.apache.hive.service.cli.HiveSQLException: Error running query: java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider not found
spark               | 	at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:89) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
spark               | 	at org.apache.spark.sql.catalyst.rules.RuleExecutor.executeAndTrack(RuleExecutor.scala:201) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:46)
spark               | 	at org.apache.spark.sql.catalyst.analysis.resolver.HybridAnalyzer.resolveInFixedPoint(HybridAnalyzer.scala:190) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:274)
spark               | 	at org.apache.spark.sql.catalyst.analysis.resolver.HybridAnalyzer.$anonfun$apply$1(HybridAnalyzer.scala:76) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:172)
tests               | 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
spark               | 	at org.apache.spark.sql.catalyst.analysis.resolver.HybridAnalyzer.withTrackedAnalyzerBridgeState(HybridAnalyzer.scala:111) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:80)
spark               | 	at org.apache.spark.sql.catalyst.analysis.resolver.HybridAnalyzer.apply(HybridAnalyzer.scala:71) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:64)
spark               | 	at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:280) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:42)
spark               | 	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:423) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:172)
spark               | 	at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:280) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:167)
tests               | 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
spark               | 	at org.apache.spark.sql.execution.QueryExecution.$anonfun$lazyAnalyzed$2(QueryExecution.scala:110) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
tests               | 	at java.base/javax.security.auth.Subject.doAs(Subject.java:439)
spark               | 	at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:148) ~[spark-catalyst_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
spark               | 	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$2(QueryExecution.scala:278) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
spark               | 	at org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:654) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
spark               | 	at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:278) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
spark               | 	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:804) ~[spark-sql-api_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:181)
spark               | 	at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:277) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
tests               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
spark               | 	at org.apache.spark.sql.execution.QueryExecution.$anonfun$lazyAnalyzed$1(QueryExecution.scala:110) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
tests               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
spark               | 	at scala.util.Try$.apply(Try.scala:217) ~[scala-library-2.13.16.jar:?]
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
spark               | 	at org.apache.spark.util.Utils$.doTryWithCallerStacktrace(Utils.scala:1378) ~[spark-core_2.13-4.0.0.jar:4.0.0]
tests               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
spark               | 	at org.apache.spark.util.Utils$.getTryWithCallerStacktrace(Utils.scala:1439) ~[spark-core_2.13-4.0.0.jar:4.0.0]
tests               | 	at java.base/java.lang.Thread.run(Thread.java:833)
spark               | 	at org.apache.spark.util.LazyTry.get(LazyTry.scala:58) ~[spark-core_2.13-4.0.0.jar:4.0.0]
tests               | Caused by: java.io.IOException: From option fs.s3a.aws.credentials.provider java.lang.ClassNotFoundException: Class software.amazon.awssdk.auth.credentials.EnvironmentVariableCredentialsProvider not found
spark               | 	at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:121) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses(S3AUtils.java:657)
tests               | 	at org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList(S3AUtils.java:680)
tests               | 	at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:631)
tests               | 	at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:877)
spark               | 	at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:80) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:534)
tests               | 	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3615)
tests               | 	at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:172)
tests               | 	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3716)
tests               | 	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3667)
tests               | 	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:557)
tests               | 	at org.apache.hadoop.fs.Path.getFileSystem(Path.java:366)
tests               | 	at org.apache.spark.sql.delta.DeltaLog$.apply(DeltaLog.scala:966)
tests               | 	at org.apache.spark.sql.delta.DeltaLog$.forTable(DeltaLog.scala:801)
tests               | 	at org.apache.spark.sql.delta.commands.CreateDeltaTableCommand.run(CreateDeltaTableCommand.scala:134)
spark               | 	at org.apache.spark.sql.classic.Dataset$.$anonfun$ofRows$5(Dataset.scala:139) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.delta.catalog.DeltaCatalog.$anonfun$createDeltaTable$1(DeltaCatalog.scala:223)
spark               | 	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:804) ~[spark-sql-api_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:171)
spark               | 	at org.apache.spark.sql.classic.Dataset$.ofRows(Dataset.scala:136) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:169)
spark               | 	at org.apache.spark.sql.classic.SparkSession.$anonfun$sql$4(SparkSession.scala:499) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.delta.catalog.DeltaCatalog.recordFrameProfile(DeltaCatalog.scala:69)
spark               | 	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:804) ~[spark-sql-api_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.delta.catalog.DeltaCatalog.org$apache$spark$sql$delta$catalog$DeltaCatalog$$createDeltaTable(DeltaCatalog.scala:105)
spark               | 	at org.apache.spark.sql.classic.SparkSession.sql(SparkSession.scala:490) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.delta.catalog.DeltaCatalog.$anonfun$createTable$1(DeltaCatalog.scala:372)
spark               | 	at org.apache.spark.sql.classic.SparkSession.sql(SparkSession.scala:504) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile(DeltaLogging.scala:171)
tests               | 	at org.apache.spark.sql.delta.metering.DeltaLogging.recordFrameProfile$(DeltaLogging.scala:169)
tests               | 	at org.apache.spark.sql.delta.catalog.DeltaCatalog.recordFrameProfile(DeltaCatalog.scala:69)
tests               | 	at org.apache.spark.sql.delta.catalog.DeltaCatalog.createTable(DeltaCatalog.scala:352)
spark               | 	at org.apache.spark.sql.classic.SparkSession.sql(SparkSession.scala:513) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.delta.catalog.DeltaCatalog.createTable(DeltaCatalog.scala:343)
spark               | 	at org.apache.spark.sql.classic.SparkSession.sql(SparkSession.scala:91) ~[spark-sql_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.execution.datasources.v2.CreateTableExec.run(CreateTableExec.scala:46)
spark               | 	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:235) ~[spark-hive-thriftserver_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)
spark               | 	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:172) ~[spark-hive-thriftserver_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43)
spark               | 	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18) ~[scala-library-2.13.16.jar:?]
tests               | 	at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49)
tests               | 	at org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$2(QueryExecution.scala:155)
tests               | 	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$8(SQLExecution.scala:162)
spark               | 	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:80) ~[spark-hive-thriftserver_2.13-4.0.0.jar:4.0.0]
spark               | 	at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:64) ~[spark-hive-thriftserver_2.13-4.0.0.jar:4.0.0]
spark               | 	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:42) ~[spark-hive-thriftserver_2.13-4.0.0.jar:4.0.0]
spark               | 	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:172) ~[spark-hive-thriftserver_2.13-4.0.0.jar:4.0.0]
tests               | 	at org.apache.spark.sql.execution.SQLExecution$.withSessionTagsApplied(SQLExecution.scala:268)
spark               | 	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:167) ~[spark-hive-thriftserver_2.13-4.0.0.jar:4.0.0]
spark               | 	at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) [?:?]
spark               | 	at java.base/javax.security.auth.Subject.doAs(Subject.java:439) [?:?]
tests               | 	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$7(SQLExecution.scala:124)
spark               | 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953) [hadoop-client-api-3.4.1.jar:?]
spark               | 	at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:181) [spark-hive-thriftserver_2.13-4.0.0.jar:4.0.0]
spark               | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
spark               | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
tests               | 	at org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:94)
spark               | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
spark               | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
spark               | 	at java.base/java.lang.Thread.run(Thread.java:833) [?:?]
tests               | 	at org.apache.spark.sql.artifact.ArtifactManager.$anonfun$withResources$1(ArtifactManager.scala:112)
spark               | 	Suppressed: org.apache.spark.util.Utils$OriginalTryStackTraceException: Full stacktrace of original doTryWithCallerStacktrace caller
tests               | 	at org.apache.spark.sql.artifact.ArtifactManager.withClassLoaderIfNeeded(ArtifactManager.scala:106)
spark               | 		at org.apache.hadoop.fs.s3a.S3AUtils.loadAWSProviderClasses(S3AUtils.java:657) ~[hadoop-aws-3.3.4.jar:?]
tests               | 	at org.apache.spark.sql.artifact.ArtifactManager.withResources(ArtifactManager.scala:111)
spark               | 		at org.apache.hadoop.fs.s3a.S3AUtils.buildAWSProviderList(S3AUtils.java:680) ~[hadoop-aws-3.3.4.jar:?]
tests               | 	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$6(SQLExecution.scala:124)
spark               | 		at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:631) ~[hadoop-aws-3.3.4.jar:?]
tests               | 	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:291)
spark               | 		at org.apache.hadoop.fs.s3a.S3AFileSystem.bindAWSClient(S3AFileSystem.java:877) ~[hadoop-aws-3.3.4.jar:?]
spark               | 		at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:534) ~[hadoop-aws-3.3.4.jar:?]
tests               | 	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$1(SQLExecution.scala:123)
spark               | 		at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3615) ~[hadoop-client-ap`

@chenjian2664
Copy link
Contributor Author

Added:
spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider
spark.hadoop.fs.s3a.connection.establish.timeout=3000
spark.hadoop.fs.s3a.connection.timeout=200000
spark.hadoop.fs.s3a.multipart.purge.age=86400

@chenjian2664 chenjian2664 mentioned this pull request Jun 12, 2025
Merged
@chenjian2664
Copy link
Contributor Author

@ebyhr Any suggestions to fix the ci failures

@ebyhr
Copy link
Member

ebyhr commented Jun 12, 2025 •
edited
Loading

Replacing curl with nc resolved the failure.

You could debug the issue by running the below command:

docker-images/bin/test.sh

Lines 69 to 71 in 0dfa3d6

function check_spark() {
environment_compose exec spark curl --http0.9 -f http://localhost:10213 -o /dev/null
}

@ebyhr ebyhr force-pushed the jack/delta-4.0.0 branch from de32c63 to 5fb4ccb Compare June 12, 2025 23:01
@ebyhr ebyhr merged commit ba3a0fa into trinodb:master Jun 12, 2025
13 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ebyhr ebyhr Awaiting requested review from ebyhr

Assignees
No one assigned
Labels
cla-signed
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@chenjian2664 @ebyhr