Skip to content

Commit 56496d4

Browse files
cxzl25yhuang-db
authored andcommitted
[SPARK-52396][SQL] Artifact Root Directory should use tmpdir
### What changes were proposed in this pull request? ### Why are the changes needed? Now the `artifacts` directory will try to be created at workdir, which may not have permission to create write. `artifacts` is a temporary directory, and although we have an exit cleanup mechanism, it's better to put it in tmpdir, driver because `artifacts` may not be cleaned up after an OOM exit. ```java org.apache.spark.sql.execution.QueryExecutionException: java.io.IOException: Failed to create a temp directory (under artifacts) after 10 attempts! at org.apache.spark.network.util.JavaUtils.createDirectory(JavaUtils.java:411) at org.apache.spark.util.SparkFileUtils.createDirectory(SparkFileUtils.scala:95) at org.apache.spark.util.SparkFileUtils.createDirectory$(SparkFileUtils.scala:94) at org.apache.spark.util.Utils$.createDirectory(Utils.scala:99) at org.apache.spark.util.Utils$.createTempDir(Utils.scala:249) at org.apache.spark.sql.artifact.ArtifactManager$.artifactRootDirectory$lzycompute(ArtifactManager.scala:468) at org.apache.spark.sql.artifact.ArtifactManager$.artifactRootDirectory(ArtifactManager.scala:467) at org.apache.spark.sql.artifact.ArtifactManager.artifactRootPath(ArtifactManager.scala:60) at org.apache.spark.sql.artifact.ArtifactManager.<init>(ArtifactManager.scala:70) at org.apache.spark.sql.internal.BaseSessionStateBuilder.$anonfun$artifactManager$2(BaseSessionStateBuilder.scala:395) ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? local test ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#51083 from cxzl25/SPARK-52396. Authored-by: sychen <sychen@ctrip.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 9fdb4a8 commit 56496d4

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

sql/core/src/main/scala/org/apache/spark/sql/artifact/ArtifactManager.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -465,7 +465,7 @@ object ArtifactManager extends Logging {
465465
val ARTIFACT_DIRECTORY_PREFIX = "artifacts"
466466

467467
private[artifact] lazy val artifactRootDirectory =
468-
Utils.createTempDir(ARTIFACT_DIRECTORY_PREFIX).toPath
468+
Utils.createTempDir(namePrefix = ARTIFACT_DIRECTORY_PREFIX).toPath
469469

470470
private[artifact] object SparkContextResourceType extends Enumeration {
471471
type ResourceType = Value

0 commit comments

Comments
 (0)