You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-52396][SQL] Artifact Root Directory should use tmpdir
### What changes were proposed in this pull request?
### Why are the changes needed?
Now the `artifacts` directory will try to be created at workdir, which may not have permission to create write.
`artifacts` is a temporary directory, and although we have an exit cleanup mechanism, it's better to put it in tmpdir, driver because `artifacts` may not be cleaned up after an OOM exit.
```java
org.apache.spark.sql.execution.QueryExecutionException: java.io.IOException: Failed to create a temp directory (under artifacts) after 10 attempts!
at org.apache.spark.network.util.JavaUtils.createDirectory(JavaUtils.java:411)
at org.apache.spark.util.SparkFileUtils.createDirectory(SparkFileUtils.scala:95)
at org.apache.spark.util.SparkFileUtils.createDirectory$(SparkFileUtils.scala:94)
at org.apache.spark.util.Utils$.createDirectory(Utils.scala:99)
at org.apache.spark.util.Utils$.createTempDir(Utils.scala:249)
at org.apache.spark.sql.artifact.ArtifactManager$.artifactRootDirectory$lzycompute(ArtifactManager.scala:468)
at org.apache.spark.sql.artifact.ArtifactManager$.artifactRootDirectory(ArtifactManager.scala:467)
at org.apache.spark.sql.artifact.ArtifactManager.artifactRootPath(ArtifactManager.scala:60)
at org.apache.spark.sql.artifact.ArtifactManager.<init>(ArtifactManager.scala:70)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.$anonfun$artifactManager$2(BaseSessionStateBuilder.scala:395)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
local test
### Was this patch authored or co-authored using generative AI tooling?
No
Closesapache#51083 from cxzl25/SPARK-52396.
Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
0 commit comments