Windows Compatibility Issue: DataFrame Materialization and File Permission Errors When Writing Parquet Files

[drockparashar]

Description

When running the SDK on Windows, DataFrame writes to Parquet files (using pandas/pyarrow) frequently fail due to file permission errors and file locking issues. This does not occur on Unix-like systems.

Reproduction Repo/Script (if any)

N/A (Issue occurs with any workflow in the SDK that writes DataFrames to Parquet on Windows)

Reproduction Steps

1. Import SDK
2. Call method that writes a DataFrame to Parquet (e.g., dataframe.to_parquet(...))
3. Observe error: PermissionError: [Errno 13] Permission denied or file locking issues

Logs / Tracebacks

PermissionError: [Errno 13] Permission denied: 'output_file.parquet'
# Or: FileNotFoundError, zero-byte files, access denied errors

Expected vs Actual

Expected:

• DataFrame writes to Parquet should succeed on Windows as they do on Unix-like systems.
• No file locking or permission errors should occur.

Actual:

• Frequent permission errors and file locking issues.
• Sometimes files are not created, are zero bytes, or require manual cleanup.

Environment

Windows

SDK Version

0.1.1rc38

[drockparashar]

Hi team,
Just to clarify, I initially posted this issue in the atlan-sample-apps repo by mistake, but realised it’s relevant to the application_sdk core.
If you need more details, logs, or a minimal reproduction, I’m happy to provide them.
I’m also willing to contribute a PR with a proposed fix if you’re open to it.

Thanks for your time and consideration!