How atomic is renameFile?

[Gabriella439]

Is System.Directory.renameFile supposed to safely handle two parallel renames that share the same destination?

The documentation says:

If the new object already exists, it is atomically replaced by the old object

... but it doesn't clarify what should happen if two concurrent processes rename two files to the same destination for the first time.

The context for this is: dhall-lang/dhall-haskell#1904

What we ran into is that we were using the atomic-write package and in the course of doing so ran into a situation where two parallel writes attempted to write to the same destination file at the same time. Under the hood, each write creates a temporary file and uses renameFile to move the temporarily file to the final destination. If they rename to the same destination in parallel we see a conflict on Windows. On Linux things work without problems.

[Rufflewind]

This is a fairly nuanced issue on Windows: https://stackoverflow.com/q/167414

According to the Niall Douglas' explanation in "Racing the File System" at CppCon and Doug E. Cook's comment on MSDN forums, MoveFileEx uses multiple strategies to move (and replace) a file. The first strategy attempted is atomic, but if it fails it will fall back to copy-and-delete. So if it succeeds, then you know it worked. But if it fails, I don't know for sure whether the destination file might be corrupted. Other than permission denied errors, have you seen instances where the file is corrupted?

Docs

I think the documentation could be clarified by weakening the guarantees of this function on Windows, as the atomicity of this function seems unclear on that platform.

Alternative 1

"Racing the File System" notes that there is a more exotic, NTFS-specific alternative:

SetFileInformationByHandle(FILE_RENAME_INFO { ReplaceIfExists: TRUE })

However, I don't know if this might break other use cases, since users might be unintentionally relying on the fallback behavior when moving files from one volume to another, so this is unlikely to work as a general purpose replacement for rename.

To allow an open file to be replaced, the FILE_RENAME_FLAG_POSIX_SEMANTICS flag (Windows 10 V1607+) would also need to be added.

(This approach is not used in the standard libraries of other languages, e.g. both Python and Rust use the more conventional MoveFileEx approach.)

Alternative 2

There is an older API ReplaceFileW that has better guarantees and is more in line with what Dhall is trying to do. However, it is not suitable as a general purpose rename function because it expects the target file to already exist.

[Gabriella439]

@Rufflewind: I think it's fine if the solution is to weaken the guarantee on Windows

[soulomoon]

What is the status of this? HLS depend on it too.