Race conditions when creating output directory

[mgorny]

Subject: Sphinx fails with EEXIST exception when multiple instances attempt to simultaneously create output directory

Problem

We run Sphinx via cmake to create a few manpages in 'docs/man' directory. If parallel make (ninja) happens to run multiple Sphinx instances in parallel, it sometimes happens that they try to simultaneously create the output directory and fail.

Procedure to reproduce the problem

Attempt to build LLVM from https://git.llvm.org/git/llvm.git; preferably with ccache and large -j value. Try as many times as you need until you hit it.

Error logs / results

[1740/2948] cd /var/tmp/portage/sys-devel/llvm-9999/work/llvm-9999-abi_x86_32.x86/docs && /usr/bin/sphinx-build -b man -d /var/tmp/portage/sys-devel/llvm-9999/work/llvm-9999-abi_x86_32.x86/docs/_doctrees-llvm-dwarfdump-man -q /var/tmp/portage/sys-devel/llvm-9999/work/llvm-9999/docs /var/tmp/portage/sys-devel/llvm-9999/work/llvm-9999-abi_x86_32.x86/docs/man
FAILED: docs/CMakeFiles/docs-llvm-dwarfdump-man 
cd /var/tmp/portage/sys-devel/llvm-9999/work/llvm-9999-abi_x86_32.x86/docs && /usr/bin/sphinx-build -b man -d /var/tmp/portage/sys-devel/llvm-9999/work/llvm-9999-abi_x86_32.x86/docs/_doctrees-llvm-dwarfdump-man -q /var/tmp/portage/sys-devel/llvm-9999/work/llvm-9999/docs /var/tmp/portage/sys-devel/llvm-9999/work/llvm-9999-abi_x86_32.x86/docs/man

Exception occurred:
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/var/tmp/portage/sys-devel/llvm-9999/work/llvm-9999-abi_x86_32.x86/docs/man'
The full traceback has been saved in /var/tmp/portage/sys-devel/llvm-9999/temp/sphinx-err-lMuLFm.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!

Traceback:

# Sphinx version: 1.6.3
# Python version: 2.7.14 (CPython)
# Docutils version: 0.12 release
# Jinja2 version: 2.9.5
# Last messages:

# Loaded extensions:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/sphinx/cmdline.py", line 305, in main
    opts.warningiserror, opts.tags, opts.verbosity, opts.jobs)
  File "/usr/lib/python2.7/site-packages/sphinx/application.py", line 163, in __init__
    os.makedirs(outdir)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/var/tmp/portage/sys-devel/llvm-9999/work/llvm-9999-abi_x86_32.x86/docs/man'

Expected results

It should not fail.

Reproducible project / your project

• https://llvm.org

Environment info

• OS: Gentoo Linux
• Python version: 2.7.14
• Sphinx version: 1.6.3 (but git master is broken as well)

[tk0miya]

#4282 is merged now :-)