sphinx-apidoc duplicate lines in genrated rst files if module is a cpython extension module

[twmr]

Describe the bug

sphinx-apidoc -M -o modules/pkg pkg generates files that contain duplicate entries if pkg.wrapper is an cpython extension module (.so file). The generated rst file looks like

pkg package
===========

.. automodule:: pkg
   :members:
   :show-inheritance:

Submodules
----------


pkg.lockable\_plugin module
---------------------------

.. automodule:: pkg.lockable_plugin
   :members:
   :show-inheritance:

pkg.wrapper module
------------------

.. automodule:: pkg.wrapper
   :members:
   :show-inheritance:

pkg.wrapper module
------------------

.. automodule:: pkg.wrapper
   :members:
   :show-inheritance:

How to Reproduce

Build a python library that contains a cpython extension using multiple interpreters, s.t. the source tree of the python lib contains so files for multiple interpreters

./pkg/wrapper.cpython-39-x86_64-linux-gnu.so
./pkg/wrapper.cpython-37m-x86_64-linux-gnu.so

(./pkg is the source directory)

Expected behavior

The generated rst file must not contain duplicate entries.

Your project

Screenshots

No response

OS

Linux

Python version

3.7

Sphinx version

4.1.2

Sphinx extensions

autodoc

Extra tools

No response

Additional context

FYI I've reported a similar (apidoc related) issue in the past: #6986

[fduxiao]

Hi @Thisch. I was facing the same problem, so I checked out the apidoc command. It is just a simple python script. I found that the script will look into every file with suffix .py .pyx .so, etc, as mentioned here. That means apidoc will generate corresponding autodoc for the .pyx files as well as the .so extension aside. You can change the name of the pyx files while keeping the name of the extension, recompile the doc and see what will happen. My solution is simply ignoring the *.so files by excluding pattern, or maybe you want to write a code generator yourself to avoid this.

[twmr]

Thx for the suggestion @fduxiao. Since sphinx-apidoc doesn't seem to be so complicated, I'll consider fixing the issue upstream when I have time.

For the current issue, I guess we can simply check if there are duplicate entries in the data-structure that is written to a rst file.
Whoever reads this - feel free to beat me creating a PR for that ;)