[3.14] gh-134939: Add the concurrent.interpreters Module (gh-135414)

PEP-734 has been accepted (for 3.14).

(FTR, I'm opposed to putting this under the concurrent package, but
doing so is the SC condition under which the module can land in 3.14.)

(cherry picked from commit 6214373, AKA gh-133958)

Author: ericsnowcurrently

.github/CODEOWNERS

@@ -281,9 +281,13 @@ Doc/howto/clinic.rst          @erlend-aasland
 # Subinterpreters
 **/*interpreteridobject.*     @ericsnowcurrently
 **/*crossinterp*              @ericsnowcurrently
-Lib/test/support/interpreters/  @ericsnowcurrently
 Modules/_interp*module.c      @ericsnowcurrently
+Lib/test/test__interp*.py     @ericsnowcurrently
+Lib/concurrent/interpreters/  @ericsnowcurrently
+Lib/test/support/channels.py  @ericsnowcurrently
+Doc/library/concurrent.interpreters.rst  @ericsnowcurrently
 Lib/test/test_interpreters/   @ericsnowcurrently
+Lib/concurrent/futures/interpreter.py  @ericsnowcurrently
 
 # Android
 **/*Android*                  @mhsmith @freakboy3742

Doc/library/concurrency.rst

@@ -18,6 +18,7 @@ multitasking). Here's an overview:
    multiprocessing.shared_memory.rst
    concurrent.rst
    concurrent.futures.rst
+   concurrent.interpreters.rst
    subprocess.rst
    sched.rst
    queue.rst

Doc/library/concurrent.interpreters.rst

@@ -0,0 +1,198 @@
+:mod:`!concurrent.interpreters` --- Multiple interpreters in the same process
+=============================================================================
+
+.. module:: concurrent.interpreters
+   :synopsis: Multiple interpreters in the same process
+
+.. moduleauthor:: Eric Snow <ericsnowcurrently@gmail.com>
+.. sectionauthor:: Eric Snow <ericsnowcurrently@gmail.com>
+
+.. versionadded:: 3.14
+
+**Source code:** :source:`Lib/concurrent/interpreters.py`
+
+--------------
+
+
+Introduction
+------------
+
+The :mod:`!concurrent.interpreters` module constructs higher-level
+interfaces on top of the lower level :mod:`!_interpreters` module.
+
+.. XXX Add references to the upcoming HOWTO docs in the seealso block.
+
+.. seealso::
+
+   :ref:`isolating-extensions-howto`
+       how to update an extension module to support multiple interpreters
+
+   :pep:`554`
+
+   :pep:`734`
+
+   :pep:`684`
+
+.. XXX Why do we disallow multiple interpreters on WASM?
+
+.. include:: ../includes/wasm-notavail.rst
+
+
+Key details
+-----------
+
+Before we dive into examples, there are a small number of details
+to keep in mind about using multiple interpreters:
+
+* isolated, by default
+* no implicit threads
+* not all PyPI packages support use in multiple interpreters yet
+
+.. XXX Are there other relevant details to list?
+
+In the context of multiple interpreters, "isolated" means that
+different interpreters do not share any state.  In practice, there is some
+process-global data they all share, but that is managed by the runtime.
+
+
+Reference
+---------
+
+This module defines the following functions:
+
+.. function:: list_all()
+
+   Return a :class:`list` of :class:`Interpreter` objects,
+   one for each existing interpreter.
+
+.. function:: get_current()
+
+   Return an :class:`Interpreter` object for the currently running
+   interpreter.
+
+.. function:: get_main()
+
+   Return an :class:`Interpreter` object for the main interpreter.
+
+.. function:: create()
+
+   Initialize a new (idle) Python interpreter
+   and return a :class:`Interpreter` object for it.
+
+
+Interpreter objects
+^^^^^^^^^^^^^^^^^^^
+
+.. class:: Interpreter(id)
+
+   A single interpreter in the current process.
+
+   Generally, :class:`Interpreter` shouldn't be called directly.
+   Instead, use :func:`create` or one of the other module functions.
+
+   .. attribute:: id
+
+      (read-only)
+
+      The interpreter's ID.
+
+   .. attribute:: whence
+
+      (read-only)
+
+      A string describing where the interpreter came from.
+
+   .. method:: is_running()
+
+      Return ``True`` if the interpreter is currently executing code
+      in its :mod:`!__main__` module and ``False`` otherwise.
+
+   .. method:: close()
+
+      Finalize and destroy the interpreter.
+
+   .. method:: prepare_main(ns=None, **kwargs)
+
+      Bind "shareable" objects in the interpreter's
+      :mod:`!__main__` module.
+
+   .. method:: exec(code, /, dedent=True)
+
+      Run the given source code in the interpreter (in the current thread).
+
+   .. method:: call(callable, /, *args, **kwargs)
+
+      Return the result of calling running the given function in the
+      interpreter (in the current thread).
+
+   .. method:: call_in_thread(callable, /, *args, **kwargs)
+
+      Run the given function in the interpreter (in a new thread).
+
+Exceptions
+^^^^^^^^^^
+
+.. exception:: InterpreterError
+
+   This exception, a subclass of :exc:`Exception`, is raised when
+   an interpreter-related error happens.
+
+.. exception:: InterpreterNotFoundError
+
+   This exception, a subclass of :exc:`InterpreterError`, is raised when
+   the targeted interpreter no longer exists.
+
+.. exception:: ExecutionFailed
+
+   This exception, a subclass of :exc:`InterpreterError`, is raised when
+   the running code raised an uncaught exception.
+
+   .. attribute:: excinfo
+
+      A basic snapshot of the exception raised in the other interpreter.
+
+.. XXX Document the excinfoattrs?
+
+.. exception:: NotShareableError
+
+   This exception, a subclass of :exc:`TypeError`, is raised when
+   an object cannot be sent to another interpreter.
+
+
+.. XXX Add functions for communicating between interpreters.
+
+
+Basic usage
+-----------
+
+Creating an interpreter and running code in it::
+
+    from concurrent import interpreters
+
+    interp = interpreters.create()
+
+    # Run in the current OS thread.
+
+    interp.exec('print("spam!")')
+
+    interp.exec("""if True:
+        print('spam!')
+        """)
+
+    from textwrap import dedent
+    interp.exec(dedent("""
+        print('spam!')
+        """))
+
+    def run():
+        print('spam!')
+
+    interp.call(run)
+
+    # Run in new OS thread.
+
+    t = interp.call_in_thread(run)
+    t.join()
+
+
+.. XXX Explain about object "sharing".

Doc/library/concurrent.rst

@@ -1,6 +1,7 @@
 The :mod:`!concurrent` package
 ==============================
 
-Currently, there is only one module in this package:
+This package contains the following modules:
 
 * :mod:`concurrent.futures` -- Launching parallel tasks
+* :mod:`concurrent.interpreters` -- Multiple interpreters in the same process

Doc/library/python.rst

@@ -27,3 +27,8 @@ overview:
    inspect.rst
    annotationlib.rst
    site.rst
+
+.. seealso::
+
+   * See the :mod:`concurrent.interpreters` module, which similarly
+     exposes core runtime functionality.

Doc/whatsnew/3.14.rst

@@ -83,6 +83,7 @@ and improvements in user-friendliness and correctness.
 .. PEP-sized items next.
 
 * :ref:`PEP 649 and 749: deferred evaluation of annotations <whatsnew314-pep649>`
+* :ref:`PEP 734: Multiple Interpreters in the Stdlib <whatsnew314-pep734>`
 * :ref:`PEP 741: Python Configuration C API <whatsnew314-pep741>`
 * :ref:`PEP 750: Template strings <whatsnew314-pep750>`
 * :ref:`PEP 758: Allow except and except* expressions without parentheses <whatsnew314-pep758>`
@@ -123,6 +124,101 @@ of Python.  See :ref:`below <whatsnew314-refcount>` for details.
 New features
 ============
 
+.. _whatsnew314-pep734:
+
+PEP 734: Multiple Interpreters in the Stdlib
+--------------------------------------------
+
+The CPython runtime supports running multiple copies of Python in the
+same process simultaneously and has done so for over 20 years.
+Each of these separate copies is called an "interpreter".
+However, the feature had been available only through the C-API.
+
+That limitation is removed in the 3.14 release,
+with the new :mod:`concurrent.interpreters` module.
+
+There are at least two notable reasons why using multiple interpreters
+is worth considering:
+
+* they support a new (to Python), human-friendly concurrency model
+* true multi-core parallelism
+
+For some use cases, concurrency in software enables efficiency and
+can simplify software, at a high level.  At the same time, implementing
+and maintaining all but the simplest concurrency is often a struggle
+for the human brain.  That especially applies to plain threads
+(for example, :mod:`threading`), where all memory is shared between all threads.
+
+With multiple isolated interpreters, you can take advantage of a class
+of concurrency models, like CSP or the actor model, that have found
+success in other programming languages, like Smalltalk, Erlang,
+Haskell, and Go.  Think of multiple interpreters like threads
+but with opt-in sharing.
+
+Regarding multi-core parallelism: as of the 3.12 release, interpreters
+are now sufficiently isolated from one another to be used in parallel.
+(See :pep:`684`.)  This unlocks a variety of CPU-intensive use cases
+for Python that were limited by the :term:`GIL`.
+
+Using multiple interpreters is similar in many ways to
+:mod:`multiprocessing`, in that they both provide isolated logical
+"processes" that can run in parallel, with no sharing by default.
+However, when using multiple interpreters, an application will use
+fewer system resources and will operate more efficiently (since it
+stays within the same process).  Think of multiple interpreters as
+having the isolation of processes with the efficiency of threads.
+
+.. XXX Add an example or two.
+.. XXX Link to the not-yet-added HOWTO doc.
+
+While the feature has been around for decades, multiple interpreters
+have not been used widely, due to low awareness and the lack of a stdlib
+module.  Consequently, they currently have several notable limitations,
+which will improve significantly now that the feature is finally
+going mainstream.
+
+Current limitations:
+
+* starting each interpreter has not been optimized yet
+* each interpreter uses more memory than necessary
+  (we will be working next on extensive internal sharing between
+  interpreters)
+* there aren't many options *yet* for truly sharing objects or other
+  data between interpreters (other than :type:`memoryview`)
+* many extension modules on PyPI are not compatible with multiple
+  interpreters yet (stdlib extension modules *are* compatible)
+* the approach to writing applications that use multiple isolated
+  interpreters is mostly unfamiliar to Python users, for now
+
+The impact of these limitations will depend on future CPython
+improvements, how interpreters are used, and what the community solves
+through PyPI packages.  Depending on the use case, the limitations may
+not have much impact, so try it out!
+
+Furthermore, future CPython releases will reduce or eliminate overhead
+and provide utilities that are less appropriate on PyPI.  In the
+meantime, most of the limitations can also be addressed through
+extension modules, meaning PyPI packages can fill any gap for 3.14, and
+even back to 3.12 where interpreters were finally properly isolated and
+stopped sharing the :term:`GIL`.  Likewise, we expect to slowly see
+libraries on PyPI for high-level abstractions on top of interpreters.
+
+Regarding extension modules, work is in progress to update some PyPI
+projects, as well as tools like Cython, pybind11, nanobind, and PyO3.
+The steps for isolating an extension module are found at
+:ref:`isolating-extensions-howto`.  Isolating a module has a lot of
+overlap with what is required to support
+:ref:`free-threading <whatsnew314-free-threaded-cpython>`,
+so the ongoing work in the community in that area will help accelerate
+support for multiple interpreters.
+
+Also added in 3.14: :ref:`concurrent.futures.InterpreterPoolExecutor
+<whatsnew314-concurrent-futures-interp-pool>`.
+
+.. seealso::
+   :pep:`734`.
+
+
 .. _whatsnew314-pep750:
 
 PEP 750: Template strings
@@ -1109,6 +1205,8 @@ calendar
 concurrent.futures
 ------------------
 
+.. _whatsnew314-concurrent-futures-interp-pool:
+
 * Add :class:`~concurrent.futures.InterpreterPoolExecutor`,
   which exposes "subinterpreters" (multiple Python interpreters in the
   same process) to Python code.  This is separate from the proposed API

Lib/concurrent/futures/interpreter.py

@@ -167,7 +167,7 @@ def run(self, task):
             except _interpqueues.QueueError:
                 continue
             except ModuleNotFoundError:
-                # interpreters.queues doesn't exist, which means
+                # interpreters._queues doesn't exist, which means
                 # QueueEmpty doesn't.  Act as though it does.
                 continue
             else: