LaTeX: improve the table column widths computed by LaTeX itself

Fix #3447
Fix #8228

With this patch the legacy notion of "problematic cell" is almost
entirely removed, and LaTeX package tabulary will be used more often so
that column widths in tables may get better fit to actual contents.

This works well when the column contains cells with multiple paragraphs,
bullet lists, enumerated lists, or line blocks.  For cells with object
descriptions or admonitions, the column may not fit tightly to the
contents, but other columns will probably be improved.

Author: jfbu

CHANGES.rst

@@ -62,6 +62,12 @@ Features added
 Bugs fixed
 ----------
 
+* #3447: LaTeX: when assigning longtable class to table for PDF, it may render
+  "horizontally" and overflow in right margin.
+  Patch by Jean-François B.
+* #8828: LaTeX: adding a footnote to a longtable cell causes table to occupy
+  full width.
+  Patch by Jean-François B.
 * #11498: LaTeX: Table in cell fails to build if it has many rows.
   Patch by Jean-François B.
 * #11515: LaTeX: longtable does not allow nested table.

doc/usage/restructuredtext/directives.rst

@@ -1483,60 +1483,72 @@ Check the :confval:`latex_table_style`.
 
 .. rst:directive:: .. tabularcolumns:: column spec
 
-   This directive influences only the LaTeX output for the next table in
-   source.  The mandatory argument is a column specification (known as an
-   "alignment preamble" in LaTeX idiom).  Please refer to a LaTeX
+   This directive influences only the LaTeX output, and only for the next
+   table in source.  The mandatory argument is a column specification (known
+   as an "alignment preamble" in LaTeX idiom).  Please refer to a LaTeX
    documentation, such as the `wiki page`_, for basics of such a column
    specification.
 
    .. _wiki page: https://en.wikibooks.org/wiki/LaTeX/Tables
 
    .. versionadded:: 0.3
 
-   .. note::
+   Sphinx renders tables with at most 30 rows using ``tabulary`` (or
+   ``tabular`` if at least one cell contains either a code-block or a nested
+   table), and those with more rows with ``longtable``.  The advantage of
+   using ``tabulary`` is that it tries to compute automatically (internally to
+   LaTeX) suitable column widths.
+
+   :rst:dir:`tabularcolumns` can serve to provide one's own "colspec" choice.
+   Here is an advanced example:
+
+   .. code-block:: latex
 
-      :rst:dir:`tabularcolumns` conflicts with ``:widths:`` option of table
-      directives.  If both are specified, ``:widths:`` option will be ignored.
+      .. tabularcolumns:: >{\raggedright}\Y{.4}>{\centering}\Y{.1}>{\sphinxcolorblend{!95!red}\centering\noindent\bfseries\color{red}}\Y{.12}>{\raggedright\arraybackslash}\Y{.38}
 
-   Sphinx renders tables with at most 30 rows using ``tabulary``, and those
-   with more rows with ``longtable``.
+   This is used in Sphinx own PDF docs at :ref:`dev-deprecated-apis`.
+   Regarding column widths, this "colspec" achieves the same as would
+   ``:widths:`` option set to ``40 10 12 38`` but it injects extra effects.
 
-   ``tabulary`` tries to compute automatically (internally to LaTeX) suitable
-   column widths.  However, cells are then not allowed to contain
-   "problematic" elements such as lists, object descriptions,
-   blockquotes... Sphinx will fall back to using ``tabular`` if such a cell is
-   encountered (or a nested ``tabulary``).  In such a case the table will have
-   a tendency to try to fill the whole available line width.
+   .. note::
 
-   :rst:dir:`tabularcolumns` can help in coercing the usage of ``tabulary`` if
-   one is careful to not employ the ``tabulary`` column types (``L``, ``R``,
-   ``C`` or ``J``) for those columns with at least one "problematic" cell, but
-   only LaTeX's ``p{<width>}`` or Sphinx ``\X`` and ``\Y`` (described next).
+      In case both :rst:dir:`tabularcolumns` and ``:widths:`` option of table
+      directives are used, ``:widths:`` option will be ignored by the LaTeX
+      builder.  Of course it is obeyed by other builders.
 
-   Literal blocks do not work at all with ``tabulary``.  Sphinx will fall back
-   to ``tabular`` or ``longtable`` environments depending on the number of
-   rows.  It will employ the :rst:dir:`tabularcolumns` specification only if it
-   contains no usage of the ``tabulary`` specific types.
+   Literal blocks do not work at all with ``tabulary`` and Sphinx will then
+   fall back to ``tabular`` LaTeX environment.  It will employ the
+   :rst:dir:`tabularcolumns` specification in that case only if it contains no
+   usage of the ``tabulary`` specific column types (which are ``L``, ``R``,
+   ``C`` and ``J``).
 
    Besides the LaTeX ``l``, ``r``, ``c`` and ``p{width}`` column specifiers,
-   one can also use ``\X{a}{b}`` which configures the column width to be a
-   fraction ``a/b`` of the total line width and ``\Y{f}`` where ``f`` is a
-   decimal: for example ``\Y{0.2}`` means that the column will occupy ``0.2``
-   times the line width.
+   and the ``tabulary`` specific ``L``, ``R``, ``C`` and ``J``, one can also
+   use (with all table types) ``\X{a}{b}`` which configures the column width
+   to be a fraction ``a/b`` of the total line width and ``\Y{f}`` where ``f``
+   is a decimal: for example ``\Y{0.2}`` means that the column will occupy
+   ``0.2`` times the line width.
 
 .. versionchanged:: 1.6
 
-   Use ``J`` (justified) by default with ``tabulary``, not ``L``
+   Sphinx uses ``J`` (justified) by default with ``tabulary``, not ``L``
    (flushed-left).  To revert, include ``\newcolumntype{T}{L}`` in the LaTeX
    preamble, as in fact Sphinx uses ``T`` and sets it by default to be an
    alias of ``J``.
 
-.. hint::
+.. versionchanged:: 8.3.0
 
-   A frequent issue with ``tabulary`` is that columns with little contents
-   appear to be "squeezed".  One can add to the LaTeX preamble for example
-   ``\setlength{\tymin}{40pt}`` to ensure a minimal column width of ``40pt``,
-   the ``tabulary`` default of ``10pt`` being too small.
+   Formerly, Sphinx did not use ``tabulary`` if the table had at least one
+   cell containing "problematic" elements such as lists, object descriptions,
+   blockquotes (etc...) because such contents are not out-of-the-box
+   compatible with ``tabulary``.  At ``8.3.0`` a technique, which was already
+   in use for merged cells, was extended to such cases, and the sole
+   "problematic" contents are code-blocks and nested tables.  So tables
+   containing (only) cells with mutliple paragraphs, bullet or enumerated
+   lists, or line blocks, will now better fit to their contents (if not
+   rendered by ``longtable``).  Cells with object descriptions or admonitions
+   will still have a tendency to induce the table to fill the full text area
+   width, but columns in that table with no such contents will be tighter.
 
 .. hint::
 

sphinx/texinputs/sphinxlatextables.sty

@@ -1,7 +1,7 @@
 %% TABLES (WITH SUPPORT FOR MERGED CELLS OF GENERAL CONTENTS)
 %
 % change this info string if making any custom modification
-\ProvidesPackage{sphinxlatextables}[2025/06/09 v8.3.0 tables]%
+\ProvidesPackage{sphinxlatextables}[2025/06/30 v8.3.0 tables]%
 
 % Provides support for this output mark-up from Sphinx latex writer
 % and table templates:
@@ -42,6 +42,13 @@
 % - \sphinxthistablewithnocolorrowsstyle
 % - \sphinxthistablewithvlinesstyle
 % - \sphinxthistablewithnovlinesstyle
+% - \sphinxbeforeendvarwidth
+
+% At 8.3.0, ALL table cell contents are wrapped into a varwidth environment.
+% This helps solve issues such as #3447, #8828, and helps use tabulary
+% in many more cases hence obtain better looking tables.
+\def\sphinxbeforeendvarwidth{\par\vskip-\baselineskip\vbox{\hbox{\strut}}}
+
 % These conditionals added at 8.3.0 for nested tables not to break row colors
 % (#13635).  Nested tables are only partially supported by Sphinx LaTeX.
 % The method here is with no changes to neither writer nor templates.
@@ -251,7 +258,29 @@
 %
 % configuration of tabulary
 \setlength{\tymin}{3\fontcharwd\font`0 }% minimal width of "squeezed" columns
-\setlength{\tymax}{10000pt}% allow enough room for paragraphs to "compete"
+\setlength{\tymax}{2\textwidth}% allow enough room for paragraphs to "compete"
+%
+% MEMO: tabulary initially renders cell contents "horizontally" to measure
+%       them and compare their relative importance.  Its goal is to choose the
+%       column width so that, roughly, all columns will look about evenly
+%       filled.  "Horizontal" rendering is incompatible with many LaTeX
+%       structures such as lists, so prior to Sphinx 8.3.0 cells with such
+%       "problematic" contents caused Sphinx to use tabular not tabulary; the
+%       tabular would then render each column in absence of :widths: option or
+%       tabularcolumns directive the same width equal to available text width
+%       divided by number of columns.  At 8.3.0, "problematic" contents is
+%       wrapped into a "varwidth" environment, as was already done formerly
+%       for merged cells, and this avoids tabulary causing errors such as
+%       "incompatible with LR mode"; \sphinxcolwidth is used which sets
+%       the initial horizontal width for "varwidth". In the first tabulary
+%       pass, \sphinxcolwidth is configured (by us) to use \tymax.
+%
+%       During testing, it was determined that our former 10000pt setting for
+%       \tymax could cause "Dimension too large" TeX error if two columns or
+%       more contained such cells.  So we use now 2\textwidth which is more
+%       than 10 times smaller but proves large enough for the tabulary
+%       algorithm to provide reasonable results.
+%
 % we need access to tabulary's final computed width. \@tempdima is too volatile
 % to hope it has kept tabulary's value when \sphinxcolwidth needs it.
 \newdimen\sphinx@TY@tablewidth
@@ -302,9 +331,7 @@
 % ****            TODO: clarify if next paragraph means we must raise an
 % ****            if LaTeX writer detects a merged cell inside nested table.
 % MEMO about nesting: if sphinxmulticolumn is encountered in a nested tabular
-% inside a tabulary it will think to be at top level in the tabulary. But
-% Sphinx generates no nested tables, and if some LaTeX macro uses internally a
-% tabular this will not have a \sphinxstartmulticolumn within it!
+% inside a tabulary it will think to be at top level in the tabulary.
 %
 % 5.3.0 adds a check for multirow as single-row multi-column will allow a row
 % colour but multi-row multi-column should not.
@@ -407,8 +434,7 @@
 }%
 \newcommand*\sphinxcolwidth[2]{%
   % this dimension will always be used for varwidth, and serves as maximum
-  % width when cells are merged either via multirow or multicolumn or both,
-  % as always their contents is wrapped in varwidth environment.
+  % width for cells whose contents are wrapped in varwidth environment.
   \ifnum#1>\@ne % multi-column (and possibly also multi-row)
   % we wrote our own multicolumn code especially to handle that (and allow
   % verbatim contents)