ENH Allows multiclass target in TargetEncoder (scikit-learn#26674)

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>

Author: lucyleeow

doc/modules/preprocessing.rst

@@ -903,6 +903,19 @@ computed as an empirical Bayes estimate: :math:`m=\sigma_i^2/\tau^2`, where
 :math:`\sigma_i^2` is the variance of `y` with category :math:`i` and
 :math:`\tau^2` is the global variance of `y`.
 
+For multiclass classification targets, the formulation is similar to binary
+classification:
+
+.. math::
+    S_{ij} = \lambda_i\frac{n_{iY_j}}{n_i} + (1 - \lambda_i)\frac{n_{Y_j}}{n}
+
+where :math:`S_{ij}` is the encoding for category :math:`i` and class :math:`j`,
+:math:`n_{iY_j}` is the number of observations with :math:`Y=j` and category
+:math:`i`, :math:`n_i` is the number of observations with category :math:`i`,
+:math:`n_{Y_j}` is the number of observations with :math:`Y=j`, :math:`n` is the
+number of observations, and :math:`\lambda_i` is a shrinkage factor for category
+:math:`i`.
+
 For continuous targets, the formulation is similar to binary classification:
 
 .. math::

doc/whats_new/v1.4.rst

@@ -178,6 +178,12 @@ Changelog
   to :ref:`metadata routing user guide <metadata_routing>`. :pr:`26789` by
   `Adrin Jalali`_.
 
+:mod:`sklearn.preprocessing`
+............................
+
+- |Enhancement| :class:`preprocessing.TargetEncoder` now supports `target_type`
+  'multiclass'. :pr:`26674` by :user:`Lucy Liu <lucyleeow>`.
+
 :mod:`sklearn.model_selection`
 ..............................