bug(skore): EstimatorReport thinks skorch model is fitted but it is not

[divakaivan]

What would you like to say?

Context

show

I tried using a skorch model in a EstimatorReport

# env google colab: pip install -U skore skorch
import numpy as np 
from torch import nn
from skorch import NeuralNetClassifier

from skore import EstimatorReport

from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=1000, n_features=10, n_informative=4, n_classes=2)
X = X.astype(np.float32)

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

class MyModule(nn.Module):
    def __init__(self, num_units=10, nonlin=nn.ReLU()):
        super().__init__()

        self.dense0 = nn.Linear(10, num_units)
        self.nonlin = nonlin
        self.dropout = nn.Dropout(0.5)
        self.dense1 = nn.Linear(num_units, num_units)
        self.output = nn.Linear(num_units, 2)
        self.softmax = nn.Softmax(dim=-1)

    def forward(self, X, **kwargs):
        X = self.nonlin(self.dense0(X))
        X = self.dropout(X)
        X = self.nonlin(self.dense1(X))
        X = self.softmax(self.output(X))
        return X

net = NeuralNetClassifier(
    MyModule,
    max_epochs=10,
    lr=0.1,
    # Shuffle training data on each epoch
    iterator_train__shuffle=True,
)

report = EstimatorReport(
    net, X_train=X_train, y_train=y_train, X_test=X_test, y_test=y_test
)

The above code runs fine. However

report.metrics.summarize() # or accuracy, roc, etc

Produces:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[/usr/local/lib/python3.11/dist-packages/skorch/classifier.py](https://localhost:8080/#) in classes_(self)
    102         try:
--> 103             return self.classes_inferred_
    104         except AttributeError as exc:

AttributeError: 'NeuralNetClassifier' object has no attribute 'classes_inferred_'

The above exception was the direct cause of the following exception:

AttributeError                            Traceback (most recent call last)
6 frames
[/usr/local/lib/python3.11/dist-packages/skorch/classifier.py](https://localhost:8080/#) in classes_(self)
    113                 f"initializing {self.__class__.__name__}"
    114             )
--> 115             raise AttributeError(msg) from exc
    116 
    117     # pylint: disable=signature-differs

AttributeError: NeuralNetClassifier could not infer the classes from y; this error probably occurred because the net was trained without y and some function tried to access the '.classes_' attribute; a possible solution is to provide the 'classes' argument when initializing NeuralNetClassifier

Root cause

show

After some digging I think the EstimatorReport thinks the skorch net is fit but actually it's not.

In EstimatorReport (line 152), we check if the estimator is fitted

class EstimatorReport
    ...
    def __init__(self, ...):
        ...
        if fit == "auto":
            try:
                # An error is not raised when not fitted skorch model is passed
                check_is_fitted(estimator) # <---- HERE. 
                self._estimator = self._copy_estimator(estimator)
            except NotFittedError:
                self._estimator, fit_time = self._fit_estimator(
                    estimator, X_train, y_train
                )
        ...

This check_is_fitted comes from sklearn and calls _is_fitted

def check_is_fitted(...):
    ...
    if not _is_fitted(estimator, attributes, all_or_any):
        raise NotFittedError(msg % {"name": type(estimator).__name__})

_is_fitted goes to check this condition:

def _is_fitted(...):
    ...
    fitted_attrs = [
        v for v in vars(estimator) if v.endswith("_") and not v.startswith("__")
    ]
    return len(fitted_attrs) > 0

• If True then the model is fitted
• If False then the model is not fitted.

Said differently, when not fitted the above check returns an empty list which in turn raises an error from check_is_fitted

This check works fine for sklearn models. Example:

from sklearn.linear_model import Ridge 

unfit_ridge = Ridge()

from sklearn.utils.validation import check_is_fitted

check_is_fitted(unfit_ridge)  # raises an error

Under, the call to _is_fitted returns an empty list, which raises an error.

The error I got at the top of the issue suggested the model is not fit. And indeed it was not. Because in the example code at the top, before I pass net (the skorch model) to the EstimatorReport, if I fit it (net.fit(X_train, y_train)), then report.metrics.summarize() works ✅ (as net is fit) (alongside other report.metrics.<function>).

Issue: EstimatorReport's check_is_fitted returns True when it should be False

show

The EstimatorReport init method calls check_is_fitted, however here the output on a not fitted skorch model is not an empty list:

# check_is_fitted calls _is_fitted which checks if the list below is empty
# reminder: if the list is empty -> model is not fit
net_fitted_attrs = [v for v in vars(unfit_net) if v.endswith("_") and not v.startswith("__")]
# ['history_', 'initialized_', 'virtual_params_'] 

So this does not raise any errors:

unfit_net = NeuralNetClassifier(
    MyModule,
    max_epochs=10,
    lr=0.1,
    # Shuffle training data on each epoch
    iterator_train__shuffle=True,
)

check_is_fitted(unfit_net) # doesnt raise anything but it should

The list is not empty, so check_is_fitted returns True and the code up to but not including report.metrics.summarize() runs fine.

Suggestion

show

I found that if we check that the model is a skorch model, we should use this to check if the model is fitted:

check_is_fitted(net, attributes=[
    'init_context_',
    'callbacks_',
    'prefixes_',
    'cuda_dependent_attributes_',
    'module_',
    'criterion_',
    'optimizer_',
    'classes_inferred_'
])

as these are the attributes that appear in the skorch model after it is fitted.

Example:

unfit_net = NeuralNetClassifier(
    MyModule,
    max_epochs=10,
    lr=0.1,
    # Shuffle training data on each epoch
    iterator_train__shuffle=True,
)

check_is_fitted(unfit_net, attributes=['init_context_', 'callbacks_', 'prefixes_', 'cuda_dependent_attributes_', 'module_', 'criterion_', 'optimizer_', 'classes_inferred_'])

Results in:

---------------------------------------------------------------------------
NotFittedError                            Traceback (most recent call last)
/tmp/ipython-input-1381609148.py in <cell line: 0>()
      9 )
     10 
---> 11 check_is_fitted(net, attributes=[
     12     'init_context_',
     13     'callbacks_',

/usr/local/lib/python3.11/dist-packages/sklearn/utils/validation.py in check_is_fitted(estimator, attributes, msg, all_or_any)
   1755 
   1756     if not _is_fitted(estimator, attributes, all_or_any):
-> 1757         raise NotFittedError(msg % {"name": type(estimator).__name__})
   1758 
   1759 

NotFittedError: This NeuralNetClassifier instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.

---

Google colab with code