Skip to content

Using PermutationImportance with Sklearn RFECV #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
apptimise opened this issue Jun 24, 2020 · 1 comment
Open

Using PermutationImportance with Sklearn RFECV #383

apptimise opened this issue Jun 24, 2020 · 1 comment

Comments

@apptimise
Copy link

apptimise commented Jun 24, 2020

I would like to use RFECV scores to be able to compare the R2 scores for different numbers of features selected based on PermutationImportance importances. My solution is as follows:

from eli5.sklearn import PermutationImportance
from sklearn.datasets import make_friedman1
from sklearn.feature_selection import RFECV
from sklearn.model_selection import KFold
from sklearn.svm import SVR

X, y = make_friedman1(n_samples=50, n_features=10, random_state=0)

splitter = KFold(n_splits=3) 

estimator = SVR(kernel="linear")
selector = RFECV(
    PermutationImportance(estimator,  scoring='r2', n_iter=10, random_state=42, cv=splitter),
    cv=splitter,
    scoring='r2',
    step=1
)
selector = selector.fit(X, y)
print(selector.grid_scores_)
  1. My understanding is that RFECV creates 3 folds and then PermutationImportance also creates 3 folds within each fold. Is that the case? The ideal situation is that PermutationImportance uses the same split as provided by RFECV, fits on the train data and provides the results for the validation set. Or maybe this is exactly what is happening in this case. Can someone confirm this, please?

  2. Also, do I need to worry about the refit parameter? I'm not exactly sure what it does

@apptimise apptimise changed the title Using Permutation Feature with Sklearn RFECV Using PermutationImportance with Sklearn RFECV Jun 24, 2020
@t10-13rocket
Copy link

Can it be hidden in plain sight?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants