-
Notifications
You must be signed in to change notification settings - Fork 182
n_jobs
support details in docs
#2453
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Flags with carried forward coverage won't be shown. Click here to find out more. see 41 files with indirect coverage changes 🚀 New features to boost your workflow:
|
@@ -0,0 +1,46 @@ | |||
.. Copyright 2025 Intel Corporation |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're using a different copyright header now: https://github.com/uxlfoundation/oneDAL/blob/main/CONTRIBUTING.md#license-and-copyright
* `n_jobs` parameter is supported for all estimators patched by |sklearnex|, | ||
while |sklearn| enables it for selected estimators only | ||
* `n_jobs` estimator parameter sets the number of threads used by the underlying |oneDAL| | ||
* |sklearnex| doesn't use `joblib` for parallelism in patched estimators and functions | ||
* The only low-level parallelism library used by |sklearnex| is oneTBB (through oneDAL) | ||
* The `threading` parallel backend of `joblib` is not supported by |sklearnex| |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
* `n_jobs` parameter is supported for all estimators patched by |sklearnex|, | |
while |sklearn| enables it for selected estimators only | |
* `n_jobs` estimator parameter sets the number of threads used by the underlying |oneDAL| | |
* |sklearnex| doesn't use `joblib` for parallelism in patched estimators and functions | |
* The only low-level parallelism library used by |sklearnex| is oneTBB (through oneDAL) | |
* The `threading` parallel backend of `joblib` is not supported by |sklearnex| | |
* `n_jobs` parameter is supported for all estimators patched by |sklearnex|, | |
while |sklearn| enables it for selected estimators only. | |
* `n_jobs` estimator parameter sets the number of threads used by the underlying |oneDAL|. | |
* |sklearnex| doesn't use `joblib` for parallelism in patched estimators and functions. | |
* The only low-level parallelism library used by |sklearnex| is oneTBB (through oneDAL). | |
* The `threading` parallel backend of `joblib` is not supported by |sklearnex|. |
* The only low-level parallelism library used by |sklearnex| is oneTBB (through oneDAL) | ||
* The `threading` parallel backend of `joblib` is not supported by |sklearnex| | ||
|
||
The only exception is multiclass LogisticRegression, which uses `joblib` for parallelism across classes. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only exception is multiclass LogisticRegression, which uses `joblib` for parallelism across classes. | |
The only exception is multiclass LogisticRegression, which uses :external+joblib:doc:`joblib <index>` for parallelism across classes. |
(perhaps it could be added in the substitutions list)
while |sklearn| enables it for selected estimators only | ||
* `n_jobs` estimator parameter sets the number of threads used by the underlying |oneDAL| | ||
* |sklearnex| doesn't use `joblib` for parallelism in patched estimators and functions | ||
* The only low-level parallelism library used by |sklearnex| is oneTBB (through oneDAL) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's also multi-threading from the MKL side.
The only exception is multiclass LogisticRegression, which uses `joblib` for parallelism across classes. | ||
|
||
|sklearnex| follows the same rules as |sklearn| for | ||
`the calculation of the 'n_jobs' parameter value <https://scikit-learn.org/stable/glossary.html#term-n_jobs>`_. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
`the calculation of the 'n_jobs' parameter value <https://scikit-learn.org/stable/glossary.html#term-n_jobs>`_. | |
`the calculation of the 'n_jobs' parameter value <https://scikit-learn.org/stable/glossary.html#term-n_jobs>`__. |
Link is repeated, single underscore makes it a named reference, which can cause with repetitions that change the name.
|sklearnex| supports the `n_jobs <https://scikit-learn.org/stable/glossary.html#term-n_jobs>`_ parameter | ||
of the original |sklearn| with the following differences: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|sklearnex| supports the `n_jobs <https://scikit-learn.org/stable/glossary.html#term-n_jobs>`_ parameter | |
of the original |sklearn| with the following differences: | |
|sklearnex| supports the `n_jobs <https://scikit-learn.org/stable/glossary.html#term-n_jobs>`_ parameter | |
of the original |sklearn|, with the following differences: |
Environment variables such as `OMP_NUM_THREADS`, `MKL_NUM_THREADS`, `OPENBLAS_NUM_THREADS`, and others used by | ||
low-level parallelism libraries are recognized by `joblib` and therefore can be used as hints by |sklearnex|. | ||
|
||
To track the actual number of threads used by sklearnex's estimators, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's also the MKL debug variable, and now the oneDAL debug variable.
|sklearnex| follows the same rules as |sklearn| for | ||
`the calculation of the 'n_jobs' parameter value <https://scikit-learn.org/stable/glossary.html#term-n_jobs>`_. | ||
|
||
When Scikit-learn's utilities with built-in parallelism are used (for example, `GridSearchCV` or `VotingClassifier`), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When Scikit-learn's utilities with built-in parallelism are used (for example, `GridSearchCV` or `VotingClassifier`), | |
When Scikit-learn's utilities with built-in parallelism are used (for example, :obj:`sklearn.model_selection.GridSearchCV` or :obj:`sklearn.model_selection.VotingClassifier`), |
`the calculation of the 'n_jobs' parameter value <https://scikit-learn.org/stable/glossary.html#term-n_jobs>`_. | ||
|
||
When Scikit-learn's utilities with built-in parallelism are used (for example, `GridSearchCV` or `VotingClassifier`), | ||
|sklearnex| tries to determine the optimal number of threads per job using hints provided by `joblib`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wasn't aware that such a system existed. Could you provide a link to the code where this happens?
* `n_jobs` estimator parameter sets the number of threads used by the underlying |oneDAL| | ||
* |sklearnex| doesn't use `joblib` for parallelism in patched estimators and functions | ||
* The only low-level parallelism library used by |sklearnex| is oneTBB (through oneDAL) | ||
* The `threading` parallel backend of `joblib` is not supported by |sklearnex| |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does it mean "not supported by sklearnex"? What happens for example if you run an sklearn metaestimator (like BaggingClassifier
) using that joblib backend with an sklearnex estimator inside?
Thanks for adding these explanations. But it's still missing important pieces of information and leaves several questions unanswered:
|
@Alexsandruss make sure to merge main for latest CI checks on docs |
Description
Adds a doc page for
n_jobs
specifics of sklearnex.Checklist to comply with before moving PR from draft:
PR completeness and readability
Testing
Performance
N/A