Seems related to this bug: https://github.com/shap/shap/issues/3363 Can be avoided with passing `shap_kwargs=dict(check_additivity=False)` to the explainer, but then you might get inaccurate shap values. Added check_additivity=False param to the skorch tests for now.