You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jun 3, 2025. It is now read-only.
Copy file name to clipboardExpand all lines: integrations/huggingface-transformers/README.md
+6-8Lines changed: 6 additions & 8 deletions
Original file line number
Diff line number
Diff line change
@@ -29,11 +29,9 @@ Once trained, SparseML enables you to export models to the ONNX format, such tha
29
29
Install with `pip`:
30
30
31
31
```bash
32
-
pip install sparseml[torch]
32
+
pip install sparseml[transformers]
33
33
```
34
34
35
-
**Note**: Transformers will not immediately install with this command. Instead, a sparsification-compatible version of Transformers will install on the first invocation of the Transformers code in SparseML.
36
-
37
35
## **Tutorials**
38
36
39
37
-[Sparse Transfer Learning with the Python API](tutorials/sparse-transfer-learning-bert-python.md)[**RECOMMENDED**]
@@ -50,13 +48,13 @@ pip install sparseml[torch]
50
48
### **Use Case Examples - Python**
51
49
52
50
-[Sparse Transfer with GLUE Datasets (SST2) for sentiment analysis](tutorials/sentiment-analysis/docs-sentiment-analysis-python-sst2.ipynb)
53
-
-[Sparse Transfer with Custom Datasets (RottenTomatoes) and Custom Teacher from HF Hub for sentiment analysis](tutorials/sentiment-analysis/docs-sentiment-analysis-python-custom-teacher-rottentomatoes)
51
+
-[Sparse Transfer with Custom Datasets (RottenTomatoes) and Custom Teacher from HF Hub for sentiment analysis](tutorials/sentiment-analysis/docs-sentiment-analysis-python-custom-teacher-rottentomatoes.ipynb)
54
52
-[Sparse Transfer with GLUE Datasets (QQP) for multi-input text classification](tutorials/text-classification/docs-text-classification-python-qqp.ipynb)
55
53
-[Sparse Transfer with Custom Datasets (SICK) for multi-input text classification](tutorials/text-classification/docs-text-classification-python-sick.ipynb)
56
54
-[Sparse Transfer with Custom Datasets (TweetEval) and Custom Teacher for single input text classificaiton](tutorials/text-classification/docs-text-classification-python-custom-teacher-tweeteval.ipynb)
57
55
-[Sparse Transfer with Custom Datasets (GoEmotions) for multi-label text classification](tutorials/text-classification/docs-text-classification-python-multi-label-go_emotions.ipynb)
58
56
-[Sparse Transfer with Conll2003 for named entity recognition](tutorials/token-classification/docs-token-classification-python-conll2003.ipynb)
59
-
-[Sparse Transfer with Custom Datasets (WNUT) and Custom Teacher for named entity recognition](tutorials/token-classification/docs-token-classification-custom-teacher-wnut.ipynb)
57
+
-[Sparse Transfer with Custom Datasets (WNUT) and Custom Teacher for named entity recognition](tutorials/token-classification/docs-token-classification-python-custom-teacher-wnut.ipynb)
60
58
- Sparse Transfer with SQuAD (example coming soon!)
61
59
- Sparse Transfer with Squadshifts Amazon (example coming soon!)
62
60
@@ -66,7 +64,7 @@ pip install sparseml[torch]
66
64
67
65
SparseZoo is an open-source repository of pre-sparsified models, including BERT-base, BERT-large, RoBERTa-base, RoBERTa-large, and DistillBERT. With SparseML, you can fine-tune these pre-sparsified checkpoints onto custom datasets (while maintaining sparsity) via sparse transfer learning. This makes training inference-optimized sparse models almost identical to your typical training workflows!
68
66
69
-
[Check out the available models](https://sparsezoo.neuralmagic.com/?repo=huggingface&page=1)
67
+
[Check out the available models](https://sparsezoo.neuralmagic.com/?repos=huggingface)
70
68
71
69
### **Recipes**
72
70
@@ -140,15 +138,15 @@ Currently supported tasks include:
140
138
141
139
Sparse Transfer is very similiar to the typical transfer learing process used to train NLP models, where we fine-tune a checkpoint pretrained on a large upstream dataset using masked language modeling onto a smaller downstream dataset. With Sparse Transfer Learning, however, we simply start the fine-tuning process from a pre-sparsified checkpoint and maintain sparsity while the training process occurs.
142
140
143
-
Here, we will fine-tune a [90% pruned version of BERT](https://sparsezoo.neuralmagic.com/models/nlp%2Fmasked_language_modeling%2Fobert-base%2Fpytorch%2Fhuggingface%2Fwikipedia_bookcorpus%2Fpruned90-none) from the SparseZoo onto SST2.
141
+
Here, we will fine-tune a [90% pruned version of BERT](https://sparsezoo.neuralmagic.com/models/obert-base-wikipedia_bookcorpus-pruned90?comparison=obert-base-wikipedia_bookcorpus-base) from the SparseZoo onto SST2.
144
142
145
143
### **Kick off Training**
146
144
147
145
We will use SparseML's `sparseml.transformers.text_classification` training script.
148
146
149
147
To run sparse transfer learning, we first need to create/select a sparsification recipe. For sparse transfer, we need a recipe that instructs SparseML to maintain sparsity during training and to quantize the model.
150
148
151
-
For the SST2 dataset, there is a [transfer learning recipe available in SparseZoo](https://sparsezoo.neuralmagic.com/models/nlp%2Fsentiment_analysis%2Fobert-base%2Fpytorch%2Fhuggingface%2Fsst2%2Fpruned90_quant-none), identified by the following SparseZoo stub:
149
+
For the SST2 dataset, there is a [transfer learning recipe available in SparseZoo](https://sparsezoo.neuralmagic.com/models/obert-base-sst2_wikipedia_bookcorpus-pruned90_quantized?comparison=obert-base-sst2_wikipedia_bookcorpus-base&tab=0), identified by the following SparseZoo stub:
Copy file name to clipboardExpand all lines: integrations/huggingface-transformers/tutorials/sentiment-analysis/docs-sentiment-analysis-python-custom-teacher-rottentomatoes.ipynb
Copy file name to clipboardExpand all lines: integrations/huggingface-transformers/tutorials/sentiment-analysis/docs-sentiment-analysis-python-sst2.ipynb
Copy file name to clipboardExpand all lines: integrations/huggingface-transformers/tutorials/text-classification/docs-text-classification-python-custom-teacher-tweeteval.ipynb
Copy file name to clipboardExpand all lines: integrations/huggingface-transformers/tutorials/text-classification/docs-text-classification-python-multi-label-go_emotions.ipynb
Copy file name to clipboardExpand all lines: integrations/huggingface-transformers/tutorials/text-classification/docs-text-classification-python-qqp.ipynb
0 commit comments