You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
refine docs for multi-backend alpha release (#1380)
* refine docs for multi-backend alpha release
* docs: further tweaks to multi-backend alpha docs
* docs: further tweaks to multi-backend alpha docs
* docs: further tweaks to multi-backend alpha docs
* docs: add multi-backend feedback links
* docs: add request for contributions
* docs: small fixes
* docs: small fixes
* docs: add info about `main` continuous build
* docs: further tweaks to multi-backend alpha docs
* docs: further tweaks to multi-backend alpha docs
Welcome to the installation guide for the `bitsandbytes` library! This document provides step-by-step instructions to install `bitsandbytes` across various platforms and hardware configurations. The library primarily supports CUDA-based GPUs, but the team is actively working on enabling support for additional backends like AMD ROCm, Intel, and Apple Silicon.
4
4
5
-
bitsandbytes is only supported on CUDA GPUs for CUDA versions **11.0 - 12.5**. However, there's a multi-backend effort under way which is currently in alpha release, check [the respective section below in case you're interested to help us with early feedback](#multi-backend).
5
+
> [!TIP]
6
+
> For a high-level overview of backend support and compatibility, see the [Multi-backend Support](#multi-backend) section.
> MacOS support is still a work in progress! Subscribe to this [issue](https://github.com/TimDettmers/bitsandbytes/issues/1020) to get notified about discussions and to track the integration progress.
20
+
## CUDA[[cuda]]
17
21
18
-
For Linux systems, make sure your hardware meets the following requirements to use bitsandbytes features.
22
+
`bitsandbytes` is currently only supported on CUDA GPUs for CUDA versions **11.0 - 12.5**. However, there's an ongoing multi-backend effort under development, which is currently in alpha. If you're interested in providing feedback or testing, check out [the multi-backend section below](#multi-backend).
> bitsandbytes >= 0.39.1 no longer includes Kepler binaries in pip installations. This requires manual compilation, and you should follow the general steps and use `cuda11x_nomatmul_kepler`for Kepler-targeted compilation.
42
+
> `bitsandbytes >= 0.39.1` no longer includes Kepler binaries in pip installations. This requires [manual compilation using](#cuda-compile)the `cuda11x_nomatmul_kepler`configuration.
27
43
28
44
To install from PyPI.
29
45
30
46
```bash
31
47
pip install bitsandbytes
32
48
```
33
49
34
-
### Compile from source[[compile]]
50
+
### `pip install` pre-built wheel from latest `main` commit
51
+
52
+
If you would like to use new feature even before they are officially released and help us test them, feel free to install the wheel directly from our CI (*the wheel links will remain stable!*):
53
+
54
+
<hfoptionsid="OS">
55
+
<hfoptionid="Linux">
56
+
57
+
```
58
+
# Note, if you don't want to reinstall BNBs dependencies, append the `--no-deps` flag!
> Don't hesitate to compile from source! The process is pretty straight forward and resilient. This might be needed for older CUDA versions or other less common configurations, which we don't support out of the box due to package size.
35
76
36
-
For Linux and Windows systems, you can compile bitsandbytes from source. Installing from source allows for more build options with different CMake configurations.
77
+
For Linux and Windows systems, compiling from source allows you to customize the build configurations. See below for detailed platform-specific instructions (see the `CMakeLists.txt` if you want to check the specifics and explore some additional options):
37
78
38
79
<hfoptionsid="source">
39
80
<hfoptionid="Linux">
40
81
41
-
To compile from source, you need CMake >= **3.22.1** and Python >= **3.8** installed. Make sure you have a compiler installed to compile C++ (gcc, make, headers, etc.). For example, to install a compiler and CMake on Ubuntu:
82
+
To compile from source, you need CMake >= **3.22.1** and Python >= **3.8** installed. Make sure you have a compiler installed to compile C++ (`gcc`, `make`, headers, etc.).
83
+
84
+
For example, to install a compiler and CMake on Ubuntu:
42
85
43
86
```bash
44
87
apt-get install -y build-essential cmake
@@ -48,16 +91,16 @@ You should also install CUDA Toolkit by following the [NVIDIA CUDA Installation
48
91
49
92
Refer to the following table if you're using another CUDA Toolkit version.
50
93
51
-
| CUDA Toolkit | GCC |
52
-
|---|---|
53
-
| >= 11.4.1 | >= 11 |
54
-
| >= 12.0 | >= 12 |
55
-
| >= 12.4 | >= 13 |
94
+
| CUDA Toolkit |GCC|
95
+
|--------------|-------|
96
+
| >= 11.4.1 | >= 11 |
97
+
| >= 12.0 | >= 12 |
98
+
| >= 12.4 | >= 13 |
56
99
57
100
Now to install the bitsandbytes package from source, run the following commands:
@@ -93,7 +136,7 @@ Big thanks to [wkpark](https://github.com/wkpark), [Jamezo97](https://github.com
93
136
</hfoption>
94
137
</hfoptions>
95
138
96
-
### PyTorch CUDA versions
139
+
### PyTorch CUDA versions[[pytorch-cuda-versions]]
97
140
98
141
Some bitsandbytes features may need a newer CUDA version than the one currently supported by PyTorch binaries from Conda and pip. In this case, you should follow these instructions to load a precompiled bitsandbytes binary.
99
142
@@ -105,7 +148,7 @@ Some bitsandbytes features may need a newer CUDA version than the one currently
105
148
Then locally install the CUDA version you need with this script from bitsandbytes:
3. Now when you launch bitsandbytes with these environment variables, the PyTorch CUDA version is overridden by the new CUDA version (in this example, version 11.7) and a different bitsandbytes library is loaded.
136
179
137
-
## Multi-backend[[multi-backend]]
180
+
## Multi-backend Support (Alpha Release)[[multi-backend]]
138
181
139
182
> [!TIP]
140
-
> This functionality is currently in preview and therefore not yet production-ready! Please reference [this guide](./non_cuda_backends) for more in-depth information about the different backends and their current status.
183
+
> This functionality is currently in preview and not yet production-ready. We very much welcome community feedback, contributions and leadership on topics like Apple Silicon as well as other less common accellerators! For more information, see [this guide on multi-backend support](./non_cuda_backends).
184
+
185
+
**Link to give us feedback** (bugs, install issues, perf results, requests, etc.)**:**
Please follow these steps to install bitsandbytes with device-specific backend support other than CUDA:
200
+
[**Github Discussion space on coordinating the kickoff of MPS backend development**](https://github.com/bitsandbytes-foundation/bitsandbytes/discussions/1340)
143
201
144
-
### Pip install the pre-built wheel (recommended for most)
To use bitsandbytes non-CUDA backends, be sure to install:
219
+
220
+
```
221
+
pip install "transformers>=4.45.1"
222
+
```
149
223
150
224
<hfoptionsid="backend">
151
225
<hfoptionid="AMD ROCm">
152
226
153
-
#### AMD GPU
154
-
155
-
bitsandbytes is fully supported from ROCm 6.1 onwards (currently in alpha release).
227
+
> [!WARNING]
228
+
> Pre-compiled binaries are only built for ROCm versions `6.1.0`/`6.1.1`/`6.1.2`/`6.2.0` and `gfx90a`, `gfx942`, `gfx1100` GPU architectures. [Find the pip install instructions here](#multi-backend-pip).
229
+
>
230
+
> Other supported versions that don't come with pre-compiled binaries [can be compiled for with these instructions](#multi-backend-compile).
231
+
>
232
+
> **Windows is not supported for the ROCm backend**; also not WSL2 to our knowledge.
156
233
157
234
> [!TIP]
158
-
> If you would like to install ROCm and PyTorch on bare metal, skip Docker steps and refer to our official guides at [ROCm installation overview](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/install-overview.html#rocm-install-overview) and [Installing PyTorch for ROCm](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/3rd-party/pytorch-install.html#using-wheels-package) (Step 3 of wheels build for quick installation). Please make sure to get PyTorch wheel for the installed ROCm version.
235
+
> If you would like to install ROCm and PyTorch on bare metal, skip the Docker steps and refer to ROCm's official guides at [ROCm installation overview](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/tutorial/install-overview.html#rocm-install-overview) and [Installing PyTorch for ROCm](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/3rd-party/pytorch-install.html#using-wheels-package) (Step 3 of wheels build for quick installation). Special note: please make sure to get the respective ROCm-specific PyTorch wheel for the installed ROCm version, e.g. `https://download.pytorch.org/whl/nightly/rocm6.2/`!
159
236
160
237
```bash
161
238
# Create a docker container with latest ROCm image, which includes ROCm libraries
@@ -165,9 +242,70 @@ apt-get update && apt-get install -y git && cd home
165
242
166
243
# Install pytorch compatible with above ROCm version
# (This is supported on Ubuntu 22.04, Python 3.10, ROCm 6.1.0/6.1.1/6.1.2/6.2.0 and gpu arch - gfx90a, gfx942, gfx1100
247
+
</hfoption>
248
+
<hfoptionid="Intel CPU + GPU">
249
+
250
+
Compatible hardware and functioning `import intel_extension_for_pytorch as ipex` capable environment with Python `3.10` as the minimum requirement.
251
+
252
+
Please refer to [the official Intel installations instructions](https://intel.github.io/intel-extension-for-pytorch/index.html#installation?platform=cpu&version=v2.4.0%2bcpu&os=linux%2fwsl2) for guidance on how to pip install the necessary `intel_extension_for_pytorch` dependency.
253
+
254
+
</hfoption>
255
+
<hfoptionid="Apple Silicon (MPS)">
256
+
257
+
> [!TIP]
258
+
> Apple Silicon support is still a WIP. Please visit and write us in [this Github Discussion space on coordinating the kickoff of MPS backend development](https://github.com/bitsandbytes-foundation/bitsandbytes/discussions/1340) and coordinate a community-led effort to implement this backend.
259
+
260
+
</hfoption>
261
+
</hfoptions>
262
+
263
+
### Installation
264
+
265
+
You can install the pre-built wheels for each backend, or compile from source for custom configurations.
> bitsandbytes does not yet support Apple Silicon / Metal with a dedicated backend. However, the build infrastructure is in place and the below pip install will eventually provide Apple Silicon support as it becomes available on the `multi-backend-refactor` branch based on community contributions.
290
+
291
+
```
292
+
# Note, if you don't want to reinstall BNBs dependencies, append the `--no-deps` flag!
bitsandbytes is fully supported from ROCm 6.1 onwards (currently in alpha release).
307
+
308
+
```bash
171
309
# Please install from source if your configuration doesn't match with these)
172
310
pip install bitsandbytes
173
311
@@ -195,10 +333,10 @@ pip install -e . # `-e` for "editable" install, when developing BNB (otherwise
195
333
196
334
Similar to the CUDA case, you can compile bitsandbytes from source for Linux and Windows systems.
197
335
198
-
The below commands are for Linux. For installing on Windows, please adapt the below commands according to the same pattern as described [the section above on compiling from source under the Windows tab](#compile).
336
+
The below commands are for Linux. For installing on Windows, please adapt the below commands according to the same pattern as described [the section above on compiling from source under the Windows tab](#cuda-compile).
199
337
200
338
```
201
-
git clone --depth 1 -b multi-backend-refactor https://github.com/TimDettmers/bitsandbytes.git && cd bitsandbytes/
339
+
git clone --depth 1 -b multi-backend-refactor https://github.com/bitsandbytes-foundation/bitsandbytes.git && cd bitsandbytes/
Copy file name to clipboardExpand all lines: docs/source/non_cuda_backends.mdx
+3Lines changed: 3 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,8 @@
1
1
# Multi-backend support (non-CUDA backends)
2
2
3
+
> [!Tip]
4
+
> If you feel these docs need some additional info, please consider submitting a PR or respectfully request the missing info in one of the below mentioned Github discussion spaces.
5
+
3
6
As part of a recent refactoring effort, we will soon offer official multi-backend support. Currently, this feature is available in a preview alpha release, allowing us to gather early feedback from users to improve the functionality and identify any bugs.
4
7
5
8
At present, the Intel CPU and AMD ROCm backends are considered fully functional. The Intel XPU backend has limited functionality and is less mature.
0 commit comments