Deepspeed Installation

Deepspeed Sparse Attention - Installation

You can also train with Microsoft Deepspeed's Sparse Attention, with any combination of dense and sparse attention that you'd like. However, you will have to endure the installation process.

Dependencies

• llvm-9-dev
• cmake
• gcc
• python3.7.x
• cudatoolkit=10.1
• pytorch=1.6.*

Installation instructions for Ubuntu 20.04 (Python3.7 is required)

First - install dependencies

sudo apt-get -y install llvm-9-dev cmake
git clone https://github.com/microsoft/DeepSpeed.git /tmp/Deepspeed
cd /tmp/Deepspeed && DS_BUILD_SPARSE_ATTN=1 ./install.sh -s # Change this to -r if you need to run as root
pip install triton
cd ~

Then you may either use conda or pip:

1. Conda

#!/bin/bash

conda create -n dalle_env python=3.7
conda activate dalle_env
conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch
pip install "git+https://github.com:lucidrains/DALLE-pytorch.git"

2. Pip

#!/bin/bash

python -m pip install virtualenv
python -m virtualenv -p=python3.7 ~/.virtualenvs/dalle_env
source ~/.virtualenvs/dalle_env/bin/activate
# Make sure your terminal shows that you're inside the virtual environment - and then run:
pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
pip install "git+https://github.com:lucidrains/DALLE-pytorch.git"

If all went well - continue to the following:

https://github.com/lucidrains/DALLE-pytorch/wiki/Deepspeed---Usage