Guide: WSL + cuda 11.6

[thistleknot]

.run files
#to match max compute capability

nano Makefile (wsl)
    NVCCFLAGS += -arch=native
    Change it to specify the correct architecture for your GPU. For a GPU with Compute Capability 5.2, you should replace it with:

    makefile
    Copy code
    NVCCFLAGS += -arch=sm_52
    
modified makefile
#https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/

apt-get remove --purge '^nvidia-.*' '^cuda-.*'
 sudo apt-get autoremove
sudo apt-get autoclean
sudo find / \( -path /home -o -path /mnt -o -path /usr/local/lib/python3.10 \) -prune -o ! -user root \( -name '*nvidia*' -o -name '*cuda*' \) -print

cat /var/log/cuda-uninstaller.log

modprobe -r nvidia

dpkg -l | grep -i nvidia | awk '{print $2}' | xargs sudo apt-get --purge remove -y

sudo rm -rf /etc/systemd/system/nvidia*
sudo rm -rf /etc/systemd/system/display-manager.service.d
sudo rm -rf /etc/modprobe.d/nvidia*
sudo rm -rf /etc/modules-load.d/nvidia*
sudo rm -rf /etc/kernel/postinst.d/nvidia*
sudo rm -rf /usr/lib/nvidia*
sudo rm -rf /usr/share/doc/nvidia*
sudo rm -rf /var/lib/dkms/nvidia*
sudo rm -rf /var/log/nvidia*
sudo rm -rf /usr/local/cuda*

Failed to initialize NVML: GPU access blocked by the operating system
Failed to properly shut down NVML: GPU access blocked by the operating system

Reinstall windows driver
=D

cd /data/llama.cpp
make LLAMA_CUBLAS=1

python -m pip install --force-reinstall --no-deps llama-cpp-python llama-cpp-python[server] --prefer-binary --extra-index-url=https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/AVX/cu122 pydantic pydantic_settings fastapi

python3 -m llama_cpp.server --model /root/text-gen-install/text-generation-webui/models/llama-2-7b-chat.Q2_K.gguf --n_gpu_layers 24

I can run a server, but text-generation-webui still doesn't work with this. Conflicting pydantic settings and llama.cpp