Didn't detected GPU when Deployed on AWS

[touqeerShah]

Hello , I am try to deployed Private GPT on AWS when I run it , it will not detected the GPU on Cloud but when i run it detected and work fine

AWS configuration

and logs are attached

[BrunoBosshard]

Did you install both CUDA and llama.cpp?

[touqeerShah]

I am using llam.cpp , and install with following commands

# Install Python 3.11
pyenv install 3.11
pyenv local 3.11

# Install dependencies
poetry install --with ui,local

# Download Embedding and LLM models
poetry run python scripts/setup

# (Optional) For Mac with Metal GPU, enable it. Check Installation and Settings section 
to know how to enable GPU on other platforms
CMAKE_ARGS="-DLLAMA_METAL=on" pip install --force-reinstall --no-cache-dir llama-cpp-python

# Run the local server  
export PORT=8080
PGPT_PROFILES=local make run

[Muxelmann]

I set up privateGPT in a VM with an Nvidia GPU passed through and got it to work. Before running make run, I executed the following command for building llama-cpp with CUDA support:

CMAKE_ARGS='-DLLAMA_CUBLAS=on' poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python

From start (fresh Ubuntu installation) to finish, these were the commands I used:

# Initial update and basic dependencies
sudo apt update
sudo apt upgrade
sudo apt install git curl zlib1g-dev tk-dev libffi-dev libncurses-dev libssl-dev libreadline-dev libsqlite3-dev liblzma-dev

# Check for GPU drivers and install them automatically
sudo ubuntu-drivers
sudo ubuntu-drivers list
sudo ubuntu-drivers autoinstall

# Install CUDA development dependencies
sudo apt install nvidia-cudnn nvidia-cuda-toolkit

# Reboot the system
reboot

# Test that GPU is correctly detected
nvidia-smi

# Verify CUDA is installed correctly
nvcc --version

# Installing pyenv
curl https://pyenv.run | bash

# Adding the following to .bashrc:
# export PYENV_ROOT="$HOME/.pyenv"
# [[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
# eval "$(pyenv init -)"
nano ~/.bashrc 
. ~/.bashrc 

# Installing Poetry
curl -sSL https://install.python-poetry.org | python3 -

# Adding the following to .bashrc:
# export PATH="$HOME/.local/bin:$PATH"
nano ~/.bashrc
. ~/.bashrc 

# Cloning privateGPT
git clone https://github.com/imartinez/privateGPT
cd privateGPT/

# Installing and setting python 3.11(.6)
pyenv install 3.11
pyenv local 3.11

# Installing dependencies
poetry install --with ui,local

# Running setup - since PR #1368:
make setup
# Running setup the old way:
# poetry run python scripts/setup

# Compiling llama-cpp with CUDA libs
CMAKE_ARGS='-DLLAMA_CUBLAS=on' poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python

# Running privateGPT
make run

Hope that helps.

[touqeerShah]

Thank you for you support

[Muxelmann]

I also tried the above inside a virtual environment using virtualenv with Python 3.11 installed via pyenv. Thereby, the local Python installation is not altered by Poetry and everything is contained within a local folder. And everything still worked with Nvidia.