How to install it and integrate with prometheus?

[Vijaygawate]

I would like to scale HPA based on GPU utilization so for that, i need to send GPU metrics to prometheus
How can I make use of nvidia_gpu_exporter for this use case?

[Karaatin]

Hi there :)

I faced a similar challenge and wanted to share the steps that worked for me:

1. Install Prometheus
   Use the following command:

   sudo apt install prometheus

2. Install the latest version of the nvidia_gpu_exporter
   Download the package and install it via:

   sudo dpkg -i <your-package-name>

3. Verify that Prometheus and the nvidia_gpu_exporter are running
   Check their status with:

   sudo systemctl status prometheus
   sudo systemctl status nvidia_gpu_exporter

4. Update Prometheus configuration
   Add the following job to your /etc/prometheus/prometheus.yml file:

   - job_name: 'nvidia_gpu_exporter'
     scrape_interval: 5s
     scrape_timeout: 5s
     static_configs:
       - targets: ['localhost:9835']

   Note: You can adjust the interval as needed. This assumes the exporter runs on port 9835.

5. (Optional) Enable persistent data storage for Prometheus
   Update /etc/default/prometheus with the following arguments:

   ARGS="--config.file=/etc/prometheus/prometheus.yml \
         --storage.tsdb.path=/var/lib/prometheus/data \
         --storage.tsdb.retention.time=30d"

   Note: Modify the storage location and retention time based on your requirements.

6. Test the data scraping process
   Run the following command to verify if queries are being scraped successfully:

   curl -G 'http://localhost:9090/api/v1/query' --data-urlencode 'query=up' | jq

   • A value of 1 means the job is up and running.
   • A value of 0 indicates it is not working / correctly configured.

7. (Optional) Check storage requirements
   You can assess the size of the persistent storage directory using:

   sudo du -sh /var/lib/prometheus/data

---

For me, these steps worked flawlessly!
To confirm that Prometheus is scraping nvidia_gpu_exporter metrics, you can:

• List all metrics scraped by Prometheus:

  curl -G 'http://localhost:9090/api/v1/label/__name__/values' | jq

• Filter NVIDIA-related metrics:

  curl -G 'http://localhost:9090/api/v1/label/__name__/values' | jq '.data[] | select(test("nvidia"))'

To dive deeper, you can test specific metrics, like this:

curl -G 'http://localhost:9090/api/v1/query' --data-urlencode 'query=nvidia_smi_memory_used_bytes' | jq

To wrap it all up, you can import the pre-configured dashboard from the provided repository into Grafana. This should make visualizing your metrics seamless. Just make sure to set up Prometheus (localhost:9090) as your data source. From there, you can analyze and scale according to your needs.

Hope this helps!