- Overview
- Use Case / Problem Description
- Software Components
- Target Audience
- Repository Structure Overview
- Documentation
- Prerequisites
- Hardware Requirements
- Quickstart Guide
- Known CVEs
- License
This repository is what powers the build experience, showcasing video search and summarization agent with NVIDIA NIM microservices.
Insightful, accurate, and interactive video analytics AI agents enable a range of industries to make better decisions faster. These AI agents are given tasks through natural language and can perform complex operations like video summarization and visual question-answering, unlocking entirely new application possibilities. The NVIDIA AI Blueprint makes it easy to get started building and customizing video analytics AI agents for video search and summarization — all powered by generative AI, vision language models (VLMs) like Cosmos Nemotron VLMs, large language models (LLMs) like Llama Nemotron LLMs, NVIDIA NeMo Retriever, and NVIDIA NIM.
The NVIDIA AI Blueprint for Video Search and Summarization addresses the challenge of efficiently analyzing and summarizing large volumes of video data. This can be used to create vision AI agents, that can be applied to a multitude of use cases such as monitoring smart spaces, warehouse automation, and SOP validation. This is important where quick and accurate video analysis can lead to better decision-making and enhanced operational efficiency.
-
NIM microservices: Here are models used in this blueprint:
-
Ingestion Pipeline:
The process involves decoding video segments (chunks) generated by the stream handler, selecting frames, and using a vision-language model (VLM) along with a caption prompt to generate detailed captions for each chunk. A computer vision pipeline enhances video analysis by providing detailed metadata on objects. In parallel, the audio is extracted and a transcription is generated. These dense captions, along with audio transcripts and CV metadata are then indexed into vector and graph databases for use in the Context-Aware Retrieval-Augmented Generation workflow.
-
CA-RAG module:
The Context-Aware Retrieval-Augmented Generation (CA-RAG) module leverages both Vector RAG and Graph-RAG as primary sources for video understanding. This module is utilized in key features such as summarization, Q&A, and sending alerts. During the Q&A workflow, the CA-RAG module extracts relevant context from the vector database and graph database to enhance temporal reasoning, anomaly detection, multi-hop reasoning, and scalability. This approach offers deeper contextual understanding and efficient management of extensive video data. Additionally, the context manager effectively maintains its working context by efficiently using both short-term memory, such as chat history, and long-term memory resources like vector and graph databases, as needed.
This blueprint is designed for ease of setup with extensive configuration options, requiring technical expertise. It is intended for:
-
Video Analysts and IT Engineers: Professionals focused on analyzing video data and ensuring efficient processing and summarization. The blueprint offers 1-click deployment steps, easy-to-manage configurations, and plug-and-play models, making it accessible for early developers.
-
GenAI Developers / Machine Learning Engineers: Experts who need to customize the blueprint for specific use cases. This includes modifying the RAG pipeline for unique datasets and fine-tuning LLMs as needed. For advanced users, the blueprint provides detailed configuration options and custom deployment possibilities, enabling extensive customization and optimization.
deploy/: Contains scripts for docker compose and helm chart deployment, along with notebook for Brev launchable deployment.src/: Source code for the video search and summarization agent.
For detailed instructions and additional information about this blueprint, please refer to the official documentation.
- NVIDIA AI Enterprise developer licence required to local host NVIDIA NIM.
- API catalog keys:
- NVIDIA API catalog or NGC (steps to generate key)
The platform requirement can vary depending on the configuration and deployment topology used for VSS and dependencies like VLM, LLM, etc. For a list of validated GPU topologies and what configuration to use, see the supported platforms.
| Deployment Type | VLM | LLM | Embedding (llama-3.2-nv-embedqa-1b-v2) | Reranker (llama-3.2-nv-rerankqa-1b-v2) | Minimum GPU Requirement |
|---|---|---|---|---|---|
| Local deployment (Default topology) | Local (VILA 1.5) | Local (Llama 3.1 70B) | Local | Local | 8xB200, 8xH200, 8xH100, 8xA100 (80GB), 8xL40S |
| Local deployment (Reduced Compute) | Local (NVILA 15b) | Local (Llama 3.1 70B) | Local | Local | 4xB200, 4xH200, 4xH100, 4xA100 (80GB), 6xL40S |
| Local deployment (Single GPU) | Local (NVILA 15b) | Local (Llama 3.1 8b low mem mode) | Local | Local | 1xB200, 1xH200, 1xH100, 1xA100 (80GB) |
| Local VLM deployment | Local | Remote | Remote | Remote | 1xB200, 1xH200, 1xH100, 2xA100 (80GB), 2xL40S |
| Complete remote deployment | Remote | Remote | Remote | Remote | Minimum 8GB VRAM GPU |
Ideal for: Quickly getting started with your own videos without worrying about hardware and software requirements.
Follow the steps from the documentation and notebook in deploy directory to complete all pre-requisites and deploy the blueprint using Brev Launchable in an 8xL040s Crusoe instance.
- deploy/1_Deploy_VSS_docker_Crusoe.ipynb: This notebook is tailored specifically for the Crusoe CSP which uses Ephemeral storage.
Ideal for: Development phase where you need to run VSS locally, test different models, and experiment with various deployment configurations. This method offers greater flexibility for debugging each component.
For custom VSS deployments through Docker Compose, multiple samples are provided to show different combinations of remote and local model deployments. The /deploy/docker directory contains a README with all the details. Link to README
- Ubuntu 22.04
- NVIDIA driver 535.161.08 (Recommended minimum version). NVIDIA driver 570.86.15 (for H200). NVIDIA driver 570.133.20 (for B200)
- CUDA 12.2+ (CUDA driver installed with NVIDIA driver)
- NVIDIA Container Toolkit 1.13.5+
- Docker 27.5.1+
- Docker Compose 2.32.4
Ideal for: Production deployments that need to integrate with other systems. Helm offers advantages such as easy upgrades, rollbacks, and management of complex deployments.
The /deploy/helm/ directory contains a nvidia-blueprint-vss-2.3.1.tgz file which can be used to spin up VSS. Refer to the documentation here for detailed instructions.
- Ubuntu 22.04
- NVIDIA driver 535.183.06 (Recommended minimum version). NVIDIA driver 570.86.15 (for H200). NVIDIA driver 570.133.20 (for B200)
- CUDA 12.2+ (CUDA driver installed with NVIDIA driver)
- Kubernetes v1.31.2
- NVIDIA GPU Operator v23.9 (Recommended minimum version)
- Helm v3.x
VSS Engine 2.3.1 Container has the following known CVEs:
| CVE | Description |
|---|---|
| CVE-2024-8966 | This impacts gradio <= 5.22.0 python package, This impacts the file upload functionality of Gradio UI where an attacker can cause Denial-of-Service (DoS) attack by appending a large number of characters to the end of a multipart boundary. This does not affect VSS since the underlying root cause is already fixed by having a newer version 0.0.18 of python-multipart which does not have this vulnerability. |
| CVE-2025-4565 | This impacts protobuf < 4.25.8 python package, This impacts parsing of untrusted Protocol Buffers data containing an arbitrary number of recursive groups, recursive messages or a series of SGROUP tags leading to unbounded recursions and potential Denial-of-Service when protobuf pure-Python backend is used. This does not affect VSS since python backend of protobuf is not used. |
| CVE-2025-3887 | This impacts GStreamer H.265 codec parser, Malicious malformed streams can cause stack overflow in H.265 codec parser causing the application to crash. Users must take care that malicious H.265 streams are not added to VSS. This can be remedied by building and installing the GStreamer1.24.2 codec parser library after applying the patch mentioned in https://gstreamer.freedesktop.org/security/sa-2025-0001.html. |
The software and materials in this repository are governed by the NVIDIA Software License Agreement and the Product-Specific Terms for NVIDIA AI Products; except for models which are governed by the NVIDIA Community Model License.
ADDITIONAL INFORMATION: Llama 3.1 Community License Agreement for Llama-3.1-70b-instruct; Llama 3.2 Community License Agreement for NVIDIA Retrieval QA Llama 3.2 1B Embedding v2 and NVIDIA Retrieval QA Llama 3.2 1B Reranking v2, Apache, Version 2.0 for https://github.com/google-research/big_vision/blob/main/LICENSE and Apache License, Version 2.0 for https://github.com/01-ai/Yi/blob/main/LICENSE. Built with Llama.