Connection problem due to lack of CORS support in Triton Server, which blocks requests from frontend web applications

[Lizbeth-2Hernandez]

Description

Lack of native support for CORS (Cross-Origin Resource Sharing) in Triton Inference Server 23.10, which prevents direct integration with web frontend applications running on different domains. HTTP requests from web browsers (localhost:3000) to Triton Server (localhost:8000) are blocked by CORS security policies.

Environment

TensorRT Version: 23.10-py3
GPU Type: Included in container 23.10
Nvidia Driver Version: 535.154.05
CUDA Version:12.2
CUDNN Version: 8.9.5
Operating System + Version: Ubuntu 22.04.3 LTS
Python Version (if applicable): 3.10
Baremetal or Container (if container which image + tag): nvcr.io/nvidia/tritonserver:23.10-py3

Relevant Files

Technical Issue Report - Triton Inference Server - Google Drive

Steps To Reproduce

1. Environment Setup

# Pull the Triton container
docker pull nvcr.io/nvidia/tritonserver:23.10-py3

# Check GPU and drivers
nvidia-smi

2. Execution of Triton Server

# Exact execution command
docker run --gpus=1 -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v /home/user/model_repository:/models \
  nvcr.io/nvidia/tritonserver:23.10-py3 \
  tritonserver --model-repository=/models \
  --allow-http=true \
  --http-port=8000 \
  --log-verbose=1

3. Server Verification

# Verify that Triton is working
curl -v http://localhost:8000/v2/health/ready

# Check loaded models
curl http://localhost:8000/v2/models/model

4. Execution of the Frontend Application

# Install dependencies
npm install

# Run application
npm run dev

5. Error Reproduction

// From the browser console (http://localhost:3030)
fetch('http://localhost:8000/v2/health/ready', {
  method: 'GET',
  headers: {
    'Content-Type': 'application/json'
  }
})
.then(response => console.log(response))
.catch(error => console.error(error));

Full Trace of Errors

Error in Browser (Console)

Access to fetch at ‘http://localhost:8000/v2/health/ready’ from origin ‘http://localhost:3030’has been blocked by CORS policy: No ‘Access-Control-Allow-Origin’ header is present on the requested resource.

GET http://localhost:8000/v2/health/ready net::ERR_FAILED 200 (OK)

Triton Server Logs (Technical Success)

I0822 22:12:11.355377 1 http_server.cc:4384] HTTP request: 8 /v2/health/ready
I0822 22:12:11.355516 1 http_server.cc:4384] HTTP request: 8 /v2/health/ready
I0822 22:12:11.367992 1 http_server.cc:4384] HTTP request: 8 /v2/health/ready
# ... múltiples solicitudes recibidas correctamente

Missing Response Headers

HTTP/1.1 200 OK
Content-Length: 25
Content-Type: application/json
Date: Wed, 22 Aug 2023 22:12:11 GMT
# ❌ Faltan headers CORS:
# Access-Control-Allow-Origin: *
# Access-Control-Allow-Methods: GET, POST, OPTIONS
# Access-Control-Allow-Headers: Content-Type

CORS Configuration Attempt (Error)

# Comando fallido - parámetros no reconocidos
docker run --gpus=1 -p 8000:8000 -p 8001:8001 -p 8002:8002 \
  -v /home/user/model_repository:/models \
  nvcr.io/nvidia/tritonserver:23.10-py3 \
  tritonserver --model-repository=/models \
  --allow-http=true \
  --cors-allow-origin="http://localhost:3030" \
  --cors-allow-methods="GET,POST,OPTIONS" \
  --log-verbose=1Error resultante:

Resulting error:

tritonserver: unrecognized option '--cors-allow-origin=http://localhost:3030'

[whoisj]

With the exception of our OpenAI Frontend, Triton Server doesn't support CORS policy. The best approach would likely be to rely on an intermediary, like a reverse proxy, to append any necessary headers. In the case of health checks, this could be more complicated than it's worth.

I am curious as to what system is requiring CORS for health checks?

[ajcasagrande]

I am curious as to what system is requiring CORS for health checks?

@whoisj The system is this:

direct integration with web frontend applications

Most if not all web browsers will use strict CORS policy checking for security purposes. So the error is being seen by the javascript client code being run on the user's browser.

[whoisj]

@ajcasagrande I understand where you're coming from, and you are not incorrect.

That said, Triton Server does not provide significant security features such as authentication or DDOS protection. Instead, we expect the hosting environment to provide these features, meaning we do not generally expect direct browser to Triton interactions. Therefore we expect the frontend application to inject security/safety headers such as CORS.

[whoisj]

@Lizbeth-2Hernandez does my explanation clarify things for you?

if so, can this issue be closed?