Skip to content

Add Containerized Benchmarking Support for GuideLLM #123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Add Containerized Benchmarking Support for GuideLLM #123

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
29aaf7b
Add Dockerfile and benchmark script for containerized GuideLLM benchm…
wangchen615 Apr 17, 2025
9ca1e8f
fix pre-commit issue?
wangchen615 Apr 17, 2025
2fd8c0c
Add Dockerfile and benchmark script for containerized GuideLLM benchm…
wangchen615 Apr 17, 2025
5abf27a
Merge branch 'main' of https://github.com/wangchen615/guidellm
wangchen615 Apr 22, 2025
661e8e4
remote original dockerfile and run_benchmark.sh
wangchen615 Apr 22, 2025
4ae2bae
Merge branch 'main' into main
markurtz Apr 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
FROM python:3.11-slim

LABEL org.opencontainers.image.source="https://github.com/neuralmagic/guidellm"
LABEL org.opencontainers.image.description="GuideLLM Benchmark Container"

# Install system dependencies
RUN apt-get update && apt-get install -y \
git \
curl \
&& rm -rf /var/lib/apt/lists/*

# Create non-root user
RUN useradd -m -u 1000 guidellm

# Set working directory
WORKDIR /app

# Install GuideLLM
RUN pip install git+https://github.com/neuralmagic/guidellm.git
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than installing main, install our current HEAD. Also cp from within the container to avoid having two COPY layers.

Suggested change
RUN pip install git+https://github.com/neuralmagic/guidellm.git
COPY . /source
RUN pip install /source && \
cp /source/build/run_benchmark.sh /app/


# Copy and set up the benchmark script
COPY run_benchmark.sh /app/
RUN chmod +x /app/run_benchmark.sh

# Set ownership to non-root user
RUN chown -R guidellm:guidellm /app

# Switch to non-root user
USER guidellm

# Healthcheck
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume this was an accidental inclusion since GuideLLM does not provide a web server.

Suggested change
# Healthcheck
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
CMD curl -f http://localhost:8000/health || exit 1

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I remove this in the new build/Dockerfile.

# Set the entrypoint
ENTRYPOINT ["/app/run_benchmark.sh"]
34 changes: 34 additions & 0 deletions run_benchmark.sh
View file
Open in desktop
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this script is going to be the ENTRYPOINT and not the CMD than it needs to be more flexible. Either add an extra arguments variable or append command line arguments to the command.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I allow it to add extra arguments.

Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
#!/bin/bash

# Required environment variables
TARGET=${TARGET:-"http://localhost:8000"}
MODEL=${MODEL:-"neuralmagic/Meta-Llama-3.1-8B-Instruct-quantized.w4a16"}
RATE_TYPE=${RATE_TYPE:-"sweep"}
DATA=${DATA:-"prompt_tokens=256,output_tokens=128"}
MAX_REQUESTS=${MAX_REQUESTS:-"100"}
MAX_SECONDS=${MAX_SECONDS:-""}

# Output configuration
OUTPUT_PATH=${OUTPUT_PATH:-"/results/guidellm_benchmark_results"}
OUTPUT_FORMAT=${OUTPUT_FORMAT:-"json"} # Can be json, yaml, or yml

# Build the command
CMD="guidellm benchmark --target \"${TARGET}\" --model \"${MODEL}\" --rate-type \"${RATE_TYPE}\" --data \"${DATA}\""

# Add optional parameters
if [ ! -z "${MAX_REQUESTS}" ]; then
CMD="${CMD} --max-requests ${MAX_REQUESTS}"
fi

if [ ! -z "${MAX_SECONDS}" ]; then
CMD="${CMD} --max-seconds ${MAX_SECONDS}"
fi

# Add output path with appropriate extension
if [ ! -z "${OUTPUT_PATH}" ]; then
CMD="${CMD} --output-path \"${OUTPUT_PATH}.${OUTPUT_FORMAT}\""
fi

# Execute the command
echo "Running command: ${CMD}"
eval "${CMD}"