Skip to content

Add Comprehensive CI Scripts and Documentation #26

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 33 commits into from
Closed

Conversation

FindHao
Copy link
Member

@FindHao FindHao commented Jul 9, 2025

🚀 Add Comprehensive CI Scripts and Documentation

📋 Overview

This PR introduces a complete CI infrastructure for tritonparse, including modular scripts for environment setup, dependency installation, and test execution, along with comprehensive documentation.

🆕 New Features

CI Scripts (.ci/ directory):

  • setup.sh - Sets up conda environment, installs system dependencies, and configures CUDA/cuDNN
  • install-triton.sh - Installs Triton from source with proper libstdc++ handling
  • install-cudnn.sh - Downloads and installs cuDNN with architecture detection
  • install-project.sh - Installs tritonparse in editable mode with test dependencies
  • run-tests.sh - Runs test suite with configurable options (CPU/CUDA/all, coverage)
  • README.md - Comprehensive documentation for all CI scripts

Key Script Features:

  • Environment Variables: Configurable defaults (Python 3.11, CUDA 12.8, cuDNN 9.10.2.21)
  • Error Handling: All scripts use set -e for fail-fast behavior
  • Modular Design: Each script has single responsibility and can be used independently
  • Verbose Logging: Detailed output for debugging and monitoring
  • Architecture Support: Automatic detection for x86_64 and ARM64/aarch64

Test Configuration:

  • CPU-only tests: TEST_TYPE=cpu
  • CUDA-only tests: TEST_TYPE=cuda
  • All tests: TEST_TYPE=all (default)
  • Coverage reporting: COVERAGE=true

📚 Documentation

  • Complete workflow examples for CI and local development
  • Environment variable reference for all scripts
  • Usage examples with different configurations
  • Script dependencies and assumptions clearly documented

🎯 Benefits

  • Reproducible Builds: Same environment setup for CI and local development
  • Easy Maintenance: Modular scripts that can be updated independently
  • Flexible Testing: Support for different test types and coverage reporting
  • Developer Experience: Clear documentation and examples for all use cases

Test Plan:

Github Actions:

https://github.com/pytorch-labs/tritonparse/actions/workflows/test.yml

Manual Test:

# Complete setup workflow
CONDA_ENV=tritonparse bash .ci/setup.sh
CONDA_ENV=tritonparse bash .ci/install-triton.sh  
CONDA_ENV=tritonparse bash .ci/install-project.sh
CONDA_ENV=tritonparse bash .ci/run-tests.sh

# Run specific test types
TEST_TYPE=cuda COVERAGE=true bash .ci/run-tests.sh

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Meta Open Source bot. label Jul 9, 2025
@facebook-github-bot
Copy link
Contributor

@FindHao has imported this pull request. If you are a Meta employee, you can view this in D77978115.

@FindHao FindHao force-pushed the findhao/setup_ci branch from ae9bad6 to a74acdf Compare July 9, 2025 16:43
@facebook-github-bot
Copy link
Contributor

@FindHao has imported this pull request. If you are a Meta employee, you can view this in D77978115.

@FindHao FindHao force-pushed the findhao/setup_ci branch from a40356e to 9319a79 Compare July 9, 2025 22:41
FindHao and others added 21 commits July 9, 2025 18:56
Summary:
- Introduced multiple CI scripts: `setup.sh`, `install-triton.sh`, `install-project.sh`, and `run-tests.sh` for setting up the tritonparse environment, installing dependencies, and running tests.
- Updated `README.md` to document the usage and features of the new scripts.
- Added a GitHub Actions workflow (`test.yml`) to automate the testing process on code pushes and pull requests.

These additions streamline the CI process, ensuring a consistent environment for development and testing.
Summary:
- Modified the `test.yml` workflow file to add `findhao/setup_ci` branch to the list of branches that trigger CI tests on push events.

This change ensures that CI tests are executed for the new setup branch, enhancing the testing coverage during development.
Summary:
- Updated `install-triton.sh` to include system `libstdc++` updates for compatibility with newer C++ features and improved error handling during Triton installation verification.
- Modified `setup.sh` to install cuDNN and verify its installation, enhancing the setup process for the Triton environment.
- Updated `test.yml` to reflect changes in CUDA version from 12.6 to 12.8 and added a new linting job to ensure code quality.

These changes improve the robustness of the CI setup and ensure compatibility with the latest dependencies.
Summary:
- Introduced `install-cudnn.sh` script to automate the download and installation of cuDNN in the CI environment.
- Updated `README.md` to document the new script, including environment variables and usage examples.
- Modified `install-triton.sh` and `setup.sh` to utilize the new cuDNN installation script, streamlining the setup process.

These changes enhance the CI setup by providing a dedicated script for cuDNN installation, improving clarity and usability for developers.
Summary:
- Removed the linting job from the `test.yml` workflow, streamlining the CI process.
- Retained the test execution step to ensure that tests are still run as part of the CI pipeline.

This change simplifies the workflow configuration, focusing on test execution while eliminating the linting steps that were previously included.
Summary:
- Added architecture detection to the `install-cudnn.sh` script, allowing it to download the appropriate cuDNN archive based on the system architecture (aarch64/arm64 or x86_64).
- Updated the download process to fetch the cuDNN archive directly from NVIDIA, improving reliability.
- Implemented error handling for both the download and extraction processes, ensuring better feedback in case of failures.
- Cleaned up temporary files after installation to maintain a tidy environment.

These changes improve the robustness and usability of the cuDNN installation process in the CI environment.
…allation

Summary:
- Modified `.ci/setup.sh` to include installation of the CUDA toolkit along with clang and clangd, enhancing the setup process for system dependencies.
- Removed the separate CUDA installation step from the GitHub Actions workflow in `test.yml`, consolidating the installation process into the setup script.

These changes improve the efficiency of the CI environment setup by ensuring all necessary dependencies are installed in a single step.
@FindHao FindHao force-pushed the findhao/setup_ci branch from 9319a79 to c6a9711 Compare July 9, 2025 22:56
@facebook-github-bot
Copy link
Contributor

@FindHao has imported this pull request. If you are a Meta employee, you can view this in D77978115.

FindHao added 8 commits July 10, 2025 01:12
…ibility; update GitHub Actions workflow to reflect changes
…d logging; update GitHub Actions to fetch latest Triton commit for caching
…ging; update GitHub Actions workflow to cache Triton source and build more effectively
…checks, and error handling; update GitHub Actions workflow to ensure jq availability and handle commit fetching errors gracefully
…rsion detection, and conditional installation of development libraries; update GitHub Actions workflow to include weekly cache timestamp and cache APT packages.
…nd pull request events, streamlining CI process
@facebook-github-bot
Copy link
Contributor

@FindHao has imported this pull request. If you are a Meta employee, you can view this in D77978115.

…rch's official cuDNN installation script; revise README to reflect changes and streamline installation instructions.
@facebook-github-bot
Copy link
Contributor

@FindHao has imported this pull request. If you are a Meta employee, you can view this in D77978115.

@facebook-github-bot
Copy link
Contributor

@FindHao has imported this pull request. If you are a Meta employee, you can view this in D77978115.

@facebook-github-bot
Copy link
Contributor

@FindHao merged this pull request in d5975f7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed This label is managed by the Meta Open Source bot. Merged
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants