04. Developer Guide

This guide is for developers who want to contribute to TritonParse, understand its architecture, or extend its functionality.

🏗️ Architecture Overview

High-Level Architecture

TritonParse consists of three main components:

┌─────────────────────┐    ┌─────────────────────┐    ┌─────────────────────┐
│   Python Backend    │    │   Processing        │    │   Frontend UI       │
│                     │    │                     │    │                     │
│ • Structured Logging│──▶│ • Log Parsing       │──▶│ • React Interface   │
│ • Triton Hooks      │    │ • Source Mapping    │    │ • IR Visualization  │
│ • Trace Generation  │    │ • Data Compression  │    │ • Code Comparison   │
└─────────────────────┘    └─────────────────────┘    └─────────────────────┘

Component Details

1. Python Backend (tritonparse/)

• Purpose: Capture Triton compilation events and generate structured logs
• Key Files:
  • structured_logging.py - Main logging infrastructure
  • utils.py - Parsing and processing utilities
  • extract_source_mappings.py - IR stage correlation
  • source_type.py - Source type definitions

2. Processing Pipeline

• Purpose: Transform raw logs into structured, analyzable format
• Key Functions:
  • Parse NDJSON logs
  • Extract source mappings between IR stages
  • Compress and package data

3. Frontend UI (website/)

• Purpose: Interactive visualization and analysis interface
• Key Technologies:
  • React 19 with TypeScript
  • Vite build system
  • Tailwind CSS for styling
  • Monaco Editor for code display

📁 Project Structure

tritonparse/
├── tritonparse/                 # Python package
│   ├── __init__.py
│   ├── structured_logging.py    # Core logging infrastructure
│   ├── utils.py                 # CLI and parsing utilities
│   ├── extract_source_mappings.py # Source mapping extraction
│   ├── source_type.py           # Source type definitions
│   ├── common.py                # Common utilities
│   ├── tp_logger.py             # Logger configuration
│   └── shared_vars.py           # Shared variables
├── website/                     # React web application
│   ├── src/
│   │   ├── components/          # React components
│   │   │   ├── CodeViewer.tsx   # Code display component
│   │   │   ├── CodeComparisonView.tsx # Side-by-side view
│   │   │   ├── WelcomeScreen.tsx # Landing page
│   │   │   └── ...
│   │   ├── pages/               # Main page components
│   │   │   ├── CodeView.tsx     # Single IR view
│   │   │   └── KernelOverview.tsx # Kernel overview
│   │   ├── utils/               # Utility functions
│   │   │   ├── dataLoader.ts    # Data loading and processing
│   │   │   └── fbDetection.ts   # Internal detection
│   │   ├── App.tsx              # Main application component
│   │   └── main.tsx             # Entry point
│   ├── public/                  # Static assets
│   ├── scripts/                 # Build scripts
│   └── package.json             # Dependencies
├── tests/                       # Test suite
├── docs/                        # Documentation
├── .github/                     # GitHub Actions
├── .ci/                         # CI scripts
├── pyproject.toml               # Python project configuration
├── Makefile                     # Development commands
└── README.md                    # Project overview

🔧 Development Environment Setup

Prerequisites

• Python >= 3.9
• Node.js >= 18.0.0
• Triton > 3.3.1 (compiled from source)
• Git for version control

1. Clone and Setup

# Clone repository
git clone https://github.com/pytorch-labs/tritonparse.git
cd tritonparse

# Install Python dependencies
make install-dev

# Install website dependencies
cd website
npm install

2. Verify Development Setup

# Check formatting and linting
make format-check
make lint-check

3. Verify Setup

# Check Python setup
make format-check
make lint-check
python -m unittest tests.test_tritonparse.TestTritonparseCPU -v

# Check website setup
cd website
npm run dev

🛠️ Development Workflow

Code Style and Formatting

We use a comprehensive formatting pipeline:

Tool	Purpose	Configuration
Black	Code formatting	pyproject.toml
usort	Import sorting	pyproject.toml
Ruff	Linting	Built-in rules

Essential Commands

# Format code
make format

# Check formatting
make format-check

# Run linting
make lint-check

# Run tests
python -m unittest tests.test_tritonparse -v

# Website development
cd website && npm run dev

Development Quality Checks

Before committing, ensure:

1. Code is formatted: make format
2. Linting passes: make lint-check
3. Tests pass: python -m unittest tests.test_tritonparse -v
4. Website builds: cd website && npm run build

🏗️ Backend Development

Core Components

1. Structured Logging (structured_logging.py)

Purpose: Capture Triton compilation and launch events in structured format

Key Functions:

• init(log_path) - Initialize logging system
• TODO:

Integration Points:

• Triton compilation hooks
• PyTorch TorchInductor integration
• Stack trace extraction

2. Log Processing (utils.py)

Purpose: Transform raw logs into analyzable format

Key Functions:

• unified_parse() - Main parsing interface
• oss_run() - OSS-specific parsing logic
• parse_logs() - Core log processing

Processing Pipeline:

1. Read raw NDJSON logs from input directory
2. Parse and validate log entries
3. Extract source mappings between IR stages
4. Compress and save processed data

3. Source Mapping (extract_source_mappings.py)

Purpose: Correlate lines between different IR stages

Key Functions:

• extract_source_mappings() - Main extraction logic
• process_kernel_logs() - Process individual kernel logs
• map_ir_stages() - Map lines between IR formats

Adding New Features

TODO

Testing Backend Changes

# Run CPU tests (no GPU required)
python -m unittest tests.test_tritonparse.TestTritonparseCPU -v

# Run GPU tests (requires CUDA)
python -m unittest tests.test_tritonparse.TestTritonparseCUDA -v

# Run specific test
python -m unittest tests.test_tritonparse.TestTritonparseCUDA.test_whole_workflow -v

# Test with real kernel
cd tests
TORCHINDUCTOR_FX_GRAPH_CACHE=0 python test_add.py

🎨 Frontend Development

Technology Stack

• React 19 - UI framework
• TypeScript - Type safety
• Vite - Build tool and dev server
• Tailwind CSS - Styling
• Monaco Editor - Code display

Key Components

1. Data Loading (utils/dataLoader.ts)

Purpose: Load and process trace files

Key Functions:

• loadLogData() - Load from URL
• loadLogDataFromFile() - Load from file
• processKernelData() - Process raw data

2. Code Viewer (components/CodeViewer.tsx)

Purpose: Display IR code with syntax highlighting

Features:

• Language-specific syntax highlighting
• Line number display
• Interactive line selection
• Source mapping visualization

3. Code Comparison (components/CodeComparisonView.tsx)

Purpose: Side-by-side IR comparison

Features:

• Synchronized scrolling
• Line mapping visualization
• Interactive highlighting
• Dropdown IR selection

Adding New Features

TODO

Testing Frontend Changes

cd website

# Development server
npm run dev

# Type checking
npm run build

# Linting
npm run lint

# Test with sample data
# Load ./public/f0_fc0_a0_cai-.ndjson in browser

📊 Data Flow

End-to-End Data Flow

Python Code
     │
     ▼
Triton Compilation
     │
     ▼
Structured Logging ──────┐
     │                   │
     ▼                   ▼
Raw NDJSON Logs     Hook Events
     │                   │
     ▼                   ▼
Log Processing      Source Mapping
     │                   │
     ▼                   ▼
Compressed Data ◄───────┘
     │
     ▼
Web Interface
     │
     ▼
Interactive Visualization

Data Formats

1. Raw NDJSON Format

{
  "event_type": "compilation_start",
  "timestamp": 1234567890,
  "kernel_name": "add_kernel",
  "metadata": {...}
}

2. Processed Format

{
  "kernels": [
    {
      "hash": "abc123",
      "name": "add_kernel",
      "metadata": {...},
      "irFiles": {
        "ttgir": "...",
        "ptx": "..."
      },
      "sourceMappings": {
        "ttgir": {...},
        "ptx": {...}
      }
    }
  ]
}

🔍 Debugging and Development Tools

Debug Logging

# Enable debug logging
export TRITONPARSE_DEBUG=1

# Run with debug output
python your_script.py

Development Utilities

# Check log file contents
head -n 10 ./logs/*.ndjson

# Inspect compressed data
zcat ./parsed_output/*.gz | head -n 20

# Test parsing pipeline
python -c "
import tritonparse.utils
tritonparse.utils.unified_parse('./logs/', './test_output/', verbose=True)
"

Browser Developer Tools

// Enable frontend debug logging
localStorage.setItem('tritonparse-debug', 'true');

// Inspect loaded data
console.log(window.tritonparseData);

// Test data processing
import { processKernelData } from './utils/dataLoader';
console.log(processKernelData(rawData));

🧪 Testing

Test Structure

tests/
├── test_tritonparse.py         # Main test suite
├── test_add.py                 # Manual test example
├── unit_tests.py               # Unit tests
└── example_output/             # Sample data

Running Tests

# All tests
python -m unittest tests.test_tritonparse -v

# CPU-only tests
python -m unittest tests.test_tritonparse.TestTritonparseCPU -v

# GPU tests (requires CUDA)
python -m unittest tests.test_tritonparse.TestTritonparseCUDA -v

# Manual test
cd tests
TORCHINDUCTOR_FX_GRAPH_CACHE=0 python test_add.py

Writing Tests

# tests/test_new_feature.py
import unittest
from tritonparse.new_feature import new_function

class TestNewFeature(unittest.TestCase):
    def test_new_function(self):
        result = new_function(input_data)
        self.assertEqual(result, expected_output)

    def setUp(self):
        # Test setup
        pass

    def tearDown(self):
        # Test cleanup
        pass

📦 Release Process

Version Management

Versions are managed in:

• pyproject.toml - Python package version
• website/package.json - Frontend version

Release Steps

1. Update version numbers
2. Update CHANGELOG.md
3. Run full test suite
4. Build and test website
5. Create GitHub release
6. Deploy to GitHub Pages

GitHub Actions

CI/CD pipeline includes:

• Format checking - Code style validation
• Linting - Code quality checks
• Testing - Python and frontend tests
• Website deployment - Automatic GitHub Pages deployment

🤝 Contributing Guidelines

Pull Request Process

1. Fork the repository
2. Create feature branch: git checkout -b feature-name
3. Make changes following coding standards
4. Add tests for new functionality
5. Run formatting: make format
6. Run tests: make lint-check && python -m unittest tests.test_tritonparse -v
7. Submit pull request

Code Review Process

• All PRs require review by core maintainers
• CI checks must pass before merge
• Documentation updates required for new features
• Tests required for new functionality

Issue Reporting

When reporting issues:

1. Use issue templates provided
2. Include system information
3. Provide reproduction steps
4. Include error messages and logs

📚 Additional Resources

Documentation

• Code Formatting Guide - Detailed formatting standards
• API Reference - Complete API documentation
• Architecture Deep Dive - Detailed architecture

Community

• GitHub Discussions - Community Q&A
• GitHub Issues - Bug reports and feature requests

External Resources

• Triton Documentation - Official Triton docs
• React Documentation - React development guide
• TypeScript Documentation - TypeScript reference

🔗 Next Steps

For new developers:

1. Complete the Installation Guide
2. Read the Usage Guide to understand the tool
3. Explore the codebase starting with simple components
4. Run the test suite to verify your setup
5. Join GitHub Discussions for community support

For experienced contributors:

1. Check GitHub Issues for open tasks
2. Review the Architecture Deep Dive for advanced topics
3. Contribute to documentation improvements
4. Propose new features through GitHub Discussions