Skip to content

Commit 1e02e70

Browse files
FindHaofacebook-github-bot
authored andcommitted
Add Makefile and format fix script for tritonparse project (#28)
Summary: # Add Linter and Formatting to CI Pipeline This PR adds comprehensive code formatting and linting checks to the CI pipeline, ensuring consistent code quality across the project. ## Key Changes ### 🔧 CI Integration - **Added format-check job** to `.github/workflows/test.yml` - **Parallel execution**: Format checks run before tests for faster feedback ### 🛠️ Formatting Tool Chain - **Black**: Primary code formatter (line-length: 88) - **usort**: Import sorting - **Ruff**: Linting only ### 📝 Configuration Updates - **pyproject.toml**: Configured black, usort, and ufmt settings - **Makefile**: Updated install-dev and lint commands - **format_fix.py**: Rewritten to use black for formatting, ruff for linting ## Usage Commands ```bash # Install development dependencies make install-dev # Fix all formatting issues make format # Check formatting (CI-compatible) make lint-check # Individual checks make format-check # Only check formatting ``` Pull Request resolved: #28 Test Plan: - ✅ `make format && make lint-check` passes cleanly - ✅ CI pipeline runs format checks before tests - ✅ All formatting tools work consistently ```bash % make format ; make lint-check Running format fix script... python -m tritonparse.tools.format_fix --verbose Running usort for import sorting... Running: usort format . ✅ usort completed Running ruff for linting... Running: ruff check . --fix All checks passed! ✅ ruff linting completed Running black for code formatting... Running: black . ✅ black completed 🎉 All formatting tools completed successfully! Checking linting... ruff check --diff . black --check --diff . All done! ✨ 🍰 ✨ 18 files would be left unchanged. ``` Reviewed By: davidberard98 Differential Revision: D78130587 Pulled By: FindHao fbshipit-source-id: db8db22697d3cef1abbbeca1e7071fcf7dbb3d3b
1 parent b781bd9 commit 1e02e70

File tree

7 files changed

+248
-21
lines changed

7 files changed

+248
-21
lines changed

.github/workflows/test.yml

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,9 +33,33 @@ on:
3333
type: boolean
3434

3535
jobs:
36+
format-check:
37+
runs-on: ubuntu-latest
38+
timeout-minutes: 10
39+
steps:
40+
- uses: actions/checkout@v4
41+
42+
- name: Set up Python 3.11
43+
uses: actions/setup-python@v4
44+
with:
45+
python-version: "3.11"
46+
47+
- name: Install development dependencies
48+
run: |
49+
make install-dev
50+
51+
- name: Check code formatting
52+
run: |
53+
make format-check
54+
55+
- name: Check linting
56+
run: |
57+
make lint-check || (echo "❌ Linting failed. Please run 'make format' to fix formatting issues, then commit the changes." && exit 1)
58+
3659
test:
3760
runs-on: 4-core-ubuntu-gpu-t4
3861
timeout-minutes: 120
62+
needs: format-check
3963
steps:
4064
- uses: actions/checkout@v4
4165

Makefile

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Makefile for tritonparse project
2+
3+
.PHONY: help format format-check lint lint-check test test-cuda clean install-dev
4+
5+
# Default target
6+
help:
7+
@echo "Available targets:"
8+
@echo " format - Format all Python files"
9+
@echo " format-check - Check formatting without making changes"
10+
@echo " lint - Run all linters"
11+
@echo " lint-check - Check linting without making changes"
12+
@echo " test - Run tests (CPU only)"
13+
@echo " test-cuda - Run tests (including CUDA tests)"
14+
@echo " clean - Clean up cache files"
15+
@echo " install-dev - Install development dependencies"
16+
17+
# Formatting targets
18+
format:
19+
@echo "Running format fix script..."
20+
python -m tritonparse.tools.format_fix --verbose
21+
22+
format-check:
23+
@echo "Checking formatting..."
24+
python -m tritonparse.tools.format_fix --check-only --verbose
25+
26+
# Linting targets
27+
lint:
28+
@echo "Running linters..."
29+
ruff check .
30+
black --check .
31+
32+
lint-check:
33+
@echo "Checking linting..."
34+
ruff check --diff .
35+
black --check --diff .
36+
37+
# Testing targets
38+
test:
39+
@echo "Running tests (CPU only)..."
40+
pytest tests/ -v -m "not cuda"
41+
42+
test-cuda:
43+
@echo "Running all tests (including CUDA)..."
44+
pytest tests/ -v
45+
46+
# Utility targets
47+
clean:
48+
@echo "Cleaning up cache files..."
49+
find . -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
50+
find . -type f -name "*.pyc" -delete 2>/dev/null || true
51+
find . -type f -name "*.pyo" -delete 2>/dev/null || true
52+
find . -type f -name ".coverage" -delete 2>/dev/null || true
53+
find . -type d -name "*.egg-info" -exec rm -rf {} + 2>/dev/null || true
54+
find . -type d -name ".pytest_cache" -exec rm -rf {} + 2>/dev/null || true
55+
56+
install-dev:
57+
@echo "Installing development dependencies..."
58+
pip install -e ".[test]"
59+
pip install black usort ruff

pyproject.toml

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,16 @@ test = [
1717
[tool.setuptools.packages.find]
1818
include = ["tritonparse*"]
1919

20-
[tool.ruff]
21-
line-length = 88
22-
2320
[tool.black]
2421
line-length = 88
2522
target-version = ["py310"]
2623

24+
[tool.ufmt]
25+
formatter = "black"
26+
sorter = "usort"
27+
28+
[tool.usort]
29+
first_party_detection = false
30+
2731
[project.urls]
2832
"Homepage" = "https://github.com/pytorch-labs/tritonparse"

tritonparse/common.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -173,9 +173,9 @@ def print_parsed_files_summary(parsed_log_dir: str) -> None:
173173
if size_bytes < 1024:
174174
file_size = f"{size_bytes}B"
175175
elif size_bytes < 1024 * 1024:
176-
file_size = f"{size_bytes/1024:.1f}KB"
176+
file_size = f"{size_bytes / 1024:.1f}KB"
177177
else:
178-
file_size = f"{size_bytes/(1024*1024):.1f}MB"
178+
file_size = f"{size_bytes / (1024 * 1024):.1f}MB"
179179
except OSError:
180180
pass
181181

tritonparse/structured_logging.py

Lines changed: 4 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -142,8 +142,6 @@ def convert(obj):
142142
Returns:
143143
A serializable version of the input object where dataclasses are converted to dictionaries
144144
"""
145-
from triton.language.core import dtype
146-
147145
# 1. primitives that JSON already supports -------------------------------
148146
if obj is None or isinstance(obj, (bool, int, str)):
149147
return obj
@@ -178,11 +176,6 @@ def convert(obj):
178176
return convert(
179177
asdict(obj)
180178
) # Convert dataclass to dict and then process that dict
181-
182-
# 4. Common Triton constexpr objects
183-
if isinstance(obj, dtype):
184-
return f"triton.language.core.dtype('{str(obj)}')"
185-
186179
log.warning(f"Unknown type: {type(obj)}")
187180
return str(obj) # Return primitive types as-is
188181

@@ -378,18 +371,16 @@ def extract_file_content(trace_data: Dict[str, Any], metadata_group: Dict[str, s
378371
# Check file size before reading to avoid memory issues
379372
file_size = os.path.getsize(file_path)
380373
if file_size > MAX_FILE_SIZE:
381-
trace_data["file_content"][ir_filename] = (
382-
f"<file too large: {file_size} bytes>"
383-
)
374+
message = f"<file too large: {file_size} bytes>"
375+
trace_data["file_content"][ir_filename] = message
384376
continue
385377

386378
with open(file_path, "r") as f:
387379
trace_data["file_content"][ir_filename] = f.read()
388380
except (UnicodeDecodeError, OSError) as e:
389381
# add more specific error type
390-
trace_data["file_content"][ir_filename] = (
391-
f"<error reading file: {str(e)}>"
392-
)
382+
message = f"<error reading file: {str(e)}>"
383+
trace_data["file_content"][ir_filename] = message
393384
log.debug(f"Error reading file {file_path}: {e}")
394385

395386

tritonparse/tools/format_fix.py

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
#!/usr/bin/env python3
2+
"""
3+
Format fix script for tritonparse project.
4+
5+
This script runs all linter tools to format and fix code issues:
6+
- usort: Import sorting
7+
- ruff: Linting only
8+
- black: Code formatting
9+
10+
Usage:
11+
python -m tritonparse.tools.format_fix [options]
12+
13+
Options:
14+
--check-only Only check for issues, don't fix them
15+
--verbose Verbose output
16+
--help Show this help message
17+
"""
18+
19+
import argparse
20+
import subprocess
21+
import sys
22+
23+
24+
def run_command(cmd: list[str], verbose: bool = False) -> bool:
25+
"""Run a command and return success status."""
26+
if verbose:
27+
print(f"Running: {' '.join(cmd)}")
28+
29+
try:
30+
result = subprocess.run(cmd, capture_output=True, text=True, check=False)
31+
32+
if result.returncode != 0:
33+
if verbose:
34+
print(f"Command failed with return code {result.returncode}")
35+
if result.stdout:
36+
print("STDOUT:", result.stdout)
37+
if result.stderr:
38+
print("STDERR:", result.stderr)
39+
return False
40+
41+
if verbose and result.stdout:
42+
print(result.stdout)
43+
44+
return True
45+
except Exception as e:
46+
if verbose:
47+
print(f"Error running command: {e}")
48+
return False
49+
50+
51+
def run_usort(check_only: bool = False, verbose: bool = False) -> bool:
52+
"""Run usort for import sorting."""
53+
cmd = ["usort"]
54+
55+
if check_only:
56+
cmd.extend(["check", "."])
57+
else:
58+
cmd.extend(["format", "."])
59+
60+
return run_command(cmd, verbose)
61+
62+
63+
def run_ruff_check(check_only: bool = False, verbose: bool = False) -> bool:
64+
"""Run ruff for linting only."""
65+
cmd = ["ruff", "check", "."]
66+
67+
if check_only:
68+
cmd.append("--diff")
69+
else:
70+
cmd.append("--fix")
71+
72+
return run_command(cmd, verbose)
73+
74+
75+
def run_black(check_only: bool = False, verbose: bool = False) -> bool:
76+
"""Run black for code formatting."""
77+
cmd = ["black"]
78+
79+
if check_only:
80+
cmd.extend(["--check", "--diff", "."])
81+
else:
82+
cmd.append(".")
83+
84+
return run_command(cmd, verbose)
85+
86+
87+
def main():
88+
"""Main function."""
89+
parser = argparse.ArgumentParser(
90+
description="Format fix script for tritonparse project",
91+
epilog="""
92+
Examples:
93+
# Fix all formatting issues
94+
python -m tritonparse.tools.format_fix
95+
96+
# Check for issues without fixing
97+
python -m tritonparse.tools.format_fix --check-only
98+
99+
# Verbose output
100+
python -m tritonparse.tools.format_fix --verbose
101+
""",
102+
)
103+
104+
parser.add_argument(
105+
"--check-only",
106+
action="store_true",
107+
help="Only check for issues, don't fix them",
108+
)
109+
parser.add_argument("--verbose", action="store_true", help="Verbose output")
110+
111+
args = parser.parse_args()
112+
113+
# Run formatters on the entire project
114+
success = True
115+
116+
# 1. Run usort for import sorting
117+
print("Running usort for import sorting...")
118+
if not run_usort(args.check_only, args.verbose):
119+
print("❌ usort failed")
120+
success = False
121+
else:
122+
print("✅ usort completed")
123+
124+
# 2. Run ruff for linting only
125+
print("Running ruff for linting...")
126+
if not run_ruff_check(args.check_only, args.verbose):
127+
print("❌ ruff linting failed")
128+
success = False
129+
else:
130+
print("✅ ruff linting completed")
131+
132+
# 3. Run black for code formatting
133+
print("Running black for code formatting...")
134+
if not run_black(args.check_only, args.verbose):
135+
print("❌ black failed")
136+
success = False
137+
else:
138+
print("✅ black completed")
139+
140+
if success:
141+
print("\n🎉 All formatting tools completed successfully!")
142+
return 0
143+
else:
144+
print("\n❌ Some formatting tools failed. Please check the output above.")
145+
return 1
146+
147+
148+
if __name__ == "__main__":
149+
sys.exit(main())

tritonparse/utils.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -6,9 +6,6 @@
66
from pathlib import Path
77
from typing import Optional
88

9-
# argument parser for OSS
10-
parser = None
11-
129
from .common import (
1310
copy_local_to_tmpdir,
1411
is_fbcode,
@@ -19,6 +16,9 @@
1916
)
2017
from .source_type import Source, SourceType
2118

19+
# argument parser for OSS
20+
parser = None
21+
2222

2323
def init_parser():
2424
global parser

0 commit comments

Comments
 (0)