Skip to content

Commit 2b7b9ed

Browse files
tonytony
committed
feat(models): Add Pydantic models and update validator to use them
1 parent 1ad1045 commit 2b7b9ed

10 files changed

+1697
-77
lines changed

‎notes/2025-03-08 - test-audit - pydantic-plan.md

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# VCSPull Test Coverage Audit and Pydantic Integration Plan
2+
3+
## Overview
4+
5+
VCSPull has a good overall test coverage of 85%, but certain modules like the validator need improvement. This updated plan outlines how to enhance the codebase using Pydantic for better validation and type safety.
6+
7+
## Coverage Metrics
8+
9+
```
10+
Name Stmts Miss Branch BrPart Cover Missing
11+
------------------------------------------------------------------------------------
12+
conftest.py 39 8 4 1 79% 31-32, 91-98
13+
src/vcspull/_internal/config_reader.py 39 5 12 3 84% 50, 69, 114, 160, 189
14+
src/vcspull/cli/sync.py 85 14 34 11 79% 29, 61, 76->78, 81, 89, 91, 109-111, 115, 129-130, 132-133, 142, 151->153, 153->155, 160
15+
src/vcspull/config.py 148 10 88 13 89% 105, 107->110, 110->117, 121, 128-131, 151->153, 220->235, 266, 281, 307, 342->344, 344->347, 424
16+
src/vcspull/log.py 55 8 4 1 85% 39, 67-96, 105-106
17+
src/vcspull/validator.py 18 6 16 6 65% 17, 21, 24, 27, 31, 34
18+
------------------------------------------------------------------------------------
19+
TOTAL 414 51 170 35 85%
20+
```
21+
22+
## Pydantic Integration Plan
23+
24+
### 1. Core Model Definitions
25+
26+
Replace the current TypedDict-based system with Pydantic models to achieve better validation and type safety:
27+
28+
1. **Base Models**
29+
- Create `RawConfigBaseModel` to replace `RawConfigDict`
30+
- Create `ConfigBaseModel` to replace `ConfigDict`
31+
- Implement field validations with descriptive error messages
32+
33+
2. **Nested Models Structure**
34+
- `Repository` model for repository configuration
35+
- `ConfigSection` model for config sections
36+
- `Config` model for the complete configuration
37+
38+
3. **Validator Replacement**
39+
- Use Pydantic validators instead of manual function-based validation
40+
- Implement field-level validators for URLs, paths, and VCS types
41+
- Create model methods for complex validation scenarios
42+
43+
### 2. Error Handling Integration
44+
45+
Enhance the exception system to work seamlessly with Pydantic validation:
46+
47+
1. **Exception Integration**
48+
- Create adapters between Pydantic validation errors and VCSPull exceptions
49+
- Enrich error messages with contextual information
50+
- Provide suggestions for fixing validation errors
51+
52+
2. **Error Reporting**
53+
- Improve error messages with field-specific context
54+
- Add schema validation details in error messages
55+
- Include path information in nested validation errors
56+
57+
### 3. Configuration Processing Updates
58+
59+
Update the configuration processing to leverage Pydantic models:
60+
61+
1. **Parsing and Loading**
62+
- Update config reader to return Pydantic models
63+
- Maintain backward compatibility for existing code
64+
- Add serialization methods for different output formats
65+
66+
2. **Path Handling**
67+
- Implement path validators with environment variable expansion
68+
- Add path normalization in model fields
69+
- Handle relative and absolute paths correctly
70+
71+
3. **URL Processing**
72+
- Add URL validators for different VCS schemes
73+
- Implement URL normalization in model fields
74+
- Add protocol-specific validation
75+
76+
## Testing Strategy
77+
78+
### 1. Model Testing
79+
80+
1. **Unit Tests for Models**
81+
- Test model instantiation with valid data
82+
- Test model validation errors with invalid data
83+
- Test model serialization and deserialization
84+
- Test backward compatibility with existing data structures
85+
86+
2. **Validation Logic Tests**
87+
- Test field validators individually
88+
- Test model validators for complex validations
89+
- Test conversion between different model types
90+
- Test error message generation and context
91+
92+
### 2. Integration Testing
93+
94+
1. **Config Loading Tests**
95+
- Test loading configurations from files with Pydantic models
96+
- Test backward compatibility with existing files
97+
- Test error scenarios and validation failures
98+
99+
2. **End-to-End Flow Tests**
100+
- Test CLI operations with Pydantic-based config handling
101+
- Test sync operations with validated models
102+
- Test error handling and recovery in full workflows
103+
104+
### 3. Regression Testing
105+
106+
1. **Migration Tests**
107+
- Ensure existing tests pass with Pydantic models
108+
- Verify that all edge cases are still handled correctly
109+
- Test performance impact of model-based validation
110+
111+
2. **Backward Compatibility Tests**
112+
- Test with existing configuration files
113+
- Ensure command-line behavior remains consistent
114+
- Verify API compatibility for external consumers
115+
116+
## Implementation Plan
117+
118+
### Phase 1: Core Model Implementation
119+
120+
1. **Create Base Pydantic Models**
121+
- Implement `models.py` with core Pydantic models
122+
- Add field validators and descriptive error messages
123+
- Implement serialization and deserialization methods
124+
125+
2. **Update Types Module**
126+
- Update type aliases to use Pydantic models
127+
- Create Protocol interfaces for structural typing
128+
- Maintain backward compatibility with TypedDict types
129+
130+
3. **Validator Integration**
131+
- Replace manual validators with Pydantic validators
132+
- Integrate with existing exception system
133+
- Improve error messages with context and suggestions
134+
135+
### Phase 2: Config Processing Updates
136+
137+
1. **Update Config Reader**
138+
- Modify config reader to use Pydantic parsing
139+
- Update config loading functions to return models
140+
- Add path normalization and environment variable expansion
141+
142+
2. **Sync Operations Integration**
143+
- Update sync operations to use validated models
144+
- Improve error handling with model validation
145+
- Add type safety to repository operations
146+
147+
3. **CLI Updates**
148+
- Update CLI modules to work with Pydantic models
149+
- Improve error reporting with validation details
150+
- Add schema validation to command line options
151+
152+
### Phase 3: Testing and Documentation
153+
154+
1. **Update Test Suite**
155+
- Update existing tests to work with Pydantic models
156+
- Add tests for model validation and error scenarios
157+
- Implement property-based testing for validation
158+
159+
2. **Documentation**
160+
- Document model schemas and field constraints
161+
- Add examples of model usage in docstrings
162+
- Create API documentation for Pydantic models
163+
164+
3. **Performance Optimization**
165+
- Profile model validation performance
166+
- Optimize critical paths if needed
167+
- Implement caching for repeated validations
168+
169+
## Expected Benefits
170+
171+
1. **Improved Type Safety**
172+
- Runtime validation of configuration data
173+
- Better IDE autocomplete and suggestions
174+
- Clearer type hints for developers
175+
176+
2. **Better Error Messages**
177+
- Specific error messages for validation failures
178+
- Context-rich error information
179+
- Helpful suggestions for fixing issues
180+
181+
3. **Reduced Boilerplate**
182+
- Less manual validation code
183+
- Automatic serialization and deserialization
184+
- Built-in schema validation
185+
186+
4. **Enhanced Maintainability**
187+
- Self-documenting data models
188+
- Centralized validation logic
189+
- Easier to extend and modify
190+
191+
## Metrics for Success
192+
193+
1. **Type Safety**
194+
- Pass mypy in strict mode with zero warnings
195+
- 100% of functions have type annotations
196+
- All configuration types defined as Pydantic models
197+
198+
2. **Test Coverage**
199+
- Overall test coverage > 90%
200+
- Core modules coverage > 95%
201+
- All public APIs have tests
202+
203+
3. **Documentation**
204+
- All public APIs documented
205+
- All Pydantic models documented
206+
- Examples for all major features

‎notes/2025-03-08 - test-audit - test plan.md

Lines changed: 12 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -15,24 +15,20 @@ Throughout this plan, we'll ensure all code follows these standards:
1515
- Always import typing as a namespace: `import typing as t`
1616

1717
2. **Mypy Configuration**
18-
- Use strict mode (`--strict`) for mypy checking
19-
- Enable all error checks in the mypy configuration:
20-
```
21-
[mypy]
18+
- ✓ Strict mode is already enabled in `pyproject.toml` under `[tool.mypy]`
19+
- ✓ The project uses the following mypy configuration:
20+
```toml
21+
[tool.mypy]
2222
python_version = 3.9
23-
warn_return_any = True
24-
warn_unused_configs = True
25-
disallow_untyped_defs = True
26-
disallow_incomplete_defs = True
27-
check_untyped_defs = True
28-
disallow_untyped_decorators = True
29-
no_implicit_optional = True
30-
strict_optional = True
31-
warn_redundant_casts = True
32-
warn_unused_ignores = True
33-
warn_no_return = True
34-
warn_unreachable = True
23+
warn_unused_configs = true
24+
files = [
25+
"src",
26+
"tests",
27+
]
28+
strict = true
3529
```
30+
- All necessary error checks are enabled via the `strict = true` setting
31+
- Remaining task: Add CI checks for type validation
3632

3733
3. **Python 3.9+ Features**
3834
- Use built-in generic types when possible (but always access typing via namespace)

‎notes/TODO.md

Lines changed: 63 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -2,18 +2,35 @@
22

33
This document outlines the tasks needed to improve the test coverage, type safety, and overall quality of the VCSPull codebase based on the test audit plan.
44

5+
## Progress Update (2025-03-08)
6+
7+
- ⬜ Initiated Pydantic integration for improved type safety and validation
8+
- Plan to replace TypedDict with Pydantic models
9+
- Will use Pydantic validators instead of manual validation functions
10+
- Will leverage Pydantic's built-in error handling
11+
12+
- ⬜ Enhanced test coverage for the validator module
13+
- Will add tests for edge cases and complex configurations
14+
- Will ensure all tests pass with mypy in strict mode
15+
- Need to update tests to work with Pydantic models
16+
517
## 1. Type Safety Improvements
618

19+
- [ ] **Implement Pydantic Models**
20+
- [ ] Replace TypedDict definitions with Pydantic models
21+
- [ ] Add field validators with meaningful error messages
22+
- [ ] Use Pydantic's built-in error handling
23+
- [ ] Create model hierarchies for nested configurations
24+
725
- [ ] **Enhance Exception Hierarchy**
826
- [ ] Expand `exc.py` with specific exception types for different error scenarios
927
- [ ] Add rich exception metadata (path, url, suggestions, risk level)
10-
- [ ] Add proper typing to all exception classes
28+
- [ ] Integrate exceptions with Pydantic validation errors
1129

1230
- [ ] **Improve Type Definitions**
13-
- [ ] Revise `types.py` to use more specific types (avoid Any)
14-
- [ ] Create type aliases for complex types to improve readability
15-
- [ ] Add Protocols for structural typing where appropriate
16-
- [ ] Ensure all TypedDict definitions are complete and accurate
31+
- [ ] Revise `types.py` to use Pydantic models instead of TypedDict
32+
- [ ] Create model aliases for complex types to improve readability
33+
- [ ] Add Protocol interfaces for structural typing where appropriate
1734

1835
- [ ] **Type Annotation Completeness**
1936
- [ ] Audit all functions for missing type annotations
@@ -22,8 +39,8 @@ This document outlines the tasks needed to improve the test coverage, type safet
2239
- [ ] Properly annotate all class methods
2340

2441
- [ ] **Configure Strict Type Checking**
25-
- [ ] Set up `mypy.ini` with strict mode enabled
26-
- [ ] Enable all recommended type checking flags
42+
- [ ] Strict mode enabled in `pyproject.toml` under `[tool.mypy]`
43+
- [ ] Recommended type checking flags enabled
2744
- [ ] Add CI checks for type validation
2845

2946
## 2. Test Coverage Improvements
@@ -47,15 +64,16 @@ This document outlines the tasks needed to improve the test coverage, type safet
4764
- [ ] Test recovery mechanisms
4865

4966
- [ ] **Validator Module**
50-
- [ ] Add tests for each validation function
67+
- [ ] Update validator tests to use Pydantic models
68+
- [ ] Add tests for each validation function and validator
5169
- [ ] Test validation of malformed configurations
5270
- [ ] Ensure all validators throw appropriate exceptions
5371

5472
## 3. Test Infrastructure
5573

5674
- [ ] **Improve Test Fixtures**
5775
- [ ] Create reusable fixtures for common test scenarios
58-
- [ ] Implement typed fixtures using Protocols
76+
- [ ] Implement typed fixtures using Protocols and Pydantic models
5977
- [ ] Add fixtures for different repository types (git, svn, etc.)
6078

6179
- [ ] **Add Property-Based Testing**
@@ -76,10 +94,11 @@ This document outlines the tasks needed to improve the test coverage, type safet
7694
- [ ] Document possible exceptions and error conditions
7795
- [ ] Add type information to docstrings (NumPy format)
7896

79-
- [ ] **Add Type Documentation**
80-
- [ ] Document complex type behavior
81-
- [ ] Add clear explanations for TypedDict usage
82-
- [ ] Document Protocol implementations
97+
- [ ] **Add Pydantic Model Documentation**
98+
- [ ] Document model schemas and field constraints
99+
- [ ] Add examples of model usage
100+
- [ ] Document validation logic and error messages
101+
- [ ] Create API documentation for Pydantic models
83102

84103
## 5. Refactoring for Testability
85104

@@ -108,29 +127,46 @@ This document outlines the tasks needed to improve the test coverage, type safet
108127
## Prioritized Tasks
109128

110129
1. **Immediate Priorities**
111-
- Enhance exception hierarchy
112-
- Complete type annotations
113-
- Configure strict type checking
114-
- Add tests for core configuration functionality
130+
- [ ] Implement base Pydantic models for configuration
131+
- [ ] Integrate Pydantic validation with existing validation logic
132+
- [ ] Configure strict type checking
133+
- [ ] Update validator tests to work with Pydantic models
115134

116135
2. **Medium-term Goals**
117-
- Improve test fixtures
118-
- Add tests for CLI operations
119-
- Improve docstrings
120-
- Refactor for better testability
136+
- [ ] Improve test fixtures
137+
- [ ] Add tests for CLI operations
138+
- [ ] Improve docstrings
139+
- [ ] Refactor for better testability
121140

122141
3. **Long-term Objectives**
123-
- Implement property-based testing
124-
- Achieve 90%+ test coverage
125-
- Complete documentation overhaul
126-
- Integrate comprehensive CI checks
142+
- [ ] Implement property-based testing
143+
- [ ] Achieve 90%+ test coverage
144+
- [ ] Complete documentation overhaul
145+
- [ ] Integrate comprehensive CI checks
146+
147+
## Next Steps
148+
149+
1. **Create Pydantic Models**
150+
- Create base models for RawConfigDict and ConfigDict
151+
- Add validators for required fields and constraints
152+
- Implement serialization and deserialization methods
153+
154+
2. **Update Validation Logic**
155+
- Replace manual validators with Pydantic validators
156+
- Integrate Pydantic error handling with existing exceptions
157+
- Update validation tests to use Pydantic models
158+
159+
3. **Update Config Processing**
160+
- Update config processing to use Pydantic models
161+
- Ensure backward compatibility with existing code
162+
- Add tests for model-based config processing
127163

128164
## Metrics and Success Criteria
129165

130166
- [ ] **Type Safety**
131167
- [ ] Pass mypy in strict mode with zero warnings
132168
- [ ] 100% of functions have type annotations
133-
- [ ] No usage of `Any` without explicit justification
169+
- [ ] All configuration types defined as Pydantic models
134170

135171
- [ ] **Test Coverage**
136172
- [ ] Overall test coverage > 90%
@@ -139,5 +175,5 @@ This document outlines the tasks needed to improve the test coverage, type safet
139175

140176
- [ ] **Documentation**
141177
- [ ] All public APIs documented
142-
- [ ] All complex types documented
178+
- [ ] All Pydantic models documented
143179
- [ ] Examples for all major features

0 commit comments

Comments
 (0)