Verification gate: test runner #23

New issue

Open

opened 2026-04-07 20:24:13 +00:00 by austin · 0 comments

austin commented

2026-04-07 20:24:13 +00:00

Owner

Implement test-based verification of generated code.

Detect the project's test framework (pytest by default)
Given a modified symbol, find relevant tests via:
- Filename convention (test_foo.py for foo.py)
- Import analysis from the dependency graph
- Configurable test path mapping
Run only the relevant test subset, not the full suite
Parse test output into structured results (pass/fail per test, failure message, relevant line)
Timeout protection for tests that hang
Support disabling via config

More expensive than parse/lint checks but catches semantic errors. Run after cheaper checks pass.

Implement test-based verification of generated code. - Detect the project's test framework (pytest by default) - Given a modified symbol, find relevant tests via: - Filename convention (test_foo.py for foo.py) - Import analysis from the dependency graph - Configurable test path mapping - Run only the relevant test subset, not the full suite - Parse test output into structured results (pass/fail per test, failure message, relevant line) - Timeout protection for tests that hang - Support disabling via config More expensive than parse/lint checks but catches semantic errors. Run after cheaper checks pass.