Testing

Overview

The repository uses Vitest, Playwright, Bruno, GitHub Actions, and evaluation runners. The goal is not to run every test on every pull request. The goal is to run the cheapest deterministic tests close to the changed code, then reserve deployed smoke checks, live provider checks, browser checks, and AI quality evaluations for explicit promotion points or scheduled maintenance.

Operator-facing results should be presented as Test Runs. The backing API and database contract currently use Evaluation Runs, but CI, Bruno, browser, live, and model-backed checks should emit or be converted into the same summary shape.

Test Tiers

Tier	Purpose	Examples	Normal trigger
t0	Static and policy gates	lint, typecheck, CI policy validation, no-live-LLM guard	Every relevant PR
t1	Deterministic behavior	package unit tests, route tests, schema tests, mocked regressions	Targeted PR lanes
t2	Small deployed smoke or critical integration	promoted Bruno smoke, minimal K12 EOP live smoke	Deploy candidate or manual PR check
t3	Broad scenario matrix	answer-quality matrix, provider drift, broader Bruno packs	Scheduled/manual
t4	Exhaustive full-suite run	full deterministic, integration, Bruno, browser, live/eval packs	Release hardening only

Live/model-backed tests must declare budgets and must not run automatically on ordinary PR pushes.

Main Commands

npm run test:plan
npm run test:shared
npm run test:k12
npm run test:sdac
npm run test:tap
npm run test:providers
npm run test:agents
npm run test:admin
npm run test:contracts
npm run test:infra
npm run test:ci:coverage

Browser and deployed checks:

npm run test:admin:e2e
npm run test:widget:e2e
npm run bruno:test:smoke
npm run bruno:test:ci

App- and workload-specific checks:

npm run test:handbook
npm run test:widget
npm run typecheck:handbook-app
npm run typecheck:health-aggregator
npm run contracts:ingestion:check
npm run contracts:widget:check
npm run contracts:bruno

Catalog and maintenance checks:

npm run evaluation:catalog:validate
npm run evaluation:suite-catalog:validate
npm run maintenance:project -- --lane tests --force

Pull Request Selection

PR CI is lane-driven. scripts/ci/resolve-validation-scope.mjs maps changed files to policy lanes from .github/code-assist/policy.yml. scripts/ci/build-test-plan.mjs converts that scope into a test-plan.json artifact with selected and skipped suites. Main pushes run the broad quality gates and six non-coverage Vitest shards; coverage remains in the separate coverage-check.yml workflow.

For normal PR pushes:

• deterministic t0 and targeted t1 checks can block the PR
• K12, SDAC, shared, and other independent lanes run in parallel when selected
• the largest deterministic lanes use stable Vitest shards
• live/model-backed suites are skipped unless manually requested
• superseded PR runs are cancelled by workflow concurrency

For example, a K12 lookup-chain change should select K12 deterministic shards and skip k12-eop-live-smoke because the PR budget for live/model calls is zero.

Bruno Contracts

Promoted Bruno requests are the executable endpoint contract for deployed API behavior. Source route/schema tests remain the implementation truth, while Bruno proves how endpoints are called from outside the app.

Bruno maintenance classifies requests as active, compatibility, candidate, stale, duplicate, false-pass risk, or manual-only. Only active promoted requests should be eligible for automated smoke gates.

Test Runs

Every deployment-influencing run should publish a summary with:

• project, environment, source, suite, tier, branch, commit, and workflow URL
• pass, fail, skipped, blocked, and warning counts
• observed duration and model cost where applicable
• scenario rows for scenario-oriented checks
• artifact pointers for JUnit, Bruno reports, screenshots, Playwright reports, coverage, summaries, and raw logs

Raw artifacts should stay in GitHub Actions or existing artifact storage. The Test Run database stores normalized metadata and pointers.

Scheduled Maintenance

Project Maintenance Nightly runs report-first maintenance lanes:

• documentation maintenance
• Bruno contract maintenance
• test maintenance
• evaluation catalog maintenance
• combined maintenance summary

Each lane is change-aware. If no relevant surface changed, it writes a small heartbeat report instead of running expensive work. Skills such as doc-update-scope, doc-update-targeted, doc-update-validate, validate-docs, and techdebt define the maintenance contracts; scheduled workflows use report mode by default.