Handbook Policy Review

Overview

Handbook policy review starts from two indexed sources:

• a selected authoritative MSBA policy set
• a selected handbook prepared for review

Both selectors are DB-backed by [HANDBOOK].[IndexedArtifactRegistry]. The registry row carries the index name, corpus/version IDs, source document IDs, policy codes, policy references, and canonical registry JSON used to build Azure AI Search filters.

Operator Sequence

1. Ingest or prepare the authoritative MSBA policy set so it appears as a policy row in [HANDBOOK].[IndexedArtifactRegistry].
2. Upload or choose a handbook PDF in the Admin Review Flow.
3. Prepare the handbook with preparationMode: "reviewable-unit-v2" so it is indexed and recorded as a handbook registry row.
4. Select the policy set and handbook in /admin/?dashboardScope=handbook&dashboardTab=policy-review-flow.
5. Start the review with POST /admin/handbook/policy-review/run.
6. Watch the 11 backend-owned stages from source resolution through profile/applicability, retrieval, evidence judgment/expansion, reasoning, criticism, remediation, validation, and publication.
7. Review both policy findings and checklist flags, record reviewer actions/feedback, and export when a saved run detail exists.

Pipeline

Hybrid review stages

The latency-aware path keeps retrieval and final decisions separate. Search produces candidate packets; batched evidence judgment decides relevance, scope, and completeness; proposed absences receive bounded neighbor/context expansion. Deterministic requirement-element and policy-relationship rules enforce the proposal, and a separate critic challenges consequential findings before any remediation language can be generated.

The UI displays these backend phases in order: resolve_sources, profile_handbook, plan_applicability, retrieve_evidence, judge_evidence, expand_evidence, reason_findings, criticize_findings, draft_remediation, validate_report, and publish_report. Reports expose hybrid provenance, cache/replay status, and source links; retrieval scores are never presented as authoritative proof.

Each semantic stage is independently configured as off, shadow, or enforce (the remediation guard is off/enforce). Turning the stages off restores the legacy decision path without deleting audit artifacts. Production uses durable SQL run state and Blob-backed stage/report artifacts; content hashes and stable fingerprints prevent incompatible or corrupt stage output from being replayed.

Policy Comparison Pass

The policy comparison pass works over policy-reviewable handbook segments. For each segment, it:

1. resolves selected handbook evidence from the handbook index
2. plans policy searches using cited policy codes, headings, concepts, keywords, transformed questions, and fallback strategies
3. searches the selected MSBA policy index with artifact-specific filters
4. sends retrieved policy evidence through the Handbook Policy Evidence Gate Agent
5. reconciles cited handbook evidence and cited MSBA policy evidence through the Handbook Policy Reconciliation Agent
6. emits a canonical finding status, severity, disposition, evidence refs, rejected evidence, and suggested wording when grounded

Canonical finding statuses include:

• aligns
• contradicts_policy
• handbook_missing_required_policy
• overstates_policy
• local_variation_allowed
• policy_expected_no_evidence
• not_policy_based
• insufficient_evidence
• needs_manual_review

Missing policy evidence is not treated as proof that the handbook is wrong. It produces unresolved/manual-review states unless the cited evidence supports a stronger classification.

Checklist Verification Pass

The checklist pass is item-driven, not chunk-driven. The runner loads student-handbook-checklist-2025 from packages/domain-handbook/src/checklists/student-handbook-checklist-2025.json and evaluates every checklist item against the full selected handbook index.

That distinction matters: content such as earthquake safety materials should not appear in every chunk. The checklist pass asks, item by item, whether the selected handbook contains evidence for that requirement anywhere in the indexed handbook. A checklist item can be marked missing only after the item-driven search cannot find supporting handbook evidence.

Checklist output is stored on runDetail.checklistReview and includes:

• expected and evaluated checklist item IDs
• one finding per checklist item
• selected handbook evidence refs and policy evidence refs where available
• metrics for expected, evaluated, satisfied, missing, contradicted, and could-not-verify counts
• coverage and checklist scores

The policy-review progress phase includes checking_checklist so operators can see when this pass is running.

Persistence

Table	Purpose
[HANDBOOK].[IndexedArtifactRegistry]	DB-backed ledger of indexed policy and handbook artifacts plus Search filters.
[HANDBOOK].[PolicyReviewJobs]	Async queue, progress state, retries, stale-running recovery, and job payloads.
[HANDBOOK].[PolicyReviewRuns]	Run records, selected corpus refs, lifecycle state, report JSON, and artifact refs.
[HANDBOOK].[PolicyReviewFindings]	Queryable findings with canonical status, severity, disposition, review state, and full finding JSON.
[HANDBOOK].[PolicyReviewReviewerActions]	Reviewer actions and quality feedback.
[HANDBOOK].[PolicyReviewUsageEvents]	Per-stage search/model/export telemetry used in run summaries.

UI Results

The Admin UI exposes two review surfaces:

Tab	URL	Role
Review Flow	/admin/?dashboardScope=handbook&dashboardTab=policy-review-flow	One-page product flow for choosing/uploading a handbook, preparing it, running review, seeing checklist flags, and exporting.
Policy Review	/admin/?dashboardScope=handbook&dashboardTab=policy-review	Detailed queue with artifact selectors, progress, saved runs, findings, evidence, reviewer actions, and feedback.

The BFF returns handbook.policy-review-admin-dashboard.v1 after every mutation so the UI can refresh policy/handbook options, active job state, saved runs, finding details, checklist review, usage summary, diagnostics, and export readiness in one projection.

Source And Verification

Primary sources:

• packages/domain-handbook/src/policy-review-run/**
• packages/domain-handbook/src/checklists/**
• packages/domain-handbook/src/agents/handbook-policy-review-agents.ts
• apps/admin-api/src/routes/admin/handbook-rag.ts
• apps/admin-ui/src/pages/dashboard/handbook-policy-review-tab.tsx
• docs/internal/handbook/policy-review-admin-api.md

Focused validation commands:

npx tsc -p packages/domain-handbook/tsconfig.json --noEmit --pretty false
npx vitest run packages/domain-handbook/tests/policy-review-run/policy-review-pipeline-runner.test.ts
npx vitest run packages/domain-handbook/tests/policy-review-run/policy-review-local-runner.test.ts
npm --prefix apps/admin-ui run build:server
npm --prefix apps/admin-ui run build:client