Ingestion And Indexing
Overview
Handbook ingestion has two related paths:
- Source-artifact ingestion stores source bytes and derived artifacts so product workflows can parse, classify, extract answers, generate, render, and compare handbooks.
- Index ingestion prepares policy and handbook evidence for Azure AI Search so RAG and policy review can retrieve source-backed evidence.
Policy review starts after the selected policy set and handbook have been indexed and recorded in [HANDBOOK].[IndexedArtifactRegistry].
Functional Operator Flow
For the one-page Review Flow, the operator path is:
- Upload or choose a handbook PDF.
- Create a PDF job with
preparationMode: "reviewable-unit-v2"orpolicyReviewPreparation: true. - Let the job run through extraction, chunking, embedding, indexing, and verification.
- Confirm the prepared handbook appears in the policy-review handbook selector.
- Select the authoritative MSBA policy set from the policy selector.
- Start policy review.
PDF Preparation Jobs
The Admin BFF route is:
POST /admin-api/admin/handbook/rag-presets/pdf-jobs
Browser code calls the same operation through:
/api/admin/handbook/rag-presets/pdf-jobs
The job body includes source information plus preset IDs:
| Field | Meaning |
|---|---|
preparationMode | reviewable-unit-v2; all Handbook uploads use the canonical reviewable-unit preparation path. |
policyReviewPreparation | Compatibility boolean equivalent to preparationMode: "reviewable-unit-v2" when true. |
chunkingPresetId | Chunking preset used by the PDF preparation pipeline. |
embeddingPresetId | Embedding preset used for vector generation. |
indexingPresetId | Indexing preset/schema selection used by the Admin workflow state. |
runNow | When true, schedules the async PDF pipeline immediately. |
sourceDocumentId / sourceDocuments | Source document identity for one file or a batch manifest. |
Job stages are:
| Stage | Meaning |
|---|---|
queued | Job exists but has not started. |
extracting | Document Intelligence or fixture extraction is running. |
chunking | Extracted spans are being normalized and chunked. |
embedding | Embeddings are being generated. |
indexing | Search documents are being written. |
verifying | Retrieval smoke checks are running and source refs are being bound. |
completed | Candidate bundle or policy-review source preparation completed. |
failed | Job failed with diagnostics. |
Source: apps/admin-api/src/routes/admin/handbook-rag.ts and apps/admin-api/src/lib/handbook-rag-workflow.ts.
Generic PDF RAG Preparation
Reviewable-unit preparation creates a candidate corpus/index bundle for RAG tuning and answer probes. The route calls runDistrictHandbookV2IngestionForSources(), which invokes the configured canonical reviewable-unit ingestion script with:
- project key
handbook - corpus id
handbook-pdf-corpus - candidate corpus bundle/version IDs
- selected chunking, embedding, and indexing presets
- Azure OpenAI embedding settings
- optional semantic configuration
- candidate index creation unless disabled
The completed job summary records index name, source document refs, counts, retrieval evidence, stage timings, sample document IDs, and artifact refs. Operators can promote or roll back candidate bundles from the RAG workbench.
Policy-Review Handbook Preparation
Policy-review handbook preparation is different from reviewable-unit candidate ingestion. When preparationMode is reviewable-unit-v2, the BFF prepares handbook evidence for the shared policy-review path:
- it requires a readable uploaded or staged PDF
- it counts all PDF pages
- it runs the profile import script with
--profile district-handbook-to-handbook-index-v2 - it uses live Document Intelligence for extraction
- it uses Azure OpenAI embeddings
- it writes to the resolved Handbook evidence index
- it validates or uses the existing Azure AI Search index according to
HANDBOOK_POLICY_REVIEW_INDEX_LIFECYCLE/HANDBOOK_AZURE_SEARCH_INDEX_LIFECYCLE - it writes candidate bundle, projected document, quality report, retrieval smoke, and end-to-end proof artifacts
The default shared handbook evidence index resolver checks:
HANDBOOK_POLICY_REVIEW_HANDBOOK_INDEX_NAMEHANDBOOK_EVIDENCE_INDEX_NAME- fallback
handbook-evidence-index-v2
Profile-Driven Index Ingestion
The package defines two active ingestion profiles in packages/domain-handbook/src/ingestion-profiles/index.ts.
| Profile ID | Corpus role | Accepted sources | Required metadata | Target schema |
|---|---|---|---|---|
msba-core-policy-to-policy-index-v2 | authoritative | pdf, text | policySetId, policySetVersionId, authorityName | policy-evidence-index-v2 |
district-handbook-to-handbook-index-v2 | supplemental | pdf, docx, html, markdown, text | districtId, handbookVersionLabel | handbook-evidence-index-v2 |
The exported profile workflow runs these stages:
resolve profile
-> preflight
-> extract
-> normalize
-> enrich
-> validate enrichment
-> chunk
-> ensure index
-> embed
-> project
-> write index
-> smoke test retrieval
-> publish
Important boundary: handbook-ingestion-profile is exported from the Handbook package for scripts/tests/direct callers, but it is not currently registered by loadHandbookArtifacts() as a runtime /handbook/workflows/* route.
Indexed Artifact Registry
Policy-review selectors read from [HANDBOOK].[IndexedArtifactRegistry]. This table is the backbone that turns indexed documents into selectable sources.
Important columns:
| Column | Meaning |
|---|---|
OwnerScopeId | Owner/scope partition for selector reads. |
ArtifactRole | policy or handbook. |
ArtifactId | Stable selectable artifact ID. |
LifecycleStatus | indexed rows appear in selectors; archived rows do not. |
SourceMode | Currently post-ingestion-source. |
Label | Operator-visible selector label. |
IngestionRunId | Ingestion run that produced the indexed artifact. |
CorpusId / CorpusVersionId / CorpusBundleId | Corpus identity used by RAG and policy-review refs. |
PipelineProfileId | Profile that produced the artifact, when applicable. |
TargetIndexName | Azure AI Search index used for retrieval. |
SourceDocumentIdsJson | Source document IDs to filter/query. |
PolicyCodesJson / ReviewablePolicyCodesJson / PolicyReferencesJson | Policy metadata used by review and filtering. |
RegistryRecordJson | Canonical dashboard registry record consumed by the Admin projection. |
The Admin policy-review selector path fails if the registry cannot be read. It does not silently scan local artifact folders as a fallback.
Technical Inputs And Outputs
| Stage | Inputs | Outputs |
|---|---|---|
| Intake | Uploaded/staged file, source IDs, preset IDs, metadata | HandbookPdfCandidateJob, candidate bundle shell, stage history |
| Extraction | PDF bytes and page range | source spans, extraction provider diagnostics |
| Enrichment | normalized spans and profile metadata | domain facts, policy refs, handbook refs |
| Chunking | enriched spans and chunking policy | chunks with source span refs |
| Embedding | chunks and embedding profile | vector values and token/cost estimates |
| Projection | chunks, facts, embeddings, target schema | Azure AI Search documents |
| Indexing | projected documents and target index | searchable policy/handbook evidence |
| Verification | keyword/vector/hybrid queries | retrieval smoke summary and sample document refs |
| Publication/registry | ingestion summary and projected docs | candidate bundle, artifact refs, DB registry row |
Required Services For Live Runs
Live PDF preparation and policy-review ingestion need:
- Azure Document Intelligence for PDF extraction
- Azure OpenAI or AI Foundry embedding configuration
- Azure AI Search endpoint and credentials or managed auth
- configured Handbook evidence/policy indexes
- SQL Handbook schema for registry-backed policy-review selectors
Local fixture paths exist for tests and demos, but production-shaped policy review should use live post-ingestion Search indexes and DB registry rows.