Reasoning Engine Integration

The Recovered project uses the reasoning engine as its core analysis component through a sophisticated adapter pattern that ensures format independence and extensibility.

Architecture Overview

note

Key Design Principle: Normalization happens in the outer workflow (recoveredAnalysisWorkflow), NOT in the reasoning engine. The reasoning engine expects pre-normalized canonical format and focuses purely on analysis.

Preflight Step: Entry Point & Orchestration

The preflight step is the entry point to the reasoning engine workflow. It performs two critical phases:

Phase 1: Deterministic Validation (No Agents)

The preflight step validates the input using pure schema validation—no LLM calls, fully deterministic:

// Required fields
✓ objective: string (user's question)
✓ series: Series[] (time-series data)
✓ datasets: NumericDataset[] (numeric metrics)

// Optional fields
✓ compactedRows?: array (raw data sample)
✓ hints?: string[] (transformation context)
✓ meta?: object (metadata)

Validation ensures:

All required fields present
Correct data types and structure
At least one series or dataset exists
Timestamps in valid ISO format

warning

If validation fails, the workflow terminates immediately with a detailed error message. The reasoning engine expects pre-normalized canonical format—normalization must happen in the outer workflow (e.g., recoveredAnalysisWorkflow).

Phase 2: AI-Driven Step Orchestration (Optional)

After validation passes, the preflight step can optionally use an AI agent to select which analysis steps to run:

When orchestration is enabled (enableStepOrchestration: true):

Step orchestrator agent analyzes the user's question and data characteristics
Returns a stepPlan with selected steps and reasoning
Typical execution time: ~2-3 seconds for orchestration decision
Time savings: 30-50% by skipping irrelevant steps

When orchestration is disabled (default):

All 6 analysis steps run (temporal → pattern → correlation → domain → explanation → validation)
Predictable, comprehensive analysis
Total execution time: 18-25 seconds

Fallback behavior:

If orchestrator agent unavailable: falls back to running all steps
If orchestrator returns invalid response: logs warning, runs all steps
Ensures workflow always completes successfully

note

The preflight step does NOT perform normalization. It only validates and orchestrates. Normalization is the responsibility of the outer workflow's intake adapter.

For complete preflight documentation, see Preflight Step Reference.

Step Orchestration Details

warning

Step orchestration is opt-in. By default, all steps run (previous behavior). Set enableStepOrchestration: true in the workflow input to enable intelligent step selection.

When enabled, a step orchestrator agent analyzes the user's question and data characteristics to determine which analysis steps are necessary. This optimization reduces execution time and API costs by skipping irrelevant steps.

Enabling Orchestration

{
  "jsonData": "...",
  "userPrompt": "How have I been doing?",
  "enableStepOrchestration": true  // Enable smart step selection
}

Without orchestration (default):

All 6 steps run: temporal → pattern → correlation → domain → explanation → validation
Total time: 18-25 seconds
Predictable, comprehensive analysis

With orchestration (opt-in):

Only necessary steps run based on user's question
Total time: 8-20 seconds (depending on query type)
Faster, optimized for specific query types

Orchestration Logic

interface StepPlan {
  steps: Array<'temporal' | 'pattern' | 'correlation' | 'domain' | 'explanation' | 'validation'>;
  reasoning: string;  // Why these steps were selected
}

Available Steps:

temporal: Time-series trends, statistics, volatility
pattern: Behavioral cycles, recurring themes
correlation: Relationships between metrics
domain: Health/wellbeing knowledge base
explanation: Narrative synthesis (ALWAYS REQUIRED)
validation: Fact-checking (ALWAYS REQUIRED)

Example Orchestration Plans

Simple factual query

User: "What was my average stress?"

{
  "steps": ["temporal", "explanation", "validation"],
  "reasoning": "Simple statistical query requires only temporal analysis"
}

Skipped: pattern, correlation, domain (not needed for basic stats) Time saved: ~5-10 seconds

Exploratory question

User: "How have I been doing?"

{
  "steps": ["temporal", "pattern", "correlation", "domain", "explanation", "validation"],
  "reasoning": "Broad question requires full analysis pipeline"
}

Skipped: None (comprehensive analysis needed)

Correlation-specific

User: "Does stress affect my sleep?"

{
  "steps": ["temporal", "correlation", "explanation", "validation"],
  "reasoning": "Correlation query requires statistical relationship analysis"
}

Skipped: pattern (not relevant), domain (user wants data-driven answer) Time saved: ~3-5 seconds

Recommendation request

User: "What should I do to feel better?"

{
  "steps": ["temporal", "pattern", "domain", "explanation", "validation"],
  "reasoning": "Advice request requires domain knowledge and pattern detection"
}

Skipped: correlation (not asking about relationships) Time saved: ~2-4 seconds

Orchestration Constraints

Always include:

explanation (generates the answer)
validation (ensures accuracy)

Smart skipping rules:

Short time periods (<7 days) → skip pattern (insufficient data)
Single metric → skip correlation (need 2+ metrics)
Specific question → skip domain (user wants data, not advice)

Performance Impact

Query Type	Steps Run	Typical Time	Steps Skipped	Time Saved
Simple stats	3	8-12s	3	~50%
Correlation	4	12-15s	2	~30%
Exploratory	6	18-25s	0	0%
Recommendation	5	15-20s	1	~20%

Outer Workflow: Normalization & Adapters

note

Important: The sections below describe the outer workflow (recoveredAnalysisWorkflow), NOT the reasoning engine itself. Normalization happens before data enters the reasoning engine.

Purpose of Intake Adapters

Transform any Recovered export format into the canonical ReasoningEngineInput format that the reasoning engine expects. This normalization happens in the outer workflow's normalize step.

Adapter Interface

interface IntakeAdapter {
  id: string;
  name: string;
  description: string;
  
  // Can this adapter handle the provided data?
  canHandle(data: unknown): boolean;
  
  // Transform to canonical format
  transform(intake: RawIntake): Promise<IntakeResult>;
}

interface IntakeResult {
  input: ReasoningEngineInput;  // Canonical format
  metadata?: Record<string, unknown>;  // Transformation details
}

Recovered Intake Adapter

The recoveredIntakeAdapter handles two export formats:

Small Format (Recovered_Small.json)

[
  {
    "Created At": "December 12, 2025, 8:32 AM",
    "Label": "emotion-checkin",
    "Value": "{\"options\":[\"depressed\"],\"date\":\"2025-12-12T08:32:47.249Z\"}",
    "Kind": "emotion-checkin"
  }
]

Large Format (Recovered_Large.json)

[
  {
    "created_at": "July 21, 2025, 1:27 PM",
    "label": "high",
    "value": "16",
    "extra": "{\"eat-9-score\": \"16\"}",
    "user_id": "5,216"
  }
]

Transformation Process

The normalization step is the most critical part of the pipeline—it converts potentially thousands of individual check-in records into a compact, structured format optimized for AI reasoning.

Inspect field names to determine Small vs Large format:

Small: "Created At", "Label", "Value"
Large: "created_at", "label", "value"

Large Dataset Example:

Input: 661 records from Recovered_Large.json (190 KB)
Spanning: 6 months of daily check-ins
Check-in types: stress, mood, sleep, energy, cycle symptoms

Extract structured data from JSON strings in value field:

const parsed = JSON.parse(row.Value);
// { options: ["depressed"], notes: "...", date: "..." }

Field Extraction:

Stress levels: "very-low" | "low" | "medium" | "high" | "very-high"
Mood options: "depressed" | "apathetic" | "anxious" | "calm" | "joyful"
Sleep notes: Free-text descriptions of sleep quality
Energy scores: Numeric ratings (1-10)
Cycle symptoms: Arrays of physical/emotional/behavioral symptoms

Group check-ins by time period and aggregate into daily summaries:

bucketBy: 'day'  // Converts 661 records → ~30 daily datapoints

Aggregation Strategy:

Count metrics: Sum check-ins per day (e.g., 661 records → 30 days)
Categorical metrics: Mode (most frequent stress level that day)
Numeric metrics: Mean (average energy score per day)
Binary flags: Presence detection (sleep disturbance noted: yes/no)

Example Aggregation:

Input:  8 check-ins on Dec 12 (3 stress, 2 mood, 1 sleep, 2 energy)
Output: stress=3.5, mood=2.0, sleep_disturbed=true, energy=6.5

Large Dataset Compression:

661 individual records → 30 daily aggregates (95% reduction)
Original size: 190 KB → Canonical size: ~8 KB (96% reduction)
Enables efficient AI processing without losing temporal patterns

Convert categorical values to numeric indices for statistical analysis:

Stress Index (1-5 scale):

very-low → 1, low → 2, medium → 3, high → 4, very-high → 5

Mood Index (1-5 scale):

depressed → 1, apathetic → 2, anxious → 3, calm → 4, joyful → 5

Sleep Disturbance (binary):

no_notes → 0, has_notes → 1

Cycle Symptom Count:

count(physical_symptoms) + count(emotional_symptoms) + count(behavioral_symptoms)

Result: Each metric becomes a time-aligned numeric array:

stress_index:  [2, 1, 3, 2, 4, 3, 2]  // 7 days
mood_index:    [1, 2, 3, 2, 1, 2, 3]
energy_score:  [5, 6, 4, 7, 3, 5, 6]

Create temporal series for chart rendering in dashboard:

{
  id: "checkins-per-day",
  label: "Check-ins per day",
  points: [
    { t: "2025-12-12", v: 8 },  // 8 check-ins that day
    { t: "2025-12-13", v: 3 },
    { t: "2025-12-14", v: 5 }
  ]
}

Multiple Series:

Check-in frequency (activity level indicator)
Stress trend over time
Mood trend over time
Energy levels
Sleep quality

Store a sample of original check-ins in compactedRows for context:

compactedRows: [
  // First 50 records for LLM context
  { "Created At": "...", "Label": "stress", "Value": "..." },
  { "Created At": "...", "Label": "mood", "Value": "..." }
  // ... (truncated if > 50)
]

Why Truncate?

LLMs have token limits (~8K-128K context window)
Full 661 records = ~120K tokens (exceeds many model limits)
Aggregated datasets + sample = ~5K tokens (efficient)
Reasoning engine works on aggregates, not individual records

Include transformation details for transparency:

meta: {
  bucket: "day",
  rowCount: 661,           // Original record count
  dayCount: 30,            // Aggregated time periods
  approxInputChars: 195000, // Original file size
  compressionRatio: 24.4   // 661 / 30 ≈ 24x compression
},

hints: [
  "Derived datasets are per-day aggregates from 661 raw check-ins",
  "Missing days use last-observation-carried-forward (LOCF)",
  "Stress/mood indices are mode of that day's check-ins",
  "Energy scores are mean of that day's ratings"
]

Large Dataset Optimization

The normalization process is specifically designed to handle large exports efficiently:

Input Constraints:

Supports 1-10,000+ records
Tested with 661 records (190 KB)
Handles 6+ months of daily data

Output Guarantees:

Always produces ≤100 daily datapoints (even with years of data)
Canonical format stays under 10 KB regardless of input size
Preserves temporal resolution while reducing dimensionality

Performance:

Normalization time: ~200ms for 661 records
Memory usage: ~15 MB peak (Node.js process)
No database required (in-memory transformation)

Canonical Output Format

The adapter produces ReasoningEngineInput:

{
  objective: "How have I been doing?",  // User's question
  
  series: [
    {
      id: "checkins-per-day",
      label: "Check-ins per day",
      points: [{ t: "2025-12-12", v: 5 }, ...]
    }
  ],
  
  datasets: [
    {
      id: "stress-index",
      label: "Stress index (higher = more stress)",
      values: [2, 1]  // Per-day aggregates
    },
    {
      id: "mood-index",
      label: "Mood index (higher = better mood)",
      values: [1, 2]
    }
  ],
  
  compactedRows: [
    // Original raw check-ins for reference
  ],
  
  hints: [
    "Derived datasets are per-day aggregates from raw check-ins",
    "Missing days use last-observation-carried-forward"
  ],
  
  meta: {
    bucket: "day",
    rowCount: 8,
    dayCount: 2,
    approxInputChars: 4175
  }
}

Smart Passthrough

If the input is already in canonical format (has series and datasets), the normalize step skips transformation:

const isAlreadyCanonical = 
  inputData.rawData && 
  typeof inputData.rawData === 'object' &&
  ('series' in inputData.rawData || 'datasets' in inputData.rawData);

if (isAlreadyCanonical) {
  // Skip normalization, pass through with user's prompt
  return {
    ...inputData.rawData,
    objective: inputData.userPrompt
  };
}

This allows users to:

Paste preprocessed JSON directly
Test with curated datasets
Bypass normalization when iterating on analysis logic

Agent Configuration & Model Selection

Per-Agent Model Selection

Each reasoning engine agent can use a different Azure OpenAI deployment from AI Foundry. This is configured via the Admin Panel using agent profiles.

Available Agents:

reasoning-orchestrator - Original orchestrator (not step orchestrator)
reasoning-step-orchestrator - Smart step selection agent
reasoning-temporal-summary - Time-series analysis
reasoning-pattern-recognition - Pattern detection
correlation-tool-normalizer and correlation-tool-analysis - Correlation preparation and interpretation
reasoning-query-generator and reasoning-domain-retrieval - Domain knowledge retrieval
reasoning-explanation - Narrative synthesis
reasoning-validation - Fact-checking
reasoning-phrasing - Optional outer-flow phrasing helper

These are profile IDs. Live registry IDs use the reasoning-engine-...-agent forms documented in Reasoning Engine Agents.

Admin Panel Configuration

Use the Admin MCP Tools or Admin UI to configure each agent:

// Example: Configure temporal agent to use GPT-4o deployment
await adminUpdateAgentProfile({
  agentId: "reasoning-temporal",
  systemPrompt: "You are a temporal analysis agent...",
  llmOverrides: {
    deploymentName: "gpt-4o",           // AI Foundry deployment name
    modelName: "gpt-4o-2024-11-20",     // Optional: specific model version
    temperature: 0.3,                    // Lower temperature for analytical tasks
    maxOutputTokens: 2048
  }
});

Model Selection Strategy

Recommended deployment mapping:

Agent	Recommended Model	Reasoning
Temporal	GPT-4o-mini	Fast statistical analysis, low cost
Pattern	GPT-4o	Better pattern recognition in behavioral data
Correlation	GPT-4o-mini	Deterministic stats + simple interpretation
Domain	GPT-4o	Needs strong medical/health knowledge
Explanation	GPT-4o	Narrative quality matters most
Validation	GPT-4o	Fact-checking requires reasoning depth
Step Orchestrator	GPT-4o-mini	Simple classification task

Cost optimization:

Use GPT-4o-mini for analytical steps (temporal, correlation, step orchestrator)
Use GPT-4o for creative/reasoning steps (explanation, validation, domain)
Total cost: ~$0.02-0.05 per analysis (with mixed deployment strategy)

LLM Override Parameters

All agent profiles support these overrides:

interface LlmOverrides {
  deploymentName?: string;      // AI Foundry deployment name
  modelName?: string;           // Model identifier (optional)
  temperature?: number;         // 0.0-2.0 (default: 0.7)
  topP?: number;                // 0.0-1.0 (default: 1.0)
  frequencyPenalty?: number;    // -2.0 to 2.0 (default: 0)
  presencePenalty?: number;     // -2.0 to 2.0 (default: 0)
  maxOutputTokens?: number;     // Max response length
}

Testing Different Models

To test which model works best for each agent:

Baseline - Run with default deployment
Override - Update agent profile via Admin Panel
Compare - Review output quality and latency
Optimize - Fine-tune temperature/tokens per agent

# View current agent profiles
curl http://localhost:4111/admin/agent-profiles

# Update temporal agent to use GPT-4o-mini
curl -X POST http://localhost:4111/admin/agent-profiles \
  -H "Content-Type: application/json" \
  -d '{
    "agentId": "reasoning-temporal",
    "deploymentName": "gpt-4o-mini",
    "temperature": 0.3
  }'

These are local operator/runtime examples. Admin UI and Replit browser callers should use /admin-api/admin/agent-profiles locally or /dashboard/admin-api/admin/agent-profiles in testing.

Reasoning Engine: Analysis Steps

After the preflight step validates and orchestrates, the canonical input flows through the reasoning engine's analysis steps:

Analysis Step Chain

Analyzes time-series trends and statistics:

Trend direction (increasing, decreasing, stable)
Volatility (consistent vs erratic)
Anomalies (sudden spikes or drops)
Statistical summary (mean, min, max, std dev)

Detects behavioral/wellbeing patterns:

Stress buildup patterns
Mood cycles (weekly, monthly)
Sleep-energy correlations
Cycle-related symptom clusters

Computes relationships between metrics:

Pearson correlation coefficients
Lag analysis (delayed effects)
Significance testing
Example: stress ↔ energy: r = -0.82 (strong negative)

Injects relevant health knowledge:

Stress management techniques
Sleep hygiene best practices
Cycle phase characteristics
Energy-boosting strategies

Synthesizes findings into human narrative:

Starts with user's question
Integrates temporal trends
Adds pattern insights
Cites correlations
References domain knowledge
Provides actionable recommendations

Fact-checks the explanation:

Verifies statistical claims
Checks for contradictions
Flags low-confidence statements
Suggests corrections if needed

State Threading

The reasoning engine uses a state envelope pattern where each tool:

Receives ReasoningEngineState containing accumulated results
Extracts what it needs from state.steps
Performs its analysis
Returns updated state with new steps.{toolId}.output

interface ReasoningEngineState {
  requestId: string;
  objective: string;
  series: Series[];
  datasets: NumericDataset[];
  
  steps: {
    temporal?: { status, input, output, startedAt, completedAt },
    pattern?: { ... },
    correlation?: { ... },
    domain?: { ... },
    explanation?: { ... },
    validation?: { ... }
  };
  
  // Human-readable trace of which steps ran or were skipped and WHY
  executedSteps?: Array<{
    name: string; // step id
    status: 'run' | 'skipped' | 'error';
    startedAt?: string; // ISO timestamp
    completedAt?: string; // ISO timestamp
    reason?: string; // Optional explanation (e.g., 'Not selected by stepPlan' or 'Short time window < 7 days')
  }>
}

Output Format

The reasoning engine returns ReasoningEngineOutput:

{
  taskId: "task-abc123",
  objective: "How have I been doing?",
  
  temporal: {
    summary: "Stress decreased, energy stable but low",
    insights: [
      { metric: "stress-index", trend: "decreasing", confidence: 0.85 }
    ]
  },
  
  pattern: {
    matches: ["stress_buildup", "cycle_fatigue_cluster"],
    confidence: 0.72
  },
  
  correlation: {
    pairs: [
      { a: "stress", b: "energy", r: -0.82, significance: 0.95 }
    ]
  },
  
  domain: {
    recommendations: [
      "Consider mindfulness practices for stress reduction",
      "Energy-boosting activities aligned with cycle phase"
    ]
  },
  
  explanation: {
    narrative: "Your stress has decreased from Dec 12 to 13...",
    keyPoints: [
      "Stress: improving (trending downward)",
      "Energy: stable but low (3/10 both days)"
    ],
    confidence: "low"
  },
  
  validation: {
    isValid: true,
    warnings: ["Limited data (2 days) - confidence is low"]
  }
}

Executed steps trace example:

"executedSteps": [
  { "name": "preflight-intake", "status": "run", "startedAt": "2025-12-13T12:00:00Z", "completedAt": "2025-12-13T12:00:00Z" },
  { "name": "orchestration-decision", "status": "run", "reason": "Short time window < 7 days; single metric - skipping pattern and correlation" },
  { "name": "temporal", "status": "run" },
  { "name": "pattern", "status": "skipped", "reason": "Short time window < 7 days" },
  { "name": "correlation", "status": "skipped", "reason": "Only one numeric metric present" },
  { "name": "explanation", "status": "run" },
  { "name": "validation", "status": "run" },
  { "name": "workflow-result", "status": "run" }
]

Output Adapters: Business Logic

note

Important: This section describes the outer workflow's output adapter (recoveredAnalysisWorkflow), NOT part of the reasoning engine. The output adapter transforms reasoning engine results into dashboard-specific format.

Purpose

Transform the canonical reasoning engine output into Recovered-specific dashboard format.

Adapter Interface

interface OutputAdapter {
  id: string;
  name: string;
  description: string;
  outputSchema: z.ZodSchema;
  
  // Can this adapter handle the provided output request?
  canHandle(context: OutputAdapterContext): boolean;
  
  // Transform canonical output to target format
  transform(
    output: ReasoningEngineOutput,
    context: OutputAdapterContext
  ): Promise<OutputResult>;
}

interface OutputAdapterContext {
  objective: string;
  intakeMetadata?: Record<string, unknown>;
  appContext?: { appId: string; userId?: string };
  targetFormat?: string;
}

Recovered Output Adapter

The recoveredOutputAdapter produces dashboard-ready JSON:

{
  summary: {
    headline: "Stress improving, energy needs attention",
    confidenceLevel: "low",
    dateRange: "Dec 12 - Dec 13, 2025"
  },
  
  insights: [
    {
      type: "trend",
      metric: "Stress",
      direction: "decreasing",
      magnitude: "moderate",
      statement: "Your stress decreased from low to very-low"
    },
    {
      type: "alert",
      metric: "Energy",
      severity: "medium",
      statement: "Energy remains consistently low at 3/10"
    }
  ],
  
  trends: [
    {
      metric: "stress-index",
      direction: "decreasing",
      change: -1,
      percentChange: -50
    }
  ],
  
  correlations: [
    {
      metricA: "stress",
      metricB: "energy",
      strength: "strong",
      direction: "negative",
      coefficient: -0.82
    }
  ],
  
  recommendations: [
    "Continue stress management practices to maintain improvement",
    "Consider energy-boosting activities during low-symptom cycle days"
  ],
  
  meta: {
    checkInCount: 8,
    dayCount: 2,
    analysisDate: "2026-01-05T12:00:00Z"
  },
  
  answer: "Your stress has decreased from Dec 12 to 13, which is positive...",
  
  backendLogging: {
    mood_tone: "cautiously_optimistic",
    session_type: "general_checkin"
  }
}

Transformation Steps

Pull top 3-5 insights from temporal, pattern, and correlation steps

Create a concise summary using explanation narrative + top trends

Convert temporal analysis into dashboard-friendly trend objects

Map correlation coefficients to strength labels (weak/moderate/strong)

Include mood tone and session type for analytics tracking

Save output to RECOVERED.data_WorkflowOutputs linked to taskId

Data Flow Example

Input: Raw Small Format

[
  {
    "Created At": "2025-12-12T08:32",
    "Label": "emotion-checkin",
    "Value": "{\"options\":[\"depressed\"],\"date\":\"2025-12-12T08:32\"}",
    "Kind": "emotion-checkin"
  },
  {
    "Created At": "2025-12-12T08:36",
    "Label": "stress-checkin",
    "Value": "{\"options\":[\"low\"],\"date\":\"2025-12-12T08:36\"}",
    "Kind": "stress-checkin"
  }
]

After Phase 1: Canonical Format

{
  "objective": "How have I been doing?",
  "datasets": [
    { "id": "stress-index", "values": [2, 1] },
    { "id": "mood-index", "values": [1, 2] }
  ]
}

After Phase 2: Reasoning Engine Output

{
  "temporal": {
    "insights": [
      { "metric": "stress-index", "trend": "decreasing" }
    ]
  },
  "explanation": {
    "narrative": "Your stress has decreased from low to very-low..."
  }
}

After Phase 3: Dashboard Output

{
  "summary": {
    "headline": "Stress improving, mood stable"
  },
  "insights": [
    { "type": "trend", "metric": "Stress", "direction": "decreasing" }
  ],
  "answer": "Your stress has decreased..."
}

Adapter Registry

Both intake and output adapters are registered in global registries for extensibility:

// Register intake adapter
intakeAdapterRegistry.register(recoveredIntakeAdapter);

// Register output adapter
outputAdapterRegistry.register(recoveredOutputAdapter);

Custom Adapters

To add a new intake format (e.g., Apple Health export):

const appleHealthIntakeAdapter = createIntakeAdapter({
  id: 'apple-health-intake',
  name: 'Apple Health Intake Adapter',
  
  canHandle: (data) => {
    return Array.isArray(data) && 
           data[0]?.type === 'HKQuantityTypeIdentifier';
  },
  
  transform: async (intake) => {
    // Convert Apple Health records to canonical format
    const series = convertToSeries(intake.rawData);
    const datasets = convertToDatasets(intake.rawData);
    
    return {
      input: { objective, series, datasets },
      metadata: { source: 'apple-health', recordCount: data.length }
    };
  }
});

intakeAdapterRegistry.register(appleHealthIntakeAdapter);

Performance Considerations

Large Dataset Handling

For files with 500+ check-ins:

Time bucketing reduces data points:
- 661 raw records → ~30 daily aggregates
- Configurable bucket size (day/week/month)
DataRef pattern for DB storage:
- Enable with DB_STEP_DATA_ENABLED=true
- Stores large payloads in DB, passes references
- Reduces in-memory footprint
Selective extraction in output adapter:
- Extracts only top 5-10 insights
- Discards intermediate calculation data
- Dashboard receives ~8 KB JSON regardless of input size

Payload limits

The intake adapter limits representative notes and compacted rows, and the output adapter enforces field-level schema limits. The implementation does not currently enforce the older 500 KB input / 50 KB per-step / 20 KB output limits previously listed here.

Testing

Test Raw Small Format

curl -X POST http://localhost:4111/tap/workflows/recoveredAnalysisWorkflow \
  -H "Content-Type: application/json" \
  -d '{
    "jsonData": "[{\"Created At\":\"...\"}]",
    "userPrompt": "How have I been doing?"
  }'

Test Canonical Format

curl -X POST http://localhost:4111/tap/workflows/recoveredAnalysisWorkflow \
  -H "Content-Type: application/json" \
  -d '{
    "jsonData": "{\"objective\":\"...\",\"series\":[...]}",
    "userPrompt": "What are my stress patterns?"
  }'

View Adapter Logs

# Server terminal shows:
INFO [recovered-workflow] Phase 1: Starting normalization
INFO [recovered-workflow] Phase 1 Complete { detectedFormat: 'Small', dayCount: 2 }
INFO [recovered-workflow] Phase 2: Starting reasoning engine
INFO [recovered-workflow] Phase 2 Complete { taskId: 'task-abc123' }
INFO [recovered-workflow] Phase 3 Complete { headline: '...', insightsCount: 5 }

Reasoning Engine Overview - Core analysis engine
Recovered Workflow Guide - Full pipeline details
Database Overview - Persistence and schema

Architecture Overview​

Preflight Step: Entry Point & Orchestration​

Phase 1: Deterministic Validation (No Agents)​

Phase 2: AI-Driven Step Orchestration (Optional)​

Step Orchestration Details​

Enabling Orchestration​

Orchestration Logic​

Example Orchestration Plans​

Orchestration Constraints​

Performance Impact​

Outer Workflow: Normalization & Adapters​

Purpose of Intake Adapters​

Adapter Interface​

Recovered Intake Adapter​

Small Format (Recovered_Small.json)​

Large Format (Recovered_Large.json)​

Transformation Process​

Large Dataset Optimization​

Canonical Output Format​

Smart Passthrough​

Agent Configuration & Model Selection​

Per-Agent Model Selection​

Admin Panel Configuration​

Model Selection Strategy​

LLM Override Parameters​

Testing Different Models​

Reasoning Engine: Analysis Steps​

Analysis Step Chain​

State Threading​

Output Format​

Output Adapters: Business Logic​

Purpose​

Adapter Interface​

Recovered Output Adapter​

Transformation Steps​

Data Flow Example​

Input: Raw Small Format​

After Phase 1: Canonical Format​

After Phase 2: Reasoning Engine Output​

After Phase 3: Dashboard Output​

Adapter Registry​

Custom Adapters​

Performance Considerations​

Large Dataset Handling​

Payload limits​

Testing​

Test Raw Small Format​

Test Canonical Format​

View Adapter Logs​

Related Documentation​

Architecture Overview

Preflight Step: Entry Point & Orchestration

Phase 1: Deterministic Validation (No Agents)

Phase 2: AI-Driven Step Orchestration (Optional)

Step Orchestration Details

Enabling Orchestration

Orchestration Logic

Example Orchestration Plans

Orchestration Constraints

Performance Impact

Outer Workflow: Normalization & Adapters

Purpose of Intake Adapters

Adapter Interface

Recovered Intake Adapter

Small Format (Recovered_Small.json)

Large Format (Recovered_Large.json)

Transformation Process

Large Dataset Optimization

Canonical Output Format

Smart Passthrough

Agent Configuration & Model Selection

Per-Agent Model Selection

Admin Panel Configuration

Model Selection Strategy

LLM Override Parameters

Testing Different Models

Reasoning Engine: Analysis Steps

Analysis Step Chain

State Threading

Output Format

Output Adapters: Business Logic

Purpose

Adapter Interface

Recovered Output Adapter

Transformation Steps

Data Flow Example

Input: Raw Small Format

After Phase 1: Canonical Format

After Phase 2: Reasoning Engine Output

After Phase 3: Dashboard Output

Adapter Registry

Custom Adapters

Performance Considerations

Large Dataset Handling

Payload limits

Testing

Test Raw Small Format

Test Canonical Format

View Adapter Logs

Related Documentation