Analysis Pipeline
Pipeline Architecture
The Reasoning Engine executes a 7-step sequential pipeline where each step builds on the accumulated output of prior stages. Data flows forward through the pipeline, with each step reading the results of previous steps and contributing its own analysis to the growing body of evidence.
Step 0: Preflight
Purpose: Validate the input structure and determine the execution plan.
The preflight step performs two functions:
-
Schema validation (deterministic, no AI) -- verifies the objective is present and non-empty, validates series/dataset structure, and initializes tracking identifiers.
-
Step orchestration (optional, AI-driven) -- when enabled, the Step Orchestrator Agent evaluates the objective against the data dimensions and produces an execution plan specifying which steps to invoke.
Orchestration Decision Factors
| Factor | Influence |
|---|---|
| Objective type | Statistical queries skip pattern/correlation; exploratory questions run all steps |
| Dataset count | Single metric skips correlation (requires 2+ datasets) |
| Data points | Short time periods (< 7 days) may skip pattern detection |
| Content of question | Recommendation requests include domain knowledge; relationship questions include correlation |
Orchestration Modes
| Mode | Behavior |
|---|---|
| Pre-computed plan | Consumer provides stepPlan -- engine follows it exactly |
| Toggle overrides | Consumer enables/disables individual steps (runTemporalStep, runCorrelationStep, etc.) |
| AI orchestration | Step Orchestrator Agent determines the plan based on objective and data shape |
| Default | All steps execute unconditionally |
When orchestration is disabled (the default), every step runs. This guarantees comprehensive analysis at the cost of higher latency for simple queries.
Step 1: Temporal Analysis
Purpose: Compute statistical baselines and identify trends across time-series data.
Processing
-
Algorithmic computation (always runs):
- Per-metric statistics: mean, min, max, standard deviation
- Trend classification: increasing, decreasing, or stable
- Volatility assessment: consistency vs. erratic variation
-
Agent enhancement (layers on top):
- Contextualizes statistical results with domain awareness
- Identifies implications of data gaps
- Generates human-readable insight strings
Output Structure
{
"summary": "Narrative summary of temporal patterns",
"insights": ["Individual temporal observations"],
"stats": [
{
"id": "metric_id",
"mean": 3.7,
"min": 2.0,
"max": 5.0,
"trend": "increasing"
}
]
}
Key Behaviors
- Data gaps are flagged as potentially significant observations (e.g., absent weekend check-ins)
- Short datasets receive appropriately tentative language
- Trend detection uses linear regression slope to classify direction
- This step is rarely skipped because all downstream agents depend on its statistical output
Step 2: Pattern Recognition
Purpose: Detect behavioral signatures that statistical summaries alone do not surface.
Processing
-
Algorithmic detection:
- Trend patterns: upward/downward trends per metric
- Volatility patterns: high volatility (range/mean > 0.5), low volatility (range/mean < 0.1)
- Cross-series patterns: inverse trends across metrics (one rising while another falls)
-
Agent enhancement:
- Semantic pattern recognition beyond what algorithms capture
- Named behavioral patterns (e.g.,
stress_cycle,sleep_mood_connection) - Compound pattern detection across multiple metrics
Output Structure
{
"matches": ["stress_cycle", "sleep_mood_connection"],
"confidence": 0.72,
"detectedPatterns": [
{
"name": "stress_cycle",
"series": ["stress_level", "sleep_quality"],
"detail": "Stress peaks midweek consistently, coinciding with elevated sleep disturbance"
}
]
}
Confidence Calibration
| Data Volume | Confidence Range | Behavior |
|---|---|---|
| < 5 data points | 0.3 - 0.5 | Preliminary observations only |
| 5-14 data points | 0.5 - 0.8 | Patterns reported with noted limitations |
| 14+ data points | 0.7 - 0.95 | High-confidence pattern detection |
The maximum confidence is capped at 0.95 -- no pattern assertion claims absolute certainty.
Step 3: Correlation Analysis
Purpose: Compute and interpret statistical relationships between metrics.
Processing
-
Normalization (Normalizer Agent):
- Extract numeric values from mixed-type arrays
- Drop non-numeric entries while maintaining cross-dataset alignment
- Report normalization outcomes
-
Computation (deterministic):
- Pearson correlation coefficient for every dataset pair
- Minimum of 2 datasets required; single-metric inputs skip this step
-
Interpretation (Analysis Agent):
- Classify relationship strength and direction
- Identify cascade effects across metric chains
- Produce up to 3 actionable recommendations
Correlation Thresholds
| Coefficient Range | Classification |
|---|---|
| Absolute value of r at or above 0.7 | Strong |
| Absolute value of r between 0.4 and 0.7 | Moderate |
| Absolute value of r below 0.4 | Weak |
Output Structure
{
"pairs": [
{ "a": "stress_level", "b": "sleep_quality", "r": -0.82 }
],
"summary": "Interpretation of correlation findings",
"insights": ["Actionable observations"],
"strongPositive": 1,
"strongNegative": 1,
"moderate": 0,
"normalizedCount": 4
}
Cascade Detection
The Analysis Agent identifies chain effects where metrics are linked through intermediate variables:
stress -> sleep -> energy
When stress correlates negatively with sleep, and sleep correlates positively with energy, addressing the upstream factor (stress) may produce downstream benefits across multiple metrics. These cascades are flagged as high-leverage intervention points.
Step 4: Domain Knowledge Retrieval
Purpose: Retrieve and filter external knowledge relevant to the detected analytical findings.
Processing
-
Query generation (Query Generator Agent):
- Synthesizes temporal insights, patterns, and correlations into 2-5 targeted search queries
- Each query includes a rationale and priority score (1-10)
- Queries are optimized for web search: 5-10 words, domain-specific terminology
-
Web search execution:
- Highest-priority queries are executed first
- Results include content, source URLs, and metadata
-
Knowledge assembly:
- Converts the synthesized Perplexity answer and citations into knowledge items
- Preserves source metadata such as citation URLs and retrieval mode
- Falls back to local context items when Perplexity is not configured or search fails
Output Structure
{
"knowledge": [
{
"content": "Extracted insight text",
"url": "https://source-url.com/article",
"title": "Source article title",
"score": 0.85,
"metadata": {
"confidence": "high",
"evidence": "Supporting quote from source",
"analyzedAt": "2025-01-10T12:00:00Z"
}
}
]
}
Key Behaviors
- Queries target the specific findings detected in prior steps, not generic topics
- Retrieved content is filtered through the user's analytical context -- generic information without specific relevance is discarded
- Multiple search results are synthesized into a consolidated set of domain knowledge items
Step 5: Explanation Generation
Purpose: Synthesize all prior findings into a structured, user-facing narrative.
Processing
The Explanation Agent receives outputs from all prior steps and produces three structured outputs:
-
Narrative (2-4 paragraphs):
- Leads with the most significant finding
- Connects related patterns into cause-and-effect relationships where data supports it
- References specific statistics and coefficients
- Acknowledges limitations and data gaps
- Concludes with forward-looking recommendations
-
Key Points (3-5 items):
- Extracted takeaways suitable for summary display or dashboard rendering
-
Confidence Level (
low,medium,high):- Based on data volume and analytical coverage
Output Structure
{
"narrative": "Multi-paragraph explanation synthesizing all findings...",
"keyPoints": [
"Key finding 1 with supporting data",
"Key finding 2 with specific metrics",
"Key finding 3 with recommendation"
],
"confidence": "medium"
}
Fallback Behavior
If the Explanation Agent is unavailable, the step generates a template-based narrative from the computed data. This ensures the pipeline always produces output, though the template-based version lacks the nuanced synthesis of the AI-generated narrative.
Step 6: Validation
Purpose: Fact-check every claim in the narrative against the computed data.
Verification Process
The Validation Agent performs five categories of checks:
- Data consistency -- narrative claims match computed statistics (trend directions, values, data spans)
- Correlation accuracy -- relationships classified at the correct strength (strong/moderate/weak)
- Pattern validity -- referenced patterns were actually detected by the Pattern Agent
- Overgeneralization -- conclusions appropriately scoped for the available data volume
- Missing caveats -- sparse data receives uncertainty disclaimers
Output Structure
{
"valid": true,
"issues": []
}
When issues are found:
{
"valid": false,
"issues": [
"Narrative claims stress is decreasing, but temporal stats show increasing trend",
"Moderate correlation (r=0.5) described as 'strong'"
]
}
Post-Validation
After validation, the step assembles the final workflow result containing all step outputs and applies any configured output formatter. The assembled result is the canonical output returned to the consumer.
Execution Model
Dual-Mode Processing
Every step that involves AI agents follows the same pattern:
The algorithmic layer always produces a result. The agent layer enhances it with semantic understanding. If the agent fails, the algorithmic result is used as-is, and the gap is flagged for the consumer.
State Management
The pipeline uses a minimal state pattern to prevent payload growth:
- Each step returns only its own output plus essential tracking fields
- Prior step outputs are stored in the persistence layer and loaded on demand by downstream steps
- Large payloads (datasets, full narratives) are stored separately with reference identifiers
This prevents the workflow state from growing unboundedly as it passes through 7 steps with potentially large datasets.
Step Skip Behavior
When the Step Orchestrator or an explicit step plan determines a step is unnecessary:
| Skipped Step | Downstream Impact |
|---|---|
| Temporal | All downstream agents receive no statistical context (rare skip) |
| Pattern | Explanation narrative lacks behavioral pattern references |
| Correlation | No cross-metric relationship data available |
| Domain | Narrative relies solely on internal analysis without external context |
| Explanation | Never skipped -- always executes |
| Validation | Never skipped -- always executes |
The Explanation and Validation steps always execute regardless of orchestration decisions. This guarantees every output includes a synthesized narrative and fact-check verification.
Performance Characteristics
| Configuration | Approximate Latency | Use Case |
|---|---|---|
| All steps (no orchestration) | 15-25 seconds | Comprehensive analysis, exploratory questions |
| Orchestrated (3-4 steps) | 8-15 seconds | Targeted queries, statistical questions |
| Minimal (temporal + explanation + validation) | 5-10 seconds | Simple statistical lookups |
Latency varies based on model response time, data volume, and web search availability. The orchestrator's step selection is the primary mechanism for latency optimization.
Output Schema
Every pipeline execution produces a structured result object with these top-level fields. Step outputs are optional because the orchestrator may skip steps that are not relevant to the objective.
Identification
| Field | Type | Description |
|---|---|---|
taskId | string | Unique task identifier for tracing |
workflowRunId | string (optional) | Workflow run identifier |
objective | string | Original analysis objective |
outputProfile | string (optional) | Output profile used for formatting |
Step Outputs
| Field | Type | Description |
|---|---|---|
temporal | object (optional) | summary (string), insights (string[]), stats (array of {id, mean, min, max, trend}) |
pattern | object (optional) | matches (string[]), confidence (number, 0-1), detectedPatterns (array of {name, series, detail}) |
correlation | object (optional) | pairs (array of {a, b, r} where r is the correlation coefficient), summary (string) |
domain | object (optional) | knowledge (array of {content, url, title, score, sourceType}), source (string) |
explanation | object (optional) | narrative (string), recommendations (string[]), sources (string[]) |
validation | object (optional) | isValid (boolean), errors (string[]), warnings (string[]), factCheck (array of {claim, verdict, note}) |
result | object (optional) | summary (string), insights (string[]), recommendations (string[]), confidence (number, 0-1) |
Metadata
| Field | Type | Description |
|---|---|---|
formattedOutput | string (optional) | Output produced by a configured output formatter |
errors | array (optional) | Per-step errors: {step, message, type, severity} |
executedSteps | array (optional) | Step trace: {name, status, startedAt, completedAt, reason} where status is run, skipped, or error |
timing | object (optional) | {startedAt, completedAt, durationMs} |
Example Output
{
"taskId": "task-a1b2c3",
"objective": "Analyze sleep and mood trends over the past month",
"temporal": {
"summary": "Sleep duration shows a declining trend over 30 days. Mood scores are stable with high variance.",
"insights": [
"Sleep duration decreased by 12% over the period",
"Mood scores show no statistically significant trend"
],
"stats": [
{ "id": "sleep_hours", "mean": 6.8, "min": 4.5, "max": 9.2, "trend": "decreasing" },
{ "id": "mood_score", "mean": 7.1, "min": 3.0, "max": 10.0, "trend": "stable" }
]
},
"pattern": {
"matches": ["Weekend recovery pattern detected in sleep data"],
"confidence": 0.82,
"detectedPatterns": [
{ "name": "weekend-recovery", "series": "sleep_hours", "detail": "Sleep duration increases by 1.5h on weekends" }
]
},
"correlation": {
"pairs": [
{ "a": "sleep_hours", "b": "mood_score", "r": 0.64 }
],
"summary": "Moderate positive correlation between sleep duration and mood scores (r=0.64)."
},
"domain": {
"knowledge": [
{ "content": "Sleep deprivation is associated with reduced emotional regulation.", "title": "Sleep and Mood Meta-Analysis", "score": 0.91, "sourceType": "citation" }
],
"source": "perplexity-ai-synthesis"
},
"explanation": {
"narrative": "Over the past 30 days, sleep duration has been declining while mood scores remain variable. A moderate positive correlation (r=0.64) suggests that nights with less sleep tend to be followed by lower mood ratings. Weekend recovery patterns indicate compensatory sleep behavior.",
"recommendations": ["Maintain consistent sleep schedule", "Monitor mood on days following short sleep"],
"sources": ["Sleep and Mood Meta-Analysis"]
},
"validation": {
"isValid": true,
"errors": [],
"warnings": ["Mood dataset has 3 missing data points"],
"factCheck": [
{ "claim": "Sleep duration decreased by 12%", "verdict": "confirmed", "note": "Computed from temporal stats" }
]
},
"result": {
"summary": "Declining sleep trend with moderate mood correlation. Weekend recovery pattern detected.",
"insights": [
"Sleep is declining by ~12% over 30 days",
"Sleep and mood are moderately correlated (r=0.64)",
"Weekend recovery compensates for weekday sleep debt"
],
"recommendations": ["Stabilize weekday sleep schedule", "Track mood after poor sleep nights"],
"confidence": 0.78
},
"executedSteps": [
{ "name": "temporal", "status": "run" },
{ "name": "pattern", "status": "run" },
{ "name": "correlation", "status": "run" },
{ "name": "domain", "status": "run" },
{ "name": "explanation", "status": "run" },
{ "name": "validation", "status": "run" }
],
"timing": {
"startedAt": "2025-03-15T10:00:00Z",
"completedAt": "2025-03-15T10:00:18Z",
"durationMs": 18200
}
}