Reasoning Engine

What Is the Reasoning Engine?

The Reasoning Engine takes structured data -- time-series metrics, categorical check-ins, numeric measurements -- and produces actionable, validated insights through a team of specialized AI agents. Each agent handles one analytical discipline (statistics, pattern recognition, correlation, narrative generation, fact-checking), and they work together in a coordinated pipeline.

Who Is This For?

Product teams use the reasoning engine to power data-driven features without building analytical logic from scratch. Feed it user data and a question, and it returns a structured narrative with confidence scores, key findings, and actionable recommendations.

Engineering teams use it as a composable building block. The engine is format-agnostic -- plug in an intake adapter for your data format and an output adapter for your target, and the same analytical pipeline works across different applications.

What Makes It Different?

Approach	Limitation	Reasoning Engine
Single LLM call	One model does everything; hard to tune individual aspects	10 registered agents, with the core workflow using the relevant subset per step
Rule-based analytics	Rigid; can't handle nuance or unexpected patterns	Dual-mode: algorithmic baselines + AI enhancement
Black box analysis	No verification of output accuracy	Built-in validation agent fact-checks every claim
Hardcoded formats	New data sources require core changes	Adapter pattern: add new formats without touching the engine

The Pipeline at a Glance

The engine runs a 7-step sequential pipeline, where each step builds on the previous:

Step	What Happens	Agent
0. Preflight	Validates input, optionally decides which steps to run	Step Orchestrator (optional)
1. Temporal	Computes statistics and identifies trends	Temporal Agent
2. Pattern	Detects behavioral patterns and semantic signals	Pattern Agent
3. Correlation	Computes and interprets metric relationships	Normalizer + Analysis
4. Domain	Retrieves relevant external knowledge	Query Generator + Perplexity search when configured
5. Explanation	Synthesizes findings into a user-facing narrative	Explanation Agent
6. Validation	Fact-checks every claim against the raw data	Validation Agent

Every step supports dual-mode processing: deterministic algorithms run first (reliable, reproducible), then AI agents layer semantic understanding on top (nuanced, contextual). This means the pipeline always produces a baseline result even if an agent is unavailable.

See detailed agent documentation for what each agent does, why it matters, and how they collaborate.

Format-Agnostic Architecture

The engine doesn't know or care where data comes from or where results go. Adapters handle all format-specific logic at both ends:

Intake Adapters transform source-specific data into a canonical schema:

JSON Array Adapter -- nested object arrays
API Payload Adapter -- structured API responses
CSV Adapter -- tabular data from external systems
Auto-Detection -- canHandle() method enables automatic adapter selection

Output Adapters transform canonical results into target formats:

Dashboard Formatter -- JSON optimized for UI rendering
Report Formatter -- structured data for PDF generation
API Response Formatter -- REST-friendly JSON
Plain Text Formatter -- human-readable summaries

Key benefit: Multiple applications can share the same analytical pipeline. Only the adapters change.

Design Principles

Source Agnosticism -- the core engine doesn't know where data originates. Intake adapters normalize any format into a canonical schema.
Target Flexibility -- output adapters transform canonical results into any required format (dashboards, reports, API responses).
Composable Intelligence -- ten registered agents are available to the reasoning artifacts, with step selection determining which agents participate in a given run.

System Architecture

High-Level Architecture Diagram

Component Summary

Component	Purpose	Location
Intake Adapters	Transform source-specific data → canonical input	adapters/intake-adapter.ts (src/mastra/workflows/reasoning-engine/adapters/intake-adapter.ts)
Canonical Input Schema	Contract for engine input	schemas/engine-input.schema.ts (src/mastra/workflows/reasoning-engine/schemas/engine-input.schema.ts)
Core Workflow	7-step orchestration pipeline	reasoning-engine.workflow.ts (src/mastra/workflows/reasoning-engine/reasoning-engine.workflow.ts)
Specialized Agents	10 registered agents for orchestration, analysis, retrieval, validation, and optional outer-flow phrasing	agents/reasoning-engine/ (src/mastra/agents/reasoning-engine/)
Canonical Output Schema	Contract for engine output	schemas/engine-output.schema.ts (src/mastra/workflows/reasoning-engine/schemas/engine-output.schema.ts)
Output Adapters	Transform canonical output → target format	adapters/output-adapter.ts (src/mastra/workflows/reasoning-engine/adapters/output-adapter.ts)
Formatters	Template-based output transformation	formatters/ (src/mastra/tools/reasoning-engine/formatters/)

Compatibility Layers: The Adapter Pattern

Why Adapters?

The Reasoning Engine serves multiple applications with vastly different data formats:

Large JSON arrays with nested object structures
Structured API payloads with defined schemas
CSV/XML files from external systems
Custom formats with application-specific structures

Rather than embedding format-specific logic in the core engine, we use adapters at both ends of the pipeline:

Intake Adapter Interface

Intake adapters implement a simple contract:

interface IntakeAdapter {
  id: string;           // Unique identifier
  name: string;         // Human-readable name
  description: string;  // What formats it handles

  // Auto-detection: Can this adapter handle the data?
  canHandle(rawData: unknown): boolean;

  // Transform raw data → canonical input
  transform(intake: RawIntake): Promise<IntakeResult>;

  // Optional: Validate before transformation
  validate?(rawData: unknown): { valid: boolean; errors?: string[] };
}

Key Design Decisions:

Auto-Detection — The canHandle() method enables automatic adapter selection. When multiple adapters are registered, the system tries each until one matches.
Metadata Preservation — Adapters return IntakeMetadata alongside the canonical input, preserving context like date ranges, record counts, and detected formats for downstream use.
Validation First — Optional validate() method provides detailed error messages before attempting transformation.

Canonical Input Schema

The canonical input is the "contract" between intake adapters and the core engine:

const reasoningEngineInputSchema = z.object({
  // Required: What should the analysis accomplish?
  objective: z.string().min(1),

  // Data inputs (at least one required)
  series: z.array(seriesSchema).optional(),           // Time-series data
  datasets: z.array(numericDatasetSchema).optional(), // Numeric arrays
  compactedRows: z.array(z.record(z.any())).optional(), // Raw context

  // Analysis guidance
  hints: z.array(z.string()).optional(),
  selectedNotes: z.array(selectedNoteSchema).optional(),
  weeklySummaries: z.array(z.record(z.any())).optional(),

  // Output configuration
  outputProfile: z.string().optional().default('default'),

  // Workflow metadata
  taskId: z.string().optional(),
  workflowRunId: z.string().optional(),
});

Output Adapter Interface

Output adapters transform canonical results into application-specific formats:

interface OutputAdapter<TOutput> {
  id: string;
  name: string;
  description: string;
  outputSchema: z.ZodType<TOutput>;  // Zod schema for validation

  // Transform canonical output → app format
  transform(
    output: ReasoningEngineOutput,
    context?: OutputAdapterContext
  ): Promise<OutputResult<TOutput>>;

  // Optional: Execute side effects (API calls, notifications)
  execute?(result: OutputResult<TOutput>, context?: OutputAdapterContext): Promise<void>;
}

Rationale: Why This Architecture?

Concern	Without Adapters	With Adapters
Adding new data source	Modify core engine	Add new intake adapter
Adding new output format	Modify core engine	Add new output adapter
Testing core logic	Need all format handling	Test with canonical mocks
Format-specific bugs	Debug entire pipeline	Debug isolated adapter
Business logic changes	Risk format regressions	Core isolated from formats

The adapter pattern ensures the Reasoning Engine remains stable while applications can evolve independently.

Orchestration Pipeline

Step-by-Step Flow

The Reasoning Engine executes 7 sequential steps, each building on previous results:

Step Definitions

Step Order	Step ID	Purpose	Agent
0	`preflight-intake`	Validate input and choose the step plan	Step Orchestrator Agent (optional)
1	`temporal-analysis`	Analyze time-series trends and statistics	Temporal Agent
2	`pattern-recognition`	Detect patterns in time-series data	Pattern Agent
3	`correlation-analysis`	Compute pairwise correlations	Analysis Agent
4	`domain-retrieval`	Retrieve domain knowledge through the domain retrieval tool	Query Generator Agent + Perplexity search when configured
5	`explanation-agent`	Generate narrative from analysis	Explanation Agent
6	`validation-agent`	Verify claims match data, assemble result	Validation Agent

note

Phrasing/tone adjustment is handled by the outer workflow (business logic layer), not by the reasoning engine. The reasoning engine produces structured analytical output; the outer workflow applies domain-specific phrasing.

State Management

The workflow uses a minimal state pattern to prevent JSON payload growth:

Key Behaviors:

Each step reads prior outputs from the database — Not from the passed state object
Each step returns only essential fields — TaskId, workflowRunId, objective, errors, and its own output
Large payloads are stored separately — Via data_ReasoningEnginePayloads with reference IDs

This prevents the workflow state from ballooning as it passes through the workflow with potentially large datasets.

Agent Architecture

Agent Registry

Ten registered agents are available to the reasoning engine artifacts:

Agent Specifications

Agent	Registry ID	Responsibility
Orchestrator	`reasoning-engine-orchestrator-agent`	Legacy/planning agent available in the artifact loader
Step Orchestrator	`reasoning-engine-step-orchestrator-agent`	Chooses enabled workflow steps during preflight when orchestration is enabled
Temporal	`reasoning-engine-temporal-summary-agent`	Analyzes time-series patterns and trends
Pattern	`reasoning-engine-pattern-recognition-agent`	Identifies behavioral patterns, trends, and volatility
Normalizer	`reasoning-engine-correlation-normalizer-agent`	Prepares data for correlation analysis
Analysis	`reasoning-engine-correlation-analysis-agent`	Interprets correlation findings
Query Generator	`reasoning-engine-query-generator-agent`	Generates focused Perplexity search queries from reasoning context
Domain Retrieval	`reasoning-engine-domain-retrieval-agent`	Available domain-knowledge agent; the committed domain step currently calls the domain retrieval tool
Explanation	`reasoning-engine-explanation-agent`	Generates narratives from analysis
Validation	`reasoning-engine-validation-agent`	Ensures narrative consistency
Phrasing Helper	`reasoning-engine-phrasing-agent`	Helper agent for outer workflows; not a committed `reasoningEngineWorkflow` step

Agent-Enhanced vs Algorithmic Processing

Each step supports dual-mode processing:

Algorithmic processing — Deterministic computation for reliable, reproducible results
Agent enhancement — AI-powered insights layered on top of algorithmic results

Example from pattern recognition:

// Algorithmic detection always runs first
const detectedPatterns = [];
for (const stat of stats) {
  if (stat.trend === 'increasing') {
    detectedPatterns.push({
      name: 'upward_trend',
      series: stat.id,
      detail: `${stat.id} is increasing...`
    });
  }
  // ... more algorithmic patterns
}

// Agent enhancement (if available)
const agent = await getPatternAgent(mastra);
if (agent) {
  const agentResponse = await callAgent(agent, {
    features: stats,
    detectedPatterns,
    patternHints: hints,
    objective,
  });
  // Merge agent insights with algorithmic results
}

Operational Configuration

Environment Variables

Variable	Purpose	Default
`AZURE_SQL_SERVER`	SQL Server hostname	(required for DB features)
`AZURE_SQL_DATABASE`	Database name	(required for DB features)
`REASONING_SQL_LOGGING`	Enable/disable SQL logging	Auto (enabled if SQL configured)
`REASONING_ASSUME_TABLES_EXIST`	Skip table-existence fallback behavior in the step data service unless set to `false`	`true`

Feature Flags

// SQL logging control
const SQL_LOGGING_ENABLED = process.env.REASONING_SQL_LOGGING
  ? process.env.REASONING_SQL_LOGGING !== 'false'
  : HAS_SQL_CONFIG;

// Step outputs are written through stepDataService to REASONING.data_WorkflowStepOutputs.
// Set REASONING_ASSUME_TABLES_EXIST=false only when local fallback creation/check behavior is needed.

Output Profiles

Control output formatting via the outputProfile input parameter:

Profile	Description	Formatter
`default`	Passthrough (canonical format)	`passthroughOutputAdapter`
`dashboard`	Dashboard-friendly format	`dashboard-formatter`
`api-v1`	REST API response format	`api-v1-formatter`
`text`	Plain text summary	`defaultTextOutputAdapter`

Phrasing Control

The phrasing agent is registered for consumers that want a warmer tone in an outer workflow or application layer. The committed reasoningEngineWorkflow does not include a phrasing step; it ends at validation and optional output formatting.

Development & Testing

Code Locations

Component	Path
Core Workflow	reasoning-engine.workflow.ts (src/mastra/workflows/reasoning-engine/reasoning-engine.workflow.ts)
Adapters	adapters/ (src/mastra/workflows/reasoning-engine/adapters/)
Schemas	schemas/ (src/mastra/workflows/reasoning-engine/schemas/)
Core Agents	agents/reasoning-engine/core/ (src/mastra/agents/reasoning-engine/core/)
Correlation Agents	agents/reasoning-engine/correlation/ (src/mastra/agents/reasoning-engine/correlation/)
Formatters	formatters/ (src/mastra/tools/reasoning-engine/formatters/)
DB DDL	Mastra.Platform.ReasoningEngineDb/ (db/sqlproj/Mastra.Platform.ReasoningEngineDb/model/Tables/)

Running Locally

# Start the dev server
npm run dev

# Run the full reasoning chain against a sample payload
# (expects a JSON file; Recovered exports are a convenient starting point)
npx tsx scripts/reasoning-engine/run-full-chain.ts examples/recovered/Recovered_Small.processed.dropIdentifiers-true.json --skip-validation

# Verify database connectivity through the admin health tool
curl -X POST "$MASTRA_BASE_URL/admin/tools/db-health-check" \
  -H "Content-Type: application/json" \
  -d '{"data":{"schema":"REASONING"}}'

Debugging Tips

Check step-by-step progress:

SELECT StepName, Status, DurationMs, Message
FROM REASONING.log_StepReasoningEngineTasks
WHERE TaskId = @taskId ORDER BY StepOrder;

View full step outputs:

SELECT StepName, OutputJson
FROM REASONING.data_WorkflowStepOutputs
WHERE TaskId = @taskId;

Find errors:

SELECT StepName, ErrorType, Severity, Message
FROM REASONING.log_Error
WHERE TaskId = @taskId ORDER BY StepOrder;

Console log prefixes:
- [reasoning-engine:preflight] — Preflight step
- [reasoning-engine:temporal] — Temporal analysis
- [reasoning-engine:pattern] — Pattern recognition
- [reasoning-engine:correlation] — Correlation analysis
- [reasoning-engine:explanation] — Explanation generation
- [reasoning-engine:validation] — Validation step

Troubleshooting

Symptom	Likely Cause	Resolution
"objective is required"	Empty or missing objective in input	Ensure intake adapter sets `objective`
Correlation step skipped	Less than 2 numeric datasets	Check dataset normalization in preflight
Agent not enhancing output	Agent not registered or unavailable	Verify agent registry IDs match
DB logging not working	SQL env vars not set	Set `AZURE_SQL_SERVER` and `AZURE_SQL_DATABASE`
Large payloads causing timeouts	Input is too large or unnecessary steps are running	Pre-aggregate data and use `stepPlan` or step toggles to skip unnecessary work
Validation failing	Narrative claims don't match data	Review explanation agent output

Next Steps

Meet the Agents

What each agent does, why it matters, and how they work together

Agent Technical Reference

Registry IDs, prompts, and schemas for all agents

Orchestration Pipeline

Step-by-step workflow coordination

Database Schema

Persistence tables and query patterns

What Is the Reasoning Engine?​

Who Is This For?​

What Makes It Different?​

The Pipeline at a Glance​

Format-Agnostic Architecture​

Design Principles​

System Architecture​

High-Level Architecture Diagram​

Component Summary​

Compatibility Layers: The Adapter Pattern​

Why Adapters?​

Intake Adapter Interface​

Canonical Input Schema​

Output Adapter Interface​

Rationale: Why This Architecture?​

Orchestration Pipeline​

Step-by-Step Flow​

Step Definitions​

State Management​

Agent Architecture​

Agent Registry​

Agent Specifications​

Agent-Enhanced vs Algorithmic Processing​

Operational Configuration​

Environment Variables​

Feature Flags​

Output Profiles​

Phrasing Control​

Development & Testing​

Code Locations​

Running Locally​

Debugging Tips​

Troubleshooting​

Next Steps​