Validation Agent
Purpose
LLMs can produce plausible but factually incorrect claims. The Explanation Agent may state that "stress has been decreasing" when the temporal statistics show an increasing trend, or characterize a moderate correlation as "strong." The Validation Agent is a dedicated verification layer that systematically compares every factual claim in the narrative against the computed data, flagging inconsistencies before the output reaches the consumer.
Value Proposition
- Output integrity -- ensures every factual claim is supported by the underlying computed data
- Correlation accuracy -- verifies that statistical relationships are classified at the correct strength
- Appropriate qualification -- ensures sparse data receives uncertainty disclaimers rather than definitive claims
- Quality gate -- serves as the final checkpoint before analytical output leaves the engine
Verification Categories
The Validation Agent performs five categories of systematic checks:
1. Data Consistency
Verifies that narrative claims match computed statistics.
| Pass | Fail |
|---|---|
| Narrative correctly states "stress has been climbing" when stats show increasing trend | Narrative claims "stress decreasing" but stats show increasing trend (mean: 4.2, range: 3.0-5.0) |
| "Averaging approximately 3/10" when actual mean is 3.2 | "Averaging 7/10" when actual mean is 3.2 |
| "Over the past 5 days" matching actual data span | "Over the past month" when data covers 5 days |
2. Correlation Accuracy
Verifies that statistical relationships are correctly classified.
| Coefficient | Correct Classification | Common Mischaracterization |
|---|---|---|
| r > 0.7 | "Strong correlation" | -- |
| r = 0.4-0.7 | "Moderate correlation" | Elevated to "strong" |
| r < 0.4 | "Weak correlation" | Presented as "meaningful relationship" |
| Any r value | "Correlation" | Characterized as "causation" |
3. Pattern Validity
Confirms that referenced patterns were actually detected by the Pattern Agent.
| Pass | Fail |
|---|---|
References "stress cycle" when stress_cycle was detected | References "mood volatility" when that pattern was not detected |
| Claims scoped to detected patterns | Introduces patterns not present in the analysis output |
4. Overgeneralization
Assesses whether conclusions are appropriately scoped for the available data.
| Pass | Fail |
|---|---|
| "Stress tends to be higher on Mondays" (appropriately tentative) | "You always feel stressed on Mondays" (from only 2 data points) |
| "Early indications suggest..." (limited data acknowledged) | Definitive claims derived from 3 days of data |
5. Missing Caveats
Checks for appropriate uncertainty disclaimers given data availability.
| Pass | Fail |
|---|---|
| "With limited data (5 days), these are preliminary observations..." | Confident assertions from sparse data without acknowledging limitations |
| "This pattern may become clearer with continued tracking" | Preliminary findings presented as established conclusions |
Verification Philosophy
The agent applies strict factual standards with flexible stylistic tolerance:
| Strict | Flexible |
|---|---|
| Statistical accuracy | Word choice and phrasing |
| Trend direction claims | Narrative structure |
| Correlation strength classification | Writing style |
| Data-claim alignment | Tone and voice |
A narrative can be structured and phrased in many ways, but every factual assertion must be verifiable against the computed data.
Example
Input:
{
"narrative": "Your stress has been decreasing over the past week, which is encouraging.",
"temporalStats": {
"stress_level": {
"trend": "increasing",
"mean": 4.2,
"min": 3.0,
"max": 5.0
}
}
}
Output:
{
"valid": false,
"issues": [
"Narrative claims stress is decreasing, but temporal stats show 'increasing' trend (mean: 4.2, range: 3.0-5.0)",
"Positive framing is inappropriate when the underlying data indicates a concerning upward trajectory"
]
}
When validation fails, the workflow can either:
- Return the validation issues alongside the narrative for consumer-side handling
- Re-invoke the explanation agent with validation feedback (retry loop, if configured)
Pipeline Position
The Validation Agent is the final agent in the pipeline (Step 6). After validation, the output exits the reasoning engine and enters the consumer's business logic layer (e.g., dashboard formatting, tone adjustment).
Technical Reference
Identifiers
| Property | Value |
|---|---|
| Registry ID | reasoning-engine-validation-agent |
| Profile ID | reasoning-validation |
| Code | src/mastra/agents/reasoning-engine/core/validation.ts |
| Accessor | getValidationAgent() |
Output Schema
{
"valid": "boolean - True if narrative is consistent with computed data",
"issues": ["string[] - Identified inconsistencies (empty if valid)"]
}
Default Prompt
"You validate that the explanation narrative is consistent with the underlying data and findings. You receive the explanation narrative and key claims made about the user's wellbeing data. You also have access to the original findings (temporal stats, patterns, correlations). Your role is to catch inconsistencies, unsupported claims, or potential misinterpretations. Return strictly JSON:
{ valid: boolean, issues: string[] }. Be strict about factual accuracy but lenient about tone/phrasing. Flag if data is sparse without acknowledgment, if correlations are misinterpreted, or if claims don't match actual numbers."
Configuration
Update the prompt via Admin tools, SQL, or API targeting the reasoning-validation profile.