Domain Retrieval

Overview

The Domain Retrieval step retrieves relevant domain knowledge from the web using AI-enhanced search. It uses a query generator agent to create focused search queries from reasoning context, then executes web searches via Perplexity to gather current, synthesized information with citations.

File: src/mastra/tools/reasoning-engine/domain-retrieval/domain-retrieval.tool.ts

note

This step uses Perplexity API as the primary search solution with AI-powered synthesis and automatic citation tracking. The implementation includes parallel query execution, diversity checking, and smart query optimization.

Architecture

The domain retrieval flow involves four components:

Query Generator Agent - Analyzes reasoning context to generate 2-5 focused search queries with priorities, model selection, and search modes
Query Validation & Diversity Checking - Validates parameters and warns on similar queries (>70% Jaccard similarity)
Perplexity Search Tool - Executes parallel web searches (top 2 queries) with AI synthesis and citation tracking
Knowledge Aggregator - Merges results, deduplicates citations, and filters empty/generic responses

Current Implementation

// 1. Generate focused queries with Perplexity params
const queryGenAgent = await getQueryGeneratorAgent();
const contextSummary = `Objective: ${objective}\n\nFindings:\n${findingsSummary}`;
const queryGenResponse = await queryGenAgent.generate([{
  role: 'user',
  content: contextSummary
}]);

// 2. Extract and validate queries
const queries = extractQueriesFromResponse(queryGenResponse);
queries.forEach(q => {
  q.model = validModels.has(q.model) ? q.model : 'sonar';
  q.searchMode = validModes.has(q.searchMode) ? q.searchMode : 'internet';
});

// 3. Check query diversity
checkQueryDiversity(queries, logger);

// 4. Execute top 2 queries in parallel
const topQueries = queries.sort((a, b) => b.priority - a.priority).slice(0, 2);
const results = await Promise.all(
  topQueries.map(q => {
    let effectiveQuery = q.query;
    if (shouldUseRecencyFilter(q.query)) {
      effectiveQuery += ' (recent research 2024-2025)';
    }
    return perplexitySearchTool.execute({
      query: effectiveQuery,
      model: q.model || 'sonar',
      searchMode: q.searchMode || 'internet'
    });
  })
);

// 5. Filter empty/generic responses
const validResults = results.filter(r => 
  'success' in r && r.success && !isEmptyOrGenericResponse(r.answer)
);

// 6. Deduplicate citations and merge answers
const allCitations = new Set<string>();
validResults.forEach(r => r.citations.forEach(c => allCitations.add(c)));

const knowledge = [
  validResults[0].answer,  // Primary answer
  ...validResults.slice(1)
    .filter(r => r.answer.length > 200)
    .map(r => r.answer),    // Secondary answers
  ...Array.from(allCitations)  // All unique citations
];

Output Structure

{
  domain: {
    knowledge: Array<{
      content: string,
      title?: string,
      url?: string,
      score?: number,
      sourceType?: string
    }>,                           // Synthesized answers, citations, metadata, or fallback context
    source: 'perplexity-ai-synthesis' | 'stub' | 'stub-error-fallback',
    searchMetadata: {
      queriesExecuted: number,    // Number of parallel queries
      totalResponseTime: number,  // Combined search time in ms
      primaryQuery: {
        query: string,            // Top priority query
        model: string,            // sonar or sonar-pro
        searchMode: string,       // internet/academic/news
        rationale: string,        // Why this query
        priority: number          // Query priority (1-10)
      },
      citationCount: number,      // Total unique citations
      diversityWarnings?: string[] // Query similarity warnings
    }
  }
}

Example Execution

Input Context

{
  "objective": "Analyze quantum computing error correction trends",
  "temporal": {
    "insights": ["Error rates declining 15% quarterly"]
  },
  "correlation": {
    "insights": ["Strong correlation between qubit coherence and error rates"]
  }
}

Generated Queries

[
  {
    "query": "latest quantum error correction breakthroughs topological qubits 2025-2026",
    "rationale": "Targets time-sensitive progress in topological qubit error correction",
    "priority": 10,
    "model": "sonar-pro",
    "searchMode": "academic"
  },
  {
    "query": "quantum coherence time improvements error rates 2025",
    "rationale": "Explores correlation between coherence and error rates",
    "priority": 9,
    "model": "sonar",
    "searchMode": "internet"
  }
]

Output

{
  "domain": {
    "knowledge": [
      "Quantum error correction achieved significant milestones in 2026 with topological qubits demonstrating unprecedented stability...",
      "Recent advances in quantum coherence show 15% quarterly improvement in error rates, primarily driven by better qubit isolation techniques...",
      "https://riverlane.com/quantum-error-correction-2026",
      "https://spectrum.ieee.org/quantum-computing-coherence",
      "https://nature.com/articles/quantum-topology-advances"
    ],
      "source": "perplexity-ai-synthesis",
    "searchMetadata": {
      "queriesExecuted": 2,
      "totalResponseTime": 3420,
      "primaryQuery": {
        "query": "latest quantum error correction breakthroughs topological qubits 2025-2026",
        "model": "sonar-pro",
        "searchMode": "academic",
        "rationale": "Targets time-sensitive progress in topological qubit error correction",
        "priority": 10
      },
      "citationCount": 3
    }
  }
}

Components

Query Generator Agent

Registry ID: reasoning-engine-query-generator-agent
Profile ID: reasoning-query-generator
Code: src/mastra/agents/reasoning-engine/core/query-generator.ts

Analyzes temporal insights, patterns, and correlations
Generates 2-5 focused search queries
Assigns rationale and priority (1-10) to each query
Returns JSON with queries array

See Query Generator Agent documentation →

Perplexity Search Tool

Tool ID: base-perplexity-search
Code: src/mastra/tools/base/perplexity-search.tool.ts

Platform-level AI-powered web search tool
Supports multiple search modes (internet, academic, news, youtube, reddit)
Returns synthesized answers with automatic citation tracking
URL extraction fallback for reliable citation capture
Default system prompt optimized for inline citations

See Perplexity Search Tool documentation →

Configuration

Required Environment Variables

PERPLEXITY_API_KEY=your-api-key

Optional Tuning

PERPLEXITY_DEFAULT_MODEL=sonar              # or sonar-pro
PERPLEXITY_TIMEOUT_MS=60000                 # Request timeout
PERPLEXITY_DEBUG_TIMINGS=true               # Enable performance logging
PERPLEXITY_DEBUG_CITATIONS=true             # Debug citation extraction

Skip Conditions

This step is skipped if:

stepPlan exists and doesn't include 'domain'
Query generator fails to produce valid queries
All generated queries fail validation

When Perplexity is not configured or search fails, the tool returns local fallback knowledge with source: "stub" or source: "stub-error-fallback" so the workflow can still complete.

Implemented Features

✅ Parallel Query Execution

Top 2 queries executed concurrently via Promise.all()
Citation deduplication across all results
Secondary answer merging (>200 chars, score 0.85)

✅ Query Validation & Diversity

Model validation (sonar/sonar-pro)
Search mode validation (internet/academic/news/youtube/reddit)
Jaccard similarity checking (warns on >70% overlap)

✅ Smart Query Enhancement

Automatic recency filter for health/wellness queries
Appends "(recent research 2024-2025)" to relevant queries
Empty/generic response detection with fallback

✅ Enhanced Query Generation

LLM generates model selection per query
LLM generates searchMode per query
Rationale and priority scoring (1-10)

Future Enhancements

Vector Search Integration

Query internal wellbeing knowledge base
Combine Perplexity search with vector search results
Add clinical guidelines and research citations

Advanced Caching

Query result caching to avoid duplicate API calls
Semantic query similarity matching
Time-based cache invalidation

Overview​

Architecture​

Current Implementation​

Output Structure​

Example Execution​

Input Context​

Generated Queries​

Output​

Components​

Query Generator Agent​

Perplexity Search Tool​

Configuration​

Required Environment Variables​

Optional Tuning​

Skip Conditions​

Implemented Features​

✅ Parallel Query Execution​

✅ Query Validation & Diversity​

✅ Smart Query Enhancement​

✅ Enhanced Query Generation​

Future Enhancements​

Vector Search Integration​

Advanced Caching​