Correlation Analysis

Overview

The Correlation Analysis step computes pairwise Pearson correlations between numeric datasets to identify relationships.

File: steps/correlation.step.ts

What It Does

1. Extracts numeric values from datasets
2. Computes Pearson correlation using Python processor (with TS fallback)
3. Classifies correlations by strength
4. Agent enhancement for interpretation

Correlation Classification

Strength	Absolute Value
Strong	|r| > 0.7
Moderate	0.4 < |r| ≤ 0.7
Weak	|r| ≤ 0.4

Correlation Processor

The workflow first tries a Python processor for accurate statistical computation:

python3 scripts/reasoning-engine/correlation-tool/correlation_processor.py

Python processor advantages:

• NumPy-based computation
• Handles missing data gracefully
• Returns p-values for significance

Falls back to TypeScript implementation if Python is unavailable.

Input

{
  series: Series[],         // Time-series with numeric values
  datasets?: Dataset[]      // Additional numeric datasets
}

Output

{
  correlation: {
    pairs: Array<{
      a: string,           // Dataset A ID
      b: string,           // Dataset B ID  
      r: number            // Pearson coefficient (-1 to 1)
    }>,
    summary: string,       // Human-readable summary
    insights?: string[]    // Agent-generated insights
  }
}

Pearson Calculation

// TypeScript fallback implementation
function pearsonCorrelation(x: number[], y: number[]): number {
  const n = x.length;
  const sumX = x.reduce((a, b) => a + b, 0);
  const sumY = y.reduce((a, b) => a + b, 0);
  const sumXY = x.reduce((acc, xi, i) => acc + xi * y[i], 0);
  const sumX2 = x.reduce((acc, xi) => acc + xi * xi, 0);
  const sumY2 = y.reduce((acc, yi) => acc + yi * yi, 0);
  
  const numerator = n * sumXY - sumX * sumY;
  const denominator = Math.sqrt(
    (n * sumX2 - sumX * sumX) * (n * sumY2 - sumY * sumY)
  );
  
  return denominator === 0 ? 0 : numerator / denominator;
}

Example Output

{
  "correlation": {
    "pairs": [
      { "a": "stress", "b": "mood", "r": -0.78 },
      { "a": "stress", "b": "sleep", "r": -0.65 },
      { "a": "mood", "b": "energy", "r": 0.82 },
      { "a": "sleep", "b": "energy", "r": 0.71 }
    ],
    "summary": "Strong negative correlation between stress and mood (r=-0.78). Strong positive correlations between mood-energy and sleep-energy.",
    "insights": [
      "Stress appears to be the primary driver of mood changes",
      "Sleep quality strongly influences next-day energy levels",
      "Consider stress management as the key intervention point"
    ]
  }
}

Interpretation Guide

r value	Interpretation
0.7 to 1.0	Strong positive - metrics move together
0.4 to 0.7	Moderate positive - some relationship
-0.4 to 0.4	Weak/no correlation
-0.7 to -0.4	Moderate negative - inverse relationship
-1.0 to -0.7	Strong negative - metrics move opposite

Skip Conditions

This step is skipped if:

• stepPlan exists and doesn't include 'correlation'
• Fewer than 2 numeric series available