QRG Generation Deep Dive

Overview

The QRG generation workflow transforms Emergency Operation Plan (EOP) annexes into Quick Reference Guides through a 4-step AI pipeline executed per annex.

info

For a non-technical overview, see the QRG Generation Overview guide.

Technical Summary

What Actually Happens:

1. Fetch K12 context from MOEOP (EOP snapshot, organization, role tags)
2. Fetch annex forms from form-management-service
3. Clean form content (strip HTML, filter short fields, validate minimum length)
4. Run 4 AI steps per annex:
   • Classification (categorize annex type)
   • Group Extraction (find mentioned roles)
   • Role Mapping (map to K12 role IDs)
   • QRG Generation (create one QRG per role)
5. Save QRGs to K12 platform immediately after each annex
6. Store results in database with traceability

Key Technical Points:

• All form fields matter - No predetermined "important" fields; AI scans entire state.data object
• Template-agnostic - Works with any form structure; field names don't need to match expected patterns
• HTML stripping reduces tokens - ~70% reduction in typical payload size
• Validation happens in workflow - Empty forms reach AI steps and fail with descriptive errors
• Database tracks everything - FormStatus column captures completion state for debugging
• Error logging - Two tables (qrg_ai_calls for LLM calls, qrg_execution_errors for workflow failures)
• No fallback content - AI failures result in error status with detailed messages; no placeholder text is ever generated
• Organization constraint - QRGs can only be saved to FACILITY organizations. DISTRICT-linked EOPs fail fast unless skipK12OrgValidation=true is set.
• EOP-scoped role tags - Role mapping uses ONLY the tags assigned to the specific EOP (legacy-compatible behavior)
• No org-role-universe fetch in QRG - QRG processing resolves only the EOP-selected tags and skips organization-wide role-universe fetches for lower latency and fewer MOEOP calls
• Resilient role assignments - Invalid or unknown role IDs are filtered out; QRGs still generate with empty generalUserTagIds when mappings fail.

---

Role Tag Handling (Legacy Compatibility)

warning

Critical Implementation Detail: The role tags sent to the AI for mapping are scoped to the EOP, not the entire organization.

How Role Tags Are Collected

// Step 1: Fetch EOP snapshot
const snapshot = await fetchSnapshotByPlanId(eopId);
// snapshot.generalUserTagIds = ["tag-1", "tag-2", ...] (EOP-specific tags)

// Step 2: Resolve tag names from IAM
const eopTags = await fetchGeneralUserTagsByIds(snapshot.generalUserTagIds);
// Result: [{ id: "tag-1", name: "Teacher" }, { id: "tag-2", name: "Admin" }, ...]

// Step 3: Only these tags are sent to AI for role mapping
const availableRoles = eopTags.map(tag => ({ id: tag.id, name: tag.name }));

Why EOP Tags Only?

The legacy Python implementation filters role tags to only those specified in the request or EOP:

# Legacy Python behavior
general_user_tags = await self.k12_client.iam_service.get_general_user_tags(
    params=schemas.iam.GeneralUserTagsListFilter(
        include_organization_ids=None,
        general_user_tag_ids=user_tag_ids  # Only EOP-specific tags
    )
)

Benefits:

• AI can only map to roles actually relevant to this EOP
• Prevents QRGs being assigned to roles not configured for the plan
• Matches exactly how the legacy system works
• Reduces AI confusion from seeing 200+ organization roles when only 30-40 are relevant

Role Tag Priority

1. Request body (generalUserTagIds parameter) - If provided, only these tags are used
2. EOP Snapshot (snapshot.generalUserTagIds) - Default source
3. Organization fallback - Only used if snapshot has no tags (rare edge case)

Validation Safety Net

Even with EOP-scoped input, a safety filter ensures AI hallucinations don't create invalid assignments:

// After AI returns role mappings
const eopTagIds = new Set(eopTags.map(t => t.id));

for (const [groupName, roleIds] of Object.entries(roleMappings)) {
  // Filter to only valid EOP tag IDs (safety net for AI hallucination)
  const validRoleIds = roleIds.filter(id => eopTagIds.has(id));
  roleMappings[groupName] = validRoleIds;
}

---

Form Structure & Data Flow

MOEOP Form Architecture

Three-tier structure:

1. EOP Snapshot (document-service)

   • Contains list of annexes (form references)
   • Links to form-management-service via externalFormId
   • Contains generalUserTagIds (EOP-assigned roles)

2. Form Instance (form-management-service)

   • Actual form data filled by users
   • Links to template via formTemplateSnapshotId
   • Contains state.status (FormStatus) and state.data (content)

3. Form Template Snapshot (form-management-service)

   • Template definition with formTypeId
   • Used to identify annex types (filter out non-annex forms)

Form Schema (What We Receive)

// From form-management-service
{
  id: string;                          // Form instance ID
  formTemplateSnapshotId: string;      // Links to template
  name: string;                        // e.g., "Fire Safety Annex"
  state: {
    status: string;                    // 'COMPLETED', 'IN_PROGRESS', 'NOT_STARTED'
    data: {                            // User-entered content
      // Field keys are template-specific
      "introduction": "<p>Fire safety procedures...</p>",
      "procedures_before": "<ul><li>Check extinguishers</li></ul>",
      "procedures_during": "<ul><li>Evacuate building</li></ul>",
      "procedures_after": "<ul><li>Account for all</li></ul>",
      "roles_responsibilities": "Teachers lead evacuation...",
      // ... more fields depending on template
    }
  }
}

Field Filtering Logic

Fields That Matter (Used in QRG Generation):

Field	Purpose
state.status	FormStatus - distinguishes empty forms (IN_PROGRESS vs COMPLETED)
state.data.*	ALL content fields are candidates for QRG content
name	Annex identifier, used for QRG naming
id	Form instance ID for traceability

Fields That Don't Matter (Ignored):

• formTemplateSnapshotId - Only used during fetch to filter annex types
• Template metadata - Not passed to AI
• Timestamps - Not relevant to QRG content
• User metadata - Not included in generation

Data Cleaning Pipeline

Step 1: HTML Stripping

function stripHtmlFormatting(input: string): string {
  let cleaned = input
    .replace(/<style[^>]*>[\s\S]*?<\/style>/gi, '')  // Remove <style> tags
    .replace(/<[^>]+>/g, '')                          // Remove all HTML tags
    .replace(/&nbsp;/gi, ' ')                         // Decode &nbsp;
    .replace(/&[a-z]+;/gi, '')                        // Remove other entities
    .replace(/\s+/g, ' ');                            // Normalize whitespace
  return cleaned.trim();
}

Step 2: Field Validation

function cleanFormDataForAI(form: FormRecord): string {
  const cleanedData: Record<string, unknown> = {};
  
  // Validation 1: Check state.data exists
  if (!form.state?.data || typeof form.state.data !== 'object') {
    throw new Error('Form has no state.data');
  }
  
  // Validation 2: Check not empty object
  const dataEntries = Object.entries(form.state.data);
  if (dataEntries.length === 0) {
    throw new Error('Form state.data is empty (0 fields)');
  }
  
  // Step 3: Clean and filter each field
  for (const [key, value] of dataEntries) {
    if (typeof value === 'string') {
      const cleaned = stripHtmlFormatting(value);
      // Only keep fields with meaningful content (>= 10 chars)
      if (cleaned.trim().length >= 10) {
        cleanedData[key] = cleaned;
      }
    } else if (value && typeof value === 'object') {
      cleanedData[key] = value;  // Keep nested objects as-is
    }
  }
  
  // Validation 3: Check minimum total content length
  if (Object.keys(cleanedData).length === 0) {
    throw new Error('Form content is empty after cleaning');
  }
  
  return JSON.stringify({
    formId: form.id,
    formName: form.name,
    formStatus: form.state?.status,
    content: cleanedData,
  }, null, 2);
}

Token Savings Example:

• Original HTML field: <p><strong style="color: red;">Evacuate building</strong> using <span class="highlight">nearest exit</span></p> (120 chars)
• Cleaned field: Evacuate building using nearest exit (37 chars)
• ~70% reduction in typical form data

---

AI Pipeline Stages

Phase 1: Data Collection (collect_k12_moeop_data)

Purpose: Fetch all context needed for QRG generation

Process:

1. Resolve EOP snapshot from plan ID
2. Fetch EOP record (contains organizationId)
3. Fetch organization details
4. Collect role tags from EOP snapshot (NOT organization)
5. Fetch annex forms from form-management-service

Key Design Decision: Forms with empty data (data: {}) are NOT filtered out early. Instead, they proceed to the workflow where validation failures are properly logged.

Phase 2: Per-Annex AI Processing

For each annex, execute 4 sequential AI steps:

Step 1: Classification (classify_annex)

Input to AI: Cleaned form content

AI Task: Determine category and type:

• Category (FIRE, EVACUATION, LOCKDOWN, etc.)
• Type (EMERGENCY vs FUNCTIONAL)
• Confidence (high/medium/low)

Output:

{
  "category": "FIRE",
  "type": "EMERGENCY",
  "confidence": "high",
  "reasoning": "Form contains fire drill procedures and evacuation routes"
}

Step 2: Group Extraction (extract_groups)

Input to AI: Full annex content + classification

AI Task: Identify distinct groups/roles mentioned

Example extraction:

Form text: "Teachers check evacuation routes. Administrators coordinate with emergency services."

Output: ["Teachers", "Administrators"]

Step 3: Role Mapping (map_to_k12_roles)

Input to AI:

• Extracted group names (comma-separated)
• Available K12 roles (EOP-scoped tags injected into system prompt)

System prompt includes:

{
  "data": [
    {"id": "tag-uuid-1", "name": "Teacher"},
    {"id": "tag-uuid-2", "name": "Administrator"},
    {"id": "tag-uuid-3", "name": "Support Staff"}
  ]
}

AI Task: Map informal group names to official role IDs

Output:

{
  "roles": [
    {"user_role_name": "Teachers", "mapped_role_id": "tag-uuid-1"},
    {"user_role_name": "Administrators", "mapped_role_id": "tag-uuid-2"}
  ]
}

note

Validation: Any mapped role IDs not matching the EOP's tags are filtered out as a safety measure.

Automatic batching: When the EOP has more than ROLE_BATCH_SIZE role tags (default 12, configurable via K12_ROLE_BATCH_THRESHOLD env var), the system splits the available roles into batches and runs the AI mapping call for each batch in parallel via Promise.all. Results are merged with deduplication -- the same group name can map to roles found in different batches. This prevents LLM refusal on oversized prompts (too many roles in a single system prompt). See k12-role-utils.ts for the constant and qrg-generation.workflow.ts line ~775 for the implementation.

Step 4: QRG Generation (generate_qrgs)

Input to AI:

• All annex data (classification, groups, mappings)
• Full cleaned form content
• Role information

AI Task: Generate one QRG per mapped role with structured content blocks

Output per QRG:

{
  "qrgName": "Fire Safety - Teachers",
  "qrgType": "EMERGENCY",
  "instructionFor": "FIRE",
  "generalUserTagIds": ["tag-uuid-1"],
  "contentBlocks": {
    "before": "<ul><li>Know evacuation routes</li></ul>",
    "during": "<ul><li>Lead students to assembly point</li></ul>",
    "after": "<ul><li>Report to incident commander</li></ul>"
  }
}

Phase 3: K12 Save & Database Persistence

Immediate K12 Save: QRGs are saved to the K12 platform immediately after generation (not batched).

Database tracking:

Table	Purpose
qrg_requests	Request-level tracking (RequestId, Status, WorkflowRunId)
qrg_generation_annexes	Per-annex processing results with FormStatus
qrg_generated_items	One row per QRG with K12 QRG ID
qrg_ai_calls	Individual LLM invocation details
qrg_execution_errors	Workflow-level failures

---

Error Handling

Error Categories

Category	Types	Logged To
form_validation	empty_content, invalid_json	qrg_execution_errors
k12_api	api_failure, timeout, not_found	qrg_execution_errors
ai_call	model_error, timeout, rate_limit	qrg_ai_calls + qrg_execution_errors
workflow	step_failure, data_transform	qrg_execution_errors

Error Recovery

Per-Annex Isolation:

• Each annex processes independently
• One annex failure doesn't stop others
• Final status: SUCCESS (all annexes) or PARTIAL_SUCCESS / ERROR

No Fallback Content: When AI fails for any step:

• That step fails completely
• No placeholder or generic content is generated
• Detailed error messages explain what failed

---

API Response Formats

Status Values

Status	Meaning
SUCCESS	All AI calls succeeded, QRGs contain AI-generated content
ERROR	AI calls failed, no QRGs generated
PROCESSING	Workflow still running
CANCELLED	Job was cancelled

Kickoff Response

The kickoff endpoint returns immediately with a lowercase status:

{
  "requestId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "sessionId": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "status": "accepted"
}

Result Response (Success)

{
  "requestId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "sessionId": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "status": "SUCCESS",
  "data": {
    "root": [
      {
        "qrgName": "Fire Safety - Principal",
        "beforeContent": "<p>AI-generated content...</p>",
        "duringContent": "<p>AI-generated content...</p>",
        "afterContent": "<p>AI-generated content...</p>",
        "generalUserTagIds": ["tag-principal-uuid"],
        "qrgType": "EMERGENCY",
        "instructionFor": "FIRE",
        "k12QrgId": "saved-qrg-uuid"
      }
    ]
  },
  "aiStats": {
    "totalAnnexes": 6,
    "aiClassifySuccess": 6,
    "aiClassifyError": 0,
    "aiGroupsSuccess": 6,
    "aiGroupsError": 0,
    "aiRolesSuccess": 6,
    "aiRolesError": 0,
    "aiGenerateSuccess": 12,
    "aiGenerateError": 0
  }
}

Result Response (Error)

{
  "requestId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "sessionId": "b2c3d4e5-f6a7-8901-bcde-f12345678901",
  "status": "ERROR",
  "message": "AI processing failed: 6/6 annexes failed AI classification",
  "errors": [
    "[classify:Fire Safety Annex] AI classification failed: PermissionDenied",
    "[classify:Evacuation Annex] AI classification failed: PermissionDenied"
  ],
  "aiStats": {
    "totalAnnexes": 6,
    "aiClassifySuccess": 0,
    "aiClassifyError": 6
  }
}

---

Debugging Queries

Find all annexes for a request with role info

SELECT
  a.RequestId,
  a.AnnexId,
  a.AnnexName,
  a.FormStatus,
  a.ProcessingStatus,
  a.ClassifiedCategory,
  a.ExtractedGroupsJson,
  a.RoleMappingsJson,
  a.GeneratedQrgCount,
  a.ErrorMessage
FROM K12SAFETY.qrg_generation_annexes a
WHERE a.RequestId = @RequestId
ORDER BY a.ProcessedAtUtc;

Check role tag source

SELECT
  r.RequestId,
  r.RoleTagsJson,  -- The actual tags sent to AI
  r.RoleTagCount,
  r.RoleTagSource  -- 'eop_snapshot' or 'organization_fallback'
FROM K12SAFETY.qrg_request_roles r
WHERE r.RequestId = @RequestId;

Role mapping validation results

SELECT
  a.AnnexName,
  a.RoleMappingsJson,
  JSON_QUERY(a.RoleMappingsJson, '$') as MappedRoles,
  (SELECT COUNT(*) FROM OPENJSON(a.RoleMappingsJson)) as GroupCount
FROM K12SAFETY.qrg_generation_annexes a
WHERE a.RequestId = @RequestId
  AND a.ProcessingStatus = 'success';

Error summary by category

SELECT
  ErrorCategory,
  ErrorType,
  COUNT(*) AS ErrorCount,
  COUNT(DISTINCT RequestId) AS AffectedRequests
FROM K12SAFETY.qrg_execution_errors
WHERE OccurredAtUtc >= DATEADD(day, -7, GETUTCDATE())
GROUP BY ErrorCategory, ErrorType
ORDER BY ErrorCount DESC;

---

Configuration

Environment Variables

Variable	Purpose	Default
MOEOP_INTERNAL_BASE_URL	K12 MOEOP API endpoint	Required
MOEOP_INTERNAL_USERNAME	Basic auth username	Required
MOEOP_INTERNAL_PASSWORD	Basic auth password	Required
QRG_SAVE_TO_K12	Enable saving QRGs to K12	true in dev/prod, false in test/staging
K12_ANNEX_FORM_TYPE_IDS	Comma-separated form type IDs for annex filtering	Required

Code Locations

Component	Path
Workflow	packages/domain-k12/src/qrg-generation.workflow.ts
AI Prompts	packages/domain-k12/src/qrg-prompts.ts
MOEOP Client	packages/domain-k12/src/moeop-internal-client.ts
Data Collector	packages/domain-k12/src/k12-data-collector.ts
Storage	packages/domain-k12/src/qrg-dedicated-storage.ts
K12 Client	packages/domain-k12/src/clients/index.ts

---

Testing

Manual Testing

# Focused local checks
npm run test:run -- packages/domain-k12/tests/tools/qrg-generation-workflow-runner.test.ts
npm run test:run -- packages/domain-k12/tests/workflows/qrg-generation-workflow.test.ts

# Check role tag source in logs
# Look for: "available_roles_loaded" with "sourceType": "eop"

Key Assertions

1. Role tags come from EOP snapshot, not organization
2. AI only sees EOP-scoped roles (check availableRoleCount in logs)
3. Mapped role IDs are validated against EOP tags (check for role_validation_filtered logs)
4. QRGs save successfully to K12 (check for k12QrgId in response)

Integration Test Checklist

• Empty form handling (skipped with proper error message)
• Form with content (generates QRGs)
• Role mapping to valid tags (QRGs assigned correctly)
• Role mapping to invalid tags (filtered out, QRG still generates)
• K12 save success (k12QrgId populated)
• K12 save failure (error logged, generation continues)

---

Monitoring

Key metrics:

Metric	Query
Success rate	ProcessingStatus = 'success' / total
Empty form rate	Annexes with AnnexContentJson = '{}'
Avg QRGs per annex	AVG(GeneratedQrgCount)
AI call duration	AVG(DurationMs) from qrg_ai_calls
Role mapping accuracy	Groups with non-empty mappings / total groups

-- Daily success rate
SELECT
  CAST(ProcessedAtUtc AS DATE) AS ProcessDate,
  COUNT(*) AS TotalAnnexes,
  SUM(CASE WHEN ProcessingStatus = 'success' THEN 1 ELSE 0 END) AS Successful,
  AVG(GeneratedQrgCount) AS AvgQrgsPerAnnex
FROM K12SAFETY.qrg_generation_annexes
WHERE ProcessedAtUtc >= DATEADD(day, -30, GETUTCDATE())
GROUP BY CAST(ProcessedAtUtc AS DATE)
ORDER BY ProcessDate DESC;