Probe LLM Provider

Overview

The admin-probe-llm-provider tool is a comprehensive diagnostic tool for testing any configured LLM provider. It supports both azure-ai-inference and azure-ai-foundry provider types, reasoning models (GPT-5.x), all sampling knobs, function calling, streaming, and a raw providerOptions escape hatch.

Tool ID: admin-probe-llm-provider

The examples use the direct runtime/operator route. Replit/Admin UI browser code should call the Admin UI BFF instead: local /admin-api/*, testing /dashboard/admin-api/*.

Use this tool to:

• Verify a new model deployment works end-to-end before routing production traffic
• Test parameter behavior for reasoning models (parameter stripping, reasoning effort)
• Diagnose latency or response issues with full diagnostic output
• Validate streaming vs. non-streaming paths independently

Input Schema

Provider Targeting

Field	Type	Required	Description
provider	azure-ai-inference | azure-ai-foundry	No	Provider type. Defaults to AZURE_DEFAULT_PROVIDER env var.
modelName	string	No	Target model name. Resolved from DB if omitted.
deploymentName	string	No	Deployment name override.
endpoint	string (URL)	No	Override the capability endpoint URL.
apiKey	string	No	Explicit API key; bypasses managed identity.

Messages

Field	Type	Required	Description
messages	array	No	Full message array (role, content). Takes precedence over prompt.
prompt	string	No	Simple user prompt shortcut.
system	string	No	System prompt. Injected before user messages.

Sampling Parameters

Field	Type	Required	Description
temperature	float 0–2	No	Sampling temperature. Automatically stripped for reasoning models.
topP	float 0–1	No	Top-P nucleus sampling. Stripped for reasoning models.
frequencyPenalty	float -2–2	No	Frequency penalty. Stripped for reasoning models.
presencePenalty	float -2–2	No	Presence penalty. Stripped for reasoning models.
maxOutputTokens	int 1–65536	No	Maximum tokens to generate.
reasoningEffort	none|minimal|low|medium|high|xhigh	No	Reasoning depth for GPT-5.x models.

Advanced Parameters

Field	Type	Required	Description
seed	int	No	Fixed seed for reproducible outputs.
stopSequences	string[] (max 4)	No	Stop sequences.
responseFormat	{type: "text"|"json"}	No	Response format control.
tools	array (max 8)	No	Function calling tool definitions to test.
providerOptions	object	No	Raw vendor-specific parameters forwarded directly (e.g. logprobs, logit_bias).

Control Flags

Field	Type	Required	Description
skipCapabilityValidation	boolean	No	Bypass DB capability check. Requires explicit provider, endpoint, and modelName.
stream	boolean	No	Test the streaming path instead of the non-streaming generate path.

Output Schema

The response includes three sections:

probe

Resolved provider details: provider, modelId, deployment, endpoint, resolvedProvider (db-capability | explicit-config).

request

What was actually submitted and resolved: submittedParams, resolvedParams (after reasoning model stripping), messages, messageCount, and parameterInfo (whether it is a reasoning model, which params were stripped, supported reasoning efforts).

result

The LLM response: text, content, finishReason, usage (input/output/total/reasoning/cached tokens), warnings, providerMetadata, responseMetadata, latencyMs.

Usage Examples

Basic probe (uses DB default capability)

curl -X POST "${MASTRA_BASE_URL}/admin/tools/probe-llm-provider" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${BEARER_TOKEN}" \
  -d '{
    "data": {
      "prompt": "Say hello",
      "temperature": 0.7
    }
  }'

Test a specific deployment

curl -X POST "${MASTRA_BASE_URL}/admin/tools/probe-llm-provider" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${BEARER_TOKEN}" \
  -d '{
    "data": {
      "provider": "azure-ai-foundry",
      "modelName": "gpt-5",
      "deploymentName": "PROD-GPT5-CHAT",
      "prompt": "Explain how caching works"
    }
  }'

Test reasoning model with effort control

curl -X POST "${MASTRA_BASE_URL}/admin/tools/probe-llm-provider" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${BEARER_TOKEN}" \
  -d '{
    "data": {
      "modelName": "gpt-5o-reasoning",
      "prompt": "Solve: x^2 - 5x + 6 = 0",
      "reasoningEffort": "medium",
      "maxOutputTokens": 2000
    }
  }'

Note: temperature, topP, frequencyPenalty, and presencePenalty are automatically stripped for reasoning models. The parameterInfo.strippedParams field in the response shows which params were removed.

Test with explicit config (bypass DB)

curl -X POST "${MASTRA_BASE_URL}/admin/tools/probe-llm-provider" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${BEARER_TOKEN}" \
  -d '{
    "data": {
      "skipCapabilityValidation": true,
      "provider": "azure-ai-inference",
      "endpoint": "https://new-resource.services.ai.azure.com/",
      "modelName": "gpt-4o",
      "deploymentName": "TEST-GPT4O",
      "prompt": "Hello",
      "apiKey": "test-key"
    }
  }'

Environment Variables

Variable	Description
AZURE_DEFAULT_PROVIDER	Default provider when provider field is omitted
ADMIN_TEST_MODEL_NAME	Default model name fallback (overrides MODEL_DEFAULT_MODEL_NAME)
ADMIN_TEST_MODEL_DEPLOYMENT	Default deployment fallback
MODEL_DEFAULT_MODEL_NAME	System-wide default model name
MODEL_DEFAULT_DEPLOYMENT	System-wide default deployment