Probe LLM Provider

Overview

The admin-probe-llm-provider tool is a comprehensive diagnostic tool for testing any configured LLM provider. It supports both azure-ai-inference and azure-ai-foundry provider types, reasoning models (GPT-5.x), all sampling knobs, function calling, streaming, and a raw providerOptions escape hatch.

Tool ID: admin-probe-llm-provider

The examples use the direct runtime/operator route. Replit/Admin UI browser code should call the Admin UI BFF instead: local /admin-api/*, testing /dashboard/admin-api/*.

Use this tool to:

Verify a new model deployment works end-to-end before routing production traffic
Test parameter behavior for reasoning models (parameter stripping, reasoning effort)
Diagnose latency or response issues with full diagnostic output
Validate streaming vs. non-streaming paths independently

Input Schema

Provider Targeting

Field	Type	Required	Description
`provider`	`azure-ai-inference` \| `azure-ai-foundry`	No	Provider type. Defaults to `AZURE_DEFAULT_PROVIDER` env var.
`modelName`	string	No	Target model name. Resolved from DB if omitted.
`deploymentName`	string	No	Deployment name override.
`endpoint`	string (URL)	No	Override the capability endpoint URL.
`apiKey`	string	No	Explicit API key; bypasses managed identity.

Messages

Field	Type	Required	Description
`messages`	array	No	Full message array (`role`, `content`). Takes precedence over `prompt`.
`prompt`	string	No	Simple user prompt shortcut.
`system`	string	No	System prompt. Injected before user messages.

Sampling Parameters

Field	Type	Required	Description
`temperature`	float 0–2	No	Sampling temperature. Automatically stripped for reasoning models.
`topP`	float 0–1	No	Top-P nucleus sampling. Stripped for reasoning models.
`frequencyPenalty`	float -2–2	No	Frequency penalty. Stripped for reasoning models.
`presencePenalty`	float -2–2	No	Presence penalty. Stripped for reasoning models.
`maxOutputTokens`	int 1–65536	No	Maximum tokens to generate.
`reasoningEffort`	`none`\|`minimal`\|`low`\|`medium`\|`high`\|`xhigh`	No	Reasoning depth for GPT-5.x models.

Advanced Parameters

Field	Type	Required	Description
`seed`	int	No	Fixed seed for reproducible outputs.
`stopSequences`	string[] (max 4)	No	Stop sequences.
`responseFormat`	`{type: "text"\|"json"}`	No	Response format control.
`tools`	array (max 8)	No	Function calling tool definitions to test.
`providerOptions`	object	No	Raw vendor-specific parameters forwarded directly (e.g. `logprobs`, `logit_bias`).

Control Flags

Field	Type	Required	Description
`skipCapabilityValidation`	boolean	No	Bypass DB capability check. Requires explicit `provider`, `endpoint`, and `modelName`.
`stream`	boolean	No	Test the streaming path instead of the non-streaming generate path.

Output Schema

The response includes three sections:

`probe`

Resolved provider details: provider, modelId, deployment, endpoint, resolvedProvider (db-capability | explicit-config).

`request`

What was actually submitted and resolved: submittedParams, resolvedParams (after reasoning model stripping), messages, messageCount, and parameterInfo (whether it is a reasoning model, which params were stripped, supported reasoning efforts).

`result`

The LLM response: text, content, finishReason, usage (input/output/total/reasoning/cached tokens), warnings, providerMetadata, responseMetadata, latencyMs.

Usage Examples

Basic probe (uses DB default capability)

curl -X POST "${MASTRA_BASE_URL}/admin/tools/probe-llm-provider" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${BEARER_TOKEN}" \
  -d '{
    "data": {
      "prompt": "Say hello",
      "temperature": 0.7
    }
  }'

Test a specific deployment

curl -X POST "${MASTRA_BASE_URL}/admin/tools/probe-llm-provider" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${BEARER_TOKEN}" \
  -d '{
    "data": {
      "provider": "azure-ai-foundry",
      "modelName": "gpt-5",
      "deploymentName": "PROD-GPT5-CHAT",
      "prompt": "Explain how caching works"
    }
  }'

Test reasoning model with effort control

curl -X POST "${MASTRA_BASE_URL}/admin/tools/probe-llm-provider" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${BEARER_TOKEN}" \
  -d '{
    "data": {
      "modelName": "gpt-5o-reasoning",
      "prompt": "Solve: x^2 - 5x + 6 = 0",
      "reasoningEffort": "medium",
      "maxOutputTokens": 2000
    }
  }'

Note: temperature, topP, frequencyPenalty, and presencePenalty are automatically stripped for reasoning models. The parameterInfo.strippedParams field in the response shows which params were removed.

Test with explicit config (bypass DB)

curl -X POST "${MASTRA_BASE_URL}/admin/tools/probe-llm-provider" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${BEARER_TOKEN}" \
  -d '{
    "data": {
      "skipCapabilityValidation": true,
      "provider": "azure-ai-inference",
      "endpoint": "https://new-resource.services.ai.azure.com/",
      "modelName": "gpt-4o",
      "deploymentName": "TEST-GPT4O",
      "prompt": "Hello",
      "apiKey": "test-key"
    }
  }'

Environment Variables

Variable	Description
`AZURE_DEFAULT_PROVIDER`	Default provider when `provider` field is omitted
`ADMIN_TEST_MODEL_NAME`	Default model name fallback (overrides `MODEL_DEFAULT_MODEL_NAME`)
`ADMIN_TEST_MODEL_DEPLOYMENT`	Default deployment fallback
`MODEL_DEFAULT_MODEL_NAME`	System-wide default model name
`MODEL_DEFAULT_DEPLOYMENT`	System-wide default deployment

Overview​

Input Schema​

Provider Targeting​

Messages​

Sampling Parameters​

Advanced Parameters​

Control Flags​

Output Schema​

probe​

request​

result​

Usage Examples​

Basic probe (uses DB default capability)​

Test a specific deployment​

Test reasoning model with effort control​

Test with explicit config (bypass DB)​

Environment Variables​