AgentOps: contain an AI agent consumption runaway before shutting down production

A new release of an internal agent reaches production. User traffic is steady and answers still arrive, but consumption climbs: more tokens per intent, more tool calls, and longer conversations. Disabling the entire service stops the spend, but it also removes healthy workflows and the traces needed to understand the incident.

This runbook treats the runaway as an operations incident. It attributes consumption to a release, journey, and stage, then contains amplification at the narrowest possible boundary. The exit decision is explicit: keep the release with a corrected budget, restore the previous bundle, or hold the affected journey in read-only mode until its behavior is understood.

Freeze the timeline and define the unit of work

A daily aggregate cannot explain a loop that lasted a few minutes. Before changing the model or prompt, record the first deviation, all active versions, and a stable unit of work: the intent. One intent may span several turns, model calls, and tool calls. Grouping by that boundary makes amplification visible.

incident_window:
start: <timestamp>
end: <timestamp>

active_versions:
agent: <agent-version>
prompt: <prompt-version>
model_deployment: <deployment-name>
tool_catalog: <catalog-version>
retrieval_index: <index-version>

unit_of_work:
intent_id: <stable-intent-id>
conversation_id: <conversation-id>
tenant_or_audience: <bounded-segment>

evidence:
- input_and_output_tokens_per_model_call
- tool_calls_and_retries_per_intent
- retrieved_items_and_bytes
- context_size_before_each_call
- latency_result_and_error_class
- release_and_configuration_timeline

Keep correlation identifiers separate from sensitive content. Prompts, tool results, and retrieved documents may contain business data. Log their sizes, versions, and fingerprints when the full text is not required for diagnosis.

Separate traffic, unit consumption, and amplification

Total consumption combines several factors. More users are different from more tokens per call. Higher tokens per intent may come from growing context, extra planning steps, or retries that the conversation view does not expose.

Total model consumption
intents
x model calls per intent
x tokens per call

Tool workload
intents
x tool calls per intent
x cost and duration of each tool

Amplification signals
stable traffic with more model calls per intent
stable intents with more tool calls or retries
stable tool results retransmitted in every turn
the same intent recreated after a timeout or client resume
a read-only journey continuing after sufficient evidence exists

Compare the suspect release with a representative baseline for the same journey, audience, and outcome. A global average can hide one tenant or one expensive tool. Derive thresholds from the observed distribution; a number copied from another agent is not an operational budget.

Emit consumption telemetry by intent

A consumption event should connect the model call, tool, version, and outcome without requiring prompt text. Use additive counters so totals can be reconstructed even when other parts of the trace are sampled.

{
"timestamp": "<timestamp>",
"intentId": "<intent-id>",
"conversationId": "<conversation-id>",
"agentVersion": "<agent-version>",
"promptVersion": "<prompt-version>",
"modelDeployment": "<deployment-name>",
"stage": "plan|retrieve|tool|synthesize",
"modelCallId": "<call-id>",
"inputTokens": 0,
"outputTokens": 0,
"contextBytes": 0,
"retrievedItems": 0,
"toolName": "<tool-or-empty>",
"toolCallId": "<tool-call-id-or-empty>",
"toolAttempt": 0,
"result": "success|denied|timeout|error|budget_exhausted",
"durationMs": 0
}

Prices may change or vary by deployment. Preserve technical units first and version the rate card used by cost reporting. That distinction separates behavior drift from a pricing or model change.

Find the multiplier with KQL

The following query assumes a normalized AgentConsumptionEvents table. Map it to your telemetry pipeline and use a bounded incident window; this is not the time for an unbounded production query.

let StartTime = datetime(2026-08-04T08:00:00Z);
let EndTime = datetime(2026-08-04T10:00:00Z);
AgentConsumptionEvents
| where TimeGenerated between (StartTime .. EndTime)
| summarize
  ModelCalls=dcountif(ModelCallId, isnotempty(ModelCallId)),
  ToolCalls=dcountif(ToolCallId, isnotempty(ToolCallId)),
  ToolAttempts=countif(isnotempty(ToolCallId)),
  InputTokens=sum(InputTokens),
  OutputTokens=sum(OutputTokens),
  MaxContextBytes=max(ContextBytes),
  RetrievedItems=sum(RetrievedItems),
  Failures=countif(Result in ("timeout", "error")),
  BudgetStops=countif(Result == "budget_exhausted")
by IntentId, AgentVersion, PromptVersion, ModelDeployment
| extend TotalTokens = InputTokens + OutputTokens
| order by TotalTokens desc

Inspect the largest intents stage by stage. Input tokens rising on every turn with unchanged evidence suggests context accumulation. ToolAttempts materially above ToolCalls points to retries. A release-correlated jump in RetrievedItems points to retrieval limits or filters. The objective is to identify the multiplier, not merely the largest total.

Contain the narrowest boundary

A global stop is appropriate when actions can no longer be attributed or production writes are uncontrolled. Otherwise, reduce the blast radius first: block one release, journey, tool, audience, or new-intent admission while preserving read-only evidence and diagnostics.

policy_version: <version>
scope:
agent: ops-assistant
environment: production

per_intent:
max_model_calls: <derived-from-baseline>
max_total_tokens: <derived-from-baseline>
max_tool_calls: <derived-from-baseline>
max_retrieved_items: <derived-from-baseline>
max_elapsed_time: <derived-from-slo>

on_budget_exhausted:
stop_new_model_and_tool_calls: true
allow_final_bounded_response: true
disable_write_tools: true
emit_reason_and_correlation_id: true

containment:
quarantine_agent_versions: []
read_only_tool_allowlist: []
blocked_audiences: []
pin_previous_prompt_version: false

Enforce the budget in the runtime or execution gateway, not in the prompt. The agent may state that it will stop, but only a deterministic layer can reject the next call. Reserve a small allowance for a useful final response without permitting another tool invocation.

Diagnose the multiplier before optimizing the model

Walk the most expensive journey in execution order. A large context may come from a tool payload being injected in full, history that is never compacted, or duplicate documents. Extra calls may come from a missing exit condition, repeated planning, a client timeout, or an error incorrectly classified as retryable.

Check the system boundaries as well:

• does the client create a new intent when its response times out?
• does the runtime resume a trace while replaying completed stages?
• does the tool enforce pagination or return the entire result set?
• is a tool result persisted and then injected repeatedly?
• did the release change the prompt, model, tool catalog, retrieval layer, or several at once?

Switching models immediately may lower unit price while leaving the loop intact. The durable fix removes the multiplier and keeps a budget that limits its return.

Exercise the budget and degraded path

Replay representative traces without write effects. Inject a large tool response, excessive pagination, a timeout, and an incomplete result. Each case must prove that the intent stops, partial state remains explainable, and no write is attempted after budget exhaustion.

cases:
- id: normal_multi_step_intent
  expected:
    result: success
    budget_exhausted: false

- id: oversized_tool_result
  fault: tool_returns_bounded_but_large_payload
  expected:
    context_compacted_or_rejected: true
    repeated_full_payload: false

- id: retry_amplification
  fault: tool_timeout_after_accept
  expected:
    duplicate_write: false
    budget_stops_additional_attempts: true

- id: model_loop
  fault: exit_condition_never_selected
  expected:
    runtime_stops_execution: true
    final_response_names_budget_exhaustion: true
    write_tools_after_stop: 0

Run the policy in observation mode first, followed by a read-only canary. Compare tokens and calls per intent, task completion, budget denials, and latency. Lower consumption is not a successful outcome if legitimate journeys are truncated or operators lose proof of final state.

Validate, roll back, or keep quarantine

Version the prompt, model deployment, tool catalog, retrieval layer, and budget policy as one release bundle. A partial rollback can restore a prompt that expects a missing tool or leave an index incompatible with the previous journey.

Keep the release when the canary returns to a baseline-comparable distribution, completes critical cases, and performs no action after exhaustion. Restore the previous bundle when amplification follows the new release or the budget cuts healthy journeys. Keep read-only quarantine when earlier tool effects remain ambiguous; reverting configuration does not cancel a backend action that was already accepted.

Conclusion

An AgentOps consumption runaway is more than a billing concern. It often shows that an intent is no longer bounded because of accumulated context, broad retrieval, retries, a lost exit condition, or uncontrolled resumption.

Make the production decision with per-intent evidence. Keep the release when its budget, canary, and business outcomes are stable. Otherwise, pin the previous bundle, block writes from the suspect version, and retain enough telemetry to remove the multiplier before resuming.