AgentOps: bound MCP tool timeouts, retries and circuit breakers before production

An operations agent calls an MCP tool to inspect a deployment, prepare an action and execute it after approval. Every test call succeeds. In production, a dependency slows down: the agent waits, the client times out, the runtime retries and the operator starts the conversation again. One intent can become several concurrent calls, followed by a late response and a final state nobody can explain confidently.

The problem is not just model latency. It sits at the boundary between agent, MCP transport, tool server and business backend. This runbook bounds that boundary before production exposure: allocate a time budget, decide which calls may be retried, stop cascades with a circuit breaker, observe one complete journey, then validate or roll back the resilience policy.

Freeze one journey and its expected outcome

Start with one representative path. The working example uses an internal agent that checks a deployment job and can request a restart. The lookup has no side effect; the restart is an approved write. They must not inherit the same timeout and retry policy without review.

journey:
name: inspect_then_restart_deployment_job
agent: ops-assistant-prod
user_deadline: 20s
tools:
  - name: get_deployment_job
    effect: read_only
    expected_result: current job state
  - name: restart_deployment_job
    effect: write
    approval: human_required
    expected_result: one accepted operation or no operation

evidence:
- one conversation and intent identifier
- one tool call identifier per attempt
- backend operation identifier when accepted
- timeout owner and elapsed time
- retry reason and idempotency key
- circuit state at call time
- final validation or rollback decision

The user deadline is a ceiling for the journey, not a value to copy into every layer. If the backend may consume all twenty seconds, the transport, agent and interface have no time left to report failure cleanly or offer a controlled recovery.

Classify tools by effect, not protocol

MCP describes a tool interface; it does not make the operation idempotent. Classify every tool by its real effect and by the backend’s ability to recognize a repeated request.

Deterministic read
Example: get one job state by identifier
Retry a transient error when enough budget remains
Still cap attempts and concurrency

Expensive search or computation
Example: query a broad log window
Retry only with a bounded query and a known backend budget
Prefer pagination, caching or explicit continuation to duplication

Idempotent write
Example: request an operation with a stable idempotency key
Retry only when the backend guarantees one result for that key
Read operation state after any ambiguous response

Non-idempotent write
Example: restart a job without a stable operation identifier
Never retry automatically
After a timeout, inspect the backend and recover manually or roll back

This control belongs in the tool catalog or execution policy, not only in the prompt. A model may suggest another attempt; the runtime remains the authority that allows or blocks it.

Build the timeout budget from the outside in

A useful timeout names the layer that gives up and leaves outer layers enough time to process that result. The following values illustrate a contract; they are not universal production thresholds.

journey_budget_ms: 20000
layers:
user_interface:
  deadline_ms: 20000
agent_runtime:
  deadline_ms: 15000
  reserve_for_final_response_ms: 3000
mcp_transport:
  timeout_ms: 10000
tool_handler:
  backend_timeout_ms: 7000

rules:
- inner_timeout_must_expire_before_outer_timeout
- no_retry_without_remaining_budget
- cancellation_must_propagate_to_backend_when_supported
- late_results_must_not_trigger_a_second_write
- timeout_events_must_name_the_layer_that_expired

Measure p50, p95 and the tail by tool, backend and outcome. One global timeout hides the difference between network wait, execution queue, backend throttling and business processing. Reserve time for an actionable final response as well: the agent should be able to report ambiguous state and return a correlation identifier instead of disappearing at the last millisecond.

Make retries explicit and bounded

A retry policy combines error class, remaining budget, attempt count and idempotency. Exponential backoff with jitter reduces client synchronization, but it does not make a write repeatable.

{
"tool": "get_deployment_job",
"max_attempts": 3,
"retry_on": ["transport_unavailable", "backend_429", "backend_503"],
"do_not_retry_on": ["invalid_arguments", "policy_denied", "approval_required", "backend_4xx_other"],
"backoff": {
  "strategy": "exponential_with_jitter",
  "initial_ms": 250,
  "max_ms": 1500
},
"constraints": {
  "remaining_budget_required": true,
  "same_intent_id": true,
  "new_tool_call_id_per_attempt": true
}
}

For a write, add an idempotency key derived from a stable intent, target and approval, without embedding secrets. The backend must persist the key with the operation and return the same identifier when the request arrives again.

tool: restart_deployment_job
automatic_retry: false
idempotency:
key_components:
  - intent_id
  - target_job_id
  - environment
  - approval_id
backend_guarantee: one_operation_per_key
after_ambiguous_timeout:
- stop_agent_retry
- query_operation_by_idempotency_key
- compare_target_state
- resume_only_with_explicit_decision

If the backend cannot find an operation by key, automatic retries remain disabled. Improve the tool contract or target API first instead of asking the model to be more careful.

Open the circuit before the cascade

The circuit breaker protects the MCP server and its dependency when a series of failures shows that retries cannot help. Scope it by tool and, when necessary, by target or backend. A global circuit may take healthy reads offline because one unrelated operation is degraded.

breaker:
scope: tool_and_backend
tool: get_deployment_job
backend: deployment-api-prod
window: 60s
minimum_calls: 20
open_when:
  failure_ratio_gte: 0.50
  slow_call_ratio_gte: 0.70
open_duration: 30s
half_open:
  probe_calls: 3
  concurrent_probes: 1
fallback:
  mode: read_only_degraded_response
  include:
    - last_known_state_timestamp
    - incident_reference
    - no_write_action_performed

Derive thresholds from measurements and controlled exercises. While the circuit is open, the agent must not bypass it through an equivalent tool or additional conversations. A degraded response must distinguish fresh data, cached data and missing evidence.

Trace the attempt, not only the conversation

A conversation-level dashboard hides retries. Emit one event per attempt with intent, tool call and backend operation identifiers, plus the timeout layer and circuit state.

let StartTime = datetime(2026-08-02T08:00:00Z);
let EndTime = datetime(2026-08-02T10:00:00Z);
AgentToolCallEvents
| where TimeGenerated between (StartTime .. EndTime)
| where Protocol == "mcp"
| summarize
  calls=count(),
  intents=dcount(IntentId),
  retries=countif(Attempt > 1),
  timeouts=countif(Result == "timeout"),
  breakerRejects=countif(CircuitState == "open"),
  p50DurationMs=percentile(DurationMs, 50),
  p95DurationMs=percentile(DurationMs, 95),
  backendOperations=dcountif(BackendOperationId, isnotempty(BackendOperationId))
by bin(TimeGenerated, 5m), ToolName, Backend, Result
| extend CallsPerIntent = todouble(calls) / iif(intents == 0, 1.0, todouble(intents))
| order by TimeGenerated asc

Adapt the table to your telemetry pipeline. The important signal is the relationship between intents, attempts and backend operations. Rising CallsPerIntent without added user traffic exposes internal amplification. An idempotent write may have several attempts, but it must create only one backend operation.

Exercise failures before real traffic

Validation should inject latency, timeout, late responses, unavailability and ambiguous outcomes. It also covers agent behavior: does it explain the state, stop when the budget is gone and preserve approval boundaries?

cases:
- id: read_transient_503
  tool: get_deployment_job
  fault: backend_503_then_success
  expected:
    attempts: 2
    backend_operations: 0
    answer_contains_fresh_state: true

- id: write_response_lost_after_accept
  tool: restart_deployment_job
  fault: response_timeout_after_backend_accept
  expected:
    automatic_retry: false
    operation_lookup_by_idempotency_key: true
    duplicate_operations: 0

- id: sustained_dependency_failure
  tool: get_deployment_job
  fault: backend_503_for_90s
  expected:
    circuit_opens: true
    probe_concurrency: 1
    write_fallback: false

- id: budget_exhausted
  tool: get_deployment_job
  fault: slow_response
  expected:
    extra_retry: false
    timeout_layer_reported: true
    correlation_id_returned: true

Run these cases in an isolated environment, then with a side-effect-free production canary. Confirm that cancelling the request stops backend work when supported. Otherwise document that late completion remains possible and forbid any automatic second write.

Roll out in stages and prepare rollback

Start in shadow mode: calculate retry and breaker decisions without enforcing them. Compare those decisions with current behavior, then enable the policy for one read-only tool. Writes come later, and only with backend idempotency, bounded approval and state lookup after an ambiguous timeout.

Rollback applies to the resilience policy, not to business operations already accepted. Version the configuration, retain the previous version and provide a kill switch that blocks writes while keeping read-only diagnostics available. If an operation was created, cancellation or compensation follows the backend’s own runbook.

Keep the new policy only when the canary proves a respected budget, no uncontrolled amplification, an observable breaker, no duplicate operation and an actionable degraded response. Roll it back when legitimate calls are cut off, latency merely moves to another layer, retries increase load or a write outcome cannot be proven.

Conclusion

MCP tool resilience is not a matter of adding three retries. It connects the user budget to internal timeouts, classifies side effects, requires idempotent writes, opens the circuit before a cascade and retains one trace per attempt.

The production decision is straightforward: enable the policy only when a failure exercise proves that one intent remains time-bounded and creates one explainable business effect. Otherwise restore the previous configuration, block the affected writes and repair the tool contract or observability first.