AI
AgentOps: diagnose Azure OpenAI rate limits before changing models
A production runbook for qualifying Azure OpenAI or Microsoft Foundry throttling with quota, deployment capacity, agent traces, retries, fallback, validation and rollback before changing models.
An AI agent that suddenly slows down or returns incomplete answers does not always need a new model. In production, the first suspect is often more operational: an Azure OpenAI deployment is throttled, a Foundry agent calls too many tools in one turn, retries amplify the load, a batch job competes with interactive traffic, or a fallback path silently changes the answer quality.
The use case is an internal assistant used by operations, support or engineering teams. It reads approved sources, calls bounded tools and uses Azure OpenAI or Microsoft Foundry for reasoning. Users report timeouts, partial answers or bursts of 429 and 503 errors during incidents. The runbook goal is to decide whether to tune traffic, split workloads, add capacity, change fallback behavior or rollback the latest agent change before changing the model.
Freeze the failing workload
Start with one agent path. A rate-limit incident becomes impossible to diagnose when chat traffic, evaluations, ingestion, summarization and tool retries are all mixed together.
Workload to qualify
Agent: ops-assistant-prod
Environment: production
User path: incident diagnosis conversation
Model deployment: gpt-4.1-prod or equivalent approved deployment
Tool path: retrieval, log query, draft action
Symptom: 429, timeout, truncated answer or fallback answer
Recent change: prompt, tool schema, retrieval index, evaluation batch or deployment quota
Evidence required before changing model
Agent trace with conversation and tool call IDs
Azure OpenAI response status and retry metadata
Token volume by request class
Concurrent traffic during the incident window
Fallback path and user-visible behavior
Validation case that reproduces or disproves the issue
Rollback path for prompt, tool, traffic or deployment changeThe first useful decision is scope. If only evaluation jobs are throttled, do not change the interactive agent. If only one tool path explodes token usage, do not move the whole assistant to another deployment.
Separate quota from behavior
Quota is not only an Azure setting. The agent behavior can consume it faster than expected: long retrieved chunks, repeated tool failures, retries without jitter, parallel conversations or an evaluation run launched during business hours.
Possible causes
Deployment capacity is too low for peak traffic
Requests per minute exceed deployment limits
Tokens per minute increased after prompt or retrieval change
Tool errors cause repeated reasoning loops
Client retry policy creates synchronized bursts
Evaluation, batch or indexing jobs share the same deployment
Fallback sends traffic to a smaller or incompatible deployment
Regional outage or dependency issue increases latency and retries
Do not conclude too early
429 does not prove the model is wrong
Higher quota does not fix a retry storm
A faster model does not fix oversized context
Fallback success does not prove answer quality is acceptableTreat the deployment limit and the agent path as two parts of the same system. Capacity may need to increase, but only after the request pattern is understood.
Capture traces before retrying
Before rerunning the same user prompt, capture the failed conversation. The trace should explain how many model calls were made, which tools were called, how large the context became and whether fallback happened.
{
"conversation_id": "conv-20260707-0812",
"agent": "ops-assistant-prod",
"environment": "production",
"model_deployment": "gpt-4.1-prod",
"request_class": "interactive_incident_diagnosis",
"retrieval_chunks": 8,
"tool_calls": ["search_runbook", "query_logs"],
"prompt_tokens": 18400,
"completion_tokens": 1200,
"response_status": 429,
"retry_count": 3,
"fallback_used": false,
"backend_correlation_id": "aoai-req-7f3c",
"user_visible_result": "timeout_before_final_answer"
}If this data is not available, the first fix is observability. Increasing quota without traces can make the same failure more expensive and harder to notice.
Check deployment and quota evidence
Compare the deployment configuration with observed traffic. The objective is not to memorize every quota value. The objective is to prove whether the failing workload exceeded its assigned envelope.
RESOURCE_GROUP="rg-ai-prod"
ACCOUNT_NAME="aoai-prod-west"
DEPLOYMENT_NAME="gpt-4-1-prod"
az cognitiveservices account deployment show --resource-group "$RESOURCE_GROUP" --name "$ACCOUNT_NAME" --deployment-name "$DEPLOYMENT_NAME" --output json
az monitor metrics list --resource "/subscriptions/<subscription-id>/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.CognitiveServices/accounts/$ACCOUNT_NAME" --metric "TotalCalls,TokenTransaction,ThrottledCalls" --interval PT1M --aggregation Total --output tableUse the metric names and dimensions available in your workspace and region. Keep the evidence tied to the deployment name and time window used by the failed agent trace.
Inspect retries and fallback
Retries are useful only when they reduce transient failure. In a quota incident, aggressive retries can multiply the problem. Fallback can protect the user experience, but it must be explicit and evaluated.
retry_policy:
applies_to:
- transient_429
- transient_503
max_attempts: 2
backoff: exponential_with_jitter
retry_budget_per_conversation: 1
do_not_retry:
- schema_validation_error
- authorization_denied
- context_too_large
fallback_policy:
allowed_when:
- primary_deployment_throttled
- fallback_evaluation_passed
fallback_deployment: gpt-4.1-mini-prod
user_visible_behavior: "answer_with_limited_confidence_and_trace"
blocked_for:
- production_write_decision
- security_exception
- rollback_recommendation_without_human_reviewFallback should never be a hidden quality downgrade for high-risk actions. For a draft or summary it may be acceptable. For a production write recommendation, it should usually hold, ask for validation or keep the action in draft.
Reduce token pressure before adding capacity
When throttling follows a prompt, retrieval or tool change, the cheapest and safest fix may be to reduce unnecessary tokens.
Token pressure checks
Retrieved chunks are relevant and deduplicated
Runbook excerpts have size limits
Tool outputs are summarized before being reintroduced
Conversation memory is bounded by task, not by full history
Evaluation cases are not running on the production deployment
Batch summarization has a separate deployment or schedule
The agent stops after a tool failure instead of looping
Block capacity increase when
One prompt change doubled context size
A tool returns raw logs without compression
Retries are unbounded
Evaluation traffic shares the same deployment as incidents
The team cannot distinguish interactive and batch loadCapacity is still a valid answer, but it should not hide a broken context strategy.
Validate with controlled traffic
Do not validate the fix with one lucky conversation. Replay a small set of representative cases: a normal question, a retrieval-heavy diagnosis, a tool failure, a fallback scenario and a refusal case.
validation_cases:
- id: normal_diagnosis
expected:
status: success
fallback: false
max_model_calls: 2
- id: retrieval_heavy_incident
expected:
status: success
retrieved_chunks_within_limit: true
answer_cites_sources: true
- id: tool_backend_unavailable
expected:
status: graceful_hold
retry_count_within_budget: true
no_write_action: true
- id: primary_deployment_throttled
expected:
status: fallback_or_hold
fallback_trace_visible: true
production_write_blocked: true
- id: broad_action_request
expected:
status: refused
no_capacity_escalation: trueThe validation should prove both availability and control. A fix that makes answers fast but allows unsafe fallback behavior is not ready.
Decide tune, scale, split or rollback
Keep the operational decision explicit. A rate-limit incident can be solved in several ways, and they do not carry the same risk.
Tune traffic
Token growth came from retrieval, prompt or tool output
Retry behavior amplified the incident
Batch jobs can move outside the peak window
Fallback rules need clearer risk boundaries
Scale or request quota
Interactive traffic is valid and sustained
Token usage matches the intended use case
Evaluation and batch traffic are already separated
Traces prove the deployment envelope is too small
Split deployments
Production incident traffic competes with evaluation or batch jobs
Different request classes need different latency and risk policies
Fallback requires a separately validated deployment
Rollback
Throttling started after prompt, retrieval, tool or routing change
The previous version restored latency and reduced 429s in test
Fallback changed answer quality without approval
Observability cannot reconstruct model calls and retriesChanging model is only one option. Often the better production move is to separate traffic classes, reduce context, fix retries or rollback the agent change that caused the surge.
Validate after rollback or promotion
After the decision, keep a short watch window. The goal is to prove that user-facing behavior recovered without widening the agent’s action boundary.
Post-change validation
Throttled calls return to baseline
Interactive latency is inside the expected envelope
Retry count stays within budget
Fallback decisions are visible in traces
Production write recommendations still require approval
Evaluation and batch jobs do not use the incident deployment
Incident note stores trace IDs, metrics, validation cases and decision
Rollback is incomplete when
429s disappear but fallback hides lower confidence answers
Token volume remains unexplained
Batch jobs still share the same deployment
The old prompt or retrieval index can be redeployed automaticallyConclusion
Azure OpenAI throttling in an agent is not only a quota problem. It is a production behavior problem that spans prompt size, retrieval, tool loops, retry policy, fallback, deployment capacity and observability.
The decision becomes safe when evidence is visible: tune traffic when the agent wastes tokens, split deployments when workloads compete, request capacity when valid demand exceeds the envelope, and rollback when a recent change created the surge. The goal is not just fewer 429 responses. It is an agent that remains available without silently weakening the controls around production actions.