Cloud
Azure Cosmos DB: diagnose 429 throttling before increasing RU/s
A production runbook for qualifying Cosmos DB 429 throttling with RU consumption, hot partitions, query shape, SDK retries, indexing, KQL evidence, scaling decision and rollback.
Cosmos DB 429 responses are easy to misread during an incident. The application is slow, the SDK retries, dashboards show throttling, and someone proposes increasing RU/s immediately. That may be the right decision, but it should not be the first unexplained change. A single hot partition, a new query shape, an indexing drift or an aggressive retry policy can burn the available throughput without proving that the whole container needs more capacity.
The use case is a production API, orders-api-prod, that writes and reads orders from the orders container. Since the last deployment, checkout latency increased and the application logs show intermittent RequestRateTooLarge errors. The runbook goal is to decide whether to scale RU/s, fix a partition or query issue, roll back the deployment, or keep the current capacity while collecting better evidence.
Freeze the incident contract
Start by recording the failure as an operational contract. The team needs a shared view of the container, workload, timeframe, symptoms and rollback path before changing throughput.
incident:
service: orders-api-prod
database: commerce
container: orders
account: cosmos-commerce-prod
region: westeurope
symptom:
- checkout latency increased
- application logs contain 429 / RequestRateTooLarge
- retry count increased after deployment
incident_window:
start: 2026-07-12T08:20:00Z
end: 2026-07-12T09:10:00Z
last_change:
deployment: orders-api-prod-20260712.3
suspected_area: order search and status update path
decision_needed:
- scale throughput
- rollback deployment
- fix query or partition usage
- hold change and keep monitoring
rollback_candidate:
deployment: orders-api-prod-20260712.2
throughput: previous container throughput or autoscale max RU/sIf the incident contract only says “Cosmos is throttling”, the next action will probably be too broad. The useful question is which operation, partition key range, query or client path consumes the capacity.
Prove the current capacity mode
Do not assume the container is provisioned the way the team remembers it. Check whether throughput is manual, autoscale, shared at database level, recently changed or below the expected baseline.
SUBSCRIPTION="00000000-0000-0000-0000-000000000000"
RG="rg-commerce-prod"
ACCOUNT="cosmos-commerce-prod"
DB="commerce"
CONTAINER="orders"
az account set --subscription "$SUBSCRIPTION"
az cosmosdb sql container throughput show --resource-group "$RG" --account-name "$ACCOUNT" --database-name "$DB" --name "$CONTAINER" --query "{resource:resource, autoscaleSettings:autoscaleSettings}" --output json
az cosmosdb sql database throughput show --resource-group "$RG" --account-name "$ACCOUNT" --name "$DB" --query "{resource:resource, autoscaleSettings:autoscaleSettings}" --output jsonOne of these commands can legitimately fail when throughput is configured at the other level. Keep that result in the incident note. It prevents a false assumption about where scaling would actually apply.
Separate global saturation from a hot partition
A container-wide 429 rate does not automatically mean the whole container lacks RU/s. If one partition key or key range is hot, increasing total throughput may reduce pressure temporarily while leaving the design problem intact.
let StartTime = datetime(2026-07-12T08:20:00Z);
let EndTime = datetime(2026-07-12T09:10:00Z);
AzureDiagnostics
| where TimeGenerated between (StartTime .. EndTime)
| where ResourceProvider == "MICROSOFT.DOCUMENTDB"
| where Category has_any ("DataPlaneRequests", "PartitionKeyStatistics", "QueryRuntimeStatistics")
| extend StatusCode = tostring(statusCode_s)
| extend Operation = tostring(operationName_s)
| extend Collection = tostring(collectionName_s)
| extend PartitionKeyRangeId = tostring(partitionKeyRangeId_s)
| extend RequestCharge = todouble(requestCharge_s)
| where Collection == "orders" or Collection == ""
| summarize
Requests = count(),
Throttled = countif(StatusCode == "429"),
TotalRU = sum(RequestCharge),
P95RU = percentile(RequestCharge, 95)
by Operation, PartitionKeyRangeId, bin(TimeGenerated, 5m)
| order by TimeGenerated asc, Throttled descTreat missing diagnostic fields as evidence too. If the workspace cannot show operation, status code or partition information, the first fix may be observability, not capacity.
Read the client retry behavior
Cosmos SDK retries are useful, but they can hide a production problem until latency becomes visible. Check whether the application changed retry count, timeout, preferred region, consistency level or query pattern.
Client checks
SDK version changed in the last deployment
Retry count and max wait time are known
Request diagnostics are logged for failed operations
Preferred regions did not change unexpectedly
Consistency level did not become stronger for the hot path
Bulk mode or parallel query settings did not amplify pressure
Block blind scaling when
429 errors are only visible after the new deployment
Retry count increased but end-to-end latency also increased
Request charge per operation changed sharply
Logs do not identify operation name or partition key context
The same user or tenant shape appears in most failuresIf retries are absorbing most 429 responses, the user symptom may be latency rather than failure. That still matters. A retry storm can consume more client threads, hold connections longer and make rollback harder to interpret.
Compare operation shape before and after deployment
The most useful evidence often comes from the delta. A new query that scans more documents, misses an index path or fans out across partitions can look like a capacity incident.
let Before = AzureDiagnostics
| where TimeGenerated between (datetime(2026-07-12T07:20:00Z) .. datetime(2026-07-12T08:10:00Z))
| where ResourceProvider == "MICROSOFT.DOCUMENTDB"
| extend Operation = tostring(operationName_s)
| extend RequestCharge = todouble(requestCharge_s)
| summarize BeforeRequests=count(), BeforeRU=sum(RequestCharge), BeforeP95RU=percentile(RequestCharge, 95) by Operation;
let After = AzureDiagnostics
| where TimeGenerated between (datetime(2026-07-12T08:20:00Z) .. datetime(2026-07-12T09:10:00Z))
| where ResourceProvider == "MICROSOFT.DOCUMENTDB"
| extend Operation = tostring(operationName_s)
| extend RequestCharge = todouble(requestCharge_s)
| extend StatusCode = tostring(statusCode_s)
| summarize AfterRequests=count(), AfterRU=sum(RequestCharge), AfterP95RU=percentile(RequestCharge, 95), After429=countif(StatusCode == "429") by Operation;
Before
| join kind=fullouter After on Operation
| extend RUChange = AfterRU - BeforeRU
| order by After429 desc, RUChange descA high After429 on a write path suggests capacity or partition pressure. A high RU delta on a read or query path suggests query shape, indexing or fan-out. The decision is different.
Validate indexing and query paths before scaling
When the incident follows a release, inspect the queries and the indexing policy before changing RU/s. The goal is not to optimize everything during the incident. The goal is to find a safe explanation for the new request charge.
Validation questions
Which endpoint or job introduced the new query?
Does it filter by the partition key?
Does it sort on an indexed path?
Does it use OFFSET LIMIT or a broad CONTAINS pattern?
Does it read many documents to compute a small response?
Did the indexing policy change near the incident window?
Is a background reindexing operation in progress?
Safe actions during incident
Disable or roll back the new query path
Reduce batch size for the affected job
Pause a non-critical backfill
Route expensive search to a safer read model
Scale only with a rollback condition and expiryIf the application introduced a cross-partition query on a hot user path, scaling can buy time, but rollback or query correction is still the durable fix.
Build a decision matrix
Make the decision explicit so the team does not treat “more RU” as the only successful outcome.
Scale RU/s or autoscale max RU/s
Container-wide RU consumption is saturated
Throttling affects several operations and partition ranges
Traffic increase is expected or business-critical
Query shape and retry behavior do not explain the spike
Scaling has owner, expiry and validation checks
Rollback the deployment
Throttling started after one release
One new endpoint, job or query dominates RU delta
Request charge per operation increased unexpectedly
User impact is tied to the changed path
Previous version is known and rollbackable
Fix partition or workload shape
One partition key range dominates throttling
One tenant, customer, campaign or batch job drives pressure
Writes or reads are uneven by key
Scaling would hide the imbalance without removing risk
Hold and improve evidence
Diagnostic logs cannot identify operation or status
Application logs omit request charge and retry count
Capacity mode is unclear
No safe test proves which path changedThe matrix should be attached to the incident record. It protects the team from a capacity change that nobody can later explain.
Apply a bounded mitigation
When scaling is justified, make it bounded. Throughput increases are operational changes: they need a target, a validation window, a rollback or reduction condition, and ownership.
change:
target: cosmos-commerce-prod / commerce / orders
action: increase_autoscale_max_ru
reason: container-wide throttling across operations and partition ranges
owner: platform-oncall
validation_window: 30 minutes
success:
- 429 rate decreases on affected operations
- checkout latency returns to expected range
- retry count decreases in application logs
- no new hot partition dominates request charge
rollback_or_reduce_when:
- rollback of application removes RU pressure
- throttling persists on one partition only
- cost guardrail or change window expires
- diagnostics prove query or indexing regressionIf the mitigation is rollback, keep the same discipline: deploy the previous version, watch 429 rate, request charge and latency, and leave throughput unchanged unless the rollback does not reduce pressure.
Keep the post-incident artifact
The final artifact should be reusable. It should explain why the team scaled, rolled back or changed the workload, and what evidence would catch the next event earlier.
Incident result
Primary cause:
Affected operation:
Affected partition evidence:
Request charge delta:
Retry behavior:
Throughput mode:
Mitigation:
Rollback or reduction condition:
Follow-up:
- add request charge and retry count to application logs
- dashboard 429 rate by operation and partition key range
- review query shape in deployment checklist
- document autoscale owner and expiry when used as mitigationThis note is more valuable than the exact RU number. It turns the incident into a repeatable diagnostic path instead of a memory of a stressful dashboard.
Conclusion
A Cosmos DB 429 incident is not automatically a capacity incident. It is a signal that the workload, partition distribution, query shape, retry policy and provisioned throughput no longer line up.
The safe runbook is to prove capacity mode, separate global saturation from hot partitions, compare operation cost before and after deployment, validate query and indexing changes, then decide: scale with a bounded window, rollback the release, fix the workload shape, or improve evidence before touching production. The best outcome is not always more RU/s. It is a decision the team can validate and reverse.