Azure Container Apps: validate revision traffic before rollback

An Azure Container Apps deployment can fail without the new revision being technically dead. The image starts, probes pass, some traffic reaches it, but errors increase only for specific paths, clients or outbound calls. The immediate reflex is often to roll back the whole revision. Sometimes that is right. Sometimes the issue is the traffic weight, a label pointing to the wrong revision, a dependency that is not ready, or a route that still sends users through the wrong assumption.

The use case is a Container Apps service running in multiple revisions mode with a progressive rollout or blue-green switch. After promotion, alerts show degradation, but the team does not yet know whether to reduce traffic, fix configuration, move back to the previous revision or keep the candidate isolated for analysis.

The goal of this runbook is to make an observable decision: keep, reduce, shift back or roll back.

Name the revision actually serving traffic

Start by capturing routing state before changing anything. Separate the active revision, traffic weight, public or internal label, and the container image actually exposed.

Service to qualify
Container App: api-orders-prod
Environment: cae-prod
Previous revision: api-orders-prod--r42
Candidate revision: api-orders-prod--r43
Mode: multiple revisions
Change: new image and new configuration variable
Strategy: 20 percent candidate, 80 percent stable

Questions before rollback
What percentage of traffic really reaches the candidate?
Does a label point to a revision different from the expected weight?
Do errors follow the revision, path, client or dependency?
Are probes enough to validate the user path?
Can traffic be reduced without deleting evidence?

This step avoids confusing rollback with diagnosis. If the candidate revision is removed immediately, the team may also remove the traces that explain why it degraded the service.

Read revisions, weights and labels

Capture Container Apps state from Azure CLI. The important part is not only which revision is active, but how traffic reaches it.

RG="rg-prod"
APP="api-orders-prod"

az containerapp show --resource-group "$RG" --name "$APP" --query "{mode:properties.configuration.activeRevisionsMode,ingress:properties.configuration.ingress.traffic,latestRevision:properties.latestRevisionName}" --output json

az containerapp revision list --resource-group "$RG" --name "$APP" --query "[].{name:name,active:properties.active,created:properties.createdTime,image:properties.template.containers[0].image,replicas:properties.replicas,trafficWeight:properties.trafficWeight}" --output table

If the service uses labels, also verify that the test, canary or production label points to the intended revision. A forgotten label can keep a validation path on an old revision, or expose a candidate to a client that should not receive it.

Correlate errors by revision

Degradation must be read by revision. A global average can hide a candidate failing at 20 percent traffic, or a stable revision suffering from a shared dependency.

let Window = 2h;
ContainerAppConsoleLogs_CL
| where TimeGenerated > ago(Window)
| where ContainerAppName_s == "api-orders-prod"
| summarize errors=countif(Log_s has_any ("ERROR", "Exception", "timeout")), rows=count() by RevisionName_s, bin(TimeGenerated, 5m)
| order by TimeGenerated asc, RevisionName_s asc

Complete this with system logs to detect restarts, scaling issues, image pull failures or resource pressure.

let Window = 2h;
ContainerAppSystemLogs_CL
| where TimeGenerated > ago(Window)
| where ContainerAppName_s == "api-orders-prod"
| summarize events=count(), messages=make_set(Log_s, 5) by RevisionName_s, Reason_s, bin(TimeGenerated, 10m)
| order by TimeGenerated asc

If errors are shared across all revisions, an application rollback may not fix anything. Look instead at a dependency, secret, identity, outbound route, upstream service or common saturation point.

Verify probes and the user path

A revision can be healthy from the platform point of view and wrong from the user point of view. Probes often validate a short endpoint. They do not prove that authentication, database calls, messaging, internal routing or downstream dependencies work.

Minimum validation
Platform probe: active revision and ready replicas
Smoke test: user endpoint with the real hostname
Correlation: request id present in application logs
Dependencies: outbound calls, identity and 401/403/5xx errors
Traffic: expected weight observed in logs
Rollback: stable revision still active and targetable

Block promotion when
The candidate only answers the technical probe
Logs cannot distinguish revisions
The validation label does not point to the candidate
A critical dependency fails only on the candidate
Rollback would delete the only usable evidence

The smoke test must follow the same path as users: domain, TLS, possible Front Door or Application Gateway handoff, authentication, primary dependency and correlation trace.

Decide reduction, shift or rollback

The decision should be proportionate. Not every post-deployment incident deserves the same action.

decision:
keep_and_watch:
  when:
    - no_revision_specific_error
    - latency_and_error_budget_within_limit
    - traces_distinguish_candidate_and_stable
  actions:
    - keep_current_traffic_weight
    - extend_observation_window
    - keep_previous_revision_active

reduce_candidate_traffic:
  when:
    - candidate_errors_are_real_but_bounded
    - stable_revision_remains_healthy
    - evidence_is_needed_before_full_rollback
  actions:
    - lower_candidate_weight
    - keep_label_for_targeted_tests
    - capture_logs_before_next_change

shift_back_to_stable:
  when:
    - user_impact_confirmed_on_candidate
    - stable_revision_has_no_same_failure
    - rollback_path_is_known
  actions:
    - set_stable_revision_to_100_percent
    - keep_candidate_active_without_public_traffic
    - validate_user_path_and_error_drop

deactivate_candidate:
  when:
    - candidate_causes_restarts_or_resource_pressure
    - no_more_evidence_is_required
    - stable_revision_is_validated
  actions:
    - deactivate_candidate_revision
    - open_fix_task_with_evidence
    - document_return_condition

The key nuance is the difference between shifting traffic back and deactivating the revision. Shifting traffic protects users. Keeping the candidate active without public traffic often preserves diagnostic value.

Apply a controlled rollback

If rollback is approved, apply the smallest reversible change. The first move can be setting the stable revision back to 100 percent, not deleting the candidate.

RG="rg-prod"
APP="api-orders-prod"
STABLE="api-orders-prod--r42"
CANDIDATE="api-orders-prod--r43"

az containerapp ingress traffic set --resource-group "$RG" --name "$APP" --revision-weight "$STABLE=100" "$CANDIDATE=0"

az containerapp show --resource-group "$RG" --name "$APP" --query "properties.configuration.ingress.traffic" --output json

Then validate with the same signals that justified the rollback: error rate, latency, logs by revision, dependency traces and user feedback when the symptom was external.

Keep usable evidence

Before closing the incident, keep a short evidence pack.

Evidence to keep
Traffic state before change
Active revisions and exposed images
First degradation time
Application logs by revision
Container Apps system logs
Smoke test before and after rollback
Decision: reduce, shift or deactivate
Candidate return condition

The return condition matters. It may be a new image, corrected variable, validated dependency, repeated load test, or canary observation with a lower traffic weight.

Conclusion

An Azure Container Apps incident after deployment should not be treated as a single rollback button. In multiple revisions mode, traffic, labels, probes and logs often tell a more precise story than the revision’s “active” state.

The right reflex is to measure which revision actually serves users, isolate errors by revision, validate the user path, then choose between observation, traffic reduction, shift back to stable or deactivation. The final decision should protect production without erasing the evidence the team needs to fix the candidate cleanly.