Automation

Azure Container Apps Jobs: diagnose a KEDA scale rule before rerunning workers

A production runbook for qualifying an Azure Container Apps Job that no longer consumes correctly with KEDA scale rule, backlog, managed identity, secrets, logs, idempotency, validation and rollback before rerunning workers.

30 Jul 2026 azurecontainer-appsjobskedaautomationqueuemanaged-identityobservabilitykqlrunbookrollbackproduction

An Azure Container Apps Job that stops consuming a queue can look like a simple worker problem. The tempting response is to rerun the job, increase parallelism or purge blocked messages. That is risky when the real issue is the KEDA rule, an identity that can no longer read the queue, an expired secret, a saturated workload profile, a broken image or a non-idempotent handler.

The use case is an invoice-worker-prod job triggered by queue backlog. Since a deployment, the backlog keeps growing, a few executions fail, and the team wants to rerun workers to catch up. Before replaying anything, it needs to decide whether the safe action is a KEDA rule fix, secret restoration, image rollback, lower parallelism, bounded replay or a deliberate pause in consumption.

Treat the job as a consumption contract

Start by writing what the job is expected to consume and what it must not break. A KEDA rule is not only a threshold. It links an event source, an identity, metadata, execution limits and application code that must tolerate retries.

yaml container-apps-job-contract.yml

job:
name: invoice-worker-prod
environment: aca-env-prod
resource_group: rg-app-prod
trigger: event_driven
workload_profile: consumption-prod

source:
type: storage_queue
account: stprodops
queue: invoices-to-process
expected_backlog_behavior: decreases_during_business_hours
poison_queue: invoices-to-process-poison

scale_rule:
name: invoices-queue
min_executions: 0
max_executions: 20
polling_interval_seconds: 30
message_threshold: 25

runtime_contract:
idempotency_key: invoice_id
retry_safe_until: before_external_payment_capture
managed_identity: aca-invoice-worker-prod
secrets:
  - queue_connection_or_identity_binding
  - downstream_api_token_reference

blocked_until_proven:
- backlog_shape_known
- keda_rule_current
- identity_can_read_queue
- latest_image_healthy
- replay_scope_bounded
- rollback_path_ready

This contract prevents the queue from becoming a plain counter. If processing is no longer idempotent after an external step, a broad replay can create a larger incident than the original backlog.

Read backlog shape before rerun

The first diagnostic is backlog shape. Continuous growth may mean KEDA no longer triggers. A few old messages may point to poison messages. Retry waves may come from a downstream outage. An empty queue with stale delay alerts may only be an observability lag.

text backlog-reading.txt

Backlog to qualify
Visible message count
Age of the oldest message
Number of messages dequeued several times
Messages moved to the poison queue
Time of the last successful consumption
Correlation with the last deployment

Possible interpretation
Backlog grows with no executions: check KEDA, identity, trigger metadata
Many failed executions: check image, code, secrets, downstream system
A few repeated messages: isolate poison messages before global replay
Intermittent backlog: check workload profile, quotas and cold start

The rerun decision depends on that reading. Do not replay an entire queue when the real symptom is concentrated on three invalid messages or on a downstream API that is still unavailable.

Compare declared and effective KEDA configuration

The useful configuration is what Container Apps sees now. Check the job, trigger type, scale rules, referenced secrets and execution limits before changing code or increasing maxExecutions.

bash 01-read-job-scale-rule.sh

RG="rg-app-prod"
JOB="invoice-worker-prod"

az containerapp job show --resource-group "$RG" --name "$JOB" --query "{trigger:properties.configuration.triggerType, replicaTimeout:properties.configuration.replicaTimeout, replicaRetryLimit:properties.configuration.replicaRetryLimit, parallelism:properties.configuration.parallelism, replicaCompletionCount:properties.configuration.replicaCompletionCount, scaleRules:properties.configuration.eventTriggerConfig.scale.rules, registries:properties.configuration.registries}" --output json

az containerapp job execution list --resource-group "$RG" --name "$JOB" --output table

Look for simple drift: queue name changed, threshold too high, renamed secret, removed identity, execution limit too low, timeout too short or a workload profile different from the expected one.

Separate trigger failure from runtime failure

A job may not start because KEDA cannot read the source. It may also start correctly and then fail inside the container. Those two states require different fixes.

text trigger-vs-runtime.txt

Trigger problem
No execution while backlog grows
Authentication errors on the event source
Missing scale rule or inconsistent metadata
Secret or identity binding not found

Runtime problem
Executions created and then Failed
Application logs show business or downstream exception
Image pull or startup probe failure
Timeout before processing completes
Retries process the same message again

Decision
Fix KEDA or identity when the job never starts
Fix image, code, secret or downstream when the job starts then fails
Do not increase scale-out until the problem class is clear

This separation protects production from an attractive but unsafe move: increasing concurrency while every worker fails for the same reason.

Read logs within a bounded window

Logs should answer three questions: did the scaler trigger, did the container start, and was the message processed or rejected? Adapt the tables to the Log Analytics setup actually enabled in the environment.

kusto 02-container-apps-job-evidence.kql

let StartTime = datetime(2026-07-30T06:00:00Z);
let EndTime = datetime(2026-07-30T09:00:00Z);
let JobName = "invoice-worker-prod";
ContainerAppSystemLogs_CL
| where TimeGenerated between (StartTime .. EndTime)
| where ContainerAppName_s == JobName
| project TimeGenerated, Reason_s, Log_s, ReplicaName_s
| union (
  ContainerAppConsoleLogs_CL
  | where TimeGenerated between (StartTime .. EndTime)
  | where ContainerAppName_s == JobName
  | project TimeGenerated, Reason_s="console", Log_s, ReplicaName_s
)
| order by TimeGenerated desc

If logs show nothing, that does not prove the job is healthy. Check Diagnostic Settings, target workspace, ingestion delay and the real job or execution name first.

Validate identity, secrets and source access

A KEDA rule can depend on a secret or managed identity. An RBAC change, secret rotation or Key Vault change can prevent backlog reads even when application code has not changed.

text identity-secret-checklist.txt

Identity and access
Expected managed identity attached to the job
Minimal role on the queue or namespace
No recent deny assignment or policy change
Secret reference still resolves
Key Vault reachable from the Container Apps environment when used
Expiration or recent rotation documented

Useful evidence
401 or 403 errors in system or application logs
Last success before rotation
Comparison with a healthy job on the same source
Exact RBAC scope instead of broad subscription role

Do not replace a managed identity with a durable connection string just to move fast. If a temporary secret is required for recovery, it needs an expiry, an owner and a removal plan.

Choose replay, correction or pause

Once evidence is collected, make the decision explicit. A manual replay is acceptable only when processing is idempotent, scope is bounded and the cause is fixed or cleanly bypassed.

text rerun-decision-matrix.txt

Fix the KEDA rule
Backlog present, no execution, scaler metadata or secret drift

Roll back image or configuration
Recent executions fail after an application or IaC deployment

Reduce parallelism
Downstream degraded, too many workers, retries amplify the incident

Isolate poison messages
Small set of repeated messages, reproducible functional errors

Rerun manually
Cause fixed, idempotency proven, window and volume bounded, validation ready

Deliberate pause
Processing is not idempotent or downstream is still unstable

The correct action can be to stop consuming for a few minutes. A controlled pause with monitored backlog is better than a replay that writes twice to a downstream system.

Validate and prepare rollback

Closure must prove that consumption resumed without hiding the cause. Keep an application-level validation, not only a decreasing backlog.

text job-closure-and-rollback.txt

Final validation
KEDA rule observes the right source
Recent executions finish as Succeeded
Backlog decreases at the expected rate
No new poison message appears
Logs show the idempotency key or business correlation
Downstream confirms expected processing

Rollback
Return to the previous job image or revision
Restore the previous scale rule from IaC
Restore initial threshold and parallelism
Revoke any temporary secret
Put isolated messages back only after the fix
Document messages that were not replayed

If an emergency change was applied manually, it remains provisional until it is either captured in IaC or removed explicitly.

Conclusion

A blocked Azure Container Apps Job is not just a worker to rerun. It is a consumption chain: event source, KEDA rule, identity, secrets, image, execution limits, logs, idempotency and downstream system.

The useful decision is bounded: fix the scaler if the job does not start, roll back the image if executions fail after deployment, reduce concurrency if the downstream system is suffering, isolate invalid messages, or rerun only after idempotency is proven. The runbook turns an urgent backlog into controlled recovery.