Azure App Configuration: diagnose a feature flag before rollback

A production incident caused by configuration rarely looks like a configuration incident at first. The deployment is green, the container image did not change, the API still answers, but one path returns 500, a new flow is visible to the wrong users, or a dependency is called with an unexpected endpoint. The reflex is often to roll back the application, disable the feature flag everywhere, or edit a value directly in the portal.

The use case is an Azure application using Azure App Configuration for feature flags and runtime settings. It may run on App Service, Azure Functions, Container Apps or AKS. Some values are labeled by environment, region or ring. Some settings reference Key Vault secrets. The runbook goal is to decide whether the regression comes from a flag, a label, a stale refresh, an identity issue, a Key Vault reference, or the application release itself.

Freeze the configuration scope

Start by naming the exact change surface. A feature flag is not only a boolean. It can have labels, filters, targeting rules, percentage rollout, time windows, cache refresh and code paths that interpret it differently.

Incident
Application: orders-api-prod
Environment: production
Region or ring: westeurope / ring-1
Symptom: checkout retries and payment validation errors
Suspected key or flag: FeatureManagement:UseNewPaymentAdapter
Expected label: prod
Last known healthy value and timestamp
Deployment version and image digest
App Configuration store and replica
Key Vault references involved

Questions before change
Which label did the runtime read?
Did the application refresh the new value?
Was the flag targeted to the intended users only?
Did a Key Vault reference fail behind the setting?
Can rollback be done by label, flag, or application version?

This scope prevents two common mistakes: disabling a flag globally when only one label is wrong, or rolling back code when the deployed version is reading stale configuration.

Read labels and effective values

The first evidence is the actual key set, not the value remembered by the team. Export the relevant keys with labels, ETags and last modified dates. If labels are used for environment or rollout rings, compare them side by side.

APP_CONFIG_NAME="appcfg-prod"
KEY_FILTER="FeatureManagement:*"

az appconfig kv list --name "$APP_CONFIG_NAME" --key "$KEY_FILTER" --label prod --fields key label value content_type last_modified etag --output table

az appconfig kv list --name "$APP_CONFIG_NAME" --key "$KEY_FILTER" --label ring-1 --fields key label value content_type last_modified etag --output table

az appconfig feature list --name "$APP_CONFIG_NAME" --label prod --output json

Keep the ETag and timestamp in the incident record. They make it possible to prove whether a value changed during the incident and to restore the previous value without guessing.

Verify the identity and network path

When an application reads App Configuration with managed identity, a 403, a stale value or a fallback to local configuration can look like an application bug. Prove the identity and the path used by the workload before editing flags.

Runtime access
Managed identity attached to the workload
App Configuration Data Reader or expected role assignment
Key Vault access for every referenced secret
Private or public network path documented
DNS and outbound path validated from runtime, not workstation

Failure signals
App reads default value because App Configuration is unreachable
Key Vault reference fails and application falls back
Wrong label is loaded after deployment slot or ring change
Cached value remains active after emergency toggle
Connection string is still used by one component

Private Endpoint may be part of the design, but it is only one control. A correct private path does not prove the workload has the right App Configuration or Key Vault permissions.

Correlate flags with application symptoms

The decision should be based on a timeline. Correlate configuration changes, application errors, dependency calls and request dimensions that match the flag targeting rule.

let Window = 4h;
let FlagName = "UseNewPaymentAdapter";
let SuspectedPath = "/checkout";
AppConfigurationChangeEvents
| where TimeGenerated > ago(Window)
| where Key has FlagName
| project TimeGenerated, Key, Label, OldValue, NewValue, ETag, Actor, CorrelationId
| order by TimeGenerated desc

Use application telemetry to see whether the symptom follows the flag, the version or the environment.

let Window = 4h;
requests
| where timestamp > ago(Window)
| where url has "/checkout"
| extend FeatureFlag = tostring(customDimensions["UseNewPaymentAdapter"])
| summarize Requests=count(),
          Failures=countif(success == false),
          P95=percentile(duration, 95)
by bin(timestamp, 10m), FeatureFlag, cloud_RoleName
| order by timestamp asc

If the failure appears only when the flag is true, rollback the flag or targeting rule. If it appears across both values after a deployment, treat the application release as the primary suspect. If telemetry does not contain the flag state, add it before making the next rollout more autonomous.

Check refresh behavior before trusting the toggle

A feature flag rollback is only useful when the runtime observes it. Applications often cache App Configuration values, refresh on sentinel keys, or need explicit refresh middleware. During an incident, verify the refresh contract.

refresh_contract:
provider: azure-app-configuration
watched_keys:
  - FeatureManagement:UseNewPaymentAdapter
  - AppConfig:Sentinel
expected_refresh_interval: 30s
required_runtime_evidence:
  - current_etag
  - loaded_label
  - last_refresh_timestamp
  - refresh_result
  - fallback_used
block_rollout_when:
  - flag_state_not_logged
  - label_not_visible_in_telemetry
  - refresh_result_unknown
  - emergency_toggle_requires_restart

If the application needs a restart to observe a flag change, write that into the rollback path. Otherwise, operators may believe they disabled a feature while the old value remains active.

Decide the smallest rollback

Rollback should match the proven failure. Do not disable every feature flag when one ring label is wrong. Do not redeploy the application if the previous App Configuration value is available and the runtime refreshes correctly.

Rollback the flag
Failures correlate with flag=true
Correct label is proven
Runtime refresh is observable
Previous ETag or value is known

Rollback the label or targeting rule
Only one ring, region or user group is affected
Global flag value is correct
Targeting conditions changed recently
Application version is otherwise healthy

Fix identity or dependency access
App Configuration or Key Vault access fails
Application uses fallback values
Logs show 401, 403, DNS or timeout before business error

Rollback the application release
Errors happen with both flag states
Telemetry ties failure to version, not configuration
The code misinterprets a valid setting
Configuration rollback does not restore the path

For App Configuration, a safe rollback is usually a new controlled change: restore the previous value, restore the previous label, or restore targeting. Direct portal edits during pressure should still leave an evidence trail.

Automate the evidence pack

The useful automation is not an automatic rollback that fires blindly. It is a repeatable evidence pack that gives the operator the state needed to decide.

evidence_pack:
collect:
  - app_configuration_keys_and_labels
  - feature_flag_definitions
  - etag_and_last_modified
  - runtime_identity
  - key_vault_reference_status
  - application_version
  - telemetry_by_flag_state
  - last_refresh_timestamp
propose:
  - restore_previous_key_value
  - restore_previous_targeting_rule
  - restart_only_if_refresh_contract_requires_it
require_human_validation:
  - production_label_change
  - global_flag_disable
  - secret_reference_change
  - application_release_rollback

This keeps automation useful without giving it permission to hide the cause. The operator should see the proposed rollback and the evidence that supports it.

Validate after rollback

After the rollback, validate from the application path, not only from App Configuration. The expected proof is: the effective value changed, the runtime refreshed it, the failing path recovered, and no other ring was unintentionally changed.

Validation
App Configuration value or targeting rule matches the rollback decision
Runtime logs show the new ETag and expected label
Error rate and latency recover on the affected path
Unaffected rings keep their previous values
Key Vault references still resolve
Incident record includes before and after snapshots

Rollback is incomplete when
Only the portal value changed
The runtime did not refresh
The flag state is absent from telemetry
Another label now carries the risky value
The application still fails after the configuration rollback

Conclusion

Azure App Configuration makes runtime change safer only when the team can explain what the application actually read. Feature flags, labels, Key Vault references and refresh behavior are production controls, not convenient switches.

The practical decision is to rollback the narrowest proven object: the flag, the label, the targeting rule, the identity path or the application release. Once the evidence pack exists, configuration stops being an invisible side channel and becomes an operable part of the production runbook.