Azure Monitor: diagnose ingestion latency before changing a KQL alert

A KQL alert that fires fifteen minutes after an incident starts often pushes teams toward the wrong fix. They widen the window, lower a threshold, increase the evaluation frequency, and only later discover that the logs were already late when they reached Log Analytics. The alert rule was not the root cause. The delay sat between the real event, the application timestamp, ingestion, table visibility and rule evaluation.

The use case is an Azure application monitored with Application Insights, Log Analytics, Diagnostic Settings, Data Collection Rules and KQL alerts. After a collection change, deployment, volume spike or logging cost optimization, alerts still exist but fire too late. The runbook goal is to decide whether to fix collection, temporarily adjust the KQL window, roll back an observability change, or accept a documented delay for a non-critical signal.

Name the delay being measured

Start by separating three clocks. TimeGenerated describes when the producer saw the event. ingestion_time(), when available, describes when the row became queryable in Log Analytics. The alert fire time describes when Azure Monitor evaluated the query and sent the action.

Signal to qualify
Alert: api-orders-errors-prod
Table: AppExceptions or AppRequests
Source resource: app-orders-prod
User symptom: HTTP 500 errors on checkout
First observed symptom time: 2026-07-14T07:42:00Z
First row visible in Log Analytics
Alert fire time
KQL window: 5m, 15m or 30m
Evaluation frequency: 1m, 5m or 10m
Action group: incident, on-call, automation or ticket

Questions before changing the rule
Are logs produced late or ingested late?
Is the affected table the only one delayed?
Does the rule query a window shorter than observed latency?
Is the issue constant or limited to a volume spike?
Did a DCR, Diagnostic Setting or sampling change just happen?

Without this separation, the team may turn the alert into an overly broad net. A longer window can hide latency, but it can also merge several incidents and delay alert recovery.

Measure latency in the affected table

The first query measures the gap between event time and ingestion time. Run it on the table used by the alert, not only on a general table that appears healthy.

let Window = 6h;
let Service = "orders-api";
AppExceptions
| where TimeGenerated > ago(Window)
| where AppRoleName == Service
| extend IngestedAt = ingestion_time()
| extend IngestionDelay = IngestedAt - TimeGenerated
| summarize
  rows = count(),
  p50 = percentile(IngestionDelay, 50),
  p95 = percentile(IngestionDelay, 95),
  p99 = percentile(IngestionDelay, 99),
  maxDelay = max(IngestionDelay)
by bin(TimeGenerated, 15m)
| order by TimeGenerated asc

If ingestion_time() is not usable for the table, compare the first time an expected event appears with the application time carried in the payload or in a controlled dimension. The point is to avoid confusing no error with no evidence.

Compare tables, resources and collection paths

Latency isolated to an application table is not handled like workspace-wide latency. Compare critical tables: requests, exceptions, traces, platform logs, WAF, Key Vault, Service Bus or VM logs depending on the scope.

let Window = 6h;
union withsource=TableName
(AppRequests | where TimeGenerated > ago(Window) | project TimeGenerated, IngestedAt=ingestion_time()),
(AppExceptions | where TimeGenerated > ago(Window) | project TimeGenerated, IngestedAt=ingestion_time()),
(AppTraces | where TimeGenerated > ago(Window) | project TimeGenerated, IngestedAt=ingestion_time()),
(AzureDiagnostics | where TimeGenerated > ago(Window) | project TimeGenerated, IngestedAt=ingestion_time())
| extend IngestionDelay = IngestedAt - TimeGenerated
| summarize rows=count(), p95=percentile(IngestionDelay, 95), maxDelay=max(IngestionDelay) by TableName, bin(TimeGenerated, 30m)
| order by TimeGenerated asc, TableName asc

If all tables drift, inspect the workspace, volume and ingestion path. If only one table drifts, look at the SDK, sampling, DCR transformation, Diagnostic Settings category or producer.

Check collection, sampling and transformations

A collection change can delay or filter the signal without breaking ingestion completely. The useful evidence is the exact configuration at incident time: Diagnostic Settings, DCRs, destinations, transformations, application sampling and agents.

RESOURCE_ID="/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-prod/providers/Microsoft.Web/sites/app-orders-prod"
WORKSPACE_ID="/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-obs/providers/Microsoft.OperationalInsights/workspaces/log-prod"

az monitor diagnostic-settings list --resource "$RESOURCE_ID" --query "[].{name:name,workspace:workspaceId,logs:logs,metrics:metrics}" --output json

az monitor data-collection rule list --resource-group rg-obs --query "[].{name:name,location:location,streams:properties.dataFlows[].streams,destinations:properties.destinations}" --output json

az monitor log-analytics workspace show --ids "$WORKSPACE_ID" --query "{name:name,retention:retentionInDays,sku:sku.name,provisioningState:provisioningState}" --output json

Also capture application changes: SDK version, log level, adaptive sampling, trace volume and any expensive enrichment. The alert can be correct while the pipeline feeding it has become too slow.

Read the alert rule as a time contract

A KQL alert is a contract between query window, evaluation frequency, threshold logic and action group. If p95 ingestion latency is longer than the query window, the alert can miss the signal or fire after the fact.

RG="rg-obs-prod"
RULE="api-orders-errors-prod"

az monitor scheduled-query show --resource-group "$RG" --name "$RULE" --query "{enabled:enabled,scopes:scopes,evaluationFrequency:evaluationFrequency,windowSize:windowSize,severity:severity,criteria:criteria,actions:actions}" --output json

Do not change the rule before proving that the signal arrives too late. If collection is slow, a wider window may be a temporary workaround, not the durable fix.

Decide fix, workaround or rollback

Write the decision with its risk level. A critical user-impact alert cannot tolerate the same latency as a daily capacity report.

decision:
correct_collection:
  when:
    - latency_increased_after_dcr_or_diagnostic_change
    - only_one_table_or_resource_is_delayed
    - alert_window_was_valid_before_change
  actions:
    - restore_previous_collection_configuration
    - remove_filter_or_transformation_that_delays_signal
    - validate_sampling_and_sdk_version

temporary_alert_adjustment:
  when:
    - ingestion_latency_is_known_and_bounded
    - signal_is_not_user_critical_or_has_secondary_alert
    - rollback_collection_is_not_immediate
  actions:
    - increase_window_size_with_expiry
    - document_expected_detection_delay
    - keep_user_symptom_alert_active

rollback_observability_change:
  when:
    - critical_alert_misses_incident_window
    - latency_source_is_recent_change
    - no_equivalent_signal_exists
  actions:
    - restore_previous_dcr_or_diagnostic_setting
    - revert_sampling_change
    - re-run alert replay query

block_rule_change:
  when:
    - no_ingestion_evidence
    - missing_table_or_ambiguous_time_source
    - action_group_delay_not_separated_from_log_delay

The common trap is compensating uncontrolled latency with an ever-larger window. It may calm today’s incident, but the team no longer knows whether it is detecting a live issue or one that already ended.

Replay the alert before closing the incident

After the fix, replay the alert logic on a historical window that contains the symptom. Validation must prove that the signal would have been detected within an acceptable delay.

let IncidentStart = datetime(2026-07-14T07:42:00Z);
let IncidentEnd = datetime(2026-07-14T08:10:00Z);
let DetectionWindow = 5m;
AppExceptions
| where TimeGenerated between (IncidentStart .. IncidentEnd)
| where AppRoleName == "orders-api"
| summarize errors=count() by bin(TimeGenerated, DetectionWindow)
| where errors >= 5
| order by TimeGenerated asc

Then compare it with ingestion latency after correction. If the signal still arrives after the decision window, the rule is not validated even if the portal eventually shows a healthy state.

Conclusion

A delayed Azure Monitor alert is not only a threshold problem. It is a timing-chain problem: log production, collection, ingestion, query, evaluation and notification. Changing the rule too early is like adjusting a clock before knowing which one is slow.

The right runbook measures latency in the actual table, compares collection paths, captures collection configuration, reads the rule as a time contract and chooses between fix, temporary workaround or rollback. The final decision should state which detection delay remains acceptable and what evidence will keep it visible.