Infrastructure

Azure Application Insights: diagnose sampling before changing alerts

A production runbook for qualifying apparent Application Insights telemetry loss with sampling, ingestion, SDK configuration, KQL, alerts, validation and rollback before changing thresholds.

19 Jul 2026 azureapplication-insightsazure-monitorobservabilitytelemetrysamplingkqlalertsrunbookrollbackproduction

When an Azure Monitor alert goes quiet after a deployment, the fastest reaction is often to lower a threshold, rewrite the KQL query or assume the application improved. That is brittle. With Application Insights, fewer requests, dependencies, traces or exceptions can mean real traffic changed, but it can also mean sampling, SDK configuration drift, a renamed role, ingestion loss or an alert query reading the wrong signal.

The use case is a billing-api service instrumented with Application Insights. After a release, the error-rate alert no longer fires while support still reports intermittent failures. The runbook goal is to decide whether the alert should change, the instrumentation should be rolled back, the sampling policy is acceptable, or the incident must remain open because evidence is incomplete.

Freeze the telemetry contract

Start by stating what telemetry must prove. Application Insights is not just a place to search logs. It is the contract that links user symptom, request, dependency, exception, application trace, version and deployment operation.

yaml telemetry-contract.yml

service:
name: billing-api
environment: production
app_insights: ai-platform-prod
expected_cloud_role: billing-api-prod

signals_required:
- requests with resultCode and success
- dependencies for sql and http backends
- exceptions linked to operation_Id
- traces carrying deployment_version
- availability or synthetic probe result

change_context:
deployment: dep-20260719-0840
suspected_change: sdk configuration and sampling policy
decision_needed: alert_change_or_instrumentation_rollback
rollback_owner: platform-operations

If this contract is unclear, the team will mix three questions: did the application improve, is telemetry still arriving, and does the alert still read the right signal?

Separate real traffic, sampling and ingestion

A volume drop has more than one explanation. Before touching the alert, classify the symptom.

text telemetry-drop-classes.txt

Real traffic drop
Requests, dependencies, traces and probes decrease together
Typical decision: check routing, frontend behavior or business calendar before changing the alert

More aggressive sampling
Stored rows decrease but ratios, operation_Id and itemCount indicate extrapolation
Typical decision: read rates with itemCount and confirm critical signals are not lost

Ingestion degradation
Some signal types disappear or arrive late
Typical decision: qualify pipeline, Diagnostic Settings, workspace, quotas and latency before changing KQL

Broken instrumentation
Role name, connection string, SDK or enrichment changed
Typical decision: roll back instrumentation or fix configuration before recalibrating alerts

Obsolete alert query
Data exists but not under the same field, role or table
Typical decision: fix the query with evidence before changing the threshold

This separation avoids the wrong repair: making the alert more sensitive while telemetry is incomplete, or rolling back the application when only the role name changed.

Read volume with itemCount

In Application Insights, sampling can reduce stored rows without reducing the number of represented events. Tables such as requests, dependencies, traces and exceptions may carry itemCount. A query that only counts rows can therefore mislead operations.

kusto 01-application-insights-volume-with-itemcount.kql

let StartTime = datetime(2026-07-19T07:30:00Z);
let EndTime = datetime(2026-07-19T10:00:00Z);
union
(requests
 | where timestamp between (StartTime .. EndTime)
 | extend signal = "requests", weight = toint(coalesce(itemCount, 1))),
(dependencies
 | where timestamp between (StartTime .. EndTime)
 | extend signal = "dependencies", weight = toint(coalesce(itemCount, 1))),
(exceptions
 | where timestamp between (StartTime .. EndTime)
 | extend signal = "exceptions", weight = toint(coalesce(itemCount, 1))),
(traces
 | where timestamp between (StartTime .. EndTime)
 | extend signal = "traces", weight = toint(coalesce(itemCount, 1)))
| summarize storedRows=count(), representedEvents=sum(weight) by signal, bin(timestamp, 15m)
| order by timestamp asc, signal asc

If storedRows drops but representedEvents stays coherent, the alert may need to read weighted signal or an adapted ratio. If both drop without explanation, continue with ingestion and instrumentation checks.

Verify role, version and operation

A deployment can break an alert without breaking the application: cloud_RoleName changes, version enrichment disappears, traces lose operation_Id, or dependencies are emitted under another name.

kusto 02-role-version-operation-drift.kql

let StartTime = datetime(2026-07-19T07:30:00Z);
let EndTime = datetime(2026-07-19T10:00:00Z);
requests
| where timestamp between (StartTime .. EndTime)
| summarize requests=count(),
          representedRequests=sum(toint(coalesce(itemCount, 1))),
          operations=dcount(operation_Id),
          versions=make_set(tostring(customDimensions.deployment_version), 10),
          sampleNames=make_set(name, 10)
by cloud_RoleName, bin(timestamp, 15m)
| order by timestamp asc, cloud_RoleName asc

A changed cloud_RoleName after deployment is strong evidence: the alert may be watching the old role while the new release emits elsewhere. In that case, changing the threshold hides the problem. Fix the filter, restore the expected role or document the migration.

Correlate request, dependency and exception

Before declaring an error gone, prove that the full chain still exists. An API can keep requests while losing dependencies, or keep exceptions that are no longer linked to operations.

kusto 03-correlate-request-dependency-exception.kql

let Window = 2h;
let FailedRequests =
requests
| where timestamp > ago(Window)
| where cloud_RoleName == "billing-api-prod"
| where success == false or toint(resultCode) >= 500
| project operation_Id, requestTime=timestamp, name, resultCode, url;
FailedRequests
| join kind=leftouter (
  dependencies
  | where timestamp > ago(Window)
  | project operation_Id, dependencyTime=timestamp, target, dependencyName=name, dependencyResultCode=resultCode, dependencySuccess=success
) on operation_Id
| join kind=leftouter (
  exceptions
  | where timestamp > ago(Window)
  | project operation_Id, exceptionTime=timestamp, exceptionType=type, problemId
) on operation_Id
| summarize failedRequests=dcount(operation_Id),
          dependencyTargets=make_set(target, 10),
          exceptionTypes=make_set(exceptionType, 10),
          missingDependencies=countif(isempty(target)),
          missingExceptions=countif(isempty(exceptionType))
by bin(requestTime, 15m), resultCode
| order by requestTime desc

This correlation shows whether the alert can still explain an incident. A signal that counts failures without usable dependencies or exceptions may fire, but it will not help on-call much.

Check configuration changes

The application deployment is not the only possible change. SDK sampling, connection string, workspace ingestion, Diagnostic Settings, Application Insights resource or environment variable may have moved.

text configuration-checklist.txt

Check before changing the alert
Connection string or instrumentation key points to the expected resource
Application Insights SDK is loaded in the new revision
Cloud role name is stable or the migration is documented
Sampling does not exclude exceptions, failed requests or critical dependencies
customDimensions used by KQL still exist
Data lands in the expected workspace
Ingestion latency matches the alert window
Quotas, filters and transformations have not changed

If one of these points changed during the incident window, the priority is to restore reliable telemetry. The alert comes next.

Qualify the alert query

A useful Application Insights alert should read like an operations decision. It should name the role, window, ratio, minimum volume and evidence that justify rollback or continued operation.

kusto 04-alert-query-with-sampling-awareness.kql

let RoleName = "billing-api-prod";
let MinimumRepresentedRequests = 50;
requests
| where timestamp > ago(15m)
| where cloud_RoleName == RoleName
| extend weight = toint(coalesce(itemCount, 1))
| summarize representedRequests=sum(weight),
          representedFailures=sumif(weight, success == false or toint(resultCode) >= 500),
          storedRows=count(),
          sampleOperationIds=make_set_if(operation_Id, success == false or toint(resultCode) >= 500, 5)
| extend failureRate = todouble(representedFailures) / todouble(representedRequests)
| where representedRequests >= MinimumRepresentedRequests
| project representedRequests, representedFailures, failureRate, storedRows, sampleOperationIds

This query is not a universal template. It shows the expected controls: do not alert on three sampled rows, do not ignore itemCount, keep example operations and make the threshold defensible.

Decide validation, rollback or alert change

The decision should separate layers. An alert must not compensate for broken instrumentation.

text application-insights-decision.txt

Validate without change
Represented volume remains coherent
Sampling is understood and documented
Errors, dependencies and exceptions remain correlated
Alert query still reads the right role and workspace

Change the alert
Telemetry is reliable
Threshold or window no longer matches expected behavior
Change keeps a minimum volume and sample operations
Query rollback is ready

Roll back instrumentation
Role name, SDK, connection string or customDimensions drifted
Critical signals disappear or are no longer correlated
The alert would become blind or misleading

Keep incident open
Ingestion is late or incomplete
Sampling cannot prove critical failures
Support still observes the symptom without usable trace evidence

This framing prevents a telemetry gap from becoming a monitoring change.

Prepare rollback

Rollback must cover two objects: instrumentation configuration and the alert rule. They do not always revert together.

yaml telemetry-alert-rollback-card.yml

rollback:
instrumentation:
  restore_connection_string: previous_secret_version
  restore_cloud_role: billing-api-prod
  restore_sampling_policy: previous_sdk_configuration
  validation: requests_dependencies_exceptions_correlated

alert:
  restore_query: alert-rule-version-before-dep-20260719-0840
  restore_threshold: previous_threshold
  validation: controlled_failure_or_synthetic_signal_visible

evidence_required:
  - before_after_kql_result
  - deployment_id
  - alert_rule_version
  - owner_decision
  - next_review_window

Post-rollback validation must show that telemetry returned and that the alert can produce a decision again. Otherwise, the rollback only restores configuration. It does not restore operability.

Conclusion

Before changing an Application Insights alert, prove that the signal still exists, that sampling is interpreted correctly, that requests, dependencies and exceptions remain correlated, and that the data arrives within the expected alert window.

The right decision may be to keep the alert, change its query, roll back instrumentation or keep the incident open. What matters is not confusing alert silence with problem resolution.