Azure Monitor: diagnose missing logs before changing alerts

An alert that stops firing is not always a bad rule. In Azure production environments, the problem often sits lower in the chain: a deleted Diagnostic Setting, a modified Data Collection Rule, a table that stopped ingesting, a moved workspace, an aggressive transformation, a retention change, or an ingestion cost reduction made too quickly. The symptom is dangerous because it looks quiet: dashboards get cleaner, alerts stop making noise, but operations lose part of their evidence.

The use case is an Azure application operated with Application Insights, Log Analytics, Diagnostic Settings on managed resources, a few DCRs for VMs or agents, and KQL alerts used during incidents. After a deployment or logging rationalization effort, the team notices that some errors no longer appear. The runbook goal is to diagnose the observability chain before changing thresholds, recreating alerts or assuming the service is healthier.

Describe the expected signal

Before searching with KQL, write down the signal that should exist. Missing logs should be diagnosed as a production path: producer, collection configuration, destination, table, transformation, alert and operational consumer.

Expected signal
Source resource: app-orders-prod / Application Gateway / Key Vault / worker VM
Expected event: application error, WAF block, Key Vault denial, runtime exception
Destination: workspace log-prod-euw
Expected table: AppRequests, AzureDiagnostics, AppExceptions, KeyVaultAuditEvents
Comparison window: 24h before change / 2h after change
Consumers: SLO alert, incident dashboard, rollback runbook, post-incident review

Questions before correction
Is the service still producing the event?
Is collection still enabled?
Did the destination change?
Is the expected table ingesting data?
Is a transformation filtering the signal?
Does the alert query the right table and window?

This contract prevents the team from starting with the alert rule. If the table no longer receives data, changing a threshold does not repair observability; it only adapts the alarm to a broken sensor.

Compare before and after the change

The first query should check signal presence, not business meaning. Compare the same scope, table and granularity before and after the suspicious window.

let changeStart = datetime(2026-06-27T07:30:00Z);
let beforeWindow = 24h;
let afterWindow = 4h;
let serviceName = "orders-api";
AppRequests
| where TimeGenerated between ((changeStart - beforeWindow) .. (changeStart + afterWindow))
| where AppRoleName == serviceName
| summarize
  rows = count(),
  failed = countif(Success == false),
  operations = dcount(OperationName)
by phase = iff(TimeGenerated < changeStart, "before", "after"), bin(TimeGenerated, 30m)
| order by TimeGenerated asc

If volume drops to zero, move down toward ingestion. If volume stays stable but errors disappear, check the service, success definition, transformations and filtered dimensions.

Check Diagnostic Settings

For many Azure resources, platform logs arrive through Diagnostic Settings. An infrastructure change can remove a category, switch the workspace, disable metrics or send logs to Storage instead of Log Analytics.

RESOURCE_ID="/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-prod/providers/Microsoft.Network/applicationGateways/agw-prod"

az monitor diagnostic-settings list --resource "$RESOURCE_ID" --query "[].{name:name,workspace:workspaceId,logs:logs[].{category:category,enabled:enabled},metrics:metrics[].{category:category,enabled:enabled}}" --output json

az monitor diagnostic-settings categories list --resource "$RESOURCE_ID" --query "value[].{name:name,type:categoryType}" --output table

Attach this state to the change ticket. A clean rollback needs to restore the Diagnostic Setting name, enabled categories, destination and collected metrics.

Inspect DCRs, agents and transformations

Data Collection Rules add a useful but sometimes less visible layer: resource association, data sources, destinations, KQL transformations and streams. A transformation can be syntactically valid while removing the signal operations need.

az monitor data-collection rule association list-by-resource --resource "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-prod/providers/Microsoft.Compute/virtualMachines/vm-worker-01" --query "[].{name:name,rule:dataCollectionRuleId,description:description}" --output table

az monitor data-collection rule show --resource-group rg-observability-prod --name dcr-linux-workers-prod --query "{destinations:destinations,streams:streamDeclarations,dataFlows:dataFlows}" --output json

Prioritize missing associations, destinations that no longer point to the expected workspace, and transformations filtering on Severity, Facility, Category, ResultType or application fields.

Read ingestion health as evidence

An alert may be quiet because the service is healthy, because collection is broken, or because ingestion is delayed. Read table health and compare several sources before deciding.

let window = 6h;
union withsource = TableName
AppRequests,
AppExceptions,
AppDependencies,
AzureDiagnostics
| where TimeGenerated > ago(window)
| summarize
  rows = count(),
  firstSeen = min(TimeGenerated),
  lastSeen = max(TimeGenerated)
by TableName, bin(TimeGenerated, 30m)
| order by TableName asc, TimeGenerated asc

Add the platform logs available in your environment: Azure activity, configuration changes, agent events, workspace metrics or administrative queries. The point is to prove whether the silence comes from no event or no collection.

Find configuration changes

Missing logs often follow a legitimate operation: Terraform cleanup, policy change, workspace migration, cost reduction, DCR change or alert redesign. Activity logs provide the entry point.

let changeWindow = 48h;
AzureActivity
| where TimeGenerated > ago(changeWindow)
| where OperationNameValue has_any (
  "MICROSOFT.INSIGHTS/DIAGNOSTICSETTINGS/WRITE",
  "MICROSOFT.INSIGHTS/DIAGNOSTICSETTINGS/DELETE",
  "MICROSOFT.INSIGHTS/DATACOLLECTIONRULES/WRITE",
  "MICROSOFT.INSIGHTS/SCHEDULEDQUERYRULES/WRITE",
  "MICROSOFT.OPERATIONALINSIGHTS/WORKSPACES/WRITE")
| project TimeGenerated,
        OperationNameValue,
        ActivityStatusValue,
        Caller,
        ResourceGroup,
        ResourceProviderValue,
        ResourceId,
        CorrelationId
| order by TimeGenerated desc

If an operation lines up with the volume drop, do not fix yet. First confirm whether it was intentional, whether it touched every expected resource, and whether a configuration rollback exists.

Separate log savings from evidence loss

Reducing ingestion can be necessary. But savings should come from an explicit decision, not from accidental loss of visibility. Classify signals before cutting them.

Keep during incidents without debate
User errors
Authentication and access denials
WAF, gateway and API gateway events
Deployments, revisions and configuration changes
Automation actions and execution identities

Reduce with validation
Verbose debug traces
Repetitive events without correlation value
Metrics already covered elsewhere
Non-critical dependency logs outside incident windows

Never remove without an alternative
Signal used by an SLO alert
Signal used by rollback validation
Signal needed for security evidence
Signal linking user, operation and version

This classification lets the team correct without going back to unlimited ingestion. The goal is to restore useful evidence, not to log everything by reflex.

Decide: restore, fix forward or adapt the alert

The decision should come after collection evidence. A silent alert may require restoring a Diagnostic Setting, fixing a DCR, rolling back Terraform or adapting a query. These actions do not carry the same risk.

Restore previous configuration
Diagnostic Setting deleted or destination changed
DCR detached from a critical resource
Table empty while the service still produces events
Known low-risk rollback exists

Fix forward
Transformation is too broad but intent is correct
Category missing on a limited scope
Target workspace is correct but stream is incomplete
Ingestion test is immediately available

Adapt the alert
Logs are present
Schema or table changed intentionally
Old signal has a documented replacement
Query is replayed on before/after data with expected result

Do not act yet
Drop explained by real absence of events
Window too short to conclude
Temporary ingestion delay confirmed
No operational consumer is affected

Do not modify the alert until the collection chain is proven. Otherwise, the team may tune the alarm to a broken sensor.

Validate after correction

Validation must replay the signal end to end: event produced, ingestion visible, alert query active and evidence attached to the change.

Minimum validation
Diagnostic Settings or DCR match the expected state
The table receives new rows after correction
The KQL query finds the test signal
The alert targets the right workspace and table
The incident dashboard is coherent again
Expected cost or volume is accepted
The correction rollback is documented

Evidence to keep
State before correction
Applied change
Timestamp when logs returned
Validation KQL query
Decision: restore, fix forward or adapt alert

If the signal returns but the alert remains quiet, move the investigation to the query, evaluation window, dimensions, action group or frequency. Collection is operable again.

Conclusion

Missing Azure Monitor logs should be handled as an observability incident, not as an alert-tuning task. The right sequence is to describe the expected signal, compare before and after, check Diagnostic Settings, DCRs, destinations, transformations and ingestion, then decide what to change.

The healthy decision is simple: restore when critical collection disappeared, fix forward when the defect is bounded, adapt the alert when the signal intentionally changed, and change nothing when the silence is genuinely explained. That keeps incidents from looking cleaner in dashboards while becoming more opaque for operations.