Azure Monitor: roll out a DCR transformation without losing telemetry

A team reduces logging cost by removing unused fields in an Azure Monitor Data Collection Rule. The KQL transformation is valid in an editor, deployment succeeds, and the table keeps receiving rows. Hours later, an incident reveals that the request correlation field is gone and one message category is no longer ingested at all.

A DCR transformation is part of the collection pipeline, not a read-time query. It can filter records, rename columns, change types, or produce a schema that does not match the destination. This runbook uses a canary rollout: freeze the telemetry contract, measure a baseline, test representative samples, isolate one source, compare the old and new paths, then retain or roll back the DCR.

Freeze the contract before changing the stream

The change record must say what may disappear and what must remain usable. “Reduce volume” is not a sufficient success criterion.

change:
dcr_current: dcr-app-prod-v12
dcr_candidate: dcr-app-prod-v13
stream: Custom-AppRuntime_CL
destination: law-platform-prod
canary_source: vm-app-canary-01
observation_window: 60m

must_preserve:
- TimeGenerated
- _ResourceId
- CorrelationId
- Severity
- MessageType

allowed_change:
- drop DebugPayload
- keep only approved MessageType values

evidence:
- dcr_json_before_after
- association_before_after
- source_and_destination_row_counts
- null_and_type_checks
- ingestion_latency
- rejected_or_missing_record_signals
- rollback_owner_and_deadline

Preserve the active DCR JSON, its associations, the stream schema, and every alert that queries the table. A field that looks secondary may be a join key in a scheduled query rule, workbook, or incident runbook.

Build a baseline that separates absence from delay

Before the change, measure volume, business categories, required fields, and ingestion latency over a comparable window. A row-count chart alone detects neither a broken type nor the loss of a low-volume category.

let StartTime = datetime(2026-08-03T05:00:00Z);
let EndTime = datetime(2026-08-03T06:00:00Z);
Custom_AppRuntime_CL
| where TimeGenerated between (StartTime .. EndTime)
| summarize
  Rows=count(),
  Sources=dcount(_ResourceId),
  MissingCorrelation=countif(isempty(CorrelationId)),
  MissingSeverity=countif(isempty(Severity)),
  P95IngestionDelay=percentile(ingestion_time() - TimeGenerated, 95)
by MessageType, bin(TimeGenerated, 5m)
| order by TimeGenerated asc

Add a synthetic marker emitted by the canary source at a known interval. It must carry a unique correlation value and follow the same collection path as real events. A missing marker is stronger evidence than a volume drop from an application whose traffic naturally varies.

Review the complete DCR path

A DCR connects sources, streams, transformations, and destinations. Check the incoming stream name, transformKql, output stream, and target table schema. A transformation that works on sample data may still be incompatible with its declared output stream.

SUBSCRIPTION_ID="<subscription-id>"
RESOURCE_GROUP="rg-observability-prod"
DCR_NAME="dcr-app-prod-v12"
API_VERSION="<supported-api-version>"

az rest --method get --url "https://management.azure.com/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/Microsoft.Insights/dataCollectionRules/$DCR_NAME?api-version=$API_VERSION" --output json > dcr-before.json

az resource list --resource-group "$RESOURCE_GROUP" --resource-type Microsoft.Insights/dataCollectionRuleAssociations --output json > dcr-associations-before.json

Use an API version already approved in the infrastructure code or delivery pipeline. The runbook should not introduce an API-version change at the same time as a transformation change; that would add another variable to the investigation.

Test the transformation as a data contract

Create samples for every category, absent fields, long values, and unexpected types. The test must prove the output record by record, not merely show that the syntax parses.

source
| where MessageType in ("request", "dependency", "exception", "audit")
| extend
  CorrelationId = tostring(CorrelationId),
  Severity = tostring(Severity),
  MessageType = tostring(MessageType)
| project
  TimeGenerated,
  _ResourceId,
  CorrelationId,
  Severity,
  MessageType,
  Message

Handle invalid input explicitly. If unknown values must be retained for investigation, send them through a quarantine path supported by the architecture or block production. Silently removing them with a filter turns a data error into missing evidence.

Nominal cases
One row for each allowed MessageType
CorrelationId and _ResourceId retained
Types match the output stream

Edge cases
Missing CorrelationId
Unknown MessageType
Numeric Severity instead of text
Empty or very long Message
Late timestamp

Block deployment when
A required row is filtered
A field used by an alert disappears
An output type does not match the schema
Unknown-value behavior is undefined

Clone the DCR and constrain the canary to one source

Do not replace a shared DCR to test the transformation. Create a candidate version with a distinct name and IaC artifact, then associate only one canary source. Avoid sending the same stream from one source through both rules when that would duplicate records or invalidate the comparison.

The canary should generate representative, reversible traffic. One VM or a small group is enough for a fleet. For a distributed service, use a dedicated instance or an environment whose events can be identified. Record the exact association before and after the switch.

steps:
- deploy candidate DCR without association
- compare candidate JSON with current DCR
- associate only vm-app-canary-01 with candidate
- emit one synthetic event every five minutes
- observe for at least one full alert evaluation cycle
- compare counts, categories, required fields and latency
- expand in bounded batches or restore previous association

stop_conditions:
- synthetic event missing
- source sends duplicate records
- required field becomes null
- unexpected MessageType disappears
- ingestion latency exceeds the agreed baseline
- alert or workbook query fails

Compare expected events with ingested records

The strongest check compares a producer or collector counter with received rows. If that counter does not exist, combine the synthetic marker, collection-agent logs, resource metrics, and the category distribution observed before the change.

let CanaryResource = "/subscriptions/<id>/resourceGroups/rg-app-prod/providers/Microsoft.Compute/virtualMachines/vm-app-canary-01";
let StartTime = datetime(2026-08-03T06:15:00Z);
Custom_AppRuntime_CL
| where TimeGenerated >= StartTime
| where _ResourceId =~ CanaryResource
| summarize
  Rows=count(),
  Correlations=dcount(CorrelationId),
  MissingCorrelation=countif(isempty(CorrelationId)),
  MissingResource=countif(isempty(_ResourceId)),
  P95IngestionDelay=percentile(ingestion_time() - TimeGenerated, 95)
by MessageType, bin(TimeGenerated, 5m)
| order by TimeGenerated asc

Replay the critical alert and workbook queries over the canary window as well. A transformation can preserve every row while breaking a join, JSON extraction, or condition that depends on the previous casing of a value.

Expand through a gate, roll back through reassociation

Expand the candidate DCR in small batches only when the synthetic marker is continuous, expected categories remain present, required fields are populated, latency stays within baseline, and KQL consumers still work. Capture the same evidence after every batch.

Rollback means reassociating sources with the previous DCR, then confirming that the synthetic marker and expected categories return. Do not immediately delete the candidate DCR: retain its JSON, final associations, and exact exposure window for analysis. Replay events only when the source or a buffer actually retained them and the replay is idempotent.

Retain and expand
Synthetic marker remains continuous
Category volumes are explainable
Required fields and types are preserved
Alerts and workbooks remain valid
Latency stays within the accepted baseline

Roll back
Expected row or category is absent
Schema or consumer query breaks
Association creates duplicates
Latency or rejected records are unexplained
Source and destination cannot be compared

After rollback
Restore the previous association
Prove the canary marker has returned
Retain candidate DCR artifacts
Bound the potential data-loss window
Fix the contract before retrying

Conclusion

A DCR transformation is validated neither by a successful ARM deployment nor by a few visible Log Analytics rows. It is validated when a canary proves that expected events arrive once, with the correct schema, within the agreed delay, and that alerts continue to make the same decision.

The production gate is straightforward: expand only with comparable evidence at source and destination. At the first missing signal or ambiguous schema, restore the previous association, measure the affected window, and correct the transformation before another attempt.