Azure Managed Grafana: diagnose a dashboard or alert before changing KQL

A Grafana dashboard that goes empty, or an alert that stops firing, quickly looks like a KQL problem. In production, that is rarely the first thing to change. The failure may come from the Azure Monitor datasource, a managed identity that lost a role, a replaced Log Analytics workspace, a dashboard variable, a time range that is too narrow, or an alert rule that no longer runs the same query as the visible panel.

The use case is an Azure Managed Grafana instance used by an operations team to monitor a critical application. Panels read Azure Monitor and Log Analytics, some Grafana alerts notify on-call engineers, and dashboards are used as evidence during deployments. The runbook must decide whether to fix access, restore a datasource, adjust a variable, roll back a dashboard, or only then change the KQL.

Freeze the observed symptom

Before touching the query, describe exactly what changed. An empty panel, a flat zero value and a silent alert do not lead to the same diagnosis.

Symptom
Grafana instance: grafana-ops-prod
Dashboard: api-production-overview
Panel or rule: 5xx rate / api-prod-high-error-rate
Expected datasource: Azure Monitor - prod
Expected workspace: law-observability-prod
First detection: 2026-07-07 09:20 UTC
Last known change: dashboard import, role assignment, workspace migration or application release

Questions before correction
Does the issue affect one panel, the whole dashboard or all alerts?
Does the query return data in Log Analytics outside Grafana?
Does the Grafana datasource point to the right tenant, subscription and workspace?
Does the identity used by Grafana still have read permissions?
Is there a previous version of the dashboard or alert rule?

This note prevents the common reflex: rewriting a KQL query that still works while Grafana can no longer query the right source.

Separate datasource, workspace and identity

Azure Managed Grafana can read Azure Monitor with a managed identity or an Entra ID application. The diagnosis must prove which identity actually runs the query and where it is authorized.

GRAFANA_NAME="grafana-ops-prod"
RESOURCE_GROUP="rg-observability-prod"
WORKSPACE_ID="/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-logs-prod/providers/Microsoft.OperationalInsights/workspaces/law-observability-prod"

az grafana show --name "$GRAFANA_NAME" --resource-group "$RESOURCE_GROUP" --query "{name:name,identity:identity,endpoint:properties.endpoint,provisioningState:properties.provisioningState}" --output json

PRINCIPAL_ID="$(az grafana show --name "$GRAFANA_NAME" --resource-group "$RESOURCE_GROUP" --query "identity.principalId" --output tsv)"

az role assignment list --assignee "$PRINCIPAL_ID" --scope "$WORKSPACE_ID" --query "[].{role:roleDefinitionName,scope:scope}" --output table

For Log Analytics, the identity must at least be able to read the required data. If the role was granted on an old workspace, on the wrong subscription or only on a resource group that does not contain the effective workspace, Grafana will show an application-looking symptom while the cause is observability access.

Replay the query outside Grafana

The next test removes Grafana from the path. If the query returns data directly from Log Analytics, the issue is probably in the datasource, variables, Grafana permissions or panel rendering. If it returns nothing, the diagnosis moves upstream to ingestion, Diagnostic Settings, table selection or time range.

let ServiceName = "api-prod";
let Window = 30m;
AppRequests
| where TimeGenerated > ago(Window)
| where AppRoleName == ServiceName
| summarize Requests = count(), Failures = countif(ResultCode startswith "5") by bin(TimeGenerated, 5m)
| extend FailureRate = todouble(Failures) / todouble(Requests)
| project TimeGenerated, Requests, Failures, FailureRate
| order by TimeGenerated asc

Then compare three things: the panel query, the alert rule query and the query executed in Log Analytics. A Grafana alert copied from a panel can keep an old variable, an old interval or a threshold that no longer matches the dashboard.

Check variables and time windows

A dashboard can become empty without a data outage. A variable such as environment, service, cluster or subscription may lose its default value, point to a renamed label, or be fed by a different datasource than the panel.

Variables to check
subscription: default value and allowed list
workspace: mapping to the Azure Monitor datasource
environment: prod, production or prd according to real logs
service: AppRoleName, cloud_RoleName or application tag
interval: resolution compatible with the alert window
region: non-empty filter after multi-region deployment

Hold a KQL change when
The variable returns an empty list
The panel uses prod but the alert uses production
The dashboard reads one workspace and the alert another one
The alert window is shorter than the observed ingestion delay

This check is especially useful after a dashboard import, a workspace migration or a tag standardization effort. The KQL may be correct while receiving an impossible filter.

Read execution and alert evidence

The decision should be evidence-based. On the Azure Monitor side, verify that data is arriving. On the Grafana side, capture rule execution, the NoData, Error or OK state, and the datasource message.

let ChangeStart = datetime(2026-07-07 08:30:00);
let ChangeEnd = datetime(2026-07-07 10:00:00);
AppRequests
| where TimeGenerated between (ChangeStart .. ChangeEnd)
| summarize Requests = count(), Failures = countif(ResultCode startswith "5") by AppRoleName, bin(TimeGenerated, 5m)
| order by TimeGenerated desc

If data exists in Log Analytics but not in Grafana, do not change tables or Diagnostic Settings yet. The next check is datasource, identity, variables and alert rule. If data is missing everywhere, the problem is upstream: ingestion, SDK, Diagnostic Settings or the logging pipeline.

Decide fix, rollback or KQL change

The decision should remain reversible. An emergency KQL change can hide the real cause and make the next incident harder to diagnose.

Fix access
The query works in Log Analytics
Grafana points to the right datasource
The Grafana identity lost Reader or Log Analytics Reader
The correction is a role assignment scoped to the expected workspace

Restore datasource or dashboard
Workspace, tenant or subscription changed unintentionally
A dashboard variable filters out every series
The alert rule no longer matches the validated panel
A previous known-good version exists in Git or Grafana history

Change KQL
The table or schema really changed
The query also fails outside Grafana
The new filter is documented and tested over a representative window
Alert and dashboard are updated together

Rollback
The imported dashboard breaks several critical panels
Alerts move to NoData or Error without application evidence
The required role assignment cannot be validated during the window
The previous version restores the expected signal

After rollback, replay the evidence query and verify at least one alert rule evaluation. A visually restored dashboard is not enough if the alert remains silent.

Conclusion

A Grafana incident is not automatically a KQL incident. In Azure Managed Grafana, the full path crosses datasource, identity, workspace, variables, queries, alert rules and ingestion delay.

The practical reflex is to qualify the signal before changing query code: replay outside Grafana, prove identity, compare panel and alert, verify variables, then choose between access correction, configuration restore, KQL change or rollback. The dashboard becomes an operations tool again, not just an interface that appears green.