Azure Managed Grafana: diagnose API access before recreating a token

The pipeline that provisions Azure Managed Grafana dashboards suddenly fails with a 401 or 403. Existing dashboards remain visible and their data sources still respond. The tempting fix is to generate another service account token, grant it Admin and rerun the full synchronization.

That response can restore the job while hiding the actual failure: an expired token, a stale secret on one runner, the wrong Microsoft Entra audience, a role assigned to another principal, or a request sent to the wrong workspace. It also creates another credential without proving that the old one was retired. The running case is a CI job that reads folders, compares versioned dashboards and publishes only the diff. The goal is to restore that path with least privilege, a canary and an explicit rollback.

Freeze the failed request

Preserve one execution before changing credentials. Record the runner, commit, Grafana endpoint, API route, HTTP method, response status, timestamp, expected identity type and secret reference. Do not paste the bearer token or complete headers into the incident ticket.

incident:
detected_utc: 2026-10-07T05:42:00Z
workspace: amg-observability-prod
endpoint: https://<workspace>.grafana.azure.com
runner: dashboard-sync-prod
pipeline_run: <run-id>
commit: <sha>
request: GET /api/org
response_status: 401

expected_identity:
mode: service_account_token
service_account: dashboard-publisher
token_name: ci-prod-2026-09
secret_reference: kv-observability/grafana-ci-token

containment:
publish_enabled: false
read_only_diagnostics: true
last_qualified_dashboard_bundle: dashboards-2026-10-06.3

A 401 or 403 is a symptom, not sufficient proof of expiration. First replay a bounded read from the same runtime. A request from an administrator’s workstation validates that person’s identity and network path, not those of the failing runner.

Identify the token family actually in use

Azure Managed Grafana supports two automation paths. A Grafana service account token is an opaque secret attached to an account and its role. A Microsoft Entra token is issued to a user, service principal or managed identity that has an Azure Managed Grafana role on the resource.

The two paths require different diagnostics. For a service account, check the account name, enabled state, role, token name and expiry. For Microsoft Entra, check the runtime identity, token audience and RBAC assignment. Do not infer the mechanism from a variable called GRAFANA_TOKEN; prove where the value came from during the affected execution.

GRAFANA_NAME="amg-observability-prod"
SERVICE_ACCOUNT="dashboard-publisher"

az grafana service-account show \
--name "$GRAFANA_NAME" \
--service-account "$SERVICE_ACCOUNT" \
--output jsonc

az grafana service-account token list \
--name "$GRAFANA_NAME" \
--service-account "$SERVICE_ACCOUNT" \
--query "[].{name:name,expiration:expiration,expired:hasExpired}" \
--output table

The list exposes metadata, not token values. Compare the expected token name with the secret version injected into the job and its deployment timestamp. If the Azure-side token remains valid but the runner has cached an older Key Vault version, creating a third token does not repair secret distribution.

Separate authentication, authorization and target

Treat the incident as four distinct contracts:

1. Target
 Endpoint, workspace, tenant and API route match the intended environment

2. Acquisition
 The runtime gets the expected token type from the approved source

3. Authentication
 The token is present, unexpired and intended for Azure Managed Grafana

4. Authorization
 The service account or principal has the minimum role required by the operation

End-to-end evidence
 execution -> identity -> audience/expiry -> role -> endpoint -> operation

For Microsoft Entra, the data-plane audience is https://dashboard.azure.com. Code should request the https://dashboard.azure.com/.default scope. A token for ARM, Microsoft Graph or Azure Monitor can be valid and signed while remaining unusable against the Grafana API.

GRAFANA_ENDPOINT="https://<workspace>.grafana.azure.com"

TOKEN=$(az account get-access-token \
--resource https://dashboard.azure.com \
--query accessToken -o tsv)

curl --silent --show-error \
--output grafana-org-response.json \
--write-out "status=%{http_code}\n" \
--header "Authorization: Bearer $TOKEN" \
"$GRAFANA_ENDPOINT/api/org"

This is an interactive test. In the runner, acquire the token with the workload identity, not an operator’s Azure CLI session. Then inspect the Grafana Viewer, Grafana Editor or Grafana Admin assignment at workspace scope. A dashboard synchronization normally needs Editor; Admin should not become the default incident workaround.

Prove which secret the runner loaded

A service account token is shown only when it is created. Azure cannot display it again for comparison with the deployed secret. Compare fingerprints inside controlled environments, never the raw values, and do not retain the fingerprint in public logs.

# Run in a restricted job; never print GRAFANA_TOKEN.
printf '%s' "$GRAFANA_TOKEN" \
| sha256sum \
| awk '{print "token_fingerprint=" substr($1,1,12)}'

# Compare with the fingerprint recorded when the secret was deployed,
# then remove the output when the incident is closed.

Also inspect the secret-resolution path: pinned or current version, refresh time, whether the runner must restart, variables overridden at pipeline scope, and effective masking. A correct secret in Key Vault that is wrong inside the process is still a deployment failure.

Rotate with an overlap window

If the old token is expired or compromised, create a named, time-bounded token on the existing service account. Do not recreate the account while its role and scope remain correct; doing so discards useful audit continuity.

az grafana service-account token create \
--name "amg-observability-prod" \
--service-account "dashboard-publisher" \
--token "ci-prod-2026-10-canary" \
--time-to-live 15d \
--output json

Capture the value once into the approved vault without sending it through logs. Deploy it to one canary runner, then perform three probes: read the organization, read a known folder, and make an idempotent write to a test dashboard in a noncritical folder. Compare the expected diff, observed identity and authentication events.

During overlap, retain the old token only if it is still safe and required for rollback. Once every consumer is inventoried and the canary is expanded, remove it explicitly with az grafana service-account token delete. Creating a token does not automatically revoke previous ones.

Prefer Microsoft Entra when the runtime supports it

When the automation runs on an Azure resource that can carry a managed identity, evaluate Microsoft Entra as the target path. Assign the minimum Grafana role to the principal on the workspace, acquire a token for https://dashboard.azure.com/.default, then replay the same probes.

Do not confuse the automation identity with the identity Grafana uses to query data sources. These are separate flows: the runner calls the Grafana API; the workspace queries Azure Monitor, Prometheus or another source. Repairing one does not prove the other.

Keep both authentication paths only for a bounded migration window. After validating Microsoft Entra, remove the runner secret, revoke the now-unused service account token and, if no other consumer depends on it, reassess whether service accounts should remain enabled on the workspace.

Decide recovery, hold or rollback

Resume publishing
The production runner uses the expected identity
GET /api/org and bounded reads return the expected result
The minimum role permits the canary write without unnecessary admin access
The published diff matches the commit and a second pass is idempotent
Unused old tokens are revoked

Keep read-only
The identity or deployed secret version remains ambiguous
The required role is not yet bounded
Some consumers of the old token have not been inventoried

Rollback
Restore the last qualified secret version only if it remains valid and safe
Or temporarily restore the previous identity path
Suspend writes and retain the last qualified dashboard bundle
Never restore an expired token or one suspected of compromise

Conclusion

An Azure Managed Grafana API denial does not automatically call for another secret. A useful diagnosis links the execution, token provenance, audience or expiry, effective role and target workspace.

The decision then becomes controlled: repair distribution when the runner loaded the wrong version, rotate with overlap when the token expired, migrate to Microsoft Entra when the runtime supports it, or remain read-only until identity is proven. Recovery is complete only after an idempotent canary and explicit revocation of access that is no longer required.