Azure Key Vault in a private network: organize secret rotation without blind spots

Putting Azure Key Vault behind a Private Endpoint reduces network exposure, but it does not solve operations by itself. Secrets still need to be named, versioned, consumed by applications, renewed, validated, and retired cleanly. Without a method, a private vault can become a safe place to store secrets that nobody dares to replace.

The scenario here is a business application hosted in Azure and consuming several secrets: a connection string, an internal API key, a certificate, or a technical password. Key Vault is reachable only from the private network, applications use managed identities, and the team wants to rotate a secret without cutting production.

Separate network, identity, and operations

A Private Endpoint answers one question: through which network path is the vault reachable? It does not answer the permissions question. A private Key Vault remains fragile if too many identities can read every secret. Conversely, well-limited rights are not enough if the vault remains publicly reachable without reason or if private DNS is inconsistent.

Network controls
Private Endpoint to the application VNet
Public network access limited or disabled depending on the model
Private DNS zone privatelink.vaultcore.azure.net

Identity controls
Managed identity per application or workload
Access limited to the secrets actually needed
Separate read, write, and administration

Operational controls
Naming convention
Documented rotation
Application validation
Access logging

The important point is not to confuse these layers. A DNS error can prevent the application from reaching the vault. A wrong role can produce a 403. An obsolete secret version can break the application even if network and permissions are correct. The runbook must distinguish these causes without reopening the vault by reflex.

Stabilize DNS before talking about rotation

With a Private Endpoint, DNS resolution becomes a production prerequisite. From the application network, the public vault name must resolve to the private IP of the Private Endpoint. If a diagnostic VM, pod, or build agent still resolves to a public address, rotation is not the main issue: the network path is inconsistent.

VAULT_NAME=kv-app-prod-001
nslookup $VAULT_NAME.vault.azure.net

az network private-endpoint-connection list --name $VAULT_NAME --resource-group rg-security-prod --type Microsoft.KeyVault/vaults --query '[].{name:name,status:privateLinkServiceConnectionState.status}' -o table

The DNS test must be executed from a representative source: an administration VM in the hub, an application pod, a private build agent, or the instance that actually consumes the vault. A local workstation connected through VPN is not always enough to validate the application path.

Give each application its own read identity

Clean rotation starts with a readable access surface. If several applications share the same identity or generic secret, nobody knows what a change will affect. The managed identity should reflect the workload, not the entire environment.

Identity app-order-api-prod
Read: sql-order-connection-string
Read: partner-billing-api-key

Identity app-backoffice-prod
Read: sql-backoffice-connection-string
Read: smtp-relay-password

Identity pipeline-secret-rotation
Controlled write on target secrets
No unnecessary broad read
No vault administration rights

This separation makes incidents simpler. If an application receives a 403, you know which identity to check. If a secret is read abnormally, you know which workload is involved. If rotation fails, analysis can be limited to the application-secret pair.

Use secret versions as a transition

Key Vault versions secrets. This capability helps deploy a rotation without abruptly replacing the active value application-side. The trap is hard-coding a precise version in application configuration. In that case, publishing a new version in Key Vault changes nothing: the application keeps asking for the old one.

Recommended stable reference
https://kv-app-prod-001.vault.azure.net/secrets/sql-order-connection-string

Pinned reference to use only when intentional
https://kv-app-prod-001.vault.azure.net/secrets/sql-order-connection-string/<version>

Transition strategy
Publish a new version
Validate that the application rereads the stable reference
Restart or reload configuration if necessary
Monitor application errors and Key Vault reads
Disable the old version after validation

The application behavior must be known before rotation. Some applications read the secret at startup. Others cache it. Others query Key Vault regularly. The reload mode determines the risk window and the way to prove that the new value is actually consumed.

Prepare a two-step runbook

Reliable rotation avoids an instant change with no return path. It first publishes the new value, verifies that the consumer can use it, then retires the old value when observation is sufficient. For a secret shared with an external service, coordination is even more important: sometimes two values must be accepted for a short period.

Before rotation
Identify consuming applications
Verify the identity used by each application
Verify private DNS and Key Vault access
Confirm the secret reload mode

During rotation
Create a new secret version
Deploy or reload the application if necessary
Test the minimal business flow
Monitor 401, 403, 500 errors and timeouts

After rotation
Keep the old version for the agreed window
Disable the old version
Document the active version and cutover time
Verify that no read targets the old version

This split makes rotation observable. The goal is not only to change a value, but to prove that the new value is available, consumed, and compatible with the target service.

Diagnose errors without mixing everything

During rotation, several errors look similar from the application side. An authentication error against the target service does not have the same cause as a Key Vault access denial. First verify whether the application reaches the vault, then whether it can read the secret, then whether the value it reads works against the target service.

VAULT_NAME=kv-app-prod-001
SECRET_NAME=sql-order-connection-string

az keyvault secret show --vault-name $VAULT_NAME --name $SECRET_NAME --query '{name:name,enabled:attributes.enabled,created:attributes.created,updated:attributes.updated}' -o json

az keyvault secret list-versions --vault-name $VAULT_NAME --name $SECRET_NAME --query '[].{id:id,enabled:attributes.enabled,updated:attributes.updated}' -o table

These commands validate the secret state, but they do not prove that the application has the same rights or uses the same network route. For that, correlate application logs, Key Vault logs, and, if needed, a test executed from the same environment as the workload.

Monitor reads and denials

Key Vault logs are useful during rotation. They show whether the application still reads the old version, whether an identity receives denials, or whether an error spike appears at the time of change. The important part is logging what helps decide, not producing unreadable volume.

AzureDiagnostics
| where ResourceProvider == "MICROSOFT.KEYVAULT"
| where OperationName in ("SecretGet", "SecretList")
| where TimeGenerated > ago(24h)
| project TimeGenerated, Resource, OperationName, ResultType, ResultSignature, identity_claim_appid_g, CallerIPAddress, requestUri_s
| order by TimeGenerated desc

During the rotation window, this query helps answer three simple questions: which identity reads the secret, with what result, and which version appears to be targeted. If the full URI is not retained or logging is filtered, adapt the observation, but keep the same principle.

Remove secrets when an identity is enough

The best rotation is sometimes the one you no longer need. When an Azure service supports managed identity or federated authentication, avoid storing an application secret. Key Vault remains useful for certificates, some external secrets, or cases where the integration does not yet support identity, but it should not become the reflex for everything.

Use a managed identity if
The target service supports it
The workload runs in a compatible environment
Permissions can be expressed cleanly with RBAC

Use Key Vault for a secret if
The external service requires a key or password
A certificate must be stored and distributed
A migration phase still requires a secret value

Review regularly
Is the secret still necessary?
Are consumers still active?
Are old versions disabled?
Are accesses still justified?

This review avoids accumulating historical secrets. A private vault filled with unused values remains an operational burden.

Conclusion

Azure Key Vault in a private network provides a solid base, provided rotation is treated as a complete operational process. The Private Endpoint secures the path. Managed identities limit access. Private DNS makes the path predictable. Secret versions enable a controlled transition. Logs provide proof that the new value is consumed.

The discipline is to never reduce rotation to a set secret command. You need to know who consumes what, through which path, with which identity, when the application reloads the value, and how the old version will be retired. That is what turns Key Vault from a simple vault into a reliable production component.