Azure Arc: diagnose an extension failure after a proxy change

A proxy policy changes overnight. The Azure Arc server still appears online in the portal, but Azure Monitor Agent fails across part of the fleet. Rerunning the deployment changes nothing. Reinstalling the Connected Machine agent recovers one server, yet provides no explanation for the machines that remain stuck.

The mistake is treating “Azure Arc connected” as end-to-end evidence. The control channel, extension manager, package download and extension handler do not necessarily follow the same path. This runbook isolates those layers before anyone opens broad outbound access or starts a fleet-wide reinstall.

Freeze a comparable incident scope

Pick one failed server and one healthy peer with the same site, operating system, agent version, extension and change window. A controlled comparison produces stronger evidence than a fleet-wide export that mixes several failure modes.

Incident: inc-2026-08-11-arc-proxy
Change: new proxy URL and bypass list
Failed server: srv-app-042
Healthy peer: srv-app-017
Extension: AzureMonitorLinuxAgent
Requested and observed versions
UTC time of last success
UTC time of first failure

Evidence required
Arc status and dependent services
Effective proxy, bypass and agent version
Connectivity results for required endpoints
Azure operation and provisioning message
Extension manager log
Handler log and status
Decision: repair, hold, retry or roll back

Do not begin with “remove and recreate” as a diagnostic test. It discards part of the useful timeline and may trigger a fresh download that hides the original fault.

Split Arc connectivity from extension delivery

On the failed server, start with the state observed by the agent itself. azcmagent show reports the connection, dependent services, version and effective proxy configuration. azcmagent check tests required endpoints and reports whether the path uses a proxy, private endpoint or Arc gateway when configured.

sudo azcmagent show
sudo azcmagent config get proxy.url
sudo azcmagent config get proxy.bypass
sudo azcmagent check
sudo azcmagent check --extensions all

# Store outputs with UTC time, server name and agent version.
# Run the exact same commands on the healthy peer.

Read the result by layer:

• disconnected agent or stopped dependent services: recover the Connected Machine agent first;
• healthy Arc control path but failed check --extensions all: qualify proxy, DNS, TLS inspection and extension-specific destinations;
• healthy network tests but failed provisioning: move to extension manager and handler logs;
• one failed handler only: avoid a global proxy or agent change.

Run the test from the affected server. A successful curl from a bastion proves neither the agent’s effective proxy, the process identity nor the certificate chain presented to the handler.

Verify the effective proxy path

When set, the Connected Machine agent’s local proxy.url takes precedence over system environment variables. A rollout can therefore change HTTPS_PROXY without changing the path actually used by the agent. The reverse also matters: some Arc extensions do not inherit agent-specific proxy settings and must be checked against their own network requirements.

Compare four items between the failed server and its healthy peer: effective URL, bypass list, proxy name resolution and the certificate chain after any TLS inspection. Confirm that the proxy allows the expected methods and destinations; generic HTTPS reachability is not sufficient evidence.

Agent disconnected and Arc endpoint blocked
Repair the Connected Machine agent path
Do not change the extension until Arc is stable

Agent connected, extension endpoints blocked
Repair the targeted proxy or firewall policy
Verify extension-specific destinations

Agent connected, network healthy, download rejected
Inspect proxy authentication, TLS inspection and package validation

Package downloaded, handler failed
Diagnose the extension, operating system and prerequisites
Do not widen outbound access without evidence

A broad bypass list is not an acceptable rollback. It may restore service by evading the intended network control. Rollback means returning to a known previous configuration with a time window and an owner.

Read both log layers

The extension manager records download, verification and orchestration failures in gc_ext.log. On Linux, inspect /var/lib/GuestConfig/ext_mgr_logs/. On Windows, use %ProgramData%\GuestConfig\ext_mgr_logs\. Only then move to the logs and status files owned by the failing extension.

sudo tail -n 250 /var/lib/GuestConfig/ext_mgr_logs/gc_ext.log
sudo find /var/lib/GuestConfig -maxdepth 4 -type f -mmin -120 -print
sudo find /var/lib/waagent -maxdepth 5 -type f -mmin -120 -print

# Search around the provisioning operation and timestamp, not only "error".
# Do not copy protected settings or secrets into the incident record.

The boundary tells you what to fix:

• failure before a handler directory exists: download, signature, proxy or orchestration;
• package present but installer returns nonzero: OS prerequisite, noexec mount, package manager or handler script;
• installation completed but health failed: extension configuration or access to its own backend;
• Azure operation absent from local logs: control channel, resource targeting or a concurrent operation.

Keep the correlation identifier, UTC timestamp and exit code. A log excerpt without them is hard to join with Azure Activity Log and provisioning state.

Apply one bounded repair

The first confirmed break determines the fix. Change one variable on the failed canary server: agent proxy configuration, destination rule, certificate chain, local mount or handler prerequisite.

If the agent-specific proxy is the fault, capture its current value first. azcmagent config set proxy.url applies a new URL without a service restart. config clear proxy.url returns control to the host’s configured fallback; do not use it until that fallback is understood.

Then rerun the same extension deployment through the authoritative IaC or automation workflow. A manual install creates a different state from the rest of the fleet and weakens the validation.

Validate, expand or roll back

An extension reporting Succeeded is necessary but not sufficient. Validate its outcome: telemetry arrives, policy is applied, the script completes or the intended security control is active. Observe the canary for a meaningful window, then expand in a small batch.

Keep the repair
Arc remains connected
Endpoint checks use the intended path
The manager downloads and verifies the package
The handler completes without error
The extension outcome is validated
No broad bypass or secret was added

Roll back
Arc connectivity regresses
The proxy is bypassed beyond the approved scope
Other extensions stop working
The repair requires unmanaged TLS trust
The canary drifts from managed configuration

Escalate without reinstalling
The package downloads but the handler fails reproducibly
Logs, versions, exit codes and identifiers are retained
Network and extension service are proven healthy

Rollback restores the previous proxy URL and bypass list, reruns azcmagent check, and confirms that other extensions did not regress. If the extension still fails with a proven healthy network layer, reinstalling it becomes a targeted, documented action rather than the default first response.

Conclusion

An Azure Arc extension failing after a proxy change does not prove that the Connected Machine agent is damaged. Prove the Arc channel, extension network path, package manager and handler separately.

Keep only a repair that restores the extension on a canary without widening bypass rules or breaking other components. Otherwise restore the previous proxy configuration, retain the evidence, and continue from the first unproven layer.