Terraform: upgrade a provider without uncontrolled drift

A team needs to upgrade a Terraform provider to fix a defect or support a new resource. The code change looks small: adjust one version constraint, run terraform init -upgrade, commit the regenerated lockfile, and review a green pipeline. The resulting plan, however, also contains recalculated values, newly sensitive attributes, replacements, or changes to resources that were never part of the request.

The provider binary is only one part of the risk. A new version may interpret state differently, evolve a schema, change a default, or call the remote API differently. This runbook treats the upgrade as a production change: freeze intent, reproduce the toolchain, separate existing drift from provider effects, canary the plan, then decide whether to deploy, fix, or roll back.

Freeze the change contract

Start by defining what is allowed to move. “Upgrade the provider” is not an operable change description. Name the provider, affected stacks, sensitive resources, and acceptable differences.

change:
provider: registry.terraform.io/<namespace>/<name>
current_lock: <current-version>
candidate_constraint: <approved-constraint>
terraform_cli: <pinned-version>
canary_stack: platform-nonprod
production_stacks:
  - platform-prod

allowed:
- lockfile checksum and selected provider version
- changes explicitly described in the pull request

block:
- unexplained delete or replacement
- identity, network or encryption change outside scope
- provider alias or subscription mapping change
- state upgrade without a tested return path
- plan that cannot be reproduced in CI

evidence:
- old and candidate lockfiles
- old and candidate plans
- plan JSON classification
- canary apply and runtime checks
- rollback owner and deadline

Retain the commit, lockfile, Terraform version, backend configuration, and execution identity used by the last successful deployment. Without this reference, a plan difference may come from the provider, CLI, CI identity, or manual drift.

Reproduce the toolchain before comparing

A comparison is trustworthy only when one variable changes. First produce a plan from the current code and lockfile in the same CI image used for production. Do not regenerate .terraform.lock.hcl yet.

set -euo pipefail

terraform version
terraform providers
terraform init -input=false
terraform validate

terraform plan -input=false -out=baseline.tfplan

terraform show -json baseline.tfplan > baseline.plan.json

A non-empty baseline must be explained before the upgrade. It represents drift, an existing application change, or a context difference. Absorbing it into the candidate provider plan makes attribution impossible.

Check provider aliases, modules that declare their own constraints, and every platform that executes Terraform. Review the lockfile as a delivery artifact; do not delete it merely to obtain a clean initialization.

Upgrade one dependency

Change only the targeted provider constraint. Run the upgrade on a dedicated branch, then inspect the lockfile diff. The selected version, constraints, and checksums should match the intended change.

set -euo pipefail

terraform init -input=false -upgrade

terraform providers lock -platform=linux_amd64 -platform=linux_arm64

git diff -- .terraform.lock.hcl
terraform validate

Match the platform list to the workstations and runners that actually execute the project. Do not add speculative targets. If multiple providers move when only one was intended, narrow the constraint, isolate the change, or document the coupling before proceeding.

Do not combine a provider upgrade with module updates, a Terraform CLI upgrade, or HCL refactoring. Each extra variable adds another plausible cause for the first unexpected plan.

Compare plans as decisions

Produce the candidate plan with the same backend, variables, and identity as the baseline. The text output supports human review; JSON makes action classification repeatable.

set -euo pipefail

terraform plan -input=false -out=candidate.tfplan

terraform show -json candidate.tfplan > candidate.plan.json

jq -r '
.resource_changes[]
| select(.change.actions != ["no-op"])
| [.address, (.change.actions | join(","))]
| @tsv
' candidate.plan.json > candidate.actions.tsv

Review every create, update, delete, and replacement, then focus on identities, secrets, diagnostics, routes, security rules, stateful stores, and shared dependencies. A value marked known after apply is not automatically harmless. Identify its consumer and the post-apply check that will prove its behavior.

Classify every difference
Expected
  Requested by the change
  Provider effect understood
  Post-apply validation defined

Existing drift
  Already visible in the baseline
  Fixed or accepted separately

Block
  Missing from baseline and outside intent
  Unexplained delete or replace
  Scope, identity or region change
  Unknown value without a post-apply test
  Critical resource with no recovery path

The gate is not “zero differences.” It is “zero unexplained differences.”

Canary against a representative scope

Start with a non-production stack that uses the same modules, aliases, resource types, and policies as production. An empty environment proves that the provider downloads, not that it behaves correctly against the existing estate.

Apply only the saved and reviewed plan. After apply, run a second plan: it should be empty or contain only precisely understood differences. Then validate operational signals for the resources that changed, including DNS resolution, application health, authentication, metrics, logs, alerts, and automated jobs.

before_apply:
- candidate plan approved
- no unexplained replacement
- state backup policy verified
- provider package available to CI
- rollback commit identified

after_apply:
- second plan is empty or explained
- application smoke tests pass
- identities still obtain expected access
- network and DNS probes pass
- logs and metrics remain continuous
- no new provider or API error appears

stop:
- state cannot be read by the previous toolchain
- post-apply plan keeps changing
- production-only resource has no canary equivalent
- runtime validation is ambiguous

Do not treat a workspace as automatic isolation. Use the repository’s established state and environment separation mechanism. A provider upgrade is not the time to introduce a different state strategy.

Roll out in batches and keep rollback honest

Deploy low-impact stacks first, then shared dependencies, and finally the most critical scopes. For each batch, retain the lockfile, approved plan, apply result, second plan, and runtime evidence.

Before any apply, the simple rollback is to restore the previous constraint and lockfile, then prove that the approved toolchain can still read the state. After apply, reverting the binary may not be enough: the provider may have changed the remote resource or evolved state. The team then needs a reviewed inverse plan or the resource-specific recovery runbook.

Approve
Lockfile is limited to the expected provider
Baseline is separated from upgrade effects
No unexplained delete or replace
Canary converges to a stable second plan
Runtime and observability checks are green

Roll back before apply
Restore constraint and lockfile
Reinitialize with the approved toolchain
Reproduce the baseline

Roll back after apply
Stop subsequent batches
Retain state, plans and logs
Qualify remote changes already applied
Build an inverse plan or follow the recovery runbook
Prove service recovery before closure

Conclusion

A Terraform provider upgrade is not validated merely because init, validate, and plan succeed. It is validated when the lockfile contains only the intended change, the baseline attributes every difference, the canary converges to a stable plan, and runtime controls stay green.

The production decision comes down to one rule: deploy only understood, reversible differences. If the provider introduces a replacement, unreadable state, or behavior the team cannot observe, restore the previous toolchain before apply or use an explicit inverse plan afterward.