GitHub Actions: prevent overlapping production deployments

Two commits are merged a few minutes apart, starting two GitHub Actions workflows against the same production service. The second run deploys the newest version. The slower first run then completes and puts the older version back online. Both pipelines are green, yet production has moved backward without any job labeled “rollback.”

This runbook treats deployment as a concurrent write to shared state. The goal is to prove which execution changed what, stabilize production, and choose an appropriate serialization strategy. The exit decision must be explicit: keep the active version, redeploy the intended commit, or roll back to a known artifact.

Reconstruct the timeline before rerunning

Do not immediately rerun the latest workflow. Another execution can join the same race and erase useful evidence. For each run, record the commit, artifact, environment, start and completion times, deployment identity, and the version observed after the job.

target: payments-api-production
expected_commit: <commit-sha>
observed_version: <version-from-runtime>

runs:
- run_id: <older-run-id>
  commit: <older-commit-sha>
  artifact_digest: <digest>
  started_at: <timestamp>
  deployment_started_at: <timestamp>
  completed_at: <timestamp>
  conclusion: success
- run_id: <newer-run-id>
  commit: <newer-commit-sha>
  artifact_digest: <digest>
  started_at: <timestamp>
  deployment_started_at: <timestamp>
  completed_at: <timestamp>
  conclusion: success

evidence:
- github_deployment_history
- cloud_activity_logs
- runtime_version_endpoint
- immutable_artifact_digest

Workflow completion time is not enough. A job may prepare an artifact long before touching the target, or start an asynchronous rollout that outlives the GitHub run. Find the point at which production state actually changed.

Identify the resource that needs a lock

The lock does not necessarily belong to the workflow. It belongs to the resource being changed: an environment, slot, cluster, namespace, resource group, or tenant. Two different workflows that write to the same service must share the same concurrency group.

Same lock required
Two branches deploy payments-api to production
Application and infrastructure workflows modify the same slot
Manual and automatic deployments target the same environment

Separate locks may be valid
Independent production environments
Services with no shared resource, database or route
Staging and production when no common step is modified

Inspect carefully
Reusable jobs called by multiple workflows
Deployments started by workflow_run or repository_dispatch
Scripts that default to a production environment
Matrix jobs that converge on the same target

A group that is too broad needlessly blocks independent services. A group that is too narrow leaves the collision in place. Do not rely on the branch name when several branches can reach the same production target.

Serialize writes with concurrency

Define concurrency on the job that writes to the target. Build and test jobs may remain parallel; deployments to the same production environment must be serialized. For a production change involving migrations, traffic switches, or non-transactional mutations, avoid canceling a running job.

name: deploy-production

on:
workflow_dispatch:
push:
  branches: [main]

jobs:
build-and-test:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@<pinned-version>
    - run: ./ci/build-and-test.sh

deploy:
  needs: build-and-test
  runs-on: ubuntu-latest
  environment: production
  concurrency:
    group: payments-api-production
    cancel-in-progress: false
    queue: single
  steps:
    - uses: actions/checkout@<pinned-version>
    - run: ./ci/deploy.sh
    - run: ./ci/verify-production.sh

With queue: single, the group limits concurrent execution and retains at most one pending run; a newer run replaces the older pending run. This strategy fits a service that should converge on the latest approved state. Use queue: max when every change must run in order, such as a sequence of migrations that cannot be compacted. The queue may then accumulate up to its limit, so each run must revalidate its base before writing.

cancel-in-progress: true is better suited to idempotent computations or disposable previews. In production, canceling halfway through a migration, rotation, or traffic transfer can leave partial state. If cancellation is enabled, the deployment script must handle termination and inspect real state before any retry.

Add a guard close to the target

GitHub concurrency reduces collisions, but it does not cover a deployment started outside GitHub or an asynchronous write that outlives the job. Add a second barrier: a protected environment, a lease in the deployment engine, a precondition on the active version, or a platform-level lock.

set -euo pipefail

EXPECTED_BASE="<version-that-was-validated>"
ACTIVE_VERSION="$(./ops/read-active-version.sh)"

if [ "$ACTIVE_VERSION" != "$EXPECTED_BASE" ]; then
echo "Production changed after this run was prepared."
echo "Expected base: $EXPECTED_BASE"
echo "Active version: $ACTIVE_VERSION"
exit 42
fi

./ops/acquire-deployment-lease.sh payments-api-production
./ops/deploy-immutable-artifact.sh <artifact-digest>

This precondition acts as an optimistic concurrency check: the run refuses to write when its validated base is no longer current. It is particularly useful when an approval keeps one job waiting while another change reaches production.

Make every deployment attributable

A latest tag or workflow-local build number is not enough. The artifact must be immutable and linked to the commit, run, and target. Production should expose at least one version that a probe or inventory can read.

{
"environment": "production",
"service": "payments-api",
"commit": "<commit-sha>",
"workflowRunId": "<run-id>",
"artifactDigest": "sha256:<digest>",
"deployedAt": "<timestamp>",
"previousVersion": "<version>",
"validation": {
  "health": "passed",
  "smokeTest": "passed",
  "errorBudgetSignal": "stable"
}
}

Retain this evidence in deployment history and platform logs. A success workflow conclusion should mean that the expected version is active and validated, not merely that the deployment command was accepted.

Decide what to do during the incident

After stopping new writes, compare the active version with the intended commit. Cancel obsolete pending runs. Rerun only an immutable artifact that has already been built; rebuilding the same commit during an incident adds another variable.

Keep the active version
It matches the newest approved commit
Probes and application signals are stable
No older deployment can still write

Redeploy the intended commit
Production runs an older or unknown artifact
The expected digest is available and verified
The target is locked and obsolete runs are canceled

Roll back
The intended newest commit fails after deployment
The previous artifact is known and data-compatible
Post-rollback validation is ready

Block resumption
The active version cannot be attributed
An asynchronous deployment may still complete
An irreversible migration is in progress
The lock does not cover every write path

A rollback must not be a blind rerun of an old workflow. It is a new controlled change to an identified artifact, with the same concurrency barriers and validation.

Validate the fix without recreating the race

Test the rule with three short runs against a validation target: one active, one pending, then a newer third run. Verify which run remains pending, that no writes overlap, and that logs make the decision understandable. Also test a manual invocation and every reusable workflow that reaches the target.

In production, observe the first serialized deployment through application validation. Confirm the commit, digest, active version, errors, and the behavior of routes or migrations. Keep a way to freeze new triggers without deleting workflow history.

Conclusion

Overlapping pipelines are a concurrency problem, not merely a CI defect. The durable fix is to lock the shared resource, keep builds parallel, serialize writes, reject stale bases, and prove the active version after every deployment.

The final decision remains operational: keep the online version, redeploy the intended artifact, or roll back to the previous one. In every case, an older run must be unable to take control again after validation.