Rollback runbook¶

Getting a bad deploy back out. This page is the exact steps; for how the pipeline is shaped and what a failed run means, see Deployments.

Read this first

Nothing here is a transaction. A Perci production deploy touches Terraform, Firestore rules, a set of Cloud Functions, Medplum bots and three hosting targets, and each of those is rolled back by a different mechanism with a different blast radius. Rolling back one component while leaving the others on the new build is a version skew of its own. Decide what you are restoring before you start typing.

Decide: roll back or fix forward¶

Situation	Do this
`verify-backend` failed and `rollback-backend` is green	HTTP traffic is already back on the last pre-deploy revision. Not finished: the Firestore rules, Medplum bots and anything Terraform applied are still on the new build, so check those against the restored revision. Then investigate and fix forward.
`deploy-backend` failed partway through	Assume a mixed-version backend. `rollback-backend` has shifted traffic for whatever the run had already changed and left the rest alone; probe every service before deciding anything else.
Backend is up but serving errors	Roll back Cloud Run traffic (below), then fix forward.
A frontend is broken but the backend is healthy	Roll back that hosting target only. Frontends do not depend on each other.
The backend is stale and the frontends are new	Do not roll the frontends back to match. Get the backend forward, or roll the frontends back to the version that matched the live backend. Check which the live backend actually is first.
Terraform apply broke something	Re-apply the previous ref (below). There is no undo.
The bug is small, understood, and low-risk	Fix forward. A rollback of a partially applied deploy is often riskier than a one-line fix.

Announce in #product-team before you start and when you finish. The deploy pipeline posts there on every run, so that channel is where the timeline is already being written.

Cloud Run rollback (backend services)¶

The four backend Cloud Run services are api, bff, bff-clinical and bff-clinical-media. A rollback shifts 100% of a service's traffic to its previous ready revision. It is the fastest and safest lever we have, because Cloud Run keeps the old revisions around and the shift is near-instant.

The pipeline does this automatically when either deploy-backend or verify-backend fails: the rollback-backend matrix job runs the same script across all four services. Run it by hand when you need it outside a failing pipeline run.

# authenticate against the right project first
gcloud auth login
gcloud config set project perci-platform-prod

# see what it would do, without doing it
pnpm run deploy:rollback -- --service api --project perci-platform-prod --dry-run

# do it
pnpm run deploy:rollback -- --service api --project perci-platform-prod

Repeat per service. Options: --region defaults to europe-west2, --current-revision overrides the revision the script resolves as currently serving, and --created-before takes unix seconds, unix milliseconds or an ISO 8601 timestamp.

The script only ever rolls backwards: candidates are restricted to revisions created before the one currently serving. That matters most when a previous rollback already pinned traffic to an older revision, because the newest non-serving revision is then newer than what is serving, and choosing it would undo the earlier rollback and restore the build it moved away from.

Split traffic stops the script

If more than one revision is serving traffic, the script refuses to act and tells you to pass --current-revision. It will not fall back to the revision with the largest share, because that is the case most likely to mislead: with the pre-deploy revision on 90% and the revision you are trying to remove on 10%, "largest share" names a revision that predates the deploy, the script reports Nothing to roll back, and the bad revision keeps serving a tenth of traffic. Check gcloud run services describe <service> --region europe-west2 before deciding which revision to name.

Pass --created-before whenever you are undoing a specific deploy, set to the moment that deploy started. One deploy-backend run mutates each service two or three times (the functions deploy, the VPC egress reapply and the Datadog instrumentation each mint a revision), so without a cutoff the "previous" revision is usually another revision of the same build you are trying to get rid of. With the cutoff the script skips every revision the deploy produced, and reports Nothing to roll back for a service whose serving revision already predates it. The pipeline passes the deploy's own start time.

pnpm run deploy:rollback -- --service api --project perci-platform-prod \
  --created-before 2026-07-24T11:00:00Z

The script (scripts/deploy/rollback-cloudrun.ts) refuses to guess. If there is no other ready revision, if every remaining candidate came from the deploy being undone, or if two candidates share a creation timestamp so the "previous" one is ambiguous, it errors out rather than picking. A rollback that lands on the wrong revision is worse than one that does not happen, so when it refuses, resolve the revision by hand:

gcloud run revisions list --service api \
  --project perci-platform-prod --region europe-west2

gcloud run services update-traffic api --to-revisions <revision>=100 \
  --project perci-platform-prod --region europe-west2

What a Cloud Run rollback does not cover

Only the HTTP services. Firestore security rules, the Medplum bots, the background and scheduled functions, and anything Terraform applied are all untouched by a traffic shift.

Hosting rollback: redeploy the retained artifact¶

The three Firebase Hosting targets are the members app, the clinicians app and the Web Professionals portal. Roll one back by redeploying the build artifact from the last good pipeline run.

Every build job uploads its output as a workflow artifact, retained for 30 days on production. That retention exists for exactly this: it means a rollback is a redeploy of a byte-identical known-good build, not a rebuild of an older commit that may no longer produce the same output.

Find the last good run. Actions → Deploy - Production → the most recent run that finished green, before the bad one.
Download the artifact from that run's summary page: members-web, clinicians-web or web-professionals-dist.
Unpack it into the app's build directory in a local checkout.
Deploy it.

gcloud auth login

# members
cd apps/perci-platform-members
# unpack members-web into build/web first
firebase deploy --only hosting:perci-platform-prod --project perci-platform-prod

# clinicians
cd apps/perci-platform-clinicians
# unpack clinicians-web into build/web first
firebase deploy --only hosting:perci-platform-clinical-prod --project perci-platform-prod

Staging uses the targets perci-platform-staging and perci-platform-clinical-staging against perci-platform-staging.

For Web Professionals, unpack web-professionals-dist into apps/perci-platform-backend/web-professionals/dist, then from apps/perci-platform-backend:

firebase target:apply hosting web-professionals perci-platform-professionals-prod \
  --project perci-platform-prod
firebase deploy --only hosting:web-professionals --project perci-platform-prod

Staging uses the site perci-platform-professionals-staging against perci-platform-staging.

Always pass --project

Without it the Firebase CLI falls back to whatever project your local checkout last selected, which during an incident is whatever you were debugging a moment ago. That applies to target:apply as much as to deploy: the target mapping is written per project, so an omitted --project can bind the production site under the staging project's target map and send the next deploy to the wrong environment.

Finally, purge the CDN so users are not served the bad build from cache:

curl -f -X PURGE https://app.perci.health/ -w 'HTTP Status: %{http_code}\n'

Firebase console rollback is faster, and less traceable

Firebase Hosting keeps a release history per site, and the console offers a one-click rollback to a previous release. In a live incident that is the quickest lever, and it is fine to pull it. Follow up by redeploying the artifact so the deployed state matches something in the pipeline's record rather than only in the console's.

Re-running the workflow will not roll you back

workflow_dispatch on Deploy - Production runs against main, so a manual re-run deploys the same bad commit again. Rolling forward through the pipeline means getting a fix onto main through a hotfix/* branch. See Releasing. Redeploying the artifact is the only way to put an older build back without moving main.

Terraform is not rollbackable¶

terraform apply has no undo. There is no saved plan to reverse and no "previous state" to restore, and a partially applied plan has already made some of its changes.

Recovery is a re-apply of the previous ref: check out the last commit whose infrastructure was known good, and apply that.

git checkout <last-good-sha>
cd infrastructure/terraform/environments/production
terraform init
terraform plan     # read this carefully, every line
terraform apply

Read the plan before applying it. Re-applying an older configuration is itself a change, and for destructive resources (anything Terraform proposes to destroy and recreate) it can be a larger change than the one you are undoing. If the plan proposes destroying a database, a bucket or a DNS record, stop and get a second pair of eyes.

Where a resource cannot be safely re-applied, fix forward with a new commit that describes the state you want, and let the pipeline apply it.

firebase deploy --only functions is non-atomic

Firebase deploys functions one at a time. A failure partway through leaves some functions on the new build and some on the old, with no transaction to unwind. Verification plus rollback is mitigation, not prevention: it shortens the window in which a half-deployed backend serves traffic; it does not stop one occurring. After any deploy-backend failure, assume the backend is mixed-version until you have confirmed otherwise:

for s in api bff bff_clinical bff_clinical_media; do
  curl -s "https://europe-west2-perci-platform-prod.cloudfunctions.net/$s/_health" \
    | jq -c '{service, version}'
done

Every service should report the same version (the deployed commit sha). A mismatch is a split backend.

Other components¶

Firestore security rules. Redeploy from a checkout of the last good commit. Rules are replaced wholesale, so this is a genuine restore:

git checkout <last-good-sha>
cd apps/perci-platform-backend
firebase deploy --only firestore:rules --project perci-platform-prod

Medplum bots. Redeploy from the last good commit. The bot pattern is *prod for production and *staging for staging:

git checkout <last-good-sha>
cd apps/perci-platform-backend/medplum/bots
pnpm install --frozen-lockfile
pnpm run build
npx medplum bot deploy '*prod'

The client id and base URL come from the environment configuration in deploy.yml, and the client secret from Secret Manager (MEDPLUM_CLIENT_SECRET in the target project).

After a rollback¶

Verify what is actually serving. Do not trust the deploy output; probe it. The same script the pipeline uses works locally:

pnpm run deploy:verify -- --target backend --environment production \
  --service api --service bff --service bff-clinical --service bff-clinical-media \
  --expected-version <sha>

pnpm run deploy:verify -- --target frontend --environment production \
  --app members --expected-version <version> --expected-commit <sha>

Say so in #product-team, including which components you rolled back and which you did not. A partial rollback that nobody recorded is the next incident.
Record the deployed state. The GitHub deployment record for the failed run stays failure, and a manual rollback does not create a new one, so the record will not tell the next person what is live. Write it in the incident thread.
Raise the Jira ticket and, for anything user-visible, an RCA. A hotfix merge into main is also what feeds the DORA change-failure signal, so route the fix through a hotfix/* branch rather than a direct push. See Releasing.