Deployments

How a commit on develop or main becomes a running environment. For the branch flow that gets code onto those branches see Branching & releases; for the release mechanics see Releasing; for getting a bad deploy back out see the Rollback runbook.

One pipeline per environment

Each environment has exactly one deploy pipeline, and it ships the whole platform in one run: backend Cloud Functions, Firestore rules, Medplum bots, the Web Professionals portal, the members app and the clinicians app.

Workflow	File	Trigger	Environment
Deploy - Staging	.github/workflows/deploy-staging.yml	Push to develop, or manual	Staging
Deploy - Production	.github/workflows/deploy-production.yml	Push to main, or manual	Production
Deploy	.github/workflows/deploy.yml	Called by the two above (workflow_call)	Either

The two entry points are thin: they set environment: staging or environment: production, pass secrets: inherit, and hand off to the reusable deploy.yml. Everything the pipeline deploys with, including every project id, hosting target, Medplum client and app URL, is resolved from that one input in the begin job.

One exception, so nobody trusts this further than it goes: the verifier keeps its own copy of the environment URLs in backendBaseUrls and frontendBaseUrls (scripts/deploy/verify-deployment.ts), because it runs as a standalone CLI and has to resolve a URL from --environment without the workflow's outputs. Change an app URL and both places need it. See Adding a component.

Why this shape exists

Production used to have four independent workflows racing on the same push. On 21 and 24 July 2026 the backend deploy failed at the Deploy functions step while both frontend deploys succeeded, and nothing said so: the frontend workflows posted to Slack, the backend ones posted nowhere. Production ran for roughly three days with the frontends two versions ahead of a stale backend. A single gated pipeline makes that particular failure impossible: the frontends cannot ship if the backend did not.

Triggers and concurrency

• Staging runs on every push to develop, with a coarse paths-ignore for docs/**, skills/** and **/*.md. There are deliberately no per-app path filters. The whole train ships from one run, so filtering by component would reintroduce the per-app skew. Concurrency group deploy-staging, cancel in progress, because a superseded staging run is not worth finishing.
• Production runs on every push to main. Concurrency group deploy-production, never cancelled. A cancelled production run can leave the backend deployed and the frontends not, which is exactly the skew the pipeline exists to prevent, so overlapping merges queue instead of interrupting each other.
• Both support workflow_dispatch for a manual re-run.

The shape of a run

flowchart TD
  begin["begin<br/>resolve config, open deployment record"] --> bb["build backend functions"]
  begin --> tbe["test backend functions"]
  begin --> bm["build Medplum bots"]
  begin --> bw["build Web Professionals"]
  begin --> bmem["build members web"]
  begin --> bcli["build clinicians web"]

  bb --> infra["deploy-infra<br/>terraform apply"]
  tbe --> infra
  bm --> infra
  bw --> infra
  bmem --> infra
  bcli --> infra

  infra --> dbe["deploy-backend<br/>rules, functions, bots, Cloud Run config"]
  dbe --> vbe["verify-backend"]

  dbe -->|"failure"| rb["rollback-backend<br/>(matrix: api, bff, bff-clinical, bff-clinical-media)"]
  vbe -->|"failure"| rb
  vbe -->|"success"| dmem["deploy members web"]
  vbe -->|"success"| dcli["deploy clinicians web"]
  vbe -->|"success"| dwp["deploy Web Professionals"]

  dmem --> vfe["verify-frontends"]
  dcli --> vfe
  dwp --> fin

  vfe --> fin["finish<br/>close record, DORA, Slack, release"]
  rb --> fin

The build gate

Every build job (build-backend, build-medplum-bots, build-web-professionals, build-members, build-clinicians) and the test-backend job must succeed before any deploy job starts. deploy-infra depends on all six, and everything else depends on deploy-infra.

That ordering is the point. Under the old workflows a backend that failed to compile could still have left Terraform applied and Firestore rules updated, because those steps ran before the build did. Now nothing touches an environment until all five components have compiled and linted, the bots and the portal have passed their tests, and the backend suite has passed.

test-backend is a separate job rather than a step inside build-backend because the two need incompatible values at the same .env path in the same working directory. pnpm run build runs the OpenAPI generators, which read the runtime configuration, so the build needs the environment's committed config; the suite needs ENV=local (which disables the Pub/Sub auth guards) and NODE_ENV=test. In one job the second would have to overwrite the first, and the hazard is a build, and the generated OpenAPI output that goes with it, produced under test-only configuration.

The .env itself never leaves either job. The artifact is lib only, and deploy-backend rebuilds the file from committed config plus the commit sha, which keeps deploy inputs out of artifact storage.

Each build job uploads its output as a workflow artifact, and the deploy jobs download it rather than rebuilding:

Artifact	Produced by	Consumed by
backend-functions-lib	build-backend	deploy-backend
medplum-bots-dist	build-medplum-bots	deploy-backend
web-professionals-dist	build-web-professionals	deploy-web-professionals
members-web	build-members	deploy-members
clinicians-web	build-clinicians	deploy-clinicians

Artifacts are retained for 30 days on production and 7 on staging. That retention is not housekeeping, it is the frontend rollback mechanism: see Hosting rollback.

The functions env file is rebuilt, not carried

deploy-backend writes .env again rather than pulling it from the artifact. It is derived deterministically from a committed config file plus the commit sha, so rebuilding it keeps deploy inputs out of artifact storage.

Deploy ordering, and why the backend goes first

The order is infrastructure → backend → verify → frontends.

1. deploy-infra runs terraform apply for the environment. Infrastructure that the application depends on has to exist before the application that needs it.
2. deploy-backend deploys Firestore rules, the Cloud Functions, the Datadog source maps, the static-IP egress configuration and Datadog tracing for the Cloud Run services, then the Medplum bots.
3. verify-backend probes the deployed backend and fails the run if it is not actually serving this commit.
4. deploy-members, deploy-clinicians, deploy-web-professionals each list verify-backend in their needs, so they only start once the backend is confirmed good.

Frontends wait on a verified backend rather than a merely deployed one because of the direction of the dependency. The apps call the backend; the backend does not call the apps. A new frontend against an old backend is the dangerous combination: it will request endpoints and response shapes that do not exist yet. An old frontend against a new backend is the ordinary state of affairs for the few minutes a deploy takes, and the backend is expected to tolerate it. So the safe sequencing is backend first, proven, then frontends.

The corollary is worth stating plainly: if the backend fails to verify, the frontends do not deploy at all. Their jobs are skipped, the environment keeps serving the previous frontend build, and the run is marked failed.

Verification

verify-backend and verify-frontends both run pnpm run deploy:verify (scripts/deploy/verify-deployment.ts). It probes with exponential backoff (8 attempts, 5s initial delay, 30s ceiling, 5 minute total budget) because a fresh deploy takes a little while to start serving.

The budget covers the whole run, not each service. Everything a run probes was deployed together and becomes ready at roughly the same time, so a per-service budget would multiply the worst case by the number of services without making a healthy deploy any likelier to pass. It would also let verification outlive the job timeout, and a timed-out job reports cancelled rather than failure, which is the one result that would leave a bad deploy live with no rollback.

Backend. For each of api, bff, bff-clinical and bff-clinical-media it fetches https://europe-west2-<project>.cloudfunctions.net/<service>/_health and requires status: "ok" and version equal to the commit sha being deployed. The version comparison is what catches the real failure mode: a deploy that reported success but left the previous build serving. Hyphens in a service name become underscores in the function path, so bff-clinical is probed at /bff_clinical/_health.

Frontends. For members and clinicians it fetches <app-url>/version.json and compares both the app version and the commit. The two apps are probed separately because the two pubspecs are versioned independently, so a single expected version would be wrong the moment they diverge. 1.1.28+1 and 1.1.28-1 are treated as equal: the pipeline carries the version with a hyphen because + is not safe everywhere it is passed through.

The expectation flags are mandatory, not optional: --expected-version for both targets and --expected-commit for frontends. deploy:verify refuses to start without them, because a probe with nothing to compare against only proves that something answered on the URL, which is precisely the signal that let a failed backend deploy sit behind two healthy frontends for three days. Leaving them optional would make that degradation a silent consequence of a future edit to this workflow.

The commit is the load-bearing half. The pubspec version is unchanged across most pushes to develop, so on its own it would pass whether or not this run's build ever reached the CDN. Each build job stamps commit_sha into build/web/version.json after flutter build web, and the probe requires it to equal the commit being deployed.

Two things are deployed but not probed

The Web Professionals portal and the Medplum bots have no verification step. A silent failure in either is still possible and will not be caught by the pipeline. Adding a probe for the portal is the obvious next increment.

Because the portal is not probed, verify-frontends does not wait for it. A failed portal deploy still fails the run, but it no longer skips verification of the two apps that did deploy. Each app's probe is gated on its own deploy for the same reason.

One deployment record, one release

One deployment record per run. begin opens a single GitHub deployment via the Deployments API and marks it in_progress; finish closes it success or failure. The pipeline deliberately does not use the environment: key on individual jobs, because GitHub opens one deployment per job that declares one, and that per-app noise is what this pipeline exists to remove. The trade-off is that individual jobs lose the native environment badge in the Actions UI. In exchange there is exactly one accurate record per run, including when the run fails.

The six old per-app environments (backend-staging, members-staging, clinicians-staging and their production counterparts) are gone. They held no environment-scoped secrets, no variables and no protection rules, so consolidating them changed no gating: every Actions secret in this repo is repo-level.

One release per production run. finish builds the notes with pnpm run deploy:release-notes (scripts/deploy/release-notes.ts) and cuts a single GitHub release covering every component in the train, replacing the three per-component releases the old workflows produced. If the tag already exists, as on a re-run of a production deploy, the existing release is left untouched rather than failing the run.

Slack, routed by environment. finish carries the app versions, commit, branch, actor, release status and a link to the run.

Environment	Channel	Posts on
production	#product-team	every run, success or failure
staging	#platform-staging	failures only

Production reports everything, because a silent production deploy is the gap that let the July incident go unnoticed for three days. Staging reports only failures: a successful staging deploy happens several times a day and says nothing anyone needs to read, and because staging queues rather than cancelling, a cancelled run means somebody stopped it deliberately and already knows.

A Slack app webhook is bound to a single channel and ignores SLACK_CHANNEL, so the two environments use two secrets: SLACK_WEBHOOK_PRODUCT_TEAM and SLACK_WEBHOOK_PLATFORM_STAGING. #platform-staging is a private channel, so the Slack app that owns its webhook has to be a member of the channel or the post is rejected.

DORA events, production only. On a successful production run, finish emits Datadog DORA deployment events for perci-platform-backend, members-app-frontend and clinicians-app-frontend. Staging is deliberately excluded: DORA measures delivery to users, so counting staging would inflate deployment frequency with events no customer saw and put lead time and change failure rate against deploys that were never deliveries. Emission is best-effort and can never fail a deploy. See Releasing.

DORA needs its Datadog source set to API

Datadog rejects these events while a service's DORA deployment source is set to anything else, with the configured deployment source was not API. The script warns and exits 0, so the deploy is unaffected, but no events are recorded until that setting is changed per service in Datadog.

What happens on failure

finish runs with if: always() and depends on every other job, so the record is closed, the Slack message is posted and the outcome is reported no matter where the run died. It resolves the outcome from the results of all its dependencies into one of three states:

Any dependency reports	Reported state	Slack
failure	failure	red, "FAILED"
cancelled and no failure	cancelled	amber, "was cancelled, usually because a later merge superseded it"
neither	success	green

(rollback-backend reports skipped on a healthy run, which counts as neither.)

A cancelled staging run is not a failure

A run can end cancelled because someone stopped it or because a job hit its timeout. Reporting that as a failure meant Slack announced deploys that nothing was wrong with, and an alert channel that cries wolf is the problem this pipeline exists to fix, so cancellation is reported as cancellation.

Both environments queue rather than cancel in progress. Staging originally cancelled superseded runs, and on the first real staging deploy that cancelled a run 6.5 minutes into firebase deploy --only functions, leaving some functions updated and some not. A pipeline built to make partial deploys impossible must not create them by superseding itself. The cost is that a burst of merges to develop queues rather than skipping to the newest commit.

rollback-backend still treats cancelled as worth rolling back. That is a safety question rather than a reporting one: a run stopped part way through the backend deploy can leave the new functions live and unverified either way.

Where it fails	What has changed in the environment	What the pipeline does
Any build job	Nothing	Every deploy job is skipped. Nothing is applied.
deploy-infra	Terraform may be partially applied	Backend and frontends skipped. See Terraform.
deploy-backend	Rules, functions or bots partially deployed	Verify and all frontends skipped. rollback-backend runs, and shifts traffic for any service this run had already changed. Manual assessment still needed for rules, bots and background functions.
verify-backend	Backend deployed but not serving the new build	rollback-backend shifts Cloud Run traffic back on all four services. Firestore rules, bots and Terraform stay on the new build. Frontends never deploy.
A frontend deploy	That app not updated; the others may be	verify-frontends skipped. The backend stays on the new build.
verify-frontends	Everything deployed, one app not serving the expected version	No automatic action. Follow the rollback runbook.

firebase deploy --only functions is not atomic

Firebase deploys functions one at a time. A failure halfway through leaves some functions on the new build and some on the old, and there is no transaction to unwind. The pipeline cannot prevent this. Verification plus rollback is mitigation, not prevention: it shortens how long a half-deployed backend serves traffic, it does not stop one happening. Treat a deploy-backend failure as a mixed-version backend until you have checked.

How to read a failed run

1. Start from the Slack message. It carries the run URL, the commit, the branch and the actor. If there is no Slack message at all, the finish job itself did not run, which means the run was cancelled at the workflow level.
2. Look at the job graph, not the logs, first. Which jobs are red and which are grey tells you what reached the environment. Grey means skipped, which means untouched. Red is where the truth is.
3. Map the first red job to the table above. The first failure is the interesting one; everything downstream of it is a consequence.
4. Check whether rollback-backend ran. It appears when either deploy-backend or verify-backend failed. Green means HTTP traffic is back on the last revision from before this run, for every service the run had touched; it logs Nothing to roll back for any service it never reached. Red for a service means that service needs a manual rollback. Either way it covers HTTP traffic only, so check Firestore rules, the Medplum bots and the background functions against the restored revision before you call it resolved.
5. Read the verify output for the actual mismatch. A verification failure logs the URL, the version it found and the version it wanted, on every attempt. is serving version X, expected Y is a deploy that silently did nothing. HTTP 5xx is a deploy that shipped something broken. They need different responses.
6. Check the GitHub deployment record. One entry per run under the repo's Environments view, with the state and a link back to the run. If the latest production record is a failure, production is not on the commit you think it is.
7. Fix forward or roll back. Staging: usually fix forward, re-push, the next run supersedes this one. Production: decide first, using the rollback runbook; a fix-forward still has to travel through develop, a release branch and a merge into main before it deploys.

Adding a component

Three lists have to be updated together when a new Cloud Run service or Firebase function appears, and none of them can be derived automatically:

• The static IP egress loop in deploy-backend, but only if the new service calls a third party that allowlists our static egress IP. This list is intentionally shorter than the set of deployed services: routing a service through the egress subnet changes its outbound path, so adding one is a networking decision to take deliberately. A service that needs the static IP and is missing from the loop egresses from an ephemeral Google pool IP instead, and the third party rejects it.
• The Datadog Cloud Run instrumentation --service list in deploy-backend.
• The verification --service list, and knownBackendServices in scripts/deploy/verify-deployment.ts, plus the rollback-backend matrix.

An environment URL is the fourth. Changing where an app or the backend is served means updating both the begin job's environment block in deploy.yml and backendBaseUrls / frontendBaseUrls in scripts/deploy/verify-deployment.ts. Miss the verifier and it probes the old host, which fails the deploy rather than passing it wrongly, but the failure names a URL nobody changed and reads as a mystery.

New functions get /_health for free by mounting createHealthRouter from functions/src/common/health/healthRouter.ts, which is what verification probes.