Datadog error triage¶
Error Tracking is only useful if env:prod means "real users in production". This page records
which environments are allowed to report, and where each gate lives.
Environment-to-reporting matrix¶
| Environment | Backend (functions) |
Members / clinicians apps | Web Professionals | Reported env tag |
|---|---|---|---|---|
Developer machine (flutter run, pnpm run dev, emulator suite) |
No | No | No | — |
| CI (unit tests, Patrol, preview builds under test) | No | No | No | — |
| Preview (per-PR deploy) | Yes | Yes | Yes | preview-pr-<run id> |
| Staging | Yes | Yes | Yes | staging |
| Staging (semantics build) | n/a | Yes | n/a | staging-semantics |
| Production | Yes | Yes | Yes | prod |
staging-semantics is the members app built from config/semantics/environment.json, which is why
env:staging-semantics events are expected rather than a sign of mis-tagging.
Medplum bots do not initialise a Datadog client at all, so they have nothing to gate.
How each gate works¶
Every gate is a positive allowlist and fails closed: an environment that cannot be identified as a deployed one reports nothing. Adding an environment to an allowlist means editing code and shipping it — no configuration value can introduce a new reporting environment on its own.
The environment name still comes from configuration (ENV / DD_ENV for the backend, EnvString
for the apps, VITE_DD_ENV for Web Professionals), so writing an allowlisted name into a local
config is not physically impossible. What matters is that no committed default does it: every
checked-in local config carries a non-reporting value, and the backend additionally demands a
GCP-injected FUNCTION_TARGET/K_SERVICE that a .env cannot fake, so nothing reports by
accident and there is no debug flag to leave switched on.
Backend — functions/src/utils/datadogReporting.ts. functions/src/index.ts only starts the
tracer, via initDatadogTracing in functions/src/observability/datadogTracing.ts, when all of
these hold: not running under the emulator, NODE_ENV=production,
FUNCTION_TARGET or K_SERVICE present (only GCP sets these, so it proves the process is
deployed), ENV is one of prod, dev or preview, and both DD_ENV and DD_VERSION are
non-blank. ENV=dev is the staging deploy, which reports as DD_ENV=staging. The GCP marker is
what makes the gate safe: a local .env is seeded from config/env/functions.staging.env, so a
developer machine legitimately holds ENV=dev and NODE_ENV=production.
A deployed run missing DD_ENV or DD_VERSION reports nothing and logs a warning at startup.
Untagged telemetry is worse than none — it lands in Error Tracking with no environment or version,
so it can neither be filtered out nor attributed to a release.
Members and clinicians apps — packages/perci-platform-frontend-shared/lib/core/observability/datadog_reporting.dart.
Each app's main calls initializeDatadogConfig only when the build is a release build and
EnvString is prod, staging, staging-semantics, or starts with preview-pr-. Gating the
call, not just the DatadogSdk.initialize that follows it, means a debug, profile, simulator or
flutter test run builds no DatadogConfiguration and no client. The checked-in
assets/environment_values/environment.json carries EnvString: dev, which is deliberately not
in the allowlist; deploys overwrite it from config/<env>/environment.json.
The Patrol harness (patrol_test/patrol_setup.dart) never initialises the SDK, so it no longer
builds a configuration either.
Both apps load the Datadog browser-agent scripts from web/index.html. Those scripts only expose
the SDK; RUM is initialised from Dart, so loading them does not report anything on its own.
Web Professionals — web-professionals/src/datadog.ts. initializeDatadog returns early
unless VITE_DD_ENV is prod, staging, or a preview-pr- value. It no longer falls back to
Vite's MODE, which used to tag local pnpm run dev sessions as env:development. The reported
tag is the normalised value, so prod and " PROD " cannot show up as two environments.
Reading Error Tracking issues¶
Two things are easy to misread when triaging:
An issue's title is one sample, not the whole group. Error Tracking groups by stack, not by
environment, so a single issue can hold events from staging and production at once and show
whichever sample it picked. Issue 56a32128-268f-11f1-8412-da7ad0900005 is titled
Fetch timed out ... https://api.staging.perci-fhir.com/oauth2/token yet legitimately matches an
env:prod search: over 30 days its staging events hit api.staging.perci-fhir.com while its
production events hit contenthub.percihealth.com, api.uk.awellhealth.com and api.cal.com.
No production service calls a staging host — that was checked directly and every error mentioning
api.staging.perci-fhir.com is env:staging in project perci-platform-staging. Filter or group
by env before reading a message as evidence about production.
Backend versions are commit SHAs by design. DD_VERSION is set to github.sha for both
staging and production deploys, so a bare SHA in last_seen_version is normal for
perci-platform-backend and is not a sign of non-prod telemetry. The Flutter apps report
<pubspec version>-<build number> instead.