Skip to content

Patrol autofix

How a failed Patrol E2E run becomes an investigated, fixed draft PR — without a human sitting in the slow edit-and-rerun loop.

Most Patrol failures are minor test-side issues (a waitUntilVisible hitting a release animation, a dropdown overlay off-screen) rather than real product bugs. Fixing them is slow only because the feedback loop is: edit the test → push → wait minutes for the Patrol job → see if it's green → repeat. This pipeline removes the human from that loop: on any Patrol failure it investigates whether the failure is a flake or a real break, and either opens a draft PR with a test fix or reports the broken functionality (and attempts a minimal product fix).

Its security model: the agent runs edit-only, the workflow owns git, and the draft PR is opened with a GitHub App token so downstream CI runs. Nothing merges without human review.

Flow

flowchart TD
  patrol[".github/workflows/e2e-web.yml<br/>(Patrol job, per app)"] -->|"job fails"| autofix[".github/workflows/e2e-web-autofix.yml<br/>(workflow_call, per app)"]
  autofix --> artifacts["download failing run's<br/>logs + Playwright report"]
  artifacts --> resolve["resolve failing titles -> Dart targets"]
  resolve -->|"no failures for this app"| noop["no-op (matrix leg for a passing app)"]
  resolve -->|"failures"| dedupe["dedupe: fingerprint = app + failing targets"]
  dedupe -->|"an autofix PR with this<br/>fingerprint is already open"| skip["skip; link the in-flight PR<br/>on the originating PR"]
  dedupe -->|"new failure"| budget["daily budget: any autofix PR<br/>opened in the last 24h?"]
  budget -->|"budget spent"| budgetskip["skip (no agent run, no PR)"]
  budget -->|"budget free"| debug["launch debug app on chrome<br/>(marionette VM service via DWDS)"]
  debug --> agent[".github/actions/run-claude-agent<br/>patrol-fix agent (edit-only)"]
  agent --> gate["verify gate:<br/>re-run failing target (workers=1)"]
  gate -->|"workflow commits + App token opens"| pr["draft PR -> develop"]
  pr --> human["human review"]

Components

  • Trigger — an autofix job in e2e-web.yml, needs: [patrol], runs when needs.patrol.result == 'failure'. It is matrixed over the same apps; the reusable workflow no-ops for an app whose run had no failures (matrix legs can't tell which app failed, so each filters itself).
  • Reusable workflow — .github/workflows/e2e-web-autofix.yml (workflow_call). Mints the GitHub App token, checks out the PR head, downloads the failing run's <app>-test-logs artifact, resolves failing test titles back to Dart target files, deduplicates against in-flight autofix PRs and enforces the daily budget (see below), branches off develop, sets up the Patrol toolchain, launches a debug app for marionette, runs the agent, re-runs the failing target as the gate, and opens the draft PR.
  • Composite action — .github/actions/run-claude-agent runs Claude Code headless in edit-only mode (strips the agent file's frontmatter, appends a run-context data block, claude -p --permission-mode acceptEdits). Shared with the Datadog pipeline so the "run the agent" step has one source of truth.
  • Agent — .github/agents/patrol-fix.agent.md (prompt) + .github/agents/patrol-fix.mcp.json (marionette MCP). Classifies flake vs real break, applies the smallest fix reusing integration_test_shared helpers and the existing page objects, validates with a scoped patrol re-run, and writes a JSON verdict the workflow reads.

How the agent decides flake vs. real break

The agent reads the failing step from the log and Playwright report, then probes the live debug app with marionette (connect, get_interactive_elements, tap, …). If the flow works interactively, it's a flake and the fix belongs in the test; if the feature is genuinely broken, it's a real break and the agent attempts a minimal first-party product fix plus a written report. Known flake root causes and their fixes are encoded in the agent prompt (dashboard card waitUntilVisible, provider-read-during-build, off-screen dropdown overlay, double-dispose, un-awaited sign-out).

The authoritative gate is the workflow re-running the failing target with --web-workers=1. The agent's own re-run is just for fast iteration; the workflow's gate decides whether the PR is labelled needs-human.

The debug app the agent probes runs on the chrome device, not web-server: web-server only serves the bundle, so nothing loads the page — main() (hence MarionetteBinding.ensureInitialized) never runs and DWDS never emits a VM service URI. The chrome device (headless Chromium under xvfb, reusing the Playwright-managed browser) boots the app so DWDS prints the ws:// URI marionette attaches to. It is still best-effort: if no URI is captured the agent falls back to the scoped patrol re-run.

Deduplication

An external cause — a down backend endpoint, a flaky third party — tends to fail the same target across every open PR at once. Without a guard, each originating PR would spawn its own draft fix PR, so one outage becomes a wall of near-identical PRs.

Each leg fingerprints the failure as app + sorted failing target files (falling back to a hash of the failing titles when no target resolves) and, before doing any expensive work, searches open patrol-autofix PRs for that fingerprint (embedded as an HTML-comment marker in the PR body). If one is already in flight, the leg skips the agent run, the gate and PR creation, and instead comments a link to the in-flight PR on the originating PR. So one broken target yields one draft PR regardless of how many PRs trip over it. Per-PR re-runs (synchronize) are also covered — the deterministic fix branch is reused, and once the in-flight PR closes a later re-run is free to open a fresh one.

Daily budget

On top of the fingerprint dedupe, at most one autofix PR is opened per rolling 24 hours, repo-wide. Fingerprints only catch exact repeats: one underlying cause can fail a different app or resolve to different targets, and a closed-unmerged duplicate frees its fingerprint — each such leg would burn a full agent run for the same thing. Before doing any expensive work, the leg checks for any patrol-autofix PR (in any state, so closing one does not refund the budget) created in the last 24 hours; if one exists it skips quietly — no agent run, no PR, a note in the job summary — and the job succeeds. The check is best-effort, not a strict lock: legs for different PRs can race past it concurrently, which at worst yields one extra PR.

Outcomes & labels

  • Flake, gate passes — draft PR to develop, labels patrol-autofix, auto-fix.
  • Real break — draft PR with the minimal fix + a report, additionally labelled needs-human; the report is also commented on the originating PR.
  • Gate fails / inconclusive / no change — draft PR (if any change) labelled needs-human; the job fails so the failure stays visible.
  • Duplicate of an in-flight failure — no PR opened; the originating PR gets a comment linking the existing autofix PR, and the job succeeds (it is not a real failure).
  • Daily budget spent — no agent run and no PR; the job summary links the most recent autofix PR and the job succeeds.

Configuration

No new secrets. Reuses the Datadog pipeline's ANTHROPIC_API_KEY, AUTOFIX_APP_ID, AUTOFIX_APP_PRIVATE_KEY, and the optional OTEL vars. The autofix job runs on every Patrol failure; to pause it, disable or remove the autofix job in e2e-web.yml.

Boundaries

  • Edit-only agent; first-party files only (no generated clients, lockfiles, vendored code).
  • Fixes always target develop (consistent with the Datadog pipeline). A flake fix won't auto-unblock the originating release/* PR — the reviewer routes/cherry-picks it.
  • Never writes to main, release/*, or hotfix/*, and never merges.