Skip to content

Autonomous Migration Pipeline

How Claude autonomously executes the frontend refactoring roadmap (PPL-2956) with layered, CI-enforced evidence that every chunk is behaviour-preserving.

Honest framing first: "100% certain nothing broke" is not achievable — not for agents, not for humans. What this pipeline buys is layered confidence: each layer mechanically kills one class of regression, and the layers that matter most are enforced by CI scripts the agent cannot influence. The central idea:

The agent writes the proof before it touches the code, and CI guarantees the proof cannot be weakened by the change it is meant to verify.

The three locked stages

Every migration chunk (typically one screen) moves through three stages, each with its own PR and its own gate. Humans review and merge every PR; the pipeline never merges.

Stage A — Characterize (test-only PR, zero risk)

Before touching the target, the agent writes characterization tests against the legacy implementation: widget tests and golden tests that pin down what the screen does today, quirks included — every visual state (loading / error / empty / data), every user action, every API call it fires, every navigation it triggers. Fakes come from the shared harness in perci_platform_test_shared (PPL-2961) so all characterization suites look alike. Tests live under test/characterization/<screen>/ and must pass on unmodified develop.

This PR is trivially safe (pure test addition) and permanently raises coverage — the characterization suite outlives the migration as the screen's regression suite.

Stage B — Migrate (the refactor PR)

The agent migrates the screen to the target architecture (feature slice, Riverpod, typed client, AppScaffold) per the standards and the architecture guide.

The hard rule, enforced by CI (test-lock, PPL-2960): a migration PR may not modify the characterization tests it relies on. The suite written against the old code must pass unchanged against the new code. That is the behaviour-preservation proof, and because the lock is a plain diff check, the agent cannot game it by "adjusting" a failing expectation.

If a characterization test is genuinely wrong (it happens), fixing it is a separate, human-reviewed test-only PR that must land before the migration PR — the change to the proof is reviewed on its own, never smuggled in with the code it verifies.

Stage C — Verify (no code changes)

Independent checks that do not trust Stage B:

  • Adversarial review: a fresh-context agent whose only instruction is to refute the claim "this diff preserves behaviour" — hunting changed strings, dropped parameters, inverted conditions, altered endpoints, swallowed errors. Its verdict goes in the PR description.
  • Behaviour-diff linter (script, not model — PPL-2960): fails the PR if the diff touches things that are never "refactoring": user-visible strings/.arb entries, API endpoint path literals, routeName/routePath values, analytics event names.
  • Golden comparison: the Stage A goldens re-run against the migrated screen.
  • Patrol subset: any E2E journey crossing the touched screen runs on the PR.
  • Legacy-guard delta (PPL-2958): the burn-down counters must go down, never up.

All evidence — test counts, golden results, adversarial verdict, counter delta — is pasted into the migration PR description so the human reviewer sees the case, not just the diff.

Chunk-type-specific proofs

The generic stack covers screen migrations (roadmap phase 6). The structural phases need stronger, mechanical proofs, each built just before its phase starts:

Roadmap phase Proof
Phase 1 — package moves Rename-only verification: git diff --find-renames must show content-identical moves; melos analyze && melos test green. Script check in PPL-2960.
Phase 3 — API layer Record/replay contract tests: a Dio interceptor captures the exact HTTP (method, path, query, body shape) each legacy api_calls.dart call emits; the typed-client replacement must emit identical HTTP. Byte-level equivalence, no judgement required.
Phase 4 — FFAppState State parity tests: drive the old singleton and the new Riverpod provider through the same event sequence, assert identical state transitions. Auth slice additionally runs behind a runtime flag with an app-by-app canary.
Phase 5 — navigation Route-table snapshot: serialize every route (name, path, guard, params) before the refactor; assert the table is byte-identical after. Deep links pinned by Patrol.

These harnesses get their own tickets when their phase is scheduled; they are deliberately not built up front.

Orchestration

Runs on GitHub Actions with claude-code-action in automation mode — the same engine as the Datadog auto-triage pipeline (PPL-2643) — implemented in PPL-2962.

  1. Queue: the phase-6 backlog under PPL-2956, one ticket per screen, labelled refactor-queue, priority-ordered by churn × incident rate (from the roadmap).
  2. A scheduled workflow claims the top unclaimed ticket. It skips any ticket whose target files are touched by an open PR — this is the "never interrupt product development" guarantee. Conflicts are avoided, not resolved.
  3. It runs Stage A and opens the test-only PR (label refactor-characterize). When that merges, it runs Stages B and C and opens the migration PR (label refactor-migration) with the evidence bundle.
  4. Humans review and merge. Merge triggers nothing — the next queue pick is independent.

Controls: concurrency starts at 1 screen in flight; raise it only after the pipeline earns trust. Kill switch: remove the refactor-queue label or disable the workflow. Post-merge, the existing Datadog RUM monitors watch error rates per app; a spike after a refactor release files a ticket straight back into the queue via the DD→Jira wiring.

Trust model — what each layer actually guarantees

Layer Catches Blind spot
melos analyze type/nullability breakage anything that compiles
Characterization suite (locked) changed widget behaviour, states, actions behaviour the suite didn't capture
Golden diffs any visual change non-visual logic
Behaviour-diff linter changed strings/endpoints/routes/events logic with identical surface
Record/replay & parity harnesses API/state semantic drift endpoints/state not exercised
Adversarial reviewer subtle logic changes, dropped edge cases what a reviewer can miss
Patrol E2E integration/wiring breakage journeys without coverage
Human review judgement calls reviewer fatigue — mitigated by the evidence bundle
RUM monitors (post-merge) anything the above missed, in production needs traffic to trigger

The stack is designed so the blind spot of each layer is covered by another. The weakest link is Stage A completeness — a behaviour the characterization suite never captured is unprotected. That is why Stage A output is human-reviewed as its own PR, and why the pilot exists.

Rollout

  1. Land PPL-2960 (test-lock + behaviour-diff linter) and PPL-2961 (harness + skill).
  2. Build the workflow (PPL-2962) but trigger it manually only.
  3. Pilot (PPL-2963): one boring, low-churn main_pages/ screen end-to-end. Measure where the agent needed help, which checks fired (or should have and didn't), review cost. Fix the skill and checks, update this doc.
  4. Go/no-go on enabling the schedule; open the queue at concurrency 1.