Skip to content

Migration, adoption and rollback plan

Temporary planning document. Planning only: no migration, backfill, inventory of production data or cutover is authorised by this page. MIG1 authorises planning outcome migration; execution, activation and live migration each need separate approval. This page covers adoption for every domain in the integrated plan, not just outcomes. The outcome specifics stay in the outcome-migration proposal, which this plan adopts as the proposal for Q-05. Owner session: Q-05 is decided-amended through R4, so outcome data converts inside each project's faithful baseline (BC-R19); the proposal keeps the outcome-specific mapping detail.

Revised after round-2 review (3 October 2026). Ownership is now enforced by markers on the documents legacy writers write, cutover follows ADR-020's lock, verify, stamp and release protocol, and rollback and restore follow the consistency model (§6, §13 and §17.6). The allocation and target rows come from programme integration §3.

Revised after the owner session (5 October 2026). Chris chose one eventual writable engine (R4 owner direction, OS-A15; consolidation §5). The baseline conversion specification (BC) is now the detailed source for this page, and its §13 is the list of changes applied here:

  • Universal faithful baseline conversion. Opt-in staging trials, then owner-approved production pilots, then scheduled conversion of every remaining project, including completed, inactive and deleted ones, once parity is proven. No project stays on the legacy path for good and there is no permanent dual writable mode. R6 becomes the universal baseline conversion waves.
  • Faithful baseline. Conversion reproduces existing behaviour with the minimum versioned structures and makes no protocol choice. Missing history is labelled with explicit legacy-gap states and is never fabricated.
  • Mapping review in the opt-in wizard, with a reviewed manifest and project-specific findings.
  • Quarantine and remediation for any project that cannot convert faithfully. Conversion is never forced through by dropping data.
  • Recovery boundary. Before the first canonical write, rollback restores legacy routing. After it, recovery is canonical-aware and moves forward.
  • Later redesign is a separate, versioned publication inside the new engine; it repeats no storage migration.
  • Q-35 is removed. No legacy reconciliation lane exists; each dry run checks the no-records premise.
  • Legacy-writer retirement (R7) is its own verified milestone that Chris approves separately.
  • D2-13 (recovery after data loss) is a brief item with a required rehearsal.

No migration, conversion wave, pilot, production inventory, dry run against production, cutover, recovery execution or legacy-writer retirement is authorised. Feature implementation remains on hold (Chris, 5 October 2026). Owner decisions here are planning approval only; brief approval and implementation authorisation are separate, per gate (D1-04), and neither has been given. Superseded text below is kept and marked "Superseded by the owner session (5 October 2026)". The dependency-aware placement of the waves is in the rollout plan, progress rows (T-BC-00 onwards) in the implementation tracker, and counts and amendment IDs in the owner-session integration.

1. Principles

  1. Additive and evidence-preserving. Never delete or overwrite reviewer data. Originals, manifests and adapters are retained after any cutover.
  2. No fabricated history. Legacy records become labelled current snapshots (historyCoverage), never invented versions, votes, adjudications, gold history or original timestamps (EX2, research §5). Values that may be untouched defaults are labelled "value or default (unknown)", never read as answers. Owner session (BC §3.2, BC-R08, BC-R09): the labels are now the named legacy-gap states below, attached to the field or record that lacks the information. They are domain states, never answer values, and stay distinct from a reviewer's Unknown, Not reported or Not applicable answer and from an unanswered new question. Nullable historical times stay null; the conversion time is always recorded separately.
Legacy-gap state (PROPOSAL names) Meaning
NotRecordedInLegacy Legacy never captured this kind of fact (earlier Save or Complete versions, screening reasons never collected, pool entries before tracking, step settings before conversion)
CurrentSnapshotOnly Only the latest value survives; earlier values were overwritten (replaces "current-only")
UnknownLegacyAuthor The record exists but its author cannot be established
UnknownLegacyTime The original time is missing or untrustworthy; the field stays null
UnknownAuthoredUnderDefinition The question wording an answer was written under cannot be proven (AuthoredUnder = Unknown)
ValueOrDefaultUnknown The stored value may be an untouched default (replaces "value or default (unknown)")
LegacyCompletionUnvalidated Legacy marked the session Completed but the server never validated it (replaces "legacy-completed, unvalidated")
  1. Per-project adoption. New projects start on the canonical path once admitted. Existing projects adopt one complete scope at a time after a reviewed manifest. A scope is never split between legacy and canonical writers (C16). Superseded by the owner session (5 October 2026), see BC §1 and BC-R01 to BC-R03, BC-R23: existing projects move through opt-in staging trials, then owner-approved production pilots (complete scopes only, with validated reader and writer coverage), then universal waves that convert whole projects. There is no permanent dual writable mode and no permanent legacy archive adapter. A scope is still never split between legacy and canonical writers.
  2. Every writer and reader is accounted for. Writers must either use canonical commands or be refused for canonical scopes through R0's ownership markers and write guard (research A27):
  3. interactive saves, screening and reconciliation;
  4. reviewer session removal (a hard delete today);
  5. question edit and the question-delete cascade through the legacy API, which the new editor and the #3934 import both use;
  6. reference-file screening import, and question-template import (#2781/#3934, through the legacy question API above; this is not annotation import). FEAT-004 annotation import from other tools, if D4-14 approves it, is a canonical lane after R2a with Imported provenance (C3), not a legacy writer (owner session: D4-14 is decided-amended; annotation imports are the later lane XA1 and external or AI-model screening decisions the later lane XS1, both canonical, RI §10);
  7. bulk study update v2 (with study locks) and batch risk of bias;
  8. FEAT-024 writers, including the fold worker that bumps Study's audit version;
  9. the inclusion recalculation and every other UpdateMany on pmStudy;
  10. the tracking writers (hub, PM consumers, claim pipelines, typed admission);
  11. the notification stack's study-issue (#3945) and checked-PDF (#3947) Study writers;
  12. the reversible-deletion scheduler once built, and import-failure compensation that deletes studies (owner session: ordinary project deletion is reversible and stamps a deletion marker on every Study, checked by the composite write guard; no scheduler physically deletes a project's data, ACD §3.3);
  13. bulk PDF finalisation; preview seeding (its delete is exempt from bulk locks);
  14. admin tools, Quartz jobs, seed and fixture loaders;
  15. support-impersonation writes, which must record the real actor.

Readers need the same treatment (route, refuse or adapt): pool filters, capacity guards, reconciliation readiness, StudyStats, exports, the AF2 reconcile source, FEAT-024 classifiers, the allocation and eligibility facts, the presence snapshot and FEAT-024's availability calculators. #3944's candidate lookup reads legacy sessions and stores their IDs, so conversations refuse canonical scopes until R4a binds them to the task. The Study canonical summary (C1), merged by R0's floor, keeps legacy readers correct during coexistence. The merged bulk-update study locks (#3909) must be honoured by canonical writes. The inventory requirements are in the consistency model §6.5. 5. Compatibility before canonical writes (R0). Old binaries must capture and write back new fields, and R0 also ships the reader merge that keeps Study's computed legacy fields correct. Legacy writers refuse canonical scopes by a marker on the documents they write, checked in aggregate methods and by a composite write guard: data, not configuration. Flags gate only new admission. Each release ADR records its minimum rollback image. 6. Canonical-aware rollback. Before canonical writes exist, roll back by routing reads back to intact legacy data. After canonical writes exist, roll back by stopping new writes, keeping new data authoritative for what it holds, and using read-only containment or a verified forward-recovery adapter. Never down-migrate destructively, never let an old writer strip fields, and never let a configuration change hand a canonical scope back to legacy writers. Owner session (BC-R27, BC-R28; BC §4.8 to §4.10): the boundary is exact per converted scope. The ownership registry entry records firstCanonicalWriteAt (PROPOSAL), set by the first command that writes new evidence or configuration, in its own transaction. Routing rollback requires that value to be null under the same compare-and-set. After it, recovery is canonical-aware and moves forward, and new work is never flattened back into legacy records (consistency model §6.4). 7. Rehearse before running. Every cutover is rehearsed on synthetic fixtures, then on an authorised non-production copy, including induced failure at each cutover step (copy, verification, lock, delta, stamp and release), and an image rollback with canonical data present. 8. Erasure and retention are designed, not assumed (E32). Account deletion's existing behaviour (the irreversible Deactivated record) is extended to immutable revisions, exposure events, receipts and inbox items; drafts, exposure events and notifications get retention rules. Answers stay attributed to an anonymised identity, and as-of exports are identical except erased identities, which manifests record (D2-14). If today's behaviour can't extend, Chris decides. Superseded by the owner session (5 October 2026), see ACD §3.1, ACD-R01 and ACD-R02: ordinary account deletion disables sign-in and access, and submitted contributions keep named attribution. No identity-erasure process is decided (T-POL-02), and nothing here claims legal sufficiency. The retention-rules requirement stands (E32): drafts, exposure events and notifications still get retention rules, and conversion keeps disabled memberships and their history exactly. 9. Conversion is faithful and never forced (owner session; BC-R04 to BC-R09, BC-R26). Conversion preserves existing functionality and workflow semantics, generates only the minimum versioned structures, keeps original identities and provenance, and never reopens a completed stage or project to convert storage. A project that cannot convert faithfully is quarantined: its evidence is kept, the blocker is recorded with a concrete remedy, an owning lane and a review milestone, and it waits on the legacy path (before cutover) or in read-only containment (after cutover) until the remedy lands. Parity is never declared and a lossy conversion is never forced. 10. Redesign is separate (owner session; BC §3.1, BC-R31). Later rearrangement of a converted project (for example several steps in one stage, or steps across stages) happens inside the new engine through a redesign wizard. It is an ordinary versioned publication with impact previews, active-work warnings, compatibility and target treatment and explicit admin confirmation. It reads and writes no legacy data and repeats no storage migration. 11. Legacy-writer retirement is its own milestone (owner session; BC-R29, BC-R37). Legacy writers are retired only after every project has converted and coverage, access, export and restore checks pass, with Chris's explicit approval. No date is set. Retirement deletes no immutable history or legacy originals; originals, manifests and aliases are retained and need no second active engine. 12. Optional betas stay off (owner session; BC-R32). Baseline conversion never switches on the inference beta (owner decision, S5 activation amendment), training admission rules or external AI screening imports (both PROPOSAL).

2. Adoption modes

Mode Who What happens When available
Greenfield New projects admitted through R0's admission service Everything created on the canonical path from the start Forms and sessions from R2a (one stage) and R2b (several stages); steps and decisions from R3a; profiles from R3b; reconciliation from R4a; schemas from O1; classification from C1. Default for all new projects only from the GA milestone.
Reviewed adoption Existing projects chosen by Chris Inventory → manifest → shadow → verify → fenced cutover → monitor, per complete scope R6 waves, each separately authorised. Superseded by the owner session (5 October 2026), see the trial, pilot and wave rows below
Legacy All other existing projects Unchanged legacy behaviour, legacy reads/exports, no partial new semantics Indefinitely, until adopted or retirement is separately approved. Superseded by the owner session (5 October 2026), see BC §14: temporary, until the project's universal wave or the remedy of its recorded blocker
Opt-in staging trial (owner session) Seeded and synthetic staging projects, as each scope becomes complete The per-project protocol of §4 on staging data, with the wizard and findings; staging data preserved where possible (D3-14) Each release's own gate (BC §10.1 phase 1)
Production pilot (owner session) Projects whose owners opt in, each with Chris's per-pilot approval The §4 protocol for a complete scope with validated reader and writer coverage; "wait" is a supported choice After the pilot's G-ADOPT check and a passed D2-13 rehearsal (BC §10.1 phase 2)
Universal baseline conversion wave (owner session) Every remaining project, including completed, inactive and deleted ones, and projects that never opted in The §4 protocol per whole project, scheduled in waves with a notice period; each project commits or rolls back on its own; a project that cannot convert is quarantined with a review milestone R6, once parity is proven on pilots and Chris approves the wave (BC §10.1 phase 3)
Converted project, later redesign (owner session) Any converted project, when its admin chooses The in-engine redesign wizard publishes new versions through the ordinary publication protocol (§1 principle 10) After R2c and R3a; scheduled by the rollout plan

Scope completeness (a G-ADOPT check): a stage is adoptable for a domain only when every reader and writer of its data is canonical. In practice:

  • annotation-only stages without reconciliation: after R2a/R2b;
  • Combined stages (screening and annotation together need atomic Complete-and-Include): after R3a;
  • anything that is reconciled: after R4a, and after R4p for profile reconciliation;
  • extraction: after O1 and O2.

A "sessions only" wave that leaves screening or reconciliation of the same stage on legacy writers would split ownership and is never offered.

Owner session (BC-R23, BC-R25): scope completeness now governs pilots and staging trials. The permitted partial boundaries for pilots are fixed in the brief. Universal waves convert whole projects and never leave a half-converted authoritative scope. A project whose scope is not yet complete gets a recorded blocker and a review milestone; it does not become permanently legacy. "Anything that is reconciled" applies only where legacy reconciliation exists, which is expected nowhere (Q-35 removed).

3. Domain mapping (what legacy data can and cannot become)

3.1 Universal baseline mapping (owner session, 5 October 2026)

This table replaces the adoption-era mapping of §3.2 for the universal baseline (BC §3.4, which owns it). Every row is a faithful mapping, verified by parity. Names are PROPOSALs until the F1a naming and storage ADRs.

Legacy thing Baseline target Legacy-gap states used Never inferred
Project annotation questions Question identity plus a v1 version keeping original IDs; counted as published (QD1) NotRecordedInLegacy for earlier wording Overwritten wording or options
System questions Pinned SystemQuestionVersion per project system version None That later code left the definition unchanged
Stage question set One form per stage set by default, its v1 version bound to that stage. A shared form across stages is an admin choice, shown with its duplicate preview (BC-R15) None That two stages historically shared evidence
Overlapping question sets across stages A shared form (admin choice) or overlapping forms once R2d ships; until R2d, a finding blocks separate forms that share an answerable question (A-19) None A silent merge
Stage target (SessionCountTarget, override or inherited) FormVersion.standardTarget in the immutable v1 form version, materialised from the effective value at conversion (target versions the form, OS-A12; BC-R16) None That the screening threshold was meant as an annotation target
Converging stages with different targets The admin chooses the shared form's standard target; the wizard shows the effect per Study None A guessed total
EnforceAnnotationTarget A CapacityCap equal to the target where legacy enforced it; no cap otherwise None A cap where legacy had none
Idle timeout, in-progress limit (stage-level) Form-owned values (Q-28, D3-18); for converging stages the admin chooses, with the most restrictive as the suggested value None Stage-owned rules after conversion
Reconciliation blinding as configured in legacy Form-owned blinding (profile-owned for screening); converging stages with different settings need an admin choice; blinded is the suggested value None A blinding setting legacy did not have
Stage mode (screening, annotation, combined) and excluded-work policy One step per stage by default with the activities and dependencies that reproduce actual behaviour, including Allow or Stop for extra screening and the excluded-work setting (Q-24; BC-R13) NotRecordedInLegacy for step settings before conversion Historical step settings
Stage filters, partitions and inclusion requirements A StageStudyFilterVersion per stage whose clauses reproduce the legacy pool; pool parity is set equality at conversion; baseline members get StagePoolBaselineMember events with coverage BaselineAtTrackingStart (BC-R33) NotRecordedInLegacy for pool history before conversion Earlier pool membership
Stage availability and completion state The canonical lifecycle state and mode that reproduce availability; a closed or completed stage stays closed and is never reopened by conversion (BC-R06) NotRecordedInLegacy for status history Reopening to convert
Annotation sessions One converted session version per session, CurrentSnapshotOnly; nullable times kept; legacy Completed becomes LegacyCompletionUnvalidated and counts as completed by default because legacy counted it (E10 admin choice) CurrentSnapshotOnly, LegacyCompletionUnvalidated, UnknownLegacyTime Prior Save or Complete versions; validity
Answers and parent and child trees Answer heads with one legacy revision each; AuthoredUnder Verified(v1) or Unknown; conflicting duplicates become one Conflicted head; E10's "membership-uncertain" label stays as a provenance flag UnknownAuthoredUnderDefinition, UnknownLegacyAuthor Context sharing from text similarity; "latest wins"
Same reviewer, same Study, converging routes Previewed in the wizard and resolved before commit under the provenance and conflict rules (BC-R14); counted once CurrentSnapshotOnly A silent double count
Screening records (project and screener) Decisions under the one legacy-compatible profile, current value only CurrentSnapshotOnly; NotRecordedInLegacy for reasons and earlier decisions Earlier decisions, reasons, adjudications; a profile split by stage
Project agreement threshold and inclusion results The legacy-compatible profile's collective rule reproducing the characterised legacy maths, with its tie policy set explicitly to ExtraReview with the bound legacy applied (RS-R50); outcomes recalculated and compared; an admin confirms the profile's scientific meaning (Q-21) None Separate historical profiles per array entry; an inferred tie default
Legacy reconciled answers or sessions None expected (Q-35 removed). Every dry run counts them; any found stop that project's case and go to quarantine, with the treatment brought back to Chris (BC-R20) n/a Reconciliation authority
Single-reviewer studies on target-one forms Converted candidate evidence only; the v1 form version carries targetOneHandling = NoAcceptance (RS-R04a, PROPOSAL), so conversion creates no accepted results and no reconciliation work; exports keep the label "single annotator, not accepted" (BC-R21; BC ambiguity A1) None An accepted result; a reconciliation
Outcome data and time points Legacy-compatible schema series and observations with stable paths (O1, Q-05, outcome-migration proposal; BC-R19) ValueOrDefaultUnknown, CurrentSnapshotOnly Missing values from defaults; a majority direction
Permissions, groups, memberships Preserved exactly, including disabled memberships and their history; owner-only actions enforced None Any broadened default
Reservations and claims Translated under a coordinated lease transition at cutover; busy Studies deferred, never force-released None Reviews, drafts or votes
Proportional allocation regime, if enabled Blocker until AL1 is live (BC-R17) n/a Disabling allocation silently
Progressive batch plans, if present Existing immutable membership kept; completion rebound to canonical requirements None Re-ordered membership
Search, import and PRISMA source data Citations and source fields only where evidence exists; otherwise unknown with coverage labels NotRecordedInLegacy A guessed source type; deduplication as a side effect
Notification-stack aggregates Listed in the manifest; references remapped through LegacyIdAlias or marked unresolved None Dangling references
Materialised statistics Rebuilt under new family source versions; legacy basis labelled None Historical checkpoints for semantics that did not exist
Project in the reversible deleted state Converts in that state, without notices to members, so that restoration stays possible after retirement (BC-R38, PROPOSAL) As for its contents A restoration; member notices

3.2 Adoption-era mapping (superseded record)

Superseded by the owner session (5 October 2026), see §3.1 and BC §3.4. This table is kept as the round-2 record. The rows for stage targets, reconciled answers, single-reviewer studies and allocation carry their own superseded notes because their wording conflicts with owner decisions.

Legacy evidence Canonical target Never inferred
Project annotation questions Question identity plus a v1 question version (Scope = project), keeping original IDs. Adopted questions count as published, so they can never be permanently deleted afterwards (QD1). Earlier wording or options that were overwritten
System questions Immutable snapshots per question and SystemQuestionVersion (v0 and v1 projects differ structurally), pinned by adopted form versions That a code change since authoring left the definition unchanged
Stage question assignments One initial form version per stage assignment set, bound to that stage. Merging identical sets across stages into one shared form is an admin-reviewed choice, never automatic. That two stages historically shared evidence
Stage targets (Stage.SessionCountTarget, an override or inherited from the project screening threshold) The form's minimum target (an operational setting, D2-05), materialised from the stage's effective value at adoption, so later changes to the agreement threshold no longer move it. When stages merged into one shared form have different targets, an admin chooses (AP-13; criterion AC-R6-14). Superseded by the owner session (5 October 2026), see §3.1: the standard target is part of the immutable form version (FormVersion.standardTarget), not an operational setting (superseded wording 3) That the screening threshold was meant as an annotation target; any target that tracks the threshold after adoption
Annotation sessions (Incomplete/Completed, Reconciliation flag) Legacy session snapshot: current explicit state only, labelled current-only (owner session: CurrentSnapshotOnly; "legacy-completed, unvalidated" becomes LegacyCompletionUnvalidated); nullable timestamps kept. Per E10: an answer whose stage differs from the session's is marked "membership-uncertain" (a stage-A session's view drops answers later re-saved from stage B); merging stages into one shared form leaves two sessions whose statuses may conflict, so an admin chooses; legacy "Completed" sessions were never validated on the server, so they adopt as "legacy-completed, unvalidated" and an admin decides whether they count Prior Save/Complete versions; an Include vote; a completed contribution under a new form version; validity
Annotation answers and parent/child trees Annotation heads with legacy IDs (kept through LegacyIdAlias) and a single legacy revision; ambiguous edges preserved and flagged; adopted revisions carry AuthoredUnder: Verified(v1) when the stored wording equals the adopted v1 wording, otherwise Unknown, which is excluded from same-version agreement and exact-match prefill; conflicting legacy duplicates adopt as one Conflicted head (C2, C3) Context sharing from text similarity; owned-decision edges from answer text; the wording an answer was authored under
Cross-stage answer replacement effects Current values only, with stage provenance where stored Values that a later save overwrote
Screening records (project + screener) Decisions under an admin-labelled "legacy project screening" compatibility profile; current value only. For projects admitted before adoption, the R2a capture log adds the screening history recorded since admission. Earlier decisions, stages, criteria, reasons or adjudications; profile split by StageId
Project agreement threshold, InclusionInfo[] The legacy compatibility profile's collective rule, reproducing the characterised legacy maths; recalculated outcomes. Decoupled from targets: after adoption the threshold supplies no stage or form target (AP-13) Separate historical profiles per array entry
Reconciled answers and reconciliation sessions Legacy authority snapshot marked LegacyAuthorityUnknown, treated per Q-35 (recommended: exportable as "legacy reconciled (authority unknown)", never a gold snapshot, not queryable, and a canonical reconciliation is needed for gold). Superseded by the owner session (5 October 2026), see §3.1: Q-35 is removed, no legacy-authority backfill or migration lane exists, and the dry run verifies the no-records premise (superseded wording 7) Source candidate vectors; resolver history; an accepted snapshot timeline
Single-reviewer studies No automatic promotion to gold (Q-29); exports label them "single reviewer, unreconciled". Superseded by the owner session (5 October 2026), see §3.1: conversion creates no accepted results; converted target-one forms carry NoAcceptance (RS-R04a), labelled "single annotator, not accepted", until the admin publishes AutoAccept or RequireHumanReconciliation Gold
Outcome data and time points Per the outcome-migration proposal: legacy-compatible schema series with stable observation paths. Each outcome measure gets a single canonical direction (ODIR1). Per-source or per-row values are evidence only: legacy GreaterIsWorse is a non-nullable bool defaulting to false, and the client defaults error type SD, average type mean and zero animals, so those values count as "value or default (unknown)" unless the dry-run proves an explicitly stored answer. The outcome-level system answer is preferred over row copies, and stale copies are flagged. Values are consolidated only within one author's (or one reconciled authority's) series for one outcome, never across reviewers. Conflicting values block binding until a reviewed mapping or separate measures. Missing values from zero/defaults; a majority direction; a direction from untouched defaults; cohort enrolment from result N
Permissions and groups Existing project/stage grants preserved exactly; owner-only actions enforced (R1b) Any broadened default (ChangeOwner, AssignPermissions and Delete stay owner-only)
Materialized statistics Rebuilt projections with the legacy basis labelled Historical checkpoints for semantics that didn't exist
Reservations and claims Drained or translated under a coordinated lease transition at cutover; enabling review eligibility requires migrating legacy reservations first Reviews, drafts or votes
Proportional allocation regime (if enabled on the stage) A frozen legacy regime record kept for provenance; allocation is disabled on adoption unless AL1 is live (D3-13a, AP-13; criterion AC-R6-14). Superseded by the owner session (5 October 2026), see §3.1: enabled allocation blocks conversion until AL1 is live, because silently disabling it would change behaviour (BC-R17) Shares, bucket assignments or plans on the canonical form; claims from historical allocation
Progressive batch plans (#3939, if merged) Existing immutable membership kept; completion stream rebound to canonical requirements Re-ordered membership; any PRISMA count from batch release that amendment A doesn't define
Search/import records and PRISMA source data Citations and source columns only where evidence exists; otherwise unknown/unclassified with coverage labels. Backfilled Citations come from re-parsed retained reference files, or are labelled as derived from current Study metadata (which bulk updates may have rewritten), never presented as "raw, as imported". Source type is inferred only where FEAT-011's table allows (for example PubMed XML → Database). A guessed "Database" source; deduplication as a side effect of any other migration
Notification-stack aggregates (StudyConversation, StudyIssue, inbox SourceIds) Listed in each adoption manifest; references to legacy sessions remapped through LegacyIdAlias or marked unresolved Silent dangling references

4. Per-project adoption protocol

  1. Inventory (read-only; production inventory needs its own authorisation). Per-project counts and checksums, duplicate natural keys, invalid reviewer IDs, dangling questions/edges, custom thresholds, missing timestamps, oversized documents, writer activity. Without an authorised aggregate-only survey, storage benchmarks rest on synthetic sizes. Owner session (BC §4.2, BC-R20, BC-R40): the inventory runs under a read-only database role, where any write attempt fails; it also counts legacy reconciled sessions and answers as the Q-35 premise check, plus stages and modes, question-set overlaps, allocation regimes, grants and the largest form's size. The dry run builds the default faithful mapping in memory and writes only a preview and findings. Production inventories are aggregate-only and never copy project content into tooling outside SyRF. Unexpected reconciled records stop that project's case and go to quarantine; authority is never fabricated.
  2. Manifest. Deterministic source → target identities, compatibility profile labels, form-binding and session-merge choices, target and threshold mapping and any legacy allocation regime's disposition (AP-13), unresolved records with dispositions. An admin approves it; source changes invalidate it. Owner session (BC §4.3, OS-A14, BC-R11 to BC-R14, BC-R30, BC-R39): the manifest is reviewed in the opt-in conversion wizard. Per legacy stage it shows the proposed steps, forms, screening profile, filter, targets and settings, labelled "created" or "reused", and how existing sessions, answers and decisions map, with counts. Where the admin converges stages onto one shared form, it lists every reviewer with work in both routes, and the manifest cannot be approved until each conflict group is resolved. The admin confirms the scientific meaning of the legacy-compatible screening profile (Q-21). Adoption findings name the affected stage, form, question or record count and a remedy, and say whether converting now is advisable or the project should wait (BC §10.3). Each edit is a new draft version with a base-version check. A wave approval covers only default faithful mappings with no unresolved findings; project-specific choices need the project owner or a delegate (PROPOSAL, BC ambiguity A3).
  3. Shadow. Idempotent, checkpointed backfill into non-authoritative storage with no serving change; reruns create nothing new. FEAT-024's staged operation fence is raised for every family for the shadow and cutover window, so nothing is served Fresh over a half-migrated population (MS-17).
  4. Verify. Semantic parity (decisions, outcomes, pools, exports, permission-filtered API output, statistics), not byte equality. Statistics parity is automated for ProjectScreening and labelled manual for other families until per-family audits exist (#3845). Pinned sessions and snapshots still read their original versions, and the integrity checker passes (consistency model §13.2).
  5. Cutover through ADR-020's protocol. An operation record with a lease and a generation; lock batches that refuse busy studies (reservations, canonical claims, open tasks, active drafts); refresh and re-verify the delta under the lock; stamp CanonicalScopes on the Project and on every Study of the scope and write the pmCanonicalOwnership registry; release with an Audit.Version bump; record the cutover revision and operator. Before the commit write a failure unstamps and releases; after it every interruption leads forward to release. Cutover is all-or-nothing through locks, not one atomic switch: legacy writes during the window are refused with a retry message that keeps drafts. Claims, presence records, connections and scheduled messages for the scope are drained or converted. Owner session (BC §4.5, BC-R28): the operation kind is baselineConversion; busy Studies are deferred and counted, never force-released, until none remain or the stall limit is reached (proposed, not approved); the commit write records the registry entry with firstCanonicalWriteAt null and writes BaselineConverted and StagePoolBaselineMember events. Conversion never offers Apply anyway.
  6. Monitor and recover. Checkpointed resume, quarantine of mismatches, originals retained; legacy writers refused for the adopted scope by the markers; FEAT-024 families reset and rebuilt under the new family source version; the checker runs nightly. Owner session (BC §3.3, §4.11, §8): a failure at any step stops that project's attempt and writes a quarantine record (blocker code, evidence, remedy, owning lane, review milestone and who is told). Before the commit write the project stays on the legacy path; after it, the scope goes to read-only containment until it is recovered forward. A quarantine closes only when the blocker is remedied and conversion succeeds; no quarantine ends by dropping data.
  7. Recovery boundary (owner session; BC §4.8 to §4.10). The first canonical command that writes new evidence or configuration sets firstCanonicalWriteAt. Until then an authorised operator may use routing rollback: a compare-and-set that requires the null value, clears the markers in batches and returns serving to the intact legacy data; the converted records stay as non-authoritative history of the attempt. After it, rollback is refused and canonical-aware forward recovery applies (§1 principle 6).

5. Release-by-release adoption and rollback

Release New writes introduced Adoption scope Rollback before canonical writes Rollback after canonical writes
R0 Compatibility floor, admission Extra-element capture on extended types; reader merge logic (inert until a canonical writer exists); CanonicalScopes markers and the composite write guard; enrolment (admission) records; the writer floor Platform-wide, behaviour-neutral Image rollback to the previous release (no canonical data yet) Not applicable: R0 precedes canonical writes. Its image becomes the minimum rollback image for R2a; once canonical data exists, binaries below it are never redeployed, and canonical commands refuse while any instance runs below it
R1a Question templates and import Template imports through the existing import contract All projects, flag per capability Flag off Imported questions stay; legacy semantics unchanged
R1b Members & groups visibility Owner-only enforcement (security fix); no new data All projects Flag off for the UI; enforcement stays Not applicable
R1c/R1d Groups and delegation Configurable groups, grants, delegation envelope All projects, behind authorization gates Flag off Groups and grants stay; flag-off hides management UI but never removes or broadens grants
R2a Versioned forms, immutable sessions Question and form versions, form sessions, revisions, drafts, session versions, the Study canonical summary, the capture log Greenfield and admitted pilots Flag off; legacy path untouched Stop new canonical writes; read-only containment for pilot data; canonical exports remain; markers keep legacy writers out; image rollback no earlier than R0, whose floor keeps the summary merged; if a statistics writer changed, FEAT-024's rollback order applies
R2b Shared sessions Multi-stage bindings, re-keyed claims, form-unique tallies Same Flag off Shared sessions stay readable from every bound stage; claims drained
R2c Publication with impact Policy records, publication operation records, impact manifests (audit only), projection rewrites. Owner session: also attributable publication-generated session versions, mapped revisions and target-only publications (D2-01 decided-amended; RD §3.15) Same Flag off Published versions and policy records stay; a running phase-2 operation stops at its next item and resumes on roll-forward, while the per-study fail-closed rule keeps gates safe; new publications stop. Owner session: generated versions stay as immutable history and are never removed
R2d Overlap, Fix, FV4 Outdated flags, Fix transitions, policy revisions Same Flag off Flags/fixes stay readable; new Fix disabled
R3a Steps, routing, decisions Canonical screening decisions, stage settings versions, steps, admission records, pool-entry events. Owner session: stage study filter versions and HistoryEvents (StagePool*, WorkFirstReleased, ReviewStarted) replace the pool-entry events (SP §3.12) Greenfield and pilots, after the screening floor step Flag off Decision- and step-aware read-only containment; legacy single-ScreeningInfo can't represent canonical decisions, so no flattening. Owner session: history writes inside canonical scopes are never switched off, because that would leave unexplainable gaps (SP §10.1)
R3b Profiles Profile versions, eligibility answers, derived decisions, reasons Same Flag off Profile-aware read-only containment
R3c Lifecycle Status events, change requests and approvals Pilots Flag off Status history stays; manual lifecycle continues
R3d Guided setup Setup drafts New projects Flag off; old wizard remains until GA Created projects stay canonical
R4a Form reconciliation and gold Tasks, matches, gold snapshots, assignments, extra-review requests Pilots Flag off Snapshots remain effective and readable; new reconciliation writes stop; candidates are never rewritten
R4p Profile reconciliation Adjudications, final screening outcomes Pilots Flag off Outcomes stay authoritative; new adjudication stops
R4b Queries Query items, concerns, resolutions, notices Pilots Flag off Open queries frozen readable; gold stays effective
R4c Outcome reconciliation Outcome series gold Pilots Flag off As R4a
R5a History and as-of export Export manifests All admitted projects Flag off Manifests are append-only; disable generation
R5c Agreement statistics None persisted beyond derived, rebuildable results All admitted projects Flag off Results are derived from canonical revisions; disable the view
R5b PRISMA reporting Frozen report snapshots All admitted projects Flag off Snapshots are append-only records; disable generation
C1/C2 Classification and inference Entity/population records, assertions, rules; inference is derived and rebuildable Pilots Flag off Explicit records stay readable/exportable; inference can be switched off at any time
O1 Outcome schemas Schema versions, bindings, schema-reference answers, measures, new-shape observations Greenfield and pilots Flag off New-shape data stays authoritative; legacy export returns a typed "unsupported shape" result rather than coercing
O2 Outcome migration Staged copies, then cutover per project Chosen projects after Q-05 and separate execution approval. Owner session: outcome conversion is part of each project's faithful baseline (Q-05 decided-amended through R4; BC-R19), delivered in pilots and universal waves, each with its own execution approval Route back to intact legacy data Forward recovery or verified reverse adapter only
AL1 Shared-form allocation Shared allocation plans Pilots Flag off (back to refusal) Plans stay readable; allocation falls back to refusal for shared forms
P1 Identification provenance Immutable Citation/source capture for new imports; retrieval and lifecycle events New imports in admitted projects, after the Study floor step Flag off Captured provenance retained; reports label coverage
P2 Identification and dedup Publications, Citation links, lifecycle status, dedup audit, reviewed merges. Owner session: consolidated merges (StudyVersion, StudyMerge, StudyUnmerge, MergeConflictTask, currentEvidence; DM spec) Admitted projects Flag off Dedup decisions reversible through the audit; no destructive unmerge. Owner session: a merge is reversed by an audited, immutable unmerge that restores the originals; merge reversal is separate from disaster recovery
GA milestone Canonical default for new projects New projects Revert the default; created projects stay canonical Not applicable
R6 Legacy adoption waves Adopted legacy snapshots per complete scope; markers and registry entries. Superseded by the owner session (5 October 2026), see the next row Projects Chris selects Per protocol step 5 (before the commit write: unstamp and release) Per protocol step 6 (forward recovery only)
R6 Universal baseline conversion waves (owner session; replaces the row above) Faithful baseline structures and converted evidence snapshots per whole project, with legacy-gap states; markers, registry entries with firstCanonicalWriteAt, LegacyIdAlias rows, BaselineConverted and StagePoolBaselineMember events; quarantine records Every remaining project, including completed, inactive and deleted ones, after pilots prove parity; each wave approved by Chris Before the commit write: unstamp and release. After it, until the first canonical write: routing rollback (§4 step 7) Read-only containment and canonical-aware forward recovery; markers never cleared (BC-R27)
R7 Retirement Removal of adapters and legacy writers Platform-wide, separately approved — Earlier images stop being valid rollback targets here; restore rehearsal is required first
R7 Legacy-writer retirement milestone (owner session; refines the row above) Removal of legacy writers and coexistence adapters; originals, manifests and aliases retained Platform-wide, only after every project has converted, with a retirement readiness record (empty consumer inventory, access, export and restore checks, retention approval) and Chris's explicit approval; no date set — Earlier images stop being valid rollback targets; the D2-13 rehearsal is required first (BC-R37, BC-AE29)
Later redesign wizard (owner session) New form, profile and stage-settings versions through ordinary publication Converted projects, by admin choice Guided rollback and republication, as for any publication Ordinary publication history; no legacy data read or written (BC-R31)

6. Rollback rehearsal checklist (every release with canonical writes)

  • An older binary at the recorded minimum reads documents containing canonical fields: no exception and no stripped field, and in a mixed fleet every persisted computed field and tally equals an authoritative recount (AC-R0-09, AC-R0-06).
  • A legacy writer retrying after canonical data exists is refused by the marker and changes nothing, including writers that open no transaction (research A19; AC-R0-11; C16-T07).
  • Removing a project from enrolment, or reverting configuration, doesn't hand its canonical scope to legacy writers.
  • Image rollback to the recorded minimum image, with canonical data present; canonical commands refuse while any instance runs below the writer floor (AC-R0-15).
  • Mixed-version API and web clients during deployment: stale clients get typed retry messages that keep drafts.
  • Interrupted backfill and rerun: idempotent, no duplicate heads or revisions.
  • Operations in flight (publication phase 2, sweeps, capture moves) survive the rollback: older binaries leave their records alone, gates fail closed on stale projections, and the operations resume on roll-forward.
  • Restore per D2-13: a point-in-time restore into an isolated database; the integrity checker across collections (history, snapshots, pins, drafts, command-bearing records, the canonical summary, markers and the registry); the history-discontinuity record; scheduled-state reconciliation. Never a selective per-project overwrite of canonical data (E31, E55; AC-R2a-29, AC-R7-02). Owner session: the rehearsal also builds a recovery manifest and recovers forward by new commands (BC-AE27).
  • Owner session: routing rollback succeeds while firstCanonicalWriteAt is null and is refused when a canonical write wins the race, under forced interleaving (BC-AE19); after the first canonical write, rollback produces read-only containment, copies no canonical write into legacy records and never clears markers (BC-AE20).
  • Owner session: induced failure at each conversion step leaves the project fully legacy (before the commit write) or fully converted (after it), with a quarantine record where the attempt stopped (BC-AE18, BC-AE21).
  • Statistics fall back to authoritative queries when a projection kind is incompatible. When the release changed a statistics writer, family or protocol, the rehearsal follows FEAT-024's rollback order and records the fold mode and stamp before and after (MS-16).

FEAT-024 rollback order (MS-16). When a release changed a statistics writer, family or protocol, rollback follows FEAT-024's order: (1) fold-disable for every fold project and wait for foldMode: "Disabled"; (2) close the project narrow gate, which takes two calls around the quarantine, and the fleet gate for a full stop; (3) turn the flags off in one cluster-gitops change for both hosts, never through runtime overrides, which the PM host does not see; (4) only for an image rollback past the fold, apply the allowlist guard alone and then change the images; rolling forward again takes two resets around guard removal. The rehearsal records the fold mode and stamp before and after (docs/features/materialized-project-statistics/phase2c-staging-proof-runbook.md, Step 10 and "Disable and rollback", read on main eb93caffa; consistency model §17.6).

Restore policy (D2-13, PROPOSAL until Chris answers). Superseded by the owner session (5 October 2026): D2-13 is a brief item, specified in BC §8.4, with a required rehearsal on an authorised non-production copy before the first production pilot. Production recovery execution needs Chris's separate authorisation per incident. The reviewed RecoveryManifest lists the records to recreate from the isolated restore, each as a new command with provenance (source restore point, original record ID, recovery operator, approver). Disaster recovery is separate from merge reversal and from conversion routing rollback (BC-R36). The policy text below stands as the specification. There is no selective per-project restore of canonical data. A point-in-time restore goes into an isolated database, and recovery into production is manifest-driven forward recovery: records are recreated as new commands with provenance, never overwritten. A whole-database restore writes a history-discontinuity record per affected project (restore point, stamps lost, reason), which manifests and as-of requests report. Writes reopen only after the integrity checker passes. Scheduled state in SQL Server (MassTransit scheduled messages, Quartz) does not rewind with Mongo, so it is reconciled from Mongo state. Commands committed after the restore point are lost; a client retrying one either succeeds against the restored bases or gets StaleBase. FEAT-024 families are rebuilt (consistency model §13.1).

7. PRISMA-specific adoption

PRISMA adoption needs amendments A–P (Q-06a, Q-06b, Q-37; owner session: A–P, with B, E, F, K, L and M now approved and O deferred). Adopted projects can record steps done outside SyRF, such as deduplication before import, as reported counts (amendment K), and can run ASySD's retroactive deduplication inside SyRF after adoption (amendment L). Legacy projects keep current-state screening counts labelled with their basis. Report snapshots for adopted projects record coverage for import, deduplication, retrieval and pool-entry history that is missing, and use evidence-based lower bounds where possible (a study with a recorded legacy decision certainly entered screening). Deduplication of reviewed records is admin-reviewed and never a side effect of outcome or session migration (amendment D). Amendment G replaces FEAT-011's platform-wide backfills (MIG-11, MIG-12) with this per-project adoption, and amendment I replaces its $unset rollbacks with canonical-aware rollback.

Owner session (5 October 2026; PRISMA amendments A, D, G and H; BC §3.4, BC-R33; RI §3.2, §3.3):

  • "Legacy projects keep current-state screening counts" now holds only until each project's conversion. Converted projects report pre-conversion facts with legacy-gap states, and their pool history starts at the conversion baseline (StagePoolBaselineMember, coverage BaselineAtTrackingStart; earlier history NotRecordedInLegacy). Reports disclose the baseline date and never fabricate earlier entries (RI-R08).
  • Amendment G's per-project adoption becomes per-project conversion inside the universal waves.
  • Converted legacy screening outcomes are recalculated under the legacy-compatible profile; there is no LegacyUnknown outcome authority (Q-35 removed; amendment H as revised).
  • Duplicate merges of reviewed records follow the consolidated merge model with reversible unmerge (amendment D as revised; DM spec). Conversion never deduplicates as a side effect.
  • Exact PRISMA box mapping for converted history is specialist input T-SI-05.

FEAT-011 release checklists mapped to this plan (FEAT-011's "Release ½/3" are unrelated to this plan's R-numbers):

FEAT-011 checklist item Must pass in
Release 1: nullable sourceType and sourceName on SystematicSearch P1
Release 1: no Study or SystematicSearch fields that clash with planned PRISMA names (lifecycleStatus, screeningOutcomes[], duplicateGroupId, publicationId, citations[], fullTextStatus, status, state, type, category) Every release that adds persisted fields, checked at F1a (storage ADR) and in R0 floor steps
Release 2: classification fields don't use lifecycle enum names C1
Release 2: reconciliation never overwrites screeningOutcomes; no single-outcome field on Study; consistent authority patterns; Reconcile covers annotation and screening R3a (outcome shape), R4a and R4p (reconciliation)
Release 3: lifecycle status enum, pool exclusion of Duplicate and Merged P2
Release 3: backfill of lifecycle status on all studies; migration of all screening to screeningOutcomes[] Per-project adoption in R6 (amendment G), not platform-wide. Owner session: per-project conversion inside the R6 universal waves, which eventually covers every project
Release 3: screeningOutcomes[] schema; structured exclusion reasons groupable by primary reason R3a and R3b, with the per-profile shape of amendment H and the reason shape of Q-22. Owner session: Q-22 is decided through S3 (OS-A17); a primary reason is template guidance, several reasons stay supported, and distinct excluded Studies are counted apart from overlapping reason counts (amendment E as revised)
Release 3: pmPublication with DOI/PMID indexes; immutable Citations with all raw fields; duplicate count derivable P1 (Citations), P2 (Publications, duplicates)
Release 3: sourceType on new imports; backfill where determinable P1 (new imports, admin classification tool); R6 (inference per project)
Release 3: flow diagram generation; all 34 fields derivable; source columns; PRISMA and dedup exports (EXP-05/06) R5b (and P2 for the dedup export)

8. Conversion phases (owner session, 5 October 2026)

The order is fixed by Chris (register "R4 owner direction"). Release placement is a PROPOSAL for the rollout plan; every entry gate is still to be passed and nothing is authorised (BC §10.1, BC-R41).

Phase What happens Entry gate
0. Foundations Writer and reader inventory (AC-M0-03); CanonicalScopes markers and the composite write guard (R0); integrity checker (R2a); D2-13 rehearsal F1a
1. Staging trials Opt-in conversion of seeded and synthetic projects as each scope becomes complete (annotation-only after R2a and R2b; combined after R3a; profiles after R3b; extraction after O1 and O2; pools and history with R3a; exports with R5a) Each release's own gate
2. Production pilots Owner opt-in projects, each with Chris's approval; complete scopes only G-ADOPT for the pilot; D2-13 rehearsal passed
3. Universal waves (R6) Scheduled conversion of every remaining project, including completed, inactive and deleted ones G-ADOPT per wave; parity proven on pilots; Chris's wave approval
4. Legacy-writer retirement (R7) Remove legacy writers and coexistence adapters after every project has converted G-RETIRE; the retirement readiness record; Chris's approval
Later. Redesign wizard In-engine rearrangement of converted workflows through ordinary publication After R2c and R3a; ordinary release gate

Convert now or wait. The wizard recommends conversion when the selected scope has a delivered and validated capability for every activity, a faithful mapping, a dry run that accounts for every record, pool and offered-work parity, resolved overlaps and target choices, passing large-form performance, manageable active work, a clear rollback boundary and a clean Q-35 premise check. It names a concrete remedy otherwise (BC §10.3). During trials and pilots "wait" is a supported choice. During universal waves it becomes a temporary quarantine with a review milestone (BC-R30). The largest project is an acceptance case and is never excluded for size; if it fails a performance check it is quarantined with a performance remedy (BC-R24; D2-16 replaced).

Flags. Conversion is gated per project by CanonicalEnrolment and the ownership markers, never by environment flags. The wizard and wave tooling sit behind default-off flags until their release gates pass.

9. Owner-session amendment record (5 October 2026)

Amendment Source Where on this page
Universal faithful baseline conversion after trials and pilots; no permanent dual writable mode; legacy status temporary R4 owner direction; OS-A15; BC §13 Status, §1 principle 3, §2, §5 R6 rows, §8
Explicit legacy-gap states replace the older labels BC §3.2, §13 §1 principle 2, §3.1, §3.2 annotation sessions row
Mapping review and findings in the opt-in wizard; Q-21 meaning confirmation; manifest approval scope R4 amendment; OS-A14; BC §4.3 §4 step 2, §8
Domain mapping replaced (target in the form version, Q-35 removed, NoAcceptance for target-one forms, allocation as a blocker, steps, cap, timeouts, blinding, stage state, overlaps, deleted projects) BC §3.4, §13, §14 §3.1, §3.2 superseded notes
Q-35 premise check in every dry run Q-35 removed; BC-R20 §4 step 1, §3.1
Quarantine and remediation instead of lossy conversion Consolidation §5; BC-R26 §1 principle 9, §4 step 6, §8
Recovery boundary: routing rollback before the first canonical write, canonical-aware forward recovery after Consolidation §5; BC-R27, BC-R28 §1 principle 6, §4 steps 5 and 7, §5 R6 row, §6
Later in-engine redesign wizard as a separate versioned publication R4 owner direction; BC-R31 §1 principle 10, §2, §5, §8
Legacy-writer retirement as its own verified, owner-approved milestone R4 owner direction; BC-R37 Status, §1 principle 11, §5 R7 rows, §8
D2-13 brief item with recovery manifest, rehearsal and per-incident authorisation Consolidation §7; BC §8.4 §6 checklist and restore policy
Named attribution on account deletion; identity erasure not decided; retention rules kept D2-14 decided-amended; ACD §13 §1 principle 8
Reversible project deletion in the writer inventory O1 final clarification; OS-A25; ACD §3.3 §1 principle 4
Publication-generated versions, HistoryEvents and consolidated merges in the release table D2-01, OS-A07, OS-A29 §5 R2c, R3a, P2 rows
PRISMA adoption notes: per-project conversion, baseline coverage, no LegacyUnknown, consolidated merge, S3 reasons PRISMA amendments as revised; BC §13 §7
No migration execution authorised Consolidation §5, §8; hold of 5 October 2026 Status, §8