Skip to content

API-wide migration from Newtonsoft.Json to System.Text.Json (October 2026)

Chris decided on 2026-10-04 that the API moves from Newtonsoft.Json to System.Text.Json (STJ) before realtime hub v2, which will use the STJ hub protocol (AddJsonProtocol; see realtime-server-plan-2026-10.md §5 and D12). This plan inventories every Newtonsoft use, defines a parity strategy, picks a rollout mechanism and sets out the releases. It is planning only; nothing starts until Chris approves and answers the decisions in §12. Code facts are from main at 700d6a93f (4 Oct 2026). Markers: VERIFIED (code path read), CORRECTED (the brief or an earlier note was wrong; says how), NOT VERIFIED (reason given). No build or test was run for this plan.

A fact that shapes the plan (VERIFIED by the coordinator against dotnet/aspnetcore main, 2026-10-04): NewtonsoftJsonHubProtocol.cs:39 and JsonHubProtocol.cs:49 both declare ProtocolName = "json". One API process cannot serve Newtonsoft on /notifications (v1) and STJ on /notifications/v2. The migration therefore moves the existing v1 hub to STJ, with golden tests for every v1 message.

1. Inventory

grep -rn 'Newtonsoft' src --include='*.cs' finds 61 lines in 36 files: 23 production files (36 lines) and 13 test files (25 lines). The brief's "about 61" is right. CORRECTED: the S3-notifier Lambda, the PDF agent, Identity and the AI request DTOs contain no Newtonsoft code (see 1.1 and 1.4).

1.1 Hosts

Host JSON today Evidence Status
API MVC on Newtonsoft with SyRF options; hub on the Newtonsoft protocol; 14 direct STJ call sites (WriteAsJsonAsync problem bodies, Redis session store) SyrfConfigureServices.cs:47-49; Program.cs:172,209,390-401; ApplicationAuthorityResultHandler.cs:45; ReviewWriteAuthorizationResultHandler.cs:35; RedisSessionStore.cs:20 VERIFIED; in scope (R1, R2)
Project Management MVC on Newtonsoft with ASP.NET defaults (no SyRF converters, no dictionary-key casing); 4 controllers: api/internal/investigators (called by Identity), reader-status, application/info, e2e-setup PM Program.cs:476; Controllers/*.cs; caller ProjectManagementInvestigatorProvisioner.cs:41 VERIFIED; in scope (R3)
Identity Already STJ: AddControllersWithViews, no AddNewtonsoftJson, no Newtonsoft code. Pulls Newtonsoft.Json transitively through SharedKernel.csproj:33 Identity Program.cs:347-348 VERIFIED; only the R4 package removal touches it
Quartz No MVC; the Quartz job store serialises JobDataMap with Newtonsoft into SQL Server QuartzServiceCollectionExtensions.cs:85; SyRF.Quartz.csproj:27 (Quartz.Serialization.Json) VERIFIED; out of scope (persisted data; §3)
S3 notifier, PDF agent STJ already S3FileReceivedFunction.cs:123,249,299; OutcomeJournal.cs:74 VERIFIED; they reference WebHostConfig.Common, so R4's package removal shrinks them

1.2 Wire serialisers and custom converters (API)

Item Today Evidence STJ target
MVC options IConfigureOptions<MvcNewtonsoftJsonOptions>: 4 converters + CamelCaseNamingStrategy { ProcessDictionaryKeys = true } ConfigureJsonOptions.cs:17-29; Program.cs:209 JsonOptions (Web defaults) + DictionaryKeyPolicy = CamelCase + 3 converters (§5)
Hub protocol AddNewtonsoftJsonProtocol: 3 converters (not AnnotationAnswerDtoConverter) + the same camelCase resolver Program.cs:390-401 AddJsonProtocol with the same options object as MVC
ParentFilterConverter Read-only. Sniffs integer filterType; returns null when missing, non-integer or undefined AnnotationConverter.cs:14-79 JsonConverter<ParentFilterDto>
ConditionalTargetParentOptionsConverter Read-only. Integer conditionType; strict validation with specific error messages; throws JsonSerializationException AnnotationConverter.cs:81-200 JsonConverter<ConditionalTargetParentOptionsDto>
AnnotationConverter Read-only. String answerType → 8 subtypes; unknown → StringAnnotationDto; missing → null AnnotationConverter.cs:202-253 JsonConverter<AnnotationDto>
AnnotationAnswerDtoConverter Read-only, DI-built, swallows every exception and returns null. No endpoint binds AnnotationAnswer: its only production users are RiskOfBiasAiJob (BSON) and the DTO file; the RoB AI tool is suspended AnnotationConverter.cs:255-333; grep for AnnotationAnswer/AnnotationInfo/AiAnnotationRequest Delete from MVC options (D4)
JSON Patch JsonPatchDocument<T> (Newtonsoft-based package) on 2 actions ProjectController.cs:370,870; API.csproj:33; JsonPatchOperationProcessor.cs:18 Microsoft.AspNetCore.JsonPatch.SystemTextJson (exists for ASP.NET Core 10, per the aspnetcore docs, read via Context7); NOT VERIFIED in the local package cache
ProblemDetails Newtonsoft ProblemDetailsConverter flattens Extensions. ProjectStatisticsAdminController.cs:299-309 adds an extension named status, so the body has two status keys (documented in phase2c-staging-proof-runbook.md:497-499) as cited STJ's converter also flattens; which duplicate wins in JSON.parse depends on write order. Golden test required (B12)

Every converter uses CanConvert = IsAssignableFrom, so Newtonsoft also routes concrete subtypes through it. All are CanWrite = false: writes use Newtonsoft's runtime-type serialisation (see B1).

1.3 Serialisation attributes

File:line Newtonsoft attribute STJ twin today?
AggregateRoot.cs:23,24,26 [JsonIgnore] ×3 (Version, SchemaVersion, DefaultSchemaVersion) No
Entity.cs:61,78 [JsonIgnore] ×2, on Events and IsDirty. These are load-bearing for the FEAT-024 submission digest (D6): BindAnnotation hashes each Annotation (Entity<Guid>; 8 concrete subtypes) whole with JObject.FromObject, minus DateTimeCreated. OutcomeData is already an explicit projection (ProjectStatisticsSubmissionDigest.cs:59-66), so these two attributes matter for Annotation only No
ProjectAgreementThreshold.cs:34 [JsonIgnore] on computed AgreementMode No
StageSecuritySettings.cs:21,26,28,36,38 and :33 [JsonIgnore] ×5; [JsonProperty] on private field _customStagePermissions No
StageReviewerStats.cs:17 (ProjectReviewerStats.CurrentProgress) [JsonProperty(NullValueHandling.Ignore)] No (test ReviewControllerFullStatsMaterializedTests.cs:548 pins its absence)
StageAllocationReadController.cs:124 [JsonConverter(StringEnumConverter)] on AllocationRegimeOrigin No
Study.cs:149,160,240; Project.cs:28; UpsertCustomAnnotationQuestionDto.cs:112,124 Ignore / ignore-null Yes, dual (6 STJ attributes; the house pattern, see UpsertCustomAnnotationQuestionDto.cs:54-58)

14 Newtonsoft-only attributes need STJ twins. No JsonStringEnumConverter, [JsonPolymorphic] or [JsonDerivedType] exists in non-test code (grep: 0).

1.4 Other Newtonsoft uses (not the HTTP or hub wire)

Use Evidence Disposition
FEAT-024 submission digest: JToken.FromObject/JObject.FromObject of anonymous objects, whole domain Annotation entities (8 subtypes) and an explicit OutcomeData projection whose TimePoints (List<TimePoint>) and ExtraElements (IDictionary<string,object>) still pass through the serializer, canonicalised → SHA-256, persisted as a source digest for idempotent replay (receipt SourceDigest, fold entry and quarantine SubmissionDigest; duplicate and conflict classification) ProjectStatisticsSubmissionDigest.cs:31-98; callers ReviewController.cs:835,852, SubmitAnnotationSessionService.cs:170, AnnotationFoldTargets.cs:105, ProjectScreeningFoldSave.cs:708; persisted at ProjectStatisticsSourceOperationReceipt.cs:113,211, StudyPendingStatistics.cs:69,191; compared at ProjectStatisticsFoldClassification.cs:136-138, ProjectScreeningFoldSave.cs:559-575, ProjectAnnotationFoldSave.cs:350-370 Keep on Newtonsoft unless the statistics session versions the digest (D6). A serializer swap changes hashes, and so does a change to the Newtonsoft attributes on the hashed types (Entity.Events/IsDirty [JsonIgnore], honoured for Annotation; any enum converter on the hashed enums ScreeningDecision (SharedKernel) and SessionStatus (PM Core), both written as integers). ExtraElements is hashed too: inherited by Annotation (empty on the review path, because AnnotationDto has none) and projected for OutcomeData. The digest relies on Newtonsoft defaults via JsonSerializer.CreateDefault(): PascalCase names, integer enums, nulls included, "D" Guids, decimal scale kept, round-trip doubles; CreateDefault() also applies JsonConvert.DefaultSettings, which nothing sets today. It hashes the bound and mapped Annotation, so request binding changes it too (R1, decimal parsing). Adding any public property to Entity<TId> or Annotation also changes every annotation digest. No test pins the absolute digest today: ProjectStatisticsSubmissionDigestTests compares only relatively (statistics session reviews, 2026-10-05)
FEAT-024 checkpoint observation digests: definition metadata serialised with STJ default options (StageStatisticsDefinitionSnapshot.Serialize :26, ReviewerAnnotationScopeCalculator.DescribeDefinition :82, QuestionAnswersScopeCalculator.DescribeDefinition :119, ReviewerProgressDefinition.Serialize :29) → stored ContentDigest and Merkle roots, recomputed on read ProjectStatisticsDerivedSummaries.cs:188; ProjectStatisticsCheckpointResolver.cs:181 Already STJ, so the MVC switch does not touch them. They must stay on default options: moving them onto SyrfJsonSerializerOptions (camelCase, relaxed encoder) or adding an STJ type-level converter or attribute to an enum they serialise (QuestionType) changes new observations' bytes, and copied and calculated observations or unchanged-day detection diverge. Allowlisted in PR-0.1 with a golden
Response byte budgets (serialised length decides accept or fallback; not persisted) Newtonsoft JsonConvert.SerializeObject(...) length: StageOverviewStatisticsQuery.cs:280; ProjectReviewerProgressQuery.cs:124; SearchPopulationResponses.cs:20. STJ Serialize() length: ProjectStatisticsBundleReader.cs:314-315 (2 sites) R3: measure with the STJ options; cover all five sites. The three Newtonsoft sites measure PascalCase today, not the camelCase wire
Email template data (SES and SMTP render, JObject merge) AwsEmailService.cs:175,244,253; SmtpEmailService.cs:52,88,175; ApplicationController.cs:204 R3, with goldens per template. Default Newtonsoft settings = declared names, which STJ's defaults match
Multipart JSON reader FileStreamingHelper.cs:70 — StreamFile<,> has 0 callers Delete (surface-area plan)
Quartz job store QuartzServiceCollectionExtensions.cs:85 Out of scope; follow-up issue
Tests 13 files, e.g. ProjectOwnershipEndpointTests.cs:893 (test host uses AddNewtonsoftJson), EndpointAuthorizationCatalogTests.cs:384, ProjectControllerTests.cs:267-349 (converter tests) Translate in the PR that changes the code under test
AI request DTOs DTOs/AiRequest/AiQuestionDto.cs has no JSON attributes; only the converter above No change

Mongo: no Newtonsoft code in src/libs/mongo or any *.Mongo.Data project (grep), so there is no JSON↔BSON bridge. VERIFIED. MassTransit already uses STJ (ADR-021 I10). Caution: adding STJ twins to AggregateRoot/Entity/ProjectAgreementThreshold can change MassTransit message JSON if a contract carries those types (backend plan C15 already touches ProjectAgreementThreshold), so R0.2 checks the contracts.

1.5 Packages (VERIFIED, grep of every csproj; no Directory.Packages.props exists yet)

Package Where End state
Microsoft.AspNetCore.Mvc.NewtonsoftJson 10.0.0 WebHostConfig.Common.csproj:16; PM.Core.csproj:18 (no Microsoft.AspNetCore.Mvc/JsonPatch namespace used in PM.Core) Removed (R4)
Microsoft.AspNetCore.SignalR.Protocols.NewtonsoftJson 10.0.0 API.Endpoint.csproj:34 Removed (R4)
Microsoft.AspNetCore.JsonPatch 10.0.0 API.Endpoint.csproj:33 Replaced by …JsonPatch.SystemTextJson (R4)
Newtonsoft.Json 13.0.3 SharedKernel.csproj:33; PM.Core.csproj:25 PM.Core keeps it only if D6 keeps the digest. Removed from SharedKernel only if the digest no longer depends on Newtonsoft attributes there (Entity.Events/IsDirty): otherwise SharedKernel keeps it for those attributes, allowlisted (D6 item 2, 4.1.5)
Quartz.Serialization.Json SyRF.Quartz.csproj:27 Unchanged (out of scope)

Newtonsoft.Json stays in the graph transitively via NSwag/NJsonSchema (swagger header: "NJsonSchema v11.1.0.0 (Newtonsoft.Json v13.0.0.0)"). The goal is no Newtonsoft on the wire and no direct use outside an allowlist, not a zero-Newtonsoft dependency graph.

1.6 NSwag and the generated TypeScript client

  • Spec: syrf-generate-spec.nswag (aspNetCoreToOpenApi, Release, noBuild: true) over AddOpenApiDocument with two schema processors and a JSON Patch operation processor (Program.cs:716-735); NSwag.AspNetCore 14.2.0.
  • NSwag already models schemas with STJ, not with the Newtonsoft settings the wire uses. Evidence: UpsertCustomAnnotationQuestionDto.cs:56-57 ("MVC uses Newtonsoft while NSwag inspects STJ"), and the schema-processor tests build SystemTextJsonSchemaGeneratorSettings with Web defaults (ConditionalTargetParentOptionsSchemaProcessorTests.cs:15-18). NJsonSchema evidently also honours Newtonsoft attributes by name: AllocationRegimeOrigin is a string enum and ProjectAgreementThreshold has no agreementMode in swagger.json. VERIFIED from output; the mechanism inside NJsonSchema is NOT VERIFIED.
  • swagger.json: 191 paths, 373 schemas, 62 enums (61 integer, 1 string). Only ConditionalTargetParentOptionsDto has a discriminator (added by the custom processor); ParentFilterDto is an empty abstract schema although the wire carries filterType and value.
  • TS generation (syrf.nswag/syrf-local.nswag): Angular template, typeStyle: Interface, dateTimeType: String. No fromJS in api-client.generated.ts (count 0), so the client is a type overlay with no runtime conversion: the wire shape is whatever the server writes.
  • Expected effect: swapping the MVC serialiser and adding STJ attribute twins should not change swagger.json. Adding [JsonPolymorphic]/[JsonDerivedType] would (new discriminators and unions), which is a reason to prefer custom converters (§5). Every PR that touches DTO attributes regenerates per the memory rules: build SyRF.API.Endpoint in Release, pnpm run nswag:all from src/services/web, pnpm run validate:generated from the repo root, and commit .generated-checksums.json. A diff must be empty or explained. No generation was run for this plan.

1.7 Implicit behaviours and their STJ settings

# Behaviour Newtonsoft (today) STJ default (MVC Web) Needed for parity Risk
B1 Abstract-typed members (ParentFilterDto?, ConditionalTargetParentOptionsDto?, IEnumerable<AnnotationDto>, StudyBaseDto? NextStudy) Writes the runtime type Writes the declared type: subtype properties silently vanish Converters whose Write serialises value.GetType() High. 5 such properties found by grep (QuestionOptionDto.cs:19, UpsertCustomAnnotationQuestionDto.cs:49, StatsWithIncompleteDto.cs:17, SessionSubmissionWebDto.cs:22, StageStudyDto.cs:45); the scanner (PR-0.1) is authoritative
B2 Property names camelCase camelCase none Low
B3 Dictionary keys camelCased (ProcessDictionaryKeys) on write unchanged DictionaryKeyPolicy = CamelCase; goldens for acronym, digit, space, enum and Guid keys Medium (22 dictionary properties in the spec)
B4 Case-insensitive reads yes yes (Web) none Low
B5 Enums numbers; reads accept names and any integer numbers; reads reject names, accept any integer JsonStringEnumConverter only where Newtonsoft had StringEnumConverter; audit hand-written callers that send names Medium (inbound)
B6 Constructors public default → single public parameterised → non-public default public default → single public parameterised → [JsonConstructor]; a type with only a non-public default throws [JsonConstructor] where the scanner finds one Medium
B7 Get-only collection properties populated in place left empty JsonObjectCreationHandling.Populate per type Medium
B8 Public fields; private members with [JsonProperty] serialised ignored [JsonInclude] (.NET 8+ supports non-public) Low (2 domain fields, 1 private field)
B9 Nulls; ignore-null attributes written; per-attribute omission written STJ twins (WhenWritingNull) Low
B10 Missing and extra members defaults; ignored defaults; ignored none Low
B11 Type coercion on read: number→string, "true"→bool, 1.0→int, loose date formats, non-"D" Guids, comments accepted rejected (Web allows numbers from strings) none; fixtures and the 8 hand-written HttpClient call sites decide Medium (inbound 400s)
B12 ProblemDetails extensions, model-state error keys and converter error text flattened; keys like target.x; message as thrown flattened; keys like $.target.x; message plus path Golden on each problem shape; check web code that reads error keys Medium
B13 DateTime/DateTimeOffset ISO, trailing zeros trimmed, Z for UTC kind ISO, trimmed, Z Goldens per kind (Mongo returns UTC) Low
B14 double whole numbers 1.0 1 Intentional difference; the comparator normalises numbers None for JS
B15 decimal scale preserved preserved none Low
B16 Reference loops; max depth error; MVC 32 error; MVC 32 (NOT VERIFIED for both; the annotation tree depth is checked by goldens) none Low

2. Scope

ID Problem Impact Evidence (file:line) Issue
J1 MVC uses Newtonsoft while the OpenAPI model and every other serialiser (MassTransit, Identity, direct writes) use STJ Two contracts. The spec and the TS types already diverge from the wire (B1, ParentFilterDto) §1.2, §1.6 new
J2 The hub uses the Newtonsoft protocol; v2 needs STJ, and both cannot coexist Blocks realtime R1.3b Program.cs:390-401; ProtocolName fact new
J3 Four read-only shape-sniffing converters built on JObject Must be rewritten with matching leniency and error text AnnotationConverter.cs new
J4 14 Newtonsoft-only attributes Hidden fields would appear (e.g. AgreementMode), and the CurrentProgress omission and string enum would be lost §1.3 new
J5 JSON Patch depends on Newtonsoft Blocks package removal ProjectController.cs:370,870 new
J6 PM's MVC host on Newtonsoft defaults; its provisioning endpoint is on Identity's sign-in path Coordination with the authentication session PM Program.cs:476; ProjectManagementInvestigatorProvisioner.cs:41 new
J7 Non-wire uses (budgets, email, dead helper, dead converter) Package removal; budgets measure the wrong casing §1.4 new
J8 Persisted digest and Quartz job data are serialiser-dependent; the digest also depends on Newtonsoft attributes on hashed SharedKernel types, and the checkpoint digests on STJ default options A careless swap, attribute change or options change breaks idempotency (retried saves across a deploy classified ConflictingReceipt or quarantined instead of duplicates), checkpoint comparison, or stored jobs ProjectStatisticsSubmissionDigest.cs:90-98; Entity.cs:61,78; §1.4 checkpoint row; QuartzServiceCollectionExtensions.cs:85 new (excluded from re-basing, §3; guarded by PR-0.1, PR-0.2, PR-4.1)

3. MVP, out of scope and flag decision

MVP boundary (R0–R2): API REST and both hubs (v1 now; v2 when R1.3b lands) run on STJ in staging behind startup switches, with parity goldens green. That unblocks realtime R1.3b.

MVP acceptance: 1. With both switches on, every enumerated DTO and hub message serialises to JSON identical to today's, or the difference is listed. 2. Every real request fixture binds to an equivalent object. 3. The hermetic E2E suite passes with both switch values.

Out of scope (tracked where noted):

Item Tracked in
Quartz job-store serialiser (persisted JobDataMap) new follow-up issue: needs a dual-read migration of stored jobs
FEAT-024 digest re-basing D6, materialised-statistics session
MassTransit contract changes and the Wolverine move ADR-021 (S0.2 contract goldens)
ProjectAgreementThreshold MassTransit fix backend plan C15 / PR-11
Hub v2 itself realtime-server plan R1.3b
Client realtime rewrite and fixture spec realtime-client plan (CR-13 consumes our recorded frames)
Identity host already STJ; nothing to do
[JsonPolymorphic] with OpenAPI discriminators for the TS client follow-up after R4 (changes the generated client)

Flag decision: flagged, with two deployed startup switches. The change rewrites the serialisation of every request and response, so its observable semantics change on a hot path, and the repo rule defaults it to flagged. - stjMvcSerializer (services: api; default false): MVC formatters. - stjHubProtocol (services: api; default false): the hub protocol for all hubs.

Both are catalogued in env-mapping.yaml (pnpm run generate:flags) and read once at start-up, from the deployed value, like the fold flag. Runtime overrides do not apply, because formatters and protocols are fixed when the host is built. Rollback is a values flip plus a pod restart through ArgoCD. The flag dependency catalog records realtimeHubV2 requires stjHubProtocol. The web is unaffected and gets no flag. PM (R3) is not flagged: four internal endpoints, covered by contract tests and E2E.

4. Parity strategy

Harness (SyRF.API.Endpoint.Tests/Serialization/, PR-0.1): 1. Enumerate. Response and body types come from ApiExplorer over the real controller set (409 [ProducesResponseType], 30 ActionResult<T>, 80 [FromBody]). Hub types come from INotificationHubClient (NotificationHub.cs:1636-1649) and the hub method signatures (15 methods, all-Guid arguments; returns StudyReviewAccessResult and bool). A cross-check fails if a swagger.json schema maps to no enumerated CLR type. 2. Scan. For the closure of every type, fail with a listed finding on: - an abstract or interface member (B1); - no usable constructor (B6); - a get-only collection (B7); - fields or [JsonProperty] on non-public members (B8); - a Newtonsoft attribute without an STJ twin (J4); - object, JToken or a non-string dictionary key. An allowlist file records each finding as fixed or intentional. 3. Instances. A deterministic builder per type covers every concrete subtype, null and non-null variants, UTC and unspecified DateTime, decimals with scale, whole doubles, dictionaries with tricky keys, and undefined enum integers. 4. Write parity. Serialise each instance with the current Newtonsoft settings (MVC and hub variants), then with the STJ options. Compare semantically: property order ignored, numbers normalised (B14). Store the Newtonsoft output as *.golden.json so later PRs diff against today, not against themselves. 5. Read parity. Real fixtures are bound through both stacks, and the object graphs are compared: - annotation sessions (SessionSubmissionWebDto); - conditional parent options (schema-v0 multi-option); - parent filters; - question upserts; - JSON Patch documents; - malformed variants for each converter error path. Sources: existing test JSON (ProjectControllerTests.cs:267-349, ConfigureJsonOptionsTests.cs), web spec fixtures, and requests captured from a hermetic E2E run. No production or participant data. 6. Shadow compare (PR-1.2, non-production only). When Json:ShadowCompare=true, an output-formatter decorator also serialises each response with Newtonsoft. It logs the route template and the first differing JSON pointer (never values) and increments syrf.json.parity_mismatch{route}. On in E2E and staging; it is never deployed to production. It catches Ok(object) paths that ApiExplorer cannot type.

Intentional differences are listed in docs/architecture/json-serialization-contract.md (new) with the reason, e.g. B14 and error-message paths (B12).

5. Converter rewrites

Newtonsoft STJ design Why not [JsonPolymorphic]
ConditionalTargetParentOptionsConverter JsonConverter<ConditionalTargetParentOptionsDto>: JsonDocument.ParseValue, the same validation and the same messages, then deserialise the element as the concrete type. CanConvert = exact base type only. Write = runtime type The integer discriminator is also a real property (name conflict); custom validation; the spec already has a discriminator via the schema processor
ParentFilterConverter Same pattern; null on missing, non-integer or undefined filterType Unknown discriminators must yield null, not throw
AnnotationConverter Same pattern over string answerType; unknown → StringAnnotationDto; missing → null Fallback semantics
AnnotationAnswerDtoConverter Not ported (D4); kept only if Chris wants RoB AI parity, then as a JsonConverter<AnnotationAnswer> that throws instead of swallowing —

Each converter gets round-trip tests on the real fixtures above, plus one test per error branch asserting the 400 body. ConfigureJsonOptions becomes SyrfJsonSerializerOptions.Apply(JsonSerializerOptions), used by MVC JsonOptions and JsonHubProtocolOptions.PayloadSerializerOptions, so the two cannot drift (realtime 1.3b.7).

6. Rollout mechanism

Option Kill switch Cost Verdict
Startup switch choosing AddNewtonsoftJson + Newtonsoft protocol, or STJ formatters + AddJsonProtocol Yes: values flip + restart Both stacks live for the transition; attributes dual Recommended
Input-formatter fallback (STJ first, Newtonsoft on failure) Partial Body buffering, hides divergence, covers input only, nothing for output or the hub Rejected. The one exception is the documented Microsoft pattern that keeps a Newtonsoft formatter only for JsonPatchDocument until R4
Big-bang with goldens Revert + redeploy only Least code Rejected: no fast rollback for a change that touches every response
Per-request or runtime flag — Formatters and the hub protocol resolver are built once per host Not feasible

7. Common acceptance criteria (every PR)

# Criterion Verification
C1 With both switches off, behaviour is unchanged: the Newtonsoft goldens pass byte-for-byte Parity harness
C2 Every new or changed DTO attribute has its twin (house dual pattern) until R4 Scanner (PR-0.1)
C3 swagger.json regenerated via a Release build and nswag:all; the diff is empty or explained; validate:generated passes from the root PR body; CI
C4 Narrow, niced test runs only (this machine is the CI host); reviews settled on the head; checks green PR body; pr-review-settled.sh
C5 Docs in the same PR: json-serialization-contract.md; the flag decision stated Docs review
C6 No production values touched; production rollout is a separate Chris-approved step cluster-gitops diff
C7 No production or participant data in fixtures or delegate prompts Review

8. Releases → PRs

R0: parity harness and attribute twins (no runtime change)

PR-0.1 Contract scanner and Newtonsoft goldens (J1, J4; effort L). Files: new Serialization/* in SyRF.API.Endpoint.Tests, fixtures folder, docs/architecture/json-serialization-contract.md.

# Criterion Verification
0.1.1 Every ApiExplorer response/body type and every hub message/argument type → has a golden Harness count ≥ the ApiExplorer count, asserted
0.1.2 A swagger.json schema with no enumerated CLR type → the test fails Unit
0.1.3 The scanner over main → reports the B1 properties, the 14 attributes and any B6/B7 types, recorded in the allowlist Unit; allowlist review
0.1.4 ProblemDetails with an extension named status → golden captures today's key order Unit
0.1.5 Golden absolute-hash pins for the screening (BindScreening, each ScreeningDecision) and annotation (BindAnnotation) submission digests → each fixed input hashes to a committed uppercase-hex value. Inputs cover: each of the 8 Annotation subtypes, one with IsDirty true and pending Events; a null Notes/ParentId and a null Answer; a decimal answer with a trailing zero; an OutcomeData with a TimePoint holding a non-integral double and with ExtraElements; each SessionStatus. Annotation inputs use post-binding values (the DTO bound through today's MVC stack and mapped), so R1 binding changes show up here. Merged before any SharedKernel serializer or attribute change (PR-0.2 onwards) Unit (in ProjectStatisticsSubmissionDigestTests or beside it; statistics-session review)
0.1.7 A guard → no code in src sets JsonConvert.DefaultSettings (the digest relies on Newtonsoft defaults: PascalCase, integer enums, nulls included, decimal scale kept); setting it anywhere fails the test Unit (source scan, plus a runtime assert that JsonConvert.DefaultSettings is null in each host's test container)
0.1.6 The scanner allowlist records the four STJ definition-metadata serialisers (§1.4) as default options only, with a golden of each one's output for a fixed definition; a change to their options, or an STJ converter/attribute added to an enum they serialise (QuestionType), fails the test Unit (golden)

PR-0.2 STJ attribute twins (J4; effort S). The 14 attributes in §1.3, plus [JsonConverter(typeof(JsonStringEnumConverter))] on AllocationRegimeOrigin and STJ ignore-null on CurrentProgress.

# Criterion Verification
0.2.1 swagger.json regenerated → no diff C3
0.2.2 Every MassTransit contract in PM.Messages/API.Messages serialised with MassTransit's STJ options → identical JSON, or the diff is listed and the backend-plan owner agrees Contract test (reuses ADR-021 S0.2 if it has landed)
0.2.3 The STJ twins on Entity, AggregateRoot and the hashed enums → the 0.1.5 digest pins and the 0.1.6 definition-metadata goldens pass unchanged; the Newtonsoft [JsonIgnore] on Entity.Events/IsDirty is kept (twins are added, never swapped) Existing pins (0.1.5, 0.1.6)

R1: REST on STJ (flagged)

PR-1.1 Shared options and converters (J3; effort M). SyrfJsonSerializerOptions, the three STJ converters, and tests. Not yet wired.

# Criterion Verification
1.1.1 Each converter fixture (valid and every error branch) → same object graph as Newtonsoft, and a 400 body naming the same problem Unit
1.1.2 A QuestionOptionDto with a BooleanParentFilterDto → parentFilter contains filterType and value Unit (B1)
1.1.3 A dictionary with keys ABC, Species name, 1a and an enum key → same keys as Newtonsoft Unit (B3)
1.1.4 Decimal binding parity. Session-submission fixtures with a DecAnnotationDto and a decimal-array answer, including 0.30000000000000004, a 16+-significant-digit value and a trailing-zero value → the STJ AnnotationConverter binds the same decimal as today. Today JObject.Load under MVC's default FloatParseHandling.Double reads decimals through a double (at most 15 significant digits, so 0.30000000000000004 binds as 0.3m), while STJ's GetDecimal is exact. The STJ converter reproduces the double round-trip, or the difference is a recorded decision with the statistics session (it changes both the stored answer and the digest) Unit (bound object graph compared; the 0.1.5 annotation pins recomputed from the STJ-bound value must match)

PR-1.2 MVC switch and shadow compare (J1, J5; effort M). Files: - SyrfConfigureServices.cs:47-49 (an option to skip AddNewtonsoftJson); - API Program.cs:209 (switch); - Newtonsoft JSON Patch formatter retained for JsonPatchDocument only; - shadow decorator; - env-mapping.yaml flag; - E2E env wiring.

# Criterion Verification
1.2.1 stjMvcSerializer=true → every golden matches (or is allowlisted) when served through TestServer Integration
1.2.6 stjMvcSerializer=true, the decimal fixtures of 1.1.4 posted to the session-submission endpoint → the stored answers and the submission digest equal the switch-off run Integration (Testcontainers; digest compared with the 0.1.5 pins)
1.2.2 JSON Patch on UpdateProject and the stage update → the same result as today with the switch on Integration (ProjectControllerStageSettingsStatisticsTests, ProjectOwnershipEndpointTests)
1.2.3 Shadow mode on, one mismatching response → one log line with route and pointer, no values, and the counter increments Unit
1.2.4 Full hermetic E2E with the switch on and off → same pass set (diff against the known-red baseline) E2E (hermetic e2e/)
1.2.5 Production values → stjMvcSerializer absent or false; shadow absent cluster-gitops grep

PR-1.3 Staging: REST on STJ (effort S; cluster-gitops staging values: switch and shadow on).

# Criterion Verification
1.3.1 3 days (PROPOSAL) with Chris and testers → parity_mismatch = 0, outside allowlisted routes Staging rollout check
1.3.2 Switch flipped back → the API restarts on Newtonsoft with no errors Staging rollout check

R2: both hubs on the STJ protocol (flagged)

PR-2.1 Hub protocol switch (J2; effort M). Files: Program.cs:390-401, env-mapping.yaml, hub goldens. Can be built in parallel with PR-1.2 after PR-1.1.

# Criterion Verification
2.1.1 stjHubProtocol=true → each message's JSON matches its golden: ProjectNotification, ProjectSummariesNotification, DataExportJobNotification, ProjectStatsNotification, StudyPresenceUpdated, ProjectStatisticsChanged, ActivityClaimRevoked, RuntimeFeatureFlagsRevision, UiVersionCheck, and InboxChanged once #3932 merges Unit over JsonHubProtocol frames vs NewtonsoftJsonHubProtocol frames
2.1.2 Each of the 15 hub methods invoked with recorded client frames → binds the same arguments; JoinStudyReview and Heartbeat return the same JSON TestServer hub integration
2.1.3 Hub and MVC serialise the same DTO → identical JSON (shared options) Unit (feeds realtime 1.3b.7)
2.1.4 Hermetic E2E realtime specs (stage-progress-live, session-lifecycle, disconnection-and-save-guard, materialized-statistics-*) with the switch on → pass E2E
2.1.5 Recorded STJ frames committed for the client plan's fixture spec (CR-13) Files in the PR

PR-2.2 Staging: hubs on STJ (effort S). Flip after PR-1.3 has passed.

# Criterion Verification
2.2.1 3 days (PROPOSAL) → no client hub errors in staging logs; live updates work in a two-browser check Staging rollout check

Production rollout of R1 and R2 = a separate Chris-approved step (cluster-gitops production values).

R3: other hosts and non-wire uses

PR-3.1 PM host on STJ (J6; effort S; not flagged). PM Program.cs:476 stops calling AddNewtonsoftJson.

# Criterion Verification
3.1.1 Identity's provisioning request (as ProjectManagementInvestigatorProvisioner sends it) → PM binds it identically Contract test, signed off by the authentication session
3.1.2 e2e-setup calls from e2e/helpers/project-setup.ts → unchanged E2E

PR-3.2 Non-wire cleanup (J7; effort M): - byte budgets measured with STJ (statistics-session review), at all five sites: the three Newtonsoft sites and the two STJ sites in ProjectStatisticsBundleReader.cs:314-315; - email template data via STJ, with a golden per template; - delete FileStreamingHelper.StreamFile and, if D4, AnnotationAnswerDtoConverter; - translate the 13 test files where their subject changed.

# Criterion Verification
3.2.1 Each email template's data JSON → identical to the Newtonsoft output Golden
3.2.2 Budget checks at their thresholds → same accept/reject outcome on the existing tests Unit
3.2.3 Each of the five budget sites (including ProjectStatisticsBundleReader.cs:314-315) → a boundary test just under and just over its threshold gives the same accept/fallback decision before and after Unit (statistics-session review)

R4: remove Newtonsoft (after production has run on STJ for a Chris-approved period)

PR-4.1 (effort M): - remove both switches, ConfigureJsonOptions, the Newtonsoft converters, AddNewtonsoftJson and AddNewtonsoftJsonProtocol; - port JSON Patch to Microsoft.AspNetCore.JsonPatch.SystemTextJson, and update JsonPatchOperationProcessor; - drop the Newtonsoft twins of attributes, except the [JsonIgnore] on Entity.Events/IsDirty while D6 hashes whole Annotation objects (4.1.5); - remove the packages listed in §1.5, except where D6 keeps them: the hashed types' Newtonsoft attributes need a Newtonsoft.Json reference in the assembly that declares them (Entity is in SharedKernel), so either SharedKernel keeps the package for those attributes (allowlisted), or the digest first moves to explicit projections of the hashed fields (as OutcomeData already does), agreed and versioned with the statistics session; - add a guard test.

# Criterion Verification
4.1.1 grep 'Newtonsoft' src --include='*.cs' → only allowlisted files (digest if D6, the hashed types' attributes if D6 keeps them, Quartz) Guard test
4.1.2 JSON Patch paths (/name, nested stage settings, active-reviewer tracking detection at ProjectController.cs:965) → same results Integration
4.1.3 swagger.json regenerated → diff empty or explained; TS client unchanged C3
4.1.4 Flags removed from env-mapping.yaml; generator output updated Generator diff
4.1.5 After the package removal → the Newtonsoft [JsonIgnore] on Entity.Events/IsDirty still exists and is still honoured by the digest, no converter attribute has been added to ScreeningDecision or SessionStatus (still hashed as integers), and JsonConvert.DefaultSettings is still unset (0.1.7); or the digest hashes explicit projections of Annotation. Either way the 0.1.5 absolute-hash pins pass unchanged (ExtraElements included) Guard test + 0.1.5 and 0.1.7 tests
4.1.6 The four STJ definition-metadata serialisers → still on default options; 0.1.6 goldens unchanged 0.1.6 goldens

9. Order and critical path

flowchart LR
  P01[PR-0.1 harness] --> P02[PR-0.2 twins] --> P11[PR-1.1 options+converters]
  P11 --> P12[PR-1.2 MVC switch] --> P13[PR-1.3 staging REST]
  P11 --> P21[PR-2.1 hub switch] --> P22[PR-2.2 staging hubs]
  P13 --> P22
  P21 --> R13b[[realtime R1.3b merges, dark]]
  P22 --> RT21[[realtime R2.1/R2.2]]
  P22 --> PROD{{Chris: production}} --> P31[PR-3.1 PM] & P32[PR-3.2 cleanup] --> P41[PR-4.1 removal]
  • Critical path into realtime R1.3b: PR-0.1 → PR-0.2 → PR-1.1 → PR-2.1. R1.3b can merge once PR-2.1 exists, because realtimeHubV2 requires stjHubProtocol. Its staging dark run needs PR-2.2.
  • Parallel: PR-1.2 ∥ PR-2.1 (they share only Program.cs, in different regions). PR-3.1 can start any time after PR-1.1, but merges after the production step. PR-3.2 is independent of R2.
  • Sequential: staging flips go REST first (PR-1.3), then hubs (PR-2.2), per Chris's order.

10. Web client impact

  • Generated client: expected unchanged (NSwag already models STJ; §1.6). Any diff rides in the PR that causes it.
  • Hand-written shape dependencies:
  • polymorphic fields read directly: filterType (3 files), conditionType (7), answerType (14), parentFilter (9), targetParentOptions (7);
  • 8 hand-written HttpClient call sites, which PR-0.1 audits for enum names, string numbers and date formats (B5, B11);
  • hub handlers in signal-r.service.ts.

B1 parity is what protects these. - Acceptance: the hermetic E2E suite with both switches on and off (1.2.4, 2.1.4). Relevant specs include schema-v0-multi-option-conditional-parent-answers, annotation*, screening*, session-lifecycle, stage-progress-live and materialized-statistics-*. E2E never runs against staging.

11. Risks

Risk Mitigation
Subtype data silently dropped (B1) in responses or hub payloads Scanner fails on abstract members; 1.1.2; shadow compare in E2E and staging
Inbound leniency loss turns accepted requests into 400s (B5, B11) Real fixtures; audit of hand-written callers; staging shadow; switch rollback
Duplicate status in ProblemDetails flips meaning for the statistics admin route 0.1.4 golden; statistics session told; fix the duplicate key as a follow-up (rename the extension)
Digest hashes change and break FEAT-024 replay idempotency (directly, because a Newtonsoft attribute on Entity or on ScreeningDecision/SessionStatus is dropped or swapped for an STJ twin, or because someone sets JsonConvert.DefaultSettings; ProjectStatisticsSubmissionDigestTests would stay green because it only compares relatively) D6; absolute-hash pins in PR-0.1 before any SharedKernel change (0.1.5); DefaultSettings guard (0.1.7); twins added, never swapped (0.2.3); package-removal guard (4.1.5)
R1 request binding changes hashed values: today AnnotationConverter reads decimals through a double (15 significant digits), STJ reads them exactly, so a decimal answer such as 0.30000000000000004 is stored and hashed differently after the switch, and a retry across the switch is classified as conflicting Decimal binding-parity fixtures (1.1.4) and the endpoint check (1.2.6); post-binding digest pins (0.1.5); any intended change is a recorded decision with the statistics session
A PR moves the STJ definition-metadata serialisers onto the shared options, or adds an STJ converter to QuestionType, so new checkpoint observations get different bytes Default-options allowlist entry with goldens (0.1.6, 4.1.6)
Byte-budget accept/fallback decisions shift at the two STJ sites in ProjectStatisticsBundleReader Covered with the Newtonsoft sites in PR-3.2 (3.2.3)
MassTransit JSON changes from new STJ twins 0.2.2 contract test
Generated client drift, or checksum conflicts with the ~20 open PRs that touch swagger.json No polymorphic attributes before R4; Release build first; rebase discipline
Two stacks maintained during the transition Short window: R4 after a Chris-approved soak (D8)
Shadow compare leaks data into logs Pointers and route templates only; unit-tested (1.2.3); never in production

12. Coordination

Stream / PR Overlap Sequencing
Realtime server plan (R1.3b, 1.3b.7) Hub protocol, shared options This plan's PR-2.1 gates R1.3b; PR-2.1.3 is the shared-options proof
Realtime client plan (CR-13) Recorded frames PR-2.1.5 supplies them
Notifications stack #3932 (hub + InboxChanged), #3938, #3941–#3945, #3947, #3965 API Program.cs, swagger.json, the generated client PR-2.1 rebases on #3932 and adds the InboxChanged golden; new DTOs use dual attributes (C2)
FEAT-024 (statistics session); #4012, #4018 (adds a Newtonsoft camelCase test helper) Digest, budgets, ProjectStatisticsChanged, the admin ProblemDetails D6; their review of PR-3.2, 0.1.4–0.1.7, the decimal binding parity (1.1.4, 1.2.6) and the PR-4.1 digest guard; #4018's test helper is translated in PR-3.2
Review eligibility #3746 (paused) Adds 5 Newtonsoft StringEnumConverter attributes (ReviewEligibilityDto.cs, SignalR/Dtos/AnnotationClaimReleaseDto.cs) Ask for STJ twins when it resumes; the scanner catches them otherwise
Authentication migration session PM provisioning endpoint (Identity → PM); SharedKernel package removal affects the Identity build 3.1.1 sign-off; R4 package removal announced
Backend bug plan PR-11 (C15) ProjectAgreementThreshold, ValueObject STJ constructors PR-0.2 lands after PR-11 or rebases on it
Surface-area plan (AutoMapper → Mapperly, dead code) DTO classes; FileStreamingHelper Mapperly changes mapping, not JSON shape, but goldens catch any shape change; dead-code deletion can happen in either plan
Build plan PR-17 (central package management) Package removal in R4 R4 edits Directory.Packages.props if PR-17 has landed
#2469, #2224, #3966 Program.cs, ProjectController.cs, NotificationHub.cs Re-check gh pr list --state open --json number,title,files before each PR

(Open-PR scan: 133 open PRs on 4 Oct 2026; 5 add Newtonsoft lines in their diffs: #4018, #3746, #3741 (docs only), #2469 (tests), and this planning PR #3961.)

13. Decisions needed from Chris

  • D1. Rollout mechanism. Two deployed startup switches (stjMvcSerializer, stjHubProtocol) with shadow compare in non-production. Recommended. Alternative: one combined switch (simpler, but REST and hub cannot be rolled back separately).
  • D2. Converters over [JsonPolymorphic] until R4, so the generated client does not change. Polymorphic attributes and OpenAPI discriminators come later as a client-typing improvement. Recommended.
  • D3. Parity bar. Byte-identical JSON after normalising numbers and property order, with a listed allowlist. Error-message text may differ only in STJ's appended path. Recommended.
  • D4. Drop AnnotationAnswerDtoConverter (no endpoint binds it; RoB AI suspended). Recommended.
  • D5. JSON Patch. Keep a Newtonsoft formatter for JsonPatchDocument only until R4, then port to Microsoft.AspNetCore.JsonPatch.SystemTextJson. Recommended.
  • D6. FEAT-024 digest. Keep it on Newtonsoft (recommended for now), or have the statistics session version it to STJ with dual-hash acceptance. It is their call. Keeping it on Newtonsoft is necessary but not sufficient (statistics session review, 2026-10-05). Whichever option is chosen, the plan adds:
  • golden absolute-hash pin tests for the screening and annotation submission digests, merged before any SharedKernel serializer or attribute change (PR-0.1, 0.1.5), covering the hashed enums ScreeningDecision and SessionStatus and using post-binding annotation values;
  • a guard that the Newtonsoft [JsonIgnore] on Entity.Events/IsDirty and the Newtonsoft attributes on the hashed enums survive the package removal, or else explicit projections of the hashed fields replace them (0.2.3, 4.1.5). Only Annotation is hashed whole; OutcomeData is already a projection (only its TimePoints and ExtraElements go through the serializer). ExtraElements is hashed, so the projection route must decide it explicitly;
  • an allowlist entry keeping the four STJ definition-metadata serialisers on default options, with a golden (0.1.6, 4.1.6);
  • coverage of the two STJ byte-budget sites in ProjectStatisticsBundleReader.cs:314-315 alongside the three Newtonsoft sites (3.2.3);
  • a guard that nothing sets JsonConvert.DefaultSettings (0.1.7), because the digest relies on Newtonsoft defaults (PascalCase, integer enums, nulls included, decimal scale kept);
  • decimal binding parity in R1 (1.1.4, 1.2.6): the digest hashes the bound Annotation, and today's converter reads decimals through a double while STJ reads them exactly.
  • D7. PM host unflagged (internal endpoints, contract test, E2E). Recommended.
  • D8. PROPOSAL values:
  • staging soaks of 3 days each for REST and hubs;
  • the R4 gate: 4 weeks in production on STJ with no rollback.
  • D9. Quartz job store stays on Newtonsoft; a follow-up issue plans a migration. Recommended.
  • D10. The duplicate status extension on the statistics admin 409: keep it byte-identical in the migration (recommended), and rename it in a separate statistics-owned PR.

14. Not verified

  • NSwag's mechanism for choosing STJ schema settings, and NJsonSchema honouring Newtonsoft attributes by name. Inferred from the code comment, tests and swagger.json output; not read in NSwag source.
  • Microsoft.AspNetCore.JsonPatch.SystemTextJson behaviour parity (path casing, ApplyTo errors). Package existence comes from the aspnetcore docs; it is not in the local cache.
  • STJ and Newtonsoft edge behaviours stated from knowledge, not tests:
  • Guid formats;
  • enum dictionary keys under DictionaryKeyPolicy;
  • MVC max depth 32 in both stacks;
  • the duplicate-key write order in both ProblemDetails converters. PR-0.1/PR-1.1 tests settle them.
  • PM's Newtonsoft settings are assumed to be ASP.NET defaults (camelCase, dictionary keys not processed); not read from the framework source.
  • Whether any MassTransit contract carries AggregateRoot/Entity/ProjectAgreementThreshold (0.2.2 settles it).
  • gh pr list --json files caps at 100 files per PR, so very large PRs (#2469, #3288) may touch more files than the scan saw.