Build hygiene and CI simplification plan (October 2026)¶
This plan covers three issues: .NET build hygiene (#3982), one service registry generated from the csproj graph (#3983), and workflow simplification (#3990). It also covers the delivery risks that remain from Appendix F of architecture-review-2026-10.md. It is planning only; nothing starts until Chris approves and answers the decisions in §9.
Every finding was re-checked against main@85e6facf7 (3 Oct 2026) by reading the code, plus a
live GitHub API read where stated. Status codes: VERIFIED, CORRECTED (the appendix was
wrong; the row says how), NOT VERIFIED (the row says why).
The work done today is excluded except as context: - #3971: release gate and production gate. - #3993: one rolling draft production PR. - cluster-gitops CODEOWNERS and code-owner review. - The superseded production PRs, now closed.
Test Summary already fails closed (pr-tests.yml:1979-1990).
1. Scope¶
1a. Delivery risks (Appendix F, re-verified)¶
| ID | Problem | Impact | Evidence | Status | Issue |
|---|---|---|---|---|---|
| F1 | Promotion copies each staging config.yaml wholesale into production, so deploymentNotification.deploymentId is copied too |
The production PostSync hook reports success against a staging GitHub Deployment | ci-cd.yml:2939; _postsync-notify.tpl:97. Production api/config.yaml:17 holds 5760286720, which the API reports as environment staging |
VERIFIED | new |
| F1b | Production snapshots cluster-gitops right after the staging PR's auto-merge is requested, with no wait | The rolling production PR proposes the previous staging state, one push behind | ci-cd.yml:2819 (gh pr merge --auto), then checkout at :2896 |
VERIFIED. "Runs when nothing was promoted" is CORRECTED: #3993's prod-diff step (:2977) handles it |
new |
| F2 | create-lambda-release never succeeds |
No s3-notifier GitHub releases | It uploads /tmp/released.zip (ci-cd.yml:1671) but attaches lambda/production.zip (:2273). It needs: create-tags without always(). API: 0 s3-notifier and 0 pdf-agent releases among the latest 200 |
VERIFIED | new |
| F3 | retag-images can pick a prerelease tag |
It could promote a PR or branch build as a release | ci-cd.yml:1385 git describe --match "${TAG_PREFIX}*" with no --exclude. 23 prerelease tags are reachable from main (e.g. api-v9.60.0-fix-preview-manual-dispatch-rebuild.1). Latent today because the nearest tags are stable |
VERIFIED (mechanism) | new |
| F4 | A failed version job still builds images |
0.0.0 images get pushed and latest moves |
build-docker allows a failed version job (ci-cd.yml:1450); the fallback is '0.0.0' (:1480); _docker-build.yml:139 rejects only an empty version |
VERIFIED | new |
| F5 | The preview database lock fails open | A "locked" preview database is dropped | _preview-cleanup.yml:90: gh pr view … 2>/dev/null \|\| echo "". The callers grant no pull-requests: read (pr-preview.yml:3863-3867, preview-expiry.yml:176-180). pr-preview.yml:2093-2098 carries on with empty labels. Substring grep -q (:2110). The lock-db label does not exist (API label list) |
VERIFIED (code and permissions; not exercised live) | new |
| F6 | Script injection: ${{ }} interpolated into shell |
Shell injection by anyone with write or dispatch rights | docs-rebuild.yml:27,32-34,164,226-227 (client_payload); snapshot-on-demand.yml:96,168 (inputs.reason, GKE credentials); e2e-tests.yml:138,160 (labels, under pull_request_target) |
VERIFIED | new |
| F7 | No pipefail. No workflow sets defaults.run.shell, so steps run as bash -e |
Failed pipelines pass silently | ci-cd has pipefail in 1 of 46 multi-line run blocks; _preview-cleanup 0 of 20; preview-sweep 0 of 16. The guard at preview-sweep.yml:61-67 takes sort's exit status, so a gh failure gives an empty "open" set |
VERIFIED | new |
| F8 | CI strips ArgoCD finalizers | GitOps bypass; managed resources can be orphaned | pr-preview.yml:2011, _preview-cleanup.yml:555,592, preview-sweep.yml:372. CORRECTED scope: it only touches Applications that already have a deletionTimestamp, plus the hook finalizers on preview Jobs. Preview only |
VERIFIED | new |
| F9 | Static AWS keys | Long-lived credentials in self-hosted PR builds | ci-cd.yml:1603, pr-preview.yml:1405 (package-lambda, a self-hosted route in validate-workflows.sh:928+), _preview-cleanup.yml:433. ci-cd.yml has no id-token permission |
VERIFIED | new |
| F10 | The Claude review agent may run any gh api call while holding a PR-write token and reading PR content |
Prompt injection could post an approval or a write | claude-code-review.yml:269 (Bash(gh api:*)), :23 (pull-requests: write) |
VERIFIED (allowance); exploit SPECULATIVE | new |
| F11 | promote-production.yml is stale |
Misleading; dispatchable | Legacy paths environments/staging/services/*.yaml (:88-95), yq latest (:67). Zero runs ever (API) |
VERIFIED | new |
| F12 | GitVersion bump regexes accept only [a-z]+ scopes |
feat(bulk-update): bumps patch, not minor |
src/services/api/GitVersion.yml:5-8; 9 configs, all different. 55 of 455 feat commits since 1 Aug have hyphenated scopes. Whether the ^BREAKING CHANGE: footer matches is NOT VERIFIED (depends on GitVersion's regex options) |
VERIFIED | new |
| F13 | Promotion branches leak in cluster-gitops | 1,353 branches | 657 staging-promotion-*, 605 production-promotion* (604 legacy plus the rolling branch). delete_branch_on_merge=false. Staging still leaks one branch per run (ci-cd.yml:2808,2819, no --delete-branch). CORRECTED: the appendix said 600+ production branches only |
VERIFIED (API) | new |
1b. Build hygiene (#3982)¶
| ID | Problem | Evidence | Status |
|---|---|---|---|
| B1 | No central package management; 12 of 117 packages drift | No Directory.Packages.props; drift in xunit, its runner, Test.Sdk, Moq, Testcontainers (3.x/4.x), coverlet, Serilog.Settings.Configuration, morelinq, Http.Abstractions, Mvc.Testing, TestHost |
VERIFIED (scan of 38 tracked csproj) |
| B2 | No global.json; setup-dotnet floats 10.0.x, and some lanes also install .NET 8 |
No root global.json; 14 setup-dotnet uses |
VERIFIED |
| B3 | Vulnerable packages are buried among warnings | 52 unique NU1903 warnings for SSH.NET 2023.0.0 and 2024.2.0 in test projects (main run 36276639638, 26 Sep) | VERIFIED. Which project pulls in SSH.NET is NOT VERIFIED (probably Testcontainers 3.x) |
| B4 | syrf.sln, not .slnx |
96 references across CI, scripts, 7 .slnf files and docs |
VERIFIED |
| B5 | No Dependabot or Renovate configuration | No .github/dependabot.yml and no renovate.json; 170 open Dependabot alerts (review §Coverage) |
VERIFIED (files); alert count not re-read |
| B6 | Warning volume with TreatWarningsAsErrors=false |
8,930 warnings in that main build. Unique counts: CA2007 7,811 (almost all in src/libs/project-management), CS8625 193, CS0618 141, CS1573 129. CA2007 is suppressed only under src/services/** (.editorconfig:246-250) |
CORRECTED: 8,930, not 14,220 (the count varies with the build scope) |
| B7 | Many analyzers muted | 45 severity = none and 2 suggestion in .editorconfig, including every culture rule (:55-60), CA2211, CA1031 |
CORRECTED: 45 muted, not about 69 |
| B8 | The NSwag tool is 14.0.8 while NSwag.AspNetCore is 14.2.0 |
.config/dotnet-tools.json |
VERIFIED |
1c. Service registry (#3983)¶
| ID | Problem | Evidence | Status |
|---|---|---|---|
| G1 | Library changes never rebuild the Lambda | The s3-notifier csproj references kernel, webhostconfig and PM.Messages (→ PM.Core → AppServices) (S3FileSavedNotifier.Endpoint.csproj:28-31), but detect-service-changes.sh:75 watches only src/services/s3-notifier, src/libs/s3-notifier, PM.Messages and Directory.Build.props |
VERIFIED |
| G2 | PM misses two libraries | PM.Endpoint.csproj:35-36 references libs/api and libs/s3-notifier; detect-service-changes.sh:69 omits both |
VERIFIED |
| G3 | A fifth drifting source: the pr-tests.yml regexes |
The API regex omits libs/project-management, which the API references; the PM regex (:130) omits libs/api and libs/s3-notifier; the .NET regex (:87) omits Directory.Build.props, .editorconfig and src/testing/ |
VERIFIED (new) |
| G4 | dependency-map.yaml has drifted from the csproj graph |
PM.Messages → PM.Core is missing; S3FileSavedNotifier.Messages says depends_on: [] but references SharedKernel. generate-dockerfiles.py --validate passes (the Dockerfiles match the map), but CI never runs it. The s3-notifier Dockerfile has EXPOSE None, and nothing builds it |
VERIFIED |
| G5 | services.json is not dead, but it isn't a trigger source either |
Its only consumers are validate-workflows.sh:47,1198-1238 (JSON shape) and test-pdf-agent-arrnc-runtime-contract.sh:6; CLAUDE.md:54,73 cites it |
CORRECTED |
| G6 | A sixth source: hard-coded build and retag matrices | ci-cd.yml builds docker_matrix/retag_matrix inline (:500-661) |
VERIFIED (new) |
| G7 | Dead reusable workflow | _detect-changes.yml has no caller (only README mentions it); _gitversion.yml is used only by test-ci-cd.yml:151 |
VERIFIED |
1d. Workflow size (#3990)¶
| ID | Problem | Evidence | Status |
|---|---|---|---|
| W1 | Logic lives in YAML | 29 workflows, 19,553 lines; 10,546 lines of inline run: in 463 steps; 1 composite action (update-preview-pr, 10 uses) |
VERIFIED (parsed with PyYAML) |
| W2 | pr-preview.yml is 4,543 lines (2,613 of them inline shell); ci-cd.yml is 3,552 (2,285) |
wc -l |
VERIFIED |
| W3 | Check noise | 98–129 check runs per PR head, of which 83–97 were skipped (#3964, #3993, #3996) | CORRECTED upward from 84–105 |
| W4 | Refactors cost more because contract scripts pin step bodies by SHA-256 | test-workflow-validator-consolidation.sh:20-22,84,97 |
VERIFIED |
2. MVP boundary, out of scope, flags¶
MVP (R0 + R1-PR-8 + R2-PR-12/13):
- releases never publish 0.0.0, prerelease or wrong-environment metadata;
- the preview database lock fails closed;
- injection sinks are closed;
- destructive preview jobs use pipefail;
- library changes rebuild the Lambda and PM.
Everything else is iteration.
Out of scope (tracked elsewhere):
| Item | Where |
|---|---|
Governance: self-approve, the default write token, workflow-level permissions: {}, the camarades-argo-cd ruleset bypass |
Owner decisions under V10; #3971 follow-ups |
Image hardening: provenance/SBOM, chiseled images, nginx as root, pod securityContext, per-image buildx cache scope (_docker-build.yml:347-348) |
New issue on approval |
SharedKernel's Http.Abstractions 2.2, MassTransit v9, AutoMapper licence |
Strategic ADRs (synthesis §5) |
msbuild.binlog, docs/planning clutter, docs-backlog-sync.yml |
Repo-hygiene issue |
| Lock files and NuGet caching (#3161 is a stale draft) | D7 |
| Web lint and strict typecheck in CI | #3981 (frontend plan PR-2) |
Flag decision: not flagged. These are CI and build changes with no runtime feature surface. The CPM lift (PR-17) is version-neutral by construction. Production package upgrades are not in this plan; only test packages change (PR-18). Rollback for every PR is a single revert. Production rollout of anything here is a separate Chris-approved step: the rolling draft production PR is never readied, approved or merged by an agent.
3. Common acceptance criteria (every PR)¶
| # | Criterion | Verification |
|---|---|---|
| C1 | .github/scripts/validate-workflows.sh passes, as do the contract scripts for every touched workflow and actionlint -shellcheck= |
Output in the PR body |
| C2 | Behavioural workflow change → a contract test (or fixture) is written first, fails on main and passes on the head |
Red and green output in the PR body |
| C3 | No runs-on change unless the PR says so. A routing change → the PR triggers the changed lane, and the live run is linked |
Run URL |
| C4 | No new runner groups or labels; timeouts stay pinned exactly; ordinary lanes keep checkout first and juniper-job-cleanup.sh |
Routing contracts |
| C5 | Nothing touches production. The production PR stays a draft; production rollout is a separate Chris-approved step | PR review |
| C6 | Docs updated: .github/workflows/README.md, docs/how-to/juniper-runner-routing.md, plus CLAUDE.md where it cites the file |
PR review; validate-docs --skip-indexes |
| C7 | pr-review-settled.sh exits 0 on the head; Test Summary green |
Gate output |
| C8 | Local commands are narrow and niced (single script or single test project). The full solution is never built on this CI host | PR body lists the commands |
4. Releases and pull requests¶
Numeric values marked PROPOSAL need Chris's confirmation.
R0: release integrity (PR-1 → PR-3 in sequence, because all three touch ci-cd.yml)¶
PR-1: version and retag integrity (F3, F4; effort S)
Files: ci-cd.yml (build-docker condition, version fallback, retag-images); a new .github/scripts/test-main-release-version-integrity.sh; validate-workflows.sh registration.
| # | Acceptance criterion | Verification |
|---|---|---|
| 1.1 | version fails or is skipped → build-docker, retag-images, create-tags and the promotions are skipped or fail; no image is pushed |
Contract script over the needs/if graph |
| 1.2 | The workflow has no '0.0.0' literal; an empty version fails the build input check |
Contract script; _docker-build.yml:139 stays |
| 1.3 | Previous tag lookup → excludes any tag containing - after the prefix (--exclude "${TAG_PREFIX}*-*") |
Fixture repo in the script with a nearer prerelease tag → the stable tag is chosen |
| 1.4 | First main run after merge → the retag/build jobs behave as before on a normal push | Run URL |
PR-2: promote only release fields, from merged staging (F1, F1b; effort M)
Files: ci-cd.yml (promote-to-production copy step); a new .github/scripts/copy-staging-promotion-to-production.sh and its test. Reuse the design from stale #2920, which has a script of this name, rather than its branch.
Approach:
1. Merge only an allowlist into production: service.chartTag, service.imageTag, gitVersion.* and deploymentNotification.commitSha.
2. Keep production's own deploymentNotification.deploymentId (or clear it; D2) and every other production key.
3. Keep the s3-notifier lambda.code rule.
4. Before snapshotting, wait for this run's staging PR to reach MERGED, with a 10-minute timeout (PROPOSAL); otherwise skip the production update with a notice.
| # | Acceptance criterion | Verification |
|---|---|---|
| 2.1 | Staging config with a deploymentId and extra keys → production keeps its own deploymentId and non-allowlisted keys; only allowlisted fields change |
Script unit fixtures |
| 2.2 | The staging PR has not merged within the timeout → the production PR is not updated, and the run summary says why | Script test with stubbed gh |
| 2.3 | Next main run → the rolling draft PR diff contains only tag and version lines | Chris reads the draft diff (it is not merged) |
| 2.4 | Existing test-main-production-promotion-pr.sh still passes |
Script output |
PR-3: Lambda release and staging branch cleanup (F2, F13 part; effort S)
Files: ci-cd.yml (create-lambda-release, staging gh pr merge). Per D1: either fix the job or delete it. Recommended: fix it.
| # | Acceptance criterion | Verification |
|---|---|---|
| 3.1 | package-lambda succeeds and create-tags succeeds → the release s3-notifier-v<ver> exists with released.zip attached |
Contract script on names and paths; first main run with an s3-notifier change (staging only) |
| 3.2 | create-tags skipped or failed → no release, and no job failure caused by a missing asset |
Contract script (if: uses explicit results) |
| 3.3 | A staging promotion PR merges → its branch is deleted (--delete-branch) |
Next main run; cluster-gitops branch count does not grow |
Ops-1 (not a syrf PR): cluster-gitops branch purge (F13). A script lists staging-promotion-* and production-promotion-<run_id> branches whose PRs are merged or closed. It runs dry first, and deletes only after Chris approves. The rolling production-promotion branch is kept. Optionally enable delete_branch_on_merge (D3). Verification: the branch count before and after is posted on the tracking issue.
R0b: preview and workflow safety (parallel; disjoint files)¶
PR-4: preview database lock fails closed (F5; effort M)
Files: _preview-cleanup.yml, pr-preview.yml (label-check step), preview-expiry.yml, plus a test script. Approach: grant pull-requests: read; treat a failed label read as "locked" (skip the drop, warn); match labels exactly; create the lock-db label (D4).
| # | Acceptance criterion | Verification |
|---|---|---|
| 4.1 | Label API fails → the database is preserved, the step warns, and the cleanup continues for the other resources | Script test with stubbed gh returning non-zero |
| 4.2 | Label not-lock-db present → treated as unlocked (exact match) |
Script test |
| 4.3 | lock-db present on a test PR, then the PR is closed → the database survives |
Live: a throwaway preview PR on staging infrastructure, then manual cleanup |
| 4.4 | Preview entry-gate and routing contracts pass | test-preview-entry-gates-runner-routing.sh |
PR-5: close the script-injection sinks (F6; effort S)
Files: docs-rebuild.yml, snapshot-on-demand.yml, e2e-tests.yml, validate-workflows.sh. Move each value to env: and quote it. Add a validator rule that rejects ${{ github.event.(client_payload|inputs|label|pull_request.labels|comment|issue).* }} inside run:.
| # | Acceptance criterion | Verification |
|---|---|---|
| 5.1 | Any workflow interpolating those contexts in run: → validate-workflows.sh fails |
Fixture in the validator test |
| 5.2 | A dispatch with reason='$(id)' → the value is printed literally |
Live snapshot-on-demand dry dispatch |
| 5.3 | E2E workflow changed → the contract-required live PR smoke and full main run are green | Run URLs |
PR-6: delete dead workflows (F11, G7; effort S)
Files: delete promote-production.yml and _detect-changes.yml; update .github/workflows/README.md:522.
| # | Acceptance criterion | Verification |
|---|---|---|
| 6.1 | The repository → no workflow references the deleted files, and the API lists them as deleted after merge | grep; gh api …/actions/workflows |
| 6.2 | validate-workflows.sh and every routing contract pass |
Script output |
PR-7: GitVersion bump regexes (F12; effort S)
Files: the 9 GitVersion.yml files, plus a test that all 9 share the identical bump section. Scopes become [a-z0-9-]+; the footer pattern becomes (?m)^BREAKING CHANGE:.
| # | Acceptance criterion | Verification |
|---|---|---|
| 7.1 | feat(bulk-update): x → minor; fix(api-v2): x → patch; feat(x)!: y → major |
Run GitVersion locally against a scratch repo for one service |
| 7.2 | Body footer BREAKING CHANGE: … → major |
Same harness (also settles the NOT VERIFIED footer claim) |
| 7.3 | The 9 files' bump sections are identical | New check in validate-workflows.sh |
R1: hardening¶
PR-8: pipefail by default, as a ratchet (F7; effort M)
Files: _preview-cleanup.yml and preview-sweep.yml first (defaults: run: shell: bash, which GitHub runs as bash -eo pipefail); fix the sweep guard; add a validate-workflows.sh rule that every workflow declares the default, with an allowlist that shrinks. ci-cd.yml and pr-preview.yml follow in separate PRs after a SIGPIPE audit.
| # | Acceptance criterion | Verification |
|---|---|---|
| 8.1 | gh pr list fails in the sweep → the job fails before deleting anything |
Script test of the extracted step |
| 8.2 | A workflow not on the allowlist without defaults.run.shell → the validator fails |
Validator fixture |
| 8.3 | Each echo … \| grep -q / head pipeline in the converted files is checked for SIGPIPE under pipefail |
Audit list in the PR body |
| 8.4 | A live manual sweep dispatch with dry_run=true → green |
Run URL |
PR-9: AWS through OIDC (F9; effort L; cross-repo)
Files:
- camarades-infrastructure: an IAM OIDC provider plus three roles scoped by sub:
- main ci-cd → put lambda-packages/*;
- preview → its preview prefix;
- cleanup → delete its preview prefix.
- ci-cd.yml, pr-preview.yml, _preview-cleanup.yml: switch to role-to-assume and add id-token: write.
- Delete the secrets after one green run of each lane.
D5 covers moving the packages out of camarades-terraform-state-aws.
| # | Acceptance criterion | Verification |
|---|---|---|
| 9.1 | Each of the three jobs authenticates with no AWS_ACCESS_KEY_ID secret |
Live runs (a preview PR with a Lambda change; main; a cleanup) |
| 9.2 | The preview role cannot write the main production.zip key |
aws s3api put-object denied in a test step, or IAM policy simulator output |
| 9.3 | After cutover, the repo has no AWS_ACCESS_KEY_ID secret |
gh secret list |
PR-10: read-only tools for the Claude review agent (F10; effort S)
Files:
- claude-code-review.yml: replace Bash(gh api:*) with Bash(.github/scripts/claude-review-read.sh:*).
- The new wrapper script. It allows GET only, and only on repos/<repo>/contents/* and compare/*.
- A test for the wrapper.
The workflow is ruleset-locked and comment-triggered from the default branch, so live proof comes after merge.
| # | Acceptance criterion | Verification |
|---|---|---|
| 10.1 | Wrapper called with -X POST, --method, -f or another path → exits non-zero without calling gh |
Script test |
| 10.2 | After merge, /claude-review on a test PR → the review completes, and its inline comments and summary are posted |
Run URL |
PR-11: finalizer policy (F8; effort S). The recommended option (D6): keep the break-glass, but only for Applications with a deletionTimestamp older than 10 minutes (PROPOSAL). Log every removal to the step summary, and document it as a preview-only exception in the GitOps rule.
| # | Acceptance criterion | Verification |
|---|---|---|
| 11.1 | An Application deleting for less than 10 min → untouched | Script test of the extracted function |
| 11.2 | The docs name the exception and its scope | docs/how-to/ page review |
R2: one service registry (#3983)¶
PR-12: generate the registry, check only (G1–G6; effort M)
Files:
- tools/service-registry/generate.py: Python, as generate-dockerfiles.py already is. It parses ProjectReference closures from each service's entry csproj.
- Hand-authored metadata: name, entry csproj, image, tag prefix, chart paths, kind, port, runtime packages. It stays in docs/architecture/dependency-map.yaml under services:, with its depends_on lists removed.
- The generated .github/service-registry.json (committed), with per-service source directories and global triggers (Directory.Build.props, .editorconfig, later Directory.Packages.props and global.json).
- A --check mode in validate-workflows.sh.
| # | Acceptance criterion | Verification |
|---|---|---|
| 12.1 | s3-notifier's closure → includes kernel, webhostconfig, appservices, PM.Core, PM.Messages and the s3-notifier libs | Generator unit test |
| 12.2 | PM's closure → includes libs/api and libs/s3-notifier |
Generator unit test |
| 12.3 | A csproj reference is added without regenerating → validate-workflows.sh fails |
Fixture |
| 12.4 | No consumer behaviour changes in this PR | Diff touches no workflow logic |
PR-13: change detection reads the registry (G1, G2; effort M). Files: detect-service-changes.sh (SOURCE_PATHS from the registry via jq; chart paths unchanged) and test-detect-service-changes.sh.
| # | Acceptance criterion | Verification |
|---|---|---|
| 13.1 | A change only under src/libs/kernel → s3-notifier action is build |
Script test |
| 13.2 | A change only under src/libs/api → project-management action is build |
Script test |
| 13.3 | Every service's previous SOURCE_PATHS ⊆ the new set (no lost triggers) |
Script test comparing against a frozen copy |
| 13.4 | A preview PR touching src/libs/kernel → the preview builds the Lambda |
Live preview run |
PR-14: pr-tests.yml detection reads the registry (G3; effort M). It replaces the per-service regexes at pr-tests.yml:87-150 with a registry lookup, and adds the global triggers. Because it edits pr-tests.yml, every lane runs, which satisfies C3.
| # | Acceptance criterion | Verification |
|---|---|---|
| 14.1 | A change only to Directory.Build.props or .editorconfig → dotnet=true |
test-pr-change-detection.sh fixture |
| 14.2 | A change only under src/libs/project-management → api=true |
Fixture |
| 14.3 | The routing contracts for the dotnet, Sonar and validation-gates lanes pass | Script output |
PR-15: Dockerfiles and release matrices from the registry; retire services.json (G4–G6; effort M)
Files:
- generate-dockerfiles.py: read the closure from the registry.
- Run generate-dockerfiles.py --validate in CI.
- ci-cd.yml:500-661: build the matrices from the registry.
- Delete or regenerate services.json, per D8, and update validate-workflows.sh:47,1198-1238, the pdf-agent contract script, and CLAUDE.md:54,73.
- Delete the unused s3-notifier Dockerfile.
Rebase onto #3947 (runtime_packages) if it lands first.
| # | Acceptance criterion | Verification |
|---|---|---|
| 15.1 | Regenerated Dockerfiles → byte-identical, except PM gains nothing it lacks today | --validate; diff in PR |
| 15.2 | The main matrix JSON before and after → equal for every service | Contract script comparing against frozen JSON |
| 15.3 | Stale Dockerfile → CI fails | Fixture |
R3: build hygiene (#3982)¶
PR-16: global.json and NSwag alignment (B2, B8; effort S–M)
Files:
- global.json: SDK 10.0.x feature band, rollForward: latestPatch (PROPOSAL; D9).
- setup-dotnet gets global-json-file everywhere.
- Upgrade the nswag tool to 14.2.0 and drop .NET 8 if the tool no longer needs it. Whether 14.2.0 drops .NET 8 is NOT VERIFIED; the PR proves it.
- Update the routing contracts that pin those steps.
- Run nswag:all after a Release build, from the repo root.
| # | Acceptance criterion | Verification |
|---|---|---|
| 16.1 | Every .NET lane → dotnet --version matches global.json |
Step output on the triggered lanes |
| 16.2 | NSwag regeneration → no diff in the generated clients | validate-generated-code lane |
PR-17: CPM lift, version-neutral (B1; effort M). Directory.Packages.props with ManagePackageVersionsCentrally. Each package takes its current version; the 12 drifting packages use VersionOverride for now. The generator and Dockerfiles COPY Directory.Packages.props, which also becomes a registry global trigger. pdf-agent's hand-written Dockerfile is updated too.
| # | Acceptance criterion | Verification |
|---|---|---|
| 17.1 | For each project, dotnet list package --include-transitive → identical before and after |
Per-project diff, run niced, one project at a time |
| 17.2 | A csproj with Version= on a PackageReference → build error NU1008 |
CI build |
| 17.3 | Every service image builds | Preview build on the PR |
PR-18: harmonise test packages, transitive pinning, audit as errors (B1, B3; effort M–L)
Upgrade the test packages to the newest version already present:
- xunit 2.9.3;
- xunit.runner.visualstudio 3.1.4;
- Test.Sdk 17.14.1;
- Moq 4.20.72;
- coverlet 6.0.4;
- Testcontainers 4.x (an API change).
Then turn on CentralPackageTransitivePinningEnabled, NuGetAudit with NuGetAuditMode=all, and WarningsAsErrors NU1903 and NU1904. Production package drift (Serilog.Settings.Configuration, morelinq, Http.Abstractions) is a follow-up.
| # | Acceptance criterion | Verification |
|---|---|---|
| 18.1 | A build → 0 NU1903/NU1904 | CI log |
| 18.2 | A test project adding a package with a high advisory → restore fails | Throwaway red commit, then reverted |
| 18.3 | Test counts per project → unchanged (± skips explained) | CI test summary before and after |
PR-19: warning ratchet (B6, B7; effort M)
Files:
- .github/scripts/warning-ratchet.sh. It parses the build output in test-dotnet (main and PR) into per-(project, code) counts.
- The committed build/warning-baseline.json.
- CA2007 handled per D10 (recommended: none repo-wide, which removes about 87% of warnings).
- The culture rules CA1305, CA1307 and CA1310 re-enabled as warnings inside the baseline.
| # | Acceptance criterion | Verification |
|---|---|---|
| 19.1 | A PR adding one new CA1305 warning → test-dotnet fails and names the file and rule |
Throwaway red commit |
| 19.2 | A PR reducing counts → passes, and prints a reminder to lower the baseline | CI log |
| 19.3 | Total warnings after D10 ≤ 1,200 (PROPOSAL) | CI log |
| 19.4 | The step adds ≤ 1 min, and the lane timeouts stay as pinned | CI timing; routing contract |
PR-20: grouped dependency updates (B5; effort S). .github/dependabot.yml (D11) covers nuget, npm (pnpm), github-actions and docker. It runs weekly, with groups for test packages, Microsoft., AWSSDK., Angular and Actions, and open-pull-requests-limit: 5 (PROPOSAL), so this CI host is not flooded.
| # | Acceptance criterion | Verification |
|---|---|---|
| 20.1 | First weekly run → at most 5 open update PRs, each grouped | GitHub UI or API |
PR-21 (optional, D12): .slnx. Run dotnet sln migrate, then update the 7 .slnf files, CI, run-tests.sh, the contract scripts and the docs (96 references). Acceptance: a build of each .slnf succeeds, and no syrf.sln reference remains outside history docs.
R4: workflow simplification (#3990), incremental¶
PR-22a/b/c: composite actions, one per PR (W1; effort M each)
- a. .NET setup: setup-dotnet + global.json + run-unique NuGet state.
- b. GKE auth: 7 google-github-actions/auth uses + credentials.
- c. cluster-gitops checkout via the App token: 12 create-github-app-token uses.
Each migrates one workflow at a time and updates the contracts that pin those steps.
| # | Acceptance criterion | Verification |
|---|---|---|
| 22.1 | Migrated lanes → the same step outcomes, and checkout is still first | Triggered lane runs; routing contracts |
| 22.2 | The step-digest pins in test-workflow-validator-consolidation.sh are replaced by behavioural assertions where the PR touches them |
Script diff |
PR-23: inline-shell ratchet and pr-preview.yml split (W1, W2, W4; effort L, in 3–4 PRs)
- Add a validator rule: a new or modified run: block of more than 40 lines fails (PROPOSAL), with existing blocks allowlisted.
- Extract the cleanup, /reseed-db and label-conflict handlers into their own workflows.
- Move the largest blocks into tested scripts.
| # | Acceptance criterion | Verification |
|---|---|---|
| 23.1 | pr-preview.yml ≤ 2,500 lines (PROPOSAL) |
wc -l |
| 23.2 | Preview create, update, reseed and close on a test PR → same outcomes as before | Live preview runs |
PR-24: fewer check runs (W3; effort L; D13). Fold the per-service Sonar jobs into a matrix, and skip whole workflows by trigger paths: where the contracts allow. The *-fork twins stay, as the fork rule requires.
| # | Acceptance criterion | Verification |
|---|---|---|
| 24.1 | A docs-only PR → ≤ 40 check runs; a .NET PR → ≤ 70 (PROPOSAL) | API count on two PR heads |
| 24.2 | Test Summary semantics are unchanged (fail-closed) |
test-pr-test-summary-fail-closed.sh |
5. Order and critical path¶
flowchart LR
PR1[PR-1 version/retag] --> PR2[PR-2 promotion copy] --> PR3[PR-3 Lambda release]
PR4[PR-4 DB lock] --> PR8[PR-8 pipefail]
PR5[PR-5 injection]
PR6[PR-6 dead workflows]
PR7[PR-7 GitVersion]
PR12[PR-12 registry] --> PR13[PR-13 detect] --> PR14[PR-14 pr-tests]
PR13 --> PR15[PR-15 Dockerfiles+matrices]
PR3 --> PR15
PR16[PR-16 global.json] --> PR17[PR-17 CPM] --> PR18[PR-18 audit] --> PR19[PR-19 ratchet]
PR15 --> PR17
PR9[PR-9 AWS OIDC]
PR10[PR-10 review tools]
PR19 --> PR22[R4]
- Start immediately, in parallel (disjoint files): PR-1, PR-4, PR-5, PR-6, PR-7, PR-10, PR-12, and the PR-9 infrastructure half.
- Sequential on
ci-cd.yml: PR-1 → PR-2 → PR-3 → (PR-9's ci-cd part) → PR-15. - Sequential on
pr-preview.yml: PR-4 → PR-8 (pr-preview part) → PR-23. - Critical path for the MVP: PR-1 → PR-2 → PR-3, alongside PR-12 → PR-13.
- PR-17 waits for PR-15, because the generator must copy
Directory.Packages.props. - R4 starts only after R3, so composites wrap the final .NET setup.
- One worker per worktree (
wt new, underpr/). This host is the CI runner, so workers never build the solution.
6. Issues¶
-
3982 → PR-16 to PR-21.¶
-
3983 → PR-12 to PR-15.¶
-
3990 → PR-6, PR-8 and PR-22 to PR-24 (the epic gains sub-issues).¶
On approval, new issues are created for PR-1 to PR-5, PR-7, PR-9 to PR-11 and Ops-1. Each copies its acceptance table from this plan.
7. Risks¶
| Risk | Mitigation |
|---|---|
| PR-2 changes what production PRs contain | Allowlist fixtures; Chris reads the next draft diff; nothing is merged by an agent |
Pipefail turns silent failures into red runs, and echo \| grep -q can fail with SIGPIPE |
Ratchet one workflow per PR; SIGPIPE audit (8.3); the destructive workflows go first |
| Registry over-triggers builds (Lambda and PM rebuild more often) | Intended; CI cost is measured on the first week of main runs. Staging only |
| CPM or Testcontainers 4 breaks tests | Version-neutral lift first (17.1); the Testcontainers upgrade is isolated in PR-18 with per-project test counts |
| Contract scripts pin step bodies, so every refactor touches them | Replace digest pins with behavioural assertions as files are touched (22.2) |
| Claude review and ruleset-locked workflows can only be proven after merge | Small diffs; wrapper unit tests; immediate post-merge /claude-review check |
| OIDC cutover breaks Lambda packaging | Keep the static keys until each lane is green on OIDC, then delete them |
| Overlap with stale CI PRs | See §8; supersede rather than rebase |
8. Coordination with other active work¶
| Open PR or stream | Overlap | Action |
|---|---|---|
#2914 (gate tags on successful builds), #2920 (no-op promotions, copy-staging-promotion-to-production.sh), #2924 |
ci-cd.yml, detect-service-changes.sh; all CONFLICTING since 1 Sep |
Reuse their ideas in PR-1 and PR-2; ask Chris to close them as superseded (D14) |
| #3160 (solution filter), #3161 (NuGet fallback cache), #3167, #3200, #3203 | pr-tests.yml, routing contracts; CONFLICTING since 2 Sep |
Close, or fold into R3/R4 (D14) |
| #2860–#2885 (Juniper migrations, drafts) | pr-preview.yml, _detect-changes.yml (#2866) |
PR-6 deletes _detect-changes.yml; close #2866 |
| #3947 (notifications stack) | dependency-map.yaml, generate-dockerfiles.py (runtime_packages) |
PR-15 keeps runtime_packages as hand metadata; rebase after it |
| #3934, #2887 | csproj files, syrf.sln, quartz.slnf |
PR-17 and PR-21 rebase as needed |
| Authentication migration session | Identity Dockerfile and image (PR-15, PR-17) | Notify before merging; no Identity behaviour change |
| FEAT-024 statistics session | PM Core holds most CA2007 warnings (PR-19) | The ratchet only blocks new warnings; no edits to its code |
9. Decisions needed from Chris¶
- D1. Lambda release (F2): fix
create-lambda-release(recommended; it gives an auditable artifact per version), or delete it, since ACK deploys from S3. - D2. Production
deploymentId(F1): keep production's existing value, or clear it so the PostSync hook skips the status update (recommended: clear it; production has no matching Deployment today). - D3. cluster-gitops
delete_branch_on_merge: enable it (recommended) and purge the about 1,260 merged or closed promotion branches after a dry-run list. - D4.
lock-db(F5): create the label and keep the feature (recommended), or remove the feature. - D5. Lambda bucket (F9): move the Lambda packages out of
camarades-terraform-state-awsinto a dedicated bucket during the OIDC work (recommended), or keep the bucket and scope the IAM role by prefix. - D6. Finalizers (F8): a narrowed, documented preview break-glass (recommended), or remove it and fix the hook-finalizer root cause.
- D7. Lock files and NuGet caching: defer (recommended) until CPM settles.
- D8.
services.json: delete it and point its two consumers at the registry (recommended), or regenerate it. - D9. SDK pin:
rollForward: latestPatchon one feature band (recommended), orlatestFeature. - D10. CA2007:
nonerepo-wide, since every host is ASP.NET Core or a worker with no SynchronizationContext (recommended), or keep it for the libraries and addConfigureAwait(false)to about 7,800 call sites. - D11. Dependabot or Renovate: Dependabot version updates (recommended: native, alerts already on, no app install), or Renovate (better grouping, but an org app install).
- D12.
.slnx: defer (recommended; 96 references, little gain), or do it after R3. - D13. Check-run target (W3): accept the PROPOSAL thresholds (24.1), and decide whether the Sonar matrix is worth the contract churn.
- D14. Stale CI PRs: close #2914, #2920, #2924, #3160, #3161, #3167, #3200, #3203 and #2866 as superseded by this plan (recommended).
- D15. PROPOSAL values:
- staging-merge wait 10 min (PR-2);
- finalizer age 10 min (11.1);
- warning total ≤ 1,200 (19.3) and ratchet cost ≤ 1 min (19.4);
- Dependabot limit 5 (20.1);
run:block ≤ 40 lines andpr-preview.yml≤ 2,500 lines (R4);- check-run targets 40/70 (24.1).