home-network-k3s-impl

772 Commits 139 Branches 0 Tags

3 Commits

Include renames

Author SHA1 Message Date

Author	SHA1	Message	Date
jakman85andClaude Opus 5	79c703bc79	fix(conformance): whole-repo checks audited nested agent worktrees as source Six discovery sites walked ROOT without pruning dot-directories, so they descended into .claude/worktrees/agent-*/ — up to 18 full copies of the repo, created for isolated agent runs and gitignored (.gitignore:43). The visible half was false FAILs: four ServiceMonitors reported as selecting no Service, and nine PrometheusRule manifests reported as "under test but not in the repo", all of them worktree paths. The whole-repo L0 gate was red for reasons that had nothing to do with authored surface. The dangerous half is quieter. A check that asks "does anything in the repo declare X" can be satisfied by a worktree copy instead of real source, so the same bug that invents failures can also manufacture a pass. That is the reason to fix all six sites rather than only the ones that happened to be failing. egress-paths-selected-by-rules already had this fix and documents it (it once saw 5 CiliumEgressGatewayPolicy objects instead of the 1 authored); this applies the same dot-directory predicate everywhere else, which skips dot-DIRECTORIES while still matching dot-FILES such as .sops.yaml. platform/monitoring/tests/run.sh is the other side of the same failure: its `--list` reported each manifest once per worktree, and the L0 check compares that listing against the tree. Both find calls now prune .claude alongside .git. `--list` returns 21 real pairs where it previously returned them ~19x over. Verified in the pinned toolbox: L0 --include-scope whole-repo is 21 cases, 0 failures, 0 skips, 0 unwired (was 4 failures, then 1). L1 3/3. The 18 stale worktrees are all on already-merged branches and remain on disk; cleaning them up is a separate change and is no longer load-bearing for the gate. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-18 22:14:51 -05:00
jakman85andClaude Opus 5	87c3d83dff	fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever `conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3 SKIPPED on every gate run since it landed, because its check keyed its whole existence on `bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is no deletion commit for it anywhere; it landed only on unmerged prod-standup branches (`work/prod-standup-campaign`, `prod/`, `feat/prod-hardening`). It was planned and never merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is the quiet version of the failure it was written to prevent. The skip was also hiding a RED. Run against the real `standup.sh` from `work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both documented there as load-bearing). It would have false-blocked the merge landing its own input. NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard: HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE cluster-vars value; an empty effective value is STATED by the overlay rather than inherited in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the 2026-09-05 outcome asserted at the source instead of at the guard — if prod's S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does not exist. Input surface is `clusters/`, all of it on `main`. HALF B (self-activating): the real predicate extracted and executed against those same values, the consumer trace, and the mid-string anchoring property. The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded: any tracked `.sh` defining `<name>_is_placeholder()` is found, so a guard landing at `bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When no guard exists the gap is printed loudly every run instead of swallowed as a skip. LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is always a defect in an overlay, while `""` is held to the explicitness rule instead — and still counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard author's declaration that empty is not a usable target. Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present, reverting `s3_is_placeholder` to the `TBD-` PREFIX glob FAILS 12/110 (B3 anchoring). With the pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34. L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged — R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps. Also: `ci/touched-requirements.sh` — `bootstrap/` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale comment corrected. It claimed standup.sh "carries the placeholder guards", describing a file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the change that lands a placeholder guard selected by nothing. `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8) locking `bootstrap/` and `clusters/`, so removing either entry is a test failure rather than a silent loss of selection. All three ci/tests pass (12 + 31 + 15). * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a placeholder-shaped overlay value" is no longer true; corrected. * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that can break it attributes a requirement it declares. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 00:10:37 -05:00
jakman85andClaude Opus 5	6a8953e288	feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing Minimal prod delivery, authored directly on main (= prod's live revision `d1a6628`) rather than by merging work/prod-standup-campaign. That branch carries the DNS apex inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun rewriting apex records dev owns while the non-k3s prod still serves 443 there. Operator sequencing is "dev migrates first". Nothing under clusters/ is touched here at all, which is what makes this safe to land now. Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of platform/velero/ and the CNPG destinationPath changes. Those land as feat/r2-backup-prod together with the endpoint and bucket knobs. WHAT THIS FIXES 1. Nothing was watching CNPG. All three prod databases have run since bootstrap with ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed anywhere. Because archive_mode=on and the archive command fails, Postgres refuses to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within the hour, and evaluates to 3 firing series on prod today. Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because nextcloud-db's exporter cannot log in to its own database (permission denied for database "nextcloud", both clusters) so the pg_stat_archiver family does not exist for it — the obvious metric would have been blind to one database in three. 2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to job="kubelet" (the apiserver copy of the same series reports namespace=default), and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report available=0 forever and would otherwise have fired critical permanently. 3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment a node carries the egress-gateway label, with dev's behaviour unchanged today. 4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored LITERALLY and the import still reports success, then sits at Done=True and never re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was green. The CronJob is the level-triggered correction; two alerts make its silence observable. Repaired live already — this stops it recurring. Also: the R-NET-04 conformance scan descended into nested agent worktrees under .claude/, counting 4 egress policies where 1 is authored. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 00:59:22 -05:00

jakman85andClaude Opus 5

79c703bc79

fix(conformance): whole-repo checks audited nested agent worktrees as source

Six discovery sites walked ROOT without pruning dot-directories, so they
descended into .claude/worktrees/agent-*/ — up to 18 full copies of the repo,
created for isolated agent runs and gitignored (.gitignore:43).

The visible half was false FAILs: four ServiceMonitors reported as selecting no
Service, and nine PrometheusRule manifests reported as "under test but not in
the repo", all of them worktree paths. The whole-repo L0 gate was red for
reasons that had nothing to do with authored surface.

The dangerous half is quieter. A check that asks "does anything in the repo
declare X" can be satisfied by a worktree copy instead of real source, so the
same bug that invents failures can also manufacture a pass. That is the reason
to fix all six sites rather than only the ones that happened to be failing.

egress-paths-selected-by-rules already had this fix and documents it (it once
saw 5 CiliumEgressGatewayPolicy objects instead of the 1 authored); this applies
the same dot-directory predicate everywhere else, which skips dot-DIRECTORIES
while still matching dot-FILES such as .sops.yaml.

platform/monitoring/tests/run.sh is the other side of the same failure: its
`--list` reported each manifest once per worktree, and the L0 check compares
that listing against the tree. Both find calls now prune .claude alongside .git.
`--list` returns 21 real pairs where it previously returned them ~19x over.

Verified in the pinned toolbox: L0 --include-scope whole-repo is 21 cases,
0 failures, 0 skips, 0 unwired (was 4 failures, then 1). L1 3/3.

The 18 stale worktrees are all on already-merged branches and remain on disk;
cleaning them up is a separate change and is no longer load-bearing for the gate.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-18 22:14:51 -05:00

jakman85andClaude Opus 5

87c3d83dff

fix(conformance): placeholder-guard-coupling actually RUNS — it was skipping forever

`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3
SKIPPED on every gate run since it landed, because its check keyed its whole existence on
`bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is
no deletion commit for it anywhere; it landed only on unmerged prod-standup branches
(`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never
merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is
the quiet version of the failure it was written to prevent.

The skip was also hiding a RED. Run against the real `standup.sh` from
`work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from
treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on
purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both
documented there as load-bearing). It would have false-blocked the merge landing its own input.

NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard:

  HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE
  cluster-vars value; an empty effective value is STATED by the overlay rather than inherited
  in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the
  2026-09-05 outcome asserted at the source instead of at the guard — if prod's
  S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does
  not exist. Input surface is `clusters/`, all of it on `main`.

  HALF B (self-activating): the real predicate extracted and executed against those same
  values, the consumer trace, and the mid-string anchoring property.

The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded:
any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at
`bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already
recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When
no guard exists the gap is printed loudly every run instead of swallowed as a skip.

LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is
always a defect in an overlay, while `""` is held to the explicitness rule instead — and still
counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard
author's declaration that empty is not a usable target.

Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the
literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's
explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER
with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present,
reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the
pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34.

L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged —
R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps.

Also:
  * `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale
    comment corrected. It claimed standup.sh "carries the placeholder guards", describing a
    file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the
    change that lands a placeholder guard selected by nothing.
  * `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8)
    locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather
    than a silent loss of selection. All three ci/tests pass (12 + 31 + 15).
  * `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a
    placeholder-shaped overlay value" is no longer true; corrected.
  * Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that
    can break it attributes a requirement it declares.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-13 00:10:37 -05:00

jakman85andClaude Opus 5

6a8953e288

feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing

Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-07 00:59:22 -05:00