From e3e56af461153ca28bbdff45f9fe22db99373964 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 19 Sep 2026 00:52:29 -0500 Subject: [PATCH] =?UTF-8?q?docs(decisions-log):=20overnight=20handoff=20?= =?UTF-8?q?=E2=80=94=20backup=20alert=20crying=20wolf,=2049=20unmapped=20a?= =?UTF-8?q?pp=20manifests,=20egress=20branch=20finished?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the three fixes and, in each case, the thing that was reporting green or red over something other than what it claimed to measure. Includes corrections to my own numbers (the '192 unmapped files' figure was my audit script's regex matching across a newline, not the repo) and the two check defects the egress branch exposed. PM HANDOFF lists open items in priority order. R-STATE-02 remains the Phase 7 exit item and is blocked on the permission classifier, not on readiness. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 178 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 178 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index ba12028..d1ce8c9 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -4538,3 +4538,181 @@ Open, in the order I would take them: 7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, 103 unmapped paths, and two rendered manifests nothing applies (the keycloak one would stand up a conflicting instance). + +## 2026-09-19 — Overnight: the backup alert was crying wolf, and 49 app manifests were gated by nothing + +Worked autonomously while the operator slept. One theme again: things that reported green, or red, over +something other than what they claimed to measure. + +### 1. VeleroNoRecentBackup was firing CRITICAL over backups that protected everything + +`velero-daily-20260919030041`: **PartiallyFailed, 3079 of 3079 items backed up, 2 errors** — both +`PodVolumeBackups` for the emptyDir `plugins` + `scratch` volumes of a +`homepage-default-kopia-maintain-job-…` pod. Velero runs transient kopia repository-maintenance Jobs in +its own namespace, one per BackupRepository; under `defaultVolumesToFsBackup: true` it tries to +fs-backup their volumes, the Job finishes, the pod disappears mid-backup, and the PVB fails. Four such +failures had accumulated. + +Nothing real was ever missed. But `PartiallyFailed` does not advance +`velero_backup_last_successful_timestamp`, so the alert guarding R-STATE-03 fired nightly over Velero +cleaning up after itself. **An alert that cries wolf every night is how a real backup failure gets +ignored** — which is precisely the failure mode that alert exists to prevent. + +**First fix was right about the bug and wrong about the layer.** I excluded ns `velero`'s volume data +(keeping its resources in schedule 2) and explicitly deferred the general fix — skip emptyDir by type — +because Velero's `volumeTypes` policy condition was unproven here and an unparseable `policies.yaml` +breaks EVERY backup. I wrote "revisit if the shape shows up elsewhere." + +It showed up elsewhere inside the hour. The verification backup came back PartiallyFailed with 4 errors, +all in ns `runbooks`: the nginx Deployment's emptyDir `tmp` + `cache`, failing with +`error to expose PVB: error getting client pod …: pods "…" not found`. **Cause worth stating plainly: I +caused it** — pushing a runbook edit changed the configMap hash, rolled the Deployment, and the backup +lost the pods mid-snapshot. Self-inflicted, not a standing defect. But the mechanism is not +self-inflicted: a PVB exposes a volume *through its pod*, so any rollout landing in the backup window +fails one, and one failed PVB makes the whole Backup PartiallyFailed. On a GitOps cluster that is +routine, not an edge case. + +So the property is what matters, not the namespace: an emptyDir is created empty with its pod and +destroyed with it, so backing one up cannot protect anything while attempting it can break the backup +that protects everything else. + +**The `volumeTypes` risk was retired by testing, not by hoping.** New policy into a THROWAWAY +ConfigMap first, live one untouched, proven on prod (velero 1.18.1) with a control: + +| ns `runbooks` | PodVolumeBackups | +|---|---| +| control, no policy | **4** (2× `tmp`, 2× `cache`), Completed | +| with the policy | **0**, Backup Completed, 0 errors | + +The control is the point — "zero PVBs" alone is equally consistent with nothing having been attempted, +which is exactly the empty green I shipped earlier in this session with a 24h baseline built from ten +minutes of data. + +Then reverted the namespace exclusion as redundant (`plugins`/`scratch` are emptyDir, verified on a live +maintain pod) rather than leaving two overlapping fixes. + +**Proof, full cluster, policy live:** `pre-memswap-202609190525` — **Completed, 3926/3926 items, 0 +errors, 21/21 PodVolumeBackups Completed, 0 emptyDir volumes backed up.** `VeleroNoRecentBackup` +cleared. `VeleroBackupPartialFailure` remains at `increase = 1.008` from the pre-fix failure and +self-clears about an hour after it, by design — the rule is `increase(...[1h]) > 0`, which is correct. + +That backup also serves as the pre-maintenance backup the memory-swap runbook requires. + +### 2. The runbook claimed the CNPG primaries sit on node 1. They do not. + +Verified live: all three CNPG clusters run 3 instances spread one per node by required anti-affinity, +and **k3s-prod-1 holds a REPLICA of each** (`keycloak-db-2`, `nextcloud-db-2`, `semaphore-db-2`). No +primary is on node 1, so draining it forces **no failover at all** — the best state this maintenance +could ask for. The original sentence was stale session state. The runbook now prints the command to +re-derive it, because primaries move after any restart. + +That also settles `CNPGPrimariesCoLocated`, firing for k3s-prod-2 (both `keycloak-db` and +`semaphore-db` primaries). **Do not tidy it before the swap.** Rebalancing means promoting a replica, +the only node with spare primary capacity is node 1, so "fixing" the alert would put a primary on the +node about to be powered off and convert a clean drain into a two-database failover. Rebalance after +node 1 is back and caught up. Warning is now in the runbook. + +### 3. 49 app manifests attributed no requirement, so the merge gate ran no behaviour test for them + +`ci/touched-requirements.sh`'s coarse path map had no arm for `apps/*`. A change adding or editing any +workload selected **zero** requirement-scoped tests — so none of the L2 behaviour tests that enforce +default-deny, egress class or auth mode ran on exactly the changes that can violate them. Kyverno +admission and the L0 egress-path check are real backstops, but admission is post-merge and the gate +exists to catch it before. + +Also unmapped: `platform/homepage` and `platform/headlamp` — **the C2 UI**. This was already documented +as a known gap in `changed_diff()`'s rule-3 comment ("the ONLY route to R-C2-01/02 was rule 3", i.e. +editing the C2 test itself), and had sat there since. + +Twelve arms added: `apps/vms/*` (before `apps/*`, since `case` takes the first match), `apps/*`, +`platform/homepage|headlamp`, `platform/kubevirt`, `platform/gateway`, `platform/coredns`, +`platform/crafty`, `platform/external-snapshotter|backup`, `runbooks/*`, `platform-config/*`, +`platform/kustomization.yaml` and `.sops.yaml`. + +`platform/kustomization.yaml` deserves its own note: it is the assembly list, so **one deleted line +disables an entire platform component** while every file it points at stays valid on disk. That is the +most consequential single-line edit in the repo and it attributed nothing. + +Measured effect for an `apps/` change: **0 → 13 requirement-scoped tests** (3 L1, 8 L2, 2 L3). Coverage +stays 51/51, so the gate's "attributed IDs must be covered" assertion does not false-block. + +Ten new assertions in `ci/tests/touched-requirements.test.sh` (26 total, all passing), including the +ORDER property — `apps/vms/*` must win over `apps/*` — since that is a real failure mode of `case` and +not merely a matter of both arms existing. + +**A correction on my own numbers.** I first reported "192 unmapped files", then "182". Both were +inflated by my audit script, not by the repo: the regex used `[^)]*`, which matched across a newline, so +the first arm parsed as `case "$1" in\n catalog/*` and never matched anything — and I separately +mis-parsed alternation arms like `.gitea/*|ci/*`, which made `ci/` look unmapped when it is mapped. +True remainder after this change is **18 files, all prose or captured artifacts** (`docs/plans/`, +root READMEs, `.standup-dns/*.txt`). + +### 4. Finished `feat/egress-gateway-staleness-alert` — it had been backlogged because it could not pass the gate + +Merged the two alerts for the silent egress-gateway failure (`VPNEgressSteeringStale` critical, +`VPNEgressUplinkLabelMissing` warning) plus the KSM allowlist entry that makes them non-inert. + +Confirmed first that they are **live on prod, not decorative**: `k3s-prod-3` carries +`egress-gateway=true` and `egress-uplink=10.150.192.86-139`, `vpn-egress-steering` has been in place 14 +days, and the label was exported on **0 series** before this change — so the allowlist entry is the +switch that turns the alerts on. + +Three whole-repo checks failed the branch, each a real gap: the runbook the alerts link to did not +exist (a firing alert would hand the operator a 404); `alerts.yaml` is an already-covered file, so new +rules in it must be unit-tested; and the scrape-target guard broke. Wrote +`runbooks/vpn-egress-stale.md` (wired into the generator, four required sections) and +`platform/monitoring/tests/vpn-egress.test.yaml` (four promtool fixtures — tests 1 and 2 are the SAME +reconnect with and without the label following it, which isolates the mechanism: a label value change +retires one series and starts another rather than changing a sample). + +**Two of the three failures were defects in the checks, not the branch:** + +- `alert-metrics-have-scrape-targets` asserted the allowlist matched `nodes=[...egress-gateway]` with a + literal closing bracket — that the list held *exactly* that one label. Correctly adding a second label + therefore failed the check while the guarded property stayed satisfied. A guard that breaks when you + correctly extend the thing it guards teaches people to edit the guard, which is how it stops guarding. +- `UNTESTED_ALERTS` flagged `VPNTunnelDown` as stale because "asserted on" is a **bare word match over + the test file** — my header *comment* mentioning it counted as coverage. I did not delete the register + entry to make that go away; instead the fixtures now assert `VPNTunnelDown` **quiet** in all four + cases. That required making them faithful: they had no `node_network_receive_bytes_total{device="tun0"}` + at all, so `VPNTunnelDown` would have been firing throughout unnoticed, because promtool only checks + the alertnames you name. A healthy tun0 passes traffic — and that silence IS the claim these rules + exist for. + +**Known, deliberately not fixed:** that word match means any alert named in prose counts as tested. +Tightening it to `alertname:` positions could expose other prose-only "coverage", and the register is +shrink-only, so the cascade wants doing deliberately — not at 01:00 unattended. + +### 5. Stale worktrees: audited, deliberately not touched + +18 agent worktrees remain under `.claude/worktrees/`. **14 clean, 4 dirty** — the dirty ones being +`fix/gate-does-not-pass-on-skipped-stages`, `fix/postbuild-substitution-catches-dollar-mangling`, +`fix/scrape-targets-for-inert-alerts` and `fix/qbittorrent-durable-webui-credential`. Every branch is +already merged into main, so the uncommitted state is leftover agent scratch rather than unmerged work. + +Left in place. They no longer affect the gate (the scanner now skips dot-directories), so removal is +disk hygiene, and deleting someone's uncommitted changes unasked is not a call to make while they are +asleep. `git worktree remove` preserves branches and their commits, so only that scratch is at risk. + +### PM HANDOFF + +1. **R-STATE-02 is still the Phase 7 exit item, and it is blocked on permission, not on readiness.** + `./conformance/run.sh --layer l3 --requirements R-STATE-02 --record`. The permission classifier + refuses it and I did not route around it — hand-rolling the cordon would defeat the guard. I read the + test first: it cordons ONE node and evicts only its OWN writer pod, never `kubectl drain`; kubelet + and etcd stay up so quorum is untouched; PVC and pods live in the per-test namespace; and `finally` + uncordons via a unique label even on failure or panic. Preconditions verified: 3 Ready nodes, + `longhorn` default SC, nothing pre-cordoned, 24/24 volumes healthy. Either run it, or add a Bash + permission rule. +2. **The memory swap is still the cheapest R-STATE-02 evidence** — a supervised single-node loss on prod + is the event that requirement describes. Node 1 currently holds no DB primary, which is the ideal + window. +3. **Node 2 DIMM re-inventory** — blocked on the vault become password. Never `ansible-inventory --list`. +4. **Pi-hole needs A records** for `unifi.prod` → `.21` and `crafty-mc.prod` → `.22` before router DHCP + option 6 → `.23` is safe. It currently answers `.20` for both, and neither service speaks HTTP + through that Gateway VIP, so the LAN-wide change would break UniFi adoption and Minecraft. Until + then the per-machine dnsmasq override is load-bearing. +5. **UniFi guest has no IP** — needs console; `cat /etc/netplan/*.yaml` is the confirming command. +6. **Five `public` services have no working public path** — waiting on WAN forwarding, deliberately. +7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, two rendered + manifests nothing applies (the keycloak one would stand up a conflicting instance).