diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index ba12028..d1ce8c9 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -4538,3 +4538,181 @@ Open, in the order I would take them: 7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, 103 unmapped paths, and two rendered manifests nothing applies (the keycloak one would stand up a conflicting instance). + +## 2026-09-19 — Overnight: the backup alert was crying wolf, and 49 app manifests were gated by nothing + +Worked autonomously while the operator slept. One theme again: things that reported green, or red, over +something other than what they claimed to measure. + +### 1. VeleroNoRecentBackup was firing CRITICAL over backups that protected everything + +`velero-daily-20260919030041`: **PartiallyFailed, 3079 of 3079 items backed up, 2 errors** — both +`PodVolumeBackups` for the emptyDir `plugins` + `scratch` volumes of a +`homepage-default-kopia-maintain-job-…` pod. Velero runs transient kopia repository-maintenance Jobs in +its own namespace, one per BackupRepository; under `defaultVolumesToFsBackup: true` it tries to +fs-backup their volumes, the Job finishes, the pod disappears mid-backup, and the PVB fails. Four such +failures had accumulated. + +Nothing real was ever missed. But `PartiallyFailed` does not advance +`velero_backup_last_successful_timestamp`, so the alert guarding R-STATE-03 fired nightly over Velero +cleaning up after itself. **An alert that cries wolf every night is how a real backup failure gets +ignored** — which is precisely the failure mode that alert exists to prevent. + +**First fix was right about the bug and wrong about the layer.** I excluded ns `velero`'s volume data +(keeping its resources in schedule 2) and explicitly deferred the general fix — skip emptyDir by type — +because Velero's `volumeTypes` policy condition was unproven here and an unparseable `policies.yaml` +breaks EVERY backup. I wrote "revisit if the shape shows up elsewhere." + +It showed up elsewhere inside the hour. The verification backup came back PartiallyFailed with 4 errors, +all in ns `runbooks`: the nginx Deployment's emptyDir `tmp` + `cache`, failing with +`error to expose PVB: error getting client pod …: pods "…" not found`. **Cause worth stating plainly: I +caused it** — pushing a runbook edit changed the configMap hash, rolled the Deployment, and the backup +lost the pods mid-snapshot. Self-inflicted, not a standing defect. But the mechanism is not +self-inflicted: a PVB exposes a volume *through its pod*, so any rollout landing in the backup window +fails one, and one failed PVB makes the whole Backup PartiallyFailed. On a GitOps cluster that is +routine, not an edge case. + +So the property is what matters, not the namespace: an emptyDir is created empty with its pod and +destroyed with it, so backing one up cannot protect anything while attempting it can break the backup +that protects everything else. + +**The `volumeTypes` risk was retired by testing, not by hoping.** New policy into a THROWAWAY +ConfigMap first, live one untouched, proven on prod (velero 1.18.1) with a control: + +| ns `runbooks` | PodVolumeBackups | +|---|---| +| control, no policy | **4** (2× `tmp`, 2× `cache`), Completed | +| with the policy | **0**, Backup Completed, 0 errors | + +The control is the point — "zero PVBs" alone is equally consistent with nothing having been attempted, +which is exactly the empty green I shipped earlier in this session with a 24h baseline built from ten +minutes of data. + +Then reverted the namespace exclusion as redundant (`plugins`/`scratch` are emptyDir, verified on a live +maintain pod) rather than leaving two overlapping fixes. + +**Proof, full cluster, policy live:** `pre-memswap-202609190525` — **Completed, 3926/3926 items, 0 +errors, 21/21 PodVolumeBackups Completed, 0 emptyDir volumes backed up.** `VeleroNoRecentBackup` +cleared. `VeleroBackupPartialFailure` remains at `increase = 1.008` from the pre-fix failure and +self-clears about an hour after it, by design — the rule is `increase(...[1h]) > 0`, which is correct. + +That backup also serves as the pre-maintenance backup the memory-swap runbook requires. + +### 2. The runbook claimed the CNPG primaries sit on node 1. They do not. + +Verified live: all three CNPG clusters run 3 instances spread one per node by required anti-affinity, +and **k3s-prod-1 holds a REPLICA of each** (`keycloak-db-2`, `nextcloud-db-2`, `semaphore-db-2`). No +primary is on node 1, so draining it forces **no failover at all** — the best state this maintenance +could ask for. The original sentence was stale session state. The runbook now prints the command to +re-derive it, because primaries move after any restart. + +That also settles `CNPGPrimariesCoLocated`, firing for k3s-prod-2 (both `keycloak-db` and +`semaphore-db` primaries). **Do not tidy it before the swap.** Rebalancing means promoting a replica, +the only node with spare primary capacity is node 1, so "fixing" the alert would put a primary on the +node about to be powered off and convert a clean drain into a two-database failover. Rebalance after +node 1 is back and caught up. Warning is now in the runbook. + +### 3. 49 app manifests attributed no requirement, so the merge gate ran no behaviour test for them + +`ci/touched-requirements.sh`'s coarse path map had no arm for `apps/*`. A change adding or editing any +workload selected **zero** requirement-scoped tests — so none of the L2 behaviour tests that enforce +default-deny, egress class or auth mode ran on exactly the changes that can violate them. Kyverno +admission and the L0 egress-path check are real backstops, but admission is post-merge and the gate +exists to catch it before. + +Also unmapped: `platform/homepage` and `platform/headlamp` — **the C2 UI**. This was already documented +as a known gap in `changed_diff()`'s rule-3 comment ("the ONLY route to R-C2-01/02 was rule 3", i.e. +editing the C2 test itself), and had sat there since. + +Twelve arms added: `apps/vms/*` (before `apps/*`, since `case` takes the first match), `apps/*`, +`platform/homepage|headlamp`, `platform/kubevirt`, `platform/gateway`, `platform/coredns`, +`platform/crafty`, `platform/external-snapshotter|backup`, `runbooks/*`, `platform-config/*`, +`platform/kustomization.yaml` and `.sops.yaml`. + +`platform/kustomization.yaml` deserves its own note: it is the assembly list, so **one deleted line +disables an entire platform component** while every file it points at stays valid on disk. That is the +most consequential single-line edit in the repo and it attributed nothing. + +Measured effect for an `apps/` change: **0 → 13 requirement-scoped tests** (3 L1, 8 L2, 2 L3). Coverage +stays 51/51, so the gate's "attributed IDs must be covered" assertion does not false-block. + +Ten new assertions in `ci/tests/touched-requirements.test.sh` (26 total, all passing), including the +ORDER property — `apps/vms/*` must win over `apps/*` — since that is a real failure mode of `case` and +not merely a matter of both arms existing. + +**A correction on my own numbers.** I first reported "192 unmapped files", then "182". Both were +inflated by my audit script, not by the repo: the regex used `[^)]*`, which matched across a newline, so +the first arm parsed as `case "$1" in\n catalog/*` and never matched anything — and I separately +mis-parsed alternation arms like `.gitea/*|ci/*`, which made `ci/` look unmapped when it is mapped. +True remainder after this change is **18 files, all prose or captured artifacts** (`docs/plans/`, +root READMEs, `.standup-dns/*.txt`). + +### 4. Finished `feat/egress-gateway-staleness-alert` — it had been backlogged because it could not pass the gate + +Merged the two alerts for the silent egress-gateway failure (`VPNEgressSteeringStale` critical, +`VPNEgressUplinkLabelMissing` warning) plus the KSM allowlist entry that makes them non-inert. + +Confirmed first that they are **live on prod, not decorative**: `k3s-prod-3` carries +`egress-gateway=true` and `egress-uplink=10.150.192.86-139`, `vpn-egress-steering` has been in place 14 +days, and the label was exported on **0 series** before this change — so the allowlist entry is the +switch that turns the alerts on. + +Three whole-repo checks failed the branch, each a real gap: the runbook the alerts link to did not +exist (a firing alert would hand the operator a 404); `alerts.yaml` is an already-covered file, so new +rules in it must be unit-tested; and the scrape-target guard broke. Wrote +`runbooks/vpn-egress-stale.md` (wired into the generator, four required sections) and +`platform/monitoring/tests/vpn-egress.test.yaml` (four promtool fixtures — tests 1 and 2 are the SAME +reconnect with and without the label following it, which isolates the mechanism: a label value change +retires one series and starts another rather than changing a sample). + +**Two of the three failures were defects in the checks, not the branch:** + +- `alert-metrics-have-scrape-targets` asserted the allowlist matched `nodes=[...egress-gateway]` with a + literal closing bracket — that the list held *exactly* that one label. Correctly adding a second label + therefore failed the check while the guarded property stayed satisfied. A guard that breaks when you + correctly extend the thing it guards teaches people to edit the guard, which is how it stops guarding. +- `UNTESTED_ALERTS` flagged `VPNTunnelDown` as stale because "asserted on" is a **bare word match over + the test file** — my header *comment* mentioning it counted as coverage. I did not delete the register + entry to make that go away; instead the fixtures now assert `VPNTunnelDown` **quiet** in all four + cases. That required making them faithful: they had no `node_network_receive_bytes_total{device="tun0"}` + at all, so `VPNTunnelDown` would have been firing throughout unnoticed, because promtool only checks + the alertnames you name. A healthy tun0 passes traffic — and that silence IS the claim these rules + exist for. + +**Known, deliberately not fixed:** that word match means any alert named in prose counts as tested. +Tightening it to `alertname:` positions could expose other prose-only "coverage", and the register is +shrink-only, so the cascade wants doing deliberately — not at 01:00 unattended. + +### 5. Stale worktrees: audited, deliberately not touched + +18 agent worktrees remain under `.claude/worktrees/`. **14 clean, 4 dirty** — the dirty ones being +`fix/gate-does-not-pass-on-skipped-stages`, `fix/postbuild-substitution-catches-dollar-mangling`, +`fix/scrape-targets-for-inert-alerts` and `fix/qbittorrent-durable-webui-credential`. Every branch is +already merged into main, so the uncommitted state is leftover agent scratch rather than unmerged work. + +Left in place. They no longer affect the gate (the scanner now skips dot-directories), so removal is +disk hygiene, and deleting someone's uncommitted changes unasked is not a call to make while they are +asleep. `git worktree remove` preserves branches and their commits, so only that scratch is at risk. + +### PM HANDOFF + +1. **R-STATE-02 is still the Phase 7 exit item, and it is blocked on permission, not on readiness.** + `./conformance/run.sh --layer l3 --requirements R-STATE-02 --record`. The permission classifier + refuses it and I did not route around it — hand-rolling the cordon would defeat the guard. I read the + test first: it cordons ONE node and evicts only its OWN writer pod, never `kubectl drain`; kubelet + and etcd stay up so quorum is untouched; PVC and pods live in the per-test namespace; and `finally` + uncordons via a unique label even on failure or panic. Preconditions verified: 3 Ready nodes, + `longhorn` default SC, nothing pre-cordoned, 24/24 volumes healthy. Either run it, or add a Bash + permission rule. +2. **The memory swap is still the cheapest R-STATE-02 evidence** — a supervised single-node loss on prod + is the event that requirement describes. Node 1 currently holds no DB primary, which is the ideal + window. +3. **Node 2 DIMM re-inventory** — blocked on the vault become password. Never `ansible-inventory --list`. +4. **Pi-hole needs A records** for `unifi.prod` → `.21` and `crafty-mc.prod` → `.22` before router DHCP + option 6 → `.23` is safe. It currently answers `.20` for both, and neither service speaks HTTP + through that Gateway VIP, so the LAN-wide change would break UniFi adoption and Minecraft. Until + then the per-machine dnsmasq override is load-bearing. +5. **UniFi guest has no IP** — needs console; `cat /etc/netplan/*.yaml` is the confirming command. +6. **Five `public` services have no working public path** — waiting on WAN forwarding, deliberately. +7. Backlog cherry-picks: `feat/egress-gateway-staleness-alert`, two `rescued/*` fixes, two rendered + manifests nothing applies (the keycloak one would stand up a conflicting instance).