docs: PM handoff — 2026-08-31 current-state, gotchas + next steps; PHASE1-STATUS 8 vCPU

Appends the definitive current-state/handoff entry to DECISIONS-LOG (SSO fixed via CVE-2025-30162
egress allow, 5 forward-auth UIs gated, 8 vCPU, Gitea Actions CI live, external-snapshotter vendored;
operational gotchas; prioritized Next Steps → Longhorn-SSD, L2/L3-CI, forward-auth long-term,
Phase-6 onboarding). Updates the reconnect guide (6→8 vCPU) and points new agents at the handoff.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

DECISIONS-LOG.md

@@ -1174,3 +1174,79 @@ toolchain kubectl bumped 1.31.4 → 1.36.4 (the 1.31 pin exceeded the ±1 versio
 cluster). Revalidation items deliberately left open: external-snapshotter v8.2.0 pin vs 1.36, the
 Cilium ext_authz/forward-auth workaround relevance under 1.20.1, and the conformance ImageVolume
 feature-gate assertions under 1.36.
+
+## 2026-08-31 — Post-upgrade stabilization: SSO fixed, all UIs gated, 8 vCPU, CI live. **PM HANDOFF.**
+
+Everything the 08-30 campaign deferred is resolved; the platform is fully green. **This entry is the
+current source of truth and the handoff for the next PM.** (Session ran on a different workstation; a
+fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` and start at Next Steps.)
+
+### LIVE STATE (verified 2026-08-31, all green)
+- 3× k3s **v1.36.4+k3s1**, Debian 13, **8 vCPU / 24 GiB** each (raised from 6 — see #5). Proxmox host 24 cores, ~idle.
+- Cilium 1.20.1 (GW-API v1.6.1), Longhorn 1.12.1, cert-manager 1.21, Velero chart 12/v1.18, kube-prometheus-stack 88
+  (Grafana 13.2), kube-vip v1.2.3, **Loki chart 7.x / v3.6.11**, **Kyverno chart 3.9 / v1.19** (the last two were the
+  08-30 "deferred" items — now done). etcd on the dedicated SSD virtio2 disk per node.
+- All Flux kustomizations + HelmReleases Ready; 0 not-ready pods.
+- **SSO end-to-end**: native OIDC for grafana/gitea/nextcloud/semaphore/headlamp; the 5 forward-auth UIs
+  (pihole/jackett/qbittorrent/radarr/sonarr) gated by per-app oauth2-proxy sidecars (fail-closed, 302→Keycloak).
+  Platform-admins require TOTP MFA.
+- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main;
+  required status check; runner `incluster-dind` online (ns gitea-actions).
+
+### What changed this session (durable)
+1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared
+   Gateway L7 proxy (`enforce_policy_on_l7lb`) re-check a pod's egress policy against Keycloak's REAL backend
+   identity on the split-horizon hairpin; `toEntities: world` no longer covers it → all app SSO 403'd
+   ("Access denied", `server: envoy`). Fix: explicit `toEndpoints` egress allow to the keycloak backend —
+   `catalog/render.py` emits it for `auth.mode ∈ {oidc,forward-auth}`; headlamp/crafty hand-written. (68690e5, 8a424bc)
+2. **Keycloak MFA config bug**: browser-mfa conditional-OTP used a non-existent config key → "Invalid role name: null"
+   → 403 for every platform-admin login. Fixed to `condUserRole: platform-admin` in git + live via kcadm. (45deac9)
+3. **VPN egress restored**: the Debian 12→13 re-provision wiped k3s-dev-3's node-host OpenVPN (PIA); re-ran
+   `vpn.egress.gateway.yml`. Re-provision runbook (infra/ansible/README.md) updated — node-host roles (k3s, kube-vip,
+   vpn.egress.gateway) must be re-run after any reprovision.
+4. **Grafana OOM** (kps 65→88 ships Grafana 13) — the hidden blocker that stalled the whole kps upgrade + platform
+   tier. limits 256→512→768Mi; Prometheus 2→3Gi. (b2b1aa5 and follow-ups)
+5. **Node vCPU 6→8** — steady CPU PSI was 15-23% / load at-or-over core count (no spike headroom); now PSI ~4.5-7.7%.
+   bpg reboots the VM on a cores change → rolled ONE node at a time, waiting Longhorn healthy between. (676c37a)
+   + node CPU PSI/load alerts + a "Nodes — CPU/Memory Pressure" Grafana dashboard. (a720c73)
+6. **Security: 5 forward-auth UIs gated** with per-app oauth2-proxy sidecars (Option B, crafty's proven pattern) +
+   per-app Keycloak clients + SOPS creds. Closes the un-gated-admin-UI exposure. (72d4720, b2752ae)
+7. **Gitea Actions CI finished**: `ci/touched-requirements.sh` now excludes comment-only R-ID prose, so the workflow
+   runs the exact aggregate `make -C ci gate`. (27bc024)
+8. **external-snapshotter vendored** — was remote github kustomize bases; the kustomize-controller's per-reconcile
+   `git fetch github.com` flaked past its 27s timeout → `BuildFailed` stalled the platform tier. Now local
+   `crds.yaml` + `snapshot-controller.yaml` (offline, deterministic build). (7a79cea)
+9. **08-30 revalidations closed**: ImageVolume gate is now ON in 1.36 → conformance rationale comments corrected
+   (d60e553); external-snapshotter v8.2.0 pin kept (CRDs served at v1, works on 1.36); Cilium native ExternalAuth
+   evaluated → **DEFER** (see plan).
+
+### Operational gotchas that STILL matter (for the next PM)
+- **Access**: `./infra/devcontainer/dev.sh run <cmd>` (kubectl/flux/helm/tofu/ansible/kcadm). Do NOT run `dev.sh`/`make
+  gate` concurrently — execs into the one container interfere → transient failures, just retry. Deploy = `git push
+  gitea main` → Flux. Gate before push: `dev.sh run make -C ci gate` → "MERGE GATE: PASS".
+- **NAT topology** (see PHASE1-STATUS): API via `192.168.2.50:6443`→VIP .1.49; node SSH `192.168.2.102:225X`;
+  Proxmox API `192.168.2.102:8006`. A single node reboot can briefly drop the kubeconfig NAT — wait ~3 min for boot.
+- **Keycloak realm import is ONE-SHOT**: the operator logs "Job already executed - not recreating" — a git change to
+  `keycloak-realm-import.yaml` does NOT re-import. Edit the live realm via `kcadm.sh` in keycloak-0 (admin creds:
+  secret `keycloak-admin-credentials`; kcadm present, curl/python3 are NOT), AND fix git for reproducibility.
+- **Cilium world-hairpin (CVE-2025-30162)**: any in-cluster→`*.home` call hairpins through the Gateway and gets its
+  egress re-checked against the DESTINATION's real cluster identity — `toEntities: world` won't cover it. render.py
+  handles the keycloak case for SSO apps; a NEW app→app-by-hostname dependency needs an explicit `toEndpoints` allow.
+- **Remote github kustomize bases flake** (27s git-fetch timeout). external-snapshotter is now vendored;
+  **platform/kubevirt still uses remote bases** — vendor it the same way if it `BuildFail`s.
+- **Longhorn on the HDD**: every node reboot degrades all 18 volumes → full rebuild (slow; 30%+ Proxmox IO-delay).
+  Roll reboots one node at a time; wait `robustness=healthy` before the next. Fix = `docs/plans/longhorn-ssd-tiering.md`.
+- **Gitea** can go read-only (ext4 remount-ro) during node churn → `kubectl -n gitea rollout restart deploy/gitea`.
+
+### NEXT STEPS (prioritized) — where to start
+1. **Longhorn → SSD hot-tier move** — `docs/plans/longhorn-ssd-tiering.md`. The "won't fit" blocker was a measurement
+   artifact (working set ~79 GB; SSD ~323 GB free). Moving the hot tier (Prometheus/CNPG DBs/Loki/Gitea ≈ 51 GB) off
+   the HDD removes the reboot-rebuild tax on every future maintenance op. **Highest-leverage infra item.** Pre-flight:
+   re-verify live SSD free + read HDD SMART.
+2. **L2/L3 conformance in CI** — `docs/plans/l2-l3-conformance-ci-wiring.md`. Needs the trigger-credential decision
+   (scoped in-cluster SA + Kyverno-pinned Job shape; never a broad kubeconfig in CI). The gate runs no-cluster stages today.
+3. **Forward-auth long-term** — `docs/plans/cilium-native-externalauth-evaluation.md` (DEFER now: experimental CRD
+   channel + fail-open history). The 5 UIs are safely sidecar-gated in the meantime.
+4. **Remaining Phase-6 onboarding** — see DESIGN-waves.md / `catalog/` (services rendered-but-not-onboarded).
+5. **Vendor platform/kubevirt** remote bases proactively (same flake class as #8).
+6. **Human action**: finish Headlamp login = complete TOTP enrollment in the browser.

infra/PHASE1-STATUS.md

@@ -1,9 +1,13 @@
 # Dev cluster — current state & reconnect guide
 
-**Last verified: 2026-08-30.** Phase 1 is **complete and long since exited**; the cluster has been
+**Last verified: 2026-08-31.** Phase 1 is **complete and long since exited**; the cluster has been
 running the full platform (phases 2–6) for some time. This doc is how a session — human or agent —
 reconnects to it and re-establishes the toolchain.
 
+> **New PM / agent taking over?** Read this doc (reconnect + toolchain), then the **latest
+> `DECISIONS-LOG.md` entry (2026-08-31)** — it is the current-state + gotchas + prioritized **Next
+> Steps** handoff — and the plans under `docs/plans/`. Start at Next Steps.
+
 > **History note.** The cluster was originally 3 libvirt/KVM VMs on the workstation
 > (`10.73.0.0/24` NAT). That was **retired** after the workstation ran out of RAM/disk; the cluster
 > now lives on **Proxmox VE**. Anything mentioning `virsh`, `qemu:///system`, `tofu`+`dmacvicar/libvirt`,
@@ -15,7 +19,7 @@ reconnects to it and re-establishes the toolchain.
 | | |
 |---|---|
 | Substrate | Proxmox VE host, node `proxmox`, disks on the `Storage` ZFS datastore, bridge `vmbr0` |
-| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 6 vCPU, 24 GiB, Debian 13 (trixie) |
+| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 8 vCPU, 24 GiB, Debian 13 (trixie) |
 | k3s | **v1.36.4+k3s1**, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik |
 | CNI | Cilium 1.20.1 (kube-proxy replacement, L2 announcements, egress gateway) |
 | kube-API | kube-vip VIP **`192.168.1.49:6443`**, tls-san'd on the serving cert |