From c18288813313262fb842ce1dfddffd36c1bf33be Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 31 Aug 2026 21:23:04 -0500 Subject: [PATCH] =?UTF-8?q?docs:=20PM=20handoff=20=E2=80=94=202026-08-31?= =?UTF-8?q?=20current-state,=20gotchas=20+=20next=20steps;=20PHASE1-STATUS?= =?UTF-8?q?=208=20vCPU?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Appends the definitive current-state/handoff entry to DECISIONS-LOG (SSO fixed via CVE-2025-30162 egress allow, 5 forward-auth UIs gated, 8 vCPU, Gitea Actions CI live, external-snapshotter vendored; operational gotchas; prioritized Next Steps → Longhorn-SSD, L2/L3-CI, forward-auth long-term, Phase-6 onboarding). Updates the reconnect guide (6→8 vCPU) and points new agents at the handoff. Co-Authored-By: Claude Opus 4.8 --- DECISIONS-LOG.md | 76 ++++++++++++++++++++++++++++++++++++++++++ infra/PHASE1-STATUS.md | 8 +++-- 2 files changed, 82 insertions(+), 2 deletions(-) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index fee2c3e..d203550 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -1174,3 +1174,79 @@ toolchain kubectl bumped 1.31.4 → 1.36.4 (the 1.31 pin exceeded the ±1 versio cluster). Revalidation items deliberately left open: external-snapshotter v8.2.0 pin vs 1.36, the Cilium ext_authz/forward-auth workaround relevance under 1.20.1, and the conformance ImageVolume feature-gate assertions under 1.36. + +## 2026-08-31 — Post-upgrade stabilization: SSO fixed, all UIs gated, 8 vCPU, CI live. **PM HANDOFF.** + +Everything the 08-30 campaign deferred is resolved; the platform is fully green. **This entry is the +current source of truth and the handoff for the next PM.** (Session ran on a different workstation; a +fresh agent should read this entry + `infra/PHASE1-STATUS.md` + `docs/plans/` and start at Next Steps.) + +### LIVE STATE (verified 2026-08-31, all green) +- 3× k3s **v1.36.4+k3s1**, Debian 13, **8 vCPU / 24 GiB** each (raised from 6 — see #5). Proxmox host 24 cores, ~idle. +- Cilium 1.20.1 (GW-API v1.6.1), Longhorn 1.12.1, cert-manager 1.21, Velero chart 12/v1.18, kube-prometheus-stack 88 + (Grafana 13.2), kube-vip v1.2.3, **Loki chart 7.x / v3.6.11**, **Kyverno chart 3.9 / v1.19** (the last two were the + 08-30 "deferred" items — now done). etcd on the dedicated SSD virtio2 disk per node. +- All Flux kustomizations + HelmReleases Ready; 0 not-ready pods. +- **SSO end-to-end**: native OIDC for grafana/gitea/nextcloud/semaphore/headlamp; the 5 forward-auth UIs + (pihole/jackett/qbittorrent/radarr/sonarr) gated by per-app oauth2-proxy sidecars (fail-closed, 302→Keycloak). + Platform-admins require TOTP MFA. +- **Gitea Actions CI is LIVE + enforcing**: `.gitea/workflows/gate.yml` runs `make -C ci gate` on push/PR to main; + required status check; runner `incluster-dind` online (ns gitea-actions). + +### What changed this session (durable) +1. **SSO cluster-wide break→fix — root cause Cilium CVE-2025-30162.** The 1.16→1.20 security fix makes the shared + Gateway L7 proxy (`enforce_policy_on_l7lb`) re-check a pod's egress policy against Keycloak's REAL backend + identity on the split-horizon hairpin; `toEntities: world` no longer covers it → all app SSO 403'd + ("Access denied", `server: envoy`). Fix: explicit `toEndpoints` egress allow to the keycloak backend — + `catalog/render.py` emits it for `auth.mode ∈ {oidc,forward-auth}`; headlamp/crafty hand-written. (68690e5, 8a424bc) +2. **Keycloak MFA config bug**: browser-mfa conditional-OTP used a non-existent config key → "Invalid role name: null" + → 403 for every platform-admin login. Fixed to `condUserRole: platform-admin` in git + live via kcadm. (45deac9) +3. **VPN egress restored**: the Debian 12→13 re-provision wiped k3s-dev-3's node-host OpenVPN (PIA); re-ran + `vpn.egress.gateway.yml`. Re-provision runbook (infra/ansible/README.md) updated — node-host roles (k3s, kube-vip, + vpn.egress.gateway) must be re-run after any reprovision. +4. **Grafana OOM** (kps 65→88 ships Grafana 13) — the hidden blocker that stalled the whole kps upgrade + platform + tier. limits 256→512→768Mi; Prometheus 2→3Gi. (b2b1aa5 and follow-ups) +5. **Node vCPU 6→8** — steady CPU PSI was 15-23% / load at-or-over core count (no spike headroom); now PSI ~4.5-7.7%. + bpg reboots the VM on a cores change → rolled ONE node at a time, waiting Longhorn healthy between. (676c37a) + + node CPU PSI/load alerts + a "Nodes — CPU/Memory Pressure" Grafana dashboard. (a720c73) +6. **Security: 5 forward-auth UIs gated** with per-app oauth2-proxy sidecars (Option B, crafty's proven pattern) + + per-app Keycloak clients + SOPS creds. Closes the un-gated-admin-UI exposure. (72d4720, b2752ae) +7. **Gitea Actions CI finished**: `ci/touched-requirements.sh` now excludes comment-only R-ID prose, so the workflow + runs the exact aggregate `make -C ci gate`. (27bc024) +8. **external-snapshotter vendored** — was remote github kustomize bases; the kustomize-controller's per-reconcile + `git fetch github.com` flaked past its 27s timeout → `BuildFailed` stalled the platform tier. Now local + `crds.yaml` + `snapshot-controller.yaml` (offline, deterministic build). (7a79cea) +9. **08-30 revalidations closed**: ImageVolume gate is now ON in 1.36 → conformance rationale comments corrected + (d60e553); external-snapshotter v8.2.0 pin kept (CRDs served at v1, works on 1.36); Cilium native ExternalAuth + evaluated → **DEFER** (see plan). + +### Operational gotchas that STILL matter (for the next PM) +- **Access**: `./infra/devcontainer/dev.sh run ` (kubectl/flux/helm/tofu/ansible/kcadm). Do NOT run `dev.sh`/`make + gate` concurrently — execs into the one container interfere → transient failures, just retry. Deploy = `git push + gitea main` → Flux. Gate before push: `dev.sh run make -C ci gate` → "MERGE GATE: PASS". +- **NAT topology** (see PHASE1-STATUS): API via `192.168.2.50:6443`→VIP .1.49; node SSH `192.168.2.102:225X`; + Proxmox API `192.168.2.102:8006`. A single node reboot can briefly drop the kubeconfig NAT — wait ~3 min for boot. +- **Keycloak realm import is ONE-SHOT**: the operator logs "Job already executed - not recreating" — a git change to + `keycloak-realm-import.yaml` does NOT re-import. Edit the live realm via `kcadm.sh` in keycloak-0 (admin creds: + secret `keycloak-admin-credentials`; kcadm present, curl/python3 are NOT), AND fix git for reproducibility. +- **Cilium world-hairpin (CVE-2025-30162)**: any in-cluster→`*.home` call hairpins through the Gateway and gets its + egress re-checked against the DESTINATION's real cluster identity — `toEntities: world` won't cover it. render.py + handles the keycloak case for SSO apps; a NEW app→app-by-hostname dependency needs an explicit `toEndpoints` allow. +- **Remote github kustomize bases flake** (27s git-fetch timeout). external-snapshotter is now vendored; + **platform/kubevirt still uses remote bases** — vendor it the same way if it `BuildFail`s. +- **Longhorn on the HDD**: every node reboot degrades all 18 volumes → full rebuild (slow; 30%+ Proxmox IO-delay). + Roll reboots one node at a time; wait `robustness=healthy` before the next. Fix = `docs/plans/longhorn-ssd-tiering.md`. +- **Gitea** can go read-only (ext4 remount-ro) during node churn → `kubectl -n gitea rollout restart deploy/gitea`. + +### NEXT STEPS (prioritized) — where to start +1. **Longhorn → SSD hot-tier move** — `docs/plans/longhorn-ssd-tiering.md`. The "won't fit" blocker was a measurement + artifact (working set ~79 GB; SSD ~323 GB free). Moving the hot tier (Prometheus/CNPG DBs/Loki/Gitea ≈ 51 GB) off + the HDD removes the reboot-rebuild tax on every future maintenance op. **Highest-leverage infra item.** Pre-flight: + re-verify live SSD free + read HDD SMART. +2. **L2/L3 conformance in CI** — `docs/plans/l2-l3-conformance-ci-wiring.md`. Needs the trigger-credential decision + (scoped in-cluster SA + Kyverno-pinned Job shape; never a broad kubeconfig in CI). The gate runs no-cluster stages today. +3. **Forward-auth long-term** — `docs/plans/cilium-native-externalauth-evaluation.md` (DEFER now: experimental CRD + channel + fail-open history). The 5 UIs are safely sidecar-gated in the meantime. +4. **Remaining Phase-6 onboarding** — see DESIGN-waves.md / `catalog/` (services rendered-but-not-onboarded). +5. **Vendor platform/kubevirt** remote bases proactively (same flake class as #8). +6. **Human action**: finish Headlamp login = complete TOTP enrollment in the browser. diff --git a/infra/PHASE1-STATUS.md b/infra/PHASE1-STATUS.md index 1a98ffc..eb18fe5 100644 --- a/infra/PHASE1-STATUS.md +++ b/infra/PHASE1-STATUS.md @@ -1,9 +1,13 @@ # Dev cluster — current state & reconnect guide -**Last verified: 2026-08-30.** Phase 1 is **complete and long since exited**; the cluster has been +**Last verified: 2026-08-31.** Phase 1 is **complete and long since exited**; the cluster has been running the full platform (phases 2–6) for some time. This doc is how a session — human or agent — reconnects to it and re-establishes the toolchain. +> **New PM / agent taking over?** Read this doc (reconnect + toolchain), then the **latest +> `DECISIONS-LOG.md` entry (2026-08-31)** — it is the current-state + gotchas + prioritized **Next +> Steps** handoff — and the plans under `docs/plans/`. Start at Next Steps. + > **History note.** The cluster was originally 3 libvirt/KVM VMs on the workstation > (`10.73.0.0/24` NAT). That was **retired** after the workstation ran out of RAM/disk; the cluster > now lives on **Proxmox VE**. Anything mentioning `virsh`, `qemu:///system`, `tofu`+`dmacvicar/libvirt`, @@ -15,7 +19,7 @@ reconnects to it and re-establishes the toolchain. | | | |---|---| | Substrate | Proxmox VE host, node `proxmox`, disks on the `Storage` ZFS datastore, bridge `vmbr0` | -| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 6 vCPU, 24 GiB, Debian 13 (trixie) | +| Nodes | `k3s-dev-1/2/3` at `192.168.1.50 / .51 / .52` — 8 vCPU, 24 GiB, Debian 13 (trixie) | | k3s | **v1.36.4+k3s1**, 3 servers, embedded etcd (HA); no flannel / kube-proxy / Traefik | | CNI | Cilium 1.20.1 (kube-proxy replacement, L2 announcements, egress gateway) | | kube-API | kube-vip VIP **`192.168.1.49:6443`**, tls-san'd on the serving cert |