Merge Phase-3/4 + infra-recovery log

DECISIONS-LOG.md

@@ -381,6 +381,46 @@ alpine/k8s) once tested, or a newer kyverno chart that drops the bitnami depende
   **Conformance green now: R-NET-01/02 (L2+L3), R-TLS-01/03, R-DNS-02, R-STATE-01.** Remaining tests need
   later-phase components (VPN P3, monitoring/backup/updates P4, SSO P6) → operator secrets.
 
+## Phase 3 + 4 ("Both") + an infra recovery (2026-08-17, main @ 4f17c53)
+
+Operator: WireGuard preferred (I confirmed PIA supports it on all regions + Gluetun derives the WG
+config from username/password, so the PIA secret shape is unchanged); keycloak creds added; oauth2-proxy
+deferred to Phase 6; "proceed to next phase." Chose **Both** (deploy Phase-4 monitoring + author Phase-3 VPN).
+
+### INFRA RECOVERY — cluster VMs + Gitea were found STOPPED (not my doing)
+Mid-task, kubectl + git push failed: the 3 k3s VMs were **shut off** and the Gitea container **stopped**
+(host up 12.5h — not a reboot; something halted them). Recovered via the operator-provisioned
+ansible-become path: `virsh -c qemu:///system start k3s-dev-{1,2,3}` → cluster back, all Kustomizations +
+HelmReleases Ready from disk in ~80s; restarted Gitea (named volume `gitea-bootstrap-data` persisted) and
+pushed the queued commits. No data lost. (The VMs are SYSTEM-libvirt — user-session virsh shows nothing;
+recovery needs sudo, which only the ansible vault-become has.)
+
+### Phase 4 — MONITORING DEPLOYED + healthy
+Enabled `monitoring` in platform/kustomization.yaml. All 3 HelmReleases Ready (kube-prometheus-stack
+65.8.1, Loki 6.55.0, Alloy 0.12.6); Prometheus + Loki PVCs bound on Longhorn (5Gi each); 38
+PrometheusRules applied; no unhealthy pods. Grafana admin + Alertmanager SMTP from the operator's SOPS
+secrets. **SMTP caveats (flagged, not blocking):** operator set smtp_smarthost=smtp.gmail.com:587,
+[email protected], [email protected]; I set the receiver `to:` to a valid
+inbox (was REPLACE-ME). Gmail needs an **App Password** and the From must be the auth user or a verified
+send-as alias, else delivery fails at alert time (stack health unaffected).
+
+### FINDING F-MON1 — monitoring PrometheusRule CRD ordering (FIXED)
+Enabling monitoring first deadlocked the platform Kustomization: `alerts.yaml` (PrometheusRule CRs) needs
+the monitoring.coreos.com CRDs the kube-prometheus-stack HelmRelease installs in the SAME pass → atomic
+'no matches for kind PrometheusRule' → NO monitoring HRs created. Same controllers/configs fix
+(F-DNS1/kyverno lineage): moved `alerts.yaml` to platform-config (dependsOn platform). This is now a
+RECURRING pattern — any component shipping CRs of a CRD it installs must split the CRs into platform-config.
+
+### Phase 3 — VPN egress gateway AUTHORED (author-first, not deployed)
+`platform/vpn-egress/`: single shared **Gluetun** gateway (PIA/WireGuard, kill-switch FIREWALL=on, native
+PF, control-API Service), non-app namespace (NET_ADMIN + /dev/net/tun), pre-wired (commented)
+pia-credentials line. Defined the PIA secret location (**platform/vpn-egress**, keys username/password →
+OPENVPN_USER/OPENVPN_PASSWORD) and updated the SOPS template. NOT wired into the graph. **Open design
+(README):** egress steering (`vpn`-class pods → the one shared tunnel) — recommended Cilium Egress Gateway
+→ the Gluetun node's `wg0` (fail-closed when the tunnel drops), deliberately NOT committed as an untested
+CiliumEgressGatewayPolicy. Deploy needs PIA creds + `PIA_REGION` (PF-capable) + `CLUSTER_POD_CIDR` +
+steering validation + the qbittorrent-natmap port-sync helper. **Cluster fully green** (4/4 KS, 8/8 HR).
+
 ## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed)
 | # | Issue | Impact | Fix |
 |---|---|---|---|