diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index 01a5b13..bc2e03c 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -381,6 +381,46 @@ alpine/k8s) once tested, or a newer kyverno chart that drops the bitnami depende **Conformance green now: R-NET-01/02 (L2+L3), R-TLS-01/03, R-DNS-02, R-STATE-01.** Remaining tests need later-phase components (VPN P3, monitoring/backup/updates P4, SSO P6) → operator secrets. +## Phase 3 + 4 ("Both") + an infra recovery (2026-08-17, main @ 4f17c53) + +Operator: WireGuard preferred (I confirmed PIA supports it on all regions + Gluetun derives the WG +config from username/password, so the PIA secret shape is unchanged); keycloak creds added; oauth2-proxy +deferred to Phase 6; "proceed to next phase." Chose **Both** (deploy Phase-4 monitoring + author Phase-3 VPN). + +### INFRA RECOVERY — cluster VMs + Gitea were found STOPPED (not my doing) +Mid-task, kubectl + git push failed: the 3 k3s VMs were **shut off** and the Gitea container **stopped** +(host up 12.5h — not a reboot; something halted them). Recovered via the operator-provisioned +ansible-become path: `virsh -c qemu:///system start k3s-dev-{1,2,3}` → cluster back, all Kustomizations + +HelmReleases Ready from disk in ~80s; restarted Gitea (named volume `gitea-bootstrap-data` persisted) and +pushed the queued commits. No data lost. (The VMs are SYSTEM-libvirt — user-session virsh shows nothing; +recovery needs sudo, which only the ansible vault-become has.) + +### Phase 4 — MONITORING DEPLOYED + healthy +Enabled `monitoring` in platform/kustomization.yaml. All 3 HelmReleases Ready (kube-prometheus-stack +65.8.1, Loki 6.55.0, Alloy 0.12.6); Prometheus + Loki PVCs bound on Longhorn (5Gi each); 38 +PrometheusRules applied; no unhealthy pods. Grafana admin + Alertmanager SMTP from the operator's SOPS +secrets. **SMTP caveats (flagged, not blocking):** operator set smtp_smarthost=smtp.gmail.com:587, +from=alertmanager@keithjackson.dev, user=keithallenjackson@gmail.com; I set the receiver `to:` to a valid +inbox (was REPLACE-ME). Gmail needs an **App Password** and the From must be the auth user or a verified +send-as alias, else delivery fails at alert time (stack health unaffected). + +### FINDING F-MON1 — monitoring PrometheusRule CRD ordering (FIXED) +Enabling monitoring first deadlocked the platform Kustomization: `alerts.yaml` (PrometheusRule CRs) needs +the monitoring.coreos.com CRDs the kube-prometheus-stack HelmRelease installs in the SAME pass → atomic +'no matches for kind PrometheusRule' → NO monitoring HRs created. Same controllers/configs fix +(F-DNS1/kyverno lineage): moved `alerts.yaml` to platform-config (dependsOn platform). This is now a +RECURRING pattern — any component shipping CRs of a CRD it installs must split the CRs into platform-config. + +### Phase 3 — VPN egress gateway AUTHORED (author-first, not deployed) +`platform/vpn-egress/`: single shared **Gluetun** gateway (PIA/WireGuard, kill-switch FIREWALL=on, native +PF, control-API Service), non-app namespace (NET_ADMIN + /dev/net/tun), pre-wired (commented) +pia-credentials line. Defined the PIA secret location (**platform/vpn-egress**, keys username/password → +OPENVPN_USER/OPENVPN_PASSWORD) and updated the SOPS template. NOT wired into the graph. **Open design +(README):** egress steering (`vpn`-class pods → the one shared tunnel) — recommended Cilium Egress Gateway +→ the Gluetun node's `wg0` (fail-closed when the tunnel drops), deliberately NOT committed as an untested +CiliumEgressGatewayPolicy. Deploy needs PIA creds + `PIA_REGION` (PF-capable) + `CLUSTER_POD_CIDR` + +steering validation + the qbittorrent-natmap port-sync helper. **Cluster fully green** (4/4 KS, 8/8 HR). + ## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed) | # | Issue | Impact | Fix | |---|---|---|---|