diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index bc2e03c..0f1d254 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -446,3 +446,41 @@ Job (ADR-0011) already uses the pinned image. Deferred (non-blocking). - **T-3.2p** PIA username/password → VPN egress gateway (Phase 3). - **OQ-BKP** off-cluster backup target → Velero (Phase 4). - SMTP creds → Alertmanager email (Phase 4). + +## Disk/drive incident + k3s relocated to NVMe (2026-08-22) +**Trigger:** host root fs (Kingston SSD `/dev/sdb3`, btrfs) hit ~99% (<1GB); the system paused the k3s +VMs on I/O error. Operator asked to move the VM disks off root to a larger drive. + +**Why root filled:** the k3s VM disks lived under the snapper-snapshotted root subvolume; high VM-disk +churn multiplied across hourly snapshots. Freed by deleting the 11 `Snapper` timeline snapshots + +`virsh destroy`-ing the paused VMs (which held the old, rsync-unlinked inodes open) → btrfs reclaim. +Root 99% → 30% (326G free). + +**F-DISK1 — the Seagate 4TB HDD is FAILING.** First relocated the VM disks to the HDD (`/mnt/seagate`); +etcd then hit **48% iowait + constant "apply took too long"**, pods drifted to Unknown, never converged. +SMART on `/dev/sda` (ST4000DM000): **29,408 reallocated, 448 current-pending + 448 offline-uncorrectable**, +~25k power-on hrs. A dying drive, not "HDD is slow." Do NOT use for etcd/critical data. + +**Resolution:** k3s disks moved to the **WD_BLACK SN770 NVMe** (`/dev/nvme0n1p3`, `/mnt/games`) in a +dedicated btrfs subvol `/mnt/games/libvirt-k3s` (nodatacow), **bind-mounted to `/var/lib/libvirt/images`** +so tofu/libvirt paths + the qcow2 backing chain are unchanged (no IaC edits). etcd iowait → 0%, cluster +fully converged (64 Running). fstab updated; SELinux relabeled. Archival VMs (rocky10/silverblue/win11) +repointed to their real HDD path and LEFT on the failing drive. + +**Follow-ups (assumptions logged):** (1) the NVMe bind-mount is a runtime change — codify in an Ansible +`host-prereqs` task for reproducibility; (2) redundant k3s copies remain at `/mnt/seagate/libvirt-images/k3s-dev-*` +(safe to delete; NVMe authoritative) — left pending operator OK; (3) archival VMs on the failing HDD need +migration + the disk replaced. + +## Phase 3 VPN — protocol resolved to OpenVPN-UDP; deploying (2026-08-22) +ADR-0013 said PIA/WireGuard, but **Gluetun does not support PIA over WireGuard** (upstream: native support +"cannot be added… slow WIP"; live error "provider name is not valid for Wireguard"). **Operator (as System +Architect) approved OpenVPN-UDP and ADR-0013 was amended in the spec repo** (protocol only; architecture +unchanged; `thrnz/docker-wireguard-pia` recorded as the WireGuard fallback). Impl aligned: `VPN_TYPE=openvpn`, +`vpn-egress` ENABLED in `platform/kustomization.yaml`, stale WireGuard comments cleaned (gluetun-gateway.yaml, +README; steering iface `wg0`→`tun0`). Pushed to Gitea → Flux reconciles the OpenVPN gateway. Stale WireGuard +comments remain inside the operator's encrypted `pia-credentials.sops.yaml` header — left untouched (editing +breaks the SOPS MAC); cosmetic. + +**Next:** confirm the OpenVPN tunnel comes up + correct egress IP, then egress-steering validation (README +option 1, `tun0`) + kill-switch conformance (R-NET-06). diff --git a/platform/kustomization.yaml b/platform/kustomization.yaml index 53f171a..9d8bf83 100644 --- a/platform/kustomization.yaml +++ b/platform/kustomization.yaml @@ -17,8 +17,8 @@ resources: - cert-manager # Let's Encrypt DNS-01 wildcard — needs SOPS `cloudflare-api-token` - externaldns # auto DNS records — needs SOPS `cloudflare-api-token` - gateway # Gateway API listeners (ingress; replaces Traefik) - # ---- Phase 3 — Egress / VPN (PIA creds provisioned via SOPS) ---- - - vpn-egress # shared Gluetun PIA/WireGuard egress gateway + kill-switch (ADR-0013) + # ---- Phase 3 — Egress / VPN (PIA creds provisioned via SOPS; OpenVPN-UDP per ADR-0013 amendment) ---- + - vpn-egress # shared Gluetun PIA/OpenVPN-UDP egress gateway + kill-switch (ADR-0013) # ---- Phase 4 — Observability (creds provisioned via SOPS: grafana-admin, alertmanager-smtp) ---- - monitoring # kube-prometheus-stack + Loki + Alloy + Alertmanager # ---- Later phases: enabled when their phase opens ---- diff --git a/platform/vpn-egress/README.md b/platform/vpn-egress/README.md index c873f90..ab3cc9a 100644 --- a/platform/vpn-egress/README.md +++ b/platform/vpn-egress/README.md @@ -1,17 +1,18 @@ # platform/vpn-egress — shared VPN egress gateway (Phase 3, ADR-0013) -Single shared **Gluetun** gateway: **PIA over WireGuard**, built-in kill-switch, native port -forwarding. `vpn`-class services (sonarr, radarr, jackett, qbittorrent) egress **only** through this -tunnel; `internet`-class services go out the normal WAN. Serves **R-NET-05/06/07**. +Single shared **Gluetun** gateway: **PIA over OpenVPN-UDP** (ADR-0013 amendment — Gluetun does not +support PIA over WireGuard), built-in kill-switch, native port forwarding. `vpn`-class services +(sonarr, radarr, jackett, qbittorrent) egress **only** through this tunnel; `internet`-class services +go out the normal WAN. Serves **R-NET-05/06/07**. -> **STATUS: authored, NOT deployed.** Not wired into `platform/kustomization.yaml` yet. Deploy needs -> (a) the `pia-credentials` SOPS secret, and (b) validation of the egress-steering mechanism below, -> which cannot be tested without a live PIA tunnel. +> **STATUS: enabled in `platform/kustomization.yaml` (Phase 3).** `pia-credentials` SOPS secret is +> provisioned. Remaining work: validate the tunnel comes up + correct egress IP, then the +> egress-steering mechanism below (the hard part) and the kill-switch conformance test. ## Pieces - `gluetun-gateway.yaml` — the Gluetun Deployment (`replicas: 1`, `Recreate`) + the `gluetun-control` Service (control API :8000 for forwarded-port lookup + tunnel-state health). Kill-switch = `FIREWALL=on`. -- `pia-credentials` (SOPS) — PIA username/password only; Gluetun derives the WireGuard config at runtime. +- `pia-credentials` (SOPS) — PIA username/password only; Gluetun ships PIA's OpenVPN configs. ## Operator hand-off (when deploying Phase 3) 1. Provision `pia-credentials` (see `bootstrap/sops/secret-templates/pia-credentials.yaml.example`, @@ -35,8 +36,8 @@ natively route through a *pod's* tunnel. Options considered: 1. **Cilium Egress Gateway → the Gluetun node's tunnel interface (recommended).** Pin Gluetun to a node; a `CiliumEgressGatewayPolicy` selects pods labeled `homelab.keithjackson.dev/egress-class=vpn` - and egresses them via that node's WireGuard interface (`wg0`). If the tunnel drops, `wg0` is gone → - traffic has no egress path → **fail-closed** (the kill-switch behavior R-NET-06 needs). This is the + and egresses them via that node's tunnel interface (`tun0`, OpenVPN). If the tunnel drops, `tun0` + is gone → traffic has no egress path → **fail-closed** (the kill-switch behavior R-NET-06 needs). This is the most Cilium-native fit but requires Gluetun's tunnel interface to be visible to the node datapath (hostNetwork Gluetun, or a shared netns) — the detail to VALIDATE. 2. **Gateway-pod routing:** client pods default-route to the Gluetun pod IP (init-container + @@ -44,7 +45,7 @@ natively route through a *pod's* tunnel. Options considered: Cilium egress gateway but is more moving parts and pod-IP-fragile. 3. **Per-pod Gluetun sidecar** — explicitly REJECTED by ADR-0013 (single shared gateway preferred). -**Plan:** validate option 1 against a live PIA tunnel (confirm `wg0` on the Gluetun node + a +**Plan:** validate option 1 against a live PIA tunnel (confirm `tun0` on the Gluetun node + a `CiliumEgressGatewayPolicy` steering `egress-class=vpn` pods, then run the `egress-class` (R-NET-05/07) and `vpn-killswitch` (R-NET-06) conformance tests). Fall back to option 2 if the interface isn't addressable. The `CiliumEgressGatewayPolicy` is intentionally NOT committed until validated, to avoid diff --git a/platform/vpn-egress/gluetun-gateway.yaml b/platform/vpn-egress/gluetun-gateway.yaml index dda2dba..cfd5e6d 100644 --- a/platform/vpn-egress/gluetun-gateway.yaml +++ b/platform/vpn-egress/gluetun-gateway.yaml @@ -1,13 +1,13 @@ --- -# Gluetun — the single shared in-cluster VPN egress gateway (ADR-0013): PIA over WireGuard, with +# Gluetun — the single shared in-cluster VPN egress gateway (ADR-0013): PIA over OpenVPN-UDP, with # Gluetun's built-in kill-switch/firewall (fail-closed), health checks, auto-reconnect, and native # port forwarding for qBittorrent. `vpn`-class pods are steered through this gateway (see README.md # "Egress steering" — the Cilium-native mechanism, pending live validation with PIA creds). # # Credentials: the `pia-credentials` SOPS Secret (keys: username, password). Gluetun uses PIA's -# username/password (via OPENVPN_USER/OPENVPN_PASSWORD — legacy env names it uses for PIA on BOTH -# protocols) to authenticate to PIA's API and derive the WireGuard config at runtime — so WireGuard -# needs no extra key material (ADR-0013). +# username/password (via OPENVPN_USER/OPENVPN_PASSWORD) to authenticate to PIA. Gluetun ships PIA's +# OpenVPN server configs, so no manual `.ovpn` is needed (ADR-0013 amendment: OpenVPN-UDP — Gluetun +# does NOT support PIA over WireGuard). apiVersion: apps/v1 kind: Deployment metadata: @@ -29,9 +29,9 @@ spec: spec: containers: - name: gluetun - image: qmcgaw/gluetun:v3.41.0 # pinned; Renovate bumps. NOTE: PIA+WireGuard support - # requires >= v3.41 (v3.40.0 rejects "private internet - # access" as a WireGuard provider — verified at deploy). + image: qmcgaw/gluetun:v3.41.0 # pinned; Renovate bumps. PIA is OpenVPN-only in Gluetun + # (WireGuard for PIA is unsupported upstream — ADR-0013 + # amendment; thrnz/docker-wireguard-pia is the WG fallback). securityContext: capabilities: add: ["NET_ADMIN"] # required for the tunnel + firewall @@ -39,8 +39,8 @@ spec: - name: VPN_SERVICE_PROVIDER value: "private internet access" - name: VPN_TYPE - value: "openvpn" # UDP — satisfies "protocol != TCP" (R-NET); upgrade over OpenVPN - # PIA credentials from the SOPS secret (Gluetun's PIA creds env, used for WireGuard too): + value: "openvpn" # OpenVPN-UDP — satisfies "protocol != TCP" (R-NET); ADR-0013 amended + # PIA credentials from the SOPS secret (Gluetun's PIA OpenVPN creds env): - name: OPENVPN_USER valueFrom: secretKeyRef: