Retire gluetun gateway + retarget VPN tunnel monitoring to the node tunnel (ADR-0015)

Operator-approved. Removes the vestigial in-cluster gluetun Deployment/Service (the node-level host tunnel steered by Cilium Egress Gateway is validated + conformance-green) and updates the VPNTunnelDown alert to key on node tun0 presence (absent(node_network_receive_bytes_total{device=tun0})) instead of the now-gone gluetun_vpn_up gauge; drops the obsolete pod-based EgressGatewayUnhealthy. Unused pia-* SOPS secrets retained pending a separate cleanup.
2026-08-24 14:43:03 -05:00
commit 9927b73a5a
@@ -213,11 +213,14 @@ spec:
  groups:
    - name: platform-health.networking
      rules:
-        # VPN tunnel health. The concrete metric depends on the egress-gateway VPN sidecar chosen by
-        # WS-NET (docs/03). Written against a gluetun-style up gauge; confirm/adjust at bring-up.
+        # VPN tunnel health (ADR-0015: node-level host OpenVPN tunnel on the egress-gateway node, steered
+        # by Cilium Egress Gateway — there is NO in-cluster gateway pod). OpenVPN creates/destroys the tun0
+        # device with the connection, so the tun0 netdev series exists iff the tunnel is up. node_network_up
+        # is unreliable here (tun0 operstate=unknown => 0), so key on the byte-counter's PRESENCE. absent()
+        # also fires if the gateway node / its node-exporter is down (the tun0 series goes stale), covering
+        # gateway-node-down as well. Metric confirmed live on the dev cluster 2026-08-24.
        - alert: VPNTunnelDown
-          # TODO(WS-NET): confirm metric name of the egress VPN exporter (docs/03).
-          expr: max(gluetun_vpn_up) == 0 or absent(gluetun_vpn_up)
+          expr: absent(node_network_receive_bytes_total{device="tun0"})
          for: 5m
          labels:
            severity: critical
@@ -225,16 +228,11 @@ spec:
            summary: "Egress VPN tunnel down"
            description: "The egress-gateway VPN tunnel is down (or its metric is absent) for 5m — outbound traffic could leak or stall (R-SEC-03 leak risk)."
            runbook_url: "https://home.keithjackson.dev/runbooks/vpn-down"
-        - alert: EgressGatewayUnhealthy
-          expr: |
-            max by (pod) (kube_pod_status_ready{namespace="egress", condition="true"}) == 0
-          for: 10m
-          labels:
-            severity: critical
-          annotations:
-            summary: "Egress gateway unhealthy"
-            description: "No ready egress-gateway pod for 10m — VPN-scoped egress is unavailable."
-            runbook_url: "https://home.keithjackson.dev/runbooks/egress-gateway"
+        # (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
+        # retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
+        # service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
+        # above (tun0 presence, which also goes absent if the gateway node is down); Cilium datapath
+        # health is covered by the Cilium platform alerts.
        - alert: HubbleDeniedFlowSpike
          # Cilium/Hubble denied (policy-drop) flows. Requires hubble metrics enabled (docs/03).
          expr: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5
@@ -1,126 +0,0 @@
---
-# Gluetun — the single shared in-cluster VPN egress gateway (ADR-0013): PIA over OpenVPN-UDP, with
-# Gluetun's built-in kill-switch/firewall (fail-closed), health checks, auto-reconnect. `vpn`-class
-# pods are steered through this gateway (see README.md "Egress steering").
-#
-# CUSTOM-PROVIDER mode (DECISIONS-LOG "Phase 3 VPN — custom .ovpn"): gluetun's PIA-provider server
-# selection random-picks server IPs, many of which don't answer this cluster's flows (intermittent
-# TLS timeouts — verified by packet capture: the pod CAN complete a full handshake, only against
-# responsive servers). So we feed gluetun the operator's own PIA `.ovpn`, which pins a known-healthy
-# Stockholm server IP (gluetun's custom parser requires an IP `remote`, not a hostname). Trade-off:
-# custom mode loses gluetun's PIA-native port forwarding (Phase-6/qBittorrent concern) — revisit then.
-#
-# Credentials: `pia-credentials` SOPS Secret (username/password) → OPENVPN_USER/OPENVPN_PASSWORD (the
-# .ovpn uses auth-user-pass). Config: `pia-custom-ovpn` SOPS Secret (key `pia.ovpn`), mounted at
-# /config/pia.ovpn. Both operator-provisioned.
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: gluetun
-  namespace: vpn-egress
-  labels:
-    app: gluetun
-spec:
-  replicas: 1                 # ONE shared tunnel (ADR-0013). Active/standby is a later enhancement.
-  strategy:
-    type: Recreate           # a tunnel is a singleton — never run two overlapping during a rollout
-  selector:
-    matchLabels:
-      app: gluetun
-  template:
-    metadata:
-      labels:
-        app: gluetun
-    spec:
-      containers:
-        - name: gluetun
-          image: qmcgaw/gluetun:v3.41.3        # pinned; Renovate bumps.
-          securityContext:
-            capabilities:
-              add: ["NET_ADMIN"]               # required for the tunnel + firewall
-          env:
-            - name: VPN_SERVICE_PROVIDER
-              value: "custom"                # custom .ovpn — see header + DECISIONS-LOG
-            - name: VPN_TYPE
-              value: "openvpn"
-            - name: OPENVPN_CUSTOM_CONFIG
-              value: "/config/pia.ovpn"      # the operator's PIA .ovpn (pia-custom-ovpn Secret, IP remote)
-            # PIA username/password for the .ovpn's auth-user-pass (pia-credentials SOPS Secret):
-            - name: OPENVPN_USER
-              valueFrom:
-                secretKeyRef:
-                  name: pia-credentials
-                  key: username
-            - name: OPENVPN_PASSWORD
-              valueFrom:
-                secretKeyRef:
-                  name: pia-credentials
-                  key: password
-            # Pod MTU is 1450 (Cilium VXLAN: 1500 − 50 encap). Shrink OpenVPN's tun-mtu so data-channel
-            # packets fit (the .ovpn likely defaults to 1500). Hardcoded strings (int-coercion gotcha).
-            - name: OPENVPN_MTU
-              value: "1300"
-            - name: OPENVPN_MSSFIX
-              value: "1300"
-            - name: FIREWALL
-              value: "on"                      # KILL-SWITCH: block all egress except the tunnel (fail-closed, R-NET-06)
-            - name: FIREWALL_OUTBOUND_SUBNETS
-              value: "${CLUSTER_POD_CIDR}"     # allow steered client pods (pod CIDR) to route out via the tunnel
-            - name: DOT
-              value: "off"                     # DNS via cluster CoreDNS, not Gluetun's DoT
-          ports:
-            - name: control
-              containerPort: 8000              # Gluetun control API (health + tunnel-state lookup)
-          livenessProbe:
-            # EXEC (localhost) not httpGet: once the tunnel redirects the default route via tun0, the pod's
-            # response to a kubelet HTTP probe would egress the tunnel and never return → probe times out →
-            # needless restarts. A local wget stays on lo, unaffected by the VPN routing.
-            exec:
-              command: ["/bin/sh", "-c", "wget -qO- http://127.0.0.1:8000/v1/vpn/status | grep -q running"]
-            initialDelaySeconds: 30
-            periodSeconds: 30
-            timeoutSeconds: 5
-          resources:
-            requests:
-              cpu: 50m
-              memory: 64Mi
-            limits:
-              memory: 160Mi
-          volumeMounts:
-            - name: tun
-              mountPath: /dev/net/tun
-            - name: gluetun-data
-              mountPath: /gluetun            # gluetun runtime state
-            - name: custom-ovpn
-              mountPath: /config             # the .ovpn (read-only)
-              readOnly: true
-      volumes:
-        - name: tun
-          hostPath:
-            path: /dev/net/tun
-            type: CharDevice
-        - name: gluetun-data
-          emptyDir: {}
-        - name: custom-ovpn
-          secret:
-            secretName: pia-custom-ovpn
-            items:
-              # operator's secret keys the .ovpn by filename; map it to /config/pia.ovpn
-              - key: sweden-aes-128-cbc-udp-ip.ovpn
-                path: pia.ovpn
---
-# Control API Service — used for tunnel-state health/alerting (ADR-0013 "alert on tunnel state").
-apiVersion: v1
-kind: Service
-metadata:
-  name: gluetun-control
-  namespace: vpn-egress
-  labels:
-    app: gluetun
-spec:
-  selector:
-    app: gluetun
-  ports:
-    - name: control
-      port: 8000
-      targetPort: 8000
@@ -3,7 +3,11 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
  - namespace.yaml
-  - gluetun-gateway.yaml
+  # gluetun-gateway.yaml RETIRED 2026-08-24 (ADR-0015): the node-level host tunnel is validated +
+  # conformance-green (egress-class + vpn-killswitch pass live); a Gluetun *pod* can't be a Cilium
+  # egress-gateway target anyway. Steering now flows through the node tun0 via egress-steering.yaml.
+  # NOTE: pia-credentials/pia-custom-ovpn secrets below are now UNUSED (creds are host-side in the
+  # Ansible vault per docs/07) — retained temporarily, pending a follow-up cleanup change.
  - pia-credentials.sops.yaml
-  - pia-custom-ovpn.sops.yaml   # operator-provisioned: the PIA .ovpn (custom-provider mode). See README.
-  - egress-steering.yaml        # ADR-0015: steer egress-class=vpn pods to the node tun0 (host tunnel validated 2026-08-24)
+  - pia-custom-ovpn.sops.yaml   # (was gluetun's PIA .ovpn; superseded host-side — see note above)
+  - egress-steering.yaml        # ADR-0015: steer egress-class=vpn pods to the node tun0 (validated 2026-08-24)