feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers)

Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/** , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-05 22:06:18 -05:00
1 parent 22c94effff
commit 1a67fc1b3c
@@ -102,6 +102,7 @@ data:
  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
  # plex-remote + pihole: PLEX_L4_VIP),
+  # vpn-ingress (WG_CLIENT_CIDR),
  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
  # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -207,6 +208,31 @@ data:
  # prod -> 192.168.2.0/24  (prod LAN; nodes .10/.11/.12)
  LAN_CIDR: "192.168.1.0/24"                 # ingress scope for the PiHole DNS listener

+  # --- Remote-access VPN client subnet (platform/vpn-ingress, WireGuard) --------------------------
+  # The tunnel subnet handed to REMOTE VPN clients. `.1` is the server (the hostNetwork DaemonSet,
+  # on every node); `.2`-`.254` are devices, allocated one /32 each in
+  # platform/vpn-ingress/peers.yaml. Consumed twice in the pod's config-assembly script: to derive
+  # the server's own address, and as the `-s` match of the MASQUERADE rule that makes LAN replies
+  # routable.
+  #
+  # IT IS A KNOB RATHER THAN A LITERAL FOR ONE REASON: it is the only address range in this cluster
+  # that must be checked against EVERY OTHER range at once, so it belongs where those are declared.
+  # 10.90.0.0/24 was chosen to collide with none of them:
+  #     10.0.0.0/16     Cilium cluster-pool pod CIDR     (${CLUSTER_POD_CIDR})
+  #     10.43.0.0/16    k3s service CIDR                 (measured live 2026-10-05: kubernetes svc
+  #                                                       clusterIP 10.43.0.1)
+  #     10.72.192.0/18  the PIA tunnel's pushed addresses (platform/vpn-egress)
+  #     192.168.1.0/24  dev LAN / pfSense LAN            (${LAN_CIDR} base default)
+  #     192.168.2.0/24  prod LAN                         (${LAN_CIDR} on prod)
+  #
+  # NO PER-ENVIRONMENT OVERRIDE, and that is deliberate rather than an omission: unlike every VIP
+  # above, this range is NOT drawn from the LAN. It is private to the tunnel, so the same value is
+  # correct in both environments and a prod patch would be a no-op that implies otherwise.
+  #
+  # IF YOU CHANGE IT, every already-issued client profile breaks (its `Address` is from this range)
+  # and must be re-rendered — so treat it as fixed once the first device is enrolled.
+  WG_CLIENT_CIDR: "10.90.0.0/24"             # remote VPN clients; .1 = server. NOT a LAN range
+
  # --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
  # THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
  # library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
@@ -71,4 +71,11 @@ resources:
  # controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
  # Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
  - backup.yaml          # Flux Kustomization -> ./platform/velero (dependsOn platform-config)
+  # WireGuard remote-access VPN (north-south INGRESS, udp/51820) — POST-platform tier
+  # (dependsOn platform-config, wait:false). NOT the same thing as platform/vpn-egress, which sends
+  # pods OUT through PIA; this lets remote CLIENTS IN. Non-gating is load-bearing rather than
+  # conventional here: its DaemonSet cannot start until the operator creates the out-of-band
+  # `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have
+  # stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch.
+  - vpn-ingress.yaml     # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config)
  - cluster-vars.yaml    # overlay knob surface (base defaults)
@@ -0,0 +1,56 @@
+---
+# Flux Kustomization: WireGuard remote-access VPN — reconciles ./platform/vpn-ingress.
+#
+# POST-PLATFORM TIER (dependsOn platform-config), NON-GATING (wait:false). Not a line in
+# platform/kustomization.yaml, for the standard reason plus one specific to this component:
+#
+#   * STANDARD: a leaf must never sit in the gating `platform` tier. While `platform` is not-Ready
+#     NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
+#     (DECISIONS-LOG: "Headlamp gating stall").
+#   * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
+#     operator creates the out-of-band `wg-server-key` Secret (runbooks/vpn-remote-access.md
+#     step 1). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
+#     the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
+#     cause (a missing manual step) invisible from any Flux condition.
+#
+# WHY dependsOn platform-config AND NOT cilium. This component needs no Gateway, no TLS cert, no
+# Keycloak and no LB-IPAM pool — it has no HTTPRoute and no LoadBalancer Service, and hostNetwork
+# means it does not even need a Cilium endpoint. It could legally reconcile much earlier. It is
+# pinned behind platform-config anyway so that the ONE thing remote clients actually consume
+# through it — split-horizon DNS and the shared Gateway — exists by the time the tunnel does.
+# A tunnel that comes up before the things on the far side of it is a support call, not a feature.
+#
+# wait:false: nothing in the graph dependsOn this, and its pods are legitimately not-Ready whenever
+# the server key is absent or the `wireguard` kernel module has not been loaded on a node. Neither
+# condition should be able to gate anything else, even if a future resource is mistakenly pointed
+# at it.
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: vpn-ingress
+  namespace: flux-system
+spec:
+  interval: 10m
+  retryInterval: 2m
+  timeout: 5m
+  prune: true
+  wait: false
+  dependsOn:
+    - name: platform-config
+  sourceRef:
+    kind: GitRepository
+    name: flux-system
+  path: ./platform/vpn-ingress
+  # REQUIRED, not boilerplate: platform/vpn-ingress/daemonset.yaml renders ${WG_CLIENT_CIDR} into
+  # the config-assembly script. Without this block that token becomes the EMPTY STRING and the
+  # Kustomization still reports Ready (Flux's substitution fails OPEN — ci:
+  # flux-postbuild-substitution), leaving an initContainer that derives a server address of ".1/"
+  # and a MASQUERADE rule with no source. Both fail loudly at pod start, but the Kustomization
+  # would look healthy.
+  postBuild:
+    substituteFrom:
+      - kind: ConfigMap
+        name: cluster-vars
+  # NO `decryption:` stanza is needed in the overlays: this path carries no *.sops.yaml. The server
+  # key is an out-of-band Secret (platform/vpn-ingress/kustomization.yaml says what to do if that
+  # ever changes).