From 60e3f09c677127a7bb1c520d21e258e50d6f355d Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sat, 12 Sep 2026 17:38:46 -0500 Subject: [PATCH] =?UTF-8?q?feat(vpn):=20prod=20egress-gateway=20group=20?= =?UTF-8?q?=E2=80=94=20k3s-prod-3,=20singleton,=20opt-in=20only?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `[vpn_egress_gateway_prod]` is a SEPARATE inventory group with its own vaulted credentials, not a second member of `[vpn_egress_gateway]`. That group is the default target of playbooks/vpn.egress.gateway.yml, so adding a prod host to it would mean a flagless `ansible-playbook playbooks/vpn.egress.gateway.yml` silently reconfigures a prod node's kill-switch datapath. Prod is reached only by asking for it, with `--list-hosts` first: -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 Node 3 because node 1's SSD was replaced yesterday (least operational history) and node 2 carries the §0b memattr history. Exactly one node, forever: Cilium OSS picks "the first node in lexical ordering" among nodeSelector matches, so a labelled k3s-prod-1 would silently steal the gateway role — and the public egress IP — from k3s-prod-3, with no event and no alert. vault.yml is a verbatim copy of the dev group's CIPHERTEXT: same PIA account per the operator, and copying the encrypted blob means the plaintext was never decrypted to disk, through a shell, or into a transcript. The two files are now independent — rotating means re-encrypting both, which is the right behaviour for two environments. Also corrects a stale inventory note that claimed k3s-prod-3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it as the gateway, because a second live uplink in a kill-switch datapath is not a footnote: there is no wl* netdev on the node at all, no phy80211 device, the /etc/network/interfaces wlp1s0 stanza is commented out, 192.168.2.195 is assigned nowhere, and the only IPv4 addresses are lo and eno1 192.168.2.12. Nothing needed bringing down. Co-Authored-By: Claude Opus 5 --- .../vpn_egress_gateway_prod/README.md | 49 +++++++ .../vpn_egress_gateway_prod/vault.yml | 131 ++++++++++++++++++ infra/ansible/inventory/hosts.ini | 37 ++++- .../ansible/playbooks/vpn.egress.gateway.yml | 26 +++- 4 files changed, 234 insertions(+), 9 deletions(-) create mode 100644 infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md create mode 100644 infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vault.yml diff --git a/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md new file mode 100644 index 0000000..f111110 --- /dev/null +++ b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/README.md @@ -0,0 +1,49 @@ +# group_vars for the `vpn_egress_gateway_prod` node — ADR-0015 host tunnel secrets (PROD) + +Prod's egress gateway is `k3s-prod-3`, in its **own** inventory group +(`[vpn_egress_gateway_prod]`) so that the flagless +`ansible-playbook playbooks/vpn.egress.gateway.yml` — which defaults to the **dev** group — +can never touch a prod node. Prod is reached only by asking for it: + +```bash +cd infra/ansible +ansible-playbook playbooks/vpn.egress.gateway.yml \ + -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts +# then drop --list-hosts +``` + +A separate group needs its own `group_vars`, because the dev group's vault is attached to the +dev group and is not visible to a prod host. The required keys are **identical** to dev's — +see `../vpn_egress_gateway/README.md` for what each one is: + +| Vault key | +|---| +| `vault_vpn_egress_gateway_pia_username` | +| `vault_vpn_egress_gateway_pia_password` | +| `vault_vpn_egress_gateway_pia_ovpn` | + +## How this file was created (2026-09-12) + +`vault.yml` here is a **verbatim copy of the dev group's encrypted blob**. That is deliberate: + +- the operator confirmed the **same PIA account** applies to prod and is well inside its + connection limit, so the values are the same values; +- copying the *ciphertext* means the plaintext was never decrypted to disk, never passed + through a shell, and never appeared in an agent transcript. The repo-wide vault passphrase + (`ansible.cfg` → `vault_password_file`) decrypts both files identically. + +The two files are now **independent**: rotating the PIA credentials means re-encrypting +*both*, and that is the intended behaviour — prod and dev are separate environments and a +credential change should be a deliberate act per environment, not an accident of sharing a +file. If prod is ever moved to its own PIA account (recommended if the connection limit ever +bites, or to keep the public egress IPs of the two environments independent), re-create this +file alone: + +```bash +cd infra/ansible +ansible-vault create inventory/group_vars/vpn_egress_gateway_prod/vault.yml +``` + +## Notes +- `vault.yml` is skipped by the yamllint gate (`ci/yamllint.yaml` ignores `**/vault.yml`). +- **Never** run `ansible-inventory --list`: it prints DECRYPTED vault values. diff --git a/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vault.yml b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vault.yml new file mode 100644 index 0000000..1948dd7 --- /dev/null +++ b/infra/ansible/inventory/group_vars/vpn_egress_gateway_prod/vault.yml @@ -0,0 +1,131 @@ +$ANSIBLE_VAULT;1.1;AES256 +38306636313935333030303664653739316233663139616563663233643262373637306331653264 +6538363539383464353964663632633561323562383038340a396233633232356137313435336231 +36333663363130643532613566643339373365626462313537653033306363336564343139373537 +3164323463616632630a373732333536613936366463313439343664363530383664636535616261 +31306332666666643637346364386537343062383033663530346163373536323639373366333661 +36303831323662333232633034393362333561643637623033656565303461336465333664366332 +36336466656437343431363735646661643837666335633464666130643866313830366465323166 +65616537343630303431623537303766383762646533616539613235343938656237613934623565 +35373835633539653333343232626336326638303961373230663561313330313632393066323130 +31383838386262393730666231376362383132646230383964313439663935613861323137646331 +36626365323566316436386662383434366664356166636431663332393665323734623735303239 +62383031613237333136613837356166343436306137353935613661613264326466346337346530 +65356630373235313131656131383965313366613965386662343961653864373964346337363562 +66626635363438656430306231323764346564666132633966353861643262353131623134656137 +39306134626163346336613732326264396438623630656534396565343763303761333538316661 +30623466366232323531633262623434643366613833353564386163303966316537623934336163 +66323933653036366535663266666531666236646431383066616262353733316165666434616534 +35386633313464363233326663363634626239616364383866653135386435636130316335666430 +38323363386462373862366230393764386338643466363161333733653065383966663935303964 +64333461333437653433373131343736333831313236306262373631353461356237386163623166 +30663761306534316439356365396663336135333537663366353866336536663934646133333332 +65353236366435326630653830346631643236326337626435333236386634333137623138316131 +61323365303461353932626336323534633533646233653135396665373765666337333934356662 +36646339376564366439646137306364623165626263383634326337333566663236346366366334 +37316166393763663139336538663266333736623661383062383732303133326233313232323836 +34356130396131346533633336373736663366366231343162633064343565613937366434613664 +31656365303439643837323135613936646133306163663265373933633231616135353335376435 +61633965356538613636663564346566313261306235313438316639626131303566363365663764 +64316562383361373663653034393333653831666239363037343636666364666536343534613066 +61373335353464633963623161353866333038643062303931653134626334393864663461653531 +65373766336437376435616138666563623337373038623566666136626637633463323965666139 +38336139613635383066336164373061663266326235363731663565613336623936373531383966 +62323330636263306665343462666639643530336633346531636263616436356631343763343432 +61636264316566663933346163373163653734623031393962343635653765653964633137346433 +62616138333536336438363564356636316531346566663066376234356230626165386165336165 +38656166633839656438323966376561306338623237303530376231393663663661343362653664 +37613433653634363534306432383061373165613266386562323632636537373736656137633639 +33666634316465383066333737616636363866626330313232326434666164316439333433633461 +35336161313039646264613835613336326332326464313339636532653837653430393939633462 +66656563626163376134623437633332633535336233656237303962623063363134616431623235 +62323265666136323366343565656236646430393237356333613836343438353164313535616138 +34613134663465333734663337303663316161313439626332326334363835376361366463363631 +30396336303761383465373439313666656237356433303532373162306166356165316164656334 +37326163393230306161653332323462333238313166373134343839323461363936353737613662 +61643938333464366433306363633531383264613030316631303034376131323135313136316461 +31666632393631376133373533616639396361633762636663323463363433626537656333653837 +38633632623139356630616637663331616461383361383733613563383165363139343035383333 +32653563303038336137336333653561363037633634323662616666356534323138353438616463 +64383639353733333537616632346230383330356166663433373734386435376563333038333934 +37616536333132303166366235613265336432616239393730353036616337386463363330306532 +65613066363434346562633630336135313736393137656230306163383730356133326561333431 +35393466323936636562616336353934626262336363306335636331666633623432343134396334 +37343765646134306638313636326531616665323136643264353666323236343239633639356537 +61343037666232386135303363363938353864396461363830373633363730323165303165633430 +32656532633761316132613463616365353066393165353331646665613239333738653861363134 +30343939333633366532656430303133643337356336336638653266653766353637356561663636 +61626632333438313436323465643434666334393034366166343930323661386664623663396435 +61633334656464623330623333346165363563356363656663373335646433343762663964323735 +34316239663164623431353164306235386661666632383730376139626666313935663462326433 +36313039326635396435646466306366343438656334346231393066326638313337373933393231 +35373731336164396165303935316366343565363631353137393333613363656564343063373933 +61643033373839656532373330666436396632626663353436356231323166326365323832346461 +31366630626635323465333238613734313265356231313338356431393338353166633939383334 +64333634373638383937663933336563663132366534643434373032333934383533383063643036 +39323534326139633931623831656565333930323362366539346239646464353536346639343436 +37663337396565363065343532616437333666316266633632653661396535636461653335363538 +37303263383030646233313239643536326434323933616135373835393965346462373739383230 +33383863353761346232643835633139363231663162353937343363653339393838616535393436 +38303737633231626236316336353832306232633032653032633365386563626434646139633063 +31313436343063643766663465626330643366313663633733663762653631633966383736336138 +36353738383865613537366364666665326464626365643032666630643039326538663531353964 +33646561303435666237303763663634613136633662633536303066363764383039373236666135 +30626639313531353134353730323334623631396463323035313733333965623665333466366164 +62646539393966626337333637646438313635373666306538356631396237323739613032353832 +31383439343531373561353038363535353266333265336434303934343239633334383865333436 +31616537666332373332623932623162333138373563363336303735396431396464613364643162 +61323737643134653763373232383134643931343366353437633632326138353237336333616238 +32613239316666343761363863376563613638366432313461643535353731383631633233366234 +39663632613062656337616561636533633264373930376437373132303830313863303734346632 +33313431336661663634663861643364363834626638666565383635343433323961623765633265 +34386631653833653865383865636364653265363162323462376364313964626633313635313765 +38633131653836396463393136333932346532333631626461373739346139306465653532366632 +65363333633964616632613830386566623865356131646338653236373861346430363164393634 +64373434663531643436616664393936396564666561313862636133306537373137636234626232 +61313861626235636235353431353738303161373939373966383139323833346662373731646434 +31356133623331323661666234613466366130336662653764363738626531346531333339396266 +32636564653732396339383533343266633730666434303161626362366134353361643362626533 +32333132656636333736616661323664316234393937313666653762316463306437633163383030 +65383962346233663061356338306164343363323736646165613730373166333533373635373039 +39343262393031626637623030373535663539613361613366333863336531663637323035376564 +30626232623661623065643136353237316333323764663633643132653264303162366134616564 +31366665313830323263373835613964386132333538396165663538653865356236646234373465 +30383533316532363133323465363063373962353763346366383563613862336465366565653637 +62346365373030383135323933366138656463323864663363626330653036623739343035336135 +62643262336436336462386430363430363139396566343963373664373334366430643830313439 +66613362396137313661666635643962346662376137393066653765636330366462633565346663 +38633864366666316534653665303664393863633464373638393463343537386437383039373530 +66623063653536666364303531373931613433383139376664633934376662336231643762353830 +31653065303430393864383532333766613965383831616563653233313338343861663334373465 +39363361313532333034613034666265386430333466396163303263636133316263313538656634 +65313464316463623134356562366261663635323530393137313233316638643364646639343935 +38313131633132636463323231393332363766653935323965373134653336376131636163376566 +33623934346431313433386430656264316566383436663535343532303763316437303436336465 +39626231303931303130643237383663666338313662663231313935643334393362323130356530 +39656330616138373962633732313436666662616165316236323930613134643161613464636536 +36613735636233616164396630333634333935393938623061313963636461316564313662353034 +33666434326165643964303561303062323061633766353761313464376262333361323532616361 +39313832396134643966353239333066346239626534303830396165396530623135333834313439 +37373661643731623935666437313465633565303232366638393837303462613035393031333061 +39666433373662633730363665313666346165366138306636623263633537346631666132306238 +34393363373335663631393135646231313064383837396164613536643533383464313536663561 +38633131303966356534636539333733336464393236623238373766616537613437343232303437 +64323636613862663135323034636239333433333165313239336239666665616631623934316637 +61303237336334316265643964663830336539353164343063636664633430326139356534313036 +65393464653330333365336335363531303536313439363066376438666133376664346336633937 +66346165666337666134333963613662303534656339373731663534656533316338353763643930 +38313239313637326435323463333863663433623366316532333065396635663966343835636630 +34643631613431333431636330623564663662383063353138656436336333333262306633666163 +38303032316133666533393663343366333162626436333762306263616362643936383637653330 +31643334613766636231333461643863333838343961383762316162636634323134313461313962 +35363062663934336632383931333262643534336563376537343265633434336439623131613337 +32626461643263623664353639613361393735663266373930366134633534343466346461313032 +34363162343464353037616637316636393139343563303331376362653238346631333137393164 +30306636366636366562656361346637356364666136623131316138393262633963356138383232 +64383932623831376632336331343333393739613663613763303035643136653965386435366663 +35313161383236356563653931313933633737646363363339336639613961343934373637643034 +63333036343532323166366132656538643331303131393737656338323939396461393764393134 +33663539633930636265393363326638636130313036333536306362663633616563316465616330 +35353539303335336565373962393433333539613439353765643061383132386133396339616137 +3265326634643062346666356263326639373130333362333064 diff --git a/infra/ansible/inventory/hosts.ini b/infra/ansible/inventory/hosts.ini index 1d0f330..4b419ce 100644 --- a/infra/ansible/inventory/hosts.ini +++ b/infra/ansible/inventory/hosts.ini @@ -45,6 +45,34 @@ localhost ansible_connection=local [vpn_egress_gateway] k3s-dev-3 +# PROD's egress-gateway node — a SEPARATE group on purpose (2026-09-12). +# +# It is NOT a member of [vpn_egress_gateway] above, and k3s-prod-3 must never be added +# there: that group is the DEFAULT target of playbooks/vpn.egress.gateway.yml, so a +# flagless `ansible-playbook playbooks/vpn.egress.gateway.yml` would then silently +# reconfigure a prod node's kill-switch datapath. Prod is reached only by asking for it: +# +# ansible-playbook playbooks/vpn.egress.gateway.yml \ +# -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 +# +# (`--list-hosts` first, every time. The play's target var is +# `vpn_egress_gateway_hosts`, NOT the cluster-wide `k3s_hosts`.) +# +# WHY NODE 3, AND WHY EXACTLY ONE: +# * node 1's SSD was replaced 2026-09-11 and it has the least operational history; +# * node 2 carries the §0b memattr history; +# * node 3 has a single uplink (eno1 192.168.2.12 — its wireless is gone: no wl* netdev +# exists, the /etc/network/interfaces wlp1s0 stanza is commented out, and the +# 192.168.2.195 address the earlier note below claimed is NOT present. Verified +# 2026-09-12) which is what a kill-switch datapath wants. +# * LEXICAL ORDERING HAZARD: Cilium OSS picks "the first node in lexical ordering" when +# several match the policy's nodeSelector, so labelling k3s-prod-1 would SILENTLY take +# the gateway role from k3s-prod-3. Exactly one host here, forever. +# Its PIA credentials are its own vaulted group_vars/vpn_egress_gateway_prod/vault.yml — +# the dev group's vault does not apply to a prod host. +[vpn_egress_gateway_prod] +k3s-prod-3 + # The Proxmox HYPERVISOR host itself (NOT a k3s node) — target of playbooks/backup-s3.yml, # which stands up the off-cluster versitygw S3 backup target for Velero + CNPG. Reached DIRECTLY # over SSH as root (its own 192.168.2.102:22, the pfSense NAT to the host's 192.168.1.9), so @@ -116,8 +144,13 @@ k3s-prod-1 ansible_host=192.168.2.10 ansible_port=22 k3s_node_ip=192.168.2.10 # needs the same >= 6.13 kernel. Its disk is a THIRD distinct model (WDC WDS500G2B0A SATA # SSD) — all three nodes differ on disk, so the §2 Longhorn/etcd symmetry gap is now # three-way, not a node-1 outlier. -# NOTE: node 3 also has wlp1s0 up on 192.168.2.195 (wifi). `ansible_host` is deliberately -# the WIRED eno1 address — the same directly-routable 192.168.2.x LAN as nodes 1 and 2. +# NOTE (CORRECTED 2026-09-12): this used to read "node 3 also has wlp1s0 up on +# 192.168.2.195 (wifi)". It does not — re-verified before making node 3 the VPN egress +# gateway, because a second live uplink in a kill-switch datapath is not a detail: there is +# NO wl* netdev on the node at all, no phy80211 device, the wlp1s0 stanza in +# /etc/network/interfaces is commented out, 192.168.2.195 is assigned nowhere, and the only +# IPv4 addresses are lo and eno1 192.168.2.12. `ansible_host` is that WIRED eno1 address — +# the same directly-routable 192.168.2.x LAN as nodes 1 and 2. k3s-prod-3 ansible_host=192.168.2.12 ansible_port=22 k3s_node_ip=192.168.2.12 # Umbrella group group_vars/k3s_prod/ attaches to (mirrors dev's [k3s_servers:children] diff --git a/infra/ansible/playbooks/vpn.egress.gateway.yml b/infra/ansible/playbooks/vpn.egress.gateway.yml index 92d4a79..751e0c6 100644 --- a/infra/ansible/playbooks/vpn.egress.gateway.yml +++ b/infra/ansible/playbooks/vpn.egress.gateway.yml @@ -11,13 +11,25 @@ # vault_vpn_egress_gateway_pia_username, vault_vpn_egress_gateway_pia_password, vault_vpn_egress_gateway_pia_ovpn # Run: ansible-playbook playbooks/vpn.egress.gateway.yml # -# TARGET (R-ENV-02): the group is a var, defaulting to dev's `vpn_egress_gateway`. Prod -# has NOT decided whether it runs an egress gateway at all, nor on which node -# (infra/PROD-PROVISIONING.md §6, last box) — so there is deliberately no prod egress -# group in the inventory yet. When there is, point this at it with -# -e vpn_egress_gateway_hosts= -# and it needs its OWN vaulted PIA credentials group_vars; this play's vault is scoped to -# the dev group. +# TARGET (R-ENV-02): the group is a var, defaulting to dev's `vpn_egress_gateway`. PROD has +# its own group as of 2026-09-12 — `vpn_egress_gateway_prod` (k3s-prod-3), with its own +# vaulted credentials in group_vars/vpn_egress_gateway_prod/: +# +# ansible-playbook playbooks/vpn.egress.gateway.yml \ +# -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 --list-hosts +# +# The var is `vpn_egress_gateway_hosts`, NOT the cluster-wide `k3s_hosts`. Prod is +# deliberately NOT a member of the default group, so a flagless run cannot reach it — and +# `--list-hosts` before every run is the habit that catches a mistyped flag, which in this +# repo has silently targeted all three dev nodes before. +# +# THE LABEL IS WHAT ARMS THE DATAPATH. The CiliumEgressGatewayPolicy is already reconciled +# on both clusters, so the instant a node carries +# `homelab.keithjackson.dev/egress-gateway=true` steering goes live. The role sequences +# kill-switch -> tunnel -> re-steer -> label, with the label LAST, on purpose. Never apply +# that label by hand, and never run this play with tags that skip the tunnel or kill-switch +# tasks. Exactly ONE node per cluster may carry it (Cilium OSS picks the lexically first +# match, so a labelled k3s-prod-1 would silently steal the role from k3s-prod-3). - name: Configure the node-level VPN egress gateway hosts: "{{ vpn_egress_gateway_hosts | default('vpn_egress_gateway') }}" become: true