From 18f741d7445c0885c7ac08a2e564a87761fbb5b9 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Mon, 5 Oct 2026 23:08:19 -0500 Subject: [PATCH] fix(runbooks): vpn-remote-access step 1 named a nonexistent script; step 2 cited the wrong gitignore rule Both reported by the operator on first use of the runbook merged in #55. Step 1 said `cd infra/ansible && ./site.sh --tags system --limit k3s_prod`. There is no site.sh anywhere in this repo. It was wrong four ways at once: this repo drives Ansible with `ansible-playbook ` and never a wrapper (playbooks/site.yml is the full bring-up chain, not this); the module work is in playbooks/node-baseline.yml; targeting here is `-e k3s_hosts=...`, not `--limit`; and Ansible exists only in the pinned toolbox, with ansible.cfg read only from infra/ansible. Replaced with the real command plus why `-e k3s_hosts=k3s_prod` matters (the default group is decommissioned dev) and why `--tags system` must never be dropped (an untagged run also executes the `disks` tasks, which assert on prod's deliberate placeholder devices). Step 2 told the operator to stage the server key at /work/.cache/server.key "inside the repo's gitignored cache". .cache/ is NOT gitignored -- the file was ignored only incidentally by `*.key` (.gitignore:11). So the path was safe for a reason I had not identified, and renaming it (server.pem, wg-server) would have left a committable WireGuard private key in the checkout, which no-plaintext-secrets does not recognise. The step now names the real rule, marks the suffix as the load-bearing part, and gives a check-ignore verification to run before trusting it. Docs only -- no manifest and no behaviour change. Touches no sensitive path. Also records the post-merge measurements for #55, whose own entry predates the merge: Flux applied ad3f56e by the real path, all 18 Kustomizations Ready, hostNetwork confirmed (pod IPs are the node IPs), and the designed fail-closed state (Init:0/1 on the missing wg-server-key Secret). Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-10-05-vpn-runbook-ansible-command.md | 112 ++++++++++++++++++ runbooks/vpn-remote-access.md | 33 +++++- 2 files changed, 140 insertions(+), 5 deletions(-) create mode 100644 decisions/2026-10-05-vpn-runbook-ansible-command.md diff --git a/decisions/2026-10-05-vpn-runbook-ansible-command.md b/decisions/2026-10-05-vpn-runbook-ansible-command.md new file mode 100644 index 0000000..8038d51 --- /dev/null +++ b/decisions/2026-10-05-vpn-runbook-ansible-command.md @@ -0,0 +1,112 @@ +--- +date: 2026-10-05 +branch: fix/vpn-runbook-ansible-command +title: vpn-remote-access runbook — step 1 named a script that does not exist, and step 2 cited the wrong reason the key is gitignored +--- + +## What changed + +Two defects in `runbooks/vpn-remote-access.md` (merged in #55), both authored by me, both reported +by the operator the first time they tried to follow it. + +**1. Step 1 named a command that does not exist.** It said: + + cd infra/ansible && ./site.sh --tags system --limit k3s_prod + +There is **no `site.sh` anywhere in this repo** (`find . -name site.sh` → nothing). The invocation +was wrong in four separate ways at once, which is what makes it worth recording rather than just +patching: wrong kind of thing (this repo drives Ansible with `ansible-playbook `, never a +wrapper script — `playbooks/site.yml` exists and is the full bring-up chain, not this); wrong +playbook (the module work lives in `playbooks/node-baseline.yml`); wrong targeting flag (`--limit` +instead of this repo's `-e k3s_hosts=...` convention, which `infra/ansible/README.md` "Targeting +prod" documents for every playbook); and no toolbox wrapper, though Ansible exists only inside the +pinned container and `ansible.cfg` is only read from `infra/ansible`. + +Replaced with the real command, plus the three things about it that are load-bearing: + + ./infra/devcontainer/dev.sh run bash -c \ + 'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system' + +- `-e k3s_hosts=k3s_prod` — the playbook defaults to `k3s_servers`, which is the **dev** group; + dev is decommissioned, so an unqualified run does nothing useful. +- `--tags system`, never untagged — the two kernel-module tasks carry that tag, and an untagged run + would also hit the `disks` tasks, which assert on block-device variables prod deliberately leaves + as placeholders (the playbook's own header warns about this). `--tags system` is broader than the + modules alone (sysctls, iscsid, qemu-guest-agent, swap masking) but each is idempotent and + already in its desired state on a live node. + +**2. Step 2 gave the wrong reason the server key is safe from git — and the right reason is +fragile.** It told the operator to stage the key at `/work/.cache/server.key` "inside the repo's +gitignored cache". `.cache/` is **not** gitignored: `git check-ignore -v .cache/serverkey.txt` +matches nothing. The file was ignored only incidentally, by `.gitignore` line 11 `*.key` — so the +path happened to be safe for a reason I had not identified, and **any operator who renamed it +(`server.pem`, `wg-server`, `privatekey`) would have had a committable WireGuard private key sitting +in the checkout.** The gate would not have caught it either; `no-plaintext-secrets` does not +recognise a bare WireGuard key. + +The step now states the real rule, marks the `.key` suffix as the thing doing the work, and gives +the operator a verification they can run before trusting it (`git check-ignore -v` must print a +rule, and `git status --short` must not list the file). + +Nothing outside `runbooks/vpn-remote-access.md` changed; no manifest, no behaviour. + +## What I verified + +- **`site.sh` does not exist**: `find . -name 'site.sh' -not -path './.git/*'` → no output. +- **The real surface**: `infra/ansible/playbooks/` contains `node-baseline.yml` and `site.yml` (and + 11 others); `node-baseline.yml`'s own header documents both + `-e k3s_hosts=k3s_prod` and the tag-scoping hazard. The `-e _hosts=` convention is + confirmed by `infra/ansible/README.md` §"Targeting prod" and by the existing + `runbooks/node-memory-swap.md`, which invokes `ansible-playbook playbooks/node-health.yml -e + node_health_hosts=...`. +- **`k3s_prod` is a real inventory group**: `[k3s_prod:children]` in `inventory/hosts.ini`, with + `[k3s_prod_primary]`/`[k3s_prod_secondary]`. +- **`--tags system` covers the kernel-module tasks**: both "Persist kernel modules…" and "Load + kernel modules now" carry `tags: [system]` in `roles/node_baseline/tasks/main.yml`. +- **The gitignore claim, measured both directions**: + `git check-ignore -v .cache/server.key` → `.gitignore:11:*.key`; + `git check-ignore -v .cache/serverkey.txt` → no match. So the suffix is the only thing ignoring it. +- **Every path the component's docs reference resolves**: all six relative markdown links in + `platform/vpn-ingress/README.md` (checked from that file's own directory), and both Ansible + playbook paths. +- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`, + `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; the rc 3 is the two + environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no `CLUSTER_AVAILABLE`). + **Not green. Reported as rc 3, not as pass.** +- **This branch touches NO sensitive path** (`runbooks/**` and `decisions/**` are not in + `infra/gitea/approval-policy.json`), so unlike #55 it needs no `/approve-override`. + +## What I did NOT verify + +- **The corrected command has not been RUN.** Ansible is an operator action, and this is the + decisive gap: I have verified the playbook, the group, the tag and the flag all exist and are + the documented convention, but not that the run succeeds end to end or that + `lsmod | grep '^wireguard'` then reports the module. It is a better-evidenced command, not a + proven one. +- **Whether `--tags system` is entirely side-effect-free on a live prod node.** Each task it runs is + idempotent by construction, but I did not dry-run it (`--check`) to confirm it reports zero + changes beyond the module load. +- **The rest of the runbook's commands past step 2** are unchanged and still unexercised — in + particular `kubectl exec ds/wireguard -- wg show`, which cannot be reached while the pods sit in + `Init` waiting on the server key. +- **Whether any OTHER document I wrote in #55 contains an invented command.** I re-checked every + file path and markdown link in the component (all resolve) and every script the docs tell an + operator to run, but "the referenced file exists" is weaker than "the command is correct" — which + is exactly the gap that produced `site.sh`. + +## Post-merge state of #55, recorded here since its own entry predates the merge + +Measured on live prod after #55 merged as `ad3f56e`: + +- Flux applied it by the real path: Kustomization `vpn-ingress` `lastAppliedRevision: + main@sha1:ad3f56e`, equal to the GitRepository's artifact revision, `Ready=True`. +- All **18** Flux Kustomizations `Ready=True` — the new tier disturbed nothing, and `wait: false` + behaved as intended (`vpn-ingress` is Ready while its own pods are not, so nothing downstream + stalled on it). +- **`hostNetwork` is confirmed in effect, not merely configured**: the three pods report pod IPs + `192.168.2.10/.11/.12` — the node addresses, one per node. +- **It fails closed exactly as designed**, with a legible cause: all three pods `Init:0/1`, + `MountVolume.SetUp failed for volume "wg-server-key" : secret "wg-server-key" not found`, and no + Secret in the namespace. The #55 entry's "the DaemonSet has never been reconciled" is superseded + by this paragraph; its remaining unverified claims (the tunnel carrying a packet, the + iptables/nftables backend question) still stand. diff --git a/runbooks/vpn-remote-access.md b/runbooks/vpn-remote-access.md index 4f4acd0..cde254c 100644 --- a/runbooks/vpn-remote-access.md +++ b/runbooks/vpn-remote-access.md @@ -34,10 +34,26 @@ deliberately **not** granted `CAP_SYS_MODULE`, so it cannot autoload the module. `wg-quick` fails outright and the pod CrashLoops. ```bash -# operator action (Ansible is not an agent action) -cd infra/ansible && ./site.sh --tags system --limit k3s_prod +# operator action (Ansible is not an agent action). Ansible lives ONLY in the pinned toolbox, +# and ansible.cfg is only picked up from infra/ansible, so cd there inside the container. +./infra/devcontainer/dev.sh run bash -c \ + 'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system' ``` +Three details in that command are load-bearing, not style: + +- **`-e k3s_hosts=k3s_prod`.** `node-baseline.yml` targets `k3s_servers` by default, which is the + **dev** group — and dev was decommissioned, so an unqualified run does nothing useful. This is + the targeting pattern `infra/ansible/README.md` "Targeting prod" documents for every playbook. +- **`--tags system`, and never an untagged run.** The two kernel-module tasks carry the `system` + tag. An untagged run would also execute the `disks` tasks, which assert on block-device variables + prod has deliberately left as placeholders (`infra/PROD-PROVISIONING.md` §2) — the playbook's own + header warns about this. `--tags system` is broader than the modules alone (it also re-asserts + sysctls, iscsid, qemu-guest-agent and the swap masking) but every one of those is idempotent and + already in its desired state on a live node. +- **It is a playbook, not a wrapper script.** There is no `site.sh` in this repo; `playbooks/site.yml` + is the full bring-up chain and is the wrong tool here. + Verify on each node — the module is in-tree since kernel 5.6, so nothing is installed, only loaded: ```bash @@ -66,9 +82,16 @@ lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE --from-file=privatekey=/work/.cache/server.key' ``` -…where `/work/.cache/server.key` is a path you copied the key to **inside the repo's gitignored -cache** (or mount it however you prefer — the requirement is only that the bytes never pass through -a log). Delete that copy afterwards: `shred -u .cache/server.key`. +…where `/work/.cache/server.key` is a path you copied the key to (`.cache/` is not tracked; create +it if absent). Delete that copy as soon as the Secret exists: `shred -u .cache/server.key`. + +> **KEEP THE `.key` SUFFIX.** That filename is what keeps the key out of git: `.gitignore` line 11 +> is `*.key`, and it is the **only** rule covering this path — `.cache/` itself is NOT ignored +> (`git check-ignore -v .cache/serverkey.txt` matches nothing). So a copy named `server.pem`, +> `wg-server` or `privatekey` sitting in that directory is **committable**, and the gate's +> `no-plaintext-secrets` stage does not know a WireGuard private key when it sees one. Verify before +> you trust it: `git check-ignore -v .cache/` must print a rule, and +> `git status --short` must not list the file. Confirm it exists **by key name only**: -- 2.54.0