jakman85andClaude Opus 5
18f741d744
Both reported by the operator on first use of the runbook merged in #55.
Step 1 said `cd infra/ansible && ./site.sh --tags system --limit k3s_prod`.
There is no site.sh anywhere in this repo. It was wrong four ways at once:
this repo drives Ansible with `ansible-playbook <playbook>` and never a
wrapper (playbooks/site.yml is the full bring-up chain, not this); the module
work is in playbooks/node-baseline.yml; targeting here is `-e k3s_hosts=...`,
not `--limit`; and Ansible exists only in the pinned toolbox, with ansible.cfg
read only from infra/ansible. Replaced with the real command plus why
`-e k3s_hosts=k3s_prod` matters (the default group is decommissioned dev) and
why `--tags system` must never be dropped (an untagged run also executes the
`disks` tasks, which assert on prod's deliberate placeholder devices).
Step 2 told the operator to stage the server key at /work/.cache/server.key
"inside the repo's gitignored cache". .cache/ is NOT gitignored -- the file was
ignored only incidentally by `*.key` (.gitignore:11). So the path was safe for
a reason I had not identified, and renaming it (server.pem, wg-server) would
have left a committable WireGuard private key in the checkout, which
no-plaintext-secrets does not recognise. The step now names the real rule,
marks the suffix as the load-bearing part, and gives a check-ignore
verification to run before trusting it.
Docs only -- no manifest and no behaviour change. Touches no sensitive path.
Also records the post-merge measurements for #55, whose own entry predates the
merge: Flux applied ad3f56e by the real path, all 18 Kustomizations Ready,
hostNetwork confirmed (pod IPs are the node IPs), and the designed fail-closed
state (Init:0/1 on the missing wg-server-key Secret).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
7.2 KiB
date, branch, title
| date | branch | title |
|---|---|---|
| 2026-10-05 | fix/vpn-runbook-ansible-command | vpn-remote-access runbook — step 1 named a script that does not exist, and step 2 cited the wrong reason the key is gitignored |
What changed
Two defects in runbooks/vpn-remote-access.md (merged in #55), both authored by me, both reported
by the operator the first time they tried to follow it.
1. Step 1 named a command that does not exist. It said:
cd infra/ansible && ./site.sh --tags system --limit k3s_prod
There is no site.sh anywhere in this repo (find . -name site.sh → nothing). The invocation
was wrong in four separate ways at once, which is what makes it worth recording rather than just
patching: wrong kind of thing (this repo drives Ansible with ansible-playbook <playbook>, never a
wrapper script — playbooks/site.yml exists and is the full bring-up chain, not this); wrong
playbook (the module work lives in playbooks/node-baseline.yml); wrong targeting flag (--limit
instead of this repo's -e k3s_hosts=... convention, which infra/ansible/README.md "Targeting
prod" documents for every playbook); and no toolbox wrapper, though Ansible exists only inside the
pinned container and ansible.cfg is only read from infra/ansible.
Replaced with the real command, plus the three things about it that are load-bearing:
./infra/devcontainer/dev.sh run bash -c \
'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
-e k3s_hosts=k3s_prod— the playbook defaults tok3s_servers, which is the dev group; dev is decommissioned, so an unqualified run does nothing useful.--tags system, never untagged — the two kernel-module tasks carry that tag, and an untagged run would also hit thediskstasks, which assert on block-device variables prod deliberately leaves as placeholders (the playbook's own header warns about this).--tags systemis broader than the modules alone (sysctls, iscsid, qemu-guest-agent, swap masking) but each is idempotent and already in its desired state on a live node.
2. Step 2 gave the wrong reason the server key is safe from git — and the right reason is
fragile. It told the operator to stage the key at /work/.cache/server.key "inside the repo's
gitignored cache". .cache/ is not gitignored: git check-ignore -v .cache/serverkey.txt
matches nothing. The file was ignored only incidentally, by .gitignore line 11 *.key — so the
path happened to be safe for a reason I had not identified, and any operator who renamed it
(server.pem, wg-server, privatekey) would have had a committable WireGuard private key sitting
in the checkout. The gate would not have caught it either; no-plaintext-secrets does not
recognise a bare WireGuard key.
The step now states the real rule, marks the .key suffix as the thing doing the work, and gives
the operator a verification they can run before trusting it (git check-ignore -v must print a
rule, and git status --short must not list the file).
Nothing outside runbooks/vpn-remote-access.md changed; no manifest, no behaviour.
What I verified
site.shdoes not exist:find . -name 'site.sh' -not -path './.git/*'→ no output.- The real surface:
infra/ansible/playbooks/containsnode-baseline.ymlandsite.yml(and 11 others);node-baseline.yml's own header documents both-e k3s_hosts=k3s_prodand the tag-scoping hazard. The-e <playbook>_hosts=convention is confirmed byinfra/ansible/README.md§"Targeting prod" and by the existingrunbooks/node-memory-swap.md, which invokesansible-playbook playbooks/node-health.yml -e node_health_hosts=.... k3s_prodis a real inventory group:[k3s_prod:children]ininventory/hosts.ini, with[k3s_prod_primary]/[k3s_prod_secondary].--tags systemcovers the kernel-module tasks: both "Persist kernel modules…" and "Load kernel modules now" carrytags: [system]inroles/node_baseline/tasks/main.yml.- The gitignore claim, measured both directions:
git check-ignore -v .cache/server.key→.gitignore:11:*.key;git check-ignore -v .cache/serverkey.txt→ no match. So the suffix is the only thing ignoring it. - Every path the component's docs reference resolves: all six relative markdown links in
platform/vpn-ingress/README.md(checked from that file's own directory), and both Ansible playbook paths. - Toolbox gate from an isolated worktree: rc 3, with
spec-immutability,decisions-log,lint,l0,render-golden,l1andapproval-policyall pass; the rc 3 is the two environmental skips (gitea-settings— no local token; conformance L2/L3 — noCLUSTER_AVAILABLE). Not green. Reported as rc 3, not as pass. - This branch touches NO sensitive path (
runbooks/**anddecisions/**are not ininfra/gitea/approval-policy.json), so unlike #55 it needs no/approve-override.
What I did NOT verify
- The corrected command has not been RUN. Ansible is an operator action, and this is the
decisive gap: I have verified the playbook, the group, the tag and the flag all exist and are
the documented convention, but not that the run succeeds end to end or that
lsmod | grep '^wireguard'then reports the module. It is a better-evidenced command, not a proven one. - Whether
--tags systemis entirely side-effect-free on a live prod node. Each task it runs is idempotent by construction, but I did not dry-run it (--check) to confirm it reports zero changes beyond the module load. - The rest of the runbook's commands past step 2 are unchanged and still unexercised — in
particular
kubectl exec ds/wireguard -- wg show, which cannot be reached while the pods sit inInitwaiting on the server key. - Whether any OTHER document I wrote in #55 contains an invented command. I re-checked every
file path and markdown link in the component (all resolve) and every script the docs tell an
operator to run, but "the referenced file exists" is weaker than "the command is correct" — which
is exactly the gap that produced
site.sh.
Post-merge state of #55, recorded here since its own entry predates the merge
Measured on live prod after #55 merged as ad3f56e:
- Flux applied it by the real path: Kustomization
vpn-ingresslastAppliedRevision: main@sha1:ad3f56e, equal to the GitRepository's artifact revision,Ready=True. - All 18 Flux Kustomizations
Ready=True— the new tier disturbed nothing, andwait: falsebehaved as intended (vpn-ingressis Ready while its own pods are not, so nothing downstream stalled on it). hostNetworkis confirmed in effect, not merely configured: the three pods report pod IPs192.168.2.10/.11/.12— the node addresses, one per node.- It fails closed exactly as designed, with a legible cause: all three pods
Init:0/1,MountVolume.SetUp failed for volume "wg-server-key" : secret "wg-server-key" not found, and no Secret in the namespace. The #55 entry's "the DaemonSet has never been reconciled" is superseded by this paragraph; its remaining unverified claims (the tunnel carrying a packet, the iptables/nftables backend question) still stand.