jakman85andClaude Opus 5
18f741d744
Both reported by the operator on first use of the runbook merged in #55.
Step 1 said `cd infra/ansible && ./site.sh --tags system --limit k3s_prod`.
There is no site.sh anywhere in this repo. It was wrong four ways at once:
this repo drives Ansible with `ansible-playbook <playbook>` and never a
wrapper (playbooks/site.yml is the full bring-up chain, not this); the module
work is in playbooks/node-baseline.yml; targeting here is `-e k3s_hosts=...`,
not `--limit`; and Ansible exists only in the pinned toolbox, with ansible.cfg
read only from infra/ansible. Replaced with the real command plus why
`-e k3s_hosts=k3s_prod` matters (the default group is decommissioned dev) and
why `--tags system` must never be dropped (an untagged run also executes the
`disks` tasks, which assert on prod's deliberate placeholder devices).
Step 2 told the operator to stage the server key at /work/.cache/server.key
"inside the repo's gitignored cache". .cache/ is NOT gitignored -- the file was
ignored only incidentally by `*.key` (.gitignore:11). So the path was safe for
a reason I had not identified, and renaming it (server.pem, wg-server) would
have left a committable WireGuard private key in the checkout, which
no-plaintext-secrets does not recognise. The step now names the real rule,
marks the suffix as the load-bearing part, and gives a check-ignore
verification to run before trusting it.
Docs only -- no manifest and no behaviour change. Touches no sensitive path.
Also records the post-merge measurements for #55, whose own entry predates the
merge: Flux applied ad3f56e by the real path, all 18 Kustomizations Ready,
hostNetwork confirmed (pod IPs are the node IPs), and the designed fail-closed
state (Init:0/1 on the missing wg-server-key Secret).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
113 lines
7.2 KiB
Markdown
113 lines
7.2 KiB
Markdown
---
|
|
date: 2026-10-05
|
|
branch: fix/vpn-runbook-ansible-command
|
|
title: vpn-remote-access runbook — step 1 named a script that does not exist, and step 2 cited the wrong reason the key is gitignored
|
|
---
|
|
|
|
## What changed
|
|
|
|
Two defects in `runbooks/vpn-remote-access.md` (merged in #55), both authored by me, both reported
|
|
by the operator the first time they tried to follow it.
|
|
|
|
**1. Step 1 named a command that does not exist.** It said:
|
|
|
|
cd infra/ansible && ./site.sh --tags system --limit k3s_prod
|
|
|
|
There is **no `site.sh` anywhere in this repo** (`find . -name site.sh` → nothing). The invocation
|
|
was wrong in four separate ways at once, which is what makes it worth recording rather than just
|
|
patching: wrong kind of thing (this repo drives Ansible with `ansible-playbook <playbook>`, never a
|
|
wrapper script — `playbooks/site.yml` exists and is the full bring-up chain, not this); wrong
|
|
playbook (the module work lives in `playbooks/node-baseline.yml`); wrong targeting flag (`--limit`
|
|
instead of this repo's `-e k3s_hosts=...` convention, which `infra/ansible/README.md` "Targeting
|
|
prod" documents for every playbook); and no toolbox wrapper, though Ansible exists only inside the
|
|
pinned container and `ansible.cfg` is only read from `infra/ansible`.
|
|
|
|
Replaced with the real command, plus the three things about it that are load-bearing:
|
|
|
|
./infra/devcontainer/dev.sh run bash -c \
|
|
'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'
|
|
|
|
- `-e k3s_hosts=k3s_prod` — the playbook defaults to `k3s_servers`, which is the **dev** group;
|
|
dev is decommissioned, so an unqualified run does nothing useful.
|
|
- `--tags system`, never untagged — the two kernel-module tasks carry that tag, and an untagged run
|
|
would also hit the `disks` tasks, which assert on block-device variables prod deliberately leaves
|
|
as placeholders (the playbook's own header warns about this). `--tags system` is broader than the
|
|
modules alone (sysctls, iscsid, qemu-guest-agent, swap masking) but each is idempotent and
|
|
already in its desired state on a live node.
|
|
|
|
**2. Step 2 gave the wrong reason the server key is safe from git — and the right reason is
|
|
fragile.** It told the operator to stage the key at `/work/.cache/server.key` "inside the repo's
|
|
gitignored cache". `.cache/` is **not** gitignored: `git check-ignore -v .cache/serverkey.txt`
|
|
matches nothing. The file was ignored only incidentally, by `.gitignore` line 11 `*.key` — so the
|
|
path happened to be safe for a reason I had not identified, and **any operator who renamed it
|
|
(`server.pem`, `wg-server`, `privatekey`) would have had a committable WireGuard private key sitting
|
|
in the checkout.** The gate would not have caught it either; `no-plaintext-secrets` does not
|
|
recognise a bare WireGuard key.
|
|
|
|
The step now states the real rule, marks the `.key` suffix as the thing doing the work, and gives
|
|
the operator a verification they can run before trusting it (`git check-ignore -v` must print a
|
|
rule, and `git status --short` must not list the file).
|
|
|
|
Nothing outside `runbooks/vpn-remote-access.md` changed; no manifest, no behaviour.
|
|
|
|
## What I verified
|
|
|
|
- **`site.sh` does not exist**: `find . -name 'site.sh' -not -path './.git/*'` → no output.
|
|
- **The real surface**: `infra/ansible/playbooks/` contains `node-baseline.yml` and `site.yml` (and
|
|
11 others); `node-baseline.yml`'s own header documents both
|
|
`-e k3s_hosts=k3s_prod` and the tag-scoping hazard. The `-e <playbook>_hosts=` convention is
|
|
confirmed by `infra/ansible/README.md` §"Targeting prod" and by the existing
|
|
`runbooks/node-memory-swap.md`, which invokes `ansible-playbook playbooks/node-health.yml -e
|
|
node_health_hosts=...`.
|
|
- **`k3s_prod` is a real inventory group**: `[k3s_prod:children]` in `inventory/hosts.ini`, with
|
|
`[k3s_prod_primary]`/`[k3s_prod_secondary]`.
|
|
- **`--tags system` covers the kernel-module tasks**: both "Persist kernel modules…" and "Load
|
|
kernel modules now" carry `tags: [system]` in `roles/node_baseline/tasks/main.yml`.
|
|
- **The gitignore claim, measured both directions**:
|
|
`git check-ignore -v .cache/server.key` → `.gitignore:11:*.key`;
|
|
`git check-ignore -v .cache/serverkey.txt` → no match. So the suffix is the only thing ignoring it.
|
|
- **Every path the component's docs reference resolves**: all six relative markdown links in
|
|
`platform/vpn-ingress/README.md` (checked from that file's own directory), and both Ansible
|
|
playbook paths.
|
|
- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
|
|
`lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; the rc 3 is the two
|
|
environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no `CLUSTER_AVAILABLE`).
|
|
**Not green. Reported as rc 3, not as pass.**
|
|
- **This branch touches NO sensitive path** (`runbooks/**` and `decisions/**` are not in
|
|
`infra/gitea/approval-policy.json`), so unlike #55 it needs no `/approve-override`.
|
|
|
|
## What I did NOT verify
|
|
|
|
- **The corrected command has not been RUN.** Ansible is an operator action, and this is the
|
|
decisive gap: I have verified the playbook, the group, the tag and the flag all exist and are
|
|
the documented convention, but not that the run succeeds end to end or that
|
|
`lsmod | grep '^wireguard'` then reports the module. It is a better-evidenced command, not a
|
|
proven one.
|
|
- **Whether `--tags system` is entirely side-effect-free on a live prod node.** Each task it runs is
|
|
idempotent by construction, but I did not dry-run it (`--check`) to confirm it reports zero
|
|
changes beyond the module load.
|
|
- **The rest of the runbook's commands past step 2** are unchanged and still unexercised — in
|
|
particular `kubectl exec ds/wireguard -- wg show`, which cannot be reached while the pods sit in
|
|
`Init` waiting on the server key.
|
|
- **Whether any OTHER document I wrote in #55 contains an invented command.** I re-checked every
|
|
file path and markdown link in the component (all resolve) and every script the docs tell an
|
|
operator to run, but "the referenced file exists" is weaker than "the command is correct" — which
|
|
is exactly the gap that produced `site.sh`.
|
|
|
|
## Post-merge state of #55, recorded here since its own entry predates the merge
|
|
|
|
Measured on live prod after #55 merged as `ad3f56e`:
|
|
|
|
- Flux applied it by the real path: Kustomization `vpn-ingress` `lastAppliedRevision:
|
|
main@sha1:ad3f56e`, equal to the GitRepository's artifact revision, `Ready=True`.
|
|
- All **18** Flux Kustomizations `Ready=True` — the new tier disturbed nothing, and `wait: false`
|
|
behaved as intended (`vpn-ingress` is Ready while its own pods are not, so nothing downstream
|
|
stalled on it).
|
|
- **`hostNetwork` is confirmed in effect, not merely configured**: the three pods report pod IPs
|
|
`192.168.2.10/.11/.12` — the node addresses, one per node.
|
|
- **It fails closed exactly as designed**, with a legible cause: all three pods `Init:0/1`,
|
|
`MountVolume.SetUp failed for volume "wg-server-key" : secret "wg-server-key" not found`, and no
|
|
Secret in the namespace. The #55 entry's "the DaemonSet has never been reconciled" is superseded
|
|
by this paragraph; its remaining unverified claims (the tunnel carrying a packet, the
|
|
iptables/nftables backend question) still stand.
|