---
date: 2026-10-05
branch: fix/vpn-runbook-ansible-command
title: vpn-remote-access runbook — step 1 named a script that does not exist, and step 2 cited the wrong reason the key is gitignored
---

## What changed

Two defects in `runbooks/vpn-remote-access.md` (merged in #55), both authored by me, both reported
by the operator the first time they tried to follow it.

**1. Step 1 named a command that does not exist.** It said:

    cd infra/ansible && ./site.sh --tags system --limit k3s_prod

There is **no `site.sh` anywhere in this repo** (`find . -name site.sh` → nothing). The invocation
was wrong in four separate ways at once, which is what makes it worth recording rather than just
patching: wrong kind of thing (this repo drives Ansible with `ansible-playbook <playbook>`, never a
wrapper script — `playbooks/site.yml` exists and is the full bring-up chain, not this); wrong
playbook (the module work lives in `playbooks/node-baseline.yml`); wrong targeting flag (`--limit`
instead of this repo's `-e k3s_hosts=...` convention, which `infra/ansible/README.md` "Targeting
prod" documents for every playbook); and no toolbox wrapper, though Ansible exists only inside the
pinned container and `ansible.cfg` is only read from `infra/ansible`.

Replaced with the real command, plus the three things about it that are load-bearing:

    ./infra/devcontainer/dev.sh run bash -c \
      'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'

- `-e k3s_hosts=k3s_prod` — the playbook defaults to `k3s_servers`, which is the **dev** group;
  dev is decommissioned, so an unqualified run does nothing useful.
- `--tags system`, never untagged — the two kernel-module tasks carry that tag, and an untagged run
  would also hit the `disks` tasks, which assert on block-device variables prod deliberately leaves
  as placeholders (the playbook's own header warns about this). `--tags system` is broader than the
  modules alone (sysctls, iscsid, qemu-guest-agent, swap masking) but each is idempotent and
  already in its desired state on a live node.

**2. Step 2 gave the wrong reason the server key is safe from git — and the right reason is
fragile.** It told the operator to stage the key at `/work/.cache/server.key` "inside the repo's
gitignored cache". `.cache/` is **not** gitignored: `git check-ignore -v .cache/serverkey.txt`
matches nothing. The file was ignored only incidentally, by `.gitignore` line 11 `*.key` — so the
path happened to be safe for a reason I had not identified, and **any operator who renamed it
(`server.pem`, `wg-server`, `privatekey`) would have had a committable WireGuard private key sitting
in the checkout.** The gate would not have caught it either; `no-plaintext-secrets` does not
recognise a bare WireGuard key.

The step now states the real rule, marks the `.key` suffix as the thing doing the work, and gives
the operator a verification they can run before trusting it (`git check-ignore -v` must print a
rule, and `git status --short` must not list the file).

Nothing outside `runbooks/vpn-remote-access.md` changed; no manifest, no behaviour.

## What I verified

- **`site.sh` does not exist**: `find . -name 'site.sh' -not -path './.git/*'` → no output.
- **The real surface**: `infra/ansible/playbooks/` contains `node-baseline.yml` and `site.yml` (and
  11 others); `node-baseline.yml`'s own header documents both
  `-e k3s_hosts=k3s_prod` and the tag-scoping hazard. The `-e <playbook>_hosts=` convention is
  confirmed by `infra/ansible/README.md` §"Targeting prod" and by the existing
  `runbooks/node-memory-swap.md`, which invokes `ansible-playbook playbooks/node-health.yml -e
  node_health_hosts=...`.
- **`k3s_prod` is a real inventory group**: `[k3s_prod:children]` in `inventory/hosts.ini`, with
  `[k3s_prod_primary]`/`[k3s_prod_secondary]`.
- **`--tags system` covers the kernel-module tasks**: both "Persist kernel modules…" and "Load
  kernel modules now" carry `tags: [system]` in `roles/node_baseline/tasks/main.yml`.
- **The gitignore claim, measured both directions**:
  `git check-ignore -v .cache/server.key` → `.gitignore:11:*.key`;
  `git check-ignore -v .cache/serverkey.txt` → no match. So the suffix is the only thing ignoring it.
- **Every path the component's docs reference resolves**: all six relative markdown links in
  `platform/vpn-ingress/README.md` (checked from that file's own directory), and both Ansible
  playbook paths.
- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
  `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; the rc 3 is the two
  environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no `CLUSTER_AVAILABLE`).
  **Not green. Reported as rc 3, not as pass.**
- **This branch touches NO sensitive path** (`runbooks/**` and `decisions/**` are not in
  `infra/gitea/approval-policy.json`), so unlike #55 it needs no `/approve-override`.

## What I did NOT verify

- **The corrected command has not been RUN.** Ansible is an operator action, and this is the
  decisive gap: I have verified the playbook, the group, the tag and the flag all exist and are
  the documented convention, but not that the run succeeds end to end or that
  `lsmod | grep '^wireguard'` then reports the module. It is a better-evidenced command, not a
  proven one.
- **Whether `--tags system` is entirely side-effect-free on a live prod node.** Each task it runs is
  idempotent by construction, but I did not dry-run it (`--check`) to confirm it reports zero
  changes beyond the module load.
- **The rest of the runbook's commands past step 2** are unchanged and still unexercised — in
  particular `kubectl exec ds/wireguard -- wg show`, which cannot be reached while the pods sit in
  `Init` waiting on the server key.
- **Whether any OTHER document I wrote in #55 contains an invented command.** I re-checked every
  file path and markdown link in the component (all resolve) and every script the docs tell an
  operator to run, but "the referenced file exists" is weaker than "the command is correct" — which
  is exactly the gap that produced `site.sh`.

## Post-merge state of #55, recorded here since its own entry predates the merge

Measured on live prod after #55 merged as `ad3f56e`:

- Flux applied it by the real path: Kustomization `vpn-ingress` `lastAppliedRevision:
  main@sha1:ad3f56e`, equal to the GitRepository's artifact revision, `Ready=True`.
- All **18** Flux Kustomizations `Ready=True` — the new tier disturbed nothing, and `wait: false`
  behaved as intended (`vpn-ingress` is Ready while its own pods are not, so nothing downstream
  stalled on it).
- **`hostNetwork` is confirmed in effect, not merely configured**: the three pods report pod IPs
  `192.168.2.10/.11/.12` — the node addresses, one per node.
- **It fails closed exactly as designed**, with a legible cause: all three pods `Init:0/1`,
  `MountVolume.SetUp failed for volume "wg-server-key" : secret "wg-server-key" not found`, and no
  Secret in the namespace. The #55 entry's "the DaemonSet has never been reconciled" is superseded
  by this paragraph; its remaining unverified claims (the tunnel carrying a packet, the
  iptables/nftables backend question) still stand.