fix(runbooks): vpn-remote-access step 1 named a nonexistent script; step 2 cited the wrong gitignore rule #58

2026-10-06T04:10:38Z

gitea-admin commented

2026-10-06 04:10:38 +00:00

Both defects were mine, in the runbook merged by #55, and both were hit by the operator on first use.

Step 1 named a command that does not exist

It said cd infra/ansible && ./site.sh --tags system --limit k3s_prod. There is no site.sh anywhere in this repo (find . -name site.sh → nothing). It was wrong in four independent ways, which is why the fix spells out the reasoning rather than just swapping the string:

wrong kind of thing — this repo drives Ansible with ansible-playbook <playbook>, never a wrapper script. playbooks/site.yml exists and is the full bring-up chain, which is emphatically not what step 1 wants.
wrong playbook — the kernel-module work is in playbooks/node-baseline.yml.
wrong targeting flag — the convention here is -e k3s_hosts=..., not --limit; infra/ansible/README.md §"Targeting prod" documents it for every playbook and runbooks/node-memory-swap.md already uses it.
no toolbox wrapper — Ansible exists only inside the pinned container, and ansible.cfg is read only from infra/ansible.

Now:

./infra/devcontainer/dev.sh run bash -c \
  'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system'

with the two load-bearing details called out: -e k3s_hosts=k3s_prod because the playbook defaults to k3s_servers — the decommissioned dev group — and --tags system never dropped, because an untagged run also executes the disks tasks, which assert on block-device variables prod deliberately leaves as placeholders (the playbook's own header warns about exactly this).

Step 2 gave the wrong reason the server key is safe from git — and the real reason is fragile

It staged the key at /work/.cache/server.key "inside the repo's gitignored cache". .cache/ is not gitignored. Measured both directions:

git check-ignore -v .cache/server.key      ->  .gitignore:11:*.key
git check-ignore -v .cache/serverkey.txt   ->  (no match)

So the path was safe for a reason I had not identified. An operator who renamed the file — server.pem, wg-server, privatekey — would have had a committable WireGuard private key sitting in the checkout, and the gate would not have caught it: no-plaintext-secrets does not recognise a bare WireGuard key.

The step now names the real rule, marks the .key suffix as the part doing the work, and gives a verification to run before trusting it (git check-ignore -v must print a rule; git status --short must not list the file).

Also recorded: the post-merge state of #55

Its own handoff entry predates the merge and says "the DaemonSet has never been reconciled". Superseded, with measurements against live prod:

Flux applied it by the real path — vpn-ingress lastAppliedRevision: main@sha1:ad3f56e, equal to the GitRepository's artifact revision, Ready=True.
All 18 Flux Kustomizations Ready=True; wait: false behaved as intended (Ready while its own pods are not, so nothing downstream stalled).
hostNetwork confirmed in effect, not just configured — the three pods report pod IPs 192.168.2.10/.11/.12, the node addresses.
Fails closed as designed, legibly: all three Init:0/1 on secret "wg-server-key" not found.

Gate

rc 3, with spec-immutability, decisions-log, lint, l0, render-golden, l1 and approval-policy all pass; rc 3 is the two environmental skips (gitea-settings, conformance L2/L3). Not green — reported as rc 3. Touched requirements: R-OBS-04.

No override needed. runbooks/** and decisions/** are not in infra/gitea/approval-policy.json, so unlike #55 this merges on a green gate alone.

Not verified

The corrected command has not been run — Ansible is an operator action. I verified the playbook, the inventory group, the tag coverage and the flag convention all exist and are documented, but not that the run succeeds or that lsmod then reports the module. Better-evidenced, not proven. And "the referenced file exists" is weaker than "the command is correct" — which is precisely the gap that produced site.sh in the first place.

🤖 Generated with Claude Code

Both defects were mine, in the runbook merged by #55, and both were hit by the operator on first use. ## Step 1 named a command that does not exist It said `cd infra/ansible && ./site.sh --tags system --limit k3s_prod`. **There is no `site.sh` anywhere in this repo** (`find . -name site.sh` → nothing). It was wrong in four independent ways, which is why the fix spells out the reasoning rather than just swapping the string: - **wrong kind of thing** — this repo drives Ansible with `ansible-playbook <playbook>`, never a wrapper script. `playbooks/site.yml` exists and is the full bring-up chain, which is emphatically not what step 1 wants. - **wrong playbook** — the kernel-module work is in `playbooks/node-baseline.yml`. - **wrong targeting flag** — the convention here is `-e k3s_hosts=...`, not `--limit`; `infra/ansible/README.md` §"Targeting prod" documents it for every playbook and `runbooks/node-memory-swap.md` already uses it. - **no toolbox wrapper** — Ansible exists only inside the pinned container, and `ansible.cfg` is read only from `infra/ansible`. Now: ```bash ./infra/devcontainer/dev.sh run bash -c \ 'cd /work/infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags system' ``` with the two load-bearing details called out: `-e k3s_hosts=k3s_prod` because the playbook defaults to `k3s_servers` — the **decommissioned dev** group — and `--tags system` **never dropped**, because an untagged run also executes the `disks` tasks, which assert on block-device variables prod deliberately leaves as placeholders (the playbook's own header warns about exactly this). ## Step 2 gave the wrong reason the server key is safe from git — and the real reason is fragile It staged the key at `/work/.cache/server.key` "inside the repo's gitignored cache". **`.cache/` is not gitignored.** Measured both directions: ``` git check-ignore -v .cache/server.key -> .gitignore:11:*.key git check-ignore -v .cache/serverkey.txt -> (no match) ``` So the path was safe for a reason I had not identified. An operator who renamed the file — `server.pem`, `wg-server`, `privatekey` — would have had a **committable WireGuard private key sitting in the checkout**, and the gate would not have caught it: `no-plaintext-secrets` does not recognise a bare WireGuard key. The step now names the real rule, marks the `.key` suffix as the part doing the work, and gives a verification to run before trusting it (`git check-ignore -v` must print a rule; `git status --short` must not list the file). ## Also recorded: the post-merge state of #55 Its own handoff entry predates the merge and says "the DaemonSet has never been reconciled". Superseded, with measurements against live prod: - Flux applied it **by the real path** — `vpn-ingress` `lastAppliedRevision: main@sha1:ad3f56e`, equal to the GitRepository's artifact revision, `Ready=True`. - All **18** Flux Kustomizations `Ready=True`; `wait: false` behaved as intended (Ready while its own pods are not, so nothing downstream stalled). - **`hostNetwork` confirmed in effect, not just configured** — the three pods report pod IPs `192.168.2.10/.11/.12`, the node addresses. - **Fails closed as designed**, legibly: all three `Init:0/1` on `secret "wg-server-key" not found`. ## Gate rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all pass; rc 3 is the two environmental skips (`gitea-settings`, conformance L2/L3). **Not green — reported as rc 3.** Touched requirements: `R-OBS-04`. **No override needed.** `runbooks/**` and `decisions/**` are not in `infra/gitea/approval-policy.json`, so unlike #55 this merges on a green gate alone. ## Not verified **The corrected command has not been run** — Ansible is an operator action. I verified the playbook, the inventory group, the tag coverage and the flag convention all exist and are documented, but not that the run succeeds or that `lsmod` then reports the module. Better-evidenced, not proven. And "the referenced file exists" is weaker than "the command is correct" — which is precisely the gap that produced `site.sh` in the first place. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gitea-admin added 1 commit 2026-10-06 04:10:39 +00:00

fix(runbooks): vpn-remote-access step 1 named a nonexistent script; step 2 cited the wrong gitignore rule

gate / gate (pull_request) Successful in 1m47s

Details

gate / conformance (pull_request) Has been skipped

Details

18f741d744

Both reported by the operator on first use of the runbook merged in #55.

Step 1 said `cd infra/ansible && ./site.sh --tags system --limit k3s_prod`.
There is no site.sh anywhere in this repo. It was wrong four ways at once:
this repo drives Ansible with `ansible-playbook <playbook>` and never a
wrapper (playbooks/site.yml is the full bring-up chain, not this); the module
work is in playbooks/node-baseline.yml; targeting here is `-e k3s_hosts=...`,
not `--limit`; and Ansible exists only in the pinned toolbox, with ansible.cfg
read only from infra/ansible. Replaced with the real command plus why
`-e k3s_hosts=k3s_prod` matters (the default group is decommissioned dev) and
why `--tags system` must never be dropped (an untagged run also executes the
`disks` tasks, which assert on prod's deliberate placeholder devices).

Step 2 told the operator to stage the server key at /work/.cache/server.key
"inside the repo's gitignored cache". .cache/ is NOT gitignored -- the file was
ignored only incidentally by `*.key` (.gitignore:11). So the path was safe for
a reason I had not identified, and renaming it (server.pem, wg-server) would
have left a committable WireGuard private key in the checkout, which
no-plaintext-secrets does not recognise. The step now names the real rule,
marks the suffix as the load-bearing part, and gives a check-ignore
verification to run before trusting it.

Docs only -- no manifest and no behaviour change. Touches no sensitive path.
Also records the post-merge measurements for #55, whose own entry predates the
merge: Flux applied ad3f56e by the real path, all 18 Kustomizations Ready,
hostNetwork confirmed (pod IPs are the node IPs), and the designed fail-closed
state (Init:0/1 on the missing wg-server-key Secret).

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

jakman85 scheduled this pull request to auto merge when all checks succeed 2026-10-06 04:11:53 +00:00

jakman85 merged commit 06ca9eeb88 into main

2026-10-06 04:12:27 +00:00

jakman85 referenced this issue from a commit

2026-10-06 04:12:28 +00:00

Merge pull request 'fix(runbooks): vpn-remote-access step 1 named a nonexistent script; step 2 cited the wrong gitignore rule' (#58) from fix/vpn-runbook-ansible-command into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#58