Merge branch 'main' into feat/pm-agent-issue-to-pr
One conflict, in clusters/base/kustomization.yaml: PR #55 (vpn-ingress) and this branch both appended a resource line to the same tail. Resolved by keeping BOTH, with main's vpn-ingress first — it is a platform-config-tier leaf, while pm-agent dependsOn apps-wave-0, which is later. Nothing else conflicted. runbooks/kustomization.yaml auto-merged and carries both the vpn-remote-access and pm-agent entries; the handoff entries are separate files and had nothing to merge, which is the whole point of decisions/ being a directory. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
commit
0569d47d49
19 files changed
+1715
-7
No files matched your search
@@ -0,0 +1,147 @@
|
||||
---
|
||||
date: 2026-10-05
|
||||
branch: docs/agents-work-in-their-own-workspace
|
||||
title: One workspace per agent run — the shared checkout is now documented as off-limits
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Agents have been doing their work in the operator's checkout at
|
||||
`/var/home/keith/source/home-network-k3s-impl` and clobbering each other. Nothing in the tree
|
||||
told them not to: worktrees appeared in CLAUDE.md only as §3's *gate workarounds*, i.e. as an
|
||||
awkward edge case, and `decisions/README.md` asserted worktree isolation as a fact about how
|
||||
agents already behave rather than as a rule they must follow. Nowhere was the rule stated.
|
||||
|
||||
Documentation only — no code, no manifests, no pipeline logic.
|
||||
|
||||
- **`CLAUDE.md`: new `## 0. Work in your OWN workspace. Never in the shared checkout.`**
|
||||
Placed before §1 so the section numbers every other file and handoff entry cites (§1, §2b,
|
||||
§3, §7) all keep their meaning. It gives the three commands (`git fetch origin main`;
|
||||
`git worktree add -b <type>/<slug> <path outside the repo> origin/main`; `cd`), an explicit
|
||||
do-not list for the shared checkout (no `checkout`/`switch`, no edits, no
|
||||
`add`/`commit`/`stash`/`restore`, no removing others' worktrees), a measurement table, the
|
||||
"put it outside the repo root" rule, cleanup, what to do if you are already standing in the
|
||||
shared checkout with edits, and what a PM must put in a brief it writes for another agent.
|
||||
- **`CLAUDE.md` intro** now says "Start with §0", with the reason: it is the rule you can
|
||||
break before you have read anything else.
|
||||
- **`CLAUDE.md` §3** retitled "Running the gate from a worktree" and reframed as the normal
|
||||
path rather than an edge case, since §0 now makes every gate run a worktree run. Its list
|
||||
grew from two breakages to three: `spec-immutability`'s skip is *avoidable*, not merely
|
||||
tolerated, if the `ENGINE` shim also mounts the sibling spec repo read-only and passes
|
||||
`-e SPEC_REPO=/home-network-k3s`.
|
||||
- **`CLAUDE.md` §4** gained "One branch, one workspace, one PR".
|
||||
- **`README.md`**: a fourth non-negotiable rule ("No work in a shared checkout"), and
|
||||
"Working on this repo" now opens with how to get out of the checkout before describing the
|
||||
dev container that runs in it.
|
||||
- **`decisions/README.md`**: the worktree-isolation sentence now cites §0 as the requirement
|
||||
instead of describing it as something agents happen to do.
|
||||
|
||||
I did this work the way the new §0 requires — a worktree at
|
||||
`<session scratchpad>/wt-workspace-rule`, outside the repo root, branched from `origin/main`,
|
||||
with the shared checkout left on the branch and with the untracked files I found in it.
|
||||
|
||||
## What I verified
|
||||
|
||||
Every number in the new §0 table was measured in the shared checkout on 2026-10-05, not
|
||||
recalled:
|
||||
|
||||
| Claim | Command | Value |
|
||||
|---|---|---|
|
||||
| shared HEAD is yanked between branches | `git reflog \| grep -c 'checkout: moving'` | **107** |
|
||||
| it is not sitting on `main` | `git worktree list` | `feat/pm-agent-issue-to-pr` |
|
||||
| other runs' uncommitted work is in it | `git status --porcelain` | **2** trees: `infra/agent/` (22:04), `platform/pm-agent/` (22:09) — `stat -c %y` |
|
||||
| its `main` is stale | `git rev-parse main` vs `origin/main` | `4386d14` vs `22c94ef`, **2 behind** |
|
||||
| worktrees registered / already merged | `git worktree list --porcelain` + `git merge-base --is-ancestor <b> main` | **42** branches, **31** merged |
|
||||
| nested worktrees are a gate hazard | `grep -rl '\.claude' conformance/tests platform/monitoring/tests` | **9** scripts carry an explicit exclusion |
|
||||
| the 18× double-count | read `platform/monitoring/tests/run.sh:115-123` | in-tree comment, observed 2026-09-12 |
|
||||
|
||||
**The failure mode was observed live, mid-session, not just reconstructed.** The two untracked
|
||||
trees I measured at 22:04 and 22:09 were committed at 22:20 by another run — `5334abe`
|
||||
`feat(pm-agent): an in-cluster AI Program Manager...`, 1,000+ lines across `infra/agent/`,
|
||||
`clusters/`, `catalog/`, `apps/`, `docs/plans/` and its own `decisions/` entry — onto the shared
|
||||
checkout's checked-out branch. Measured after the fact: the shared checkout's HEAD moved
|
||||
`22c94ef` → `5334abe` and its working tree went clean between my first `git status` and my last,
|
||||
with `git log -1` naming a commit that is not mine. That is §0's "shared HEAD" row happening
|
||||
during the session that wrote §0, and it is now cited in the doc.
|
||||
|
||||
**A claim of mine was wrong and is corrected in the doc.** I had it recorded that
|
||||
`git worktree remove` is blocked by the agent sandbox. Provoked rather than assumed: created
|
||||
a throwaway detached worktree in the scratchpad and removed it — **rc 0**, and
|
||||
`git worktree list` no longer shows it. §0 therefore tells agents to remove their own, and
|
||||
scopes the "blocked" note to what it actually covers: *other* sessions'
|
||||
`.claude/worktrees/agent-*`, some of which are additionally `locked` on git's own terms. The
|
||||
`rework/<pr-number>` escape hatch is now scoped to someone else's stale worktree holding a
|
||||
branch, which is a real and separate problem (31 of them). Both of my own worktrees in this
|
||||
session were removed cleanly at rc 0 after pushing.
|
||||
|
||||
`spec-immutability` is avoidable-not-tolerated: taken from
|
||||
`decisions/2026-10-01-render-selftest-runs-in-gate.md`, which records
|
||||
`stage spec-immutability: pass` under the two-mount shim plus `-e SPEC_REPO=/home-network-k3s`.
|
||||
|
||||
**Gate — `rc 3`, UNDETERMINED, and I am not calling it green.** Pinned toolbox, from this
|
||||
worktree, `DEV_KUBECONFIG=/dev/null` and an `ENGINE` shim bind-mounting both the main
|
||||
checkout's `.git` (at its own host path) and the sibling spec repo read-only with
|
||||
`-e SPEC_REPO=/home-network-k3s`:
|
||||
|
||||
ENGINE=<two-mount shim> DEV_KUBECONFIG=/dev/null ./infra/devcontainer/dev.sh run bash -c './ci/gate.sh' # rc 3
|
||||
|
||||
| stage | result |
|
||||
|---|---|
|
||||
| spec-immutability | **pass** — it genuinely RAN; the second mount is what makes it run from a worktree |
|
||||
| approval-policy | pass, but **N/A**: `event='none'`. It cannot see a PR locally, so this green says nothing. See below |
|
||||
| decisions-log | pass — the three required headings and the frozen-archive pin |
|
||||
| lint · l0 · render-golden · l1 | pass (L1: 3 offline checks, 0 failures, 0 skips, 0 unwired) |
|
||||
| gitea-settings | **SKIPPED** — `SETTINGS_API_TOKEN` unset; it exists only in CI |
|
||||
| conformance (L2/L3) | SKIPPED — `CLUSTER_AVAILABLE!=true`; the decision notes no cluster-needing requirement was touched |
|
||||
|
||||
`gitea-settings` is the *only* stage that did not run, and it cannot be made to run from a
|
||||
workstation, so rc 3 is the ceiling for any local run — not a defect in this branch. The
|
||||
first gate attempt returned rc 3 for a different and real reason (`HEAD is not ahead of
|
||||
origin/main`: I had not committed yet, so the touched-requirement set was empty by default
|
||||
rather than by verification); it was re-run after committing, and the table above is the
|
||||
post-commit run. §3 now records both expected skips, because it previously named only
|
||||
`spec-immutability`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **GATE: UNVERIFIED at the time of writing.** The pinned-toolbox result for this branch and
|
||||
its exit code are reported in the PR, not predicted here. **A local rc 3 is not green** and
|
||||
will not be described as one.
|
||||
- **CI gate: RED, measured — `MERGE GATE: BLOCKED`, one failing stage.** Run 245 on PR #56,
|
||||
head `b988da8b`, 1m47s. The only failure is `approval-policy`: *"a sensitive change lacks a
|
||||
non-author approval"*, `sensitive paths touched: CLAUDE.md`, author `gitea-admin`. Every other
|
||||
stage passed — `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, and `gitea-settings`
|
||||
(which can only run in CI, and reported `required_approvals = 0`,
|
||||
`dismiss_stale_approvals = true`). `spec-immutability` SKIPPED in CI (no sibling spec repo in
|
||||
the CI workspace — the inverse of the local run, where the two-mount shim made it pass) and
|
||||
`conformance` SKIPPED. **Note `gate / conformance (pull_request)` reports status `success`
|
||||
with description "Has been skipped"** — CLAUDE.md §2b's trap, live on this very PR. Nothing
|
||||
in this branch is broken; the red is the structural human gate, below. *Measured at
|
||||
`b988da8b`; this entry was then amended to record it, so the head SHA in the PR is newer than
|
||||
the run quoted here and CI re-ran on the amended commit.*
|
||||
- **This PR is a sensitive-path PR and is CI-RED on `approval-policy` until a human acts.**
|
||||
`CLAUDE.md` is in `infra/gitea/approval-policy.json`'s `sensitive_paths` ("the delivery
|
||||
contract agents follow"), so the stage requires an approving review from someone who is not
|
||||
the author, or an `/approve-override <reason>` posted by an administrator **as a PR review
|
||||
of type "Comment"**, not in the plain comment box. `ci/approval-policy.sh` run locally
|
||||
reports N/A (it cannot see a PR), so the local gate cannot catch this and a green local run
|
||||
says nothing about it. **I have not posted an override and must not** — CLAUDE.md §1 forbids
|
||||
an agent clearing this on its own PR even though `gitea-admin` holds `owner` and could.
|
||||
- **Nothing here is enforced.** §0 is prose. There is no gate stage that detects work done in
|
||||
the shared checkout, no check that a worktree is outside the repo root, and no check that a
|
||||
branch was cut from `origin/main`. Every other rule in CLAUDE.md that agents kept breaking
|
||||
(`DECISIONS-LOG.md` appends, the three handoff headings) ended up as a gate stage, and that
|
||||
is the honest follow-up here. I did not write one: the ask was documentation, and an
|
||||
enforcement stage lands in `ci/**`, which is itself a sensitive path and deserves its own PR.
|
||||
- **I did not clean up the 31 stale merged worktrees or the 35 nested under
|
||||
`.claude/worktrees/`.** They are other sessions' workspaces; removing them is an operator
|
||||
action, and §0 explicitly tells agents not to remove worktrees they did not create.
|
||||
- **I did not touch the untracked `infra/agent/` and `platform/pm-agent/` trees** sitting in the
|
||||
shared checkout. They are another run's in-flight work. Whose, and whether they are wanted,
|
||||
is unknown to me.
|
||||
- **`docs/plans/ssd-etcd-execution-runbook.md:18` still points at a `.claude/worktrees/` path**
|
||||
inside the repo. Left as-is deliberately: that runbook is marked COMPLETED/historical and
|
||||
describes the now-decommissioned dev cluster, so the path is a record of what was done, not
|
||||
an instruction. It does mean a grep for in-repo worktree paths is not yet clean.
|
||||
- **No runtime, cluster or Flux behaviour was touched or checked.** This branch changes three
|
||||
Markdown files; `kubectl` was not run.
|
||||
@@ -0,0 +1,225 @@
|
||||
---
|
||||
date: 2026-10-05
|
||||
branch: feat/vpn-remote-access-wireguard
|
||||
title: Remote-access VPN — WireGuard on udp/51820, hostNetwork because a pod identity would 403 every Gateway hostname
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
New platform component `platform/vpn-ingress/`: a **WireGuard** remote-access VPN so the operator's
|
||||
Android / Linux / Windows / macOS devices can dial in from anywhere on the internet and behave like
|
||||
machines on the LAN — full reach to `192.168.2.0/24` plus internet egress out the house WAN.
|
||||
|
||||
This is the **opposite direction** to the existing `platform/vpn-egress` (pods *out* through PIA,
|
||||
ADR-0015). They share nothing but the word "VPN"; both READMEs now say so, and `README.md`'s layout
|
||||
block labels the two.
|
||||
|
||||
**Transport: WireGuard, and the security argument is the handshake's failure mode, not "it is
|
||||
encrypted".** WireGuard's first handshake message carries a MAC keyed with the responder's own
|
||||
public key, so a sender that does not already know it gets **no reply at all** — no error, no reset,
|
||||
no banner. From the internet, udp/51820 is indistinguishable from a closed port: nothing to
|
||||
brute-force, no version string, no amplification. That is what makes an unfiltered WAN port
|
||||
defensible here, and it is why there is deliberately no `loadBalancerSourceRanges`-style allowlist
|
||||
(which would also break roaming clients). It also has first-party clients on all four named
|
||||
platforms, which OpenVPN (needs a CA + per-device certs + a CRL) and IKEv2 do not.
|
||||
|
||||
**`hostNetwork: true`, and this is THE decision — forced by a measured property of this cluster.**
|
||||
A pod-networked WireGuard server must SNAT client traffic to its own pod IP (Cilium drops a packet
|
||||
whose source is not the endpoint's address), so every remote client would have arrived wearing *that
|
||||
pod's* security identity. `decisions/2026-10-04-plex-advertise-plexdirect.md` measured what that
|
||||
costs: a request to the shared Gateway VIP **from inside a pod returns `403 Access denied` from
|
||||
Envoy for all 16 Gateway hostnames**, because Cilium's Gateway preserves the original source
|
||||
identity and enforces the *destination's* ingress policy, and every app admits `reserved:ingress`
|
||||
plus named peers — never an arbitrary pod. So the obvious build would have produced a tunnel that
|
||||
came up healthy, showed connected peers, routed packets, and 403'd every cluster web UI — and
|
||||
`*.home.keithjackson.dev` would not even have resolved, because `apps/wave-0/pihole-dns.yaml`'s
|
||||
ingress policy deliberately omits the `cluster` entity. In the host netns there is no pod identity
|
||||
to carry: forwarded traffic is SNATed to the node address and arrives as `host`/`remote-node`,
|
||||
byte-for-byte what a LAN client looks like. Three further problems vanish with it: `ip_forward` is
|
||||
already 1 on these nodes (so no `--allowed-unsafe-sysctls` kubelet change — it is *not* a safe
|
||||
sysctl); no LoadBalancer VIP, so no L2-announcement lease lottery and no `externalTrafficPolicy`
|
||||
trade-off; and no double encapsulation, which is what makes `MTU = 1420` simply correct.
|
||||
|
||||
`hostNetwork` is permitted by `restrict-host-namespaces`' own stated carve-out (it matches
|
||||
`app-namespace=true` only, and names Cilium/node-exporter as the precedent); `namespace.yaml` is
|
||||
deliberately unlabelled. **The cost is stated, not buried:** a hostNetwork pod has no Cilium
|
||||
endpoint, so **no NetworkPolicy constrains this workload**. `networkpolicy.yaml` says that at
|
||||
length rather than shipping an `allow world → 51820` rule that would read as the control and
|
||||
enforce nothing; the real boundary is WireGuard's peer authentication.
|
||||
|
||||
**A DaemonSet, not a Deployment.** The WAN forward names one node address by hand, off-repo (same
|
||||
shape as the plex tcp/32400 forward). A floating single replica would silently invalidate it on
|
||||
every reschedule; an identical listener on all three nodes makes **failover a one-line router
|
||||
change** with no cluster work and no re-issued client profiles. A pinned LoadBalancer VIP was the
|
||||
considered alternative and was rejected: it reintroduces the L2 lease, SNAT and an extra node hop
|
||||
into the one path whose failure mode is "I am away from home and cannot get in".
|
||||
|
||||
**Authorisation is split so that it stays reviewable.** Peer **public** keys live in
|
||||
`platform/vpn-ingress/peers.yaml` in **plaintext** — a public key grants nothing by itself, and this
|
||||
makes "who can get into my house" a PR diff with `git log` as its audit trail. The only real secret,
|
||||
the server's private key, is the `wg-server-key` Secret, **created out of band by the operator**: an
|
||||
agent must not mint a long-lived private key (CLAUDE.md §6), and a key that passed through a session
|
||||
transcript would have to be treated as compromised. The two halves are joined only at pod start, by
|
||||
an init container, into a `medium: Memory` emptyDir — so the assembled config holding the key never
|
||||
touches a node's disk and dies with the pod.
|
||||
|
||||
**There are no peers yet, and that is the correct initial state**, for the same reason. The VPN ships
|
||||
INERT-BUT-HEALTHY: a zero-peer interface listens and refuses every handshake.
|
||||
|
||||
Files:
|
||||
|
||||
- `platform/vpn-ingress/` (new): `namespace.yaml`, `peers.yaml` (the plaintext peer register),
|
||||
`daemonset.yaml` (hostNetwork listener + config assembly), `networkpolicy.yaml` (namespace
|
||||
baseline, with the honest caveat), `kustomization.yaml`, `README.md` (the design record).
|
||||
- `clusters/base/vpn-ingress.yaml` (new): its own **non-gating post-platform** Flux Kustomization
|
||||
(`dependsOn platform-config`, `wait:false`). Non-gating is load-bearing rather than conventional
|
||||
here: the DaemonSet *cannot* start until the operator creates the server key, so in the gating
|
||||
`platform` tier an unfinished manual step would have stalled every apps wave on merge.
|
||||
- `clusters/base/kustomization.yaml`: the new tier added.
|
||||
- `clusters/base/cluster-vars.yaml`: new `WG_CLIENT_CIDR` (`10.90.0.0/24`), declared next to
|
||||
`LAN_CIDR` because it is the one range that must be collision-checked against every other; the
|
||||
comment enumerates the five it avoids. **No prod override, deliberately** — unlike every VIP, this
|
||||
range is private to the tunnel, so the same value is correct in both environments and a prod patch
|
||||
would imply otherwise. Substitution inventory updated.
|
||||
- `infra/ansible/roles/node_baseline/defaults/main.yml`: `wireguard` added to
|
||||
`node_baseline_kernel_modules`. The pod has `NET_ADMIN` but deliberately **not** `SYS_MODULE`, so
|
||||
it cannot autoload the module — it CrashLoops with a clear error rather than quietly degrading to
|
||||
userspace wireguard-go (which would need `/dev/net/tun`, a device a hostPath mount cannot actually
|
||||
grant, as the `/dev/dri` attempt proved).
|
||||
- `infra/vpn-ingress/new-peer.sh` (new): enrols a device. Generates the keypair on the operator's
|
||||
machine, writes a 0600 profile **outside** the checkout (it refuses to write into the repo),
|
||||
prints a QR code for phones, and prints only the public `[Peer]` stanza. It also refuses a
|
||||
duplicate address, because two devices on one address does not error anywhere — the server routes
|
||||
to whichever handshook last, so both appear to work intermittently.
|
||||
- `runbooks/vpn-remote-access.md` (new) + its `kustomization.yaml` generator entry + `README.md`
|
||||
index line: the three manual steps, the four off-LAN verification checks, revocation, and why no
|
||||
connection alert exists.
|
||||
- `README.md`: `vpn-ingress` added to the layout, with the egress/ingress distinction spelled out.
|
||||
|
||||
**THREE MANUAL STEPS REMAIN AND THE VPN DOES NOT WORK UNTIL ALL THREE ARE DONE** (all off-repo by
|
||||
necessity; `runbooks/vpn-remote-access.md`): run `node_baseline` to load the kernel module, create
|
||||
the `wg-server-key` Secret, and forward WAN udp/51820 to one node address on the SOHO router.
|
||||
|
||||
**SENSITIVE PATHS — this PR touches `clusters/**` (three files), so `ci/approval-policy.sh` will
|
||||
BLOCK it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
|
||||
"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
|
||||
PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
|
||||
|
||||
## What I verified
|
||||
|
||||
Live prod, read-only, 2026-10-05, plus the offline gate.
|
||||
|
||||
- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
|
||||
`spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
|
||||
**pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
|
||||
and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
|
||||
Run from an isolated git worktree with both CLAUDE.md §3 workarounds applied
|
||||
(`DEV_KUBECONFIG=/dev/null` plus an `ENGINE` shim bind-mounting the main checkout's `.git` and
|
||||
the sibling spec repo) — which is why `spec-immutability` is **green here and skipped in CI**,
|
||||
where the runner has no spec checkout. `approval-policy` passing locally is an offline N/A, not a
|
||||
verdict; the real verdict is the CI run below.
|
||||
- **The CI gate ran on PR #55 (run 244, head `1a67fc1b`) and `approval-policy` is the ONLY failing
|
||||
stage** — read from the job log, not from the red tick. Verbatim: "This PR changes the pipeline or
|
||||
the security posture it rests on, and has NO approving review from anyone other than its author
|
||||
('gitea-admin')", naming exactly the three files I predicted:
|
||||
`clusters/base/cluster-vars.yaml`, `clusters/base/kustomization.yaml`,
|
||||
`clusters/base/vpn-ingress.yaml`. Every other stage that ran reported green —
|
||||
`gitea-settings`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`. So the branch is blocked
|
||||
on the operator's `/approve-override <reason>` (posted as a PR **review** of type "Comment") and
|
||||
on nothing else. **I have not posted it** — CLAUDE.md §1.
|
||||
- **Branch protection is live, measured by CI's own `gitea-settings` stage** (it has the token this
|
||||
workstation lacks): `required_approvals = 0`, `enable_status_check = true`,
|
||||
`status_check_contexts = ["gate / gate (pull_request)"]`, `block_admin_merge_override = true`,
|
||||
`enable_push = false`, `dismiss_stale_approvals = true`, and **1 runner online**. That is the §2b
|
||||
re-measurement of CLAUDE.md §2's table for this branch, not a restatement of it.
|
||||
- **`flux-postbuild-substitution` (L0) actually exercised the new wiring, and caught a real defect
|
||||
in it.** On the first run it FAILED, naming `platform/vpn-ingress/daemonset.yaml:118` and
|
||||
attributing it to "Flux Kustomization 'vpn-ingress' (clusters/base/vpn-ingress.yaml, path
|
||||
./platform/vpn-ingress)". Two things are proven by that message: the new tier's
|
||||
`postBuild.substituteFrom` declaration is correctly detected (so `${WG_CLIENT_CIDR}` will be
|
||||
substituted, not shipped literally), and the check reads comments as well as values — the hazard
|
||||
was a comment that spelled out the forbidden braced form literally while claiming not to use it.
|
||||
Fixed; the check now passes 444/444 with 0 hazards.
|
||||
- **The `wireguard` kernel module is available on these nodes by construction**: all three run
|
||||
Debian 13 (trixie) on kernel `7.1.8+deb13-amd64` (measured), and WireGuard has been in-tree since
|
||||
5.6, so it is present in the distro kernel and only needs loading.
|
||||
- **`net.ipv4.ip_forward = 1` is already written by `node_baseline`**
|
||||
(`roles/node_baseline/tasks/main.yml`, `/etc/sysctl.d/99-k3s-cilium.conf`), and **no kubelet
|
||||
carries `--allowed-unsafe-sysctls`** (grepped across `roles/k3s_server`, `roles/node_baseline` and
|
||||
`group_vars`: zero hits). That pair is what makes the host netns the only option that does not
|
||||
require a node-restart change.
|
||||
- **Cilium's datapath, measured from `cilium-config` rather than from the values file**:
|
||||
`enable-ipv4-masquerade=true`, `enable-bpf-masquerade=true`, `routing-mode=tunnel`,
|
||||
`tunnel-protocol=vxlan`, `ipv4-native-routing-cidr` EMPTY, Cilium `v1.20.1`. The vxlan underlay is
|
||||
where `MTU = 1420` comes from, and the empty native-routing CIDR is why pod→LAN is masqueraded
|
||||
today.
|
||||
- **`WG_CLIENT_CIDR` collides with nothing.** Pod CIDR `10.0.0.0/16` (`CLUSTER_POD_CIDR`), service
|
||||
CIDR `10.43.0.0/16` (measured: `kubernetes` Service clusterIP is `10.43.0.1`), the PIA tunnel's
|
||||
`10.72.192.0/18` (`platform/vpn-egress/README.md`), and both LANs `192.168.1.0/24` /
|
||||
`192.168.2.0/24`. `10.90.0.0/24` is outside all five.
|
||||
- **The egress-class label matters and is set to `internet`.** Read live:
|
||||
`CiliumEgressGatewayPolicy vpn-egress-steering` selects `podSelector:
|
||||
homelab.keithjackson.dev/egress-class: vpn` with **no namespace scope**, so `vpn` here would have
|
||||
steered every remote client's internet traffic out PIA.
|
||||
- **All three nodes are untainted** (`spec.taints: None` on each), so the DaemonSet covers the
|
||||
cluster with no tolerations.
|
||||
- **`new-peer.sh` was run end to end** against a throwaway server key in the scratchpad (deleted
|
||||
immediately after; it never existed in the repo). It rendered a correct profile — `Address
|
||||
10.90.0.7/32`, `DNS 192.168.2.23`, `Endpoint wan.home.keithjackson.dev:51820`, `MTU 1420`,
|
||||
`PersistentKeepalive 25`, `AllowedIPs 0.0.0.0/0` — at mode 0600, printed the matching public-key
|
||||
stanza, and both argument-validation paths (missing name, out-of-range octet) refuse correctly.
|
||||
The private key was redacted from every line I displayed.
|
||||
- **The endpoint name is resolver-independent, which is the specific trap this avoids.**
|
||||
`wan.${BASE_DOMAIN}` is an unproxied public Cloudflare A record kept current by
|
||||
`platform/externaldns/ddns.yaml` (checked every 5 min), so it resolves from a cellular connection
|
||||
— unlike any name PiHole answers, which a remote client cannot resolve until it is already
|
||||
connected. (This is the `plex.direct` lesson applied.)
|
||||
- **Velero does cover this namespace.** `platform/velero/helmrelease.yaml`'s `daily` schedule
|
||||
excludes only `monitoring` and `unifi`, so `vpn-ingress` and the `wg-server-key` Secret are
|
||||
captured off-cluster to R2. Read from the manifest, not from a backup listing.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **THE ACTUAL FIX. Nothing in this branch has been proven to carry a single packet.** The
|
||||
DaemonSet has never been reconciled, no `wg0` has ever existed, and no handshake has ever been
|
||||
attempted. Three blockers are off-repo and manual (kernel module, server key, WAN forward), and
|
||||
this branch removes none of them — it only makes them the *only* remaining work.
|
||||
- **Outside-in reachability, at all.** This repo has **no off-LAN vantage point**, which is exactly
|
||||
the gap the plex remote-access work hit. Step 6 of the runbook (a phone on cellular) is the only
|
||||
honest test and it has not been run.
|
||||
- **The in-pod `iptables` → host-netfilter interaction, which is my biggest single doubt.** Debian 13
|
||||
uses nf_tables; the container writes MASQUERADE/FORWARD rules into the **host's** tables via the
|
||||
image's own `iptables`. If that binary is the legacy backend while the host is nft, the rules can
|
||||
apply to a table the kernel does not consult — and the symptom would be the nastiest available:
|
||||
handshakes succeed, `wg show` looks perfect, and every forwarded connection hangs. **Check this
|
||||
first if the tunnel comes up but nothing is reachable:** `iptables-save -t nat | grep MASQUERADE`
|
||||
and `nft list table ip nat` on the node, from the host, not from the pod.
|
||||
- **Whether `allowPrivilegeEscalation: false` is compatible with the image's s6 init.** It is the
|
||||
correct posture and I set it, but this repo has already been bitten by `no-new-privileges`
|
||||
misbehaving (`platform/externaldns/ddns.yaml`'s README note). If the pod fails before
|
||||
`wg-quick` runs, that flag is the first thing to try flipping.
|
||||
- **Whether `NET_ADMIN` alone is sufficient.** Reasoned from what `wg-quick` does (create the link,
|
||||
set a sysctl, write two netfilter rules), not measured. If `iptables` complains, `NET_RAW` is the
|
||||
likely addition — deliberately not granted pre-emptively.
|
||||
- **The image's custom-config path.** `lscr.io/linuxserver/wireguard:1.0.20260223-r0-ls124` (tag
|
||||
confirmed to exist and to be the current one, published 2026-10-01) is expected to bring up
|
||||
whatever is in `/config/wg_confs/` when `PEERS` is unset. That is its documented behaviour; it has
|
||||
not been run here, and the init container's assembly of that file has never executed.
|
||||
- **Whether the probe is right.** `wg show wg0` is asserted to exit non-zero when the interface is
|
||||
absent. Not executed.
|
||||
- **Cilium's reaction to a new `wg0` in the host netns.** `platform/cilium/values.yaml` does not pin
|
||||
`devices`, so Cilium auto-detects; whether it attaches to `wg0` — and whether that matters — is
|
||||
unknown. `wg` interfaces are already filtered out by `node_health_collect.py`, so something in
|
||||
the estate anticipates them, but that is not evidence about Cilium.
|
||||
- **Zero peers means the end-to-end authorisation path is untested**, including revocation. The
|
||||
first enrolment is also the first test of `peers.yaml`'s format being what `wg-quick` accepts.
|
||||
- **No peer-level observability exists.** An exporter publishing
|
||||
`wireguard_latest_handshake_seconds` is the top follow-up; until then "who is connected" is an
|
||||
on-demand `wg show` and nothing alerts on an unexpected session. Pod-down is covered only by
|
||||
kube-prometheus-stack's generic `KubePodNotReady` / `KubeDeploymentReplicasMismatch`, which I did
|
||||
not confirm fire for a DaemonSet in this namespace.
|
||||
- **The security consequence I am most keen for a human to review rather than take from me:** an
|
||||
enrolled device reaches the three k3s nodes themselves — apiserver 6443, kubelet 10250, SSH 22 —
|
||||
because "reach the entire local network" includes them. That is stated in the README and the
|
||||
runbook with the narrowing recipe, but it is a posture decision, not a verified fact, and it is
|
||||
the right thing to argue about on this PR.
|
||||
+2
-1
@@ -26,7 +26,8 @@ The fix is the obvious one for concurrent appenders: **stop sharing a file**. Ea
|
||||
handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
|
||||
nothing for git to merge and nothing to conflict. This scales to any number of parallel
|
||||
branches with zero coordination — which is the point, since the agents are deliberately
|
||||
isolated from each other in separate worktrees and cannot coordinate.
|
||||
isolated from each other in separate worktrees (required by `CLAUDE.md` §0) and cannot
|
||||
coordinate.
|
||||
|
||||
There is deliberately **no index file** and no counter. An index would be a shared
|
||||
mutable file — exactly the thing that just failed — and a sequence number would have
|
||||
|
||||
Reference in new issue
Block a user