Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5 7fa5e84d25 feat(pm-agent): the kubelet's pull credential, and the seventh field admission had not read
gate / gate (pull_request) Failing after 1m53s
gate / conformance (pull_request) Has been skipped
The registry had two blockers. #59 fixed the first (nodes resolved the hostname to the
non-hairpinned WAN and failed TLS against the router's CN=%D cert). This is the second:
the pull is unauthenticated, and this registry refuses unauthenticated clients outright —
measured 2026-10-05, its anonymous token carries access: null and every path 401s.

pm-agent-registry is a dockerconfigjson Secret (SOPS, placeholders) referenced by
imagePullSecrets on the dispatcher Deployment and on the committed worker Job template. It
holds the SAME Gitea token as pm-agent-credentials, because the two consumers need two
formats: the application reads a raw token from env, the kubelet reads a dockerconfigjson
before any container starts. Same one-value-in-two-files pattern as the Keycloak client
secret, documented in both .example files. The operator's part is the value plus one new
scope: read:package.

ADMISSION GAINED A SEVENTH RULE. imagePullSecrets is a pod-spec field no container
references, which is exactly why it is easy to leave unread — the same shape of omission
as initContainers, ephemeralContainers, env and envFrom before it. The kubelet reads the
named Secret, in this namespace, before anything runs. Rule 1's pattern now REQUIRES the
committed entry; rule 7 forbids any other and any second one. Both halves are needed: a
foreach over an absent list yields no result.

The leverage is small and the policy says so: the image is digest-pinned, so a different
pull credential cannot change WHAT runs. Pinned anyway — 'only the committed shape' is the
premise, and the rule costs four lines.

kyverno test: 33 passed, 0 failed (was 28). Three new fixtures, each a real path: the
committed entry plus a second one, a different Secret, and none at all.

The gate also caught me shipping the Secret in plaintext — I never ran sops -e -i on the
placeholder. Recorded in the handoff entry, because the file looked finished.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-06 01:03:54 -05:00
39 changed files with 749 additions and 1477 deletions

No files matched your search

-8
View File
@@ -37,14 +37,6 @@ coverage/
infra/tofu/backend.hcl
infra/tofu/.r2.env
# Operator scratch, writable from inside the dev container (the repo is its only persistent
# read-write mount). Holds things that must never be committed: a staged copy of the SOPS age or
# WireGuard key, rendered VPN client profiles (infra/vpn-ingress/new-peer.sh writes here by
# default). `infra/ansible/.gitignore` already does this for its own .cache/; this is the same
# convention at the root. Do NOT rely on `*.key` alone to protect a staged key — that only covers
# one filename shape, which is why this rule exists.
.cache/
# Claude Code local scratch: agent worktrees + session state. Nothing here is tracked;
# ignoring it keeps `git add -A` from staging embedded worktree repos and keeps the lint
# stage (which scans untracked files) from tripping on .claude/worktrees/.
+1 -1
View File
@@ -77,7 +77,7 @@ spec:
storageClassName: longhorn
resources:
requests:
storage: 16Gi
storage: 8Gi
---
# PVC plex-media (R-STATE)
apiVersion: v1
+1 -1
View File
@@ -351,7 +351,7 @@ exposure: public # edge port-forward + public DNS; clients + Seerr
egress: internet # normal WAN egress (docs/03; R-NET-05)
storage:
- name: config
size: 16Gi
size: 8Gi
class: longhorn # Plex DB/metadata: replicated app state on the 3-replica tier (docs/04; R-STATE-01/02)
mountPath: /config # plexinc/pms-docker writes its identity/claim + "Library" DB under /config, NOT the
# renderer default /data/config. Without this the PVC sat unused and Plex wrote to the
+2 -2
View File
@@ -359,7 +359,7 @@ storage:
# The live /config contents were copied onto each PVC before this landed,
# so API keys and credentials other apps reference were preserved.
# THE DOWNLOADS AREA — ONE mount of the SAME export sonarr/radarr/plex already consume
# (192.168.2.30:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
# (192.168.2.102:/mnt/Storage/plex), narrowed by subPath to the `downloads` subtree and presented at
# /downloads, which is precisely where this app is ALREADY configured to write.
#
# WHY subPath: downloads AND NOT THE EXPORT ROOT AT /media (the shape sonarr/radarr use):
@@ -434,7 +434,7 @@ dependencies: [] # default-deny: reaches only DNS; arr reach it n
# for weeks and neither has any CNP naming the NFS server or port 2049
# (grep apps/wave-1/sonarr.yaml: none). The SAME fact is what keeps the VPN
# egress gateway off this traffic (its selector is a podSelector), and
# 192.168.2.30 is inside the 192.168.0.0/16 excludedCIDR besides — see
# 192.168.2.102 is inside the 192.168.0.0/16 excludedCIDR besides — see
# platform/vpn-egress/egress-steering.yaml, "BULK NFS".
ingressRoutes:
- host: qbittorrent.${BASE_DOMAIN}
+15 -17
View File
@@ -234,28 +234,26 @@ data:
WG_CLIENT_CIDR: "10.90.0.0/24" # remote VPN clients; .1 = server. NOT a LAN range
# --- Bulk (external NFS) storage host (docs/04 bulk tier) ---------------------------------------
# THE address of the NAS whose ZFS datasets back every `class: bulk` volume: the media library
# (Storage/plex, 8.33 T — plex, sonarr, radarr, qbittorrent) and the legacy Nextcloud archive
# THE address of the Proxmox host whose ZFS datasets back every `class: bulk` volume: the media
# library (Storage/plex, 8.33 T — plex, sonarr, radarr) and the legacy Nextcloud archive
# (Storage/nextcloud, 270 G). The renderer emits it into each static NFS PersistentVolume's
# `spec.nfs.server` (catalog/render.py, BULK_NFS_SERVER); the export PATH is per-dataset and is
# declared in the ServiceDefinition, because the path is identical in every environment and only
# the ADDRESS is not.
#
# IT IS NO LONGER A CROSS-NETWORK KNOB (2026-10-07). This used to read "the two clusters are on
# OPPOSITE SIDES of pfSense": the datasets lived on a Proxmox host whose only interface was
# vmbr0 192.168.1.9/24 on pfSense's LAN, which prod (192.168.2.0/24, pfSense's WAN) could reach
# only through a DNAT at 192.168.2.102. Both pools were physically moved onto a standalone
# TrueNAS at 192.168.2.30 — prod's OWN /24 — so there is no crossing to arrange and no candidate
# values to choose between. Measured 2026-10-07: 192.168.2.102:2049 no longer answers;
# 192.168.2.30:2049 and :111 do. There is now exactly ONE storage host and one correct value;
# this stays a knob only because the renderer reads it per-environment.
#
# *** CHANGING THIS VALUE IS NOT A PLAIN RECONCILE. *** A bound PersistentVolume's spec.nfs is
# immutable — measured by server dry-run 2026-10-07:
# spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
# so Flux's apply FAILS rather than updating, and the pods keep their old mount. The PVs must be
# deleted and recreated; the operator sequence is docs/BULK-NFS-STORAGE.md §10.
BULK_NFS_SERVER: "192.168.2.30"
# WHY THIS IS A KNOB AND NOT A LITERAL — the two clusters are on OPPOSITE SIDES of pfSense.
# The storage host has exactly one interface, vmbr0 192.168.1.9/24, on pfSense's LAN.
# dev nodes are 192.168.1.40/.41/.42 — the SAME L2 segment. They reach it directly, and one
# NFS export to them already exists on that host (/mnt/Storage/k8s-volumes in
# /etc/exports), so this path is proven, not theoretical. Base default = the dev value,
# per the convention of the LAN block above.
# prod nodes are 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does NOT traverse that today;
# 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening
# the storage path is a PENDING OPERATOR DECISION with two candidate outcomes that give
# DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route).
# So prod holds a loud TBD-PROD-* placeholder — see clusters/prod/kustomization.yaml for the
# blast radius of shipping it unset, which is NOT the usual "fails to apply".
BULK_NFS_SERVER: "192.168.1.9"
# Off-cluster S3 backup target (R-STATE-03, R-IDP-05) — the on-Proxmox MinIO, plain HTTP, path-style.
# Consumed by BOTH backup paths: every CNPG Cluster's barmanObjectStore.endpointURL (keycloak-db,
+5 -8
View File
@@ -8,8 +8,8 @@
# NOTHING downstream reconciles — the incident behind the headlamp/oauth2-proxy/homepage moves
# (DECISIONS-LOG: "Headlamp gating stall").
# * SPECIFIC, AND THE REASON THIS IS NOT A JUDGEMENT CALL: this DaemonSet cannot start until the
# operator replaces the `REPLACE-ME` placeholder in platform/vpn-ingress/wg-server-key.sops.yaml
# with a real key (runbooks/vpn-remote-access.md step 2). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
# operator creates the out-of-band `wg-server-key` Secret (runbooks/vpn-remote-access.md
# step 1). That is the correct fail-closed state — an unenrolled VPN must not come up — but in
# the gating tier it would have been a self-inflicted platform-wide outage ON MERGE, with the
# cause (a missing manual step) invisible from any Flux condition.
#
@@ -51,9 +51,6 @@ spec:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
# SOPS decryption is layered per-environment by the overlay (clusters/dev and clusters/prod each
# patch THIS Kustomization by name), identical to headlamp/crafty/velero — this path carries
# platform/vpn-ingress/wg-server-key.sops.yaml, the WireGuard server private key. The `sops-age`
# Secret is created out of band and referenced by NAME only. Without the overlay patch the Secret
# applies cleanly full of ENC[...] and only the tunnel is broken, so the patch is as load-bearing
# as the encrypted file itself.
# NO `decryption:` stanza is needed in the overlays: this path carries no *.sops.yaml. The server
# key is an out-of-band Secret (platform/vpn-ingress/kustomization.yaml says what to do if that
# ever changes).
+7 -25
View File
@@ -122,24 +122,6 @@ patches:
provider: sops
secretRef:
name: sops-age
# vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
# platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
# secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
# Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
# the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
# validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
# The `sops-age` Secret is out-of-band (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: vpn-ingress
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
# pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
# account and the Anthropic subscription token from `claude setup-token`. Without this
@@ -279,15 +261,15 @@ patches:
- op: replace
path: /data/CRAFTY_L4_VIP
value: "192.168.1.55"
# Bulk (external NFS) storage host. THIS VALUE IS INERT — dev is decommissioned, its nodes
# (192.168.1.40/.41/.42) no longer exist, and the Proxmox host this used to name (192.168.1.9)
# went with the pools it held. It is restated per this overlay's "supply every knob"
# convention, not because anything consumes it. It carries the real address so that no file
# in the tree renders a host that is gone; it has NOT been proven from a dev node, because
# there is no dev node to prove it from.
# Bulk (external NFS) storage host: the Proxmox host itself, on the SAME LAN as dev's nodes
# (192.168.1.40/.41/.42), which already hold an NFS export from it (/mnt/Storage/k8s-volumes).
# Restated verbatim rather than inherited, per this overlay's "supply every knob" convention.
# Dev is therefore the environment in which the bulk manifests can actually be VALIDATED
# end-to-end before prod's network path exists — provided the additive `zfs set sharenfs=`
# change described in docs/BULK-NFS-STORAGE.md is applied for the dev node addresses.
- op: replace
path: /data/BULK_NFS_SERVER
value: "192.168.2.30"
value: "192.168.1.9"
# Off-cluster backup target: the on-Proxmox MinIO (plain HTTP, path-style) that every CNPG
# barmanObjectStore and Velero's backupStorageLocation write to.
#
+47 -53
View File
@@ -113,24 +113,6 @@ patches:
provider: sops
secretRef:
name: sops-age
# vpn-ingress (the WireGuard remote-access VPN, clusters/base/vpn-ingress.yaml) carries
# platform/vpn-ingress/wg-server-key.sops.yaml — the WireGuard SERVER private key, the one
# secret in that component. Without this stanza kustomize hands Flux the raw ciphertext and the
# Secret applies full of ENC[...]: the APPLY SUCCEEDS and every Flux condition stays green, and
# the only symptom is a VPN that no client can handshake with. The DaemonSet's init container
# validates the value with `wg pubkey` specifically so that state is a named CrashLoop instead.
# The `sops-age` Secret is out-of-band (name only).
- patch: |-
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: vpn-ingress
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
# pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
# pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
# account and the Anthropic subscription token from `claude setup-token`. Without this
@@ -540,48 +522,60 @@ patches:
path: /data/LAN_CIDR
value: "192.168.2.0/24"
# ------------------------------------------------------------------------------------------
# BULK (EXTERNAL NFS) STORAGE HOST — the TrueNAS at 192.168.2.30.
# BULK (EXTERNAL NFS) STORAGE HOST — *** NOT YET DECIDED. THIS PLACEHOLDER IS LOAD-BEARING. ***
# ------------------------------------------------------------------------------------------
# Backs every `class: bulk` volume: the 8.33 T media library (Storage/plex — plex, sonarr,
# radarr, qbittorrent) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
# radarr) and the 270 G legacy Nextcloud archive (Storage/nextcloud).
#
# THIS WAS THE REPO'S ONE LOAD-BEARING PLACEHOLDER, AND THE REASON IT COULD BE IS NOW GONE.
# The datasets lived on a Proxmox host with a single interface on pfSense's LAN
# (192.168.1.9), which prod could reach only through a DNAT at 192.168.2.102 — so the correct
# value here depended on an undecided operator choice, and guessing it would have authored a
# cluster that mounts nothing. Both pools were physically moved onto a standalone TrueNAS at
# 192.168.2.30, prod's OWN /24. No crossing to arrange, no candidates to choose between.
# Measured 2026-10-07: 192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and :111 do.
# THE STORAGE HOST IS NOT REACHABLE FROM PROD AT ALL RIGHT NOW. It has one interface,
# vmbr0 192.168.1.9/24, on pfSense's LAN; prod's nodes are on 192.168.2.0/24, pfSense's WAN.
# 22 and 8006 answer from prod only because pfSense port-forwards them via 192.168.2.102;
# 2049 and 111 do not. Opening that path is an OPERATOR decision, and the two candidate
# outcomes give DIFFERENT values here:
# port-forward 192.168.2.102:2049 -> 192.168.1.9:2049 => "192.168.2.102"
# a WAN->LAN route to 192.168.1.0/24 => "192.168.1.9"
# Guessing either one authors a prod cluster that mounts nothing, or — worse, if the guess
# ever became reachable — mounts something unintended. So it stays a placeholder.
#
# ONE PORT IS STILL ALL IT NEEDS: tcp/2049. The PVs mount nfsvers=4.1 and NFSv4 carries mount
# and lock in-protocol — no rpcbind/111, no mountd, no statd. The NAS advertises
# `+3 +4 +4.1 +4.2` (/proc/fs/nfsd/versions, measured), so 4.1 negotiates.
# WHICH PORTS THE DECISION MUST OPEN: exactly ONE, tcp/2049. The PVs mount nfsvers=4.1, and
# NFSv4 carries mount and lock in-protocol — no rpcbind/111, no mountd, no statd. Choosing v4
# over v3 was in part this decision's cost: v3 would have needed four ports and three of them
# on ephemeral numbers.
#
# WHAT THE EXPORT MUST ALLOW IS ALREADY SETTLED, and it is the same under BOTH outcomes.
# Measured 2026-09-07 from this workstation (192.168.2.132) through the existing port-forward:
# the storage host logged the connection as coming FROM 192.168.2.132, not from a pfSense LAN
# address (`ss -tn` peer 192.168.2.132, and the host's only route off-LAN is its default via
# 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server will see prod's REAL node
# addresses either way. The additive `zfs set sharenfs=` values are in docs/BULK-NFS-STORAGE.md.
#
# *** BLAST RADIUS — READ BEFORE MERGING THIS TO A BRANCH PROD RECONCILES. ***
# A PersistentVolume's spec.nfs.server is an unvalidated string: the API server ACCEPTS this
# placeholder, the PV is created and the PVC BINDS to it (binding is pure control-plane). The
# failure surfaces only when a pod mounts — MountVolume.SetUp "Failed to resolve server
# TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in
# ContainerCreating on prod. Unlike the LAN knobs above, this placeholder does NOT fail at
# apply time and there is no CRD validation to lean on. Set this value BEFORE prod reconciles
# a revision containing the bulk volumes; the dev overlay renders a working value, which is
# where these manifests are meant to be proven first.
- op: replace
path: /data/BULK_NFS_SERVER
# VERIFIED ON THE NAS 2026-10-07, before any cluster touched it (`exportfs -v`):
# /mnt/Storage/plex 192.168.2.10/.11/.12 rw,sec=sys,secure,root_squash,no_all_squash
# /mnt/Storage/nextcloud 192.168.2.10/.11/.12 ro,sec=sys,secure,root_squash,no_all_squash
# 192.168.2.102 is pfSense's WAN address, which DNATs tcp/2049 through to the Proxmox
# host's REAL address 192.168.1.9 on pfSense's LAN. Prod sits on the WAN side and cannot
# reach 192.168.1.9 directly, so the cluster mounts the NAT address. Dev is on the LAN and
# keeps the base value 192.168.1.9.
#
# no_all_squash is the flag the entire permission model rests on: client uids pass through
# untouched, so the arrs' PUID=1000 arrives as the owner of a 0755 1000:3000 tree. sec=sys
# keeps that numeric rather than principal-mapped. The export deliberately does NOT carry
# no_root_squash, which the Proxmox one did: nothing here needs it, because every write to
# the tree happens after the LSIO drop to uid 1000 and the images only chown their Longhorn
# /config. Both subPath directories (plex/downloads, nextcloud/nextcloud) already exist at
# 0755 1000:3000, so a squashed root can still traverse them for subPath resolution.
#
# The ro= posture on the archive is now SERVER-enforced, which the old export also managed.
# The media tree's read-only half (plex) stays a CLIENT-side guarantee — all four media pods
# can land on any node, so the server sees the same three addresses for reader and writers
# alike and cannot tell them apart. Unchanged by the move; see docs/BULK-NFS-STORAGE.md §5.
#
# *** CHANGING THIS VALUE REQUIRES DELETING AND RECREATING THE PVs. *** A bound PV's
# spec.nfs is immutable — measured by server dry-run 2026-10-07:
# spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation
# so Flux's apply FAILS rather than updating and the pods keep their old, dead mount. The
# operator sequence is docs/BULK-NFS-STORAGE.md §10. Retain + the renderer's pre-set
# claimRef make it safe: no data is touched and each recreated PV rebinds its own PVC.
value: "192.168.2.30"
# PROVEN from k3s-prod-1 on 2026-09-12, not inferred:
# - mount of 192.168.2.102:/mnt/Storage/plex negotiated nfs4 vers=4.1 proto=tcp
# - a write as uid 1000 succeeded. Tested as 1000 and NOT root deliberately: the export
# carries no_root_squash, so a root write would have passed and proved nothing, while
# LSIO images default to PUID 911 and would have gone Ready then failed every import
# with EACCES. The tree is 0755 1000:3000, so uid 1000 is the only writer.
# - ln() across downloads/ -> tv/ returned link count 2 (same inode), so the arrs get
# real atomic moves instead of silently degrading to copy-then-delete on EXDEV.
# pfSense DNATs WITHOUT source NAT (the host logs prod's own node addresses as the peer),
# so the ZFS export ACL names 192.168.2.10/.11/.12 directly rather than one translated IP.
value: "192.168.2.102"
# ==========================================================================================
# OFF-CLUSTER DR TARGET — Cloudflare R2 (operator, 2026-09-05; bucket + token 2026-09-07).
# ==========================================================================================
@@ -1,187 +0,0 @@
---
date: 2026-10-05
branch: fix/vpn-server-key-in-sops
title: VPN server key moves into SOPS-encrypted git — the out-of-band Secret broke R-SEC-01 and the DR story
---
## What changed
**Operator correction (2026-10-05): "secrets are kept in source through sops".** They are right, and
#55 shipped the WireGuard server private key the wrong way: as a Secret the operator was told to
`kubectl create` out of band, referenced by name only. This branch moves it into the tree as
`platform/vpn-ingress/wg-server-key.sops.yaml`, where every other secret in this repo lives.
**Why the original choice was wrong, not merely unconventional.** Two concrete costs, both of which
#55's own README half-admitted and then shipped anyway:
1. **R-SEC-01 / spec docs/07 says secrets live in source, SOPS-encrypted, reaching the cluster only
through Flux.** An out-of-band Secret is outside the GitOps tree by construction.
2. **It broke DR for the one credential that cannot be re-derived.** A cluster rebuild restores
everything in git; it does not restore a hand-made Secret. Losing this key means **re-enrolling
every device**, because each client profile pins its public half. #55 papered over this with
"keep a copy in your password manager", which is a workaround for a self-inflicted gap.
Note `platform/kyverno/admission/secret-provenance.yaml` did **not** forbid the original shape —
it scopes to `app-namespace=true` namespaces and `vpn-ingress` is deliberately a platform namespace,
so admission would have accepted the hand-applied Secret. The rule it violated was the written
requirement, not an enforced control. Recording that because "the gate let it through" was never
evidence it was correct.
**How the unfillable-key problem is solved.** An agent still must not mint a long-lived private key
(CLAUDE.md §6), so the file is committed **already encrypted, with `privatekey: REPLACE-ME`**. That
makes the wiring complete and the gate green from the first commit — the resource line and both
overlay decryption patches are present — and leaves the operator exactly one act: replace the value.
The alternative, a commented-out resource line to uncomment later, is precisely the "file in git with
no generator entry: reviewed, merged, apparently done, still broken" shape this repo keeps getting
bitten by (`alert-runbooks-resolvable` exists because of it).
**The init-container guard is now a validation, not an emptiness check.** It was `[ -z "$key" ]`.
Because the committed placeholder is non-empty, the likely failure is now a value that is present
and *wrong*, so the guard pipes the value through `wg pubkey` — which accepts only a real 32-byte
curve25519 private key — and names which of four cases it hit:
| case | why it matters |
|---|---|
| `REPLACE-ME` placeholder | the expected fresh-install state |
| empty | blanked value |
| **starts with `ENC[`** | **the Secret reached the cluster UNDECRYPTED.** A missing overlay decryption patch applies cleanly and leaves every Flux condition green — this is the silent failure the guard exists for |
| rejected, none of the above | mangled value, or a *public* key pasted by mistake |
Files:
- `platform/vpn-ingress/wg-server-key.sops.yaml` (new) — SOPS-encrypted to the repo's age recipient,
`stringData.privatekey` only, placeholder value.
- `platform/vpn-ingress/wg-server-key.sops.yaml.example` (new) — plaintext template + operator
hand-off, following the `crafty-oauth2-proxy-credentials.sops.yaml.example` convention.
- `platform/vpn-ingress/kustomization.yaml` — the resource added; the "DELIBERATELY ABSENT: a
Secret" block replaced with the overlay-patch warning.
- `platform/vpn-ingress/daemonset.yaml` — the `wg pubkey` guard; volume comment rewritten.
- `clusters/dev/kustomization.yaml`, `clusters/prod/kustomization.yaml` — **the `vpn-ingress`
`decryption:` patch, in both.** This is the load-bearing half: #55's own kustomization comment
predicted that forgetting it applies ciphertext cleanly.
- `clusters/base/vpn-ingress.yaml` — its "NO decryption stanza is needed" comment was now false.
- `platform/vpn-ingress/README.md`, `runbooks/vpn-remote-access.md` — step 2 rewritten.
- `infra/vpn-ingress/new-peer.sh` — stale "step 1" pointer.
**A second finding: `wg` and `sops` were not in the same place — and the first fix for that was
also wrong.** Measured: `wg` existed only on the host (`/usr/bin/wg`), `sops` only in the dev
container, `qrencode` on neither. So `sops -d … | wg pubkey` could not run in one shell, and #55's
step 2 — which assumed it could — would have failed.
My first fix was to **split every step by where it runs**, and to tell the operator to
`dnf install qrencode` on their workstation. **Operator correction: "sop, wg, and any other tool
requirement is supposed to be baked into the devcontainer."** That is CLAUDE.md §5 and the
devcontainer README's "Don't install the toolchain on the host", and the split fix broke the
invariant rather than honouring it — it also produced a procedure that could never be reproduced in
CI, and which no one should have to follow.
**The real fix:** `wireguard-tools` and `qrencode` are now distro packages in
`infra/devcontainer/Containerfile`, alongside ansible/openssh/diffutils. Both are in Fedora 44's
repos (`wireguard-tools` 1.0.20260223 — the same upstream version as the container image this
component runs — and `qrencode` 4.1.1), unpinned like every other dnf package there, because the
pinned thing is the Fedora base and neither is a gate tool whose version changes what green means.
Three things got *simpler* rather than merely relocated, which is the tell that the operator's rule
was the right one:
1. **Step 2 is now one pipeline in one shell:** `sops set … "\"$(wg genkey)\""` generates the key
straight into the already-encrypted document. The plaintext never lands on disk, never reaches a
terminal, and there is no staged copy to shred. The previous version needed a copy in
`.cache/server.key`, a `git check-ignore` warning about the `*.key` suffix being the only thing
protecting it, and a sha256 comparison to prove the right bytes arrived. All of that is gone.
2. **`new-peer.sh` now derives the server's public key from `wg-server-key.sops.yaml` itself**
(`sops -d --extract … | wg pubkey`), so there is no local key copy to keep in step with the
encrypted one. It previously required the operator to keep the private key at a magic host path.
3. **`.cache/` is now gitignored at the repo root** (mirroring `infra/ansible/.gitignore`), so the
container has a persistent, non-committable scratch directory. `new-peer.sh` writes client
profiles to `.cache/wg-peers/` by default — necessary because the repo bind-mount is the
container's ONLY persistent read-write mount, so a profile written to `/tmp` in there is gone
before it can be moved to a desktop. Its old blanket "refuse to write inside the checkout" guard
is replaced by `git check-ignore`, which tests the actual property instead of a proxy.
**SENSITIVE PATHS — this PR touches `clusters/**` (two overlays) AND adds a `**/*.sops.yaml`, so
`ci/approval-policy.sh` will BLOCK it.** It needs the operator's `/approve-override <reason>` as a
PR **review** of type "Comment". I have deliberately not posted it (CLAUDE.md §1).
## What I verified
- **The encrypted file is real and correctly targeted.** `sops -e -i` in the pinned toolbox
produced `stringData.privatekey: ENC[AES256_GCM,...]` with cleartext keys, `encrypted_regex
^(data|stringData)$`, and exactly one age recipient:
`age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg` — the recipient in the root
`.sops.yaml`, i.e. the key Flux holds as `sops-age`.
- **Round-trip, without exposing anything**: `sops -d` then a Python read of key names only →
`stringData keys: ['privatekey']`, and `value is still the placeholder: True`.
- **Both overlay patches are present and name the right Kustomization** (`name: vpn-ingress`,
`namespace: flux-system`, `decryption.secretRef.name: sops-age`), inserted after each overlay's
`velero` stanza, mirroring headlamp/crafty/velero.
- **`kubectl kustomize platform/vpn-ingress` builds** and now emits five objects — Namespace,
ConfigMap, **Secret**, DaemonSet, NetworkPolicy — with `${WG_CLIENT_CIDR}` still an unexpanded
token for Flux.
- **The tool gap, measured on both host and container before the fix**: `wg` host-only, `sops`
container-only, `qrencode` on neither.
- **Both packages exist in the image's repos**, queried from inside the running container:
`wireguard-tools` 1.0.20260223 (repo `updates`) and `qrencode` 4.1.1 (repo `fedora`).
- **The image REBUILDS with them and both tools resolve inside it** —
`./infra/devcontainer/dev.sh build` succeeded, and `wg --version` / `qrencode --version` /
`sops --version` all answer in the rebuilt container. This is the claim the change rests on, so
it was executed rather than reasoned.
- **`new-peer.sh`'s new SOPS path fails correctly against the placeholder.** Run in the rebuilt
container it refused with "could not derive the server public key … it still holds the REPLACE-ME
placeholder", which is both the expected pre-key behaviour and a live test of the
`sops -d | wg pubkey` pipeline the whole design now depends on.
- **`.cache/` is ignored regardless of filename** now: `git check-ignore -v .cache/anything.conf`
and `.cache/serverkey.txt` both match `.gitignore:46:.cache/`. Before this change only `*.key`
matched, which is the fragility the previous runbook had to warn about.
- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
`lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
environmental skips (`gitea-settings` — no local token; conformance L2/L3 — no cluster).
**Not green. Reported as rc 3, not as pass.**
- **The L0 gate caught THREE real defects in this branch before it could merge — including one
in the fix for another — and I had asserted one of them was fine without running it.** Both are recorded because the gate's value here is the
whole point of the exercise:
1. **`no-plaintext-secrets` FAILED**: "`wg-server-key.sops.yaml.example`: an unencrypted .example
template is referenced by a kustomization — it would be applied as a plaintext Secret". My
kustomization named the template in a *comment*, and the check matches comments too (correctly:
naming one is how a plaintext Secret gets applied by accident). `platform/crafty/kustomization.yaml`
never names its own example either — the convention was there to copy and I missed it. Fixed by
not naming the file; the hand-off lives inside the example, where it belongs.
**An earlier draft of this entry claimed this check passed. It did not. The claim was written
from expectation rather than from a run.**
2. **`flux-postbuild-substitution` FAILED** on `${key#ENC[}` in my new guard
(`daemonset.yaml:155`), flagged `[SILENT-EMPTY]`: envsubst consumes every `${...}`, so my
shell prefix-strip would have rendered to the empty string — the ciphertext branch of the guard
would never have fired, which is precisely the silent failure the guard was added to catch.
Rewritten with `case`, which uses no braces and so has nothing to escape.
3. **And then it failed a THIRD time, on the fix's own comment** — `[FATAL-PARSE]
daemonset.yaml:156 ${...}`, the prose explaining the rule, written with the literal form. That
class is worse than it looks: a `FATAL-PARSE` hazard fails the **entire** Kustomization, every
resource in the path, not just the line. Rewritten as "dollar-brace expansion". This is the
second time in this component that writing the hazard literally inside a comment about the
hazard has been the defect (the first was in #55), so the habit to build is: **in any
substituted path, describe the construct in words, never type it.**
- **After both fixes, `no-plaintext-secrets` and `flux-postbuild-substitution` pass** (446 and 198
assertions respectively), and `l0` is green.
## What I did NOT verify
- **That Flux decrypts this file on prod.** The overlay patches have never reconciled. This is the
whole risk of the change and it has exactly one observable outcome: after merge the pods should
CrashLoop reporting the **`REPLACE-ME` placeholder** (proving decryption worked), *not* reporting
**`ENC[`** (which would prove a patch is wrong). I built the guard to tell those apart because I
cannot test them.
- **That `wg pubkey` accepts the operator's real key through this path.** The guard is reasoned from
`wg pubkey`'s behaviour and has never run — `wg` is absent from both this workstation's container
and the gate, so the first execution will be in the pod.
- **The `sops set` command in the hand-off.** I verified `sops -e -i` and `sops -d` in the toolbox;
I did **not** run `sops set` with the quoting shown, and nested quoting inside
`dev.sh run bash -c` is exactly where a documented command tends to break. If it misbehaves, the
editor form (`sops <file>`) is the fallback and does not depend on quoting.
- **The sha256 comparison in step 2 step 4** is untested end to end (it needs the real key).
- **Whether anything else in #55 assumed a tool is available where it is not.** I checked `wg`,
`sops` and `qrencode` because step 2 depended on them. I did not audit every command in the
runbook the same way — and that class of defect has now produced two corrections in a row
(`site.sh`, then `sops | wg`), so assume more remain until each step has actually been run.
- **The live VPN still carries no traffic**, and the earlier gaps stand unchanged: the
iptables-vs-nftables question, `allowPrivilegeEscalation: false` versus the image's s6 init, and
any off-LAN reachability.
@@ -1,73 +0,0 @@
---
date: 2026-10-06
branch: fix/new-peer-ignores-template-comments
title: new-peer.sh refused the first VPN address because its duplicate check matched its own commented-out example
---
## What changed
The operator's first real enrolment failed:
```
$ ./infra/vpn-ingress/new-peer.sh keith-phone 2
new-peer.sh: 10.90.0.2/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet.
```
**Nothing was allocated.** `peers.yaml` had zero peers — the pod's own init container reported
`wg0.conf assembled: 0 peer(s)`. The duplicate-address check grepped the WHOLE file, and
`peers.yaml` line 66 is a **commented-out example stanza** I wrote, containing literally
`# AllowedIPs = 10.90.0.2/32`. So `.2` — the first address anybody would ask for, and the one my
own runbook uses in its example — was permanently un-allocatable, with a message asserting the
opposite of the truth.
Both halves fixed, because either alone leaves the trap half-set:
- **`infra/vpn-ingress/new-peer.sh`**: the check strips comment lines before matching
(`grep -vE '^[[:space:]]*#' | grep -q …`). That is the correct predicate rather than a patch: a
real stanza's `AllowedIPs` line is never commented, while both the template and every peer's
`# <name> — <owner>` label always are.
- **`platform/vpn-ingress/peers.yaml`**: the example now uses placeholders that cannot match
anything real — `# <device-name> — <owner>, enrolled <YYYY-MM-DD>` and
`AllowedIPs = 10.90.0.<octet>/32`. A future grep cannot be fooled by it either.
The bug class is worth naming: **a documentation example that is byte-identical to real data will
eventually be parsed as real data.** The same shape caused two earlier defects in this component —
`flux-postbuild-substitution` flagged a `${...}` written inside a comment *explaining* that hazard,
and `no-plaintext-secrets` flagged the `.sops.yaml.example` because the kustomization named it in a
comment. Three instances now; the rule is to make examples structurally unmatchable, not merely
commented out.
No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `decisions/**`), so this
merges on a green gate with no override.
## What I verified
Run in the pinned toolbox against the live tree, 2026-10-06.
- **`.2` now works**, and it went through the REAL key path: `new-peer.sh keith-phone 2` rendered a
profile with `Address 10.90.0.2/32`, `AllowedIPs = 10.90.0.2/32` in the printed stanza, full
tunnel, at `.cache/wg-peers/keith-phone.conf` (0600). The throwaway keypair was deleted after.
- **This incidentally closes a gap left open in #60**: the profile could only have been rendered by
deriving the server's public key via `sops -d --extract … | wg pubkey` from
`wg-server-key.sops.yaml` — so that pipeline is now proven **against the operator's real key**,
not just against a placeholder it was expected to reject.
- **A real duplicate is still refused.** Appending an uncommented
`AllowedIPs = 10.90.0.5/32` stanza and asking for octet 5 produced the correct refusal; the tree
was restored afterwards and `git diff --stat` showed only the intended template edit.
- **Zero active allocations remain**: `grep -vE '^[[:space:]]*#' peers.yaml | grep -c AllowedIPs`
→ `0`.
- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
`l1` and `approval-policy` all pass; rc 3 is the two environmental skips (`gitea-settings`,
conformance L2/L3). **Not green. Reported as rc 3.**
## What I did NOT verify
- **No peer is enrolled by this branch.** The keypair I generated was a throwaway and was deleted;
the operator must run the script themselves, since the device's private key must exist only on
the device. So the end of the enrolment path — a stanza merged into `peers.yaml`, Flux rolling
the DaemonSet, and `wg show` listing a peer — is still unexercised.
- **The duplicate-NAME warning** (`grep -qi "^ *# *${NAME} "`) deliberately still inspects
comments, because per-peer labels are comments. It is no longer tripped by the template, but I
did not test it against a real enrolled peer.
- **Nothing about client traffic.** The WAN port-forward is still not configured and there is still
no off-LAN vantage point, so the tunnel has carried no packet.
@@ -1,103 +0,0 @@
---
date: 2026-10-06
branch: fix/peers-configmap-content-hash
title: A merged VPN peer never reached the tunnel — the peer register was a fixed-name ConfigMap, so nothing rolled the pods
---
## What changed
The operator enrolled a phone, merged the peer, and **`wg show` listed no peers.** Measured on prod
2026-10-06, with the peer already on `main`:
| observation | value |
|---|---|
| peer stanza on `main` | present (`AllowedIPs = 10.90.0.2/32`) |
| GitRepository artifact | `main@sha1:e1bebb8` — Flux had fetched it |
| `vpn-ingress` Kustomization `lastAppliedRevision` | `main@sha1:5928d81` — **behind**, 10m interval |
| live `wg-peers` ConfigMap, active `AllowedIPs` lines | **0** |
| DaemonSet volume reference | `configMap.name: wg-peers` — a **fixed** name |
| DaemonSet `generation` / `observedGeneration` | **1 / 1** — it had **never rolled** |
| init container, all three pods | `wg0.conf assembled: 0 peer(s)` |
The lag was only the reconcile interval. **The defect is the fixed name.** When Flux does update a
fixed-name ConfigMap it updates it *in place*: the DaemonSet's pod template is byte-identical, so
no rollout is triggered — and because the init container assembles `wg0.conf` **once at pod start**
into a tmpfs, the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer
would have had **no effect at all** until something unrelated happened to restart the pods. For a
*revocation* that is a security defect, not an inconvenience: the operator would delete a stanza,
see a green PR, and the device would keep working.
My runbook and README both asserted "PR → merge → Flux rolls the DaemonSet (~1 min)". That was
never true.
**This repo already documents the fix, and I had read the file.** `runbooks/kustomization.yaml`:
*"The generated name carries kustomize's content hash, so editing any runbook rolls the server
automatically and a stale projection is impossible."*
So the register becomes a **content-hashed generator**:
- `platform/vpn-ingress/peers.yaml` (inline ConfigMap) → **`platform/vpn-ingress/peers.conf`**, a
plain file, plus a `configMapGenerator` in `kustomization.yaml`. The operator's enrolled peer was
carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the
documentation now travels *inside the mounted file* instead of in a YAML wrapper that gets
stripped.
- **`namespace: vpn-ingress` added to the kustomization — REQUIRED, not tidiness.** A generated
ConfigMap inherits no namespace; every other object here declares `vpn-ingress`; and kustomize's
`nameReference` transformer will not rewrite a reference across a namespace boundary. Without it
the build emitted `ConfigMap wg-peers-m9kcckcb5t` while the DaemonSet still mounted the unhashed
`wg-peers`, which would not exist — the pods would wedge on a missing volume. I caught this only
by building the output and reading the volumes block.
- **`infra/vpn-ingress/new-peer.sh`**: `PEERS_FILE` repointed to `peers.conf`, and a missing
register is now **fatal** instead of skipped. The old guard was
`[ -r "$PEERS_FILE" ] && grep …`, so after the rename the only symptom would have been
duplicate-address detection quietly never firing again — the silent-skip shape this repo treats
as a defect in its own right.
- Stale `peers.yaml` references swept from the README, runbook, `daemonset.yaml`,
`networkpolicy.yaml`, the SOPS example, and the script's own **user-facing output** (it told the
operator to paste into "peers.yaml, under data.peers.conf", which would no longer exist).
- The runbook gained the verification that actually distinguishes this failure — the DaemonSet's
`generation` must *increase* after the merge — and a troubleshooting row for `0 peer(s)` after a
merge.
Flux's `prune: true` removes the orphaned fixed-name `wg-peers` ConfigMap on the next reconcile.
No sensitive paths (`platform/vpn-ingress/**`, `infra/vpn-ingress/**`, `runbooks/**`,
`decisions/**`), so this merges on a green gate with no override.
## What I verified
Built and run in the pinned toolbox, 2026-10-06 (kustomize v5.8.1 via `kubectl kustomize`).
- **The reference is now rewritten end to end**: generated `ConfigMap wg-peers-m9kcckcb5t` in
namespace `vpn-ingress`, and the DaemonSet mounts **`wg-peers-m9kcckcb5t`**. Before adding
`namespace:` the same build mounted the unhashed `wg-peers` — so the namespace line is load
bearing, measured both ways.
- **The hash tracks content, which is the whole mechanism.** `wg-peers-m9kcckcb5t` →
append a second peer → `wg-peers-dgg7ht9c74` → restore → `wg-peers-m9kcckcb5t`. Deterministic
and reversible, so an edit necessarily changes the pod template.
- **The operator's enrolled peer survived the file conversion byte-for-byte**: the active stanza in
`peers.conf` (`PublicKey = pBRSHrRy…`, `AllowedIPs = 10.90.0.2/32`) is identical to the one on
`main`, compared directly against `git show origin/main:platform/vpn-ingress/peers.yaml`.
- **The build still emits all five objects**: Namespace, ConfigMap, Secret, DaemonSet,
NetworkPolicy — and the `Namespace` object's own name is still `vpn-ingress` (the namespace
transformer did not rewrite it).
- **`new-peer.sh` works against the new register**: `spare-laptop 7` renders a profile and prints
"append this to platform/vpn-ingress/peers.conf"; `dupe 2` is **refused** because the operator's
real `.2` now occupies it — the first time the duplicate check has been exercised against real
data rather than against its own example.
- **Toolbox gate**: rc 3, with `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`,
`l1` and `approval-policy` all pass; rc 3 is the two environmental skips. **Not green. Reported
as rc 3.**
## What I did NOT verify
- **That the roll actually happens on the cluster.** The whole point of the change is unproven
until Flux applies it: the success criterion is the DaemonSet's `generation` increasing and
`wg0.conf assembled: 1 peer(s)` in the init log, followed by `wg show` listing the peer.
- **That the tunnel then carries traffic.** No handshake has ever been observed. The WAN
port-forward is reportedly configured but has never been exercised by a packet I can see, and I
have no off-LAN vantage point — runbook step 6 from cellular remains the only honest test.
- **Revocation**, which is the case where this defect was a security problem. Deleting a stanza
should now roll the pods and drop the peer; untested.
- **Whether the orphaned fixed-name `wg-peers` ConfigMap is actually pruned.** Reasoned from
`prune: true` on the Kustomization, not observed.
@@ -0,0 +1,88 @@
---
date: 2026-10-06
branch: feat/pm-agent-registry-pull-secret
title: The kubelet's pull credential — the second half of making agent-runner pullable, and the seventh field the admission policy had not read
---
## What changed
The registry had two blockers. The first — the nodes resolving the registry hostname to the
non-hairpinned WAN and failing TLS against the router's `CN=%D` certificate — was diagnosed
and fixed by an `/etc/hosts` pin, merged in #59 and applied by the operator. This is the
second: **the pull is unauthenticated, and this registry refuses unauthenticated clients
outright.**
`pm-agent-registry` is a new `kubernetes.io/dockerconfigjson` Secret, SOPS-encrypted with
placeholder values, referenced by `imagePullSecrets` on both the dispatcher Deployment and
the committed worker Job template. It is a **second Secret holding the same Gitea token** as
`pm-agent-credentials`, because the two consumers need two formats: the application reads a
raw token from env, the kubelet reads a dockerconfigjson before any container starts. That
duplication is the established pattern here (the Keycloak client secret is mirrored the same
way, for the same reason) and it is documented in both `.example` files.
The operator's part is the value, and one scope: the Gitea token must now carry
**`read:package`** alongside `write:repository` and `write:issue`. Steps are in
`platform/pm-agent/pm-agent-registry.sops.yaml.example`, including the trap that
`.dockerconfigjson`'s registry key must be the bare host as the image reference spells it —
with a scheme or a port it matches nothing, the kubelet pulls anonymously, and the failure is
indistinguishable from the Secret being absent.
**The admission policy gained a seventh rule.** `imagePullSecrets` is a pod-spec field that
no container references, which is exactly why it is easy to leave unread — the same shape of
omission as `initContainers`, `ephemeralContainers`, `env` and `envFrom` before it. The
kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now
REQUIRES the committed entry and rule 7 forbids any other or any second one; both halves are
needed because a `foreach` over an absent list yields no result.
The leverage is small and worth saying so: the image is digest-pinned, so a different pull
credential cannot change *what* runs, and the only Secrets in this namespace are the agent's
own. It is pinned anyway, because "only the committed shape" is this policy's whole premise
and the rule costs four lines.
## What I verified
- **`kyverno test platform/kyverno/tests/pm-agent` → 33 passed, 0 failed** under the pinned
CLI (was 28 before this change). Three new fixtures, each a real path rather than a style
violation: `extra-pull-secret` (the committed entry PLUS a second one — rule 1's array
pattern is satisfied by the first element, which is why the count is asserted separately),
`wrong-pull-secret` (a different Secret in this namespace), and `no-pull-secret` (absent
entirely, refused by rule 1's pattern because rule 7's foreach cannot see it).
- **`PM_SELFTEST=1 pm-dispatch` → exit 0**, now also asserting that the committed Job
template names exactly `pm-agent-registry` — so the template and the policy cannot drift
apart without the image build failing.
- **`./ci/gate.sh` in the pinned toolbox, from a §0 worktree → exit 3 (UNDETERMINED)**, with
`gitea-settings` the only stage that did not run (no `SETTINGS_API_TOKEN` on this
workstation). Every other stage green, `spec-immutability` included. rc 3 is not a pass.
- **The gate caught me shipping the Secret in plaintext.** I wrote the placeholder
`pm-agent-registry.sops.yaml` and never ran `sops -e -i` on it. `no-plaintext-secrets`
refused it with both halves named — "has no top-level `sops:` block with a mac" and
"Secret value '.dockerconfigjson' is not SOPS ciphertext" — which is exactly the check
earning its place: the file looked finished, and the value was a placeholder, so nothing
about reading the diff would have made it obvious. Encrypted, and the stage is green.
- **The registry really does refuse anonymous clients**, measured before building this
(2026-10-05): the anonymous registry token's `access` claim is `null`, every `/v2/…`
path answers 401 whether it exists or not, and `/api/v1/packages/Home-Network` replies
`{"message":"token is required"}`. That is why this is a credential and not a posture
tweak.
## What I did NOT verify
- **That the pull now works.** Nothing has been pushed to that registry and the committed
digest is still the all-zero placeholder, so neither half of the registry fix has been
exercised end to end. The first real pull is what proves both at once — and if it fails, the
error names which half: `x509 … not valid for any names` is the resolution pin, `401` /
`unauthorized` is this credential.
- **That the `/etc/hosts` pin from #59 took effect.** The operator applied it; I could not
confirm the effect, because the dispatcher pod's retained events are all repeating
`BackOff` with no fresh `Pulling`, so the `x509` message still visible is the one from
before the play ran. Not evidence either way.
- **Whether `read:package` is sufficient** for a pull from a Gitea package of this type. It is
the documented read scope for packages; nothing here has exercised it, because no package
exists yet.
- **The dockerconfigjson placeholder is not a valid credential and is not meant to be.** Its
`auth` field is a `REPLACE-ME` string rather than base64, so a pull with it will fail at
authentication rather than be mistaken for working.
- **Anything about runtime.** Unchanged from the previous entries: the image has never been
built or pushed, no dispatcher has authenticated to Gitea, no issue has been triaged, no
worker Job has ever been created, and the admission policy — live, `Enforce`, now seven
rules — has still never admitted or refused a real Job.
@@ -1,113 +0,0 @@
---
date: 2026-10-06
branch: fix/vpn-ingress-ip-forward
title: The VPN tunnel never came up — a hostNetwork pod cannot write net.ipv4.ip_forward, and wg-quick treats a failed PostUp as fatal
---
## What changed
The remote-access VPN reconciled for the first time with a real server key (#60 then #62) and
**the tunnel did not come up**. One line in my own manifest was the whole cause.
Measured in the `wireguard` container on prod, 2026-10-06:
```
[#] ip link add dev wg0 type wireguard
[#] wg addconf wg0 /dev/fd/63
[#] ip -4 address add 10.90.0.1/24 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] sysctl -q -w net.ipv4.ip_forward=1
sysctl: permission denied on key "net.ipv4.ip_forward"
[#] ip link delete dev wg0
**** Tunnel /config/wg_confs/wg0.conf failed, will stop all others! ****
```
Everything up to that point worked. **The cause is not a missing capability.** With
`hostNetwork: true` the runtime mounts `/proc/sys` **read-only** in the container — confirmed in the
pod's own `/proc/mounts`: `proc /proc/sys proc ro,nosuid,nodev,noexec,relatime` — precisely because
those sysctls are the *host's* globals and a container must not rewrite them. `CAP_NET_ADMIN` cannot
lift a read-only mount, so this can never succeed from here. And `wg-quick` treats any failed
`PostUp` as fatal, so it deleted the interface it had just finished building.
**The line was never needed.** `node_baseline` writes `net.ipv4.ip_forward=1` to
`/etc/sysctl.d/99-k3s-cilium.conf`, and reading the value from inside the pod returns `1`. I had
written the header comment *correctly* — "ip_forward is ALREADY 1 on these nodes … so no pod sysctl
is needed" — and then emitted the sysctl anyway, justified in its own comment as "re-asserted so
this component does not silently depend on another role's sysctl file surviving". That defensive
re-assertion is the entire outage. **The lesson is narrow and reusable: in a hostNetwork pod,
verify host state, never write it.**
Fix:
- **`platform/vpn-ingress/daemonset.yaml`**: the `PostUp = sysctl …` line is gone, replaced by a
**read** in the init container — `grep -qx 1 /proc/sys/net/ipv4/ip_forward` — which fails closed
with a message naming the `node_baseline` role. Reading `/proc/sys` is permitted; only writing is
not. Fail-closed still matters: a tunnel that comes up but cannot forward would satisfy every
probe in this component and silently drop every client's traffic.
- The header bullet that said "no pod sysctl is needed" now also says **do not attempt one**, with
the mechanism, so the next author does not re-add it for the same reason I did.
- The `securityContext` comment claimed NET_ADMIN was for "create the wg0 link, set the host sysctl,
and write the netfilter rules". The middle clause was false; corrected.
- The image's `chown: … Operation not permitted` startup run — plus its own "Permissions could not
be set … we will not provide support for it" — is now documented as **expected and harmless**, in
both the manifest and the runbook, so nobody chases it. Capabilities are dropped to `NET_ADMIN`
only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs as root,
and `wg-quick` proceeds past it (as it did here — the failure was 30 lines later).
- **`runbooks/vpn-remote-access.md`**: a "what a healthy start looks like" block with the real log
lines; `ip_forward is 0` added to the init-failure table; and the note that `0/1 Running` with
`Startup probe failed: Unable to access interface: No such device` is the signature of `wg-quick`
tearing the tunnel down — read the `[#]` line it died on.
- **`platform/vpn-ingress/README.md`**: a table of what the first reconcile settled (below).
No sensitive paths: `platform/vpn-ingress/**`, `runbooks/**` and `decisions/**` only. This merges on
a green gate with no override.
## What I verified
Live prod, read-only, 2026-10-06. **Four questions this component's design rested on are now
measured rather than assumed — three went the right way, one was the defect above.**
- **The GitOps path is proven end to end.** `Kustomization vpn-ingress` `lastAppliedRevision:
main@sha1:cf7d5e4` equals the GitRepository artifact revision. The Secret carries
`kustomize.toolkit.fluxcd.io/name: vpn-ingress`, so Flux delivered it.
- **SOPS decryption works, proven by digest rather than by eye.** The value in
`platform/vpn-ingress/wg-server-key.sops.yaml` on `main`, decrypted, is 44 bytes, is **not** the
placeholder, and **`wg pubkey` accepts it**. Its SHA-256 equals the SHA-256 of the live Secret's
decoded `data.privatekey` (`d7c8f8de6f252649…` both sides). No key material was printed at any
point — lengths, booleans and digests only.
- **The `wg pubkey` guard fired correctly, and its `ENC[` branch was never needed.** During the
kubelet window before the updated Secret propagated, all three pods reported *"It is still the
committed REPLACE-ME PLACEHOLDER"* — the guard distinguishing a stale/unfilled value from
ciphertext, exactly as designed. It cleared on its own once the mount refreshed; **a transient
`REPLACE-ME` CrashLoop for a minute or two after the key PR merges is expected, not a fault.**
- **The kernel module reaches the pod**: `ip link add dev wg0 type wireguard` succeeded and
`/proc/modules` lists `wireguard`. So the `node_baseline` addition works and `CAP_SYS_MODULE` is
correctly not required.
- **`allowPrivilegeEscalation: false` does NOT break the image's s6 init** — the container reaches
`wg-quick` normally. The flag stays; the fear recorded in #55 is closed.
- **The iptables backends MATCH, which was my single biggest documented doubt.** The image ships
`iptables v1.8.13 (nf_tables)` via `/usr/sbin/xtables-nft-multi`, and Debian 13 is nf_tables. The
nightmare case — rules accepted into a table the kernel never consults, so handshakes succeed and
every forwarded connection hangs — cannot occur on this pairing.
- **`net.ipv4.ip_forward` is `1`** in the host netns the pod shares, and `/proc/sys` is `ro` there.
Both read directly from the pod.
- **Toolbox gate** from an isolated worktree: **rc 3**, with `spec-immutability`, `decisions-log`,
`lint`, `l0`, `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two
environmental skips (`gitea-settings`, conformance L2/L3). **Not green. Reported as rc 3.**
## What I did NOT verify
- **That the tunnel now comes up.** This fix has not reconciled. The next reconcile is the test, and
its success criterion is specific: pods `1/1 Running`, and `wg show` reporting `interface: wg0`
with `listening port: 51820`.
- **That the MASQUERADE and FORWARD rules actually apply.** They are the next statements after the
one that failed, so **they have never executed**. The backends matching (above) removes the worst
failure mode but proves nothing about the rules themselves. First check after the pods go Ready:
`iptables -t nat -S POSTROUTING | grep 10.90.0.0/24` on a node.
- **Anything about client traffic.** Still zero peers enrolled, still no WAN port-forward, still no
off-LAN vantage point. The VPN has carried no packet.
- **Whether the transient `REPLACE-ME` window has an upper bound.** It cleared within ~2 minutes
here; I did not determine the kubelet refresh behaviour, so a longer stall after a key change is
uncharacterised.
- **The `sops set` hand-off command**, still. The operator filled the key in by their own route
(#62), so the exact quoting I documented inside `dev.sh run bash -c` remains unexercised.
-85
View File
@@ -1,85 +0,0 @@
---
date: 2026-10-07
branch: fix/bulk-nfs-truenas
title: Repoint bulk NFS at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
---
## What changed
Both ZFS pools were physically moved off the Proxmox host onto a standalone TrueNAS Community
Edition at **192.168.2.30**, which sits on prod's own `192.168.2.0/24`. `BULK_NFS_SERVER` moves from
`192.168.2.102` (the pfSense DNAT prod used to mount through) to `192.168.2.30`.
- `clusters/prod/kustomization.yaml` — `BULK_NFS_SERVER` → `192.168.2.30`. The comment block that
justified the DNAT, and the "NOT YET DECIDED / THIS PLACEHOLDER IS LOAD-BEARING" banner, are
replaced with the measured export configuration and the recreate requirement below.
- `clusters/base/cluster-vars.yaml` — base default `192.168.1.9` → `192.168.2.30`. That host no
longer exists.
- `clusters/dev/kustomization.yaml` — same; marked explicitly **inert**, since dev is decommissioned
and its nodes are gone. It carries the real address only so nothing in the tree renders a dead one.
- `docs/BULK-NFS-STORAGE.md` — §1 and §5 rewritten for TrueNAS (exports, service settings, the ZFS
`sharenfs` properties that had to be cleared, and the ACL hazard); §7 rewritten now that the
placeholder is gone; §9 retargeted from dev to prod; **new §10** documents the PV
delete-and-recreate. §2/§3/§4/§6 are unchanged by the move.
- `catalog/qbittorrent.yaml`, `platform/monitoring/node-pressure-dashboard.yaml` — two comment/prose
references to `192.168.2.102` as the live export address.
**The headline operational fact: repointing this knob is NOT a plain reconcile.** A bound PV's NFS
source is immutable, so Flux's apply fails rather than updating and the pods keep their old, dead
mount. The five PVs must be deleted and recreated. That sequence is §10 and is an operator action;
it is **not** performed by this PR.
## What I verified
**On the NAS (192.168.2.30), measured 2026-10-07 by the operator at my request:**
- `zfs list` — `Storage` → `/mnt/Storage`, `Storage/plex` → `/mnt/Storage/plex`,
`Storage/nextcloud` → `/mnt/Storage/nextcloud`. The PV paths resolve unchanged.
- Ownership intact: `/mnt/Storage/plex`, `tv/`, `downloads/`, `movies/`,
`downloads/complete/`, `nextcloud/nextcloud/` all `drwxr-xr-x 1000:3000`. `movies/` 1171 entries
and `tv/` 107, against 1165 / 107 measured on Proxmox.
- `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`), `aclinherit=passthrough`, and no
non-trivial ACLs — mode bits still govern.
- Three stale `sharenfs` dataset properties (`plex`, `nextcloud`, `docker-volumes`) carried over from
Proxmox; cleared with `zfs inherit`, re-checked empty.
- `exportfs -v`: both shares to `192.168.2.10/.11/.12` with `sec=sys,secure,root_squash,no_all_squash`,
`rw` on plex and `ro` on nextcloud.
- `/proc/fs/nfsd/versions` → `+3 +4 +4.1 +4.2`, so the PVs' `nfsvers=4.1` negotiates.
- **A defect was found and fixed during setup:** the plex share listed `192.168.2.13` — an address no
node holds — and so omitted `192.168.2.12` (k3s-prod-3). Corrected and re-verified.
**From the cluster:**
- Node InternalIPs are `192.168.2.10/.11/.12` (`kubectl get nodes`), confirming `.13` was wrong.
- All five NFS PVs currently name `192.168.2.102`.
- `192.168.2.30:2049` and `:111` accept connections; `192.168.2.102:2049` does not.
- **`spec.nfs` immutability, by server-side dry-run** — this is the finding that shapes the change:
`spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation`.
- The pods report `Ready` while their mounts are dead: a `stat /media` answers from attribute cache,
but a `readdir` blocks indefinitely (the PVs mount `hard`). Hence "scale to zero first" in §10.
**Gate:** see the PR; run in the pinned toolbox from this worktree.
## What I did NOT verify
- **Nothing has mounted 192.168.2.30.** No pod has touched the new server — the PVs still point at
the old address and cannot be updated in place. Every §9 probe (numeric-id passthrough, the uid-1000
write, the hardlink/EXDEV check, Plex's read-only posture) is **outstanding**, not passing. The
equivalents passed against the old host on 2026-09-12; that is not evidence about this one.
- **The §10 sequence is reasoned, not executed.** Retain + the renderer's pre-set `claimRef` are read
from the manifests and from `catalog/render.py`; I have not watched a PVC rebind after a PV delete
in this cluster.
- **`root_squash` is a behaviour change from the Proxmox export**, which carried `no_root_squash`. The
argument that nothing needs root on the tree (LSIO drops to 1000; the images chown only their
Longhorn `/config`; both subPath directories already exist at 0755) is reasoned from the images and
the measured permissions. It has not been exercised. If a mount or subPath resolution fails with
EACCES, Maproot `root:root` reproduces the old export and is the first thing to try.
- **Whether NFSv3 can be disabled.** Enabled Protocols is `NFSv3, NFSv4`; the PVs only need v4.1, so
narrowing would close the rpcbind/mountd/statd surface. Not done — it is a change to make after the
cutover is proven, not during it.
- **Whether pfSense / the legacy stack is actually retired.** I measured only that
`192.168.2.102:2049` stopped answering. The other legacy consumers of these datasets
(`192.168.1.6`, `10.123.1.10`, `192.168.1.18`) were dropped from the new exports; if any is still
live, it has lost access. Not checked.
- **L2/L3 conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline subset
only. Nothing in the gate exercises an NFS mount.
+116 -222
View File
@@ -1,43 +1,61 @@
# Bulk NFS storage — the media library and the legacy Nextcloud archive
The operator-facing companion to the `class: bulk` storage entries in
`catalog/{plex,sonarr,radarr,qbittorrent,nextcloud}.yaml`.
**Rewritten 2026-10-07 for the move off Proxmox onto a standalone TrueNAS at 192.168.2.30.** The
layout (§2), the mount design (§3, §4) and the uid model (§6) are unchanged by the move and were
measured on the old host on 2026-09-07. Everything about the storage host itself — §1, §5, §10 —
was re-measured on the NAS on 2026-10-07. Nothing here is assumed. It resolves the long-standing
**OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
The operator-facing companion to the `class: bulk` storage entries in `catalog/{plex,sonarr,radarr,nextcloud}.yaml`.
Everything here was **measured** on the storage host on 2026-09-07, over read-only SSH; nothing is
assumed. It resolves the long-standing **OQ-2** stub (`platform/longhorn/storageclasses.yaml`) for the
case that actually exists here — attaching datasets that already hold data — and leaves dynamic bulk
provisioning unimplemented, because nothing needs it.
---
## 1. The storage host
## 1. The storage host, and why prod cannot reach it
The bulk data lives on a **standalone TrueNAS** (Community Edition) at **192.168.2.30** — the same
`192.168.2.0/24` as prod's nodes (`192.168.2.10/.11/.12`), so it is reached directly, with no NAT
and no routing decision.
The bulk data lives on the **Proxmox host's ZFS pool**. That host has exactly one interface:
**This section used to be the hard part of this page.** The datasets were on the Proxmox host, whose
only interface was `vmbr0 192.168.1.9/24` on pfSense's **LAN**, while prod sits on pfSense's **WAN**.
tcp/2049 did not traverse that, so prod mounted through a DNAT at `192.168.2.102`, and which address
belonged in `BULK_NFS_SERVER` depended on an undecided operator choice. Both pools were physically
moved on 2026-10-07 (`docs/plans/proxmox-to-n5-air-migration.md`, Phase C/D) and all of that is
simply gone. Measured 2026-10-07: **192.168.2.102:2049 no longer answers; 192.168.2.30:2049 and
:111 do.**
```
vmbr0 192.168.1.9/24 default via 192.168.1.1 (pfSense LAN)
```
| | nodes | reaches 192.168.1.9? |
|---|---|---|
| **dev** | 192.168.1.40 / .41 / .42 | **Yes** — same L2 segment, and it already holds an NFS export from this host (`/mnt/Storage/k8s-volumes`, in `/etc/exports`) |
| **prod** | 192.168.2.10 / .11 / .12 | **No** — 192.168.2.0/24 is pfSense's **WAN**. Ports 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. 2049 and 111 do not. |
Opening the storage path for prod is a **pending operator decision**. Its two candidate outcomes give
different values for the `BULK_NFS_SERVER` cluster-var:
| outcome | `BULK_NFS_SERVER` for prod |
|---|---|
| port-forward `192.168.2.102:2049` → `192.168.1.9:2049` | `192.168.2.102` |
| a WAN→LAN route to 192.168.1.0/24 | `192.168.1.9` |
Until it lands, `clusters/prod` holds `TBD-PROD-BULK-NFS-SERVER`. **See §7 for why that placeholder
behaves differently from the repo's other ones.**
### It only needs ONE port
The PVs mount `nfsvers=4.1`. **NFSv4 carries mount and lock in-protocol**, so the only thing that
must be reachable is **tcp/2049** — no rpcbind/111, no mountd, no statd. NFSv3 would have needed
four ports, three of them on ephemeral numbers. The NAS advertises `+3 +4 +4.1 +4.2`
(`/proc/fs/nfsd/versions`, measured 2026-10-07), so 4.1 is available and is the conservative floor.
four ports, three of them on ephemeral numbers, which is a materially worse firewall story. The
server advertises `+3 +4 +4.1 +4.2` (`/proc/fs/nfsd/versions`), so v4.1 is available and is the
conservative floor.
**NFSv4 is not on by default and its absence does not look like a storage problem.** TrueNAS ships
NFSv3-only until NFSv4 is added to *Services → NFS → Enabled Protocols*. Without it the mount never
completes and the pods sit in `ContainerCreating` — no permission error, nothing in the app logs,
just no volume.
### The export's client list is the SAME under both outcomes — measured, not assumed
**pfSense DNATs the existing port-forward WITHOUT source NAT.** Connecting from this workstation
(192.168.2.132) to `192.168.2.102:22`, the storage host logged the peer as `192.168.2.132`:
```
SSH_CONNECTION=192.168.2.132 39768 192.168.1.9 22
ss -tn -> 192.168.1.9:22 192.168.2.132:39768
```
The original WAN-side source address survives the translation (return traffic finds its way back via
the host's default route, 192.168.1.1). So under **either** outcome the NFS server will see prod's
**real** node addresses — 192.168.2.10/.11/.12 — and the `sharenfs` change in §5 is the same either
way. **It does not have to wait for the routing decision.**
---
## 2. The layout, as found
@@ -89,18 +107,16 @@ why it is mounted read-only; see §6.
| plex | `/mnt/Storage/plex` | `/media` | — | **ro** | ReadOnlyMany |
| sonarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
| radarr | `/mnt/Storage/plex` | `/media` | — | rw | ReadWriteMany |
| qbittorrent | `/mnt/Storage/plex` | `/downloads` | `downloads` | rw | ReadWriteMany |
| nextcloud | `/mnt/Storage/nextcloud` | `/mnt/legacy-nextcloud` | `nextcloud` | **ro** | ReadOnlyMany |
In-app paths that follow: Plex libraries at `/media/movies`, `/media/tv`, `/media/photos`; Sonarr
Root Folder `/media/tv`; Radarr Root Folder `/media/movies`; both importing from
`/media/downloads/complete/{tv-sonarr,radarr}`.
**The three library consumers use the identical mountPath on purpose**, and qbittorrent has since
been given the same export (narrowed by `subPath: downloads`, presented at `/downloads` to match its
existing `Session\DefaultSavePath`). The arrs and the download client therefore agree on absolute
paths, which is the second precondition for hardlinked imports — otherwise every import needs a
Remote Path Mapping and quietly falls back to copying when one is missing.
**All three media consumers use the identical mountPath on purpose.** When qbittorrent is eventually
given the same export (it is not in this change), the arrs and the download client will already agree
on absolute paths, which is the second precondition for hardlinked imports — otherwise every import
needs a Remote Path Mapping and quietly falls back to copying when one is missing.
**`/media` and not `/data`, and that is not cosmetic.** The arrs' Longhorn config PVC renders at the
renderer default `/data/config`. A media mount at `/data` would be mounted **over** it and mask the
@@ -131,123 +147,62 @@ Narrowing that would cost hardlinks, which is the worse trade.
---
## 5. The exports, as configured on TrueNAS — and the settings that decide permissions
## 5. The export change the operator must apply — STRICTLY ADDITIVE
**Do not apply this from the repo.** It is operator configuration on the NAS, recorded here so the
exact values are reviewable. Everything below was **measured on 192.168.2.30 on 2026-10-07**, after
the move.
**Do not apply this from the repo.** It is an operator step on the storage host, recorded here so the
exact value is reviewable. The existing consumers **must keep working**: 192.168.1.6, 10.123.1.10 and
192.168.1.18 on the media tree, and 192.168.1.6 and 192.168.1.5 on the Nextcloud dataset, are the
live non-k3s prod stack.
### The exports
Current values (`zfs get sharenfs Storage`):
```
/mnt/Storage/plex 192.168.2.10 192.168.2.11 192.168.2.12
(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)
/mnt/Storage/nextcloud 192.168.2.10 192.168.2.11 192.168.2.12
(sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,root_squash,no_all_squash)
Storage/plex rw=@192.168.1.6:@10.123.1.10:@192.168.1.18,no_subtree_check,no_root_squash
Storage/nextcloud [email protected]:@192.168.1.5,no_subtree_check,no_root_squash
```
Per-node addresses rather than `@192.168.2.0/24`, following the precedent of the old host. A `/24`
would work and would survive adding a node; it also grants the whole subnet.
**All three nodes must appear on BOTH shares.** Any media pod can be scheduled on any node, so a
missing node is a latent failure, not an immediate one: it surfaces the first time a pod reschedules
there, as `access denied by server while mounting`, with the cause weeks stale. One was caught
exactly this way during setup — the plex share listed `192.168.2.13`, an address no node holds, and
so was missing `192.168.2.12` (k3s-prod-3). Nothing was failing at the time.
### The option flags ARE the permission contract
| flag | why it matters here |
|---|---|
| **`no_all_squash`** | The one the whole model rests on. Client uids pass through untouched, so the arrs' `PUID=1000` arrives as the owner of a `0755 1000:3000` tree. In the UI this is **Mapall User/Group left EMPTY** — setting either squashes every client to one identity and erases the reader/writer distinction entirely. |
| **`sec=sys`** | AUTH_SYS. Numeric uid/gid, not Kerberos principals. §6 is only true because of this. |
| **`root_squash`** | Maproot left empty. The Proxmox export carried `no_root_squash`; **nothing here needs it** — every write to the tree happens after the LSIO drop to uid 1000, and the images only chown their Longhorn `/config`. This is a deliberate tightening over the old export, not parity. |
| **`rw` / `ro`** | The archive's read-only posture is **server-enforced**, as it was before. The media tree cannot get the same treatment — see below. |
| **`secure`** | Reserved source ports required; "Allow non-root mount" off. The kubelet mounts as root from a reserved port and does not need the exception. |
**Plex's read-only posture is still a CLIENT-side guarantee.** Plex is ro and the arrs are rw, but
all four can land on any node, so the server sees the same three source addresses for all of them
and cannot distinguish the reader from the writers. `spec.nfs.readOnly: true` on `plex-media` plus
the volumeMount flag is the whole enforcement. Unchanged by the move; an honest limitation, not an
oversight. If it ever needs to be server-enforced, the lever is a dedicated node pool for Plex.
### Service settings that are NOT obvious from the export list
- **Enabled Protocols must include NFSv4** (above).
- **Require Kerberos for NFSv4: off.** The numeric AUTH_SYS pass-through is the model.
- **Manage Groups Server-side: off.** This is `rpc.mountd --manage-gids`: the server *discards the
client's supplied group list* and looks that uid's groups up in TrueNAS's own user database
instead. With no local uid 1000 defined, the arrs' `PGID=3000` would simply vanish. Writes would
survive anyway — uid 1000 **owns** the tree, so the owner bits grant access without the group —
but it is a free way to lose group semantics.
- **NFSv4 DNS Domain: blank.** It only feeds `idmapd.conf`, which numeric mapping bypasses.
- **Bind IP 192.168.2.30** means loopback does not answer: check with `showmount -e 192.168.2.30`,
not `localhost`.
There is **no** "NFSv3 ownership model for NFSv4" control on current TrueNAS — it was an older
TrueNAS/CORE option, and no replacement is needed: on a Linux client against a Linux server over
AUTH_SYS, both the `nfs` and `nfsd` modules default to `nfs4_disable_idmapping=Y`, so NFSv4 carries
raw numeric ids rather than `user@domain` principals. §9's `ls -n` probe is how you confirm it.
### The ZFS `sharenfs` properties must be CLEARED, not edited
The pools arrived with the old host's exports still set as dataset properties:
```
Storage/plex [email protected]:@10.123.1.10:@192.168.1.18:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
Storage/nextcloud [email protected]:@192.168.1.5,[email protected]:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash
Storage/docker-volumes [email protected]:@192.168.1.5:@192.168.1.10:@192.168.1.18,no_subtree_check,no_root_squash
```
TrueNAS generates `/etc/exports` from its own middleware database and will fight a property it did
not set — the failure mode is two differently-configured exports of one path, with no indication
which is in force. Cleared 2026-10-07:
### `Storage/plex` — append the six k3s node addresses to the existing `rw=` list
```sh
zfs inherit sharenfs Storage/plex
zfs inherit sharenfs Storage/nextcloud
zfs inherit sharenfs Storage/docker-volumes
zfs set sharenfs='[email protected]:@10.123.1.10:@192.168.1.18:@192.168.1.40:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/plex
```
`Storage` is `off`, so each resolves to `off`.
Every pre-existing host is preserved verbatim and no option changes; only clients are added.
### Do NOT touch the dataset's permissions or ACLs
### `Storage/nextcloud` — add the k3s nodes to a **`ro=`** list, not to `rw=`
The tree's ownership **is** the access story (§6), and TrueNAS offers two ways to destroy it
irreversibly across 8.33 T.
```sh
zfs set sharenfs='[email protected]:@192.168.1.5,[email protected]:@192.168.1.41:@192.168.1.42:@192.168.2.10:@192.168.2.11:@192.168.2.12,no_subtree_check,no_root_squash' Storage/nextcloud
```
`Storage/plex` carries `acltype=nfsv4` (SCALE's built-in default, `SOURCE: default`) and
`aclinherit=passthrough` (inherited). Measured 2026-10-07: the directories carry **trivial ACLs
only** — `nfs4xdr_getfacl` shows just `owner@`/`group@`/`everyone@` matching the mode — so `0755
1000:3000` governs exactly as it did under Proxmox.
This is the one place read-only can be enforced **server-side**, and it is worth taking: only
Nextcloud consumes this dataset from k3s, so the export itself can refuse writes. Even a
misconfigured client cannot then damage the archive.
- **Stay out of the dataset's Edit Permissions / Edit ACL screen.** On an `acltype=nfsv4` dataset
that is a full ACL editor, and its presets write *real* NFSv4 ACLs, recursively if permitted.
There is no undo.
- `aclinherit=passthrough` means a single ACL written at `/mnt/Storage/plex` would be **inherited by
everything created under it afterwards** — the blast radius is the whole future library, not one
directory.
- Note GNU `ls`'s `+` marker is **not** a reliable ACL indicator here: it looks for a POSIX ACL
xattr, and this dataset stores NFSv4 ACLs elsewhere. Use `nfs4xdr_getfacl`.
**The media tree cannot get the same treatment.** Plex is read-only and the arrs are read-write, but
all three may be scheduled on any node, so the server sees the same source addresses for all of
them and cannot distinguish the reader from the writers. **Plex's read-only posture is therefore a
client-side guarantee** (`spec.nfs.readOnly: true` on `plex-media`, plus the volumeMount flag) — an
honest limitation, not an oversight. If it ever needs to be server-enforced, the lever is a
dedicated node pool for Plex, which is a much larger change.
`xattr=on` rather than `sa` is a performance detail, not a permissions one, and changing it would
not affect existing files.
Per-node addresses rather than `@192.168.2.0/24` follows the precedent already on this host
(`/etc/exports` lists the dev nodes individually, as `/32`). A `/24` would work and would survive
adding a node; it also grants the whole subnet. Per-node is the tighter default.
### Verifying, without changing anything
```sh
cat /proc/fs/nfsd/versions # +4.1 must be present
exportfs -v # the two shares, three nodes each, flags as tabulated above
showmount -e 192.168.2.30 # NOT localhost -- the service is bound to one address
zfs get -r -o name,property,value,source sharenfs Storage # every row off or -
exportfs -v # the new client lines appear, existing ones unchanged
zfs get sharenfs Storage/plex Storage/nextcloud
```
---
## 6. uid / gid — the failure this design exists to avoid
The export uses `sec=sys` and `no_all_squash`, so **the client's numeric uid is passed through
verbatim** and the server's on-disk ownership is the entire access story. (It no longer carries
`no_root_squash`, which the Proxmox export did — see §5. Nothing here needs it, and the uid that
matters is 1000, not 0.) The media tree is
The export uses `sec=sys` and `no_root_squash`, so **the client's numeric uid is passed through
verbatim** and the server's on-disk ownership is the entire access story. The media tree is
`0755 1000:3000` throughout (some directories `1000:1000`), so **only uid 1000 can write it**.
**`fsGroup` cannot fix this, and reaching for it is how this gets misdiagnosed.** The in-tree NFS
@@ -278,27 +233,24 @@ Nextcloud needs no uid work at all, which is one more reason its mount is read-o
---
## 7. Why this knob is unvalidated at create AND immutable at update
**The `TBD-PROD-BULK-NFS-SERVER` placeholder is gone** (2026-10-07) — prod now carries a real
address, because the NAS is on prod's own subnet and there is no longer a decision to defer. The
hazard that made it load-bearing is worth keeping on the page, because it is the half that explains
why §10 exists.
## 7. The prod placeholder does NOT fail at apply time — read this before merging
Every other `TBD-PROD-*` in `clusters/prod/kustomization.yaml` is rejected loudly by CRD validation
(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one was not.**
(Cilium, LB-IPAM, CNPG, Velero) because it is not a valid IP or URL. **This one is not.**
A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepted
`TBD-PROD-BULK-NFS-SERVER`, the PV was created, and the PVC **bound** to it — binding is pure
control-plane and never touches the server. The failure appeared only when a pod mounted:
A PersistentVolume's `spec.nfs.server` is an **unvalidated string**. The API server accepts
`TBD-PROD-BULK-NFS-SERVER`, the PV is created, and the PVC **binds** to it — binding is pure
control-plane and never touches the server. The failure appears only when a pod mounts:
```
MountVolume.SetUp failed ... Failed to resolve server TBD-PROD-BULK-NFS-SERVER
```
**The same field is immutable on UPDATE** (§10), which is the other half of the same awkwardness: a
wrong value is accepted silently at create, and then cannot be corrected in place. Together they are
why repointing this knob is a delete-and-recreate rather than a reconcile.
so **plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod.**
**Consequently: set `BULK_NFS_SERVER` in `clusters/prod/kustomization.yaml` BEFORE prod reconciles a
revision that carries these bulk volumes.** Dev renders a working value and is where these manifests
are meant to be proven first.
### The static guard this convention leans on is missing from `main` — but the gate now refuses the value anyway
@@ -335,9 +287,10 @@ Without the arm, HALF A is what holds the line, by design.
EPHEMERAL layer"). Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it
means those apps start from an empty config volume, so it belongs in its own change with the
operator present. **This change does not make it worse** — that is why the media mount is `/media`.
2. **RESOLVED — qbittorrent now has the bulk mount** (`subPath: downloads` at `/downloads`), so the
import source is written by the k3s download client rather than the legacy non-k3s one. See
`catalog/qbittorrent.yaml`.
2. **qbittorrent has no bulk mount.** Today the arrs' import source is written by the **legacy
non-k3s** download client, which is coherent while the legacy stack stays live. Giving the k3s
qbittorrent the same export at the same `/media` path is the natural follow-up, and the paths were
chosen so it needs no Remote Path Mapping.
3. **`/mnt/Storage/plex/downloads/downloads` is a symlink to `/shared/downloads`**, which is outside
the export. Over NFS it is a dangling symlink on every client. Harmless, but expect Sonarr to log
it while scanning `downloads/`.
@@ -347,10 +300,10 @@ Without the arm, HALF A is what holds the line, by design.
5. **`nconnect` was not set.** It would likely help throughput on the media tree, but it is a tuning
change that should be measured, not guessed, and it changes client behaviour under server
failure.
6. **Every claim about the NAS in §5 is measured (2026-10-07); the post-move MOUNT is not.** The PVs
still have to be recreated (§10) before any pod mounts 192.168.2.30, so the §9 probes are the
outstanding verification, not a formality. The pre-move equivalents did pass against the old host
on 2026-09-12.
6. **Nothing was verified against a live cluster.** No `kubectl` context is usable from this
workstation (browser-OIDC only), and this change is manifest-authoring under an explicit
do-not-apply constraint. Every claim about the *host* is measured; every claim about *binding and
mounting* is reasoned from the manifests.
7. **The bulk-tier requirement ID is deliberately not typed on any non-`.md` line of this change.**
`ci/touched-requirements.sh` attributes requirements from R-IDs appearing in added lines, and the
in-cluster conformance test for the bulk tier is still a gap, so typing it would select a
@@ -359,95 +312,36 @@ Without the arm, HALF A is what holds the line, by design.
---
## 9. Validating, after the PVs are recreated
## 9. Validating on dev, once the export is extended
These run against **prod**; dev is decommissioned. They are the outstanding verification for the
move — nothing below has been run against 192.168.2.30. Read-only checks first:
Dev can prove all of this before prod's network path exists. Read-only checks first:
```sh
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get pv -o custom-columns=\
NAME:.metadata.name,SERVER:.spec.nfs.server,PATH:.spec.nfs.path,STATUS:.status.phase'
infra/devcontainer/dev.sh run kubectl --context k3s-dev get pv
infra/devcontainer/dev.sh run kubectl --context k3s-dev get pvc -A | grep -E 'media|legacy-archive'
infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr get pod -o wide
```
Expect **five** NFS PVs — `plex-media`, `sonarr-media`, `radarr-media`, `qbittorrent-downloads`,
`nextcloud-legacy-archive` (four of them naming the *same* export, one per namespace, which is the
cross-namespace model) — each `Bound`, `RECLAIM POLICY Retain`, empty `STORAGECLASS`, and
**`SERVER 192.168.2.30`**. A PV still showing `192.168.2.102` means the recreate in §10 did not
happen; Flux will NOT have updated it in place.
Expect **four** PVs — `plex-media`, `sonarr-media`, `radarr-media`, `nextcloud-legacy-archive` (three
of them naming the *same* export, one per namespace, which is the cross-namespace model) — each
`Bound` to the same-named PVC, with `RECLAIM POLICY Retain` and an empty `STORAGECLASS`.
Then the four things that actually matter, from inside a pod:
Then the two things that actually matter, from inside a pod:
```sh
# (a) numeric ids survived the wire -- the idmapping check, new since the TrueNAS move
kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- sh -c 'ls -n /media | head -3'
# expect uid 1000 / gid 3000 on the entries. 65534 / nobody means ids are being mapped, not
# passed through, and every import will fail EACCES while the pod still reports Ready.
# (a) the writer can write — this is the uid check, and it is the one that fails in practice
infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
sh -c 'id; touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
# (b) the writer can write -- the uid check, and the one that fails in practice
kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
sh -c 'touch /media/tv/.k3s-write-probe && rm /media/tv/.k3s-write-probe && echo WRITE-OK'
# (c) hardlinks work BETWEEN the library and the download path -- the root-mount check
kubectl --context k3s-prod -n sonarr exec deploy/sonarr -c sonarr -- \
# (b) hardlinks work BETWEEN the library and the download path — the root-mount check
infra/devcontainer/dev.sh run kubectl --context k3s-dev -n sonarr exec deploy/sonarr -- \
sh -c 'set -e; d=/media/downloads/complete/tv-sonarr/.probe; t=/media/tv/.probe-link;
echo x > "$d"; ln "$d" "$t"; ls -i "$d" "$t"; rm -f "$d" "$t"; echo HARDLINK-OK'
# (d) the reader really is read-only
kubectl --context k3s-prod -n plex exec deploy/plex -- sh -c 'touch /media/.probe 2>&1 | head -1'
# expect "Read-only file system"
# (c) the reader really is read-only
infra/devcontainer/dev.sh run kubectl --context k3s-dev -n plex exec deploy/plex -- \
sh -c 'touch /media/.probe 2>&1 | head -1' # expect "Read-only file system"
```
`(c)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
`(b)` printing two identical inode numbers is the proof that §4's argument holds in practice. If it
prints `EXDEV`, something has split the single mount back into two.
Reading the failures together is what localises the fault: **(b) EACCES with (a) showing `nobody`**
is idmapping (§5 service settings). **(b) EACCES with (a) showing `1000`** is on-disk ownership —
stop and check `ls -ldn` on the NAS before changing anything else.
---
## 10. Repointing BULK_NFS_SERVER means DELETING AND RECREATING THE PVs
**A bound PersistentVolume's NFS source is immutable.** Measured by server-side dry-run on
2026-10-07:
```
$ kubectl patch pv sonarr-media --type=json --dry-run=server \
-p '[{"op":"replace","path":"/spec/nfs/server","value":"192.168.2.30"}]'
The PersistentVolume "sonarr-media" is invalid: spec.persistentvolumesource: Forbidden:
spec.persistentvolumesource is immutable after creation
```
Note the pairing with §7: `spec.nfs.server` is **unvalidated at create** (any string is accepted and
the PVC binds) and **immutable at update**. So editing `BULK_NFS_SERVER` alone does not fail loudly
and does not work either — Flux's apply errors on the five PVs and the pods keep their previous,
now-dead mount.
This is safe to do because `persistentVolumeReclaimPolicy: Retain` means deleting a PV never touches
the data, and `catalog/render.py` emits a **pre-set `claimRef`** on each one, so a recreated PV
rebinds its own PVC with no manual step.
**Operator sequence** (mutating `kubectl` is an operator action, not an agent action):
```sh
# 1. Stop the consumers. This also releases the hung hard-mounts against the old server:
# the pods report Ready, but any real I/O on /media blocks indefinitely once 2049 stops
# answering, because the PVs mount `hard`.
for ns in plex sonarr radarr qbittorrent nextcloud; do
kubectl --context k3s-prod -n $ns scale deploy --all --replicas=0
done
# 2. Delete the five PVs. Retain + no live claim means nothing is reclaimed or erased.
kubectl --context k3s-prod delete pv plex-media sonarr-media radarr-media \
qbittorrent-downloads nextcloud-legacy-archive
# 3. Let Flux reconcile the revision carrying BULK_NFS_SERVER=192.168.2.30, which recreates them.
# Confirm .status.lastAppliedRevision matches the GitRepository artifact before continuing.
# 4. Bring the consumers back, then run every probe in §9.
```
The PVCs pass through `Lost` between steps 2 and 3; that is expected and resolves when the PV with
the matching `claimRef` reappears. If a PVC is still `Lost` after Flux reconciles, the PV name or
its `claimRef` namespace does not match — do **not** delete the PVC to "fix" it.
+5 -4
View File
@@ -241,10 +241,11 @@ about runtime behaviour in this document is a design claim, not a measurement.
`gitea.{{ k3s_base_domain }}` to the Gateway VIP in `node_baseline_hosts_pins`, the same
pin the Keycloak issuer already carries, applied by
`ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`;
(ii) the pull is still unauthenticated and no `imagePullSecret` exists in the tree — with
the credential decision now being local Gitea accounts with PATs, that means a
`dockerconfigjson` for an account whose token carries `read:package`. Only then does the
digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
(ii) the pull must be authenticated — **now wired**: `pm-agent-registry`
(`dockerconfigjson`) is referenced by the Deployment and the committed worker Job
template, and the admission policy pins `imagePullSecrets` to exactly that name. The
operator supplies the value, from a Gitea token that also carries `read:package`. Only
then does the digest matter. Operator runs `infra/agent/build-image.sh`, pushes, and one PR replaces both
placeholder digests. Expect the build itself to need iteration too: the nodejs module
stream and the npm install have never run against this base. The same two blockers sit in
front of `conformance-runner`, which shares the registry path and has never been pulled.
+2
View File
@@ -922,6 +922,8 @@ def selftest_template(path: str) -> None:
assert pod.get("automountServiceAccountToken") is False, \
"template must not mount a ServiceAccount token"
assert pod.get("restartPolicy") == "Never", "a retried agent would double-push"
assert [x.get("name") for x in pod.get("imagePullSecrets") or []] == ["pm-agent-registry"], \
"the template must name exactly the committed pull secret (admission pins it)"
assert len(pod.get("containers") or []) == 1, "exactly one container"
c = pod["containers"][0]
assert c.get("command") == ["/usr/local/bin/pm-worker"], "command must be the pinned entrypoint"
+7
View File
@@ -68,6 +68,13 @@ spec:
# guarantee it cannot use one is for it not to have one.
automountServiceAccountToken: false
serviceAccountName: pm-agent-worker
# The kubelet's pull credential, the same one the dispatcher uses. Pinned by the
# admission policy to exactly this name: `imagePullSecrets` is a pod-spec field, so a
# Job could otherwise name any Secret in the namespace and have the kubelet read it
# before a container ever starts. The image is digest-pinned, so the leverage is small
# — but "only the committed shape" is the rule this template exists to express.
imagePullSecrets:
- name: pm-agent-registry
securityContext:
runAsNonRoot: true
runAsUser: 65532
-14
View File
@@ -32,23 +32,9 @@ ARG TARGETARCH=amd64
# Distro packages: libvirt client libs (harmless disk-image utilities), ansible, ssh,
# disk-image + cloud-init seed tooling, python3-pip (for the pip-installed test tools),
# diffutils (provides `diff`, used by L1 golden render checks), and common utilities.
#
# wireguard-tools + qrencode serve the REMOTE-ACCESS VPN (platform/vpn-ingress):
# * wireguard-tools gives `wg genkey` / `wg pubkey`, needed to mint the server key and to derive
# every client profile's `[Peer] PublicKey` (infra/vpn-ingress/new-peer.sh).
# * qrencode renders a client profile as a QR code, which is how a PHONE gets enrolled.
# They are here rather than on the operator's host because this image is THE toolchain
# (CLAUDE.md §5; README "Don't install the toolchain on the host"). That rule was briefly broken
# by a runbook that told the operator to run `wg` on the host and `sops` in here, and then split
# every step across the boundary to work around it — a procedure nobody should have to follow, and
# one that cannot be reproduced in CI. Both tools in one image means `sops -d … | wg pubkey` is
# just a pipeline again.
# Unpinned on purpose, like every other dnf package here: the Fedora base is what is pinned
# (FEDORA_VERSION), and these are not gate tools whose version changes what green means.
RUN dnf -y install --setopt=install_weak_deps=False \
libvirt-client qemu-img cloud-utils genisoimage openssh-clients \
ansible-core python3-libvirt python3-pip diffutils \
wireguard-tools qrencode \
git make jq unzip tar gzip xz which ca-certificates curl \
&& dnf clean all
-1
View File
@@ -14,7 +14,6 @@ baseline, and the *same* binaries CI uses.
| Provisioning | OpenTofu (`bpg/proxmox` provider), `ansible-core`, `openssh-clients` |
| GitOps / cluster | `flux`, `kubectl`, `helm` |
| Secrets | `sops`, `age` + `age-keygen` |
| VPN (`platform/vpn-ingress`) | `wg` (wireguard-tools) — mint the server key, derive client public keys; `qrencode` — render a client profile as a phone-scannable QR |
| **CI gate** | `yamllint`, `check-jsonschema`, `kyverno` CLI, `kyverno-json`, `chainsaw`, `yq`, `diffutils`, `make` |
| Misc | `qemu-img`, `cloud-utils`/`genisoimage`, `libvirt-client` (disk-image utilities only — leftovers from the libvirt era, harmless) |
+30 -85
View File
@@ -2,24 +2,10 @@
# =================================================================================================
# new-peer.sh — enrol one device on the remote-access VPN (platform/vpn-ingress).
#
# Generates a WireGuard keypair, renders the device's client profile, prints a QR code for a phone,
# and prints the one `[Peer]` stanza to paste into platform/vpn-ingress/peers.conf. The device's
# private half is written to ONE 0600 file and nowhere else: not sent to the cluster, not committed,
# never logged.
#
# RUN IT IN THE DEV CONTAINER — that is where the toolchain lives (CLAUDE.md §5):
#
# ./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2'
#
# It needs `wg` (wireguard-tools), `qrencode` and `sops`, all three baked into the image. It will
# run on a host that happens to have them too, but the container is the supported way and the only
# one that is reproducible.
#
# THE SERVER'S PUBLIC KEY IS DERIVED FROM THE ENCRYPTED FILE, not from a copy you keep lying around:
# `sops -d` the private half out of platform/vpn-ingress/wg-server-key.sops.yaml and pipe it
# straight into `wg pubkey`. The plaintext never touches disk and never reaches a terminal. (This
# is why both tools being in ONE image matters — see the Containerfile comment.) Override with
# WG_SERVER_PUBKEY or WG_SERVER_KEY_FILE if you are working off-tree.
# Generates a WireGuard keypair ON THIS MACHINE, renders the device's client profile, optionally
# prints a QR code for a phone, and prints the one `[Peer]` stanza to paste into
# platform/vpn-ingress/peers.yaml. The private half NEVER leaves this machine: it is not sent to the
# cluster, not committed, and not written to any log.
#
# WHY A SCRIPT AND NOT A RUNBOOK STEP. Three of these fields are easy to get wrong in ways that are
# either silently insecure or silently broken, and all three are mechanical:
@@ -39,30 +25,24 @@
# <last-octet> 2-254, the device's address inside WG_CLIENT_CIDR. Must be unused.
# --split split tunnel: route ONLY the LAN and the tunnel subnet. Default is a FULL
# tunnel (0.0.0.0/0) so the device also egresses the house WAN.
# --out DIR where to write the profile. Default: `.cache/wg-peers/` in the repo, which is
# GITIGNORED (`.gitignore` `.cache/`) and, unlike /tmp inside the container,
# survives the container exiting — the repo is its only persistent read-write
# mount, so a profile written anywhere else is gone before you can move it to a
# desktop. The script refuses any destination git does not ignore.
# --out DIR where to write the profile (default: a 0700 dir under $TMPDIR).
#
# ENVIRONMENT
# WG_SERVER_PUBKEY the server's public key, if you want to skip the SOPS decrypt entirely.
# WG_SERVER_KEY_FILE a local copy of the server PRIVATE key to derive the public half from,
# instead of reading the SOPS file.
# WG_SERVER_SOPS_FILE default platform/vpn-ingress/wg-server-key.sops.yaml
# WG_SERVER_KEY_FILE server PRIVATE key (default ~/.config/wireguard-home/server.key); its public
# half is derived from it. Alternatively set WG_SERVER_PUBKEY directly.
# WG_ENDPOINT_HOST default wan.home.keithjackson.dev
# WG_CLIENT_CIDR default 10.90.0.0/24 (must match clusters/base/cluster-vars.yaml)
# WG_LAN_CIDR default 192.168.2.0/24 (prod LAN_CIDR)
# WG_DNS default 192.168.2.23 (prod PIHOLE_DNS_VIP)
#
# REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container.
# REQUIRES: wireguard-tools (`wg`). `qrencode` is optional and only used for the QR code.
# =================================================================================================
set -euo pipefail
die() { printf 'new-peer.sh: %s\n' "$*" >&2; exit 1; }
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.yaml"
# --- defaults, each mirroring a value that is DECLARED elsewhere in this repo -------------------
# These are duplicated here rather than parsed out of the YAML on purpose: this script is meant to
@@ -71,7 +51,7 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
: "${WG_CLIENT_CIDR:=10.90.0.0/24}" # clusters/base/cluster-vars.yaml
: "${WG_LAN_CIDR:=192.168.2.0/24}" # clusters/prod LAN_CIDR
: "${WG_DNS:=192.168.2.23}" # clusters/prod PIHOLE_DNS_VIP
: "${WG_SERVER_SOPS_FILE:=${REPO_ROOT}/platform/vpn-ingress/wg-server-key.sops.yaml}"
: "${WG_SERVER_KEY_FILE:=${HOME}/.config/wireguard-home/server.key}"
# wan.<base-domain> is an UNPROXIED A record that platform/externaldns/ddns.yaml keeps pointed at
# the current WAN IP (checked every 5 min). Use it, never a literal address: this house has a
# DYNAMIC WAN IP, and a profile pinned to today's address stops working without warning and with no
@@ -103,28 +83,14 @@ printf '%s' "$NAME" | grep -Eq '^[a-z0-9][a-z0-9-]{0,30}$' \
printf '%s' "$OCTET" | grep -Eq '^[0-9]+$' || die "octet must be a number: got '$OCTET'"
[ "$OCTET" -ge 2 ] && [ "$OCTET" -le 254 ] || die "octet must be 2-254 (.1 is the server): got '$OCTET'"
command -v wg >/dev/null || die "'wg' not on PATH. Run this inside the dev container:
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh $NAME $OCTET'"
command -v wg >/dev/null || die "wireguard-tools not installed ('wg' not on PATH). Debian/Ubuntu: apt install wireguard-tools"
# --- the server's public key --------------------------------------------------------------------
# Order: an explicit pubkey, then a local private-key file, then the SOPS file in the tree. The
# last is the normal path and needs no local copy of anything.
if [ -z "${WG_SERVER_PUBKEY:-}" ]; then
if [ -n "${WG_SERVER_KEY_FILE:-}" ]; then
[ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read WG_SERVER_KEY_FILE '$WG_SERVER_KEY_FILE'"
WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
else
command -v sops >/dev/null || die "'sops' not on PATH. Run this inside the dev container (see --help)."
[ -r "$WG_SERVER_SOPS_FILE" ] || die "cannot read '$WG_SERVER_SOPS_FILE'"
# The decrypted private key goes straight into `wg pubkey` — never to a file, never to a tty.
# `wg pubkey` fails on the committed REPLACE-ME placeholder, which is the useful error here.
WG_SERVER_PUBKEY="$(sops -d --extract '["stringData"]["privatekey"]' "$WG_SERVER_SOPS_FILE" \
| tr -d ' \r\n' | wg pubkey 2>/dev/null)" || WG_SERVER_PUBKEY=""
[ -n "$WG_SERVER_PUBKEY" ] || die "could not derive the server public key from
$WG_SERVER_SOPS_FILE
Either it still holds the REPLACE-ME placeholder (see runbooks/vpn-remote-access.md step 2),
or this shell cannot decrypt it (the age keyring is mounted at ~/.config/sops in the container)."
fi
[ -r "$WG_SERVER_KEY_FILE" ] || die "cannot read server key '$WG_SERVER_KEY_FILE'.
Either point WG_SERVER_KEY_FILE at it, or set WG_SERVER_PUBKEY to the public half.
If the server key does not exist yet, see runbooks/vpn-remote-access.md step 1."
WG_SERVER_PUBKEY="$(tr -d ' \r\n' < "$WG_SERVER_KEY_FILE" | wg pubkey)"
fi
CLIENT_BASE="${WG_CLIENT_CIDR%/*}"; CLIENT_BASE="${CLIENT_BASE%.*}"
@@ -133,41 +99,22 @@ CLIENT_ADDR="${CLIENT_BASE}.${OCTET}"
# --- refuse a collision, because the failure is otherwise invisible -----------------------------
# Two devices sharing an address does not error anywhere: the server routes that address to
# whichever peer handshook most recently, so BOTH devices appear to work intermittently.
# COMMENT LINES ARE NOT ALLOCATIONS. This used to grep the whole file and so matched the
# COMMENTED-OUT EXAMPLE STANZA in the register, which made `.2` — the first address anyone would
# ask for — permanently un-allocatable with a message claiming it was already taken. Strip
# comments first: a real stanza's `AllowedIPs` line is never commented, while both the template
# and every peer's `# <name> — <owner>` label are.
# FAIL if the register is missing rather than skipping the check. The path was silently wrong once
# (it still said peers.yaml after the file became peers.conf), and because the old guard was
# `[ -r "$PEERS_FILE" ] && ...` the only symptom would have been duplicate detection quietly
# never firing again — the exact silent-skip shape this repo treats as a defect.
[ -r "$PEERS_FILE" ] || die "cannot read the peer register '$PEERS_FILE'.
Duplicate-address detection cannot run, so this script refuses to issue a profile."
if grep -vE '^[[:space:]]*#' "$PEERS_FILE" | grep -q "AllowedIPs *= *${CLIENT_ADDR}/32"; then
die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.conf. Pick another octet."
if [ -r "$PEERS_FILE" ] && grep -q "AllowedIPs *= *${CLIENT_ADDR}/32" "$PEERS_FILE"; then
die "${CLIENT_ADDR}/32 is already allocated in platform/vpn-ingress/peers.yaml. Pick another octet."
fi
if grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.conf.\n' "$NAME" >&2
if [ -r "$PEERS_FILE" ] && grep -qi "^ *# *${NAME} " "$PEERS_FILE"; then
printf 'new-peer.sh: WARNING a peer comment already mentions "%s" in peers.yaml.\n' "$NAME" >&2
fi
# --- output location: 0700, and never inside the repo -------------------------------------------
[ -n "$OUT_DIR" ] || OUT_DIR="${REPO_ROOT}/.cache/wg-peers"
mkdir -p "$OUT_DIR"
if [ -z "$OUT_DIR" ]; then
OUT_DIR="$(mktemp -d "${TMPDIR:-/tmp}/wg-peer-XXXXXX")"
else
mkdir -p "$OUT_DIR"
fi
chmod 700 "$OUT_DIR"
OUT_ABS="$(cd "$OUT_DIR" && pwd)"
# A profile holds a private key, so the destination must not be committable. Inside the repo that
# means git must IGNORE it — test the real property rather than a proxy, because the repo is the
# container's only persistent read-write mount and so the only sane place to write. Outside the
# repo, anywhere goes.
case "$OUT_ABS" in
"$REPO_ROOT"|"$REPO_ROOT"/*)
if ! git -C "$REPO_ROOT" check-ignore -q "$OUT_ABS/probe.conf" 2>/dev/null; then
die "refusing to write a private key to '$OUT_ABS': it is inside the git checkout and NOT
gitignored, so the profile would be committable. Use the default (.cache/wg-peers) or add an
ignore rule first. Verify with: git check-ignore -v <path>"
fi
;;
case "$(cd "$OUT_DIR" && pwd)" in
"$REPO_ROOT"|"$REPO_ROOT"/*) die "refusing to write a private key inside the git checkout ($OUT_DIR)" ;;
esac
PROFILE="${OUT_DIR}/${NAME}.conf"
@@ -215,7 +162,7 @@ cat <<EOF
Profile : ${PROFILE} (0600; contains the private key)
------------------------------------------------------------------------------------------
STEP 1 — append this to platform/vpn-ingress/peers.conf, then PR it:
STEP 1 — add this to platform/vpn-ingress/peers.yaml, under data.peers.conf, then PR it:
# ${NAME} — <owner>, enrolled $(date -u +%Y-%m-%d)
[Peer]
@@ -234,7 +181,7 @@ cat <<EOF
STEP 3 — delete the profile from this machine once the device has it:
shred -u ${PROFILE} 2>/dev/null || rm -f ${PROFILE}
The tunnel will not work until the peers.conf PR is MERGED and Flux has rolled the DaemonSet
The tunnel will not work until the peers.yaml PR is MERGED and Flux has rolled the DaemonSet
(~1 min). Until then the device handshakes and gets silence, exactly like an unknown key.
------------------------------------------------------------------------------------------
EOF
@@ -243,8 +190,6 @@ if command -v qrencode >/dev/null; then
printf ' QR code (this IS the private key — your terminal scrollback now holds a credential):\n\n'
qrencode -t ansiutf8 < "$PROFILE"
else
# Should not happen in the dev container: qrencode is baked into the image. Kept as a real
# fallback for a host run, rather than failing after the profile has already been written.
printf ' qrencode not on PATH, so no QR code was printed. It is in the dev container —\n'
printf ' rerun there, or transfer %s to the device by another route.\n' "$PROFILE"
printf ' qrencode not installed, so no QR code was printed (Debian/Ubuntu: apt install qrencode).\n'
printf ' Transfer %s to the device by another route instead.\n' "$PROFILE"
fi
@@ -111,6 +111,12 @@ spec:
=(hostNetwork): "false"
=(hostPID): "false"
=(hostIPC): "false"
# The kubelet reads this Secret BEFORE any container starts, so it is part
# of the shape even though no container references it. A pattern on an array
# asserts "at least one element matches", which is enough to REQUIRE the
# committed entry; rule 7 is what forbids a second one.
imagePullSecrets:
- name: pm-agent-registry
containers:
- image: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:*"
command:
@@ -336,6 +342,54 @@ spec:
# dispatcher itself uses envFrom (it is a Deployment, not a Job, and not matched by this
# policy); a worker must not.
# =====================================================================================
# =====================================================================================
# Rule 7 — exactly one imagePullSecret, and it is the committed one.
#
# `imagePullSecrets` is a pod-spec field that no container references, which is precisely
# why it is easy to leave unread — the same shape of omission as `initContainers` and
# `env` before it. The kubelet reads the named Secret, in this namespace, before any
# container starts. A Job naming a different one would have the kubelet read a Secret the
# worker was never meant to touch.
#
# The leverage is genuinely small: the image is digest-pinned, so a different credential
# cannot change WHAT runs, and the only Secrets in this namespace are the agent's own.
# It is pinned anyway, because "only the committed shape" is the whole premise of this
# policy, and because the cost of the rule is four lines while the cost of rediscovering
# the omission is an incident.
#
# Rule 1's pattern REQUIRES the committed entry; this forbids any OTHER entry and any
# second entry. A foreach over an absent list yields no result, which is why both halves
# are needed.
# =====================================================================================
- name: worker-pull-secret-must-be-the-committed-one
match:
all:
- resources:
kinds:
- Job
namespaces:
- pm-agent
subjects:
- kind: ServiceAccount
name: pm-agent
namespace: pm-agent
validate:
message: >-
A worker Job may declare exactly ONE imagePullSecret, named `pm-agent-registry`.
The kubelet reads that Secret before any container starts, so naming another one
reaches a credential the worker was never meant to touch.
foreach:
- list: "request.object.spec.template.spec.imagePullSecrets"
deny:
conditions:
any:
- key: "{{ length(request.object.spec.template.spec.imagePullSecrets) }}"
operator: NotEquals
value: 1
- key: "{{ element.name }}"
operator: AnyNotIn
value: ["pm-agent-registry"]
- name: worker-may-not-use-envfrom
match:
all:
@@ -56,6 +56,11 @@ results:
kind: Job
resources: [good-run, real-template]
result: pass
- policy: pm-agent-job-shape
rule: worker-pull-secret-must-be-the-committed-one
kind: Job
resources: [good-run, real-template]
result: pass
# ---------------- THE escalation, and the shape rule's other refusals ----------------
# `evil-command` is the case the whole model exists for: right namespace, right creator,
@@ -73,6 +78,14 @@ results:
- host-namespaces
- restart-onfailure
- second-container
- no-pull-secret
result: fail
# ---------------- the pull secret: a second one, or a different one ----------------
- policy: pm-agent-job-shape
rule: worker-pull-secret-must-be-the-committed-one
kind: Job
resources: [extra-pull-secret, wrong-pull-secret]
result: fail
# ---------------- args, and the second container ----------------
@@ -31,6 +31,8 @@ spec:
restartPolicy: Never
automountServiceAccountToken: false
serviceAccountName: pm-agent-worker
imagePullSecrets:
- name: pm-agent-registry
securityContext:
runAsNonRoot: true
runAsUser: 65532
@@ -95,6 +97,8 @@ spec:
restartPolicy: Never
automountServiceAccountToken: false
serviceAccountName: pm-agent-worker
imagePullSecrets:
- name: pm-agent-registry
containers:
- name: worker
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
@@ -492,3 +496,75 @@ spec:
readOnlyRootFilesystem: true
envFrom:
- secretRef: { name: pm-agent-credentials }
---
# A SECOND pull secret beside the committed one. containers[0] is perfectly compliant, and
# rule 1's array pattern is satisfied by the first element — which is why the count is
# asserted separately.
apiVersion: batch/v1
kind: Job
metadata:
name: extra-pull-secret
namespace: pm-agent
spec:
template:
spec:
restartPolicy: Never
automountServiceAccountToken: false
serviceAccountName: pm-agent-worker
imagePullSecrets:
- name: pm-agent-registry
- name: pm-agent-credentials
containers:
- name: worker
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/pm-worker"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
---
# A DIFFERENT pull secret — the kubelet would read a Secret the worker was never meant to
# touch, before any container starts.
apiVersion: batch/v1
kind: Job
metadata:
name: wrong-pull-secret
namespace: pm-agent
spec:
template:
spec:
restartPolicy: Never
automountServiceAccountToken: false
serviceAccountName: pm-agent-worker
imagePullSecrets:
- name: pm-agent-credentials
containers:
- name: worker
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/pm-worker"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
---
# NO pull secret at all. Rule 7's foreach yields nothing over an absent list, so this is
# rule 1's pattern to refuse — which is why the requirement lives in both.
apiVersion: batch/v1
kind: Job
metadata:
name: no-pull-secret
namespace: pm-agent
spec:
template:
spec:
restartPolicy: Never
automountServiceAccountToken: false
serviceAccountName: pm-agent-worker
containers:
- name: worker
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/usr/local/bin/pm-worker"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
@@ -199,7 +199,7 @@ data:
"id": 15,
"type": "timeseries",
"title": "CAUSE 4 — NFS RPC retransmission ratio, per node",
"description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.30, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
"description": "node_nfs_rpc_retransmissions_total / node_nfs_rpcs_total, from /proc/net/rpc/nfs. Alert line at 1% (NodeNFSRetransmissionRateHigh). MEASURED BASELINE IS EXACTLY ZERO on all three nodes over 48h, including throughout the 2026-09-14 spike — so any visible line here means the export at 192.168.2.102, or the path to it, is dropping requests. A flat zero does NOT clear NFS: it only clears the TRANSPORT. A merely SLOW export needs per-mount RPC round-trip time, which arrives with the mountstats collector (see kube-prometheus-stack.yaml) and is the known gap recorded in runbooks/node-io.md.",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 39},
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "custom": {"drawStyle": "line", "fillOpacity": 5, "thresholdsStyle": {"mode": "line"}}, "thresholds": {"mode": "absolute", "steps": [
{"color": "green", "value": null}, {"color": "red", "value": 0.01}
+30 -16
View File
@@ -71,21 +71,33 @@ Both are deliberate, and both fail **visibly** rather than silently:
inventory, never typed on a node. Apply it scoped to that one task:
`cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod
--tags hosts`.
- **The pull is still unauthenticated** once the node reaches the right endpoint. An
unauthenticated client gets nothing from this Gitea's registry, and there is no
`imagePullSecret` anywhere in the tree. Consistent with the agent credential decision
(local Gitea accounts with PATs), the fix is a `dockerconfigjson` for a local Gitea
account whose token carries `read:package`, referenced from the Deployment and the
committed worker Job template — not built yet, because minting that token is yours.
- **The pull must be authenticated** once the node reaches the right endpoint: an
unauthenticated client gets nothing from this registry. **Now wired** —
`pm-agent-registry` is a `dockerconfigjson` Secret referenced by both the Deployment and
the committed worker Job template, and the admission policy pins it to exactly that
name. What is left to you is the value: the Gitea token must carry **`read:package`** in
addition to `write:repository` and `write:issue`, and the dockerconfigjson is built from
it. Steps:
[`pm-agent-registry.sops.yaml.example`](./pm-agent-registry.sops.yaml.example).
The same two apply to `conformance-runner`: same registry path, same push-then-pin model,
never pulled either.
2. **The credentials are placeholders.** `pm-agent-credentials.sops.yaml` is encrypted to the
repository's age recipient but holds `REPLACE-ME-…` strings. The hand-off —
which Gitea account to create, which scopes, and why it must *not* be an administrator —
is in [`pm-agent-credentials.sops.yaml.example`](./pm-agent-credentials.sops.yaml.example).
Until then the dispatcher CrashLoops with `GET /user returned no login — is GITEA_TOKEN
valid?`.
2. **The credentials are placeholders — two Secrets, one token.** Both are encrypted to the
repository's age recipient and both hold `REPLACE-ME-…` strings:
| Secret | type | read by | hand-off |
|---|---|---|---|
| `pm-agent-credentials` | Opaque | the application, as env | [`.example`](./pm-agent-credentials.sops.yaml.example) |
| `pm-agent-registry` | dockerconfigjson | the **kubelet**, before any container starts | [`.example`](./pm-agent-registry.sops.yaml.example) |
They hold the **same** Gitea token in two formats, because a Secret cannot be both — the
same one-value-in-two-files pattern the Keycloak client secret already uses here. So mint
it once, with all three scopes: `read:package` (the pull), `write:repository` (clone,
branch push, pull request) and `write:issue` (issue comments). The first `.example`
explains which account to create and why it must *not* be an administrator.
Until they are filled in: the pull fails `401`, and if you fix only the pull, the
dispatcher CrashLoops with `GET /user returned no login — is GITEA_TOKEN valid?`.
Its Flux Kustomization is `wait: false` precisely because of those two states: a workload
that is *expected* to be unhealthy for a while must never be able to gate anything.
@@ -148,6 +160,7 @@ Job from a committed template. **The model's output is data, never a command.**
| **cannot** | clone from or push to anywhere but `gitea.gitea.svc.cluster.local:3000`: `pm-worker` validates `PM_GITEA_URL` against a one-entry allowlist baked into the image. In-cluster consumers reach a service by its cluster-local name — a different path by design from how a client outside the cluster reaches it — so the external hostname is not an accepted value |
| **cannot** | authenticate a `git push` **during the model session**: `GIT_ASKPASS` is unset for its duration, so the restriction is structural rather than a request in a prompt |
| **cannot** | be created in any shape but the committed one — see the admission policy |
| **cannot** | have the kubelet read any Secret but `pm-agent-registry` on its behalf: `imagePullSecrets` is pinned to exactly that one name |
### What is *not* defended, stated plainly
@@ -193,14 +206,15 @@ Two things answer it, and neither is sufficient alone:
`command` with no args, that powerless SA, no mounted token, `restartPolicy: Never`,
non-root with no privilege escalation and a read-only root filesystem, only the
`workspace`/`tmp` **emptyDir** volumes, no init or ephemeral containers, only the
committed env **names**, and no `envFrom`.
committed env **names**, no `envFrom`, and exactly one `imagePullSecret` — the committed
one.
Six rules, because pinning `command` only moves the injection to whatever adjacent field
Seven rules, because pinning `command` only moves the injection to whatever adjacent field
nobody enumerated — on the sibling policy that was `initContainers` (measured: a Job with a
compliant `containers[0]` and a privileged init container was **admitted**, pass 3 fail 0),
then `ephemeralContainers`, then `env`, then `envFrom`. All of them are closed here from the
start, and `platform/kyverno/tests/pm-agent/` proves each refusal offline: **28 assertions,
28 passing** under the pinned `kyverno` CLI. The worker's own input guards have their own
start, and `platform/kyverno/tests/pm-agent/` proves each refusal offline: **33 assertions,
33 passing** under the pinned `kyverno` CLI. The worker's own input guards have their own
offline probe — `infra/agent/probe-validation.sh`, **17 refusals** — because a guard nothing
exercises is a claim rather than a control.
+4
View File
@@ -51,6 +51,10 @@ spec:
pm-agent.homelab.keithjackson.dev/config-generation: "1"
spec:
serviceAccountName: pm-agent
# The kubelet's credential for pulling this image. Not the agent's Gitea credential —
# see pm-agent-registry.sops.yaml.example for why they are two Secrets of one token.
imagePullSecrets:
- name: pm-agent-registry
# The dispatcher DOES need its token: `kubectl create job` is its one privileged act,
# and platform/pm-agent/rbac.yaml is what bounds it to that. Contrast the worker, whose
# Job template sets this false because it needs no API access at all.
+6
View File
@@ -35,6 +35,12 @@ resources:
# the distance between naming a file in a comment and listing it under `resources:` is one
# careless edit.
- pm-agent-credentials.sops.yaml
# The KUBELET's pull credential for agent-runner — a separate Secret from the one above
# because it is a different consumer in a different format: dockerconfigjson, read before
# any container starts. Same underlying Gitea token, which must also carry `read:package`.
# Measured 2026-10-05: this registry refuses unauthenticated clients outright, so without
# this the pull 401s. Also placeholders until the operator fills it in.
- pm-agent-registry.sops.yaml
- deployment.yaml
# default-deny baseline + the two egress allow policies (gitea, kube-apiserver for the
# dispatcher only, tcp/443 for the model). NO ingress allow exists, by design.
@@ -0,0 +1,39 @@
# PLACEHOLDER VALUES — the kubelet cannot pull agent-runner until the operator replaces them.
# Full hand-off steps: pm-agent-registry.sops.yaml.example (same directory).
# SOPS-encrypted to the repository's age recipient; keys stay cleartext for reviewable diffs.
#
# This is the KUBELET's pull credential, not the agent's Gitea credential. The agent's is
# pm-agent-credentials (Opaque, read as env by the application); this one is
# dockerconfigjson and is read before any container starts. Same underlying Gitea token,
# which must carry read:package as well as write:repository and write:issue.
apiVersion: v1
kind: Secret
metadata:
name: pm-agent-registry
namespace: pm-agent
labels:
app.kubernetes.io/name: pm-agent
app.kubernetes.io/part-of: pm-agent
type: kubernetes.io/dockerconfigjson
stringData:
.dockerconfigjson: ENC[AES256_GCM,data:4OJS0P/oTcaXYuqlrc/936l+5/ZSqA9dlNMPx8HkDX9zmQ4q1qIumYmlXYMbcVm2c2byeYt0II0mGVLVVNOBjb5jgDB4vjyhSLN4HhoYBxgB+//hNcYe+uobmrdJTBJLEbd+lXamf1vfsEUTmLL7I4VQ2l/mjJbvqu8p2YNw25Ska9iO9mkxCzjOTfRdzMD0Pn1Z09fjo8GpYVo=,iv:AEysZLEqrybR+3l9kReHEeTbVFApzKPb8kAkibEYerE=,tag:TgE7T1zasye1bsTZ2BTFrA==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvdG1lQm16V0tJdWZXRXlU
Z2R4RXVyTXd6VTVJUWtkbUVjZldnb1JFOWhVCjVMMDdDb3AwVkt3TzJzdjR0VnNs
SFlncWg2cWlFTy9IZ1YxSC9Zd3ZMaEkKLS0tIEZFQWhIMW5zMURWYmlXaFg1ZjhD
cG1Fb0V3VncrTm9SQ091a0J0Z3RURWMKpdzNVub0zH5S8nnCSCt2H6NcCqDbV24L
Mf8K2eIfu/ixn2NfwoVVj2QO+n+3/P6PLphdoiKMrL+ms0+JqWFpkQ==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-10-06T06:01:43Z"
mac: ENC[AES256_GCM,data:zwmmMbM9Me7wE8EOvrKD6hEMLVHNCxI85FHvMjtRcHHm+mtdACjofDNfpynFVc1xFMMchcoOfipxEfAAJcwbH0FtfLeZq6M+XNHAf81qIk3NJxrS5Kgi/9FXZJAAwhqo/eced89Uozq8ntwt+mgxOuyjMds5z4h/VfN7/RmsWHg=,iv:VlvAEqozM9VV50vVx7r5aq3BdaaLLubhHJyHt8SSgS0=,tag:4ji7UqfRfm9P+qJrezFpmw==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
@@ -0,0 +1,88 @@
# =============================================================================
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (`.yaml.example` is ignored by kustomize and is
# never auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
#
# WHAT THIS IS: the credential the KUBELET uses to pull `agent-runner` from the in-cluster
# Gitea registry. It is NOT the credential the agent uses to talk to Gitea — that one is
# `pm-agent-credentials` and it is read by the application. Two Secrets, because they are
# consumed by two different things in two different formats:
#
# pm-agent-credentials Opaque read by pm-dispatch / pm-worker as env vars
# pm-agent-registry dockerconfigjson read by the KUBELET, before any container runs
#
# WHY A PULL CREDENTIAL IS NEEDED AT ALL. Measured 2026-10-05: an unauthenticated client
# gets nothing from this Gitea's registry — it issues an anonymous token whose `access`
# claim is `null`, and every path answers 401 whether it exists or not
# (`/api/v1/packages/Home-Network` replies `{"message":"token is required"}`). So a pull
# with no credential cannot succeed, and before this Secret there was no `imagePullSecret`
# anywhere in the tree. See decisions/2026-10-05-pm-agent-registry-pull-prereq.md.
#
# THE OTHER HALF, ALREADY FIXED: the nodes used to resolve this registry's hostname to the
# non-hairpinned WAN address and fail TLS against the router's `CN=%D` placeholder
# certificate. That is the `/etc/hosts` pin in
# infra/ansible/inventory/group_vars/k3s_prod/vars.yml
# (decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md). Both halves are needed; a
# pull fails on whichever is missing.
#
# =============================================================================
# HAND-OFF STEPS (operator)
# =============================================================================
#
# (1) USE THE SAME `pm-agent` ACCOUNT the agent already has (created per
# pm-agent-credentials.sops.yaml.example — the dedicated, NON-ADMIN one). A second
# account would be a second thing to keep non-admin for no gain: this token reads one
# package and nothing else.
#
# (2) ADD THE PACKAGE SCOPE. Gitea token scopes are fixed at creation, so this means
# generating a token with all three scopes and using it in BOTH Secrets:
#
# read:package <- NEW, what the kubelet needs to pull
# write:repository <- the agent's clone / branch push / pull-request create
# write:issue <- the agent's issue comments
#
# Gitea -> (as pm-agent) Settings -> Applications -> Generate New Token.
# Put the SAME value in `pm-agent-credentials` (`gitea-token`) and in the
# `.dockerconfigjson` below. One token, two consumers, two formats — the same
# one-value-in-two-files pattern the Keycloak client secret already uses here, and for
# the same reason: a Secret cannot be shared across formats.
#
# (3) BUILD THE DOCKERCONFIGJSON. `auth` is base64 of `<username>:<token>` — NOT a hash,
# just encoding, so treat this file exactly as secret as the token itself:
#
# USER=pm-agent
# TOKEN='<the token from step 2>'
# AUTH=$(printf '%s:%s' "$USER" "$TOKEN" | base64 -w0)
# cat > /tmp/dockerconfig.json <<JSON
# {"auths":{"gitea.home.keithjackson.dev":{"username":"$USER","password":"$TOKEN","auth":"$AUTH"}}}
# JSON
#
# THE REGISTRY KEY MUST BE THE HOST AS THE IMAGE REFERENCE SPELLS IT —
# `gitea.home.keithjackson.dev`, no scheme, no port, no path. The kubelet matches the
# pull host against these keys literally; `https://…` or a trailing `/v2/` silently
# matches nothing and the pull is attempted anonymously, which 401s exactly as if this
# Secret were absent.
#
# (4) COPY, FILL, ENCRYPT, COMMIT:
#
# cp platform/pm-agent/pm-agent-registry.sops.yaml.example \
# platform/pm-agent/pm-agent-registry.sops.yaml
# # paste the one-line JSON from step 3 as the .dockerconfigjson value
# sops -e -i platform/pm-agent/pm-agent-registry.sops.yaml
#
# The root `.sops.yaml` rule matches this path and encrypts `stringData`, so the diff
# stays reviewable. NEVER `sops -d` it into a terminal or a transcript (CLAUDE.md §6).
# It is already listed under `resources:` in platform/pm-agent/kustomization.yaml.
#
# (5) ROTATION. Rotating the Gitea token means updating BOTH Secrets, because it is one
# token. The dispatcher must then be restarted to re-read its env; worker Jobs and the
# kubelet pick the new value up on their own.
# =============================================================================
apiVersion: v1
kind: Secret
metadata:
name: pm-agent-registry
namespace: pm-agent
type: kubernetes.io/dockerconfigjson
stringData:
# One line. Registry key = the host exactly as the image reference spells it.
.dockerconfigjson: '{"auths":{"gitea.home.keithjackson.dev":{"username":"pm-agent","password":"REPLACE-ME-the-same-gitea-token","auth":"REPLACE-ME-base64-of-username:token"}}}'
+9 -34
View File
@@ -13,8 +13,8 @@ egress out the house WAN.
| Transport | WireGuard, **udp/51820**, kernel implementation |
| Where it runs | `hostNetwork` **DaemonSet** — one warm listener on every node |
| Client subnet | `${WG_CLIENT_CIDR}` = `10.90.0.0/24`; `.1` is the server, `.2`–`.254` devices |
| Authorisation | a device's **public key** is in [`peers.conf`](./peers.conf), or it cannot connect |
| The one secret | the server's private key — **SOPS-encrypted in this tree**, `wg-server-key.sops.yaml` |
| Authorisation | a device's **public key** is in [`peers.yaml`](./peers.yaml), or it cannot connect |
| The one secret | the server's private key, Secret `wg-server-key`, **created out of band** |
| Exposure | one hand-made WAN port-forward on the SOHO router, off-repo |
| Enrolling a device | [`infra/vpn-ingress/new-peer.sh`](../../infra/vpn-ingress/new-peer.sh) |
| Procedure / bring-up | [`runbooks/vpn-remote-access.md`](../../runbooks/vpn-remote-access.md) |
@@ -129,18 +129,12 @@ translations win here.
| half | where | why |
|---|---|---|
| peer **public** keys + addresses | [`peers.conf`](./peers.conf), **plaintext in git**, as a content-hashed `configMapGenerator` so an edit rolls the pods | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
| the server's **private** key | `wg-server-key.sops.yaml`, **SOPS-encrypted in git** | the only real secret (R-SEC-01, spec docs/07). Ships with a `REPLACE-ME` placeholder: an agent must not mint a long-lived private key (CLAUDE.md §6), so the operator `sops set`s it in |
| peer **public** keys + addresses | [`peers.yaml`](./peers.yaml), **plaintext in git** | a public key grants nothing by itself, and this makes *"who can get into my house"* a reviewable PR diff with `git log` as its audit trail |
| the server's **private** key | Secret `wg-server-key`, **out of band** | the only real secret; an agent must not mint a long-lived private key (CLAUDE.md §6) |
The two are joined only at pod start, by the init container, into a **`medium: Memory` emptyDir** —
so the assembled config holding the private key never touches any node's disk and dies with the pod.
Both halves are therefore in git, which is the point: a cluster rebuild restores the server key with
everything else. An out-of-band `kubectl create secret` — the first shape this component shipped in,
corrected after review — would have put the one credential every enrolled device depends on *outside*
the GitOps tree, and losing it means re-enrolling every device, because each client profile pins this
key's public half.
Encrypting the peer register would have hidden exactly the change a reviewer most needs to see: a
new line granting a new device access to the whole LAN.
@@ -166,40 +160,21 @@ NetworkPolicy applies.
## Operating it
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
stanza into `peers.conf` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
* **Revoke a device:** delete its stanza from `peers.conf` → PR → merge. No other step; the key
stops working as soon as the pods roll (the generator's content hash guarantees they do).
stanza into `peers.yaml` → PR → merge. Phones scan a QR code; desktops import a `.conf`.
* **Revoke a device:** delete its stanza → PR → merge. No other step; the key stops working.
* **Is it up?** `kubectl --context k3s-prod -n vpn-ingress get ds wireguard` — three Ready.
* **Who is connected?** `kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show`
(`latest handshake` under ~3 min means live). Never dump `wg showconf` — it prints the private key.
* **Full bring-up and verification:** `runbooks/vpn-remote-access.md`.
## Measured on prod, 2026-10-06 — what is now known rather than assumed
The first real reconcile settled four things this component's design rested on. Three went the
right way; one was a defect in my own manifest.
| question | answer |
|---|---|
| Does the **kernel module** reach the pod? | **Yes.** `ip link add dev wg0 type wireguard` succeeds, and `/proc/modules` lists `wireguard`, after the `node_baseline` run. `CAP_SYS_MODULE` is correctly not needed. |
| Does `allowPrivilegeEscalation: false` break the image's **s6 init**? | **No.** The container reaches `wg-quick` normally. The flag stays. |
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
## Prerequisites outside this directory
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
(`infra/ansible/roles/node_baseline/defaults/main.yml`); the operator runs the role. The pod has
`NET_ADMIN` but deliberately **not** `SYS_MODULE`, so it cannot load the module itself — it
CrashLoops with a clear `wg-quick` error instead of quietly degrading to userspace WireGuard.
2. **The server private key.** `wg-server-key.sops.yaml` is committed, encrypted, with a
`REPLACE-ME` placeholder, and both overlays already carry its decryption patch — so the wiring
is complete and the only outstanding act is replacing the value (runbook step 2). Until then the
DaemonSet CrashLoops with an explicit FATAL naming the file: the init container validates the
value with `wg pubkey`, which rejects the placeholder, an empty value, a public key pasted by
mistake, **and** a Secret that arrived as undecrypted `ENC[...]` ciphertext. That last case is
the one worth having a guard for — a missing overlay decryption patch applies cleanly and leaves
every Flux condition green.
2. **The `wg-server-key` Secret.** Runbook step 1. Until it exists the DaemonSet cannot start —
the correct fail-closed state for an unenrolled VPN.
3. **The WAN port-forward**, udp/51820 → one node address. Hand-made on the SOHO router, off-repo,
and the half CI can never check.
@@ -212,7 +187,7 @@ right way; one was a defect in my own manifest.
covered generically by `KubeDeploymentReplicasMismatch`/`KubePodNotReady` from
kube-prometheus-stack. **This is the top follow-up.**
* **No preshared keys.** They would add a symmetric post-quantum hedge on top of the Noise
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
handshake — and would be a real secret, forcing `peers.yaml` into SOPS and costing the
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
`wg-server-key` Secret.
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
+17 -94
View File
@@ -37,10 +37,6 @@
# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
# required `--allowed-unsafe-sysctls` on all three kubelets (a node-restart change).
# COROLLARY LEARNED THE HARD WAY (prod, 2026-10-06): this pod must not merely avoid NEEDING
# the sysctl, it must not ATTEMPT it. hostNetwork implies /proc/sys is mounted read-only, so
# `sysctl -w` fails with EPERM regardless of capabilities, and wg-quick treats a failed PostUp
# as fatal and deletes the interface. Verify the value; never write it.
# * no LoadBalancer VIP, so no CiliumL2AnnouncementPolicy lease lottery (the defect that cost
# this LAN an 8-day Minecraft outage) and no `externalTrafficPolicy` trade-off at all.
# * no double encapsulation. WireGuard's UDP lands on the node's own NIC, so the underlay is a
@@ -108,7 +104,7 @@ spec:
dnsPolicy: ClusterFirstWithHostNet
automountServiceAccountToken: false
# ── Assemble the running config: PUBLIC peer register (git) + PRIVATE server key (Secret) ──
# The two halves are deliberately stored apart (peers.conf explains why) and are only ever
# The two halves are deliberately stored apart (peers.yaml explains why) and are only ever
# joined here, at pod start, into a tmpfs that is destroyed with the pod. The server's private
# key is therefore never written to any node's disk, never committed, and never logged.
initContainers:
@@ -140,51 +136,9 @@ spec:
# The key arrives as a Secret file; strip any trailing newline the operator's shell
# added, or wg-quick rejects the config with "Key is not the correct length or format".
key=$(tr -d ' \r\n' < /run/wg-server-key/privatekey)
# VALIDATE, don't just check for empty. The Secret ships committed with a REPLACE-ME
# placeholder so the GitOps wiring is complete before the key exists, so the common
# failure is a value that is present and WRONG — a placeholder, a public key pasted by
# mistake, or a SOPS document that reached the cluster undecrypted (ENC[...], which
# happens if an overlay is missing its `vpn-ingress` decryption patch). `wg pubkey`
# accepts a value only if it is a real 32-byte curve25519 private key, so it rejects
# all four. Without this the bad value would be written into wg0.conf and the pod would
# fail later and less legibly, or come up with a server nobody can handshake with.
if ! printf '%s' "$key" | wg pubkey >/dev/null 2>&1; then
echo "FATAL: the server private key in Secret 'wg-server-key' is not a valid" >&2
echo " WireGuard private key (wg pubkey rejected it)." >&2
if [ -z "$key" ]; then
echo " It is EMPTY." >&2
elif [ "$key" = "REPLACE-ME" ]; then
echo " It is still the committed REPLACE-ME PLACEHOLDER — the key has not" >&2
echo " been generated yet. This is the expected state on a fresh install." >&2
else
# `case`, NOT a dollar-brace prefix-strip. This manifest is rendered through
# envsubst, which consumes every dollar-brace expansion it sees, so a shell
# parameter expansion here renders to the empty string and silently disables this
# very branch. `case` uses no braces, so there is nothing to escape.
# (ci: flux-postbuild-substitution caught both the original expansion AND the
# first version of this comment, which spelled the hazard out literally.)
case "$key" in
ENC\[*)
echo " It is still SOPS CIPHERTEXT (starts with ENC[). The Secret" >&2
echo " reached the cluster undecrypted — check that this environment's" >&2
echo " overlay carries the 'vpn-ingress' decryption patch." >&2
;;
esac
fi
echo " Fix: runbooks/vpn-remote-access.md step 2." >&2
exit 1
fi
# The pod shares the HOST netns, so this reads the node's real setting. Reading
# /proc/sys is permitted even though writing is not (see the PostUp note below).
# Fail closed: a tunnel that comes up but cannot forward would look healthy to every
# probe here and simply drop every client's traffic.
if ! grep -qx 1 /proc/sys/net/ipv4/ip_forward; then
echo "FATAL: net.ipv4.ip_forward is 0 on this node, so forwarded VPN traffic would" >&2
echo " be dropped. This pod CANNOT set it: with hostNetwork the runtime mounts" >&2
echo " /proc/sys read-only, because it is the host's global setting." >&2
echo " Fix on the NODE: run the node_baseline role (it writes" >&2
echo " /etc/sysctl.d/99-k3s-cilium.conf) — runbooks/vpn-remote-access.md step 1." >&2
if [ -z "$key" ]; then
echo "FATAL: /run/wg-server-key/privatekey is empty." >&2
echo " See runbooks/vpn-remote-access.md step 1 (create the server key)." >&2
exit 1
fi
@@ -201,20 +155,9 @@ spec:
echo 'MTU = 1420'
printf 'PrivateKey = %s\n' "$key"
# ── Forwarding + SNAT, applied in the HOST netns when wg0 comes up ──
# NO `sysctl -w net.ipv4.ip_forward=1` HERE. It is IMPOSSIBLE from this pod and it was
# the only thing stopping the tunnel coming up — MEASURED on prod 2026-10-06:
# [#] sysctl -q -w net.ipv4.ip_forward=1
# sysctl: permission denied on key "net.ipv4.ip_forward"
# [#] ip link delete dev wg0
# and wg-quick then tore the whole tunnel down. The cause is not a missing
# capability: with `hostNetwork: true` the runtime mounts /proc/sys READ-ONLY in the
# container (confirmed in /proc/mounts: `proc /proc/sys proc ro,...`), precisely
# because those sysctls are the HOST's globals and a container must not rewrite
# them. CAP_NET_ADMIN does not and cannot lift a read-only mount.
# It was never needed anyway: node_baseline writes net.ipv4.ip_forward=1 to
# /etc/sysctl.d/99-k3s-cilium.conf, and reading it from this pod returns 1. The
# init container now VERIFIES that (a read, which /proc/sys does allow) and fails
# closed with a pointed message, which is what the write was pretending to achieve.
# ip_forward is already 1 from node_baseline; re-asserted so this component does not
# silently depend on another role's sysctl file surviving.
echo 'PostUp = sysctl -q -w net.ipv4.ip_forward=1'
# SNAT client traffic to the node's address. REQUIRED: without it a LAN host would
# reply to 10.90.0.x, find no route, hand the packet to the router, and the router
# would drop it — so every connection would hang rather than fail. Matching on
@@ -274,10 +217,8 @@ spec:
# `kubectl get pod -o wide` shows what this listener owns.
- { name: wireguard, containerPort: 51820, hostPort: 51820, protocol: UDP }
securityContext:
# NET_ADMIN is the whole privilege budget: create the wg0 link and write the netfilter
# rules above. (It does NOT let the pod set a host sysctl — /proc/sys is read-only under
# hostNetwork — which is why nothing here tries.) Everything else is dropped, and the
# container is
# NET_ADMIN is the whole privilege budget: create the wg0 link, set the host sysctl, and
# write the two netfilter rules above. Everything else is dropped, and the container is
# NOT privileged — which also keeps it clear of restrict-host-namespaces' second rule.
# SYS_MODULE is deliberately NOT granted: the `wireguard` module is loaded by the node
# (infra/ansible/roles/node_baseline, node_baseline_kernel_modules), so this container
@@ -291,16 +232,6 @@ spec:
- NET_ADMIN
# readOnlyRootFilesystem intentionally UNSET: the image's s6 init writes under /run and
# /var during startup. Same deferral as crafty/headlamp.
#
# EXPECTED, HARMLESS STARTUP NOISE — do not "fix" it. Because capabilities are dropped
# to NET_ADMIN only, the image's s6 init cannot chown /config (no CAP_CHOWN) and logs a
# run of `chown: ... Operation not permitted` plus "Permissions could not be set. This
# is probably because your volume mounts are remote or read-only" and a scary "we will
# not provide support for it". It is wrong about the cause and it does not matter:
# /config is a writable tmpfs emptyDir, everything here runs as root (uid 0) anyway, and
# wg-quick proceeds normally — verified on prod 2026-10-06, where the tunnel's only
# failure was the sysctl above, well past these warnings. Granting CAP_CHOWN to silence
# log noise would widen the privilege set of an internet-facing pod for nothing.
resources:
requests:
cpu: 25m
@@ -341,22 +272,14 @@ spec:
# during startup rather than at a point anyone would connect to the volume size. Still
# counted against the container's 128Mi limit, and still far more than wg0.conf needs.
sizeLimit: 8Mi
# The ONLY secret in this component. SOPS-encrypted IN THIS TREE (R-SEC-01, spec docs/07) as
# wg-server-key.sops.yaml, delivered by Flux, referenced here by name only (R-IDP-04).
#
# It ships committed with a REPLACE-ME placeholder: an agent must not mint a long-lived
# private key (CLAUDE.md §6), so the operator generates it and `sops set`s it in
# (runbooks/vpn-remote-access.md step 2). Shipping the encrypted placeholder rather than no
# file at all is what lets the resource line and both overlay decryption patches be present
# and GREEN from the first commit — the alternative, a commented-out resource to uncomment
# later, is exactly the "file in git with no generator entry: reviewed, merged, apparently
# done, still broken" shape this repo keeps getting bitten by.
#
# DR: in git means a cluster rebuild restores it with everything else — the property an
# out-of-band `kubectl create secret` would NOT have had, and losing this key means
# re-enrolling every device because each client profile pins its public half. Velero also
# captures this namespace (its `daily` schedule excludes only `monitoring` and `unifi`).
# Keep a password-manager copy too: that is the one that survives losing cluster AND backups.
# The ONLY secret in this component. Created OUT OF BAND by the operator and referenced by
# name (R-IDP-04) — runbooks/vpn-remote-access.md step 1. It is NOT a *.sops.yaml in this
# tree on purpose: an agent must not mint a long-lived private key (CLAUDE.md §6), and a key
# that passed through a session transcript would have to be treated as compromised.
# DR: Velero's `daily` schedule excludes only `monitoring` and `unifi`, so this namespace
# and this Secret ARE captured off-cluster (platform/velero/helmrelease.yaml). Keep a copy
# in the operator's password manager as well — restoring it is what makes every already
# enrolled client work again without re-enrolling.
- name: wg-server-key
secret:
secretName: wg-server-key
+13 -52
View File
@@ -4,66 +4,27 @@
# same placement as crafty/headlamp/homepage/runbooks, for the same reason: a leaf whose pod can be
# legitimately unhealthy must never sit in the gating `platform` tier, where one component's
# unhealth stalls platform-config and every apps wave behind it (DECISIONS-LOG: "Headlamp gating
# stall"). That rationale is unusually concrete here: until the operator replaces the `REPLACE-ME`
# placeholder in wg-server-key.sops.yaml (runbooks/vpn-remote-access.md step 2) this DaemonSet
# CANNOT start, so putting it in the gating tier would have taken the whole platform down on merge.
# stall"). That rationale is unusually concrete here: until the operator creates the `wg-server-key`
# Secret (runbooks/vpn-remote-access.md step 1) this DaemonSet CANNOT start, so putting it in the
# gating tier would have taken the whole platform down on merge.
#
# NOT in platform/kustomization.yaml — see clusters/base/vpn-ingress.yaml.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# REQUIRED for the configMapGenerator below to be wired up at all, not merely tidy. A generated
# ConfigMap inherits NO namespace, every other object here declares `vpn-ingress`, and kustomize's
# nameReference transformer will not rewrite a reference across a namespace boundary — so without
# this the DaemonSet keeps mounting the unhashed name `wg-peers`, which does not exist, and the
# pods wedge on a missing volume. Verified by building it both ways (kustomize v5.8.1).
# Same placement as runbooks/kustomization.yaml, which uses `namespace: runbooks` for its own
# content generator.
namespace: vpn-ingress
resources:
- namespace.yaml
# The peer register: which devices may dial in. PLAINTEXT and reviewable on purpose (see the file).
- peers.yaml
# The listener itself: hostNetwork DaemonSet, udp/51820 on every node.
- daemonset.yaml
# Namespace default-deny baseline. NOTE: does NOT constrain the hostNetwork DaemonSet — the file
# says so at length rather than implying a protection it cannot provide.
- networkpolicy.yaml # The server's WireGuard private key — the ONE secret here. SOPS-encrypted (R-SEC-01, spec
# docs/07), decrypted per-env by the `vpn-ingress` Kustomization, referenced by name only from
# daemonset.yaml (R-IDP-04). Its plaintext `.example` sibling carries the operator hand-off --
# deliberately NOT named here: conformance/tests/l0-static/no-plaintext-secrets fails any
# kustomization that mentions an unencrypted template, because naming one is how a plaintext
# Secret gets applied by accident. (It matches comments too. That is correct, not overzealous.)
#
# SHIPS WITH A `REPLACE-ME` PLACEHOLDER, already encrypted. An agent must not
# mint a long-lived private key (CLAUDE.md §6), and the init container validates the value with
# `wg pubkey`, so the unfilled state is a loud CrashLoop naming this file, never a server that
# quietly cannot handshake.
- wg-server-key.sops.yaml
- networkpolicy.yaml
#
# THIS COMPONENT CARRIES AN ENCRYPTED SECRET, so BOTH overlays must patch the `vpn-ingress`
# Kustomization with a `decryption:` stanza — clusters/dev and clusters/prod both do, added by the
# same change as this file. That patch is as load-bearing as the file itself: without it kustomize
# hands Flux the raw ciphertext, the Secret APPLIES CLEANLY full of ENC[...], and nothing in any
# Flux condition looks wrong. The init container's `wg pubkey` guard exists partly to turn exactly
# that into a named error rather than a silently non-working tunnel.
# ── THE PEER REGISTER, as a CONTENT-HASHED ConfigMap ─────────────────────────────────────────
# peers.conf (plaintext, reviewable — see the file) becomes the `wg-peers` ConfigMap. It is a
# GENERATOR, not a plain resource, and that distinction is load-bearing rather than stylistic:
# kustomize appends a hash of the content to the generated name and rewrites the DaemonSet's
# volume reference to match, so adding or revoking a peer changes the pod template and ROLLS THE
# PODS automatically.
#
# A fixed-name ConfigMap (which is what this component shipped first) does not do that. Flux
# updates it in place, the pod template is byte-identical, no rollout is triggered — and because
# the init container assembles wg0.conf ONCE at pod start into a tmpfs, the running tunnel keeps
# the old peer list indefinitely. Measured on prod 2026-10-06: a merged peer reached the
# ConfigMap while `wg show` listed no peers and the DaemonSet still read generation=1, having
# never rolled. The runbook's "PR -> merge -> Flux rolls the DaemonSet" was false until this.
#
# DO NOT set `disableNameSuffixHash` here. The hash IS the mechanism.
configMapGenerator:
- name: wg-peers
files:
- peers.conf
options:
labels:
app.kubernetes.io/name: vpn-ingress
# DELIBERATELY ABSENT: a Secret. The server's private key (`wg-server-key`) is created out of band
# by the operator and referenced BY NAME from daemonset.yaml. There is no `*.sops.yaml` here, so
# this component needs NO SOPS decryption stanza in clusters/dev or clusters/prod — unlike
# crafty/headlamp/velero. If that key is ever migrated into SOPS, BOTH halves must land together:
# the file here, and a `decryption:` patch naming the `vpn-ingress` Kustomization in each overlay.
# Without the patch, kustomize hands Flux the raw ciphertext and the pod mounts a key full of
# ENC[...] — which applies cleanly and fails only at handshake time.
+1 -1
View File
@@ -19,7 +19,7 @@
# protects this port, and it would be enforcing nothing. So there isn't one.
#
# WHAT ACTUALLY PROTECTS THE EXPOSED PORT, then, since the CNI does not:
# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.conf gets
# 1. WireGuard's cryptographic peer authentication. A packet whose key is not in peers.yaml gets
# NO REPLY — see README "Why this is safe to expose". This is the real control.
# 2. The SOHO router's forward: only udp/51820 is forwarded inbound, to one node.
# 3. The blast radius once authenticated is deliberately the whole LAN — that IS the requirement
@@ -1,16 +1,4 @@
# THIS FILE IS THE MOUNTED CONTENT. It is turned into the `wg-peers` ConfigMap by
# `configMapGenerator` in kustomization.yaml, whose generated name carries a CONTENT HASH — so
# editing this file changes the ConfigMap name, which changes the DaemonSet's pod template, which
# ROLLS THE PODS. That is the whole reason it is a generator and not an inline ConfigMap manifest:
# a fixed-name ConfigMap updates in place, the pod template never changes, no rollout happens, and
# the init container's wg0.conf (built once at pod start, in tmpfs) keeps the OLD peer list
# forever. Measured 2026-10-06: a merged peer sat in the ConfigMap while `wg show` listed none and
# the DaemonSet reported generation=1 — it had never rolled. Same mechanism and same fix as
# runbooks/kustomization.yaml.
#
# Every line here is WireGuard config. `#` is a legal comment in it, so the documentation below
# travels inside the file that is actually mounted, rather than in a wrapper that gets stripped.
#
---
# ============================================================================================
# THE PEER REGISTER — every device allowed to dial in, in PLAINTEXT, in git, on purpose.
# ============================================================================================
@@ -53,40 +41,33 @@
# the Noise handshake and would be a genuine secret — which would force this register into SOPS
# and cost the reviewability above. Revisit only if the threat model changes; if it does, the PSK
# belongs in the `wg-server-key` Secret, not here. See README "Hardening not taken".
#
# name: wg-peers
# namespace: vpn-ingress
# labels:
# app.kubernetes.io/name: vpn-ingress
# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
#
# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
# (`wg show` reports an interface with no peers).
#
# COPY THIS TEMPLATE, uncomment, one block per device:
#
# # <device-name> — <owner>, enrolled <YYYY-MM-DD>
# [Peer]
# PublicKey = <the public key new-peer.sh printed>
# AllowedIPs = 10.90.0.<octet>/32
#
# The placeholders above are deliberately NOT valid values. An earlier version of this template
# used a real-looking `10.90.0.2/32`, and new-peer.sh's duplicate-address check — which grepped
# the whole file — matched it, so `.2` could never be allocated and the script insisted it was
# already taken. The check now ignores comments; keeping the example unmatchable too means a
# future grep cannot be fooled either.
#
# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
# it an incident responder cannot tell which stanza to delete.
#
# No peers enrolled yet. See the template in this file's comments and
# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
# keith-phone — <owner>, enrolled 2026-10-06
[Peer]
PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
AllowedIPs = 10.90.0.2/32
apiVersion: v1
kind: ConfigMap
metadata:
name: wg-peers
namespace: vpn-ingress
labels:
app.kubernetes.io/name: vpn-ingress
data:
# Appended VERBATIM to the generated [Interface] block, in this order, at pod start.
#
# *** THERE ARE NO PEERS YET. THAT IS THE CORRECT INITIAL STATE, NOT AN OVERSIGHT. ***
# An agent must not mint key material (CLAUDE.md §6), so no device could be enrolled as part of
# authoring this component. A zero-peer WireGuard interface is valid and starts cleanly: the
# DaemonSet will run and listen, and every handshake will be refused because no key is trusted.
# The VPN is therefore INERT-BUT-HEALTHY until the first stanza lands — fail-closed, and visible
# (`wg show` reports an interface with no peers).
#
# COPY THIS TEMPLATE, uncomment, one block per device:
#
# # pixel-8 — Keith's phone, enrolled 2026-10-05
# [Peer]
# PublicKey = <the public key new-peer.sh printed>
# AllowedIPs = 10.90.0.2/32
#
# Keep the `# <name> — <owner>, enrolled <date>` comment line. It is the only place a key is
# tied to a human-recognisable device; `wg show` can only ever print the key itself, so without
# it an incident responder cannot tell which stanza to delete.
peers.conf: |
# No peers enrolled yet. See the template in this file's comments and
# ./infra/vpn-ingress/new-peer.sh. One [Peer] block per device.
@@ -1,34 +0,0 @@
# WireGuard SERVER private key for platform/vpn-ingress — SOPS-ENCRYPTED (R-SEC-01, spec docs/07).
# Keys stay cleartext for reviewable diffs; only `stringData` values are encrypted (.sops.yaml
# `encrypted_regex`). Plaintext template + operator hand-off: wg-server-key.sops.yaml.example.
apiVersion: v1
kind: Secret
metadata:
name: wg-server-key
namespace: vpn-ingress
type: Opaque
stringData:
#ENC[AES256_GCM,data:fUoPkfWvasisdTtwdhjeOVOlmdQkTl6Bp9YxYyji8g6k+ClATds3BnHJWOj2iplaGpT2XaNakq1+D4YyOi7QbFvNJui+e4qRpqcY8hWnazvZIHJYxjxfAfzW,iv:xtKd6oMZQumAQJAsNLHqJB+o0wdQJ+vL1kI1mc9jL7w=,tag:aImqLgbSmhsRI36/t5kpyg==,type:comment]
#ENC[AES256_GCM,data:npDsCW6rQleolIpur5gFkX2jSRrIQzBlGaIPs2Schv0YU3q8AanG39z4QqT3ASDmskV5YuH29Bo0ujEAgBf52YIjQCp58PDDETofrVKOJzhHMa7bV+HCxUFE,iv:7E7thyFBleyqZpqJRGApKvIq3ubCFOkshF/mxim3YPc=,tag:joIrhMe2I6kUnx1ouOPdZw==,type:comment]
#ENC[AES256_GCM,data:wHUFx09FgP/2+LTyw61EFbKA5UaqfjU7CeGk6Xc5av2y4r4AvQJRefDM5EGlahKecDFEFsf7fSQRFRHtUJqrQTtw+6tcj3ubSCPW4iJvhyJJv0xdb/0puRmcgj+2NfgF,iv:xgDOSDLqMQ1wcI9vrkKqfNMgIDFEusFnnO7Rzq/aOa4=,tag:m1m/fgjkBsG91c+3daqOpA==,type:comment]
privatekey: ENC[AES256_GCM,data:ML8CPNm4JwhQHUTUUxZO/I+9X15BCIGqOb1Uw9EmX5RTAsxQ9N97YvkYlbY=,iv:5ZUsFdMMmeGsy4fNbk6qbJkzweMVWXZ5g6E7NciAGUg=,tag:pAHv3ieLjoEyFbqbWXsVgg==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMays2OXY0VkpRc2FQQmhU
Q0owM3N0VkhYR0lYTUdNZWRqeTFIMjNHOVJ3CnJSMnc5NFEyU0IzWTFtUkw2YVFS
MU02VFo1TE14YVpZRzNhVjZwRTZwd0UKLS0tIDFaMWF4NUZjZmhRdXRuZFJYNkg3
SUUzN1FnMlI4bkkzbVNlMWhUVnJZODAKQ+Iy86CIv/Oj9E9j4E/0y5hDbcctRuqB
OR825KRPRaTdF/G2hudE1R1JbM4wGbPLMzv078R4ZXHDPGvDttSEkA==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-10-06T06:10:31Z"
mac: ENC[AES256_GCM,data:6wAGAwWL7jI/NEeswBnUiWoRty+bb5DEsGEAPXajKNNYCwtm5yuXBrcOI/q4w6H8FG+nqVAECkdpAOI3S9mjDfc2zlWAzfCJJdDic92RKPXOC0tOH/M1BBSGJ1fVdJGAZZT1yAVxmtYy9WohJpawv3kmsVv+JnD+TIqmZ1CJHXo=,iv:srs4o0olSwmraQQgmdPxb+4h/t6fHHeFKJ2z914kGgY=,tag:jjoIKhU8Tu9i5SJkvkfcCg==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
@@ -1,76 +0,0 @@
# =============================================================================
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
#
# WHAT THIS IS: the WireGuard SERVER's long-term private key for the remote-access VPN
# (platform/vpn-ingress). It is the ONE secret in the whole component. The init container in
# daemonset.yaml reads it at `/run/wg-server-key/privatekey` and joins it to the PLAINTEXT peer
# register (peers.conf) in a tmpfs, so the assembled wg0.conf never touches a node's disk.
#
# WHY IT IS IN GIT, SOPS-ENCRYPTED, AND NOT HAND-CREATED. R-SEC-01 / spec docs/07: secrets live in
# source, encrypted, and reach the cluster only through Flux. An out-of-band `kubectl create secret`
# would put the single credential every enrolled device depends on OUTSIDE the GitOps tree, where a
# cluster rebuild silently loses it — and losing it means RE-ENROLLING EVERY DEVICE, because each
# client profile pins this key's public half.
#
# *** THE REAL FILE IS COMMITTED WITH `privatekey: REPLACE-ME`, ALREADY ENCRYPTED. ***
# That is deliberate, so the wiring is complete and green before the key exists: the resource is
# already listed in kustomization.yaml and both overlays already carry the `vpn-ingress` decryption
# patch. Your job is ONLY to replace the value (step 3 below). Until you do, the DaemonSet
# CrashLoops with an explicit FATAL naming this file — the init container validates the value with
# `wg pubkey` before writing wg0.conf, so a placeholder fails CLOSED and LOUDLY rather than
# producing a server that silently cannot complete a handshake.
#
# ---------------------------------------------------------------------------------------------
# EVERYTHING RUNS IN THE DEV CONTAINER. `wg`, `sops` and `qrencode` are all baked into the image
# (infra/devcontainer/Containerfile) because that image IS the toolchain — CLAUDE.md §5, and the
# devcontainer README's "Don't install the toolchain on the host". One shell, one toolchain, so
# `wg genkey | sops set` is just a pipeline.
# ---------------------------------------------------------------------------------------------
#
# HAND-OFF STEPS (operator — an agent must not mint a long-lived private key, CLAUDE.md §6).
# Full procedure with verification: runbooks/vpn-remote-access.md step 2.
#
# 1. Generate the key STRAIGHT INTO this file. The plaintext never lands on disk and never
# reaches a terminal; `sops set` edits the already-encrypted document in place:
# ./infra/devcontainer/dev.sh run bash -c '
# cd /work
# umask 077
# sops set platform/vpn-ingress/wg-server-key.sops.yaml \
# "[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
# '
# (Or edit it: `./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml`.)
#
# 2. Record BOTH halves in your PASSWORD MANAGER. The encrypted file is in git and Velero captures
# this namespace to R2, so this is belt-and-braces — but it is the only copy that survives
# losing the cluster AND the backups, and the key cannot be regenerated without invalidating
# every client profile.
# # public half (safe to display; it is what every client profile carries)
# ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
# --extract "[\"stringData\"][\"privatekey\"]" \
# platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
#
# 3. Confirm the value is a real key without printing it — `wg pubkey` succeeds only on a valid
# 32-byte curve25519 private key, which is exactly the check the pod's init container runs:
# ./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d \
# --extract "[\"stringData\"][\"privatekey\"]" \
# platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null \
# && echo "valid WireGuard key"'
#
# 4. PR it. Nothing else to wire: the resource line and both overlay decryption patches already
# exist. Flux rolls the DaemonSet and the pods leave CrashLoop.
#
# NOTE infra/vpn-ingress/new-peer.sh reads THIS FILE to derive each client profile's
# `[Peer] PublicKey`, so there is no separate copy of the key to keep in step with anything.
# =============================================================================
apiVersion: v1
kind: Secret
metadata:
name: wg-server-key
namespace: vpn-ingress
type: Opaque
stringData:
# The server's WireGuard private key: `wg genkey` output — 44 chars of base64 ending in '='.
# NOT a passphrase, and NOT the public key. The init container validates it with `wg pubkey`
# before writing wg0.conf, so a placeholder or a mangled value fails closed with a clear message.
privatekey: REPLACE-ME
+1 -1
View File
@@ -69,7 +69,7 @@ Work through these in order; the first three are by far the most likely.
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
| `ImagePullBackOff`, digest `sha256:0000…` | the image has never been pushed; the committed digest is the deliberate placeholder | `infra/agent/build-image.sh`, operator pushes, then one PR updates the digest in **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) — but read the row below first |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `401 Unauthorized` once TLS works | the pull is unauthenticated. An anonymous client gets nothing from this Gitea's registry, and no `imagePullSecret` exists anywhere in the tree | add one (a `dockerconfigjson`, SOPS-encrypted like `pm-agent-credentials`, referenced from the Deployment **and** the committed worker Job template) or allow anonymous reads of the package in Gitea — a posture decision, so yours |
| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o jsonpath='{.data}' \| tr ',' '\n' \| cut -d'"' -f2` should list `.dockerconfigjson`. Then confirm the token carries `read:package` and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
| `PM_WORKER_IMAGE must be digest-pinned` | someone put a tag in the ConfigMap | use the digest from `podman push --digestfile` |
| `scope: ONLY issues labelled 'agent'` | `PM_REQUIRE_LABEL` is set; unlabelled issues are out of scope by configuration | label the issue, or clear the knob |
| `cycle: 2 worker(s) active` and nothing new | at `PM_MAX_ACTIVE_WORKERS` | wait, or raise it (RAM is the binding constraint: each worker's limit is 2Gi) |
+40 -116
View File
@@ -21,7 +21,7 @@ behaves like a machine on the LAN: full reach to `192.168.2.0/24`, the Gateway's
DNS, and internet egress out the house WAN.
Authorisation is one thing only: the device's **public key is in
`platform/vpn-ingress/peers.conf`**. An unlisted key gets no reply at all.
`platform/vpn-ingress/peers.yaml`**. An unlisted key gets no reply at all.
> **An enrolled device is as privileged as a machine sitting in the house** — including the three
> k3s nodes (`192.168.2.10/.11/.12`: apiserver 6443, kubelet 10250, SSH 22). That is what "reach
@@ -62,109 +62,53 @@ lsmod | grep '^wireguard'
grep -w wireguard /etc/modules-load.d/k3s-cilium-longhorn.conf
```
## Step 2 — fill in the server key (once)
## Step 2 — create the server key (once)
The key is **SOPS-encrypted in this tree** (R-SEC-01, spec docs/07), not created by hand:
`platform/vpn-ingress/wg-server-key.sops.yaml`. It is already committed — encrypted, with a
`REPLACE-ME` placeholder — and it is already listed in the component's `resources:` with a
`decryption:` patch in both overlays. **Nothing needs wiring. You are only replacing one value.**
Why in git rather than `kubectl create secret`: an out-of-band Secret puts the one credential every
enrolled device depends on outside the GitOps tree, where a cluster rebuild silently loses it — and
losing it means re-enrolling every device, because each client profile pins this key's public half.
Everything below runs **inside the dev container** — `wg`, `sops` and `qrencode` are all baked into
the image (CLAUDE.md §5: the toolchain lives only in the pinned container; don't install it on the
host). One shell, one toolchain.
**1. Generate the key straight into the encrypted file.** One pipeline: `wg genkey` makes it,
`sops set` writes it into the already-encrypted document in place. The plaintext never lands on
disk and never reaches a terminal.
This is the **only** secret in the component, and the one thing that must never be generated in an
agent session or pasted into a transcript. Generate it on your own machine:
```bash
./infra/devcontainer/dev.sh run bash -c '
cd /work
umask 077
sops set platform/vpn-ingress/wg-server-key.sops.yaml \
"[\"stringData\"][\"privatekey\"]" "\"$(wg genkey)\""
'
mkdir -p ~/.config/wireguard-home && chmod 700 ~/.config/wireguard-home
umask 077
wg genkey > ~/.config/wireguard-home/server.key # keep this; see "Back it up" below
```
**2. Record it in your PASSWORD MANAGER.** Derive both halves from the file you just wrote:
Load it into the cluster. **Do not echo it, do not pass it on an interactive command line that
lands in shell history, and never print the resulting Secret's `.data`** (CLAUDE.md §6):
```bash
# the PUBLIC half — safe to display, and what every client profile carries
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey'
# the PRIVATE half — paste into the password manager, then clear your scrollback
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
platform/vpn-ingress/wg-server-key.sops.yaml'
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress create secret generic wg-server-key \
--from-file=privatekey=/work/.cache/server.key'
```
The second command is the one place this key is deliberately shown. The encrypted file is in git and
Velero captures this namespace to R2, so the password-manager copy is belt-and-braces — but it is
the only copy that survives losing the cluster *and* the backups, and the key cannot be regenerated
without invalidating every client profile.
…where `/work/.cache/server.key` is a path you copied the key to (`.cache/` is not tracked; create
it if absent). Delete that copy as soon as the Secret exists: `shred -u .cache/server.key`.
**3. Confirm the value is a real key**, without printing it — if `wg pubkey` prints a key, the
value is valid; if it errors, it is still wrong:
> **KEEP THE `.key` SUFFIX.** That filename is what keeps the key out of git: `.gitignore` line 11
> is `*.key`, and it is the **only** rule covering this path — `.cache/` itself is NOT ignored
> (`git check-ignore -v .cache/serverkey.txt` matches nothing). So a copy named `server.pem`,
> `wg-server` or `privatekey` sitting in that directory is **committable**, and the gate's
> `no-plaintext-secrets` stage does not know a WireGuard private key when it sees one. Verify before
> you trust it: `git check-ignore -v .cache/<your-filename>` must print a rule, and
> `git status --short` must not list the file.
Confirm it exists **by key name only**:
```bash
./infra/devcontainer/dev.sh run bash -c 'cd /work && sops -d --extract "[\"stringData\"][\"privatekey\"]" \
platform/vpn-ingress/wg-server-key.sops.yaml | wg pubkey >/dev/null && echo "valid WireGuard key"'
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress get secret wg-server-key -o jsonpath="{.data}" | tr "," "\n" | cut -d: -f1'
# expected: {"privatekey -- a NAME. If you see base64 payload, you printed the key. Rotate it.
```
That is the same check the pod's init container runs, so a pass here predicts a pass there.
**Why out of band and not SOPS.** An agent must not mint a long-lived private key, and a key that
passed through a session transcript would have to be treated as compromised. The trade-off is that
this Secret is **not reconstructible from git** — so:
**4. PR it, and merge.** Flux rolls the DaemonSet and the pods leave CrashLoop.
If you would rather use an editor than `sops set`:
`./infra/devcontainer/dev.sh run sops platform/vpn-ingress/wg-server-key.sops.yaml` (replace
`REPLACE-ME`, save, exit — it re-encrypts on write).
### What the pod tells you if this went wrong
The init container validates the value with `wg pubkey` before writing `wg0.conf`, and names the
specific failure in its log — so you never get a server that silently cannot complete a handshake:
| pod log says | meaning |
|---|---|
| `still the committed REPLACE-ME PLACEHOLDER` | step 3 not done (the expected fresh-install state) |
| `It is EMPTY` | the value was blanked |
| `still SOPS CIPHERTEXT (starts with ENC[)` | the Secret reached the cluster undecrypted — this environment's overlay is missing its `vpn-ingress` decryption patch. **This is the dangerous one:** the Secret applies cleanly and every Flux condition stays green |
| `wg pubkey rejected it` with none of the above | a mangled value, or a *public* key pasted by mistake |
| `net.ipv4.ip_forward is 0 on this node` | the node_baseline role has not run here (step 1). The pod **cannot** set this itself — under `hostNetwork` the runtime mounts `/proc/sys` read-only, because it is the host's global setting |
| `0 peer(s)` after you merged a peer | the DaemonSet did not roll. Check its `generation` (above). Pre-2026-10-06 this was guaranteed, because the register was a fixed-name ConfigMap; if it recurs, something has reverted `peers.conf` to a plain resource instead of a `configMapGenerator` |
### What a healthy start looks like
Measured on prod 2026-10-06. The init container logs one line, naming the shape and never the key:
```
wg0.conf assembled: 0 peer(s), server 10.90.0.1/24, udp/51820.
```
then the `wireguard` container brings the tunnel up:
```
**** Found WG conf /config/wg_confs/wg0.conf, adding to list ****
**** Activating tunnel /config/wg_confs/wg0.conf ****
[#] ip link add dev wg0 type wireguard
[#] wg addconf wg0 /dev/fd/63
[#] ip -4 address add 10.90.0.1/24 dev wg0
[#] ip link set mtu 1420 up dev wg0
```
**Ignore the `chown: ... Operation not permitted` run before that**, and the image's own
"Permissions could not be set ... we will not provide support for it". Capabilities are dropped to
`NET_ADMIN` only, so its s6 init has no `CAP_CHOWN`; `/config` is a writable tmpfs, everything runs
as root, and `wg-quick` proceeds. It is wrong about the cause and it does not matter.
**A pod stuck `0/1 Running` with `Startup probe failed: Unable to access interface: No such device`
means `wg-quick` tore the tunnel down** — read the `wireguard` container log for the `[#]` line it
died on. That is the shape of a real failure; the probe is doing its job of refusing to call a
server without a `wg0` "ready".
**Back it up.** Keep `server.key` in your password manager. Velero's `daily` schedule excludes only
`monitoring` and `unifi` (`platform/velero/helmrelease.yaml`), so this namespace *is* captured
off-cluster to R2 as well. Restoring this one key is what makes every already-enrolled device work
again after a cluster rebuild — **losing it means re-enrolling every device.**
## Step 3 — forward udp/51820 on the router (once)
@@ -205,37 +149,17 @@ unauthenticated traffic either way. If the pod is Ready, the interface exists.
## Step 5 — enrol a device
```bash
# in the dev container, like everything else
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh pixel-8 2' # full tunnel
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh thinkpad 3 --split' # LAN only
./infra/vpn-ingress/new-peer.sh pixel-8 2 # full tunnel (LAN + WAN egress)
./infra/vpn-ingress/new-peer.sh thinkpad 3 --split # split tunnel (LAN only)
```
It derives the server's public key from `wg-server-key.sops.yaml` itself, so there is no local key
copy to keep in step with anything. Profiles land in `.cache/wg-peers/` — gitignored, and persistent
because the repo is the container's only read-write mount.
It generates the keypair **on your machine**, writes a `0600` client profile outside the checkout,
prints a QR code if `qrencode` is installed, and prints the `[Peer]` stanza to add. Then:
It generates the device's keypair, writes a `0600` client profile, prints a QR code, and prints the
`[Peer]` stanza to add. It refuses to write a profile anywhere git does not ignore.
Then:
1. Append the stanza to `platform/vpn-ingress/peers.conf`, keeping the
1. Paste the stanza into `platform/vpn-ingress/peers.yaml` under `data.peers.conf`, keeping the
`# <name> — <owner>, enrolled <date>` comment. `wg show` can only ever print the key, so that
comment is the only record of which stanza belongs to which device.
2. PR → merge. Flux rolls the DaemonSet (~1 min), because `peers.conf` is a
**content-hashed** `configMapGenerator`: editing it changes the ConfigMap's generated name,
which changes the pod template, which forces the roll. (It did **not** work this way at first —
a fixed-name ConfigMap updates in place, the pod template never changes, and the init container
keeps the old peer list forever. Measured on prod 2026-10-06.) Confirm it actually rolled:
```bash
./infra/devcontainer/dev.sh run bash -c \
'kubectl --context k3s-prod -n vpn-ingress get ds wireguard \
-o jsonpath="{.metadata.generation}{\"/\"}{.status.observedGeneration}{\"\n\"}"'
```
The generation must have **increased** since before the merge. If it is still `1`, the roll did
not happen and `wg show` will list no peer no matter how long you wait.
2. PR → merge. Flux rolls the DaemonSet (~1 min).
3. Get the profile onto the device (QR scan for phones; "Import tunnel from file" on
Windows/macOS; `/etc/wireguard/<name>.conf` + `wg-quick@<name>` on Linux).
4. **Delete the profile from your machine**: `shred -u <path>`. It is the only copy of that
@@ -275,7 +199,7 @@ if the router is not delivering the packets, and a `handshake` with no completed
## Revoking a device
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
Delete its `[Peer]` stanza from `peers.yaml`, PR, merge. Nothing else — the key stops working on the
next reconcile. **Never re-use that address octet**: an old profile would silently start working
again as a different device.