.gitea/workflows/renovate.yaml shipped with `cron: "15 3 * * *"`. Gitea Actions
cron is UTC, and renovate.json carries schedule ["after 3am and before 6am"]
with timezone America/New_York -- and THAT inner schedule is authoritative:
Renovate re-checks it and declines to open PRs outside it.
03:15 UTC -> 23:15 EDT, the PREVIOUS EVENING in New York
window -> 03:00-06:00 local => NOT INSIDE (3h45m early)
The original comment claimed the cron sat "slightly ahead" of the window. It
sat 3h45m behind it, on the previous day. Every nightly run would have started,
logged "not within schedule", opened ZERO PRs and exited SUCCESSFULLY -- a green
tick forever, indistinguishable from "nothing needed updating", hiding the ~30
pending updates PR #73 existed to surface.
Changed to `15 8 * * *`, chosen to survive DST rather than to look tidy:
EDT (UTC-4): local 03:00-06:00 == 07:00-10:00 UTC
EST (UTC-5): local 03:00-06:00 == 08:00-11:00 UTC
08:15 UTC is inside BOTH (04:15 EDT / 03:15 EST), verified by computation at
both extremes, so neither switchover needs an edit. A cron cannot follow DST.
Also confirmed while here: RENOVATE_TOKEN now exists as a repo Actions secret,
so the workflow's fail-hard preflight will pass.
STILL UNRESOLVED and the likelier remaining bug: renovate.json's
`enabledManagers` lists bare `regex`, where modern Renovate names the custom
manager `custom.regex`. If the bare name is ignored, every inline `# renovate:`
marker in the tree does nothing. Deliberately NOT changed -- guessing at a key
that could fail validation and take the whole run down is worse than a question
one dry run answers. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster
DNS was a single pod on a single node, so draining or rebooting that node was a
CLUSTER-WIDE in-cluster DNS outage until the pod rescheduled -- every Service
lookup, every CNPG replica finding its primary, every operator addressing the
API by name. It was invisible because 1/1 Ready looks healthy.
It is urgent rather than latent now: the scheduled reboot automation on
feat/automated-upgrades drains and reboots one node at a time, unattended,
overnight -- turning a single point of failure into a recurring one.
WHY AN HPA AND NOT replicas: 2 -- the whole decision. k3s OWNS the coredns
Deployment: AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml,
managedFields managers deploy@k3s-prod-1 + k3s, objectset.rio.cattle.io/hash
label. k3s re-stages its bundled manifests on startup, so replicas: 2 written
there -- or a kubectl scale -- is reverted on the next restart or upgrade.
Rejected alternatives:
* coredns.yaml.skip + our own templated copy (the roles/kube_vip pattern).
Deterministic, but takes ownership of coredns from k3s permanently so k3s
upgrades stop updating it. Too high a price for a replica count.
* A Flux patch of the Deployment. Rejected outright: Flux and k3s's deploy
controller would both own one object and fight -- the default-deny
NetworkPolicy drift-loop shape (~2500 events/10min) this cluster already hit.
An HPA is a separate object k3s does not own, driving /scale rather than the
Deployment spec, so there is no shared-ownership fight; if k3s re-stages
replicas: 1 it is re-asserted within a reconcile. Disclosed cost: a brief
window at 1 replica after a k3s restart.
It is a FLOOR-HOLDER, not an autoscaler -- the CPU metric exists only so the
autoscaling/v2 object is valid. Measured: 5m/4m against a 100m request (~5%),
so the 80% target is never approached. scaleDown stabilization 600s because DNS
is the worst thing to flap; scaleUp immediate, since that restores redundancy.
The kustomization edit is ADDITIVE: platform/coredns/ already existed and holds
the coredns-custom split-horizon ConfigMap (R-DNS-01/03), which is untouched.
UNAPPLIED. The HPA has never run, and the claim that k3s reverts replicas on
restart is reasoned, not observed. After merge check `describe hpa coredns` for
ScalingActive=True -- an inert HPA looks identical to a working one because the
hand scale already left 2 replicas. See the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two halves of one repeatable upgrade process. Neither works alone.
(a) OS upgrades and reboots
platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own
Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml:
that tier gates platform-config and every apps wave, so a wedged reboot daemon there
could stall every application. A component added to make maintenance safer must not be
able to cause an outage.
infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other
open branches, and this is a different lifecycle), enabled for prod in group_vars.
The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT
REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's
rejection of automating away the gate between nodes: u-u is security-only with the
kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file
appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then
does only the mechanical half, including the ordering rule node_baseline has to shout.
Quorum: concurrent reboots are structurally impossible, not merely unlikely.
concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency),
and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node
holds it forever and nothing cascades.
Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait,
serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate
must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m
(unconditional) plus kured's Prometheus alert gate (semantic).
alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and
under the chart default kured would have been blocked on day one by the five alerts
measured firing with no end in sight, and never rebooted anything.
Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes
from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel
upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it.
Asserted against apt-config dump, not against the file just written. 6.12 kernels are
still installed and bootable on all three nodes, and nothing was held.
Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING
could create the sentinel. kured alone would have idled forever at 1/1 Running.
platform/kured/alerts.yaml — three rules over kured's only metric, with a companion
promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md.
Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this
cluster's 7d Prometheus retention.
(b) Image/dependency automation
Renovate was aspirational: renovate.json present and complete, with no workflow, no
in-cluster workload, and its docker+kubernetes managers disabled — so every plain
image: tag was maintained by nothing while seven files claimed otherwise.
.gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe
first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a
skipped Gitea job reports success and that is how this stayed inert.
Renovate now owns both halves; Flux image-automation is formally closed: it pushes
commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here
anyway — main is protected user_can_push:false and the flux-system GitRepository has
no secretRef. Its controllers are installed and have idled 43 days with zero CRs.
conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing;
tightens the check for this component (requires service.create+metrics.create, both
chart-default false, plus a companion NetworkPolicy requirement). Required because
assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is
shrink-only.
Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author
approving review; /approve-override deliberately NOT posted.
NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role
never executed, and whether CNPG switches its primary on a drain-induced eviction (the
most likely way the first real reboot fails). Full list in the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Adds the realm's WebAuthn Passwordless policy and restructures the browser-mfa
flow so a passkey is an ALTERNATIVE beside the password form. Additive and
optional by design: this IdP gates every SSO app plus kubectl, so a mandatory
passkey flow that misfires is a total lock-out. Enrolment is not forced.
Policy: RP ID is ${BASE_DOMAIN} (not the keycloak host) because a passkey's RP ID
is fixed at enrolment and must not be bound to a name that can move; resident key
required (a non-discoverable credential is a second factor, not a passkey); user
verification required (without it, possession alone is a complete login).
Flow: auth-username-password-form is split into auth-username-form +
auth-password-form, because the server must know who is logging in before it can
decide whether a passkey exists. Conditional OTP for platform-admins is relocated
intact onto the password branch only — a UV-required passkey is already two
phishing-resistant factors.
No requiredActions block is declared, deliberately: Keycloak 26.0.6 treats
requiredActions as REPLACE not merge, so declaring one would delete the default
set including CONFIGURE_TOTP and lock every platform-admin out of the OTP step
this flow still enforces. webauthn-register-passwordless is already registered
enabled-but-not-default by Keycloak's own defaults.
UNAPPLIED: the realm import is a one-shot resting at Done=True/Started=False, so
this reaches the live realm never. See the handoff entry and the README section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Exposed by the 2026-10-17 whole-cluster shutdown. oauth2-proxy performs OIDC
discovery ONCE at startup and exits 1 if it fails. On a cold boot Keycloak is
not serving yet, the shared Gateway answers 503, and the proxy dies:
[provider.go:55] Performing OIDC Discovery...
[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover
OIDC configuration: unexpected status "503": no healthy upstream
It then sits in exponential CrashLoopBackOff long after Keycloak is healthy --
one replica was still down minutes later and only a manual pod delete cleared
it. There is no retry flag, a readinessProbe cannot help (the container is
already dead), and CrashLoopBackOff backoff is not tunable per-pod.
Add an extraInitContainers gate (wait-for-idp) that polls the discovery doc
until it answers. A pod in Init accrues no CrashLoopBackOff backoff and retries
promptly. BOUNDED and FAILS OPEN: after ~5m it starts the proxy anyway, so the
worst case is exactly today's behaviour and never something new.
extraInitContainers is the chart's OWN hook for this -- its commented example is
literally `- name: wait-for-idp` -- and the key was CONFIRMED against the pinned
chart (helm show values, 10.7.1) rather than assumed, because a wrong Helm values
key is silently ignored. No new base image: the chart already runs an alpine init
container for its built-in waitForRedis gate; pinned to 3.22 (verified upstream)
rather than the chart's `latest`.
Rejected: Flux postRenderers (no precedent in this tree, and the chart has a
hook); --skip-oidc-discovery (removes the fetch entirely, but needs four
hardcoded endpoints and I could not verify lazy-JWKS offline -- this component
gates SSO for seven apps, so the additive option wins).
Shell vars are written BARE with no braced form: platform declares
postBuild.substituteFrom, and envsubst would blank a braced shell variable and
silently break the loop. Same convention as the gitea/nextcloud OIDC hooks.
UNAPPLIED; the init container has never run, and the race was not re-created to
prove the fix. Check the init container's log on first apply. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
All three nodes had to be powered off to move them onto a UPS, and there was
NO procedure: infra/cluster.sh stop is virsh-based and only ever applied to the
decommissioned dev cluster. The sequence was improvised live. Two defects came
out of doing it.
1. GRACEFUL NODE SHUTDOWN WAS UNCONFIGURED ON EVERY PROD NODE. No
shutdownGracePeriod anywhere in infra/ansible/, so `shutdown -h` stopped k3s,
containerd went with it, and every pod was SIGKILLed where it stood -- 9 CNPG
instances mid-checkpoint, the UniFi VM, 25 attached Longhorn volumes.
roles/k3s_server now sets kubelet-arg shutdown-grace-period=90s and
shutdown-grace-period-critical-pods=20s.
The half that silently defeats the other half: this is enforced by LOGIND,
not the kubelet. The kubelet holds an inhibitor DELAY lock and systemd waits
at most InhibitDelayMaxSec -- DEFAULT 5 SECONDS -- then proceeds regardless,
truncating the drain with no error. The role ships a logind.conf.d drop-in
(120) and ASSERTS cap >= grace period, because the failure is invisible.
2. SCALING A StatefulSet TO 0 DELETED ITS PVC AND VOLUME. monitoring/loki
carries persistentVolumeClaimRetentionPolicy whenScaled: Delete, so the
quiesce destroyed storage-loki-0 outright. Loki stores type: filesystem, so
that PVC was the only copy; 72h retention bounded the loss, the mechanism
does not. Symptom was the volume count going 25 -> 24, first misread as a
miscount. Repaired: scale back to the declared 1 recreated the PVC and
loki-0 is 2/2 Running.
New infra/maintenance/quiesce.sh (quiesce|resume|status) encodes the sequence
that worked and REFUSES to scale any whenScaled: Delete StatefulSet -- it stays
running, its volume hard-detaches, and it rebuilds on restart. A rebuild beats
a deletion. Discovery is computed from the live cluster, so a newly onboarded
service needs no edit here.
runbooks/cluster-maintenance-shutdown.md is the operator page: order in both
directions, TrueNAS-after-nodes (hard NFS mount wedges kubelet), the
un-hibernate-Postgres step Flux will NOT do for you, and a "looks broken but is
fine" section separating the 4/19 Flux cascade, the degraded-and-rebuilding
volume and the self-recovering keycloak-operator from the one thing that needs
a human (a stuck oauth2-proxy replica).
THE KUBELET CHANGE HAS NEVER BEEN APPLIED OR TESTED -- it needs a rolling k3s
restart, which is an operator action. Review config.yaml's kubelet-arg before
applying: a bad config.yaml is how k3s fails to come up. See the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Both ZFS pools moved off the Proxmox host onto a standalone TrueNAS on
192.168.2.30 -- prod's OWN /24, so the pfSense crossing that made this knob
load-bearing is gone, along with the DNAT at 192.168.2.102 it mounted through.
Measured 2026-10-07: 192.168.2.102:2049 no longer answers; .30:2049 and :111 do.
BULK_NFS_SERVER -> 192.168.2.30 in prod, base and dev. The dev value is marked
inert (dev is decommissioned); it carries the real address only so nothing in
the tree renders a host that no longer exists.
THE OPERATIONAL POINT, and why this is not just a value change: a bound
PersistentVolume's NFS source is immutable. Server dry-run, 2026-10-07:
spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is
immutable after creation
so Flux's apply FAILS rather than updating, and the pods keep their old, dead
mount. The five PVs must be deleted and recreated -- safe, because Retain never
touches data and render.py emits a pre-set claimRef so each PV rebinds its own
PVC. The sequence is docs/BULK-NFS-STORAGE.md section 10 and is an OPERATOR
action; this commit does not perform it.
Pairs with the hazard already on the page: spec.nfs.server is unvalidated at
create (any string binds) and immutable at update. A wrong value is accepted
silently and then cannot be corrected in place.
docs/BULK-NFS-STORAGE.md: sections 1 and 5 rewritten for TrueNAS -- the measured
exports, the service settings that decide whether uid 1000 survives the wire,
the three stale ZFS sharenfs properties that had to be cleared, and the ACL
hazard (acltype=nfsv4 + aclinherit=passthrough means one click in the ACL editor
propagates to the whole future library). Section 7 rewritten now that the
placeholder is gone, section 9 retargeted from dev to prod, section 10 new.
Sections 2/3/4/6 are unchanged by the move.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The operator enrolled a phone, merged the peer, and `wg show` listed none.
Measured on prod 2026-10-06 with the peer already on main:
live wg-peers ConfigMap, active AllowedIPs .. 0
DaemonSet volume reference ................. configMap.name: wg-peers (FIXED)
DaemonSet generation/observedGeneration .... 1 / 1 (never rolled)
init container, all three pods ............. "0 peer(s)"
Part of it was just the 10m reconcile interval. The defect is the fixed name:
Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is
byte-identical, so no rollout is triggered -- and the init container assembles
wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old
peer list indefinitely. Adding or revoking a peer had no effect until something
unrelated restarted the pods. For a REVOCATION that is a security defect, not
an inconvenience: delete a stanza, see a green PR, device keeps working.
The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet".
Never true. And this repo already documents the fix, in a file I had read --
runbooks/kustomization.yaml: "The generated name carries kustomize's content
hash, so editing any runbook rolls the server automatically and a stale
projection is impossible."
So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a
configMapGenerator. The operator's enrolled peer is carried across verbatim.
Every line is WireGuard config and `#` is a legal comment there, so the
documentation now lives inside the file that is actually mounted.
`namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a
generated ConfigMap inherits no namespace, everything else here declares
vpn-ingress, and kustomize's nameReference will not rewrite across a namespace
boundary -- without it the build mounts the unhashed name and the pods wedge on
a missing volume. Caught only by building the output and reading the volumes.
new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather
than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that
after the rename the only symptom would have been duplicate detection quietly
never firing again. Stale peers.yaml references swept from six files including
the script's own user-facing output.
Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content
(edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical
to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's
real allocation.
No sensitive paths; merges on a green gate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The first real enrolment failed:
$ ./infra/vpn-ingress/new-peer.sh keith-phone 2
new-peer.sh: 10.90.0.2/32 is already allocated ... Pick another octet.
Nothing was allocated -- peers.yaml had zero peers, as the pod's init
container reported (`0 peer(s)`). The duplicate-address check grepped the
whole file, and peers.yaml line 66 is a COMMENTED-OUT example stanza I wrote
containing literally `# AllowedIPs = 10.90.0.2/32`. So .2 -- the first
address anyone would ask for, and the one my own runbook uses as its example
-- was permanently un-allocatable, with a message asserting the opposite of
the truth.
Fixed on both sides, because either alone leaves the trap half-set:
* the check strips comment lines before matching, which is the correct
predicate rather than a patch: a real stanza's AllowedIPs line is never
commented, while the template and every peer's `# <name> — <owner>` label
always are.
* the example now uses placeholders that cannot match anything real
(`10.90.0.<octet>/32`, `<device-name>`), so a future grep cannot be fooled
by it either.
The bug class is worth naming: a documentation example that is byte-identical
to real data will eventually be parsed as real data. This is its third
instance in this component -- flux-postbuild-substitution flagged a `${...}`
written inside a comment explaining that hazard, and no-plaintext-secrets
flagged the .sops.yaml.example because a kustomization named it in a comment.
Make examples structurally unmatchable, not merely commented out.
Verified by running it: .2 now renders a profile (via the real
`sops -d | wg pubkey` path, which this also proves against the operator's
actual key for the first time), and an uncommented duplicate stanza is still
correctly refused.
No sensitive paths; merges on a green gate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The tunnel never came up on its first reconcile with a real key. One line of my
own manifest was the whole cause. Measured on prod 2026-10-06:
[#] ip link add dev wg0 type wireguard <- fine
[#] wg addconf wg0 /dev/fd/63 <- fine
[#] ip -4 address add 10.90.0.1/24 dev wg0 <- fine
[#] ip link set mtu 1420 up dev wg0 <- fine
[#] sysctl -q -w net.ipv4.ip_forward=1
sysctl: permission denied on key "net.ipv4.ip_forward"
[#] ip link delete dev wg0
**** Tunnel failed, will stop all others! ****
Not a missing capability: with hostNetwork the runtime mounts /proc/sys
READ-ONLY (confirmed in the pod's /proc/mounts), because those sysctls are the
HOST's globals. CAP_NET_ADMIN cannot lift a read-only mount, so the write can
never succeed here -- and wg-quick treats a failed PostUp as fatal, so it
deleted the interface it had just built.
The line was never needed. node_baseline already sets ip_forward=1, and reading
it from the pod returns 1. My header comment said exactly that ("so no pod
sysctl is needed") and then emitted the sysctl anyway, justified as "re-asserted
so this component does not silently depend on another role's sysctl file". That
defensive re-assertion is the entire outage. In a hostNetwork pod: VERIFY host
state, never write it.
Replaced with a read in the init container -- `grep -qx 1
/proc/sys/net/ipv4/ip_forward` -- failing closed with a message naming the
node_baseline role. Still fail-closed on purpose: a tunnel that comes up unable
to forward satisfies every probe here and silently drops all client traffic.
Also documents, in the manifest and the runbook, that the image's `chown:
Operation not permitted` run and its "we will not provide support for it"
warning are expected and harmless -- capabilities are dropped to NET_ADMIN so
its s6 init has no CAP_CHOWN, /config is a writable tmpfs, everything runs as
root, and wg-quick proceeds (it did; the failure was 30 lines later).
Three other unknowns closed by the same reconcile, now recorded in the README:
the kernel module reaches the pod (ip link add type wireguard works, no
CAP_SYS_MODULE needed); allowPrivilegeEscalation:false does NOT break the s6
init; and the image's iptables is nf_tables (xtables-nft-multi) matching Debian
13, which closes the worst documented fear -- rules landing in a table the
kernel never consults.
No sensitive paths; merges on a green gate with no override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator correction: "sop, wg, and any other tool requirement is supposed to be
baked into the devcontainer."
My previous commit discovered that `wg` existed only on the host, `sops` only in
the container, and `qrencode` on neither -- and fixed it the wrong way, by
splitting every step of the key procedure across the boundary and telling the
operator to `dnf install qrencode` on their workstation. That breaks CLAUDE.md §5
and the devcontainer README's "Don't install the toolchain on the host", produces
a procedure that cannot be reproduced in CI, and is miserable to follow.
wireguard-tools + qrencode are now distro packages in the Containerfile. Both are
in Fedora 44's repos (wireguard-tools 1.0.20260223 -- the same upstream version as
the image this component runs -- and qrencode 4.1.1), unpinned like every other
dnf package there: the pinned thing is the Fedora base, and neither is a gate tool
whose version changes what green means.
Three things got SIMPLER rather than merely relocated, which is the tell that the
rule was right:
* Step 2 is one pipeline in one shell -- `sops set ... "$(wg genkey)"` writes the
key straight into the already-encrypted document. No staged copy in .cache, no
`*.key`-suffix warning, no sha256 comparison to prove the right bytes landed.
* new-peer.sh derives the server public key from wg-server-key.sops.yaml itself
(`sops -d --extract | wg pubkey`), so there is no local key copy to keep in
step with the encrypted one, and no magic host path it must live at.
* `.cache/` is gitignored at the repo root (mirroring infra/ansible/.gitignore),
giving the container a persistent non-committable scratch dir. new-peer.sh
writes profiles to .cache/wg-peers/ -- necessary because the repo bind-mount is
the container's only persistent read-write mount, so a profile in /tmp there is
gone before it can reach a desktop. Its blanket "refuse to write inside the
checkout" guard is now a `git check-ignore` test of the real property.
Verified by execution, not reasoning: the image rebuilds; wg/qrencode/sops all
resolve in it; `sops -d | wg pubkey` runs in one shell; new-peer.sh refuses the
REPLACE-ME placeholder with an actionable message; and a full run with a throwaway
server key renders a correct profile at 0600 under a 0700 gitignored dir, prints
the QR, and refuses a non-ignored destination inside the repo.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Node configuration in this project is Ansible's — declared in the inventory, reviewed,
version-controlled — and for prod it is the ONLY surface: infra/PROD-PROVISIONING.md
records prod as 3x bare metal with the dev OpenTofu -> Proxmox -> cloud-init path
explicitly not applying, so there is no tofu/cloud-init half to keep in parity for an
/etc/hosts pin.
The pin was already in the right place; what the docs lacked was the command. All four
places now carry the scoped, idempotent one:
cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
-e k3s_hosts=k3s_prod --tags hosts
--tags hosts runs ONLY the /etc/hosts task, so applying this pin cannot drift any other
baseline setting.
Also verified the mechanism is not inert, which is the failure mode this repo keeps
meeting: roles/node_baseline/tasks/main.yml renders node_baseline_hosts_pins into a marked
blockinfile with backup: true, and state: absent when the list empties — so a removed
entry really does leave the node.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The pm-agent image pull failed TLS verification, and the previous entry on this branch
recorded it honestly as not diagnosed. Diagnosed now, and it is not about the image.
containerd pulls as a NODE process, so it resolves through node_baseline_dns_servers (the
LAN gateway, deliberately not PiHole) and gets the public answer: a CNAME to
wan.home.keithjackson.dev -> 136.51.115.252, measured via both 1.1.1.1 and 8.8.8.8. This
router does not hairpin, so WAN:443 answers with its own placeholder certificate —
measured with the right SNI from the workstation:
subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
No extensions in certificate
No extensions means no subjectAltName, which is exactly 'certificate is not valid for any
names'. Meanwhile 192.168.2.20 IS the shared Gateway's VIP and its https listener serves
the real Let's Encrypt *.home.keithjackson.dev. The Gateway, the wildcard cert and the
digest were never implicated — only the node's answer.
This file already documents the identical failure for the Keycloak issuer, measured from
k3s-prod-2/-3 on 2026-10-03 (curl exit 60, ssl_verify_result=18), and already carries the
mechanism that fixes it. gitea is simply the second node process to need it; containerd is
a node process and nobody had added it. An operator must run the node_baseline play.
Also records, where an operator reads them, that the agent authenticates to Gitea with a
local account and a scoped PAT by decision — and why the federated alternative is not
merely unchosen but unavailable: Gitea implements only the Authorization Code Grant and
validates no third-party OIDC token, so a Keycloak service-account token cannot
authenticate to its API or to git-over-HTTP.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator correction: secrets are kept in source through sops. #55 shipped the
WireGuard server private key as a Secret to `kubectl create` by hand. Two real
costs, not just a convention break:
* R-SEC-01 / spec docs/07 -- secrets live in source, SOPS-encrypted, and
reach the cluster only through Flux.
* It broke DR for the one credential that cannot be re-derived. A rebuild
restores everything in git; it does not restore a hand-made Secret, and
losing this key means re-enrolling EVERY device, because each client
profile pins its public half. #55 papered over that with "keep a copy in
your password manager".
(Note secret-provenance did NOT forbid the original shape -- it scopes to
app-namespace=true and vpn-ingress is deliberately a platform namespace. The
rule broken was the written requirement, not an enforced control.)
An agent still must not mint a long-lived private key (CLAUDE.md §6), so the
file is committed ALREADY ENCRYPTED with `privatekey: REPLACE-ME`. That makes
the wiring complete and green from the first commit -- resource line plus both
overlay decryption patches -- leaving the operator exactly one act: replace the
value. A commented-out resource to uncomment later is the "in git, no generator
entry, reviewed, merged, still broken" shape this repo keeps hitting.
The init guard is now a validation rather than an emptiness check: the value is
piped through `wg pubkey`, which accepts only a real curve25519 private key, and
the pod names which case it hit -- placeholder, empty, mangled, or (the silent
one) still `ENC[` ciphertext because an overlay is missing its decryption patch,
which otherwise applies cleanly with every Flux condition green.
Second, independent finding that also fixes#55's step 2: `wg` is on the HOST
only and `sops` is in the CONTAINER only, so `sops -d | wg pubkey` cannot run in
one shell -- the command #55 documented could not have worked. Every step is now
split by where it runs, and the written key is verified by comparing sha256
digests rather than printing anything. `qrencode` is on neither, so the QR step
says so and gives the install command.
Touches clusters/** and adds a **/*.sops.yaml, so approval-policy will block
pending the operator's /approve-override. Deliberately not posted by me.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
An in-cluster consumer reaches a service by its *.svc.cluster.local name, which is by
design a different path from how a client outside the cluster reaches the same service.
pm-worker's host allowlist accepted TWO values: the cluster-local name and the external
hostname gitea.home.keithjackson.dev. The second is removed.
It was never needed — nothing sets it — it would route in-cluster traffic out through the
Gateway and back for no reason, and as an ACCEPTED value it widened the allowlist to a
host reachable over the namespace's world:443 egress rule, which is precisely the
injection shape the allowlist exists to refuse.
probe-validation.sh now pins that: 20 refusals, including the external hostname, the
cluster-local name over https, and the cluster-local name on the wrong port.
Also corrects a remark in the handoff entry: I cited cluster-vars' GATEWAY_VIP
(192.168.1.53) as suspicious next to the nodes' 192.168.2.x. That was noise — those base
cluster-vars values are dev-era leftovers for a decommissioned cluster and say nothing
about prod. None of the measured facts depended on it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
pm-agent reconciled exactly as designed after #57: Flux applied main@sha1:95597bd to
platform-config, pm-agent and apps-wave-0 (all Ready), the ClusterPolicy
pm-agent-job-shape is live with Enforce + failurePolicy Fail and all six rules, SOPS
decryption works through the new overlay patch (checked by key names and value lengths,
never values), and pm-agent-worker really is bound to nothing.
Measuring it turned up two prerequisites for the NEXT operator step that nothing in this
repository declares, and that #57's handoff could not have known — it listed the image
pull as unverified, and that is exactly where it bit:
- the nodes cannot verify the registry's TLS certificate (containerd: 'x509: certificate
is not valid for any names'), while the same name serves a healthy Let's Encrypt
wildcard from the LAN. A correct digest fails identically;
- an unauthenticated client gets nothing from this Gitea's registry (its anonymous token
carries no access scope) and no imagePullSecret exists anywhere in the tree.
Not specific to pm-agent: conformance-runner shares the registry path and has never been
pulled either. The dispatcher is the first workload in the cluster ever to reference it.
Documentation only — no manifest, no policy, no code. Both fixes need an operator action
and one of them is a posture decision, so the docs present both rather than pick.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Both reported by the operator on first use of the runbook merged in #55.
Step 1 said `cd infra/ansible && ./site.sh --tags system --limit k3s_prod`.
There is no site.sh anywhere in this repo. It was wrong four ways at once:
this repo drives Ansible with `ansible-playbook <playbook>` and never a
wrapper (playbooks/site.yml is the full bring-up chain, not this); the module
work is in playbooks/node-baseline.yml; targeting here is `-e k3s_hosts=...`,
not `--limit`; and Ansible exists only in the pinned toolbox, with ansible.cfg
read only from infra/ansible. Replaced with the real command plus why
`-e k3s_hosts=k3s_prod` matters (the default group is decommissioned dev) and
why `--tags system` must never be dropped (an untagged run also executes the
`disks` tasks, which assert on prod's deliberate placeholder devices).
Step 2 told the operator to stage the server key at /work/.cache/server.key
"inside the repo's gitignored cache". .cache/ is NOT gitignored -- the file was
ignored only incidentally by `*.key` (.gitignore:11). So the path was safe for
a reason I had not identified, and renaming it (server.pem, wg-server) would
have left a committable WireGuard private key in the checkout, which
no-plaintext-secrets does not recognise. The step now names the real rule,
marks the suffix as the load-bearing part, and gives a check-ignore
verification to run before trusting it.
Docs only -- no manifest and no behaviour change. Touches no sensitive path.
Also records the post-merge measurements for #55, whose own entry predates the
merge: Flux applied ad3f56e by the real path, all 18 Kustomizations Ready,
hostNetwork confirmed (pod IPs are the node IPs), and the designed fail-closed
state (Init:0/1 on the missing wg-server-key Secret).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
I recorded that merging main in would dismiss the operator's approval and that
approval-policy would therefore go red until they re-approved. Measured: wrong.
dismiss_stale_approvals is true, so the review did go stale — but
ci/approval-policy.sh selects on .state == "APPROVED" and a non-author login and reads
nothing about staleness, and required_approvals is 0 so Gitea never required one either.
CI run #257 on the merged head: "approved by 'jakman85' (author 'gitea-admin') —
sensitive change is human-reviewed", approval-policy pass, gitea-settings pass, required
check green, PR mergeable.
The 'push again and you must override again' rule applies to the /approve-override
directive, which IS staleness-checked against the head commit — not to a review.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PR #56 added §0 (work in your own worktree, never the shared checkout) while this branch
was open. A worker agent reads CLAUDE.md first, and §0's instruction is to create a git
worktree outside the shared checkout — which does not exist in a worker's container. It
holds a fresh clone of its own, made for one assignment and thrown away with the pod,
which is what §0 is actually asking for.
A worker that obeyed §0 literally would 'git worktree add', edit somewhere the wrapper
that started it does not look, and have its work committed by nobody while the issue got
a 'changed nothing' report. worker-rules.md now says §0 is satisfied and not to do that.
Also records in the handoff entry: the merge and its one conflict, and that this push
dismisses the operator's approving review — measured, dismiss_stale_approvals=true — so
the approval-policy stage needs them to re-approve. That is theirs to do, not an agent's.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
One conflict, in clusters/base/kustomization.yaml: PR #55 (vpn-ingress) and this branch
both appended a resource line to the same tail. Resolved by keeping BOTH, with main's
vpn-ingress first — it is a platform-config-tier leaf, while pm-agent dependsOn
apps-wave-0, which is later.
Nothing else conflicted. runbooks/kustomization.yaml auto-merged and carries both the
vpn-remote-access and pm-agent entries; the handoff entries are separate files and had
nothing to merge, which is the whole point of decisions/ being a directory.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The PR gate ran and BLOCKED on approval-policy alone — the sensitive-path rule, with no
non-author review — while every technical stage passed, gitea-settings included (the one
stage a workstation cannot run). That is the gate working, and it is the one thing an
agent must not clear for itself.
Also recorded: the commit-status API reported 'pending — Waiting to run' for a run that
had already completed, so the log is the source of truth here. §2b again.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
pm-worker's command is pinned by admission with no args, so everything a caller could
influence arrives in the environment — and the guards at the top of that script are what
stand between an injected variable and a clone from the wrong host or a push to the
wrong ref. Nothing exercised them, which makes them a claim rather than a control.
probe-validation.sh asserts 17 refusals, each on BOTH a non-zero exit and the specific
message, so a guard that stopped firing cannot pass by failing somewhere else. It covers
a redirected git host, a real and reachable WRONG one (the egress policy permits tcp/443,
so that case is not hypothetical), the right host with a smuggled path, 'main' and five
other bad branch shapes, a malformed owner/repo, a bad mode, a bad issue number, and each
missing credential. build-image.sh runs it inside the built image too.
Writing it found three things:
- the PM_BRANCH != main guard is unreachable while the pattern check above it holds, so
the comment claiming it as the defence was wrong about which line does the work. Kept
as an assertion independent of the regex; the comment now says so.
- the EXIT trap ended in a failing test, which under set -e can replace the script's
real exit code with that failure.
- a provenance LOG LINE read a file through a redirect that set -o pipefail would have
turned into a dead worker if the file were absent.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Agents have been working in the operator's checkout and clobbering each other.
Nothing in the tree told them not to: worktrees appeared in CLAUDE.md only as
§3's gate workarounds, and decisions/README.md described worktree isolation as
something agents already do rather than as a rule.
New CLAUDE.md §0 states the rule, with the commands and an explicit do-not list
for the shared checkout. Placed before §1 so every existing §-reference in the
repo keeps its meaning.
Measured in the shared checkout on 2026-10-05, each row a distinct failure mode:
107 `checkout: moving` reflog entries; sitting on feat/pm-agent-issue-to-pr, not
main; two other runs' untracked trees left in it; local main two commits behind
origin/main; 42 registered worktrees of which 31 are on already-merged branches;
9 gate scripts carrying a `.claude/` exclusion because an in-tree worktree made
find-based discovery count the same manifest up to 18 times.
Corrects a claim carried in earlier briefs: `git worktree remove` is NOT blocked
by the agent sandbox. Provoked on a throwaway worktree — rc 0.
§3 reframed as the normal path and grown from two breakages to three:
spec-immutability's skip is avoidable, not merely tolerated, with the spec repo
mounted read-only in the same ENGINE shim plus -e SPEC_REPO=/home-network-k3s.
CLAUDE.md is a sensitive path, so this PR needs a non-author review or an
administrator /approve-override posted as a PR review. Not posted by me: §1
forbids an agent clearing that on its own PR.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
last_activity() is the whole trigger: the PM acts iff the newest thing in a thread was
not written by it. It found that newest comment by comparing Gitea's RFC 3339
created_at STRINGS, which is only correct while every offset is identical — true today,
silently wrong after a server timezone change, and the failure would be the PM answering
its own last comment forever.
The comments endpoint returns them in creation order, so the last element IS the newest.
Using that removes the dependency entirely. Found by self-review, before anything ran.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Lets the operator's Android / Linux / Windows / macOS devices dial in from
anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus
internet egress out the house WAN. Opposite direction to platform/vpn-egress,
which sends pods OUT through PIA; the two share nothing but the word "VPN".
WireGuard because it is the only option with a first-party client on all four
named platforms, and because its handshake MAC is keyed with the responder's
own public key: an unauthenticated packet gets NO reply, so udp/51820 is
indistinguishable from a closed port. That, not "it is encrypted", is what
makes an unfiltered WAN port defensible, and why there is no source allowlist.
hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT
clients to its own pod IP, so every remote client would carry that pod's Cilium
identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from
Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway
preserves the original source identity and enforces the destination's ingress
policy. The obvious build would have shipped a healthy-looking tunnel on which
every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve.
In the host netns traffic is SNATed to the node and arrives as host/remote-node,
exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the
L2-announcement lease lottery, and double encapsulation (hence MTU 1420).
Permitted by restrict-host-namespaces' own carve-out for platform namespaces.
The cost is stated in networkpolicy.yaml rather than papered over: no
NetworkPolicy constrains a hostNetwork pod.
A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule;
all three nodes answer the same keys, so failover is a one-line router change.
Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my
house" a reviewable PR diff. The only secret -- the server's private key -- is
an out-of-band Secret: an agent must not mint a long-lived private key, and the
two halves are joined only at pod start into a tmpfs that dies with the pod.
Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain
(kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md.
Touches clusters/** , so ci/approval-policy.sh will block pending the
operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A service that reads this repository's Gitea issues, decides what to do about each
thread as the Program Manager persona, and runs one worker agent per assignment that
authors a change, runs the merge gate and opens a pull request. It reacts to issue
creation and to comments; the assignment and all correspondence travel through the
issue thread and the PR. It authenticates with an Anthropic subscription token. It
never pushes to main and never merges.
Three choices carry the design, with alternatives recorded in docs/plans/:
- it POLLS rather than receiving webhooks (Gitea's webhook SSRF guard, lost
deliveries, and no inbound reach needed at all);
- it keeps NO state: the trigger is "the newest thing in this thread was not
written by me", so the issue thread is the state — no PVC, no backup obligation;
- the model holds NO tools: its reply is one JSON object this code validates, so
nothing in the path executes text.
The boundary is platform/kyverno/admission/pm-agent-job-shape.yaml (Enforce,
failurePolicy: Fail), not the RBAC: Kubernetes places no `use` check on
serviceAccountName, so "may create Jobs" is "may run arbitrary code as the
namespace's most privileged identity". Six rules admit only the committed shape, and
the worker identity is bound to nothing with no token mounted, so there is nothing to
escalate to. 28/28 offline Kyverno assertions.
Nothing has run: the image has never been built or pushed, and both the image digest
and the two credentials are deliberate placeholders. See the handoff entry's
"What I did NOT verify".
Refs CLAUDE.md §1/§3/§6, R-SEC-02, R-GITOPS-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A merged Cilium config change did nothing, and everything reported success.
The Hubble flow exporter (#50, merged 2026-10-04) never took effect:
HelmRelease cilium Ready=True, "Helm upgrade succeeded ... cilium.v6"
cilium-config hubble-export-file-path: stdout + the allowlist, PRESENT
cilium agent pods all three still PRE-MERGE, startTime 2026-10-04T07:11Z
exported flow lines 0, 0, 0
Flux every Kustomization Ready=True
Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the
ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent
reads that ConfigMap only at STARTUP. Merged, believed delivered, green
everywhere, doing nothing. The same class of failure this repo already guards
against elsewhere: a green signal over something that never ran.
Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply --
both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed
the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why
the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has
rolled them since. The inertness is specific to ConfigMap-only changes.
Set in all three value files to hold the [parity] contract.
Cost, and it is the point: every future Cilium config change now rolls the
agents instead of waiting for a manual restart. Strictly more honest -- it
applies when merged or fails visibly -- but config edits become disruptive by
default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding
node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge
at a quiet moment.
Merging this IS one rollout (the ConfigMap checksum annotation changes the pod
template) -- which is what finally activates the Hubble exporter.
UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable
after the agents restart (check for flow lines immediately); and the rollout's
actual impact on LAN DNS is unmeasured.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85andClaude Opus 5