The in-cluster runner ran at the default capacity 1 with no config file, so
gate / images / renovate runs on different refs queued behind each other
(29-759s waits; run 472 waited 759s behind a renovate burst).
- ConfigMap act-runner-config with ONLY runner.capacity: 2, selected via
CONFIG_FILE (honoured by the v0.6.1 image's run.sh). v0.6.1 LoadDefault()
applies the same defaults to a partial file as to no file; the
generate-config example is deliberately not used (its labels would override
the registered ones, force_pull differs from the default).
- Pod-template config-revision annotation so ConfigMap-only edits can roll it.
- dind: cpu request 100m -> 2 (no limit), memory 256Mi/2Gi -> 1Gi/4Gi.
Measured at capacity 1: peak 0.69 core, 447Mi working set (30d), 0 OOMs.
- runner: memory 128Mi/512Mi -> 256Mi/1Gi (peak 314Mi at capacity 1).
apps/wave-1/gitea-actions.yaml is a hand-owned EXTRA_UNITS file (the
renderer only places it in a wave), so it is the source, not generated output.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Gitea dropped the "1." prefix; 28.x is the successor of 1.27.
- image 28.3.0, update.range >=28.0.0 <29.0.0, Renovate ceiling <29.0.0
- drop GITEA__server__DOMAIN (no longer read in 28, ROOT_URL is the source)
- declare RUN_RETENTION_DAYS=400, EGRESS_MODE=lax (no allow list: the OIDC
login source is not gated by the egress policy in 28.3.0), and
DISABLE_REGISTRATION=true (28's new default)
- document autoRollbackOnFailedHealth as inert (no ImagePolicy is rendered);
rollback is a PVC restore because the DB migration is one-way
Co-Authored-By: Claude Opus 5.5 <[email protected]>
pihole/pihole (no tag) -> 2026.10.0 sha256:d3413471...
lscr.io/linuxserver/jackett:latest -> v0.24.2865-ls7 sha256:4fcf3cc2...
plexinc/pms-docker:latest -> 1.43.4.10903-e5521bd8c sha256:e0ab2739...
Each tag was resolved against its registry and matched the pods' imageID, so the image is
unchanged (the pod template string does change, so Flux rolls the three workloads).
renovate.json5: regex versioning for jackett (-lsN) and plex (build-hex); jackett is looked
up on ghcr.io via overridePackageName, because lscr.io is capped at dockerMaxPages=20 pages
and jackett's 32k tags put the running one at #31,980 (dry run proposed nothing without it);
jackett weekly (13 builds in 12 days); pihole deliberately unheld, year rollover = major.
Stale "publishes only latest" prose fixed in renovate.json5 and three runbooks.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The Keycloak CR has no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the
server image; the three keycloak-k8s-resources release URLs in
platform/keycloak/kustomization.yaml are therefore THE Keycloak version, and Renovate
could not see them (no `# renovate:` marker). They now carry markers under one depName
(one branch, groupName keycloak-operator), with a packageRule: automerge false, labels
needs-review + blast-radius; majors still wait on the dashboard.
The untagged quay.io/keycloak/keycloak in catalog/keycloak.yaml renders
apps/wave-1/keycloak.yaml, which no kustomization lists. It is pinned to the operator's
26.0.6 as documentation, re-rendered, and Renovate is disabled for that image so it
cannot raise PRs against a file nothing deploys. The render is kept (not deleted): it is
the fixture for apps/wave-1/tests/app-slis.test.yaml's keycloak SLI rule tests.
README: an "Upgrading Keycloak" section — an operator bump must be checked against
passkey-reconciler.yaml's 26.0.6 Admin API assumptions.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator reported DNS flapping between the cluster ingress and the router's
public IP. PiHole is not at fault -- both replicas answer 192.168.2.20
consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in
CACHEABILITY that we created. Measured:
pihole 192.168.2.23 -> 192.168.2.20 TTL=0
public 1.1.1.1 -> wan.home.keithjackson.dev TTL=300
router 192.168.2.1 -> wan.... TTL=153 -> 148 -> 143 (decrementing = caching)
dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router
forwards LAN queries to PiHole and caches the result, so the CORRECT answer is
uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while
the WRONG answer caches for 300s. The instant the router falls back to upstream
(standard forwarder behaviour when the target does not answer) it latches the
WAN address for the whole LAN for five minutes, then expires, re-queries PiHole,
gets TTL 0, and is immediately re-exposed.
So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured
reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That
asymmetry, not the gap, is why it reads as "flapping".
Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies
local-ttl to replies made from configuration, so one directive covers all five
address= names. 60s rides out a pod restart and a lease re-election while a real
VIP change still propagates inside a minute; these VIPs are pinned and static.
apps/wave-0/pihole.yaml regenerated with the canonical render command
(--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves);
exactly two files changed and the rendered diff is the single local-ttl prefix.
UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected
L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If
the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough
line -- see the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A service that reads this repository's Gitea issues, decides what to do about each
thread as the Program Manager persona, and runs one worker agent per assignment that
authors a change, runs the merge gate and opens a pull request. It reacts to issue
creation and to comments; the assignment and all correspondence travel through the
issue thread and the PR. It authenticates with an Anthropic subscription token. It
never pushes to main and never merges.
Three choices carry the design, with alternatives recorded in docs/plans/:
- it POLLS rather than receiving webhooks (Gitea's webhook SSRF guard, lost
deliveries, and no inbound reach needed at all);
- it keeps NO state: the trigger is "the newest thing in this thread was not
written by me", so the issue thread is the state — no PVC, no backup obligation;
- the model holds NO tools: its reply is one JSON object this code validates, so
nothing in the path executes text.
The boundary is platform/kyverno/admission/pm-agent-job-shape.yaml (Enforce,
failurePolicy: Fail), not the RBAC: Kubernetes places no `use` check on
serviceAccountName, so "may create Jobs" is "may run arbitrary code as the
namespace's most privileged identity". Six rules admit only the committed shape, and
the worker identity is bound to nothing with no token mounted, so there is nothing to
escalate to. 28/28 offline Kyverno assertions.
Nothing has run: the image has never been built or pushed, and both the image digest
and the two credentials are deliberate placeholders. See the handoff entry's
"What I did NOT verify".
Refs CLAUDE.md §1/§3/§6, R-SEC-02, R-GITOPS-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
I took Plex down for the whole LAN. ADVERTISE_IP was
http://plex-vip.${BASE_DOMAIN}:32400 -- a name that exists ONLY in PiHole's
split-horizon answers and in no public zone.
A Plex client resolves the advertised host ITSELF, with whatever resolver it
uses, and on a real LAN that is frequently not PiHole: browsers ship DoH on by
default, phones have "private DNS", guest/IoT devices take the router's DNS.
Those clients could not resolve the name, the router answered the failed lookup
with ITSELF, and they all landed on its web UI -- hence the %D certificate.
Measured 2026-10-05:
192.168.2.1:443 -> C=ZZ, O=%D342e7e6b, CN=%D (router, LAN side)
136.51.115.252:443 -> the same %D cert (router, WAN side)
pihole-0 AND pihole-1 -> plex -> .20, plex-vip -> .24 (both CORRECT)
gateway .20:443 -> CN=*.home.keithjackson.dev, valid
http://192.168.2.24:32400/identity -> 200
Plex's published connections: all port 32400, nothing on 443
The DNS was right; the DEPENDENCY on it was the bug. And since nothing Plex
advertises listens on 443, the clients had left the Plex path entirely.
Fix: <dashed-ip>.<serverhash>.plex.direct, which is RESOLVER-INDEPENDENT -- a
public zone returning the private address embedded in the label, so every
resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers it with
192.168.2.24. No split-horizon entry, no Cloudflare record and no certificate
work needed: the wildcard cert Plex already serves on 32400 covers that name.
Re-verified today with full validation (no curl -k): 200, TLS verified.
The nicer hostname cannot be made correct for clients that bypass PiHole -- a
public record resolves to the WAN address and there is no NAT hairpin, so it is
right only for PiHole users and wrong for the rest. There is no configuration of
it that is right for both. PiHole's plex-vip answer is KEPT; it is just not what
Plex advertises.
Cost restated, not buried: the value embeds the VIP and the cert hash as a
literal and nothing checks they still agree. Accepted because an unresolvable
name is a worse failure -- silent, LAN-wide, and it points users at the router.
UNVERIFIED: unreconciled, and the client-side resolver path is INFERRED -- my
public-resolver probe was broken (NXDOMAIN even for a name that resolves) so I
discarded it. The chain is the only explanation consistent with the
measurements, but the NXDOMAIN-hijack step was not observed directly.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
I broke the app tier with #48. Two faults, both mine, both measured after merge:
1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key
Secrets (plex-vip-<random>) were denied by
secret-provenance/secrets-must-be-flux-managed, which matches every Secret in
a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes
only those labelled secret-managed-by=controller or carrying a Flux label.
`plex` HAS that label; `gateway` -- where the working wildcard cert lives --
does NOT. That asymmetry is why the existing Certificate was never affected
and mine was, and I did not check it. secretTemplate cannot fix it: those
Secrets come from the keymanager controller, not the Certificate template. So
no CertificateRequest, Order or Challenge was ever created.
2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod
rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so
Flux waits on the Certificate OBJECT:
timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']
apps-wave-0 never went Ready; wave-1/-2 and everything downstream of
platform-config sat in DependencyNotReady behind it. I guarded the pod and
missed the object. That distinction is the whole bug.
Third, independent: the advertised URL was https with no servable certificate --
Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip --
the exact failure mode of the outage earlier the same day.
So: delete the Certificate (pruning it clears the health gate) and advertise
http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the
name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there.
http is explicitly temporary and only tolerable because remote access does not
work yet; the restore condition is written at the call site.
DELETED, not merely unregistered: my first attempt kept the file as "tracked but
not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02)
fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a
TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization
+ the https flip, as ONE change) is recorded in catalog/plex.yaml.
Kept and verified working from #48: the hostname design, PiHole's split-horizon
answer, and the postStart hook -- LanNetworksBandwidth is now set live.
Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that
have never once scraped. Measured on prod 2026-10-04: all six read health=down with
lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the
configured scrapeTimeout, i.e. packets DROPPED rather than refused.
Cause: every app namespace ships a default-deny NetworkPolicy with
policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only
ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180.
Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr,
qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object.
render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None
unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this
catalog declares one. It was wired in at the call site and had never emitted a single
object on any branch: correct, and entirely latent.
It now also fires for the forward-auth gate sidecar's own metrics port, keyed on
_needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since
podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the
sidecar exists. Keying on observability.metrics instead would let metrics: false produce a
discovered-but-blocked target: the same "two halves authored apart" failure this function
exists to prevent.
Rendered here rather than in platform/oauth2-proxy because that tier would have to name app
namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster
and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering
per-app dissolves it.
Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring.
apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified
byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved.
Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an
object necessarily drifts the goldens that pin the renderer's output. The first gate run
caught container-workload-render and egress-class-render as FAIL with exactly this CNP as
the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the
evidence that the regen is faithful and not a rubber stamp.
That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author
approving review. The local gate cannot see that — offline it reports approval-policy green
(N/A).
Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went
FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no
GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements.
Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two symptoms after the publishing fix landed:
1. LAN clients were reported as Remote. Measured root cause:
LanNetworksBandwidth was ABSENT, so Plex had no notion of a local
network and classified everything WAN -- 468 requests logged
"Allowed Network (WAN)", ZERO "(LAN)", including Seerr from inside the
cluster. allowedNetworks only waives AUTH; it does not make a client
local. Two settings, and only one has an env hook.
2. The advertised URL hard-coded both the VIP and the server's cert hash
into a cluster-var -- two values nothing kept in sync.
Operator design: ONE hostname, plex-vip.${BASE_DOMAIN}, split-horizon --
PiHole answers it with ${PLEX_L4_VIP} on the LAN, Cloudflare answers it
publicly with the WAN address NATed to the same VIP.
- catalog/plex.yaml: ADVERTISE_IP -> https://plex-vip.${BASE_DOMAIN}:32400,
plus a level-triggered postStart hook (pihole/qbittorrent idiom) that
re-asserts LanNetworksBandwidth and the three customCertificate* prefs
every start, converting PEM -> PKCS#12 so cert renewals need no extra part.
- apps/wave-0/plex-cert.yaml: cert-manager Certificate for exactly that
name from letsencrypt-prod, so the advertised https URL is servable.
- schema 1.11.0: `optional` on secretKeyRef/configMapKeyRef. Load-bearing --
the cert is issued INTO the plex namespace, which the rendered plex.yaml
itself creates, so it lands in the same reconcile; hard refs would have
held Plex in CreateContainerConfigError and, since apps-wave-0 is
wait:true, stalled the whole app tier on a slow DNS01 challenge.
- catalog/pihole.yaml: the plex-vip split-horizon answer. Required --
measured, the name resolved to the Gateway VIP where nothing listens
on 32400 (http=000).
- PLEX_ADVERTISE_URL deleted; the hostname removes the cert-hash fragility.
Corrects my own earlier claim that a SAN match is REQUIRED: it is not
(httpsRequired=0, secureConnections defaults to Preferred, so clients can
fall back to cleartext). A matching cert keeps the remote leg ENCRYPTED.
Hence: obtain a real cert for the advertised name rather than pick the name
to suit Plex's built-in one.
UNVERIFIED, and stated plainly in the handoff: nothing here has run. Cert
issuance for this name is inferred from the wildcard's success, and whether
Plex accepts an OpenSSL-3 PKCS#12 bundle is untested -- the largest risk.
Touches clusters/** -> sensitive paths -> needs a non-author approval or the
operator's /approve-override; deliberately not self-posted (CLAUDE.md 1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
customConnections held two hand-typed URLs and neither could ever complete, so
Plex had nothing publishable and the server disappeared from the account:
- https://192.168.2.24:32400 -- the server's cert is
*.130072f20c334d6c8f63d0dfef87309b.plex.direct with NO IP SAN, so bare-IP
https fails TLS for every client. Measured: a continuous storm of
"CERT: incomplete TLS handshake ... certificate unknown" from real LAN
clients arriving through the plex-l4-tcp VIP.
- https://plex.home.keithjackson.dev:32400 -- nothing listens; the Gateway
serves that host on 443 (measured http=000 against 192.168.2.20:32400).
With no completable connection plex.tv published nothing ("Mapped - Not
Published"), Plex fell back to Relay, and clients reported "not authorized".
Fix: advertise <dashed-ip>.<serverhash>.plex.direct -- public DNS that resolves
to the embedded address, so it points at the LAN VIP and matches the wildcard
cert. Verified with full TLS validation (no curl -k): 200 + verify OK, where
the bare-IP form fails to connect.
Declared as ADVERTISE_IP so git owns the value instead of a web form: the
image's 40-plex-first-run writes it into customConnections on every pod start,
outside the claim guard.
Does NOT fix off-LAN access -- the advertised address is RFC1918 and the WAN
tcp/32400 forward remains unverified. ManualPortMappingMode=1 with an absent
port is left as-is (no GitOps path exists for it).
Touches clusters/** -> sensitive paths -> needs the operator's
/approve-override, deliberately not self-posted (CLAUDE.md 1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one
node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no
LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO
force-detach), while losing the node holding the pihole-dns L2 lease -- a
DIFFERENT node -- meant tens of seconds. Two of three nodes were single points
of failure with wildly different blast radii.
Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent
only; the mechanism is render.py's, documented in the renderer contract. 1 or
absent renders a Deployment exactly as before -- verified byte-identical across
the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a
multi-replica Deployment never schedules the second pod), per-replica
volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy
Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the
rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever).
Pinned by ~20 new --selftest assertions rather than a conformance change.
ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default
/data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M --
gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn
volume held lost+found and 24K of nothing. So every restart discarded all state
and re-downloaded 72k gravity domains, and backup.policy has been backing up an
empty volume while reporting success. Fixed here because otherwise the new
per-replica volumes would hold nothing while state stayed ephemeral -- and
because persisting gravity.db is what makes a restart cheap.
The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as
uid 0 and chowns the directory recursively (no fsGroup needed), and it calls
migrate_gravity which creates the database when absent.
The gate caught a real regression on the first run: the wave-0 alert unit test
fed pihole a Deployment-shaped series. Updated; promtool SUCCESS.
Touches no sensitive paths, so this one needs no override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Prerequisite for running more than one PiHole replica. A second replica with a
fresh PVC would have come up serving DNS and blocking nothing, with a different
admin password -- answers varying by which pod replied.
pihole.toml's own ### CHANGED markers gave the exact inventory: four settings
already came from env (declared), one came from the image, and exactly ONE
operator-set value lived only on the PVC -- webserver.api.pwhash, a one-way hash
nobody could recover. Separately the adlists were undeclared: one list and 72,234
gravity domains that nothing in this repo named.
Declares both:
- A SOPS secret feeding FTLCONF_webserver_api_password. webserver.api.password
takes cleartext and FTL hashes it itself (verified: pihole setpassword IS
setFTLConfigValue on that key), so the PVC's pwhash becomes derived state.
- A level-triggered postStart hook that reconciles the declared adlist set into
gravity.db -- adds, re-enables, and disables (never deletes) undeclared lists,
running pihole -g only when something moved. A file would not work: v6's
adlists.list import is migration-only, guarded on the DB not existing.
The rendered hook was run in the live pod against throwaway copies of gravity.db,
exercising all four paths. A fresh DB built from the image's own schema template
has ZERO adlists, confirming this is load-bearing rather than cosmetic.
Password kept rather than set empty: the pihole Service still publishes port 80
past the oauth2-proxy, closed only by policy, and default-deny has drifted here
before.
Multi-replica is phase 2 and deliberately not in this branch -- it needs a schema
field, a StatefulSet (the config PVC is RWO), anti-affinity and a PDB.
Adds a *.sops.yaml -- a sensitive path -- so approval-policy will block pending
an operator /approve-override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Remote Plex clients had no path in, for two measured reasons: Plex advertised
no remote connection at all (customConnections and ManualPortMappingMode both
ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so
32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves
Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients
discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this
server on a client's connection list.
Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS
with *.plex.direct and streams bypass the Envoy L7 hop.
Adds a hand-written non-catalog unit (the SD schema cannot express a
LoadBalancer Service or a pinned lbipam VIP), registered in render.py's
EXTRA_UNITS so --emit-waves places it in plex's own wave.
externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather
than re-deriving it. Known consequence, documented: SNAT puts remote clients
inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change.
Remote access does NOT work until the operator does two manual steps that
cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually
specify public port" toggle (no env var exists for it; verified NONE in the
image's init surface). runbooks/plex-remote-access.md is the procedure.
Touches clusters/** -- a sensitive path -- so the approval-policy stage will
block pending an operator /approve-override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The hostPath mechanism was MEASURED NOT TO WORK. In the live Plex pod on
k3s-prod-2, 2026-10-01: the device node was VISIBLE in the container
(`crw-rw---- root video2 226,128 renderD128`) and could not be opened --
`uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE
CGROUP, not EACCES from file permissions, proven three ways: root on the HOST
opens the same device rc=0; the node is unified cgroup2fs; and the container's
runc config.json read `linux.resources.devices: [{allow: False, access: 'rwm'}]`
with `linux.devices: None` -- one deny-all rule, no allow entry for major 226.
A hostPath is a filesystem operation: it binds the device node in but adds no
cgroup allow rule, because Kubernetes cannot infer one from a volume.
The Intel GPU device plugin produces that allow rule
(`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`).
It shipped as #36 and is live: all three prod nodes advertise
`gpu.intel.com/i915: 1`, DaemonSet 3/3/3 (measured 2026-10-01).
ZERO ServiceDefinition field changes, which was the entire point of making the
field an enum of capability classes rather than a device path. `catalog/plex.yaml`'s
`hardware:` block comes through byte-identical; only its comment changes, because
it restated the old mechanism. Contract invariant 8 holds, now demonstrated rather
than claimed.
- catalog/render.py: `DRI_VOLUME_NAME`/`DRI_DEVICE_DIR` retired for
`QUICKSYNC_RESOURCE`; the hardware branch now sets
`resources.limits["gpu.intel.com/i915"] = 1` on the APP container only (never a
sidecar) and emits no volume, no mount, no securityContext, no nodeSelector.
`limits` not `requests` is mandatory: Kubernetes mirrors requests from limits,
and a requests-only extended resource is invalid. This is the only app-container
`resources` render path and is deliberately narrow -- no general CPU/memory
feature was invented.
- selftests: resources dict asserted EXACTLY; hostPath/mount asserted ABSENT; the
no-`hardware` negative extended to `resources`; the podSecurityContext
pass-through and the no-synthesised-securityContext assertions kept; one new
assertion that the limit never reaches the oauth2-proxy sidecar. Mutation-tested:
8/8 mutants caught (requests-instead-of-limits, emission dropped, hostPath
re-added, emitted unconditionally, supplementalGroups synthesised, wrong resource
name, quantity as a string, limit leaked onto the sidecar).
- apps/wave-0/plex.yaml: six hostPath lines removed, a three-line resources block
added. Only golden touched; keycloak/vm-demo unchanged.
THE REGRESSION, STATED RATHER THAN SOLD. Before: device missing ->
`DirectoryOrCreate` mounted an empty dir -> Plex ran and SILENTLY
software-transcoded (invisible, harmless to everything else). After: resource
unavailable (plugin broken, removed, or not yet reconciled) -> Plex is
UNSCHEDULABLE, and `apps-wave-0` is `wait: true` with `apps-wave-1` dependsOn it,
so that STALLS THE APP TIER. Unavoidable: an extended resource is a hard
scheduling constraint, and it is also the only thing that produces the cgroup
allow rule. There is no graceful-degradation variant. Plex's schedulability is now
coupled to the intel-gpu-plugin DaemonSet. Escape hatch, one field and no code
change: delete the `hardware:` block from catalog/plex.yaml and re-render.
Documented in the contract, the runbook (its own top-of-page section and a
"Plex will not schedule" procedure) and the handoff.
HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING. The decisive open() test
cannot be re-run until this merges and Flux reconciles, and the Plex-side
`HardwareAcceleratedCodecs` toggle is still unset. A green gate proves the
manifest renders.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PR #28 fixed the cookie collision, the 5-minute session and the missing metrics
listener in one renderer function, and deliberately declined `--api-route`
because the config model could not express the difference between two apps.
This adds the hook it was missing and sets the flag only where a measurement
says to.
THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to
Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT
`Accept: application/json`, which these apps' JavaScript does not consistently
send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied
silently by the live SSO session, and resolves 200 carrying the app's HTML where
the JavaScript expected JSON. On a write it is worse: a browser following a 302
re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
discarded and the UI reports success.
MEASURED, on two apps and two different paths -- which is the whole argument:
jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs
answered by the proxy and never reaching Jackett (upstream `-`).
radarr 2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh,
0 restarts): 232 unauthenticated requests in the pod's whole life and
ALL 232 are the same call -- POST /signalr/messages/negotiate from a
browser, 302, upstream `-`. A left-open tab whose session the cookie
rename orphaned, polling forever, silently redirected every time and
NEVER re-prompting. The reported symptom, still in production.
So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES
NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a
constant -- and a constant applies one app's measurement to four apps nobody
measured. `auth` was `additionalProperties: false`, so there was no per-app
hook. Adding one named, documented key is the change; the block stays closed.
schemas/servicedefinition.schema.json 1.8.0 -> 1.9.0: optional
`auth.apiRoutes`, a list (radarr needs two, and --api-route is natively
repeatable) of anchored path regexes. Plus the shape invariant that a
non-empty list requires mode: forward-auth, since no sidecar is rendered
for oidc/native/none and declaring it there would be a SILENT NO-OP.
catalog/render.py emits one --api-route per entry, in order, and RAISES on
an unanchored entry. The flag is SINGULAR -- verified against the pinned
v7.8.1 binary's own --help; --api-routes does not exist and the proxy
refuses to start with it.
catalog/{jackett,radarr}.yaml the two measured apps.
catalog/{pihole,sonarr,qbittorrent}.yaml comments recording that they get
NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated
requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no
need, and sonarr being nearly radarr's codebase is what makes the guess
tempting and still a guess.
apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml regenerated. Only TWO
goldens change; pihole/qbittorrent/sonarr are byte-identical to main,
which is the per-app property proving itself on the real tree.
BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing
--api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with
401, so a logged-out operator gets a bare error page and is never offered a
sign-in -- the service looks permanently broken. Each entry must therefore
match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$`: anchored at ^/ then literal path
characters. `^/`, `^/.*`, `.*`, `/api/` and `^/api/|^/` are all rejected, all
five provoked. Alternation is refused rather than supported because the flag is
repeatable, so one entry per route costs nothing.
NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app
(whole-repo scope, so ci/l0.sh's --include-scope union runs it on every
invocation; no bespoke stage, which whole-repo-invariants-wired would fail).
It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it:
that check's subject is cookie IDENTITY and shares neither inputs nor failure
mode with this, so bundling them would make its name a lie and let a future
api-route decision be expressed only by weakening a COOKIE check.
The check exists for one reason the other gates cannot cover: render-golden
compares the render to the goldens, so a RENDERER CONSTANT plus regenerated
goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits
0 -- render-golden is happy -- and this check exits 1 with 15 failures naming
pihole, qbittorrent, radarr and sonarr individually. It also asserts the
anchoring pattern is the SAME in the schema and the renderer, because three
surfaces guard this and if they drift the loosest wins silently.
Tested in both directions on throwaway trees outside the repo: compliant tree
rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four
targeted mutations (hand-edited golden, widened regex, renderer constant,
deleted schema pattern) each firing their own assertion class and nothing else.
ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest()
reached _cookie_name() through a bare `next(genexpr)` with NO default, so
deleting the --cookie-name emission -- the exact mutation that assertion exists
to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected
`SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it
masked every other assertion the same mutation broke (found by Desmond Oyelaran
while wiring the selftest into the render-golden stage; his PR #33 is not in this
base -- origin/main @ cdf3e14 drives the selftest from no stage, checked). Fixed
with `next(..., None)` plus an explicit check(), so a missing flag reports as a
NAMED assertion. Provoking it exposed three more instances of the same class in
the same block -- `side = fa_containers[1]`, `side["env"][0]`, and the fa_svc
lookup all raise rather than assert when the sidecar is absent -- so all four are
now None-tolerant, and the two fixtures added for --api-route were written that
way from the start. Measured on throwaway copies:
--cookie-name deleted: before rc 1 / 0 FAIL lines / StopIteration
after rc 1 / 3 FAIL lines / no traceback
sidecar not appended: before rc 1 / 0 FAIL lines / IndexError
after rc 1 / 20 FAIL lines / no traceback
Goldens are unchanged by this part; it touches _selftest() only.
NOT verified, in full in the handoff entry: no browser round-trip by anyone for
this flag -- that a 401 REPAIRS the operator's experience is reasoned from HTTP
semantics, not watched. `^/api/` on radarr is PR #30's measurement of radarr's
pre-fix pod, not mine; the current pod has zero /api/ traffic. Sonarr and
qBittorrent are UNMEASURED, not cleared. --cookie-refresh is still unexercised
(0 AuthSuccess on all five pods). Nothing was applied; the live pods are still
the pre-change spec.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Plex has been transcoding in software on hardware that can do it in the iGPU. This
opts its ServiceDefinition into the `hardware` capability from #27 and re-renders.
catalog/plex.yaml gains four lines of YAML and the comment that explains them:
hardware:
gpu: intel-quicksync
which renders exactly two things into apps/wave-0/plex.yaml and nothing else — a
`dri` hostPath volume (/dev/dri, type: DirectoryOrCreate) and a read-write /dev/dri
volumeMount on the app container. Whole-tree render diff: 6 added lines, one file.
The comment records what is not obvious from the four lines: the device exists on
ALL THREE prod nodes with i915 bound (so no nodeSelector is rendered and none is
wanted); why `supplementalGroups` would be INERT rather than a hardening (the
image's cont-init hook adds the plex user to the device's group inside the
container, and s6-setuidgid then REPLACES the process group list from /etc/group,
discarding anything Kubernetes supplied); and why `DirectoryOrCreate` is
load-bearing (apps-wave-0 is wait:true with apps-wave-1 depending on it, so a
failed mount would stall the whole app tier for a feature that is not).
THE FEATURE IS INERT UNTIL A HUMAN CLICKS A BOX, and that is the part this commit
is really about. Mounting the device does not enable hardware transcoding; Plex's
`HardwareAcceleratedCodecs` preference does, and it is ABSENT from Preferences.xml
— the toggle has never been on. It cannot be automated: a bounded grep across the
running image's entire init surface (/etc/cont-init.d, /etc/services.d,
/etc/fix-attrs.d, /plex-common.sh, /usr/local/bin) found ZERO PLEX_PREFERENCE*
support, and the six setPref keys the image does write are all unrelated. So
runbooks/plex-quicksync.md is the procedure, registered in the configMapGenerator
(a runbook in git with no generator entry is reviewed, merged, and still a 404) and
indexed in runbooks/README.md. It covers the GitOps/manual boundary, the toggle and
the real preference keys (including the widely-cited one that does not exist and
the upstream misspelling nobody should "fix"), verification via `Transcode (hw)` on
the Plex dashboard — stated plainly as the ONLY proof, because Plex renders no
ServiceMonitor and an alert could never fire — the device-cgroup failure to expect
first with its Intel-device-plugin fallback, and the floating-`:latest` caveat.
Stacked on #27 (feat/sd-hardware-capability), which is open and unmerged; that
branch's commit is in this range by design and this must merge after it.
NOT VERIFIED, and the handoff says so at length: hardware transcoding has not been
observed working. Whether containerd's device cgroup lets a non-privileged pod
open() a bind-mounted char device is an open question that only deploying settles,
and the Plex-side toggle was still unset at the time of writing. A green gate
proves the manifest renders; it proves nothing about transcoding.
Gate (pinned toolbox, worktree): rc 3 — UNDETERMINED, not green. Passed:
decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped (the known
local ceiling): spec-immutability (no sibling spec repo), gitea-settings (no
SETTINGS_API_TOKEN), conformance (CLUSTER_AVAILABLE unset, so L2/L3 did not run).
No fourth skip, nothing failed.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Widens the cookie-collision fix to the whole sidecar contract, because all three
defects live in the same function and land on the same five goldens, so they are one
atomic change or they are five conflicting ones.
Added to _oauth2_sidecar_container(), on top of the cookie-name/cookie-domain fix:
--cookie-refresh=2m30s
The session died FIVE MINUTES after sign-in. Keycloak's homelab realm issues
300-second access tokens (MEASURED on a live session: expires - created =
300.000s exactly, with refresh_token:true on the same line), and
--cookie-refresh defaults to 0 -- which the pinned binary's own help spells
"0 to disable". So the proxy held a refresh token it was never told to spend.
Refresh at HALF the lifetime: two chances to renew before expiry.
This, not the collision, is what dominates the rejection VOLUME -- ~95% of
radarr's 155 rejections in 12h carry no cookie at all.
--metrics-address=0.0.0.0:44180 + a `metrics` container port
Without the flag the /metrics listener never starts, so no per-app sidecar was
serving metrics at all (`ports` was ['http'] on all six). 44180 is the
platform's existing convention, measured not chosen: the shared proxy already
runs the same flag and port with 2 ready endpoints.
PREREQUISITE ONLY -- NOT scraped. A container port is not a Service port, and
each namespace's default-deny baseline is `ingress: []`, so this is unreachable
until a Service port (or PodMonitor), a monitoring allow and a ServiceMonitor
land. Do not read this as "the sidecars have metrics now".
EVERY FLAG VERIFIED AGAINST THE PINNED v7.8.1 BINARY (`--help`), not against docs or
the shared proxy -- which runs v7.15.3 and could not have vouched for v7.8.1:
* --metrics-address exists in 7.8.1, so this cannot crashloop the pods
* --cookie-refresh ... "0 to disable" confirms the default
* --cookie-name ... (default "_oauth2_proxy") -- this confirms from the
IMPLEMENTATION the constant the new L0 check is built on
* --api-route is SINGULAR; the plural does not exist and the proxy refuses to start
* --session-cookie-minimal exists and is mutually exclusive with --cookie-refresh,
so dropping --pass-access-token is a least-privilege argument, never a size fix
NOT done, deliberately: --api-route=^/api/ renderer-wide. It was measured on jackett
and PiHole does not serve /api/, so a renderer constant would apply one app's path
shape to five. It needs a per-app hook the `auth` schema forbids
(additionalProperties: false) -- a schema change, separate PR.
Also records the mechanism that made this INTERMITTENT, as a renderer comment, because
nobody would reconstruct it from the flags: the loser does not just fail to read the
cookie, it calls ClearSession, which expires the cookie on its own --cookie-domain --
the whole zone. The *arr UIs poll (/api/v3/command, /api/v3/queue/details,
/signalr/negotiate), so every poll carrying an undecryptable cookie clears it again:
A LEFT-OPEN RADARR TAB DELETED JACKETT'S COOKIE EVERY FEW SECONDS, from another
namespace, with no user action. That is why signing in again never helped for longer
than the gap between two polls.
Crafty's counter-evidence resolved: the 81:1 ratio was a 24h window straddling a
13h52m-old pod. Bounded to the fixed process, crafty's ENTIRE life is 3 rejections in
6743 log lines, with ZERO `removing session` and ZERO `cookie signature not valid`
against unfixed radarr's 8 and 8. The eviction fingerprint is gone where the fix is
deployed. It logged 0 AuthSuccess too, so that is absence-of-the-failure-mode rather
than a session watched working -- stated as such in the handoff.
Renderer selftest gains six assertions, including a --cookie-refresh check that PARSES
the duration and requires 0 < refresh < 300s (so 2m30s/150s/2.5m all stay green while a
silent widening past the token lifetime does not). NOTE, and it is in the handoff as the
top follow-up: NOTHING IN ci/ RUNS `render.py --selftest`, so those assertions are
dev-time only. conformance/tests/l0-static/auditability-proofs-producible/check.sh:291
already drives other tools' selftests for exactly this reason and is the one-line fix.
Gate (toolbox, worktree shim with the spec repo mounted at /home-network-k3s): rc 3
UNDETERMINED. Green: spec-immutability, decisions-log, lint, l0, render-golden, l1,
approval-policy. Skipped: gitea-settings (no local token), conformance (declared).
The new L0 check PASSES on this branch -- all five apps fixed in one commit.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Every oauth2-proxy on this platform is an independent instance with its OWN
cookie-secret. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven
DISTINCT cookie-secret values (compared in-cluster, never printed). So no instance
can DECRYPT another's cookie -- it can only OVERWRITE it.
All of them were emitting oauth2-proxy's DEFAULT names (`_oauth2_proxy`,
`_oauth2_proxy_csrf`) scoped to `.${BASE_DOMAIN}`, so the browser kept ONE cookie
pair for SIX live instances (jackett, pihole, qbittorrent, radarr, sonarr, and the
shared forward-auth proxy). Whichever authenticated last clobbered the rest; the
loser logged `No valid authentication in request`, bounced to Keycloak, and -- the
SSO session still being live -- was redirected straight back and clobbered it again.
The symptom is "I signed into Keycloak but I'm still not signed in", which sends you
hunting in Keycloak, where nothing is wrong. Crafty was fixed on its own side on
2026-09-29 (#22); this is that fix generalized to the pattern.
THE DEFECT WAS NOT IN THE MANIFESTS. apps/wave-*/{jackett,pihole,qbittorrent,radarr,
sonarr}.yaml are GENERATED goldens; one function emits all five. So the fix is in
catalog/render.py::_oauth2_sidecar_container(), and both flags are now DERIVED from
the ServiceDefinition rather than defaulted:
--cookie-name=_oauth2_proxy_{s['name']} unique by construction
--cookie-domain={host} the app's own host, not the zone
A per-service table was rejected for the reason twelve ServiceMonitors once acquired
one identical wrong port name in a single edit: it would leave the NEXT app onboarded
defaulting straight back into the collision. --whitelist-domain stays zone-wide; it
governs post-login REDIRECT targets, not cookie scope.
Host-scoping is defence in depth and is asserted deliberately: a zone-wide cookie is
transmitted by the browser to every host in the zone, so each app's session token was
being sent to every sibling -- and it is what would make a FUTURE name collision
platform-wide instead of contained.
New gate check (whole-repo scope, so it runs on EVERY gate invocation, not only when
a touched path happens to attribute an identity requirement):
conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/
It resolves each instance's EFFECTIVE cookie name -- declared, or oauth2-proxy's
built-in default when omitted, because an omitted flag is not an absence but a CLAIM
on a name, and reading it as an absence is exactly why this survived five reviews.
Then: no two collide; none is another's name + `_csrf` (oauth2-proxy derives the CSRF
cookie that way); the default has exactly ONE owner, registered with its reason in a
shrink-only register -- the SHARED proxy, which MUST keep `_oauth2_proxy` because
renaming it signs every other gated surface out at once; every other instance is
host-scoped; render.py derives rather than lists; and the SHIPPED renderer is proven
end-to-end on two synthetic ServiceDefinitions, so the check stays falsifiable once
the tree is clean.
Tested in both directions: rc 0 on this tree (39 assertions, 7 instances), rc 1 with
26 failures against the pre-fix tree, plus one targeted mutation per assertion class
(duplicate name, CSRF derivation, zone-wide domain, unregistered default heir, stale
register entry) -- each fired, and nothing else did.
Also regenerates the three L1 rendering goldens, which are expected-output fixtures
of the same renderer.
UNVERIFIED: no browser round-trip. The collision has still never been REPRODUCED in a
browser -- doing so needs two gated apps visited in one session -- so this is reasoned
from measured configuration and from the mechanism, not from a failure anyone watched.
Nothing was applied; merging rolls five Deployments and invalidates every existing
session on them once. See decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The branch-protection + PR-gate + sidecar sequence is complete and verified. This records
the finished state and fixes what the sequence itself proved wrong.
CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the
one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT.
Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the
startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut
off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased:
raising the timeout to hide one benign startup line would slow every subsequent poll. The
manifest comment now states this instead of the wrong prediction.
Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from
a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed
act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online,
and a real workflow goes green on it, before enabling enforcement and probing the push.
Re-run it after any future act-runner change.
Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were
deleted, so no prefix produces a duplicate run any more.
CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with
all four rows marked verified and the push rejection noted as PROVOKED rather than assumed.
It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3
never runs in CI and only the L0/L1 offline subset is covered.
infra/gitea/README.md marks the sequence done and names the root cause that remains
unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by
the conformance gate, so NOTHING detects drift from what this directory declares. These
scripts assert desired state; they are not a detector. The check that reads the live API
and fails on divergence is the highest-value work left, and it is the difference between
this being fixed and this being unable to recur.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85
renovate-bot