3ee663dbcc05382d0359b616824689d7f39eec47
66
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jakman85
|
3ee663dbcc |
Merge pull request 'docs(agents): one workspace per agent run — the shared checkout is off-limits' (#56) from docs/agents-work-in-their-own-workspace into main
Reviewed-on: #56 Reviewed-by: Keith Jackson <[email protected]> |
||
 jakman85andClaude Opus 5
|
2cb891c05d |
docs(agents): one workspace per agent run — the shared checkout is off-limits
Agents have been working in the operator's checkout and clobbering each other. Nothing in the tree told them not to: worktrees appeared in CLAUDE.md only as §3's gate workarounds, and decisions/README.md described worktree isolation as something agents already do rather than as a rule. New CLAUDE.md §0 states the rule, with the commands and an explicit do-not list for the shared checkout. Placed before §1 so every existing §-reference in the repo keeps its meaning. Measured in the shared checkout on 2026-10-05, each row a distinct failure mode: 107 `checkout: moving` reflog entries; sitting on feat/pm-agent-issue-to-pr, not main; two other runs' untracked trees left in it; local main two commits behind origin/main; 42 registered worktrees of which 31 are on already-merged branches; 9 gate scripts carrying a `.claude/` exclusion because an in-tree worktree made find-based discovery count the same manifest up to 18 times. Corrects a claim carried in earlier briefs: `git worktree remove` is NOT blocked by the agent sandbox. Provoked on a throwaway worktree — rc 0. §3 reframed as the normal path and grown from two breakages to three: spec-immutability's skip is avoidable, not merely tolerated, with the spec repo mounted read-only in the same ENGINE shim plus -e SPEC_REPO=/home-network-k3s. CLAUDE.md is a sensitive path, so this PR needs a non-author review or an administrator /approve-override posted as a PR review. Not posted by me: §1 forbids an agent clearing that on its own PR. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f92707f38c |
feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers)
Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/** , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
d0db63911a |
feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)
A merged Cilium config change did nothing, and everything reported success. The Hubble flow exporter (#50, merged 2026-10-04) never took effect: HelmRelease cilium Ready=True, "Helm upgrade succeeded ... cilium.v6" cilium-config hubble-export-file-path: stdout + the allowlist, PRESENT cilium agent pods all three still PRE-MERGE, startTime 2026-10-04T07:11Z exported flow lines 0, 0, 0 Flux every Kustomization Ready=True Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent reads that ConfigMap only at STARTUP. Merged, believed delivered, green everywhere, doing nothing. The same class of failure this repo already guards against elsewhere: a green signal over something that never ran. Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply -- both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has rolled them since. The inertness is specific to ConfigMap-only changes. Set in all three value files to hold the [parity] contract. Cost, and it is the point: every future Cilium config change now rolls the agents instead of waiting for a manual restart. Strictly more honest -- it applies when merged or fails visibly -- but config edits become disruptive by default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge at a quiet moment. Merging this IS one rollout (the ConfigMap checksum annotation changes the pod template) -- which is what finally activates the Hubble exporter. UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable after the agents restart (check for flow lines immediately); and the rollout's actual impact on LAN DNS is unmeasured. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
81f627167d |
docs(obs): record that the gate-sidecar scrape works — four places still said it did not
PR #49 (merged
|
||
|
jakman85andClaude Opus 5
|
874c6703f0 |
fix(plex): advertise a resolver-independent plex.direct name again (LAN-wide outage)
I took Plex down for the whole LAN. ADVERTISE_IP was http://plex-vip.${BASE_DOMAIN}:32400 -- a name that exists ONLY in PiHole's split-horizon answers and in no public zone. A Plex client resolves the advertised host ITSELF, with whatever resolver it uses, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default, phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve the name, the router answered the failed lookup with ITSELF, and they all landed on its web UI -- hence the %D certificate. Measured 2026-10-05: 192.168.2.1:443 -> C=ZZ, O=%D342e7e6b, CN=%D (router, LAN side) 136.51.115.252:443 -> the same %D cert (router, WAN side) pihole-0 AND pihole-1 -> plex -> .20, plex-vip -> .24 (both CORRECT) gateway .20:443 -> CN=*.home.keithjackson.dev, valid http://192.168.2.24:32400/identity -> 200 Plex's published connections: all port 32400, nothing on 443 The DNS was right; the DEPENDENCY on it was the bug. And since nothing Plex advertises listens on 443, the clients had left the Plex path entirely. Fix: <dashed-ip>.<serverhash>.plex.direct, which is RESOLVER-INDEPENDENT -- a public zone returning the private address embedded in the label, so every resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers it with 192.168.2.24. No split-horizon entry, no Cloudflare record and no certificate work needed: the wildcard cert Plex already serves on 32400 covers that name. Re-verified today with full validation (no curl -k): 200, TLS verified. The nicer hostname cannot be made correct for clients that bypass PiHole -- a public record resolves to the WAN address and there is no NAT hairpin, so it is right only for PiHole users and wrong for the rest. There is no configuration of it that is right for both. PiHole's plex-vip answer is KEPT; it is just not what Plex advertises. Cost restated, not buried: the value embeds the VIP and the cert hash as a literal and nothing checks they still agree. Accepted because an unresolvable name is a worse failure -- silent, LAN-wide, and it points users at the router. UNVERIFIED: unreconciled, and the client-side resolver path is INFERRED -- my public-resolver probe was broken (NXDOMAIN even for a name that resolves) so I discarded it. The chain is the only explanation consistent with the measurements, but the NXDOMAIN-hijack step was not observed directly. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a0e6ef145f |
fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http
I broke the app tier with #48. Two faults, both mine, both measured after merge: 1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key Secrets (plex-vip-<random>) were denied by secret-provenance/secrets-must-be-flux-managed, which matches every Secret in a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes only those labelled secret-managed-by=controller or carrying a Flux label. `plex` HAS that label; `gateway` -- where the working wildcard cert lives -- does NOT. That asymmetry is why the existing Certificate was never affected and mine was, and I did not check it. secretTemplate cannot fix it: those Secrets come from the keymanager controller, not the Certificate template. So no CertificateRequest, Order or Challenge was ever created. 2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so Flux waits on the Certificate OBJECT: timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress'] apps-wave-0 never went Ready; wave-1/-2 and everything downstream of platform-config sat in DependencyNotReady behind it. I guarded the pod and missed the object. That distinction is the whole bug. Third, independent: the advertised URL was https with no servable certificate -- Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip -- the exact failure mode of the outage earlier the same day. So: delete the Certificate (pruning it clears the health gate) and advertise http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there. http is explicitly temporary and only tolerable because remote access does not work yet; the restore condition is written at the call site. DELETED, not merely unregistered: my first attempt kept the file as "tracked but not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02) fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization + the https flip, as ONE change) is recorded in catalog/plex.yaml. Kept and verified working from #48: the hostname design, PiHole's split-horizon answer, and the postStart hook -- LanNetworksBandwidth is now set live. Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
c163deaef9 |
Merge pull request 'feat(plex): one advertised hostname for local+remote, its own LE cert, and a LAN definition so LAN clients stop reading as remote' (#48) from feat/plex-vip-hostname-and-lan-networks into main
Reviewed-on: #48 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
642a7752d8 |
Merge pull request 'feat(cilium): export Plex's pre-SNAT client address via Hubble, scoped to tcp/32400' (#50) from feat/hubble-flow-export-plex into main
Reviewed-on: #50 |
||
|
jakman85andClaude Opus 5
|
989649e64a |
feat(cilium): export Plex's pre-SNAT client address via Hubble, scoped to tcp/32400
The requirement is log visibility only -- which address connected to Plex.
Plex's own log cannot show it. Every LoadBalancer Service here is
externalTrafficPolicy: Cluster, so the node holding the VIP's L2 lease SNATs
the client before forwarding to the backend pod. Measured: Plex had logged
3,214 requests from 10.0.1.x (lease holder k3s-prod-3's own address, Cilium
identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No
X-Forwarded-For either -- it serves raw tcp/32400, not HTTP via the Gateway.
Nothing on the router does this; there is no hairpin NAT. It is Cilium.
The address is not lost, only unrecorded. Cilium observes the flow at ingress,
BEFORE its own translation, so Hubble already has it:
192.168.2.133 -> 10.0.0.135 real LAN client -> the Plex pod
10.0.1.78 -> 10.0.0.135 same traffic post-SNAT, what Plex sees
What was missing is RETENTION: 4095/4095 flows at ~303/s is about thirteen
seconds of history, metrics disabled, no exporter configured.
So enable the static exporter, port-scoped, to the agent's stdout -- Alloy
already tails every pod into Loki, so this needs no new mount, no rotation
handling and no Alloy change.
{namespace="kube-system", app="cilium"} |= "32400"
Scoped BY PORT, not namespace, on evidence: --to-namespace plex returned only
the post-SNAT flows; the record carrying the real client address did not match
it. A namespace filter would drop exactly what this exists to capture.
Rejected, with reasons: eTP:Local is incompatible with cilium 1.20's L2
announcer per upstream docs for this version ("may cause service IPs to be
announced on nodes without pods causing traffic drops") -- and the VIP is on
prod-3 while the pod is on prod-1, so it would black-hole 32400 immediately;
making it safe costs Plex its ability to reschedule. DSR: routing-mode is
vxlan, not native/geneve, and DSR returns bypass the ingress node, breaking
the router's stateful NAT on the remote path.
Also corrects the repo's own record: 2026-10-03 says the SNAT lands in
192.168.0.0/16 (the node's LAN address). It is 10.0.1.x, the cilium_host
inside the pod CIDR -- which is why LanNetworksBandwidth can never separate
local from remote while eTP is Cluster.
UNVERIFIED and stated in the handoff: nothing is reconciled; `filePath: stdout`
is taken from a verbatim upstream sentence but not confirmed against 1.20.1
source (a wrong notation is a silent no-op, with the file+Alloy fallback
documented); the port FlowFilter has no upstream example; and this change
ROLLS CILIUM -- prod-3 announces the Gateway, PiHole DNS and Plex VIPs, so a
lease move during the rollout could briefly interrupt LAN DNS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
b156ee6cfa |
fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable
platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that have never once scraped. Measured on prod 2026-10-04: all six read health=down with lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the configured scrapeTimeout, i.e. packets DROPPED rather than refused. Cause: every app namespace ships a default-deny NetworkPolicy with policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180. Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr, qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object. render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this catalog declares one. It was wired in at the call site and had never emitted a single object on any branch: correct, and entirely latent. It now also fires for the forward-auth gate sidecar's own metrics port, keyed on _needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the sidecar exists. Keying on observability.metrics instead would let metrics: false produce a discovered-but-blocked target: the same "two halves authored apart" failure this function exists to prevent. Rendered here rather than in platform/oauth2-proxy because that tier would have to name app namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering per-app dissolves it. Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring. apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved. Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an object necessarily drifts the goldens that pin the renderer's output. The first gate run caught container-workload-render and egress-class-render as FAIL with exactly this CNP as the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the evidence that the regen is faithful and not a rubber stamp. That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author approving review. The local gate cannot see that — offline it reports approval-policy green (N/A). Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements. Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0aa1a0a78a |
feat(plex): one advertised hostname for local+remote, its own LE cert, and a LAN definition
Two symptoms after the publishing fix landed:
1. LAN clients were reported as Remote. Measured root cause:
LanNetworksBandwidth was ABSENT, so Plex had no notion of a local
network and classified everything WAN -- 468 requests logged
"Allowed Network (WAN)", ZERO "(LAN)", including Seerr from inside the
cluster. allowedNetworks only waives AUTH; it does not make a client
local. Two settings, and only one has an env hook.
2. The advertised URL hard-coded both the VIP and the server's cert hash
into a cluster-var -- two values nothing kept in sync.
Operator design: ONE hostname, plex-vip.${BASE_DOMAIN}, split-horizon --
PiHole answers it with ${PLEX_L4_VIP} on the LAN, Cloudflare answers it
publicly with the WAN address NATed to the same VIP.
- catalog/plex.yaml: ADVERTISE_IP -> https://plex-vip.${BASE_DOMAIN}:32400,
plus a level-triggered postStart hook (pihole/qbittorrent idiom) that
re-asserts LanNetworksBandwidth and the three customCertificate* prefs
every start, converting PEM -> PKCS#12 so cert renewals need no extra part.
- apps/wave-0/plex-cert.yaml: cert-manager Certificate for exactly that
name from letsencrypt-prod, so the advertised https URL is servable.
- schema 1.11.0: `optional` on secretKeyRef/configMapKeyRef. Load-bearing --
the cert is issued INTO the plex namespace, which the rendered plex.yaml
itself creates, so it lands in the same reconcile; hard refs would have
held Plex in CreateContainerConfigError and, since apps-wave-0 is
wait:true, stalled the whole app tier on a slow DNS01 challenge.
- catalog/pihole.yaml: the plex-vip split-horizon answer. Required --
measured, the name resolved to the Gateway VIP where nothing listens
on 32400 (http=000).
- PLEX_ADVERTISE_URL deleted; the hostname removes the cert-hash fragility.
Corrects my own earlier claim that a SAN match is REQUIRED: it is not
(httpsRequired=0, secureConnections defaults to Preferred, so clients can
fall back to cleartext). A matching cert keeps the remote leg ENCRYPTED.
Hence: obtain a real cert for the advertised name rather than pick the name
to suit Plex's built-in one.
UNVERIFIED, and stated plainly in the handoff: nothing here has run. Cert
issuance for this name is inferred from the wildcard's success, and whether
Plex accepts an OpenSSL-3 PKCS#12 bundle is untested -- the largest risk.
Touches clusters/** -> sensitive paths -> needs a non-author approval or the
operator's /approve-override; deliberately not self-posted (CLAUDE.md 1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
1f4552d409 |
fix(plex): advertise the plex.direct name, not the bare VIP (server vanished from the plex.tv account)
customConnections held two hand-typed URLs and neither could ever complete, so Plex had nothing publishable and the server disappeared from the account: - https://192.168.2.24:32400 -- the server's cert is *.130072f20c334d6c8f63d0dfef87309b.plex.direct with NO IP SAN, so bare-IP https fails TLS for every client. Measured: a continuous storm of "CERT: incomplete TLS handshake ... certificate unknown" from real LAN clients arriving through the plex-l4-tcp VIP. - https://plex.home.keithjackson.dev:32400 -- nothing listens; the Gateway serves that host on 443 (measured http=000 against 192.168.2.20:32400). With no completable connection plex.tv published nothing ("Mapped - Not Published"), Plex fell back to Relay, and clients reported "not authorized". Fix: advertise <dashed-ip>.<serverhash>.plex.direct -- public DNS that resolves to the embedded address, so it points at the LAN VIP and matches the wildcard cert. Verified with full TLS validation (no curl -k): 200 + verify OK, where the bare-IP form fails to connect. Declared as ADVERTISE_IP so git owns the value instead of a web form: the image's 40-plex-first-run writes it into customConnections on every pod start, outside the claim guard. Does NOT fix off-LAN access -- the advertised address is RFC1918 and the WAN tcp/32400 forward remains unverified. ManualPortMappingMode=1 with an absent port is left as-is (no GitOps path exists for it). Touches clusters/** -> sensitive paths -> needs the operator's /approve-override, deliberately not self-posted (CLAUDE.md 1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
c95b1c7121 |
Merge pull request 'fix(cilium): point cgroup.hostRoot at the host cgroup2 root (socket-LB never covered host processes)' (#42) from fix/cilium-cgroup-hostroot into main
Reviewed-on: #42 |
||
|
jakman85andClaude Opus 5
|
254985baa6 |
docs(decisions): record that PiHole HA is verified live, failover drill included
No manifests change. The three entries merged earlier today each say, correctly
at the time, that nothing had been reconciled -- which is now stale in exactly
the way CLAUDE.md section 2b warns about: a reader would conclude the work is
unproven. This entry is the current state and does not edit the earlier files.
Measured on live prod after
|
||
|
jakman85andClaude Opus 5
|
028853c87c |
feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes
Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO force-detach), while losing the node holding the pihole-dns L2 lease -- a DIFFERENT node -- meant tens of seconds. Two of three nodes were single points of failure with wildly different blast radii. Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent only; the mechanism is render.py's, documented in the renderer contract. 1 or absent renders a Deployment exactly as before -- verified byte-identical across the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a multi-replica Deployment never schedules the second pod), per-replica volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever). Pinned by ~20 new --selftest assertions rather than a conformance change. ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default /data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M -- gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn volume held lost+found and 24K of nothing. So every restart discarded all state and re-downloaded 72k gravity domains, and backup.policy has been backing up an empty volume while reporting success. Fixed here because otherwise the new per-replica volumes would hold nothing while state stayed ephemeral -- and because persisting gravity.db is what makes a restart cheap. The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as uid 0 and chowns the directory recursively (no fsGroup needed), and it calls migrate_gravity which creates the database when absent. The gate caught a real regression on the first run: the wave-0 alert unit test fed pihole a Deployment-shaped series. Updated; promtool SUCCESS. Touches no sensitive paths, so this one needs no override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3b1d6ae115 |
feat(pihole): declare the admin password and adlists in git
Prerequisite for running more than one PiHole replica. A second replica with a fresh PVC would have come up serving DNS and blocking nothing, with a different admin password -- answers varying by which pod replied. pihole.toml's own ### CHANGED markers gave the exact inventory: four settings already came from env (declared), one came from the image, and exactly ONE operator-set value lived only on the PVC -- webserver.api.pwhash, a one-way hash nobody could recover. Separately the adlists were undeclared: one list and 72,234 gravity domains that nothing in this repo named. Declares both: - A SOPS secret feeding FTLCONF_webserver_api_password. webserver.api.password takes cleartext and FTL hashes it itself (verified: pihole setpassword IS setFTLConfigValue on that key), so the PVC's pwhash becomes derived state. - A level-triggered postStart hook that reconciles the declared adlist set into gravity.db -- adds, re-enables, and disables (never deletes) undeclared lists, running pihole -g only when something moved. A file would not work: v6's adlists.list import is migration-only, guarded on the DB not existing. The rendered hook was run in the live pod against throwaway copies of gravity.db, exercising all four paths. A fresh DB built from the image's own schema template has ZERO adlists, confirming this is load-bearing rather than cosmetic. Password kept rather than set empty: the pihole Service still publishes port 80 past the oauth2-proxy, closed only by policy, and default-deny has drifted here before. Multi-replica is phase 2 and deliberately not in this branch -- it needs a schema field, a StatefulSet (the config PVC is RWO), anti-affinity and a PDB. Adds a *.sops.yaml -- a sensitive path -- so approval-policy will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fbaeb3e038 |
feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP
Remote Plex clients had no path in, for two measured reasons: Plex advertised no remote connection at all (customConnections and ManualPortMappingMode both ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so 32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this server on a client's connection list. Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS with *.plex.direct and streams bypass the Envoy L7 hop. Adds a hand-written non-catalog unit (the SD schema cannot express a LoadBalancer Service or a pinned lbipam VIP), registered in render.py's EXTRA_UNITS so --emit-waves places it in plex's own wave. externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Known consequence, documented: SNAT puts remote clients inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change. Remote access does NOT work until the operator does two manual steps that cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually specify public port" toggle (no env var exists for it; verified NONE in the image's init surface). runbooks/plex-remote-access.md is the procedure. Touches clusters/** -- a sensitive path -- so the approval-policy stage will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
9a69f6906a |
fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root
hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing mounts cgroup2 there. The agent mounted one from inside its own private cgroup namespace, so socket-LB attached to the agent's own cgroup and host processes got no Service/LB translation. k3s-prod-1 (Gateway L2 lease holder + kube-vip VIP) therefore could not reach the Gateway VIP, and its apiserver's OIDC authenticator has failed since 2026-09-12. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
788eaeae62 |
fix(node-baseline): pin the OIDC issuer to the Gateway VIP on prod nodes
After public DNS moved to the WAN IP (#40), prod nodes resolve keycloak.home.keithjackson.dev to a WAN address the router does not hairpin, so the apiserver could not reach its OIDC issuer. Adds an opt-in node_baseline_hosts_pins /etc/hosts block; prod pins keycloak to GATEWAY_VIP. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
a1e808e697 |
feat(dns): publish Gateway names to the WAN via in-cluster DDNS
Gateway-published records now CNAME to wan.${BASE_DOMAIN}, which a
favonia/cloudflare-ddns Deployment keeps at the home WAN's current public
IP. Operator decision: every Gateway-published name goes public; internal
services (controller: none) remain unpublished.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
||
|
jakman85
|
0d7f94098b | Merge branch 'main' into fix/plex-quicksync-verification | ||
|
jakman85andClaude Opus 5
|
f0994c11a0 |
docs(decisions): record the PR's measured CI result — gitea-settings has drifted
Read PR #39's own gate log rather than the tick (CLAUDE.md §2b). Every stage
this branch could affect is green: decisions-log, lint, l0, render-golden, l1,
and approval-policy (which evaluated for real on the pull_request event and
passed — no sensitive path touched). The one failure is `gitea-settings`:
required_approvals: DECLARED 1 but LIVE is 0
Gitea has DRIFTED from infra/gitea/expected-branch-protection.json
Every other declared setting matched, and one runner is online. So live `main`
currently requires ZERO approvals while the declaration says 1, and because
.gitea/workflows/gate.yml sets REQUIRE_GITEA_SETTINGS=true, that drift blocks
EVERY PR in the repo, not only this one. The check that would notice approvals
are unenforced is the check that is failing.
Not touched here: infra/gitea/** is a sensitive path (non-author approval
required) and outside this branch's documentation-only scope. Re-asserting the
protection or changing the declaration is an operator decision in its own PR;
local branch fix/admin-approval-override (
|
||
|
jakman85andClaude Opus 5
|
b52c8d1af6 |
fix(ci): accept the override from a PR review, and stop misdiagnosing an unreadable comments API
The override did not work on its first real test, and the reason was invisible from the outside. Run 203 on PR #38: a well-formed `/approve-override` posted by jakman85 in the PR comment box, 29 minutes newer than the head commit, and the stage still blocked with: approval-override: COMMENTS_JSON is not a list SETTINGS_API_TOKEN is minted [read:repository, read:admin], and Gitea puts /issues/{n}/comments behind the *issue* scope — so Gitea answered with an error OBJECT, `jq -e .` accepted it as valid JSON, and the stage fell through to 'no approving review'. A permission failure wearing the costume of a missing override. /pulls/{n}/files and /pulls/{n}/reviews are under read:repository, which is why every other part of the same stage worked. 1. The diagnosis no longer lies. COMMENTS_JSON leaves the must-parse-or-block loop and is checked for being an ARRAY; when it is not, the stage prints Gitea's own .message, names the missing scope and says where to post instead. An unreadable source can only cause a false BLOCK, never a false pass, so a loud warning is the honest behaviour and blocking was not. 2. An override is accepted from a PR REVIEW body as well as a comment, so the hatch is not hostage to one API scope — a review of type 'Comment' works on the token CI has today. The gate names which source it used; PENDING (unsubmitted, author-only) reviews are dropped. 3. set-ci-token.sh requests read:issue, making the ordinary comment box work — on a RE-RUN only, since an existing token's scopes cannot be widened. ci/tests/approval-override.test.sh: 23 -> 38 assertions, covering the reviews source, PENDING reviews, an error object from the comments API, 'an unreadable comments source must not suppress a valid review override', latest-wins across both sources, and that the diagnostic quotes Gitea's message. Also measured and recorded: run 203 shows gitea-settings PASS after the operator ran the live PATCH, with block_admin_merge_override still true — the narrow change held, the approval count moved and the gate's teeth did not. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
1677dff568 |
fix(runbooks): Plex hw transcoding WORKS — the dashboard was never the proof
Hardware transcoding is confirmed working on prod (2026-10-02): VAAPI for both decode and encode on /dev/dri/renderD128, with zero-copy, measured in `Plex Media Server.log` in the running pod. The device-plugin mechanism from #37 works end to end. `runbooks/plex-quicksync.md` said the Plex dashboard showing `Transcode (hw)` was "the *only* proof available, and nothing else will ever tell you". That is false: Dashboard -> Activity showed nothing while the feature was fully working, so the page's authoritative check was silent on a healthy system and sent the operator hunting a fault that did not exist. Three inference errors corrected, two kept as named false leads: 1. The LOG is the primary check, not the dashboard. The grep ships in its REDACTING form -- that log carries API URLs with `X-Plex-Token` (CLAUDE.md §6). Also recorded: `Plex Transcoder Statistics.log` is the WRONG file (XML <Variant/> records, no ffmpeg output), and why `grep -a` is required. The dashboard is kept but demoted to best-effort, with its silence stated to prove nothing. 2. An fd snapshot is not evidence. A /proc/*/fd sweep found no holder of renderD128 while hardware transcoding was active -- sessions are short and the scan landed in a gap. The reliable process-level check is the transcoder's argv (-hwaccel, -hwaccel_device, vaapi, renderD128). 3. A missing preference key means nothing. `HardwareAcceleratedCodecs` was absent from all 27 attributes with the toggle on and working, while `HardwareDevicePath` was written and the log said "hardware transcoding: enabled". Only `="0"` is conclusive; the log settles it. Also stale and now fixed: GID 992 is a hostPath-era number. Under the plugin the injected device is root:root, so the image's hook resolves GID 0 and adds `plex` to the root group (`Groups: 0 100 1000`, `groups=1000(plex),0(root),100(users)`). Marked as an inherent property of the plugin's DeviceSpec, not a defect, and explicitly not a reason to reintroduce supplementalGroups (still a no-op -- s6-setuidgid rebuilds the group list from /etc/group). Added a per-title fallback entry (UHD 630 has no AV1 decode), labelled a HYPOTHESIS: the title that fell back was never checked. `runbooks/README.md`'s index entry was stale the same two ways and is corrected. Unchanged on purpose: the "STOP HERE IF PLEX IS Pending" section, the escape hatch, and the no-alert-is-possible reasoning (no metricsEndpoint, no ServiceMonitor, nothing a rule could evaluate). Documentation only -- no catalog/, schemas/, apps/ or platform/ change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8ccfdab180 |
fix(ci): let an administrator merge their own green PR
`required_approvals: 1` on `main` was a lock-out, not a protection. Gitea
refuses to let a PR's author approve their own PR, and `jakman85` is the only
collaborator — so no account in existence could approve a PR the operator
opened, including the PR that would fix that. Measured 2026-10-01 on PR #35:
gate green in 1m41s, reviews `[]`, `required_approvals: 1`, unmergeable.
It was never the documented design either: `ci/approval-policy.sh`'s header and
`ci/gate.sh:194` both already describe the model as "required_approvals drops
to 0 and the GATE decides", and `require-gate.sh`'s own closing note named this
failure and prescribed this remedy. 2026-09-30's entry flagged the 1-vs-0
disagreement as unresolved; this settles it at 0.
Layer 1 — branch protection. required_approvals 1 -> 0 in the declaration, plus
infra/gitea/allow-admin-self-merge.sh to apply it live (target read FROM the
declaration, so the script cannot drift from what the gate checks).
require-gate.sh moves to 0 too; left at 1 it would re-impose the lock-out on its
next run. The narrow choice is deliberate: the other way to unblock an owner is
block_admin_merge_override: false, which would re-open what
block-admin-override.sh was written to close (PRs #5-#9, zero approvals). This
drops the approval COUNT only — block_admin_merge_override and
enable_status_check stay true and are asserted after the PATCH, so a red gate
stays unmergeable by everyone, administrators included.
Layer 2 — ci/approval-policy.sh. Sensitive paths still need a non-author
approval the sole admin cannot get, so an administrator may waive it with an
`/approve-override <reason>` comment: directive must start a line, reason
mandatory, permission re-checked LIVE, invalidated by a new commit, and the
overrider + reason printed in the gate log and job summary. Candidate selection
is the pure, offline ci/approval-override.py (23 unit assertions, all passing);
authorisation stays in the fail-closed stage. Every new failure mode blocks.
The cost, stated not buried: for an administrator, "a change to the gate cannot
be gated by the gate" is now advisory rather than structural. Unavoidable with
one human; what is bought back is an attributed, commit-scoped record instead of
a silent bypass. And the uncomfortable part, accepted as an operator decision:
authorisation is by live permission alone, and the AGENT account `gitea-admin`
also holds `owner` — so an agent can clear its own PR this way. Documented in
approval-policy.json, the stage header, and CLAUDE.md (which tells agents not
to). Narrowing needs no code change: add `override_admins` and the stage already
requires membership.
Adjacent fix, in scope because `make -C ci test` is the only runner for the
suite added here: ci/tests/gate-decision.test.sh hand-maintained a stub list
that "must name EVERY stage gate.sh runs" and omitted `approval-policy` (added
to gate.sh by PR #17), so run_stage hit exit 127 and 15 of 50 assertions failed.
Verified pre-existing on pristine origin/main @
|
||
|
jakman85
|
948875aad3 |
Merge pull request 'Added arr suite ability to connect to plex' (#35) from bugfix/arr-to-plex into main
Reviewed-on: #35 |
||
|
jakman85
|
bdfa72b1c7 |
Merge pull request 'feat(gpu): satisfy hardware.gpu: intel-quicksync with gpu.intel.com/i915 instead of a /dev/dri hostPath (the hostPath was measured not to work: device-cgroup EPERM)' (#37) from feat/quicksync-via-device-plugin into main
Reviewed-on: #37 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
4f16e83eec |
Merge pull request 'feat(obs): make oauth2-proxy auth failures visible — the gates emit nothing' (#29) from feat/oauth2-proxy-auth-failure-visible into main
Reviewed-on: #29 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
0c7e8dd3ba |
Merge pull request 'fix(ci): catalog/render.py's 106 selftest assertions never ran in the gate' (#33) from fix/render-selftest-runs-in-gate into main
Reviewed-on: #33 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
8c12a1e525 |
Merge pull request 'feat(auth): declare API paths per service, so --api-route can ship at all' (#34) from feat/oauth2-proxy-api-route-per-app into main
Reviewed-on: #34 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
9babe5768f |
docs(decisions): handoff for the quicksync mechanism swap, including the regression it trades into
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
8a094c46f7 | Added decision record for allowing arr apps to connect to plex | ||
|
jakman85andClaude Opus 5
|
2983378496 |
chore(decisions): record the CI reading, and why the metrics policy is NOT here
Follow-up to the branch's one substantive commit; no code change.
CI on
|
||
|
jakman85andClaude Opus 5
|
4e1b169256 |
feat(gpu): Intel GPU device plugin v0.37.0, in its own Flux Kustomization
Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured
fix for the Plex hardware-transcoding blocker, and it only makes the resource
EXIST — no consumer is switched over here.
WHY. The `/dev/dri` hostPath merged on main @
|
||
|
jakman85andClaude Opus 5
|
1e71b578d5 |
docs(obs): record why CI is red — approval-policy on the one conformance entry
The gate job reported failure while the local run was rc 3 with nothing failed. Measured cause is a single sensitive path (the KNOWN_EXTERNAL_MONITORS entry) with no non-author approving review, which ci/approval-policy.sh blocks fail-closed by design. Also records the general trap: approval-policy short-circuits to N/A off a pull_request event, so a clean LOCAL gate is a false reassurance for any branch touching a sensitive path. Only the PR run tells the truth. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
99560c2d22 |
feat(auth): declare API paths per service, so --api-route can ship at all
PR #28 fixed the cookie collision, the 5-minute session and the missing metrics
listener in one renderer function, and deliberately declined `--api-route`
because the config model could not express the difference between two apps.
This adds the hook it was missing and sets the flag only where a measurement
says to.
THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to
Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT
`Accept: application/json`, which these apps' JavaScript does not consistently
send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied
silently by the live SSO session, and resolves 200 carrying the app's HTML where
the JavaScript expected JSON. On a write it is worse: a browser following a 302
re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
discarded and the UI reports success.
MEASURED, on two apps and two different paths -- which is the whole argument:
jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs
answered by the proxy and never reaching Jackett (upstream `-`).
radarr 2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh,
0 restarts): 232 unauthenticated requests in the pod's whole life and
ALL 232 are the same call -- POST /signalr/messages/negotiate from a
browser, 302, upstream `-`. A left-open tab whose session the cookie
rename orphaned, polling forever, silently redirected every time and
NEVER re-prompting. The reported symptom, still in production.
So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES
NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a
constant -- and a constant applies one app's measurement to four apps nobody
measured. `auth` was `additionalProperties: false`, so there was no per-app
hook. Adding one named, documented key is the change; the block stays closed.
schemas/servicedefinition.schema.json 1.8.0 -> 1.9.0: optional
`auth.apiRoutes`, a list (radarr needs two, and --api-route is natively
repeatable) of anchored path regexes. Plus the shape invariant that a
non-empty list requires mode: forward-auth, since no sidecar is rendered
for oidc/native/none and declaring it there would be a SILENT NO-OP.
catalog/render.py emits one --api-route per entry, in order, and RAISES on
an unanchored entry. The flag is SINGULAR -- verified against the pinned
v7.8.1 binary's own --help; --api-routes does not exist and the proxy
refuses to start with it.
catalog/{jackett,radarr}.yaml the two measured apps.
catalog/{pihole,sonarr,qbittorrent}.yaml comments recording that they get
NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated
requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no
need, and sonarr being nearly radarr's codebase is what makes the guess
tempting and still a guess.
apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml regenerated. Only TWO
goldens change; pihole/qbittorrent/sonarr are byte-identical to main,
which is the per-app property proving itself on the real tree.
BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing
--api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with
401, so a logged-out operator gets a bare error page and is never offered a
sign-in -- the service looks permanently broken. Each entry must therefore
match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$`: anchored at ^/ then literal path
characters. `^/`, `^/.*`, `.*`, `/api/` and `^/api/|^/` are all rejected, all
five provoked. Alternation is refused rather than supported because the flag is
repeatable, so one entry per route costs nothing.
NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app
(whole-repo scope, so ci/l0.sh's --include-scope union runs it on every
invocation; no bespoke stage, which whole-repo-invariants-wired would fail).
It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it:
that check's subject is cookie IDENTITY and shares neither inputs nor failure
mode with this, so bundling them would make its name a lie and let a future
api-route decision be expressed only by weakening a COOKIE check.
The check exists for one reason the other gates cannot cover: render-golden
compares the render to the goldens, so a RENDERER CONSTANT plus regenerated
goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits
0 -- render-golden is happy -- and this check exits 1 with 15 failures naming
pihole, qbittorrent, radarr and sonarr individually. It also asserts the
anchoring pattern is the SAME in the schema and the renderer, because three
surfaces guard this and if they drift the loosest wins silently.
Tested in both directions on throwaway trees outside the repo: compliant tree
rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four
targeted mutations (hand-edited golden, widened regex, renderer constant,
deleted schema pattern) each firing their own assertion class and nothing else.
ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest()
reached _cookie_name() through a bare `next(genexpr)` with NO default, so
deleting the --cookie-name emission -- the exact mutation that assertion exists
to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected
`SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it
masked every other assertion the same mutation broke (found by Desmond Oyelaran
while wiring the selftest into the render-golden stage; his PR #33 is not in this
base -- origin/main @
|
||
|
jakman85andClaude Opus 5
|
bce101d03f |
feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it
PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
ba79a1b59b |
Merge remote-tracking branch 'origin/main' into feat/oauth2-proxy-auth-failure-visible
# Conflicts: # runbooks/kustomization.yaml |
||
|
jakman85andClaude Opus 5
|
d44490736c |
fix(ci): catalog/render.py's 106 selftest assertions never ran in the gate
`catalog/render.py --selftest` executes 106 assertions on the renderer's output
contract. Nothing in ci/ or .gitea/ had ever invoked it — grepped and confirmed —
so all 106 were dev-time only, and the merge gate could not tell a sound renderer
from a gutted one. Recorded as the top follow-up of
decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.
`ci/render-golden.sh` now runs it and exits 1 when it fails. One file, six lines
of invocation; the rest is the rationale comment.
WHY HERE. The lead was auditability-proofs-producible/check.sh:291, which drives
conformance/lib/{record_run,status}.py --selftest "because nothing else drives
them". That is the wrong hook: the check serves R-SEC-03 and its subject is the
evidence chain — can a forged or stale run become an audit proof? The renderer is
not in that chain, and hanging it there would make the check's name, its intent
annotation and its requirement mapping untrue. This stage's subject already IS
catalog/render.py, so the renderer's own contract test belongs in it.
The two assertions are independent, and that is the point: render-golden catches
golden-vs-renderer drift but is satisfied by ANY renderer whose output was
re-rendered — including one that just lost a contract invariant and had all ten
goldens regenerated in the same commit. That is precisely the hole the follow-up
named. The selftest catches it; render-golden cannot.
It runs before the empty-catalog exit 3 because the selftest renders synthetic
ServiceDefinitions in memory and is catalog-independent — an empty catalog must
not be able to skip it. Failure is a hard rc 1, never a skip: the renderer is
source in this repo, so once python3 and render.py are both present (asserted
above it) there is no legitimate environmental cause for "could not run".
PROVEN TO FAIL, not merely to pass. Against mutated COPIES of the renderer via
the stage's existing RENDER_PY override, so catalog/render.py — another branch's
file this session — was never touched:
* deleting the --cookie-name emission (reverting PR #28's fleet-wide cookie
fix) -> stage rc 1;
* OAUTH2_COOKIE_REFRESH 2m30s -> 10m -> stage rc 1 with the named assertion
"parses to 600.0s, which is not inside (0, 300s)" — one of the two flags the
cookie entry recorded as UNGATED, and one render-golden's diff would miss.
AND PROVEN TO PROPAGATE, in the pinned toolbox: selftest rc 1 -> stage
render-golden FAIL -> `MERGE GATE: BLOCKED`, gate.sh rc 1. Worth provoking: this
repo flattens exit codes, and .gitea/workflows/gate.yml:138 runs
`./ci/gate.sh || [ "$?" -eq 3 ]`, so a failure degraded to a skip would be
swallowed by CI.
render.py itself is unchanged. No conformance/ path is touched, no gate stage is
added, nothing is deployed by this.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
df1fd1b3d1 |
chore(decisions): record the rebase onto main @ 08deb20
The runbooks/README.md conflict was two index entries appended at the same point, not a semantic one. Both kept, in the file's grouped order. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0fee584e3a |
feat(plex): opt into hardware.gpu: intel-quicksync, and write the runbook for the step GitOps cannot reach
Plex has been transcoding in software on hardware that can do it in the iGPU. This opts its ServiceDefinition into the `hardware` capability from #27 and re-renders. catalog/plex.yaml gains four lines of YAML and the comment that explains them: hardware: gpu: intel-quicksync which renders exactly two things into apps/wave-0/plex.yaml and nothing else — a `dri` hostPath volume (/dev/dri, type: DirectoryOrCreate) and a read-write /dev/dri volumeMount on the app container. Whole-tree render diff: 6 added lines, one file. The comment records what is not obvious from the four lines: the device exists on ALL THREE prod nodes with i915 bound (so no nodeSelector is rendered and none is wanted); why `supplementalGroups` would be INERT rather than a hardening (the image's cont-init hook adds the plex user to the device's group inside the container, and s6-setuidgid then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied); and why `DirectoryOrCreate` is load-bearing (apps-wave-0 is wait:true with apps-wave-1 depending on it, so a failed mount would stall the whole app tier for a feature that is not). THE FEATURE IS INERT UNTIL A HUMAN CLICKS A BOX, and that is the part this commit is really about. Mounting the device does not enable hardware transcoding; Plex's `HardwareAcceleratedCodecs` preference does, and it is ABSENT from Preferences.xml — the toggle has never been on. It cannot be automated: a bounded grep across the running image's entire init surface (/etc/cont-init.d, /etc/services.d, /etc/fix-attrs.d, /plex-common.sh, /usr/local/bin) found ZERO PLEX_PREFERENCE* support, and the six setPref keys the image does write are all unrelated. So runbooks/plex-quicksync.md is the procedure, registered in the configMapGenerator (a runbook in git with no generator entry is reviewed, merged, and still a 404) and indexed in runbooks/README.md. It covers the GitOps/manual boundary, the toggle and the real preference keys (including the widely-cited one that does not exist and the upstream misspelling nobody should "fix"), verification via `Transcode (hw)` on the Plex dashboard — stated plainly as the ONLY proof, because Plex renders no ServiceMonitor and an alert could never fire — the device-cgroup failure to expect first with its Intel-device-plugin fallback, and the floating-`:latest` caveat. Stacked on #27 (feat/sd-hardware-capability), which is open and unmerged; that branch's commit is in this range by design and this must merge after it. NOT VERIFIED, and the handoff says so at length: hardware transcoding has not been observed working. Whether containerd's device cgroup lets a non-privileged pod open() a bind-mounted char device is an open question that only deploying settles, and the Plex-side toggle was still unset at the time of writing. A green gate proves the manifest renders; it proves nothing about transcoding. Gate (pinned toolbox, worktree): rc 3 — UNDETERMINED, not green. Passed: decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped (the known local ceiling): spec-immutability (no sibling spec repo), gitea-settings (no SETTINGS_API_TOKEN), conformance (CLUSTER_AVAILABLE unset, so L2/L3 did not run). No fourth skip, nothing failed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f716e9d167 |
chore(decisions): record the rebase onto main @ 08deb20
The render.py conflict was textual adjacency with PR #28's constants block, not a semantic one. Both sides kept. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fa4765bb3e |
feat(schema): declare a host hardware capability, hardware.gpu: intel-quicksync
A ServiceDefinition could not say it needs the host's Intel QuickSync render
device. The schema is `additionalProperties: false`, so there was no authorable
form for it at all, and `apps/` is generated — a hand-edit would be reverted by
the next render.
Adds optional top-level `hardware`, with one optional member `gpu`, an enum
whose only current value is `intel-quicksync` (schema 1.7.0 -> 1.8.0).
The enum is the point. The SD declares WHAT CLASS of device it needs; the
hostPath and the mount live only in catalog/render.py. Migrating to the Intel
GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) is then
a renderer change with zero catalog edits. Letting an SD name `/dev/dri`
directly would look simpler and make that migration a catalog-wide edit, so
both the schema description and the renderer contract say so explicitly.
When `hardware.gpu == intel-quicksync`, r_deployment emits exactly two things,
on the app container only:
- a hostPath volume `dri` -> /dev/dri, type DirectoryOrCreate;
- a read-write volumeMount of it at /dev/dri (VA-API opens the render node
read-write, so deliberately not readOnly).
Mounting the device is sufficient. In particular NO
podSecurityContext.supplementalGroups is emitted, and adding one would be a
regression rather than a hardening. Measured in the running container on
prod-2, 2026-09-30:
PID 1 s6-svscan Uid: 0 Gid: 0 Groups: 0
pid 246 Plex Media Server Uid: 1000 Gid: 1000 Groups: 100 1000
The service is started with `exec s6-setuidgid plex ...`, and the child's group
list is exactly the /etc/group account-database list — root's gid 0, which
PID 1 holds, is gone. That is a setgroups() that REPLACES rather than appends,
so anything Kubernetes supplies is discarded before the app starts. What works
is the image's own /etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner,
which stats each /dev/dri char device, groupadds a matching group inside the
container and usermods the app user into it, every start, nothing hardcoded.
An inert supplementalGroups would have read as the mechanism to whoever
debugged this next, so the absence is documented at the emission site and
asserted by --selftest. Corollary: the hook re-stats on every start, so host
GID drift is self-healing and no GID is pinned anywhere in this repo.
`DirectoryOrCreate` is load-bearing. A node that comes back without the render
device (rebuild, BIOS change, driver regression, hardware swap) must degrade
the workload to software transcoding, not fail the mount: apps-wave-0 is
`wait: true` and apps-wave-1 dependsOn it, so a failed hostPath mount stalls
the whole app tier for a feature that is not load-bearing. /dev is tmpfs, so
the empty directory the kubelet creates leaves nothing durable behind. Its cost
— a missing device is silent, and no metric exists to alert on — is recorded in
the contract rather than papered over with an alert that could not fire.
Also deliberately NOT emitted: no `privileged: true` (Kyverno restrict-host-
namespaces is Enforce and it is not needed), no nodeSelector (placement is a
separate concern with its own justification), no resource requests (there is no
app-container resource render path).
No catalog entry uses the field. Re-rendering `--all catalog -o apps` is a zero
diff, which is the proof the change is inert until opted into.
Self-tests in the existing check(...) style: the positive (volume + mount, and
no securityContext / privileged / nodeSelector), the negative (an SD without
`hardware` gains no volume, no mount, no securityContext — mirroring the
podSecurityContext negative) and a pass-through case (an SD's own
podSecurityContext survives byte-for-byte). Note that NO gate stage runs
`render.py --selftest`; they were run by hand and the gap is recorded in the
handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
08deb208a1 |
Merge pull request 'fix(jackett): root-cause the oauth2-proxy session loss; the fix cannot be scoped to one app' (#30) from fix/jackett-oauth2-proxy-session into main
Reviewed-on: #30 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
b763f11e19 |
Merge pull request 'feat(node-health): probe the Intel render device, read-only' (#31) from feat/node-health-render-device into main
Reviewed-on: #31 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c89537c465 |
docs(auth): record what PR #30's CI actually ran, not what its tick said
Read the commit's /statuses rather than the check mark: gate / gate (pull_request) state=success desc='Successful in 1m41s' gate / conformance (pull_request) state=success desc='Has been skipped' The second is CLAUDE.md 2b's trap in the wild -- a SKIPPED Gitea job reports state=success, so that green tick certifies nothing. It agrees exactly with the local rc 3: the offline L0/L1 subset passed, L2/L3 never ran. Anyone reading the PR's check mark alone would conclude the opposite. Also re-measured branch protection instead of trusting the table in CLAUDE.md: GET /repos/.../branches/main -> protected=true, user_can_push=false. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
7df72db487 |
docs(decisions): record what the CI gate actually reported, not the tick
The PR's required check went green as "Successful in 1m40s" — a real run, not "Has been skipped". But .gitea/workflows/gate.yml:138 is `./ci/gate.sh || [ "$?" -eq 3 ]`, so a green tick means exit 0 OR 3 and cannot distinguish them. Fetched the job log and read the gate's own headline: MERGE GATE: UNDETERMINED, one stage that did not run (spec-immutability). So CI is rc 3 too, and the tick is the tolerance swallowing it. Recorded that plainly rather than leaving "CI green" to be read as rc 0. Also corrects a claim that had gone stale within the same entry: CI ran two stages my worktree run could not, and both passed — gitea-settings (the live Gitea matches infra/gitea/expected-branch-protection.json; runners online 1) and approval-policy with real PR context (no sensitive paths touched, so infra/ansible/** needs no human reviewer). My local run skipped both, and the handoff said branch protection was UNVERIFIED. It is now measured, so saying otherwise would have left a known-false line in the log. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
dc136a4943 |
fix(auth): the forward-auth sidecar also died at 5 minutes and served no metrics
Widens the cookie-collision fix to the whole sidecar contract, because all three
defects live in the same function and land on the same five goldens, so they are one
atomic change or they are five conflicting ones.
Added to _oauth2_sidecar_container(), on top of the cookie-name/cookie-domain fix:
--cookie-refresh=2m30s
The session died FIVE MINUTES after sign-in. Keycloak's homelab realm issues
300-second access tokens (MEASURED on a live session: expires - created =
300.000s exactly, with refresh_token:true on the same line), and
--cookie-refresh defaults to 0 -- which the pinned binary's own help spells
"0 to disable". So the proxy held a refresh token it was never told to spend.
Refresh at HALF the lifetime: two chances to renew before expiry.
This, not the collision, is what dominates the rejection VOLUME -- ~95% of
radarr's 155 rejections in 12h carry no cookie at all.
--metrics-address=0.0.0.0:44180 + a `metrics` container port
Without the flag the /metrics listener never starts, so no per-app sidecar was
serving metrics at all (`ports` was ['http'] on all six). 44180 is the
platform's existing convention, measured not chosen: the shared proxy already
runs the same flag and port with 2 ready endpoints.
PREREQUISITE ONLY -- NOT scraped. A container port is not a Service port, and
each namespace's default-deny baseline is `ingress: []`, so this is unreachable
until a Service port (or PodMonitor), a monitoring allow and a ServiceMonitor
land. Do not read this as "the sidecars have metrics now".
EVERY FLAG VERIFIED AGAINST THE PINNED v7.8.1 BINARY (`--help`), not against docs or
the shared proxy -- which runs v7.15.3 and could not have vouched for v7.8.1:
* --metrics-address exists in 7.8.1, so this cannot crashloop the pods
* --cookie-refresh ... "0 to disable" confirms the default
* --cookie-name ... (default "_oauth2_proxy") -- this confirms from the
IMPLEMENTATION the constant the new L0 check is built on
* --api-route is SINGULAR; the plural does not exist and the proxy refuses to start
* --session-cookie-minimal exists and is mutually exclusive with --cookie-refresh,
so dropping --pass-access-token is a least-privilege argument, never a size fix
NOT done, deliberately: --api-route=^/api/ renderer-wide. It was measured on jackett
and PiHole does not serve /api/, so a renderer constant would apply one app's path
shape to five. It needs a per-app hook the `auth` schema forbids
(additionalProperties: false) -- a schema change, separate PR.
Also records the mechanism that made this INTERMITTENT, as a renderer comment, because
nobody would reconstruct it from the flags: the loser does not just fail to read the
cookie, it calls ClearSession, which expires the cookie on its own --cookie-domain --
the whole zone. The *arr UIs poll (/api/v3/command, /api/v3/queue/details,
/signalr/negotiate), so every poll carrying an undecryptable cookie clears it again:
A LEFT-OPEN RADARR TAB DELETED JACKETT'S COOKIE EVERY FEW SECONDS, from another
namespace, with no user action. That is why signing in again never helped for longer
than the gap between two polls.
Crafty's counter-evidence resolved: the 81:1 ratio was a 24h window straddling a
13h52m-old pod. Bounded to the fixed process, crafty's ENTIRE life is 3 rejections in
6743 log lines, with ZERO `removing session` and ZERO `cookie signature not valid`
against unfixed radarr's 8 and 8. The eviction fingerprint is gone where the fix is
deployed. It logged 0 AuthSuccess too, so that is absence-of-the-failure-mode rather
than a session watched working -- stated as such in the handoff.
Renderer selftest gains six assertions, including a --cookie-refresh check that PARSES
the duration and requires 0 < refresh < 300s (so 2m30s/150s/2.5m all stay green while a
silent widening past the token lifetime does not). NOTE, and it is in the handoff as the
top follow-up: NOTHING IN ci/ RUNS `render.py --selftest`, so those assertions are
dev-time only. conformance/tests/l0-static/auditability-proofs-producible/check.sh:291
already drives other tools' selftests for exactly this reason and is the one-line fix.
Gate (toolbox, worktree shim with the spec repo mounted at /home-network-k3s): rc 3
UNDETERMINED. Green: spec-immutability, decisions-log, lint, l0, render-golden, l1,
approval-policy. Skipped: gitea-settings (no local token), conformance (declared).
The new L0 check PASSES on this branch -- all five apps fixed in one commit.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
a78f010368 |
feat(node-health): probe the Intel render device, read-only
The prod nodes already have a working Intel render device — measured by hand on
all three on 2026-09-30. Nothing here installs, loads or changes anything; the
point is that the measurement now re-runs on every health run instead of decaying
silently in a handoff note.
The failure being hunted is silent by construction: Plex hardware-transcodes
through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS
change that disables the onboard IGD, a kernel that binds a different DRM driver)
Plex falls back to software transcoding on its own and the only symptom is a hot
CPU.
All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*,
grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver
symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class
devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the
vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are
probed but never depended on — neither is in node_baseline's package floor.
Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:
graphics.render_device the outcome — a usable character-special renderD* node.
FAIL only when a display device IS on the PCI bus and
there is still no render node. SKIP when the machine has
no display device at all (a VM guest), which is why the
PCI scan is separate from the DRM scan: /sys/class/drm/
card* exists only once a driver has BOUND.
graphics.drm_driver what is bound, against the new
node_health_graphics_expected_driver default (i915), so
the comparison is data and not task logic. `xe` bound in
place of i915 is a WARN, not a FAIL: the device works but
the userspace media stack differs.
graphics.media_stack INFO only, deliberately. GuC/HuC off (i915.enable_guc=0)
is an accepted state that costs low-power H.264 encode
quality only, and no host vainfo is not a defect at all
because Plex ships its own libva inside its container.
The render node's owning group and GID are REPORTED, never asserted: `render` is a
systemd dynamic allocation rather than a Debian static gid, so a rebuild can
legitimately move it off the 992 measured today, and Plex's container entrypoint
stats the device and joins whichever group owns it. Drift stays visible without
being an error.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|