ae772e050dc468f6829671fa5bcc1fd672ab178f
65
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 renovate-bot
|
640edb758d | Merge pull request 'chore(deps): Update plexinc/pms-docker Docker tag to v1.43.6.11085-d1e52e169' (#111) from renovate/plexinc-pms-docker-1.x into main | ||
|
renovate-bot
|
dd9d8da86a | chore(deps): Update plexinc/pms-docker Docker tag to v1.43.6.11085-d1e52e169 | ||
|
renovate-bot
|
c3ce4db767 | chore(deps): Update lscr.io/linuxserver/jackett Docker tag to v0.24.3009-ls26 | ||
|
renovate-bot
|
9c0292001e | chore(deps): Update pihole/pihole Docker tag to v2026.10.1 | ||
  jakman85andClaude Opus 5.5
|
183bca6897 |
feat(images): pin pihole, jackett and plex to the digests already running; Renovate follows them
pihole/pihole (no tag) -> 2026.10.0 sha256:d3413471... lscr.io/linuxserver/jackett:latest -> v0.24.2865-ls7 sha256:4fcf3cc2... plexinc/pms-docker:latest -> 1.43.4.10903-e5521bd8c sha256:e0ab2739... Each tag was resolved against its registry and matched the pods' imageID, so the image is unchanged (the pod template string does change, so Flux rolls the three workloads). renovate.json5: regex versioning for jackett (-lsN) and plex (build-hex); jackett is looked up on ghcr.io via overridePackageName, because lscr.io is capped at dockerMaxPages=20 pages and jackett's 32k tags put the running one at #31,980 (dry run proposed nothing without it); jackett weekly (13 builds in 12 days); pihole deliberately unheld, year rollover = major. Stale "publishes only latest" prose fixed in renovate.json5 and three runbooks. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fda5c31a94 |
fix(pihole): make the split-horizon answer cacheable, or one blip latches the WAN IP for 5 minutes
Operator reported DNS flapping between the cluster ingress and the router's public IP. PiHole is not at fault -- both replicas answer 192.168.2.20 consistently (6/6 via the VIP, 8/8 from a client). The bug is an asymmetry in CACHEABILITY that we created. Measured: pihole 192.168.2.23 -> 192.168.2.20 TTL=0 public 1.1.1.1 -> wan.home.keithjackson.dev TTL=300 router 192.168.2.1 -> wan.... TTL=153 -> 148 -> 143 (decrementing = caching) dnsmasq answers `address=` entries with TTL 0 unless told otherwise. The router forwards LAN queries to PiHole and caches the result, so the CORRECT answer is uncacheable -- re-queried on every lookup, maximum exposure to any gap -- while the WRONG answer caches for 300s. The instant the router falls back to upstream (standard forwarder behaviour when the target does not answer) it latches the WAN address for the whole LAN for five minutes, then expires, re-queries PiHole, gets TTL 0, and is immediately re-exposed. So ONE sub-second PiHole gap -- pod restart, L2 lease move, ARP flap, kured reboot -- becomes FIVE MINUTES of the house resolving to the WAN IP. That asymmetry, not the gap, is why it reads as "flapping". Fix: local-ttl=60 prepended to FTLCONF_misc_dnsmasq_lines. dnsmasq applies local-ttl to replies made from configuration, so one directive covers all five address= names. 60s rides out a pod restart and a lease re-election while a real VIP change still propagates inside a minute; these VIPs are pinned and static. apps/wave-0/pihole.yaml regenerated with the canonical render command (--all catalog -o apps --wave-layout --no-image-automation, NOT --emit-waves); exactly two files changed and the rendered diff is the single local-ttl prefix. UNAPPLIED. Does NOT remove the trigger: .23 still has two independently-elected L2 leases (as do crafty .22 and unifi .21), which the operator has parked. If the TTL is still 0 after merge, suspect FTL silently rejecting the passthrough line -- see the handoff. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
0132c680b3 | Update apps/wave-0/plex.yaml | ||
|
jakman85andClaude Opus 5
|
c5b5b8672a |
feat(pm-agent): an in-cluster AI Program Manager that turns Gitea issues into pull requests
A service that reads this repository's Gitea issues, decides what to do about each thread as the Program Manager persona, and runs one worker agent per assignment that authors a change, runs the merge gate and opens a pull request. It reacts to issue creation and to comments; the assignment and all correspondence travel through the issue thread and the PR. It authenticates with an Anthropic subscription token. It never pushes to main and never merges. Three choices carry the design, with alternatives recorded in docs/plans/: - it POLLS rather than receiving webhooks (Gitea's webhook SSRF guard, lost deliveries, and no inbound reach needed at all); - it keeps NO state: the trigger is "the newest thing in this thread was not written by me", so the issue thread is the state — no PVC, no backup obligation; - the model holds NO tools: its reply is one JSON object this code validates, so nothing in the path executes text. The boundary is platform/kyverno/admission/pm-agent-job-shape.yaml (Enforce, failurePolicy: Fail), not the RBAC: Kubernetes places no `use` check on serviceAccountName, so "may create Jobs" is "may run arbitrary code as the namespace's most privileged identity". Six rules admit only the committed shape, and the worker identity is bound to nothing with no token mounted, so there is nothing to escalate to. 28/28 offline Kyverno assertions. Nothing has run: the image has never been built or pushed, and both the image digest and the two credentials are deliberate placeholders. See the handoff entry's "What I did NOT verify". Refs CLAUDE.md §1/§3/§6, R-SEC-02, R-GITOPS-02. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
874c6703f0 |
fix(plex): advertise a resolver-independent plex.direct name again (LAN-wide outage)
I took Plex down for the whole LAN. ADVERTISE_IP was http://plex-vip.${BASE_DOMAIN}:32400 -- a name that exists ONLY in PiHole's split-horizon answers and in no public zone. A Plex client resolves the advertised host ITSELF, with whatever resolver it uses, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default, phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve the name, the router answered the failed lookup with ITSELF, and they all landed on its web UI -- hence the %D certificate. Measured 2026-10-05: 192.168.2.1:443 -> C=ZZ, O=%D342e7e6b, CN=%D (router, LAN side) 136.51.115.252:443 -> the same %D cert (router, WAN side) pihole-0 AND pihole-1 -> plex -> .20, plex-vip -> .24 (both CORRECT) gateway .20:443 -> CN=*.home.keithjackson.dev, valid http://192.168.2.24:32400/identity -> 200 Plex's published connections: all port 32400, nothing on 443 The DNS was right; the DEPENDENCY on it was the bug. And since nothing Plex advertises listens on 443, the clients had left the Plex path entirely. Fix: <dashed-ip>.<serverhash>.plex.direct, which is RESOLVER-INDEPENDENT -- a public zone returning the private address embedded in the label, so every resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers it with 192.168.2.24. No split-horizon entry, no Cloudflare record and no certificate work needed: the wildcard cert Plex already serves on 32400 covers that name. Re-verified today with full validation (no curl -k): 200, TLS verified. The nicer hostname cannot be made correct for clients that bypass PiHole -- a public record resolves to the WAN address and there is no NAT hairpin, so it is right only for PiHole users and wrong for the rest. There is no configuration of it that is right for both. PiHole's plex-vip answer is KEPT; it is just not what Plex advertises. Cost restated, not buried: the value embeds the VIP and the cert hash as a literal and nothing checks they still agree. Accepted because an unresolvable name is a worse failure -- silent, LAN-wide, and it points users at the router. UNVERIFIED: unreconciled, and the client-side resolver path is INFERRED -- my public-resolver probe was broken (NXDOMAIN even for a name that resolves) so I discarded it. The chain is the only explanation consistent with the measurements, but the NXDOMAIN-hijack step was not observed directly. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a0e6ef145f |
fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http
I broke the app tier with #48. Two faults, both mine, both measured after merge: 1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key Secrets (plex-vip-<random>) were denied by secret-provenance/secrets-must-be-flux-managed, which matches every Secret in a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes only those labelled secret-managed-by=controller or carrying a Flux label. `plex` HAS that label; `gateway` -- where the working wildcard cert lives -- does NOT. That asymmetry is why the existing Certificate was never affected and mine was, and I did not check it. secretTemplate cannot fix it: those Secrets come from the keymanager controller, not the Certificate template. So no CertificateRequest, Order or Challenge was ever created. 2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so Flux waits on the Certificate OBJECT: timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress'] apps-wave-0 never went Ready; wave-1/-2 and everything downstream of platform-config sat in DependencyNotReady behind it. I guarded the pod and missed the object. That distinction is the whole bug. Third, independent: the advertised URL was https with no servable certificate -- Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip -- the exact failure mode of the outage earlier the same day. So: delete the Certificate (pruning it clears the health gate) and advertise http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there. http is explicitly temporary and only tolerable because remote access does not work yet; the restore condition is written at the call site. DELETED, not merely unregistered: my first attempt kept the file as "tracked but not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02) fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization + the https flip, as ONE change) is recorded in catalog/plex.yaml. Kept and verified working from #48: the hostname design, PiHole's split-horizon answer, and the postStart hook -- LanNetworksBandwidth is now set live. Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
c163deaef9 |
Merge pull request 'feat(plex): one advertised hostname for local+remote, its own LE cert, and a LAN definition so LAN clients stop reading as remote' (#48) from feat/plex-vip-hostname-and-lan-networks into main
Reviewed-on: #48 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b156ee6cfa |
fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable
platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that have never once scraped. Measured on prod 2026-10-04: all six read health=down with lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the configured scrapeTimeout, i.e. packets DROPPED rather than refused. Cause: every app namespace ships a default-deny NetworkPolicy with policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180. Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr, qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object. render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this catalog declares one. It was wired in at the call site and had never emitted a single object on any branch: correct, and entirely latent. It now also fires for the forward-auth gate sidecar's own metrics port, keyed on _needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the sidecar exists. Keying on observability.metrics instead would let metrics: false produce a discovered-but-blocked target: the same "two halves authored apart" failure this function exists to prevent. Rendered here rather than in platform/oauth2-proxy because that tier would have to name app namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering per-app dissolves it. Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring. apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved. Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an object necessarily drifts the goldens that pin the renderer's output. The first gate run caught container-workload-render and egress-class-render as FAIL with exactly this CNP as the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the evidence that the regen is faithful and not a rubber stamp. That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author approving review. The local gate cannot see that — offline it reports approval-policy green (N/A). Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements. Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0aa1a0a78a |
feat(plex): one advertised hostname for local+remote, its own LE cert, and a LAN definition
Two symptoms after the publishing fix landed:
1. LAN clients were reported as Remote. Measured root cause:
LanNetworksBandwidth was ABSENT, so Plex had no notion of a local
network and classified everything WAN -- 468 requests logged
"Allowed Network (WAN)", ZERO "(LAN)", including Seerr from inside the
cluster. allowedNetworks only waives AUTH; it does not make a client
local. Two settings, and only one has an env hook.
2. The advertised URL hard-coded both the VIP and the server's cert hash
into a cluster-var -- two values nothing kept in sync.
Operator design: ONE hostname, plex-vip.${BASE_DOMAIN}, split-horizon --
PiHole answers it with ${PLEX_L4_VIP} on the LAN, Cloudflare answers it
publicly with the WAN address NATed to the same VIP.
- catalog/plex.yaml: ADVERTISE_IP -> https://plex-vip.${BASE_DOMAIN}:32400,
plus a level-triggered postStart hook (pihole/qbittorrent idiom) that
re-asserts LanNetworksBandwidth and the three customCertificate* prefs
every start, converting PEM -> PKCS#12 so cert renewals need no extra part.
- apps/wave-0/plex-cert.yaml: cert-manager Certificate for exactly that
name from letsencrypt-prod, so the advertised https URL is servable.
- schema 1.11.0: `optional` on secretKeyRef/configMapKeyRef. Load-bearing --
the cert is issued INTO the plex namespace, which the rendered plex.yaml
itself creates, so it lands in the same reconcile; hard refs would have
held Plex in CreateContainerConfigError and, since apps-wave-0 is
wait:true, stalled the whole app tier on a slow DNS01 challenge.
- catalog/pihole.yaml: the plex-vip split-horizon answer. Required --
measured, the name resolved to the Gateway VIP where nothing listens
on 32400 (http=000).
- PLEX_ADVERTISE_URL deleted; the hostname removes the cert-hash fragility.
Corrects my own earlier claim that a SAN match is REQUIRED: it is not
(httpsRequired=0, secureConnections defaults to Preferred, so clients can
fall back to cleartext). A matching cert keeps the remote leg ENCRYPTED.
Hence: obtain a real cert for the advertised name rather than pick the name
to suit Plex's built-in one.
UNVERIFIED, and stated plainly in the handoff: nothing here has run. Cert
issuance for this name is inferred from the wildcard's success, and whether
Plex accepts an OpenSSL-3 PKCS#12 bundle is untested -- the largest risk.
Touches clusters/** -> sensitive paths -> needs a non-author approval or the
operator's /approve-override; deliberately not self-posted (CLAUDE.md 1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
1f4552d409 |
fix(plex): advertise the plex.direct name, not the bare VIP (server vanished from the plex.tv account)
customConnections held two hand-typed URLs and neither could ever complete, so Plex had nothing publishable and the server disappeared from the account: - https://192.168.2.24:32400 -- the server's cert is *.130072f20c334d6c8f63d0dfef87309b.plex.direct with NO IP SAN, so bare-IP https fails TLS for every client. Measured: a continuous storm of "CERT: incomplete TLS handshake ... certificate unknown" from real LAN clients arriving through the plex-l4-tcp VIP. - https://plex.home.keithjackson.dev:32400 -- nothing listens; the Gateway serves that host on 443 (measured http=000 against 192.168.2.20:32400). With no completable connection plex.tv published nothing ("Mapped - Not Published"), Plex fell back to Relay, and clients reported "not authorized". Fix: advertise <dashed-ip>.<serverhash>.plex.direct -- public DNS that resolves to the embedded address, so it points at the LAN VIP and matches the wildcard cert. Verified with full TLS validation (no curl -k): 200 + verify OK, where the bare-IP form fails to connect. Declared as ADVERTISE_IP so git owns the value instead of a web form: the image's 40-plex-first-run writes it into customConnections on every pod start, outside the claim guard. Does NOT fix off-LAN access -- the advertised address is RFC1918 and the WAN tcp/32400 forward remains unverified. ManualPortMappingMode=1 with an absent port is left as-is (no GitOps path exists for it). Touches clusters/** -> sensitive paths -> needs the operator's /approve-override, deliberately not self-posted (CLAUDE.md 1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
028853c87c |
feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes
Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO force-detach), while losing the node holding the pihole-dns L2 lease -- a DIFFERENT node -- meant tens of seconds. Two of three nodes were single points of failure with wildly different blast radii. Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent only; the mechanism is render.py's, documented in the renderer contract. 1 or absent renders a Deployment exactly as before -- verified byte-identical across the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a multi-replica Deployment never schedules the second pod), per-replica volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever). Pinned by ~20 new --selftest assertions rather than a conformance change. ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default /data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M -- gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn volume held lost+found and 24K of nothing. So every restart discarded all state and re-downloaded 72k gravity domains, and backup.policy has been backing up an empty volume while reporting success. Fixed here because otherwise the new per-replica volumes would hold nothing while state stayed ephemeral -- and because persisting gravity.db is what makes a restart cheap. The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as uid 0 and chowns the directory recursively (no fsGroup needed), and it calls migrate_gravity which creates the database when absent. The gate caught a real regression on the first run: the wave-0 alert unit test fed pihole a Deployment-shaped series. Updated; promtool SUCCESS. Touches no sensitive paths, so this one needs no override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3b1d6ae115 |
feat(pihole): declare the admin password and adlists in git
Prerequisite for running more than one PiHole replica. A second replica with a fresh PVC would have come up serving DNS and blocking nothing, with a different admin password -- answers varying by which pod replied. pihole.toml's own ### CHANGED markers gave the exact inventory: four settings already came from env (declared), one came from the image, and exactly ONE operator-set value lived only on the PVC -- webserver.api.pwhash, a one-way hash nobody could recover. Separately the adlists were undeclared: one list and 72,234 gravity domains that nothing in this repo named. Declares both: - A SOPS secret feeding FTLCONF_webserver_api_password. webserver.api.password takes cleartext and FTL hashes it itself (verified: pihole setpassword IS setFTLConfigValue on that key), so the PVC's pwhash becomes derived state. - A level-triggered postStart hook that reconciles the declared adlist set into gravity.db -- adds, re-enables, and disables (never deletes) undeclared lists, running pihole -g only when something moved. A file would not work: v6's adlists.list import is migration-only, guarded on the DB not existing. The rendered hook was run in the live pod against throwaway copies of gravity.db, exercising all four paths. A fresh DB built from the image's own schema template has ZERO adlists, confirming this is load-bearing rather than cosmetic. Password kept rather than set empty: the pihole Service still publishes port 80 past the oauth2-proxy, closed only by policy, and default-deny has drifted here before. Multi-replica is phase 2 and deliberately not in this branch -- it needs a schema field, a StatefulSet (the config PVC is RWO), anti-affinity and a PDB. Adds a *.sops.yaml -- a sensitive path -- so approval-policy will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fbaeb3e038 |
feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP
Remote Plex clients had no path in, for two measured reasons: Plex advertised no remote connection at all (customConnections and ManualPortMappingMode both ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so 32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this server on a client's connection list. Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS with *.plex.direct and streams bypass the Envoy L7 hop. Adds a hand-written non-catalog unit (the SD schema cannot express a LoadBalancer Service or a pinned lbipam VIP), registered in render.py's EXTRA_UNITS so --emit-waves places it in plex's own wave. externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Known consequence, documented: SNAT puts remote clients inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change. Remote access does NOT work until the operator does two manual steps that cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually specify public port" toggle (no env var exists for it; verified NONE in the image's init surface). runbooks/plex-remote-access.md is the procedure. Touches clusters/** -- a sensitive path -- so the approval-policy stage will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
948875aad3 |
Merge pull request 'Added arr suite ability to connect to plex' (#35) from bugfix/arr-to-plex into main
Reviewed-on: #35 |
||
|
jakman85
|
bdfa72b1c7 |
Merge pull request 'feat(gpu): satisfy hardware.gpu: intel-quicksync with gpu.intel.com/i915 instead of a /dev/dri hostPath (the hostPath was measured not to work: device-cgroup EPERM)' (#37) from feat/quicksync-via-device-plugin into main
Reviewed-on: #37 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
78bf68493d |
feat(gpu): satisfy hardware.gpu: intel-quicksync with gpu.intel.com/i915, not a /dev/dri hostPath
The hostPath mechanism was MEASURED NOT TO WORK. In the live Plex pod on
k3s-prod-2, 2026-10-01: the device node was VISIBLE in the container
(`crw-rw---- root video2 226,128 renderD128`) and could not be opened --
`uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE
CGROUP, not EACCES from file permissions, proven three ways: root on the HOST
opens the same device rc=0; the node is unified cgroup2fs; and the container's
runc config.json read `linux.resources.devices: [{allow: False, access: 'rwm'}]`
with `linux.devices: None` -- one deny-all rule, no allow entry for major 226.
A hostPath is a filesystem operation: it binds the device node in but adds no
cgroup allow rule, because Kubernetes cannot infer one from a volume.
The Intel GPU device plugin produces that allow rule
(`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`).
It shipped as #36 and is live: all three prod nodes advertise
`gpu.intel.com/i915: 1`, DaemonSet 3/3/3 (measured 2026-10-01).
ZERO ServiceDefinition field changes, which was the entire point of making the
field an enum of capability classes rather than a device path. `catalog/plex.yaml`'s
`hardware:` block comes through byte-identical; only its comment changes, because
it restated the old mechanism. Contract invariant 8 holds, now demonstrated rather
than claimed.
- catalog/render.py: `DRI_VOLUME_NAME`/`DRI_DEVICE_DIR` retired for
`QUICKSYNC_RESOURCE`; the hardware branch now sets
`resources.limits["gpu.intel.com/i915"] = 1` on the APP container only (never a
sidecar) and emits no volume, no mount, no securityContext, no nodeSelector.
`limits` not `requests` is mandatory: Kubernetes mirrors requests from limits,
and a requests-only extended resource is invalid. This is the only app-container
`resources` render path and is deliberately narrow -- no general CPU/memory
feature was invented.
- selftests: resources dict asserted EXACTLY; hostPath/mount asserted ABSENT; the
no-`hardware` negative extended to `resources`; the podSecurityContext
pass-through and the no-synthesised-securityContext assertions kept; one new
assertion that the limit never reaches the oauth2-proxy sidecar. Mutation-tested:
8/8 mutants caught (requests-instead-of-limits, emission dropped, hostPath
re-added, emitted unconditionally, supplementalGroups synthesised, wrong resource
name, quantity as a string, limit leaked onto the sidecar).
- apps/wave-0/plex.yaml: six hostPath lines removed, a three-line resources block
added. Only golden touched; keycloak/vm-demo unchanged.
THE REGRESSION, STATED RATHER THAN SOLD. Before: device missing ->
`DirectoryOrCreate` mounted an empty dir -> Plex ran and SILENTLY
software-transcoded (invisible, harmless to everything else). After: resource
unavailable (plugin broken, removed, or not yet reconciled) -> Plex is
UNSCHEDULABLE, and `apps-wave-0` is `wait: true` with `apps-wave-1` dependsOn it,
so that STALLS THE APP TIER. Unavoidable: an extended resource is a hard
scheduling constraint, and it is also the only thing that produces the cgroup
allow rule. There is no graceful-degradation variant. Plex's schedulability is now
coupled to the intel-gpu-plugin DaemonSet. Escape hatch, one field and no code
change: delete the `hardware:` block from catalog/plex.yaml and re-render.
Documented in the contract, the runbook (its own top-of-page section and a
"Plex will not schedule" procedure) and the handoff.
HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING. The decisive open() test
cannot be re-run until this merges and Flux reconciles, and the Plex-side
`HardwareAcceleratedCodecs` toggle is still unset. A green gate proves the
manifest renders.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
9048c2f5e8 | Forgot to render the golden from the catalog -- fixed | ||
|
jakman85andClaude Opus 5
|
99560c2d22 |
feat(auth): declare API paths per service, so --api-route can ship at all
PR #28 fixed the cookie collision, the 5-minute session and the missing metrics
listener in one renderer function, and deliberately declined `--api-route`
because the config model could not express the difference between two apps.
This adds the hook it was missing and sets the flag only where a measurement
says to.
THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to
Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT
`Accept: application/json`, which these apps' JavaScript does not consistently
send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied
silently by the live SSO session, and resolves 200 carrying the app's HTML where
the JavaScript expected JSON. On a write it is worse: a browser following a 302
re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
discarded and the UI reports success.
MEASURED, on two apps and two different paths -- which is the whole argument:
jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs
answered by the proxy and never reaching Jackett (upstream `-`).
radarr 2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh,
0 restarts): 232 unauthenticated requests in the pod's whole life and
ALL 232 are the same call -- POST /signalr/messages/negotiate from a
browser, 302, upstream `-`. A left-open tab whose session the cookie
rename orphaned, polling forever, silently redirected every time and
NEVER re-prompting. The reported symptom, still in production.
So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES
NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a
constant -- and a constant applies one app's measurement to four apps nobody
measured. `auth` was `additionalProperties: false`, so there was no per-app
hook. Adding one named, documented key is the change; the block stays closed.
schemas/servicedefinition.schema.json 1.8.0 -> 1.9.0: optional
`auth.apiRoutes`, a list (radarr needs two, and --api-route is natively
repeatable) of anchored path regexes. Plus the shape invariant that a
non-empty list requires mode: forward-auth, since no sidecar is rendered
for oidc/native/none and declaring it there would be a SILENT NO-OP.
catalog/render.py emits one --api-route per entry, in order, and RAISES on
an unanchored entry. The flag is SINGULAR -- verified against the pinned
v7.8.1 binary's own --help; --api-routes does not exist and the proxy
refuses to start with it.
catalog/{jackett,radarr}.yaml the two measured apps.
catalog/{pihole,sonarr,qbittorrent}.yaml comments recording that they get
NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated
requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no
need, and sonarr being nearly radarr's codebase is what makes the guess
tempting and still a guess.
apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml regenerated. Only TWO
goldens change; pihole/qbittorrent/sonarr are byte-identical to main,
which is the per-app property proving itself on the real tree.
BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing
--api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with
401, so a logged-out operator gets a bare error page and is never offered a
sign-in -- the service looks permanently broken. Each entry must therefore
match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$`: anchored at ^/ then literal path
characters. `^/`, `^/.*`, `.*`, `/api/` and `^/api/|^/` are all rejected, all
five provoked. Alternation is refused rather than supported because the flag is
repeatable, so one entry per route costs nothing.
NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app
(whole-repo scope, so ci/l0.sh's --include-scope union runs it on every
invocation; no bespoke stage, which whole-repo-invariants-wired would fail).
It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it:
that check's subject is cookie IDENTITY and shares neither inputs nor failure
mode with this, so bundling them would make its name a lie and let a future
api-route decision be expressed only by weakening a COOKIE check.
The check exists for one reason the other gates cannot cover: render-golden
compares the render to the goldens, so a RENDERER CONSTANT plus regenerated
goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits
0 -- render-golden is happy -- and this check exits 1 with 15 failures naming
pihole, qbittorrent, radarr and sonarr individually. It also asserts the
anchoring pattern is the SAME in the schema and the renderer, because three
surfaces guard this and if they drift the loosest wins silently.
Tested in both directions on throwaway trees outside the repo: compliant tree
rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four
targeted mutations (hand-edited golden, widened regex, renderer constant,
deleted schema pattern) each firing their own assertion class and nothing else.
ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest()
reached _cookie_name() through a bare `next(genexpr)` with NO default, so
deleting the --cookie-name emission -- the exact mutation that assertion exists
to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected
`SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it
masked every other assertion the same mutation broke (found by Desmond Oyelaran
while wiring the selftest into the render-golden stage; his PR #33 is not in this
base -- origin/main @
|
||
|
jakman85andClaude Opus 5
|
0fee584e3a |
feat(plex): opt into hardware.gpu: intel-quicksync, and write the runbook for the step GitOps cannot reach
Plex has been transcoding in software on hardware that can do it in the iGPU. This opts its ServiceDefinition into the `hardware` capability from #27 and re-renders. catalog/plex.yaml gains four lines of YAML and the comment that explains them: hardware: gpu: intel-quicksync which renders exactly two things into apps/wave-0/plex.yaml and nothing else — a `dri` hostPath volume (/dev/dri, type: DirectoryOrCreate) and a read-write /dev/dri volumeMount on the app container. Whole-tree render diff: 6 added lines, one file. The comment records what is not obvious from the four lines: the device exists on ALL THREE prod nodes with i915 bound (so no nodeSelector is rendered and none is wanted); why `supplementalGroups` would be INERT rather than a hardening (the image's cont-init hook adds the plex user to the device's group inside the container, and s6-setuidgid then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied); and why `DirectoryOrCreate` is load-bearing (apps-wave-0 is wait:true with apps-wave-1 depending on it, so a failed mount would stall the whole app tier for a feature that is not). THE FEATURE IS INERT UNTIL A HUMAN CLICKS A BOX, and that is the part this commit is really about. Mounting the device does not enable hardware transcoding; Plex's `HardwareAcceleratedCodecs` preference does, and it is ABSENT from Preferences.xml — the toggle has never been on. It cannot be automated: a bounded grep across the running image's entire init surface (/etc/cont-init.d, /etc/services.d, /etc/fix-attrs.d, /plex-common.sh, /usr/local/bin) found ZERO PLEX_PREFERENCE* support, and the six setPref keys the image does write are all unrelated. So runbooks/plex-quicksync.md is the procedure, registered in the configMapGenerator (a runbook in git with no generator entry is reviewed, merged, and still a 404) and indexed in runbooks/README.md. It covers the GitOps/manual boundary, the toggle and the real preference keys (including the widely-cited one that does not exist and the upstream misspelling nobody should "fix"), verification via `Transcode (hw)` on the Plex dashboard — stated plainly as the ONLY proof, because Plex renders no ServiceMonitor and an alert could never fire — the device-cgroup failure to expect first with its Intel-device-plugin fallback, and the floating-`:latest` caveat. Stacked on #27 (feat/sd-hardware-capability), which is open and unmerged; that branch's commit is in this range by design and this must merge after it. NOT VERIFIED, and the handoff says so at length: hardware transcoding has not been observed working. Whether containerd's device cgroup lets a non-privileged pod open() a bind-mounted char device is an open question that only deploying settles, and the Plex-side toggle was still unset at the time of writing. A green gate proves the manifest renders; it proves nothing about transcoding. Gate (pinned toolbox, worktree): rc 3 — UNDETERMINED, not green. Passed: decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped (the known local ceiling): spec-immutability (no sibling spec repo), gitea-settings (no SETTINGS_API_TOKEN), conformance (CLUSTER_AVAILABLE unset, so L2/L3 did not run). No fourth skip, nothing failed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
dc136a4943 |
fix(auth): the forward-auth sidecar also died at 5 minutes and served no metrics
Widens the cookie-collision fix to the whole sidecar contract, because all three
defects live in the same function and land on the same five goldens, so they are one
atomic change or they are five conflicting ones.
Added to _oauth2_sidecar_container(), on top of the cookie-name/cookie-domain fix:
--cookie-refresh=2m30s
The session died FIVE MINUTES after sign-in. Keycloak's homelab realm issues
300-second access tokens (MEASURED on a live session: expires - created =
300.000s exactly, with refresh_token:true on the same line), and
--cookie-refresh defaults to 0 -- which the pinned binary's own help spells
"0 to disable". So the proxy held a refresh token it was never told to spend.
Refresh at HALF the lifetime: two chances to renew before expiry.
This, not the collision, is what dominates the rejection VOLUME -- ~95% of
radarr's 155 rejections in 12h carry no cookie at all.
--metrics-address=0.0.0.0:44180 + a `metrics` container port
Without the flag the /metrics listener never starts, so no per-app sidecar was
serving metrics at all (`ports` was ['http'] on all six). 44180 is the
platform's existing convention, measured not chosen: the shared proxy already
runs the same flag and port with 2 ready endpoints.
PREREQUISITE ONLY -- NOT scraped. A container port is not a Service port, and
each namespace's default-deny baseline is `ingress: []`, so this is unreachable
until a Service port (or PodMonitor), a monitoring allow and a ServiceMonitor
land. Do not read this as "the sidecars have metrics now".
EVERY FLAG VERIFIED AGAINST THE PINNED v7.8.1 BINARY (`--help`), not against docs or
the shared proxy -- which runs v7.15.3 and could not have vouched for v7.8.1:
* --metrics-address exists in 7.8.1, so this cannot crashloop the pods
* --cookie-refresh ... "0 to disable" confirms the default
* --cookie-name ... (default "_oauth2_proxy") -- this confirms from the
IMPLEMENTATION the constant the new L0 check is built on
* --api-route is SINGULAR; the plural does not exist and the proxy refuses to start
* --session-cookie-minimal exists and is mutually exclusive with --cookie-refresh,
so dropping --pass-access-token is a least-privilege argument, never a size fix
NOT done, deliberately: --api-route=^/api/ renderer-wide. It was measured on jackett
and PiHole does not serve /api/, so a renderer constant would apply one app's path
shape to five. It needs a per-app hook the `auth` schema forbids
(additionalProperties: false) -- a schema change, separate PR.
Also records the mechanism that made this INTERMITTENT, as a renderer comment, because
nobody would reconstruct it from the flags: the loser does not just fail to read the
cookie, it calls ClearSession, which expires the cookie on its own --cookie-domain --
the whole zone. The *arr UIs poll (/api/v3/command, /api/v3/queue/details,
/signalr/negotiate), so every poll carrying an undecryptable cookie clears it again:
A LEFT-OPEN RADARR TAB DELETED JACKETT'S COOKIE EVERY FEW SECONDS, from another
namespace, with no user action. That is why signing in again never helped for longer
than the gap between two polls.
Crafty's counter-evidence resolved: the 81:1 ratio was a 24h window straddling a
13h52m-old pod. Bounded to the fixed process, crafty's ENTIRE life is 3 rejections in
6743 log lines, with ZERO `removing session` and ZERO `cookie signature not valid`
against unfixed radarr's 8 and 8. The eviction fingerprint is gone where the fix is
deployed. It logged 0 AuthSuccess too, so that is absence-of-the-failure-mode rather
than a session watched working -- stated as such in the handoff.
Renderer selftest gains six assertions, including a --cookie-refresh check that PARSES
the duration and requires 0 < refresh < 300s (so 2m30s/150s/2.5m all stay green while a
silent widening past the token lifetime does not). NOTE, and it is in the handoff as the
top follow-up: NOTHING IN ci/ RUNS `render.py --selftest`, so those assertions are
dev-time only. conformance/tests/l0-static/auditability-proofs-producible/check.sh:291
already drives other tools' selftests for exactly this reason and is the one-line fix.
Gate (toolbox, worktree shim with the spec repo mounted at /home-network-k3s): rc 3
UNDETERMINED. Green: spec-immutability, decisions-log, lint, l0, render-golden, l1,
approval-policy. Skipped: gitea-settings (no local token), conformance (declared).
The new L0 check PASSES on this branch -- all five apps fixed in one commit.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
9a4489b733 |
fix(auth): five oauth2-proxy sidecars shared one cookie and evicted each other
Every oauth2-proxy on this platform is an independent instance with its OWN
cookie-secret. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven
DISTINCT cookie-secret values (compared in-cluster, never printed). So no instance
can DECRYPT another's cookie -- it can only OVERWRITE it.
All of them were emitting oauth2-proxy's DEFAULT names (`_oauth2_proxy`,
`_oauth2_proxy_csrf`) scoped to `.${BASE_DOMAIN}`, so the browser kept ONE cookie
pair for SIX live instances (jackett, pihole, qbittorrent, radarr, sonarr, and the
shared forward-auth proxy). Whichever authenticated last clobbered the rest; the
loser logged `No valid authentication in request`, bounced to Keycloak, and -- the
SSO session still being live -- was redirected straight back and clobbered it again.
The symptom is "I signed into Keycloak but I'm still not signed in", which sends you
hunting in Keycloak, where nothing is wrong. Crafty was fixed on its own side on
2026-09-29 (#22); this is that fix generalized to the pattern.
THE DEFECT WAS NOT IN THE MANIFESTS. apps/wave-*/{jackett,pihole,qbittorrent,radarr,
sonarr}.yaml are GENERATED goldens; one function emits all five. So the fix is in
catalog/render.py::_oauth2_sidecar_container(), and both flags are now DERIVED from
the ServiceDefinition rather than defaulted:
--cookie-name=_oauth2_proxy_{s['name']} unique by construction
--cookie-domain={host} the app's own host, not the zone
A per-service table was rejected for the reason twelve ServiceMonitors once acquired
one identical wrong port name in a single edit: it would leave the NEXT app onboarded
defaulting straight back into the collision. --whitelist-domain stays zone-wide; it
governs post-login REDIRECT targets, not cookie scope.
Host-scoping is defence in depth and is asserted deliberately: a zone-wide cookie is
transmitted by the browser to every host in the zone, so each app's session token was
being sent to every sibling -- and it is what would make a FUTURE name collision
platform-wide instead of contained.
New gate check (whole-repo scope, so it runs on EVERY gate invocation, not only when
a touched path happens to attribute an identity requirement):
conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/
It resolves each instance's EFFECTIVE cookie name -- declared, or oauth2-proxy's
built-in default when omitted, because an omitted flag is not an absence but a CLAIM
on a name, and reading it as an absence is exactly why this survived five reviews.
Then: no two collide; none is another's name + `_csrf` (oauth2-proxy derives the CSRF
cookie that way); the default has exactly ONE owner, registered with its reason in a
shrink-only register -- the SHARED proxy, which MUST keep `_oauth2_proxy` because
renaming it signs every other gated surface out at once; every other instance is
host-scoped; render.py derives rather than lists; and the SHIPPED renderer is proven
end-to-end on two synthetic ServiceDefinitions, so the check stays falsifiable once
the tree is clean.
Tested in both directions: rc 0 on this tree (39 assertions, 7 instances), rc 1 with
26 failures against the pre-fix tree, plus one targeted mutation per assertion class
(duplicate name, CSRF derivation, zone-wide domain, unregistered default heir, stale
register entry) -- each fired, and nothing else did.
Also regenerates the three L1 rendering goldens, which are expected-output fixtures
of the same renderer.
UNVERIFIED: no browser round-trip. The collision has still never been REPRODUCED in a
browser -- doing so needs two gated apps visited in one session -- so this is reasoned
from measured configuration and from the mechanism, not from a failure anyone watched.
Nothing was applied; merging rolls five Deployments and invalidates every existing
session on them once. See decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e72050c690 |
docs: correct CI/platform documentation that was stale, wrong, or load-bearing
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no runner -- while the tree asserted both. This corrects the documents that carried, and in several cases CAUSED, that belief. CLAUDE.md gains section 2b: never infer platform state from this repository. It carries the specific API and kubectl checks for protection, runners, CI status, workloads and Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea job reports success). This is the rule whose absence produced every other finding. .gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host, this file is dormant". Gitea had been live for weeks. Both were written as plans and never revisited once the plan completed. The Transition section is now marked COMPLETE and says what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI). DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously hit "required status check; runner incluster-dind online" with nothing marking it false until a month later. The correction now sits on the bullet itself. ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an expected phase. That framing is why the gap went unexamined. apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to `sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded that a fresh token alone does not re-register a runner holding a stale /data/.runner. apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder. It has not since the claim -- it holds the spent token, which is inert and deliberately left in place. catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was uncited, upstream documents no expiry, and it was the basis for a since-disproven claim that the pipeline was too slow to fit inside it. Measured: ~2m45s. Both .sops.yaml files were verified to still decrypt after the comment edits (`sops -d > /dev/null`), and render==golden confirms the catalog comment does not propagate into the rendered manifest. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
bbd4f682a6 |
chore(plex): fresh plex.tv claim token
Injected inside the plex.tv claim window per runbooks/plex-claim.md. |
||
|
jakman85andClaude Opus 5
|
4fd75ccdaa |
feat(plex): wire PLEX_CLAIM from a SOPS Secret so the server can be claimed
Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving PLEX_CLAIM unwired; the operator has decided to override it. This implements the wiring in the way that respects the half of that argument which is still true. WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers. So the committed value is a documented PLACEHOLDER and the live token is injected by the operator inside that window, with the reconcile FORCED rather than waited for. runbooks/plex-claim.md is that timed procedure. WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret would crashloop the pod". Half of that is wrong, and the difference is the whole design. Established from plexinc/pms-docker's own first-run script (branch `master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked byte-identical against the same file inside the running :latest image: * the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the PlexOnlineToken pref in /config being empty. Once the server is claimed the block is skipped entirely, so the env is a NO-OP on every later start whatever it holds ("If server is already logged in, this parameter is ignored" -- upstream README). * with an invalid/expired token on a still-unclaimed /config, plex.tv answers 403 with an empty body; curl's status is never checked, the token sed yields empty, the success branch at line 120 does not fire, there is no else branch and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up UNCLAIMED AND SERVING. It does not crashloop. * the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even if a cont-init script failed), so even a non-zero exit would not halt startup. An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it is CreateContainerConfigError), which is why the Secret is committed, encrypted and listed in the wave kustomization rather than hand-created at claim time. NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had to tolerate a missing Secret or could not start without a VALID token. Neither holds: the Secret always exists in the Flux graph, and an invalid token is non-fatal. schemas/servicedefinition.schema.json is untouched, and so are catalog/render.py's env emission and the renderer contract. Changes: * apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex, key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header carries NO dollar-brace expressions on purpose: kustomize preserves body comments into the stream envsubst reads (per conformance/tests/l0-static/flux-postbuild-substitution), so quoting the upstream shell conditions verbatim would have been a live substitution hazard on apps-wave-0, not prose. * catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the "deliberately NOT set" block rewritten around the TTL + one-shot design, and the stale crashloop claim in the /config storage comment corrected. * apps/wave-0/plex.yaml -- re-rendered with the canonical command (`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden under conformance/tests/l1-rendering covers plex, so this is the only rendered surface that moves. * catalog/render.py -- plan_apps_waves places `<svc>-claim-secret.sops.yaml` in the app's OWN wave, like the bootstrap secret, plus a selftest assertion. Without this, the next `--emit-waves` would have silently dropped the Secret from apps/wave-0/kustomization.yaml and put the pod into CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one new line was applied without re-emitting the rest. * runbooks/plex-claim.md + its kustomization.yaml generator entry + README index -- the operator procedure that fits in 4 minutes, including the single forced-reconcile command and the mandatory pod roll (a changed Secret does NOT restart anything: PLEX_CLAIM is read only at container start and the pod template is byte-identical across this change). Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02 (ServiceDefinition onboarding contract). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
b184a42729 |
merge: internal services stop advertising themselves in public DNS
Six exposure:internal services published A records into the public Cloudflare zone answering 192.168.2.20, disclosing the internal subnet, the Gateway VIP and a full service inventory, and forming a DNS-rebind primitive that is why the SOHO router strips these answers in the first place. The renderer now stamps external-dns.alpha.kubernetes.io/controller: none on every HTTPRoute it renders for an internal service, derived from the exposure field alone — so a service that omits exposure defaults to internal and is likewise unpublishable. Established against the deployed v0.21.0's own source, where the controller check runs before hostname resolution in both configured sources, then proven live on a throwaway hostname including that adding the annotation to an existing record triggers the DELETE. Deletes exactly 12 records: the A and TXT pair for jackett, pihole, qbittorrent, radarr, semaphore and sonarr. A 39-record recovery baseline is committed alongside. Safe to land before the DHCP change, contrary to my earlier caution: the router returns NOTHING for all six names today, so these records serve nobody on the LAN, while Pi-hole at 192.168.2.23 already answers for anyone pointed at it. The five exposure:public services are untouched. They remain unreachable publicly, which is a separate problem: the legacy edge at 136.51.115.252 returns HTTP 403 for our hostnames and serves a placeholder certificate, so nothing is reverse-proxying them today. |
||
|
jakman85andClaude Opus 5
|
fd43f332d6 |
feat(pihole): authoritative LAN DNS for the zone on udp/53 + tcp/53 at a pinned VIP
platform/coredns/coredns-custom.yaml has always said its split-horizon is for
PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS:
its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03
that docs/09 and the README's Phase 2 row both claim did not exist, so LAN
clients ran on hand-maintained per-client dnsmasq files - one went stale after
the dev decommission and cost the operator sonarr.home.keithjackson.dev. The
SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips
RFC1918 answers for public domains as rebind protection and returns NOERROR
with no answer.
EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by
render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a
pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason
apps/vms/unifi and platform/crafty hand-write theirs):
- two Services, udp/53 and tcp/53, sharing one pinned VIP
${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the
.20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22
crafty; arping from a LAN host got no reply).
- no external-dns hostname annotation: nothing public is created, and
platform/externaldns/ is untouched.
ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes
every FTL setting as an FTLCONF_ environment variable and treats it as
env-forced, applied ahead of pihole.toml on every cold start and read-only in
the UI. That is a real declarative entry point, which the Keycloak and CNPG
CronJob reconcilers exist precisely because their state lacks, so a reconciler
here would be ceremony. misc.dnsmasq_lines carries a wildcard
address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service
(unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching
domain, so precedence is structural and a newly onboarded Gateway-fronted
service needs no DNS change. dns.upstreams is declared too - the live pod
carried hand-set upstreams that no manifest knew about and that every restart
discarded.
externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the
LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not
filter lease candidates by which nodes hold a backend - the two Services landed
on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps
renewing after the pod moves away, so with Local a reschedule leaves LAN DNS
dead indefinitely. Cluster serves the VIP from any node; the LAN restriction
moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP
before load-balancing (verified by narrowing the range to exclude the test
client and back). The cost is SNAT: forwarded queries reach PiHole from the
ingress node, so PiHole's per-client rate limit is disabled in the catalog or
the whole LAN would count as one client against its 1000-per-60s default.
Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and
both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex
-> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion
(cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA
NODATA, NXDOMAIN outside the zone, and still answering after a forced
prod-1 -> prod-2 reschedule.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
4080065aa7 |
feat(dns): exposure: internal can no longer publish a public record
docs/09 R-DNS-03 and schemas/renderer-contract.md both said an `internal` service renders no public DNS record. Nothing enforced either sentence. ExternalDNS's `gateway-httproute` source admitted EVERY HTTPRoute on the shared Gateway, so six `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — published `<name>.home.keithjackson.dev` into the public authoritative Cloudflare zone answering with the Gateway VIP 192.168.2.20. That published the internal subnet, the Gateway VIP and a full service inventory to anyone who could guess a label, and made each of those names a DNS-rebind primitive — which is why the SOHO router strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own services. THE MECHANISM, established against the DEPLOYED version rather than from memory. The live prod Deployment runs registry.k8s.io/external-dns/external-dns:v0.21.0. In v0.21.0, source/annotations/processors.go IsControllerMismatch() skips a source object whose `external-dns.alpha.kubernetes.io/controller` annotation is present and != its own `dns-controller`, and source/gateway.go applies that check inside Endpoints() BEFORE any hostname is resolved; source/service.go applies the identical check, covering the second configured source. catalog/render.py now emits that annotation on every HTTPRoute it renders for an `internal` service, derived from `exposure` alone — no per-service list — so a service that omits `exposure` inherits the internal default and is unpublishable too. Verified LIVE on prod on a throwaway hostname with no existing record at risk: the annotated route published nothing, and adding the annotation to a route that already had a record produced action=DELETE for both its A record and its `a-` TXT registry entry one ~60s reconcile later, moving nothing else in the zone. WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. `labelFilter` is the stronger default posture and v0.21.0 supports it for both configured sources, but nine publishing objects here are hand-authored platform/apps manifests with no `exposure` field, seven of which hold live records (auth, crafty, crafty-mc, grafana, headlamp, the apex, unifi). An opt-in filter deletes all seven at once, and whether each should be publicly resolvable is an operator decision per name. Per-resource exclusion has exactly the blast radius of the defect; the seven are enumerated in a shrink-only register so the unadjudicated set cannot grow silently, and the filter stays available as hardening afterwards. `policy: sync`, `txtOwnerId` and `domainFilters` are UNTOUCHED. Per-resource exclusion only ever removes an object from this instance's view, so nothing here makes prod capable of deleting a record it does not own. WHAT THIS DELETES, and it is not reversible by a revert alone: 12 records — the A record and the `a-`-prefixed TXT registry entry for each of jackett, pihole, qbittorrent, radarr, semaphore and sonarr. The zone has no `*.home.keithjackson.dev` wildcard and `*.keithjackson.dev` does not match two labels, so each becomes a public NXDOMAIN with no fallback. The complete pre-state is captured at the authority and committed as a recovery baseline (.standup-dns/dns-20260914-pre-externaldns-scope.txt), with the restore procedure for both halves of a record. MERGE ORDER IS NOT OPTIONAL: feat/pihole-lan-dns-authority must land and be verified first. Removing these records while the LAN still depends on them, with the router stripping RFC1918 and no internal authority yet, breaks name resolution outright. The five `exposure: public` services are LEFT UNCHANGED and still answer with the Gateway VIP, so they remain publicly unreachable. That is two open operator decisions, both written up in platform/externaldns/helmrelease.yaml: what address they should answer with (136.51.115.252 is the legacy non-k3s stack serving a placeholder cert, not established as correct for this cluster), and how to separate them from the non-public names that share the Gateway — because in v0.21.0 the target annotation is read only from the GATEWAY and a ListenerSet, never from an HTTPRoute, so a per-service override is not expressible while they all attach to `shared`. No knob is added for it yet: one that cannot take effect invites someone to set it and expect one. Guard: conformance/tests/l0-static/internal-services-publish-no-public-dns, a whole-repo-scoped L0 static check (no ci/l0.sh stage — the whole-repo-invariants-wired meta-test forbids one). It asserts both directions (present on internal, absent on public), the `service`-source half, that the renderer DERIVES the annotation from `exposure` and cannot emit external-dns's own claiming value, proves the shipped renderer end-to-end on internal/public/omitted-field fixtures, and ratchets the non-catalog register. Proven to fail on the pre-fix tree (9 of 53 assertions, naming all six services) and pass after (56/56). Schema 1.6.0 -> 1.7.0: no field changes, a mapping change to `exposure`. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
cf7d97e546 |
fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
72f0ba4e1f |
fix(qbittorrent): the comment warning about doubled dollars contained one
The new flux-postbuild-substitution guard failed the merge on
apps/wave-0/qbittorrent.yaml:155 — a run of two dollars in a substituted
path that nothing justifies. It was inside the postStart hook's own comment
EXPLAINING the hazard: prose about what envsubst does to a doubled dollar,
written with a literal doubled dollar.
Functionally harmless — it is a bash comment, and collapsing it changes no
behaviour — but the guard cannot distinguish a comment from content that
reaches the workload, and it is right not to try: that is precisely how the
nextcloud-db break slipped through. Reworded in words instead.
Worth recording that this is the second time today the same trap caught an
explanation of itself: my own fix for the nextcloud-db incident (
|
||
|
jakman85andClaude Opus 5
|
e4bd76070e |
fix(qbittorrent): a durable WebUI credential, and the two bugs the WIP hid
Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.
Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.
Two real bugs found in the WIP, both the postBuild-substitution trap:
* PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
(envsubst collapses it) and the renderer accepts it, but run any other way
bash reads the doubled dollar as its PID, so every curl got an unparseable
URL and the hook logged "the WebUI rejected the durable credential" three
times having never asked it anything — see the marker log at 05:35Z. Now a
literal PORT=8080; the script contains no braced expansion at all and
behaves identically however it is run.
* One shared wait budget covered both the s6 wait and the API wait, so the
first consumed all 90s and attempts 2-3 fired instantly. Separate counters
now; "the API never answered" is reported as a distinct fault from "the
password was rejected"; and a rejection restarts qbittorrent-nox so a retry
can actually change its outcome.
apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.
Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.
NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
204f0ec28e |
WIP: durable qBittorrent WebUI credential
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Problem: qBittorrent 5.1.2 has no WebUI\Username or WebUI\Password_PBKDF2 in
its config, so it mints a NEW temporary admin password on every process start
and announces it only on stdout. Two consequences, both real: the Sonarr and
Radarr download clients configured earlier will silently stop working at the
next pod restart, and a human operator has no stable login either.
Adds a SOPS-encrypted qbittorrent-bootstrap Secret (verified encrypted: both
values are ENC[AES256_GCM]) plus the catalog/rendered wiring to consume it.
WHAT REMAINS:
- The agent's last note: the postStart hook works, but its output never
reaches kubectl logs, so it was switching to the marker-file logging
pattern gitea's hook already uses. That change is NOT here.
- The Secret exists on prod as kubectl-client-side-apply, i.e. live-only;
merging adopts it into git.
- Sonarr's and Radarr's stored download-client passwords were to be updated
to the durable value — NOT done, so the chain may break on restart until
it is.
- The required proof is unfinished: restart the pod, then show the WebUI
accepts the credential AND both arrs still report isValid: true.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
16b7fcd677 |
fix(cnpg): the grant reconciler died on every Flux-managed run
The SQL asserted convergence in a dollar-quoted PL/pgSQL DO block. That cannot survive this delivery path: the ConfigMap is rendered through Flux's postBuild substitution, and envsubst collapses a doubled dollar into one, so Postgres received a half-quoted DO and failed with a syntax error. Three scheduled runs had left nine Error pods in nextcloud-db. It passed verification because it was hand-applied with kubectl, which runs no envsubst — the defect only existed once Flux owned the object. The GRANT on the first line still succeeded, so the privilege held and the blind spot stayed closed; what was broken was the assertion meant to prove it, which is the part that makes a silent regression visible. Rewritten in plain SQL with no dollar sign anywhere in the file, comments included. Escaping by doubling again would also work and would be the next person's trap. has_database_privilege()::int is 1 or 0, so the division raises division_by_zero and ON_ERROR_STOP=1 fails the Job; the divisor is not constant so it cannot be folded away at plan time. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
78d76fa089 |
merge: qBittorrent can finally write where the arrs can see it
qBittorrent's SavePath was /downloads and no such directory existed in the container, so the download client had no working destination and nothing downstream could see its output. It now mounts the same NFS export the arrs use, narrowed to the downloads subtree via subPath — one mount, so the EXDEV hardlink trap the bulk-tier test guards stays closed. PUID/PGID are load-bearing: the export is owned 1000:1000 and the image's default 911 is denied. |
||
|
jakman85andClaude Opus 5
|
78bc53afc7 |
feat(qbittorrent): attach the downloads area of the bulk export — the chain was broken end to end
qBittorrent was the one member of the download chain with NO media volume.
Its live config said
Session\DefaultSavePath=/downloads/
Session\TempPath=/downloads/incomplete/
Downloads\SavePath=/downloads/
while `ls /downloads` in the container returned "No such file or directory".
The catalog entry declared exactly one volume — config at /config on longhorn —
and carried a comment deferring the downloads volume "until OQ-2 is resolved".
OQ-2 WAS resolved (schema 1.5.0 static bulk NFS PVs); plex, sonarr, radarr and
nextcloud all took their mounts and qBittorrent's was simply never added. So the
download client was writing, or failing to write, a path that did not exist —
and anything it did create would have been invisible to sonarr/radarr, lost on
every pod recreation, and consuming the node's ephemeral disk until then.
WHAT THIS ADDS to catalog/qbittorrent.yaml (rendered into apps/wave-0):
* storage[downloads]: class bulk, export /mnt/Storage/plex, subPath downloads,
mountPath /downloads, 9Ti advisory. ONE mount of the SAME export sonarr,
radarr (rw) and plex (ro) already consume, narrowed to the handoff subtree.
* env PUID=1000 / PGID=3000. LOAD-BEARING, not cosmetic: the container ran as
uid 911 and /mnt/Storage/plex/downloads is 0755 owned 1000:1000, so the
export would have mounted cleanly and then failed every write with EACCES.
Measured both ways on prod: a write as uid 1000 succeeded, the same write as
uid 911 was denied.
* podSecurityContext.fsGroup 3000, for the LONGHORN config PVC only, exactly as
sonarr/radarr set it — fsGroup is a no-op on an in-tree NFS volume.
WHY subPath AND NOT THE EXPORT ROOT AT /media (sonarr/radarr's shape): a download
client has no business writing the library. The arrs need the root mount because
THEY hardlink downloads/ -> tv/|movies/ and link() is a same-MOUNT operation;
qBittorrent performs no such link, so narrowing to downloads/ removes movies/,
tv/ and photos/ from its mount namespace without removing any operation it
performs. It is still ONE mount of the export, so the L0 assertion that no
workload mounts one export twice (the EXDEV trap) holds — and the arrs' own
link() stays intra-mount. Proven live: nlink 1 -> 2, same inode, no EXDEV, every
step executed as uid 1000.
NETWORK: nothing to grant and nothing to steer. An NFS PersistentVolume is
mounted by the NODE KERNEL in the host netns — the established socket to
192.168.2.102:2049 is on the node's own LAN address, the pod netns has no socket
to 2049 at all, and no CiliumNetworkPolicy on this cluster names that host or
port. `ip route get 192.168.2.102` picks eno1 even on the tunnel-owning node, and
Cilium's own egress map lists 192.168.0.0/16 as an Excluded CIDR for every
vpn-class pod while 0.0.0.0/0 goes to the gateway. Public egress still exits the
PIA address; the VPN guarantee is untouched.
NOT CLOSED BY THIS COMMIT, deliberately: absolute paths still differ between this
pod (/downloads/x) and the arrs (/media/downloads/x), so Completed Download
Handling needs one Remote Path Mapping per arr. That lives in the arrs' own
databases, not in git.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
7880d84bce |
fix(cnpg): grant cnpg_metrics_exporter CONNECT on the nextcloud database
The CNPG metrics exporter could not CONNECT to the `nextcloud` application
database, so every user-defined collector query against that target failed on
all three instances:
FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
DETAIL: User does not have CONNECT privilege.
The whole cnpg_pg_stat_* / pg_settings / pg_postmaster family was therefore
MISSING for nextcloud-db while the scrape target still reported up=1 — the blind
spot platform/cnpg/alerts.yaml's CNPGMetricsCollectorErroring was written to
catch. It had been firing on all three pods; measured before this change,
cnpg_pg_settings_setting had 765 series for keycloak, 765 for semaphore-db and
ZERO for nextcloud-db. Nextcloud's database had never been monitored.
Root cause is the live ACL, not the spec — all three Cluster specs bootstrap
identically:
nextcloud : datacl = {nextcloud=CTc/nextcloud} <- explicit, no PUBLIC entry
keycloak : datacl = NULL
semaphore : datacl = NULL
A NULL datacl means PostgreSQL's defaults, which include CONNECT for PUBLIC, so
keycloak and semaphore admit the exporter only by inheriting PUBLIC's implicit
CONNECT. On `nextcloud` something ran the equivalent of REVOKE ALL ON DATABASE
nextcloud FROM PUBLIC, which materialises datacl and drops that implicit entry.
That REVOKE is in neither this repo nor DECISIONS-LOG.md — it was applied out of
band, which is why the remedy has to be declarative rather than another hand-run
psql.
CNPG 1.30.0 cannot express a database-level GRANT (verified against the live
CRDs): `Database` has no ACL field (`allowConnections` is the datallowconn flag,
not a per-role grant), and managed.roles/DatabaseRole cover role attributes plus
`inRoles` membership only — and membership would hand the exporter the owner's
CREATE/TEMP and all table access to buy one CONNECT. postInitApplicationSQL runs
once at bootstrap, so it cannot repair the already-bootstrapped instances that
are actually broken. So this follows the pattern the repo already uses for state
Flux cannot reach: the level-triggered CronJob reconciler alongside
platform/keycloak/client-secret-reconciler.yaml. A one-shot Job is deliberately
avoided — a completed Job never re-runs, so a rebuild or PITR restore would
silently reopen the blind spot.
Exactly one privilege, and nothing else:
GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
giving `cnpg_metrics_exporter=c/nextcloud` — CONNECT only, no CREATE, no
TEMPORARY, no ownership, no superuser, no table or schema grants. Nothing more is
needed because CNPG already creates the role as a member of the built-in
pg_monitor role (verified on all three clusters), which carries the read access
the collector's pg_catalog queries want. It runs as the database OWNER, not a
superuser: enableSuperuserAccess stays off, and a database owner holds the grant
option on its own database.
Verified live on prod:
* the grant issued under SET ROLE nextcloud, proving no superuser is needed;
* permission-denied and "Error collecting user query" both went to 0 on
nextcloud-db-1/-2/-3, i.e. every collector query now succeeds;
* cnpg_pg_settings_setting now 765 series for nextcloud-db, matching the other
two, and cnpg_pg_postmaster_start_time present on all three instances;
* cnpg_last_error = 0 on all NINE instances, so keycloak-db and semaphore-db
are unaffected and their datacl is still NULL;
* convergence proven by revoking the grant again and letting the committed
CronJob restore it, then re-running it against a converged database to show
idempotence;
* CNPGMetricsCollectorErroring, firing on all three pods beforehand, has
cleared (`for: 6h` delays firing, not resolution).
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
d617983e83 |
feat(gitea): render gitea-allow-flux-ingress — prod's GitOps source, in git at last
The CNP that lets flux-system's source-controller clone from the in-cluster Gitea Service was APPLIED LIVE on prod on 2026-09-13 and existed nowhere in this repo. It carried a provenance annotation saying so. Until now it could not be authored: the gitea CNPs are GENERATED from the catalog entry, and the contract had no field for "also admit this in-cluster client" (fixed in the previous commit). Why it matters more than a missing allow normally would. prod's GitRepository points at the Service, not a public name (clusters/prod/kustomization.yaml: `url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git`), deliberately, so prod's GitOps depends on neither public DNS, Cloudflare, prod's own Gateway nor ${BASE_DOMAIN}. But gitea-allow-gateway-ingress admits only `fromEntities: [ingress]`, so source-controller's clone was dropped — and the HTTPRoute that would have served the post-apex-flip public name instead lives in a DOWNSTREAM tier that cannot apply while the source is down. Source down -> routes never rename -> source stays down. Every `kubectl delete cnp` and every cluster rebuild re-armed that deadlock, and it would have been extremely hard to diagnose cold. The rendered object is EQUIVALENT to the live one — same apiVersion/kind, same name/namespace/labels, and a byte-identical spec (endpointSelector app.kubernetes.io/name=gitea; fromEndpoints matchLabels k8s:io.kubernetes.pod.namespace=flux-system; toPorts 3000/TCP), verified against `kubectl --context k3s-prod -n gitea get cnp gitea-allow-flux-ingress -o json`. The name matches, so Flux ADOPTS the live object in place: no spec change, so no policy recompute and no flap on prod's own source. The only difference is that the rendered object carries no `homelab.keithjackson.dev/provenance` annotation — the reasoning now lives where the object is authored, in catalog/gitea.yaml, next to the field that generates it. Golden re-rendered with the canonical command; the whole apps/ diff is the one new document. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
5da116b34d |
fix(oauth2-proxy): platform-admins may reach the media UIs — the immediate 403
The four forward-auth media UIs (jackett, qbittorrent, radarr, sonarr) gate on `--allowed-group=media-users` alone. On prod the `homelab` realm has exactly one user — the operator's `jakman85` — and it is a member of `platform-admins` ONLY. `media-users` has zero members, so those four hosts were reachable by nobody: oauth2-proxy completed OIDC discovery, redirected to Keycloak, got a token whose `groups` claim is `["platform-admins"]`, found no allowed group in it, and answered 403. No login form appeared because the browser already held a Keycloak SSO session from a platform-admins login — hence "403 immediately". pihole and crafty were unaffected: they already gate on platform-admins. `--allowed-group` is repeatable and OR-ed, so listing both groups means "media consumers OR platform operators". This grants no real privilege — platform-admins already maps to cluster-admin via k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods — and it keeps media-users meaningful for non-admin household accounts. Ruled out while diagnosing: client-secret drift (all 7 sidecar secrets hash-match keycloak-realm-clients), stale apex URLs (every live sidecar carries keycloak.prod.home.keithjackson.dev), missing group mappers (all 7 clients emit claim=groups, full.path=false), and the central oauth2-proxy pod restarts — those were Keycloak returning 503 "no healthy upstream" to OIDC discovery at startup, a consequence of the Keycloak roll, not a cause. Goldens re-rendered with the canonical command: python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation R-IDP-06 Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6b953cc56b |
fix(arrs): the config PVC was mounted where the app never looks
sonarr, radarr, jackett and qbittorrent all mounted their Longhorn config PVC at the renderer's default /data/config, but every one of these images runs with `-data=/config`. So the PVC held nothing but lost+found while the app wrote its entire configuration to the container's EPHEMERAL writable layer — destroyed on every pod recreation: restart, reschedule, image bump. Measured in-cluster before fixing, per app, rather than inferred: /data/config dev=2080 [lost+found] <- the Longhorn PVC /config dev=2053 [Sentry asp config.xml logs ...] <- the node's own disk and a marker file written to /config did not appear in /data/config, so they are demonstrably different storage. qbittorrent's /config sat on dev=66309 (/dev/nvme0n1p5), i.e. whichever node it happened to land on. This is the SAME failure catalog/plex.yaml:54 already records having fixed for Plex; plex was the only one of the five that was right. The live /config contents were copied onto each PVC BEFORE this landed, so jackett's API key and qbittorrent's credentials — which other apps reference — survive the move rather than being regenerated. catalog/sonarr.yaml:61 documented this as a latent bug and deliberately left it because it changes live on-disk state. It is being fixed now because all four pods had just been recreated, so their configs were freshly generated and nothing had accumulated to lose. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
f5e62d9169 | merge: bulk NFS volumes for plex, sonarr, radarr and nextcloud | ||
|
jakman85
|
cac413e87a |
feat(cnpg): synchronous replication, quorum of 1, dataDurability preferred
Three instances replicating ASYNCHRONOUSLY still lose data on failover: COMMIT returns as soon as the primary has flushed locally, so a primary that dies before its standbys catch up takes the last few transactions with it and the promoted standby comes up quietly missing them. On keycloak-db that is a just-created user, a rotated credential or a consent vanishing at exactly the moment of a failover. `method: any` + `number: 1` makes COMMIT wait for ONE of the two standbys to confirm — the point of having them. dataDurability: preferred, NEVER required. `required` is a strict quorum: fewer than `number` healthy standbys and writes BLOCK. Postgres here shares three nodes with etcd and the CNI, so one degraded standby — a node reboot, a Longhorn replica rebuild, a Cilium restart — would turn a partial degradation into a total write outage, and on the SSO database that takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth with it. `preferred` keeps synchronous commit while a healthy standby is there and relaxes the required count dynamically when one is not. dataDurability is only honoured while standbyNamesPre/standbyNamesPost are unset — both are. NON-DISRUPTIVE: this compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter. No pod restart, no switchover, no client disconnect. It is the last of the three commits because it wants standbys that already exist and are already on their final nodes. Field shape verified against the live clusters.postgresql.cnpg.io CRD before relying on it (CNPG 1.30.0, byte-identical schema on k3s-prod and k3s-dev): spec.postgresql.synchronous requires `method` (enum any|first) and `number` (x-kubernetes-validation self > 0); `dataDurability` is enum required| preferred. None of the three Clusters previously had a `postgresql:` block, so this adds one rather than merging into existing parameters. DEV IMPACT: identical and non-disruptive, same reload-only path. Verified: yamllint clean; both tiers render the block. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit f1ce19461415e77eb2f1ca5109ebab49edf0860b) |
||
|
jakman85andClaude Opus 5
|
82aff4d053 |
feat(apps): attach the media library and the legacy Nextcloud dataset over NFS
plex, sonarr, radarr and nextcloud each gain one bulk volume. Authored in the catalog and re-rendered, never hand-edited into apps/ (render-golden: 10/10). Layout, the export change the operator must apply, the uid analysis and the validation probes are in docs/BULK-NFS-STORAGE.md; everything there was measured on the host over read-only SSH, not assumed. WHAT THE LAYOUT TURNED OUT TO BE, AND WHY IT DECIDED THE DESIGN /mnt/Storage/plex holds movies/ tv/ photos/ AND downloads/ as SIBLINGS in one dataset — so the arrs' import source and their libraries are one filesystem on the server, and the download client's completed folders are already named radarr/ and tv-sonarr/. The k3s arrs are joining a tree an arr stack has been driving for a long time. That sibling fact is the whole argument for the arrs mounting the export ROOT rather than a subPath per directory. Sonarr and Radarr import by hardlinking or atomically renaming, both same-filesystem operations, and to the kernel "same filesystem" means SAME MOUNT. Two subPath mounts of one export are two mounts: link() across them returns EXDEV regardless of what the server knows, and the arrs do not fail loudly — they degrade to copy-then-delete, rewriting every file over NFS and destroying the seeding copy. The smallest subtree containing both tv/ and downloads/ IS the export root, so the root mount is the MINIMAL mount that preserves hardlinks. Least privilege bounded below by a hard requirement. The cost is stated, not hidden: sonarr can also write /media/movies. plex /mnt/Storage/plex -> /media ro ReadOnlyMany sonarr /mnt/Storage/plex -> /media rw ReadWriteMany radarr /mnt/Storage/plex -> /media rw ReadWriteMany nextcloud /mnt/Storage/nextcloud -> /mnt/legacy-nextcloud ro ReadOnlyMany (subPath: nextcloud) Plex is read-only because it indexes and streams; every write on that tree belongs to the arrs, and saying so removes Plex from the set of things that can damage 8.33 T. All three media consumers use the SAME mountPath deliberately: when qbittorrent is eventually given this export, the arrs and the download client will already agree on absolute paths, which is the other precondition for hardlinked imports. *** /media AND NOT /data — A COLLISION CAUGHT WHILE RENDERING. *** The arrs' Longhorn config PVC renders at the renderer default /data/config. A media mount at /data is mounted OVER it and MASKS the config volume: the pod comes up with its app database shadowed by an 8 T media tree. That discovery also exposed a SEPARATE PRE-EXISTING BUG, reported and deliberately NOT fixed here: LinuxServer arr images run with -data=/config, so /data/config is very likely the wrong home for that PVC and the app has been writing its EPHEMERAL layer — the exact failure catalog/plex.yaml's mountPath comment records already having been bitten by. Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it starts those apps from an empty config volume, so it needs its own change with the operator present. This change does not make it worse. UID/GID — THE FAILURE THIS TIER ALWAYS HAS The export is sec=sys + no_root_squash, so the client's numeric uid passes through verbatim and the server's ownership is the entire access story. The tree is 0755 owned 1000:3000, so ONLY uid 1000 can write it. LinuxServer images default to 911. Without intervention sonarr and radarr would have mounted cleanly, gone Ready, and failed EVERY import with EACCES. And fsGroup cannot fix that — the in-tree NFS plugin does NO ownership management, so podSecurityContext.fsGroup is a no-op on an NFS volume. Reaching for it is how this gets misdiagnosed. So PUID=1000/PGID=3000 are added to both arrs (PGID 3000 keeps what they write group-readable to the live non-k3s consumers of the same export), and fsGroup: 3000 is added for the LONGHORN config PVC only, so changing PUID does not depend on the image re-chowning /config. plex already runs as 1000 and reads a 0755 tree; nextcloud reads as www-data. Neither needs work. NEXTCLOUD IS MOUNTED READ-ONLY, WHICH IS A DEVIATION FROM THE BRIEF — ON EVIDENCE The task said nextcloud writes. The dataset says otherwise: of 270 G, 267 G is config_backup/, the rest is config/ (103 M) and config_old/ (3.4 G). There is NO data/ directory. This is the OLD non-k3s instance's configuration plus a backup blob — an archive — and that instance is STILL LIVE on it (exported rw to 192.168.1.6 and .5, which the operator has said must stay active). Two Nextclouds writing one instance's config directory, each with its own database and therefore its own oc_filecache, is unrecoverable corruption. "Do not grant more than needed" lands on ro, and ro is the trivially reversible choice. If the intent is in fact a MIGRATION, that is a larger and more deliberate change than a mount; the one-line flip is documented. Read-only for nextcloud can additionally be enforced SERVER-side (only nextcloud consumes that dataset from k3s), and the sharenfs value in the doc uses ro= for the k3s clients. The media tree cannot: plex and the arrs may land on any node, so the server sees the same addresses for reader and writers. Plex's ro is therefore a client-side guarantee — stated as a limitation, not left implied. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
e8d8601ca6 |
feat(cnpg): required pod anti-affinity + switchover primary rolls
`instances: 3` alone does not keep the three instances apart. All three Clusters used CNPG's default `podAntiAffinityType: preferred`, which is a scheduler HINT — two instances may share a node whenever that scores better, and they WILL on the path that matters: when CNPG recreates an instance after a node loss, the surviving nodes are the only candidates and one of them already runs an instance. `preferred` is how a 3-instance cluster decays back into the single-node failure domain the previous commit exists to remove. `required` makes it a scheduling constraint. On a 3-node cluster with no spare, the cost is that an instance whose node is gone stays Pending until a node returns. That is correct: a replacement squeezed onto an already-occupied node dies with that node while making the cluster LOOK three-way replicated. 2 of 3 instances still serve and CNPG still promotes. ALSO SETS primaryUpdateMethod: switchover. This is not cosmetic. Changing the affinity changes the instance pod spec, so CNPG performs a rolling update, and the CRD default for primaryUpdateMethod is `restart` (verified against the live CNPG 1.30.0 CRD on both clusters) — a delete-and-recreate of the PRIMARY pod, i.e. tens of seconds of no Postgres. On keycloak-db that is an SSO outage, which also means kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. With standbys now present, `switchover` promotes an already-updated streaming replica instead, reducing the roll to a connection reset. Known constraint documented in the manifest: under `switchover` the operator rejects an update that changes imageName and a postgresql parameter in the same reconcile. Verified satisfiable before merge (read-only): k3s-prod and k3s-dev each have 3 Ready, schedulable, untainted nodes, and every pgdata PVC already has a Longhorn replica on all three (replica-soft-anti-affinity=false), so any instance can attach from any node. DEV IMPACT: identical, and dev can absorb it. Longhorn dev nodes are 124.9 GiB each with 152.0 GiB already scheduled under a 200% over-provisioning ceiling (249.8 GiB); the six new 5Gi volumes add 30 GiB scheduled per node -> 182.0, still inside the ceiling, and only ~600 MiB per volume is really consumed. Prod is far looser: 239 -> 269 GiB scheduled against a 829 GiB ceiling. THIS IS THE DISRUPTIVE COMMIT of the three. Land it on its own reconcile, after the scale-out has produced two Ready standbys per Cluster, so the blast radius is one clean switchover per database rather than a scale-out and a roll at once. Expect a brief SSO interruption for keycloak-db. Verified: yamllint clean; both tiers render the new fields; field names, types and enum values checked against the live clusters.postgresql.cnpg.io CRD (CNPG 1.30.0, identical on prod and dev). Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit e1d364238d73e6ee99c9a50108fe8b4e09481a7e) |
||
|
jakman85
|
f338e91770 |
feat(cnpg): instances 3 on all three Clusters — a standby to fail over to
All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication saves the DATA when a node dies; it does not save the SERVICE — a single instance has to be rescheduled, re-attach its volume and replay WAL before Postgres accepts a connection again. With nothing in the spec keeping them apart, all three primaries had drifted onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give each Cluster a streaming standby CNPG can promote in seconds, repointing the -rw Service with no volume re-attach and no restore. UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate corruption and deletions faithfully; only the Barman archive is a backup, and adding standbys while archive_mode=on with a failing archive_command would have multiplied the disk-full clock, not hedged it. All three Clusters on both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real firstRecoverabilityPoint (verified read-only, 2026-09-07). Scaling out is non-disruptive on its own: CNPG creates -2 then -3 sequentially, each seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are not disconnected. The anti-affinity change that DOES cause a switchover is deliberately the next commit. DEV IMPACT: identical — dev is on the same shape (three single-instance DBs, all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes and Longhorn headroom for it (see the next commit's message for the numbers). Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning. Verified: yamllint clean; `kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0` both render `instances: 3`. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02) |
||
|
jakman85andClaude Opus 5
|
161430d915 |
feat(backup): the overlay picks the credential — BACKUP_CREDENTIALS_SUFFIX
The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).
cluster-vars BACKUP_CREDENTIALS_SUFFIX base "" | dev "" | prod "-prod"
velero HelmRelease credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
3x CNPG Cluster s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.
THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.
Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).
VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:
strict envsubst PASSES for all 8 (env x tier) combinations
dev velero existingSecret velero-cloud-credentials
dev keycloak/nextcloud/semaphore-db <db>-backup-credentials (both key refs each)
prod velero existingSecret velero-cloud-credentials-prod
prod keycloak/nextcloud/semaphore-db <db>-backup-credentials-prod (both key refs each)
both all EIGHT Secret objects present in the rendered output (dev and prod alike)
no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
escapes in apps/wave-0, which are pre-existing and unrelated.
Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
dev velero s3://velero (root) + cnpg s3://cnpg @ http://192.168.1.9:9000 region us-east-1
prod velero k3s-prod-backup/velero + barman prefix @ https://<acct>.r2.cloudflarestorage.com region auto
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e7c4a95c2d |
feat(backup): the four backup credentials become TWO objects each, one per environment
The four backup-credential SOPS Secrets are SHARED BASE manifests: platform/ and apps/ are
rendered by the same Flux tiers on both clusters, and the ONLY per-environment difference in
this repo is cluster-vars substitution (R-ENV-02). So writing the R2 token into all four
(0a6a862) did not "give prod credentials" — it REPLACED DEV'S. Merging that as-is would have
handed dev's Velero and all three dev CNPG Clusters an R2 key id to sign MinIO requests with:
403 on every WAL segment, and archive_mode=on turns that into a disk-full clock on the pgdata
PVC, not a degraded backup. Dev's PROVEN DR restore (R-STATE-03) is the thing that would have
broken.
This commit splits the material. Each credential is now TWO Secret objects, both applied to
BOTH clusters, differing only in name suffix:
dev -> velero-cloud-credentials keycloak/nextcloud/semaphore-db-backup-credentials
prod -> velero-cloud-credentials-prod ...-db-backup-credentials-prod
The consumer picks one by name; that wiring is the NEXT commit, so this one changes no
rendered reference and is a no-op for both live clusters.
WHY DEV KEEPS THE UNSUFFIXED NAME. Dev is live. Renaming its Secret would change
`s3Credentials.accessKeyId.name` on three running CNPG Clusters — which projects into the
instance pods' environment and therefore ROLLS keycloak-db, nextcloud-db and semaphore-db —
and would roll the velero Deployment, all to rename an object whose contents do not change.
This repo already makes exactly this call for dev's live identifiers (EXTERNALDNS_TXT_OWNER_ID
"default", VELERO_BACKUP_PREFIX ""): the live cluster keeps the name it is running, and the
NEW environment takes the new one. The asymmetry is deliberate and documented at each
kustomization.
WHERE THE MATERIAL CAME FROM. Neither side is invented:
- the four unsuffixed files are restored byte-for-byte from `main` (the MinIO material as
it stood before 0a6a862), so dev's Secrets are IDENTICAL to what dev is running;
- the four `-prod` files carry the R2 token from 0a6a862 (already on this branch as
|
||
|
jakman85andClaude Opus 5
|
0a0527e7db |
feat(backup): prod's off-cluster DR lands on R2 — and stops the WAL clock
Prod has been archiving WAL to `TBD-PROD-S3-BACKUP-ENDPOINT` since bootstrap. Verified live 2026-09-07 on all three CNPG Clusters: ContinuousArchiving=False, "unexpected failure invoking barman-cloud-wal-archive: exit status 4". Because archive_mode is ON, that is not a degraded backup — it is a DISK-FULL CLOCK. Postgres will not recycle a segment it has not archived, so keycloak-db's pgdata PVC is at 579 MiB used / 4360 MiB free and climbing monotonically. When it fills, the primary PANICs and takes SSO down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Wiring the endpoint is what lets Postgres drain the whole accumulated backlog in one pass. THE KNOB LAYOUT (three new, one repurposed) and why it is shaped this way. The operator created exactly ONE R2 bucket, `k3s-prod-backup`, but the two backup tools want differently-SHAPED values: the Velero chart takes `bucket` and `prefix` as separate fields, while CNPG's barmanObjectStore.destinationPath is one full s3:// URL whose first path segment IS the bucket. So: S3_BACKUP_ENDPOINT dev http://192.168.1.9:9000 prod https://<acct>.r2... S3_BACKUP_REGION dev us-east-1 prod auto VELERO_BACKUP_BUCKET dev velero prod k3s-prod-backup VELERO_BACKUP_PREFIX dev "" prod velero CNPG_BACKUP_PATH dev s3://cnpg prod s3://k3s-prod-backup/barman There is deliberately NO separate `S3_BACKUP_BUCKET`: CNPG's bucket already lives inside CNPG_BACKUP_PATH, and a second knob holding the same bucket name twice is a rotation waiting to drift. Three knobs, each shaped for its own consumer, and no two of them can contradict each other. TENANCY. Inside k3s-prod-backup, velero and barman get disjoint TOP-LEVEL prefixes — s3://k3s-prod-backup/velero/ and s3://k3s-prod-backup/barman/<cluster>/ — with the bucket ROOT left empty so a misrendered prefix lands somewhere obviously wrong instead of silently colliding. This is not tidiness: Velero ADOPTS every backup under its location and TTL-DELETES it on retention, so an overlapping prefix would let it reap WAL segments; and interleaving two Postgres servers' WAL into one prefix corrupts the chain outright. Dev↔prod isolation is doubly independent — different endpoint AND different bucket names — so neither cluster can reach the other's objects even with a mis-substituted prefix. DEV IS UNCHANGED, BYTE-FOR-BYTE, and that is asserted rather than assumed. Base defaults are dev's LIVE values (verified against the running cluster: BSL bucket=velero with NO prefix, region us-east-1; all three CNPG Clusters ContinuousArchiving=True at s3://cnpg on the MinIO), and the dev overlay now restates all five explicitly for the same reason LONGHORN_DATA_PATH is restated — a silent drift in base would ORPHAN the backups dev's PROVEN restore depends on. The empty-prefix case was the one real risk, and it is settled empirically: with prefix "" the chart omits objectStorage.prefix from the rendered BackupStorageLocation ENTIRELY (helm template, chart 12.1.0), so dev's live BSL is unchanged, not merely equivalent. R2's TWO GOTCHAS, both wired up front: - region must be exactly `auto`. head-bucket answers {"BucketRegion":"WNAM"}, which is a PHYSICAL LOCATION HINT, not a signing region. This was hardcoded us-east-1 inside the Velero HelmRelease — correct-by-accident for MinIO, wrong for anything else — so it is a knob now. - checksumAlgorithm: "". velero-plugin-for-aws v1.14.2 bundles an AWS SDK with flexible checksums on by default and sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT; R2 rejects those with HTTP 400 InvalidArgument, failing every backup and every kopia blob upload. It must be PRESENT-but-empty (the plugin branches on the key existing), and the chart does emit an empty value — verified by rendering it. Harmless against MinIO, so it is unconditional rather than a fourth knob. WHAT THIS DOES NOT DO. Velero stays SUSPENDED on prod. `spec.suspend: true` is owned by field manager `flux` (verified: kustomize-controller's managedFields do not include f:suspend), so server-side apply will not strip it, and nothing here declares suspension — which would also trip R-ENV-03, since a prod overlay may patch only cluster-vars and a Flux Kustomization's `decryption`. Unsuspending is the operator's, after `standup.sh --s3-endpoint <r2> s3check` proves a round trip. NO DNS KNOB IS TOUCHED. BASE_DOMAIN stays prod.home.keithjackson.dev and every EXTERNALDNS_* value is untouched. The apex inversion on work/prod-standup-campaign was deliberately NOT carried: this branch ports that branch's R2 reasoning knob by knob, and merges nothing. VERIFIED: ci/l0.sh green (flux-postbuild-substitution 272 assertions, 0 hazards, 33 knobs defined / 28 referenced, every new token resolved in BOTH overlays); render-golden, lint, schema, spec-immutability green; touched requirement R-STATE-03 covered. Refs R-STATE-03, R-IDP-05, R-ENV-02, R-ENV-03. Co-Authored-By: Claude Opus 5 <[email protected]> |