8891df07b0804806c69c14b65cf4860d111e3019
80
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
cf7d97e546 |
fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
72f0ba4e1f |
fix(qbittorrent): the comment warning about doubled dollars contained one
The new flux-postbuild-substitution guard failed the merge on
apps/wave-0/qbittorrent.yaml:155 — a run of two dollars in a substituted
path that nothing justifies. It was inside the postStart hook's own comment
EXPLAINING the hazard: prose about what envsubst does to a doubled dollar,
written with a literal doubled dollar.
Functionally harmless — it is a bash comment, and collapsing it changes no
behaviour — but the guard cannot distinguish a comment from content that
reaches the workload, and it is right not to try: that is precisely how the
nextcloud-db break slipped through. Reworded in words instead.
Worth recording that this is the second time today the same trap caught an
explanation of itself: my own fix for the nextcloud-db incident (
|
||
|
jakman85andClaude Opus 5
|
e4bd76070e |
fix(qbittorrent): a durable WebUI credential, and the two bugs the WIP hid
Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.
Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.
Two real bugs found in the WIP, both the postBuild-substitution trap:
* PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
(envsubst collapses it) and the renderer accepts it, but run any other way
bash reads the doubled dollar as its PID, so every curl got an unparseable
URL and the hook logged "the WebUI rejected the durable credential" three
times having never asked it anything — see the marker log at 05:35Z. Now a
literal PORT=8080; the script contains no braced expansion at all and
behaves identically however it is run.
* One shared wait budget covered both the s6 wait and the API wait, so the
first consumed all 90s and attempts 2-3 fired instantly. Separate counters
now; "the API never answered" is reported as a distinct fault from "the
password was rejected"; and a rejection restarts qbittorrent-nox so a retry
can actually change its outcome.
apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.
Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.
NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
204f0ec28e |
WIP: durable qBittorrent WebUI credential
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.
Problem: qBittorrent 5.1.2 has no WebUI\Username or WebUI\Password_PBKDF2 in
its config, so it mints a NEW temporary admin password on every process start
and announces it only on stdout. Two consequences, both real: the Sonarr and
Radarr download clients configured earlier will silently stop working at the
next pod restart, and a human operator has no stable login either.
Adds a SOPS-encrypted qbittorrent-bootstrap Secret (verified encrypted: both
values are ENC[AES256_GCM]) plus the catalog/rendered wiring to consume it.
WHAT REMAINS:
- The agent's last note: the postStart hook works, but its output never
reaches kubectl logs, so it was switching to the marker-file logging
pattern gitea's hook already uses. That change is NOT here.
- The Secret exists on prod as kubectl-client-side-apply, i.e. live-only;
merging adopts it into git.
- Sonarr's and Radarr's stored download-client passwords were to be updated
to the durable value — NOT done, so the chain may break on restart until
it is.
- The required proof is unfinished: restart the pod, then show the WebUI
accepts the credential AND both arrs still report isValid: true.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
16b7fcd677 |
fix(cnpg): the grant reconciler died on every Flux-managed run
The SQL asserted convergence in a dollar-quoted PL/pgSQL DO block. That cannot survive this delivery path: the ConfigMap is rendered through Flux's postBuild substitution, and envsubst collapses a doubled dollar into one, so Postgres received a half-quoted DO and failed with a syntax error. Three scheduled runs had left nine Error pods in nextcloud-db. It passed verification because it was hand-applied with kubectl, which runs no envsubst — the defect only existed once Flux owned the object. The GRANT on the first line still succeeded, so the privilege held and the blind spot stayed closed; what was broken was the assertion meant to prove it, which is the part that makes a silent regression visible. Rewritten in plain SQL with no dollar sign anywhere in the file, comments included. Escaping by doubling again would also work and would be the next person's trap. has_database_privilege()::int is 1 or 0, so the division raises division_by_zero and ON_ERROR_STOP=1 fails the Job; the divisor is not constant so it cannot be folded away at plan time. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
78d76fa089 |
merge: qBittorrent can finally write where the arrs can see it
qBittorrent's SavePath was /downloads and no such directory existed in the container, so the download client had no working destination and nothing downstream could see its output. It now mounts the same NFS export the arrs use, narrowed to the downloads subtree via subPath — one mount, so the EXDEV hardlink trap the bulk-tier test guards stays closed. PUID/PGID are load-bearing: the export is owned 1000:1000 and the image's default 911 is denied. |
||
|
jakman85andClaude Opus 5
|
78bc53afc7 |
feat(qbittorrent): attach the downloads area of the bulk export — the chain was broken end to end
qBittorrent was the one member of the download chain with NO media volume.
Its live config said
Session\DefaultSavePath=/downloads/
Session\TempPath=/downloads/incomplete/
Downloads\SavePath=/downloads/
while `ls /downloads` in the container returned "No such file or directory".
The catalog entry declared exactly one volume — config at /config on longhorn —
and carried a comment deferring the downloads volume "until OQ-2 is resolved".
OQ-2 WAS resolved (schema 1.5.0 static bulk NFS PVs); plex, sonarr, radarr and
nextcloud all took their mounts and qBittorrent's was simply never added. So the
download client was writing, or failing to write, a path that did not exist —
and anything it did create would have been invisible to sonarr/radarr, lost on
every pod recreation, and consuming the node's ephemeral disk until then.
WHAT THIS ADDS to catalog/qbittorrent.yaml (rendered into apps/wave-0):
* storage[downloads]: class bulk, export /mnt/Storage/plex, subPath downloads,
mountPath /downloads, 9Ti advisory. ONE mount of the SAME export sonarr,
radarr (rw) and plex (ro) already consume, narrowed to the handoff subtree.
* env PUID=1000 / PGID=3000. LOAD-BEARING, not cosmetic: the container ran as
uid 911 and /mnt/Storage/plex/downloads is 0755 owned 1000:1000, so the
export would have mounted cleanly and then failed every write with EACCES.
Measured both ways on prod: a write as uid 1000 succeeded, the same write as
uid 911 was denied.
* podSecurityContext.fsGroup 3000, for the LONGHORN config PVC only, exactly as
sonarr/radarr set it — fsGroup is a no-op on an in-tree NFS volume.
WHY subPath AND NOT THE EXPORT ROOT AT /media (sonarr/radarr's shape): a download
client has no business writing the library. The arrs need the root mount because
THEY hardlink downloads/ -> tv/|movies/ and link() is a same-MOUNT operation;
qBittorrent performs no such link, so narrowing to downloads/ removes movies/,
tv/ and photos/ from its mount namespace without removing any operation it
performs. It is still ONE mount of the export, so the L0 assertion that no
workload mounts one export twice (the EXDEV trap) holds — and the arrs' own
link() stays intra-mount. Proven live: nlink 1 -> 2, same inode, no EXDEV, every
step executed as uid 1000.
NETWORK: nothing to grant and nothing to steer. An NFS PersistentVolume is
mounted by the NODE KERNEL in the host netns — the established socket to
192.168.2.102:2049 is on the node's own LAN address, the pod netns has no socket
to 2049 at all, and no CiliumNetworkPolicy on this cluster names that host or
port. `ip route get 192.168.2.102` picks eno1 even on the tunnel-owning node, and
Cilium's own egress map lists 192.168.0.0/16 as an Excluded CIDR for every
vpn-class pod while 0.0.0.0/0 goes to the gateway. Public egress still exits the
PIA address; the VPN guarantee is untouched.
NOT CLOSED BY THIS COMMIT, deliberately: absolute paths still differ between this
pod (/downloads/x) and the arrs (/media/downloads/x), so Completed Download
Handling needs one Remote Path Mapping per arr. That lives in the arrs' own
databases, not in git.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
7880d84bce |
fix(cnpg): grant cnpg_metrics_exporter CONNECT on the nextcloud database
The CNPG metrics exporter could not CONNECT to the `nextcloud` application
database, so every user-defined collector query against that target failed on
all three instances:
FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
DETAIL: User does not have CONNECT privilege.
The whole cnpg_pg_stat_* / pg_settings / pg_postmaster family was therefore
MISSING for nextcloud-db while the scrape target still reported up=1 — the blind
spot platform/cnpg/alerts.yaml's CNPGMetricsCollectorErroring was written to
catch. It had been firing on all three pods; measured before this change,
cnpg_pg_settings_setting had 765 series for keycloak, 765 for semaphore-db and
ZERO for nextcloud-db. Nextcloud's database had never been monitored.
Root cause is the live ACL, not the spec — all three Cluster specs bootstrap
identically:
nextcloud : datacl = {nextcloud=CTc/nextcloud} <- explicit, no PUBLIC entry
keycloak : datacl = NULL
semaphore : datacl = NULL
A NULL datacl means PostgreSQL's defaults, which include CONNECT for PUBLIC, so
keycloak and semaphore admit the exporter only by inheriting PUBLIC's implicit
CONNECT. On `nextcloud` something ran the equivalent of REVOKE ALL ON DATABASE
nextcloud FROM PUBLIC, which materialises datacl and drops that implicit entry.
That REVOKE is in neither this repo nor DECISIONS-LOG.md — it was applied out of
band, which is why the remedy has to be declarative rather than another hand-run
psql.
CNPG 1.30.0 cannot express a database-level GRANT (verified against the live
CRDs): `Database` has no ACL field (`allowConnections` is the datallowconn flag,
not a per-role grant), and managed.roles/DatabaseRole cover role attributes plus
`inRoles` membership only — and membership would hand the exporter the owner's
CREATE/TEMP and all table access to buy one CONNECT. postInitApplicationSQL runs
once at bootstrap, so it cannot repair the already-bootstrapped instances that
are actually broken. So this follows the pattern the repo already uses for state
Flux cannot reach: the level-triggered CronJob reconciler alongside
platform/keycloak/client-secret-reconciler.yaml. A one-shot Job is deliberately
avoided — a completed Job never re-runs, so a rebuild or PITR restore would
silently reopen the blind spot.
Exactly one privilege, and nothing else:
GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
giving `cnpg_metrics_exporter=c/nextcloud` — CONNECT only, no CREATE, no
TEMPORARY, no ownership, no superuser, no table or schema grants. Nothing more is
needed because CNPG already creates the role as a member of the built-in
pg_monitor role (verified on all three clusters), which carries the read access
the collector's pg_catalog queries want. It runs as the database OWNER, not a
superuser: enableSuperuserAccess stays off, and a database owner holds the grant
option on its own database.
Verified live on prod:
* the grant issued under SET ROLE nextcloud, proving no superuser is needed;
* permission-denied and "Error collecting user query" both went to 0 on
nextcloud-db-1/-2/-3, i.e. every collector query now succeeds;
* cnpg_pg_settings_setting now 765 series for nextcloud-db, matching the other
two, and cnpg_pg_postmaster_start_time present on all three instances;
* cnpg_last_error = 0 on all NINE instances, so keycloak-db and semaphore-db
are unaffected and their datacl is still NULL;
* convergence proven by revoking the grant again and letting the committed
CronJob restore it, then re-running it against a converged database to show
idempotence;
* CNPGMetricsCollectorErroring, firing on all three pods beforehand, has
cleared (`for: 6h` delays firing, not resolution).
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
d617983e83 |
feat(gitea): render gitea-allow-flux-ingress — prod's GitOps source, in git at last
The CNP that lets flux-system's source-controller clone from the in-cluster Gitea Service was APPLIED LIVE on prod on 2026-09-13 and existed nowhere in this repo. It carried a provenance annotation saying so. Until now it could not be authored: the gitea CNPs are GENERATED from the catalog entry, and the contract had no field for "also admit this in-cluster client" (fixed in the previous commit). Why it matters more than a missing allow normally would. prod's GitRepository points at the Service, not a public name (clusters/prod/kustomization.yaml: `url: http://gitea.gitea.svc.cluster.local:3000/Home-Network/home-network-k3s-impl.git`), deliberately, so prod's GitOps depends on neither public DNS, Cloudflare, prod's own Gateway nor ${BASE_DOMAIN}. But gitea-allow-gateway-ingress admits only `fromEntities: [ingress]`, so source-controller's clone was dropped — and the HTTPRoute that would have served the post-apex-flip public name instead lives in a DOWNSTREAM tier that cannot apply while the source is down. Source down -> routes never rename -> source stays down. Every `kubectl delete cnp` and every cluster rebuild re-armed that deadlock, and it would have been extremely hard to diagnose cold. The rendered object is EQUIVALENT to the live one — same apiVersion/kind, same name/namespace/labels, and a byte-identical spec (endpointSelector app.kubernetes.io/name=gitea; fromEndpoints matchLabels k8s:io.kubernetes.pod.namespace=flux-system; toPorts 3000/TCP), verified against `kubectl --context k3s-prod -n gitea get cnp gitea-allow-flux-ingress -o json`. The name matches, so Flux ADOPTS the live object in place: no spec change, so no policy recompute and no flap on prod's own source. The only difference is that the rendered object carries no `homelab.keithjackson.dev/provenance` annotation — the reasoning now lives where the object is authored, in catalog/gitea.yaml, next to the field that generates it. Golden re-rendered with the canonical command; the whole apps/ diff is the one new document. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
5da116b34d |
fix(oauth2-proxy): platform-admins may reach the media UIs — the immediate 403
The four forward-auth media UIs (jackett, qbittorrent, radarr, sonarr) gate on `--allowed-group=media-users` alone. On prod the `homelab` realm has exactly one user — the operator's `jakman85` — and it is a member of `platform-admins` ONLY. `media-users` has zero members, so those four hosts were reachable by nobody: oauth2-proxy completed OIDC discovery, redirected to Keycloak, got a token whose `groups` claim is `["platform-admins"]`, found no allowed group in it, and answered 403. No login form appeared because the browser already held a Keycloak SSO session from a platform-admins login — hence "403 immediately". pihole and crafty were unaffected: they already gate on platform-admins. `--allowed-group` is repeatable and OR-ed, so listing both groups means "media consumers OR platform operators". This grants no real privilege — platform-admins already maps to cluster-admin via k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods — and it keeps media-users meaningful for non-admin household accounts. Ruled out while diagnosing: client-secret drift (all 7 sidecar secrets hash-match keycloak-realm-clients), stale apex URLs (every live sidecar carries keycloak.prod.home.keithjackson.dev), missing group mappers (all 7 clients emit claim=groups, full.path=false), and the central oauth2-proxy pod restarts — those were Keycloak returning 503 "no healthy upstream" to OIDC discovery at startup, a consequence of the Keycloak roll, not a cause. Goldens re-rendered with the canonical command: python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation R-IDP-06 Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6b953cc56b |
fix(arrs): the config PVC was mounted where the app never looks
sonarr, radarr, jackett and qbittorrent all mounted their Longhorn config PVC at the renderer's default /data/config, but every one of these images runs with `-data=/config`. So the PVC held nothing but lost+found while the app wrote its entire configuration to the container's EPHEMERAL writable layer — destroyed on every pod recreation: restart, reschedule, image bump. Measured in-cluster before fixing, per app, rather than inferred: /data/config dev=2080 [lost+found] <- the Longhorn PVC /config dev=2053 [Sentry asp config.xml logs ...] <- the node's own disk and a marker file written to /config did not appear in /data/config, so they are demonstrably different storage. qbittorrent's /config sat on dev=66309 (/dev/nvme0n1p5), i.e. whichever node it happened to land on. This is the SAME failure catalog/plex.yaml:54 already records having fixed for Plex; plex was the only one of the five that was right. The live /config contents were copied onto each PVC BEFORE this landed, so jackett's API key and qbittorrent's credentials — which other apps reference — survive the move rather than being regenerated. catalog/sonarr.yaml:61 documented this as a latent bug and deliberately left it because it changes live on-disk state. It is being fixed now because all four pods had just been recreated, so their configs were freshly generated and nothing had accumulated to lose. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
f5e62d9169 | merge: bulk NFS volumes for plex, sonarr, radarr and nextcloud | ||
|
jakman85
|
cac413e87a |
feat(cnpg): synchronous replication, quorum of 1, dataDurability preferred
Three instances replicating ASYNCHRONOUSLY still lose data on failover: COMMIT returns as soon as the primary has flushed locally, so a primary that dies before its standbys catch up takes the last few transactions with it and the promoted standby comes up quietly missing them. On keycloak-db that is a just-created user, a rotated credential or a consent vanishing at exactly the moment of a failover. `method: any` + `number: 1` makes COMMIT wait for ONE of the two standbys to confirm — the point of having them. dataDurability: preferred, NEVER required. `required` is a strict quorum: fewer than `number` healthy standbys and writes BLOCK. Postgres here shares three nodes with etcd and the CNI, so one degraded standby — a node reboot, a Longhorn replica rebuild, a Cilium restart — would turn a partial degradation into a total write outage, and on the SSO database that takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth with it. `preferred` keeps synchronous commit while a healthy standby is there and relaxes the required count dynamically when one is not. dataDurability is only honoured while standbyNamesPre/standbyNamesPost are unset — both are. NON-DISRUPTIVE: this compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter. No pod restart, no switchover, no client disconnect. It is the last of the three commits because it wants standbys that already exist and are already on their final nodes. Field shape verified against the live clusters.postgresql.cnpg.io CRD before relying on it (CNPG 1.30.0, byte-identical schema on k3s-prod and k3s-dev): spec.postgresql.synchronous requires `method` (enum any|first) and `number` (x-kubernetes-validation self > 0); `dataDurability` is enum required| preferred. None of the three Clusters previously had a `postgresql:` block, so this adds one rather than merging into existing parameters. DEV IMPACT: identical and non-disruptive, same reload-only path. Verified: yamllint clean; both tiers render the block. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit f1ce19461415e77eb2f1ca5109ebab49edf0860b) |
||
|
jakman85andClaude Opus 5
|
82aff4d053 |
feat(apps): attach the media library and the legacy Nextcloud dataset over NFS
plex, sonarr, radarr and nextcloud each gain one bulk volume. Authored in the catalog and re-rendered, never hand-edited into apps/ (render-golden: 10/10). Layout, the export change the operator must apply, the uid analysis and the validation probes are in docs/BULK-NFS-STORAGE.md; everything there was measured on the host over read-only SSH, not assumed. WHAT THE LAYOUT TURNED OUT TO BE, AND WHY IT DECIDED THE DESIGN /mnt/Storage/plex holds movies/ tv/ photos/ AND downloads/ as SIBLINGS in one dataset — so the arrs' import source and their libraries are one filesystem on the server, and the download client's completed folders are already named radarr/ and tv-sonarr/. The k3s arrs are joining a tree an arr stack has been driving for a long time. That sibling fact is the whole argument for the arrs mounting the export ROOT rather than a subPath per directory. Sonarr and Radarr import by hardlinking or atomically renaming, both same-filesystem operations, and to the kernel "same filesystem" means SAME MOUNT. Two subPath mounts of one export are two mounts: link() across them returns EXDEV regardless of what the server knows, and the arrs do not fail loudly — they degrade to copy-then-delete, rewriting every file over NFS and destroying the seeding copy. The smallest subtree containing both tv/ and downloads/ IS the export root, so the root mount is the MINIMAL mount that preserves hardlinks. Least privilege bounded below by a hard requirement. The cost is stated, not hidden: sonarr can also write /media/movies. plex /mnt/Storage/plex -> /media ro ReadOnlyMany sonarr /mnt/Storage/plex -> /media rw ReadWriteMany radarr /mnt/Storage/plex -> /media rw ReadWriteMany nextcloud /mnt/Storage/nextcloud -> /mnt/legacy-nextcloud ro ReadOnlyMany (subPath: nextcloud) Plex is read-only because it indexes and streams; every write on that tree belongs to the arrs, and saying so removes Plex from the set of things that can damage 8.33 T. All three media consumers use the SAME mountPath deliberately: when qbittorrent is eventually given this export, the arrs and the download client will already agree on absolute paths, which is the other precondition for hardlinked imports. *** /media AND NOT /data — A COLLISION CAUGHT WHILE RENDERING. *** The arrs' Longhorn config PVC renders at the renderer default /data/config. A media mount at /data is mounted OVER it and MASKS the config volume: the pod comes up with its app database shadowed by an 8 T media tree. That discovery also exposed a SEPARATE PRE-EXISTING BUG, reported and deliberately NOT fixed here: LinuxServer arr images run with -data=/config, so /data/config is very likely the wrong home for that PVC and the app has been writing its EPHEMERAL layer — the exact failure catalog/plex.yaml's mountPath comment records already having been bitten by. Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it starts those apps from an empty config volume, so it needs its own change with the operator present. This change does not make it worse. UID/GID — THE FAILURE THIS TIER ALWAYS HAS The export is sec=sys + no_root_squash, so the client's numeric uid passes through verbatim and the server's ownership is the entire access story. The tree is 0755 owned 1000:3000, so ONLY uid 1000 can write it. LinuxServer images default to 911. Without intervention sonarr and radarr would have mounted cleanly, gone Ready, and failed EVERY import with EACCES. And fsGroup cannot fix that — the in-tree NFS plugin does NO ownership management, so podSecurityContext.fsGroup is a no-op on an NFS volume. Reaching for it is how this gets misdiagnosed. So PUID=1000/PGID=3000 are added to both arrs (PGID 3000 keeps what they write group-readable to the live non-k3s consumers of the same export), and fsGroup: 3000 is added for the LONGHORN config PVC only, so changing PUID does not depend on the image re-chowning /config. plex already runs as 1000 and reads a 0755 tree; nextcloud reads as www-data. Neither needs work. NEXTCLOUD IS MOUNTED READ-ONLY, WHICH IS A DEVIATION FROM THE BRIEF — ON EVIDENCE The task said nextcloud writes. The dataset says otherwise: of 270 G, 267 G is config_backup/, the rest is config/ (103 M) and config_old/ (3.4 G). There is NO data/ directory. This is the OLD non-k3s instance's configuration plus a backup blob — an archive — and that instance is STILL LIVE on it (exported rw to 192.168.1.6 and .5, which the operator has said must stay active). Two Nextclouds writing one instance's config directory, each with its own database and therefore its own oc_filecache, is unrecoverable corruption. "Do not grant more than needed" lands on ro, and ro is the trivially reversible choice. If the intent is in fact a MIGRATION, that is a larger and more deliberate change than a mount; the one-line flip is documented. Read-only for nextcloud can additionally be enforced SERVER-side (only nextcloud consumes that dataset from k3s), and the sharenfs value in the doc uses ro= for the k3s clients. The media tree cannot: plex and the arrs may land on any node, so the server sees the same addresses for reader and writers. Plex's ro is therefore a client-side guarantee — stated as a limitation, not left implied. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
e8d8601ca6 |
feat(cnpg): required pod anti-affinity + switchover primary rolls
`instances: 3` alone does not keep the three instances apart. All three Clusters used CNPG's default `podAntiAffinityType: preferred`, which is a scheduler HINT — two instances may share a node whenever that scores better, and they WILL on the path that matters: when CNPG recreates an instance after a node loss, the surviving nodes are the only candidates and one of them already runs an instance. `preferred` is how a 3-instance cluster decays back into the single-node failure domain the previous commit exists to remove. `required` makes it a scheduling constraint. On a 3-node cluster with no spare, the cost is that an instance whose node is gone stays Pending until a node returns. That is correct: a replacement squeezed onto an already-occupied node dies with that node while making the cluster LOOK three-way replicated. 2 of 3 instances still serve and CNPG still promotes. ALSO SETS primaryUpdateMethod: switchover. This is not cosmetic. Changing the affinity changes the instance pod spec, so CNPG performs a rolling update, and the CRD default for primaryUpdateMethod is `restart` (verified against the live CNPG 1.30.0 CRD on both clusters) — a delete-and-recreate of the PRIMARY pod, i.e. tens of seconds of no Postgres. On keycloak-db that is an SSO outage, which also means kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. With standbys now present, `switchover` promotes an already-updated streaming replica instead, reducing the roll to a connection reset. Known constraint documented in the manifest: under `switchover` the operator rejects an update that changes imageName and a postgresql parameter in the same reconcile. Verified satisfiable before merge (read-only): k3s-prod and k3s-dev each have 3 Ready, schedulable, untainted nodes, and every pgdata PVC already has a Longhorn replica on all three (replica-soft-anti-affinity=false), so any instance can attach from any node. DEV IMPACT: identical, and dev can absorb it. Longhorn dev nodes are 124.9 GiB each with 152.0 GiB already scheduled under a 200% over-provisioning ceiling (249.8 GiB); the six new 5Gi volumes add 30 GiB scheduled per node -> 182.0, still inside the ceiling, and only ~600 MiB per volume is really consumed. Prod is far looser: 239 -> 269 GiB scheduled against a 829 GiB ceiling. THIS IS THE DISRUPTIVE COMMIT of the three. Land it on its own reconcile, after the scale-out has produced two Ready standbys per Cluster, so the blast radius is one clean switchover per database rather than a scale-out and a roll at once. Expect a brief SSO interruption for keycloak-db. Verified: yamllint clean; both tiers render the new fields; field names, types and enum values checked against the live clusters.postgresql.cnpg.io CRD (CNPG 1.30.0, identical on prod and dev). Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit e1d364238d73e6ee99c9a50108fe8b4e09481a7e) |
||
|
jakman85
|
f338e91770 |
feat(cnpg): instances 3 on all three Clusters — a standby to fail over to
All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication saves the DATA when a node dies; it does not save the SERVICE — a single instance has to be rescheduled, re-attach its volume and replay WAL before Postgres accepts a connection again. With nothing in the spec keeping them apart, all three primaries had drifted onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give each Cluster a streaming standby CNPG can promote in seconds, repointing the -rw Service with no volume re-attach and no restore. UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate corruption and deletions faithfully; only the Barman archive is a backup, and adding standbys while archive_mode=on with a failing archive_command would have multiplied the disk-full clock, not hedged it. All three Clusters on both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real firstRecoverabilityPoint (verified read-only, 2026-09-07). Scaling out is non-disruptive on its own: CNPG creates -2 then -3 sequentially, each seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are not disconnected. The anti-affinity change that DOES cause a switchover is deliberately the next commit. DEV IMPACT: identical — dev is on the same shape (three single-instance DBs, all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes and Longhorn headroom for it (see the next commit's message for the numbers). Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning. Verified: yamllint clean; `kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0` both render `instances: 3`. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02) |
||
|
jakman85
|
09265e0381 | merge: OIDC issuer/callback URLs become per-environment | ||
|
jakman85andClaude Opus 5
|
6548f9a89c |
fix(semaphore): per-environment OIDC URLs — same latent break as Headlamp
Semaphore's whole SEMAPHORE_OIDC_PROVIDERS JSON blob lived inside the SOPS-encrypted
semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
client secret. That buried two environment-specific, NON-SECRET URLs in ciphertext —
provider_url (the realm issuer) and redirect_url — where Flux's ${BASE_DOMAIN} postBuild
substitution cannot reach. Both were frozen at `home.keithjackson.dev`, so prod's Semaphore
was pointed at dev's Keycloak and dev's own redirect URI: identical root cause to the
Headlamp login timeout, not yet reported only because nobody had tried prod SSO.
FIX: split the blob. The URLs are public and now live as a plaintext literal in
catalog/semaphore.yaml (rendered to the apps/wave-1 golden), ${BASE_DOMAIN}-substituted like
every other hostname. Only the client secret stays encrypted, in its own key
`oidc-client-secret`, and the KUBELET splices it back via dependent-env-var expansion:
$(SEMAPHORE_OIDC_CLIENT_SECRET) resolves against the secretKeyRef env var declared
immediately above it (order is load-bearing — the kubelet resolves env in list order and
leaves a forward reference as literal text).
Verified: the new key holds the same value as keycloak-realm-clients/semaphore-client-secret
(the mirror invariant), Semaphore v2.19.10's util.OidcProvider consumes client_secret as
before, and `flux envsubst --strict` leaves $(...) untouched while resolving ${BASE_DOMAIN} —
dev renders keycloak.home.keithjackson.dev, prod keycloak.prod.home.keithjackson.dev.
apps/wave-1/semaphore.yaml is a render golden; regenerated with the canonical
`catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
161430d915 |
feat(backup): the overlay picks the credential — BACKUP_CREDENTIALS_SUFFIX
The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).
cluster-vars BACKUP_CREDENTIALS_SUFFIX base "" | dev "" | prod "-prod"
velero HelmRelease credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
3x CNPG Cluster s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.
THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.
Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).
VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:
strict envsubst PASSES for all 8 (env x tier) combinations
dev velero existingSecret velero-cloud-credentials
dev keycloak/nextcloud/semaphore-db <db>-backup-credentials (both key refs each)
prod velero existingSecret velero-cloud-credentials-prod
prod keycloak/nextcloud/semaphore-db <db>-backup-credentials-prod (both key refs each)
both all EIGHT Secret objects present in the rendered output (dev and prod alike)
no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
escapes in apps/wave-0, which are pre-existing and unrelated.
Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
dev velero s3://velero (root) + cnpg s3://cnpg @ http://192.168.1.9:9000 region us-east-1
prod velero k3s-prod-backup/velero + barman prefix @ https://<acct>.r2.cloudflarestorage.com region auto
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e7c4a95c2d |
feat(backup): the four backup credentials become TWO objects each, one per environment
The four backup-credential SOPS Secrets are SHARED BASE manifests: platform/ and apps/ are
rendered by the same Flux tiers on both clusters, and the ONLY per-environment difference in
this repo is cluster-vars substitution (R-ENV-02). So writing the R2 token into all four
(0a6a862) did not "give prod credentials" — it REPLACED DEV'S. Merging that as-is would have
handed dev's Velero and all three dev CNPG Clusters an R2 key id to sign MinIO requests with:
403 on every WAL segment, and archive_mode=on turns that into a disk-full clock on the pgdata
PVC, not a degraded backup. Dev's PROVEN DR restore (R-STATE-03) is the thing that would have
broken.
This commit splits the material. Each credential is now TWO Secret objects, both applied to
BOTH clusters, differing only in name suffix:
dev -> velero-cloud-credentials keycloak/nextcloud/semaphore-db-backup-credentials
prod -> velero-cloud-credentials-prod ...-db-backup-credentials-prod
The consumer picks one by name; that wiring is the NEXT commit, so this one changes no
rendered reference and is a no-op for both live clusters.
WHY DEV KEEPS THE UNSUFFIXED NAME. Dev is live. Renaming its Secret would change
`s3Credentials.accessKeyId.name` on three running CNPG Clusters — which projects into the
instance pods' environment and therefore ROLLS keycloak-db, nextcloud-db and semaphore-db —
and would roll the velero Deployment, all to rename an object whose contents do not change.
This repo already makes exactly this call for dev's live identifiers (EXTERNALDNS_TXT_OWNER_ID
"default", VELERO_BACKUP_PREFIX ""): the live cluster keeps the name it is running, and the
NEW environment takes the new one. The asymmetry is deliberate and documented at each
kustomization.
WHERE THE MATERIAL CAME FROM. Neither side is invented:
- the four unsuffixed files are restored byte-for-byte from `main` (the MinIO material as
it stood before 0a6a862), so dev's Secrets are IDENTICAL to what dev is running;
- the four `-prod` files carry the R2 token from 0a6a862 (already on this branch as
|
||
|
jakman85andClaude Opus 5
|
0a0527e7db |
feat(backup): prod's off-cluster DR lands on R2 — and stops the WAL clock
Prod has been archiving WAL to `TBD-PROD-S3-BACKUP-ENDPOINT` since bootstrap. Verified live 2026-09-07 on all three CNPG Clusters: ContinuousArchiving=False, "unexpected failure invoking barman-cloud-wal-archive: exit status 4". Because archive_mode is ON, that is not a degraded backup — it is a DISK-FULL CLOCK. Postgres will not recycle a segment it has not archived, so keycloak-db's pgdata PVC is at 579 MiB used / 4360 MiB free and climbing monotonically. When it fills, the primary PANICs and takes SSO down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Wiring the endpoint is what lets Postgres drain the whole accumulated backlog in one pass. THE KNOB LAYOUT (three new, one repurposed) and why it is shaped this way. The operator created exactly ONE R2 bucket, `k3s-prod-backup`, but the two backup tools want differently-SHAPED values: the Velero chart takes `bucket` and `prefix` as separate fields, while CNPG's barmanObjectStore.destinationPath is one full s3:// URL whose first path segment IS the bucket. So: S3_BACKUP_ENDPOINT dev http://192.168.1.9:9000 prod https://<acct>.r2... S3_BACKUP_REGION dev us-east-1 prod auto VELERO_BACKUP_BUCKET dev velero prod k3s-prod-backup VELERO_BACKUP_PREFIX dev "" prod velero CNPG_BACKUP_PATH dev s3://cnpg prod s3://k3s-prod-backup/barman There is deliberately NO separate `S3_BACKUP_BUCKET`: CNPG's bucket already lives inside CNPG_BACKUP_PATH, and a second knob holding the same bucket name twice is a rotation waiting to drift. Three knobs, each shaped for its own consumer, and no two of them can contradict each other. TENANCY. Inside k3s-prod-backup, velero and barman get disjoint TOP-LEVEL prefixes — s3://k3s-prod-backup/velero/ and s3://k3s-prod-backup/barman/<cluster>/ — with the bucket ROOT left empty so a misrendered prefix lands somewhere obviously wrong instead of silently colliding. This is not tidiness: Velero ADOPTS every backup under its location and TTL-DELETES it on retention, so an overlapping prefix would let it reap WAL segments; and interleaving two Postgres servers' WAL into one prefix corrupts the chain outright. Dev↔prod isolation is doubly independent — different endpoint AND different bucket names — so neither cluster can reach the other's objects even with a mis-substituted prefix. DEV IS UNCHANGED, BYTE-FOR-BYTE, and that is asserted rather than assumed. Base defaults are dev's LIVE values (verified against the running cluster: BSL bucket=velero with NO prefix, region us-east-1; all three CNPG Clusters ContinuousArchiving=True at s3://cnpg on the MinIO), and the dev overlay now restates all five explicitly for the same reason LONGHORN_DATA_PATH is restated — a silent drift in base would ORPHAN the backups dev's PROVEN restore depends on. The empty-prefix case was the one real risk, and it is settled empirically: with prefix "" the chart omits objectStorage.prefix from the rendered BackupStorageLocation ENTIRELY (helm template, chart 12.1.0), so dev's live BSL is unchanged, not merely equivalent. R2's TWO GOTCHAS, both wired up front: - region must be exactly `auto`. head-bucket answers {"BucketRegion":"WNAM"}, which is a PHYSICAL LOCATION HINT, not a signing region. This was hardcoded us-east-1 inside the Velero HelmRelease — correct-by-accident for MinIO, wrong for anything else — so it is a knob now. - checksumAlgorithm: "". velero-plugin-for-aws v1.14.2 bundles an AWS SDK with flexible checksums on by default and sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT; R2 rejects those with HTTP 400 InvalidArgument, failing every backup and every kopia blob upload. It must be PRESENT-but-empty (the plugin branches on the key existing), and the chart does emit an empty value — verified by rendering it. Harmless against MinIO, so it is unconditional rather than a fourth knob. WHAT THIS DOES NOT DO. Velero stays SUSPENDED on prod. `spec.suspend: true` is owned by field manager `flux` (verified: kustomize-controller's managedFields do not include f:suspend), so server-side apply will not strip it, and nothing here declares suspension — which would also trip R-ENV-03, since a prod overlay may patch only cluster-vars and a Flux Kustomization's `decryption`. Unsuspending is the operator's, after `standup.sh --s3-endpoint <r2> s3check` proves a round trip. NO DNS KNOB IS TOUCHED. BASE_DOMAIN stays prod.home.keithjackson.dev and every EXTERNALDNS_* value is untouched. The apex inversion on work/prod-standup-campaign was deliberately NOT carried: this branch ports that branch's R2 reasoning knob by knob, and merges nothing. VERIFIED: ci/l0.sh green (flux-postbuild-substitution 272 assertions, 0 hazards, 33 knobs defined / 28 referenced, every new token resolved in BOTH overlays); render-golden, lint, schema, spec-immutability green; touched requirement R-STATE-03 covered. Refs R-STATE-03, R-IDP-05, R-ENV-02, R-ENV-03. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
9739e9be91 |
chore(backup): R2 credentials for the off-cluster DR target
Written by platform/backup/set-r2-credentials.sh (operator, 2026-09-07T05:41:34Z) into all four consumers, in the two shapes they need: velero wants an AWS INI file, CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY. All four verified to carry the SAME 32-character key id, so no consumer was missed. Proven from a pod INSIDE prod against https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com: head-bucket on k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Byte-integrity of GET is NOT yet proven (the aws-cli image has no cmp); s3check is what will establish it. Credentials only — no endpoint or bucket wiring. That is feat/r2-backup-prod. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
4c4f5355d9 |
fix(sso): postStart hooks must exit 0 — nextcloud crashlooped the moment OIDC ran
LIVE REGRESSION, caught on the reconcile after merging #5. nextcloud went
CrashLoopBackOff with FailedPostStartHook, 7 restarts, and apps-wave-1/2 stalled.
ROOT CAUSE — a latent bug that could never fire until now. The final statement of
nextcloud's postStart script was:
[ "$p" -ge 15 ] && nextcloud_oidc_fail "could not promote ... after retries"
On the SUCCESS path (p < 15) the test returns 1, and as the script's LAST command that
becomes the hook's exit status. A non-zero postStart exit makes the kubelet KILL the
container. This was harmless for as long as the OIDC guard above it was dead code that
exited 0 first (see
|
||
|
jakman85andClaude Opus 5
|
8a2be824f6 |
feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE
Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only
L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would
overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a
path renders" has exactly one definition. It asserts: every ${...} on a substituted path
is defined-or-escaped (comments included); no default-operator form on an undefined name;
no unparseable expansion; the rendering path actually substitutes cluster-vars; every
overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing.
WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to
touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 —
so an edit to a substituted manifest under apps/ or platform/ would have filtered this
check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh.
Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form,
missing substituteFrom, un-wired gate); each fails as intended, baseline green.
STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted
paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst
--strict` exits 0 against cluster-vars base defaults. Negative control confirms the
incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict
exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped
here — that is a live-cluster change and the operator's call.
Escapes the six comment hazards that stood in the way (apps/vms/unifi x4,
platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own
contract is that $${...} emits a literal ${...}, escaping them is both precise and
self-consistent. kustomize strips these particular comments today, but comment survival is
an implementation detail — a body comment carrying $ WAS observed surviving into built
apps/wave-1 — and a survivor would kill the whole Kustomization.
DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an
indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the
literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place.
Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and
R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the
trigger for removing them.
Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV,
CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are
documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames
are hardcoded elsewhere) and are worth a follow-up.
Gate: MERGE GATE: PASS. 102 assertions, 0 hazards.
Serves: R-ENV-02, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
3484955903 |
fix(sso): gitea + nextcloud OIDC bootstrap was DEAD CODE — envsubst ate the guard
Both ServiceDefinitions guarded their Keycloak OIDC registration with an UNESCAPED
braced expansion:
if [ -z "${GITEA_OIDC_CLIENT_SECRET:-}" ]; then # gitea
if [ -z "${NEXTCLOUD_OIDC_CLIENT_SECRET:-}" ]; then # nextcloud, then exit 0
apps-wave-0/1 both declare postBuild.substituteFrom: cluster-vars, and neither name is a
cluster-vars key — so Flux's envsubst substituted them away AT RECONCILE TIME, long
before the container ever ran. Verified in the LIVE cluster: the deployed Deployments
literally contain
if [ -z "" ]; then
which is always true. The entire OIDC registration block has been skipped on EVERY boot
of both apps.
WHY NOBODY NOTICED: both apps DO have the provider registered — `gitea admin auth list`
shows keycloak/OAuth2/true and `occ user_oidc:provider` shows the keycloak provider.
Those live in each app's own database, persisted from an earlier successful registration.
SSO works today and DECISIONS-LOG is not wrong. But the idempotent re-registration is
dead, so any PVC rebuild, DB restore, or fresh cluster build comes up with NO OIDC and
only the break-glass local admin. A PROD BUILD FROM THIS REPO WOULD SILENTLY HAVE NO SSO
on gitea or nextcloud.
The irony: the surrounding code was hardened twice — see its own comments about a "silent
first-boot RACE", the verify-then-retry loops, the marker files. All of that hardening was
unreachable, because the guard above it always evaluated true.
Not a regression from the recent R-ENV-02 tokenization — introduced 2026-08-27 (
|
||
|
jakman85andClaude Opus 5
|
1157c1b071 |
fix(gitea-actions): wait for dind to ACCEPT connections, not just write certs
Re-authors the fix stranded on origin/fix/act-runner-dind-readiness (
|
||
|
jakman85andClaude Opus 5
|
3f980d2a17 |
feat(env): tokenize hardcoded dev LAN addresses into cluster-vars (R-ENV-02)
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).
New cluster-vars knobs (base default = dev literal, per the file's existing convention):
K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml
CILIUM_LB_POOL_STOP "
UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml
CRAFTY_L4_VIP platform/crafty/service-l4.yaml
S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
semaphore}-db.yaml, and platform/velero/helmrelease.yaml
Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.
clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.
PROD ADDRESS PLAN (operator, 2026-08-31):
nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
Cilium LB pool .20-.30
CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.
Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.
DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.
Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
b2752aec48 |
feat(auth): gate jackett/qbittorrent/radarr/sonarr UIs with oauth2-proxy sidecar (rollout)
Roll the verified Option-B per-app oauth2-proxy sidecar out to the remaining four forward-auth apps after the pihole canary was confirmed fail-closed (unauthenticated -> 302 to Keycloak, pod 2/2, clean sidecar OIDC discovery). Each app's Gateway route now terminates at its oauth2-proxy sidecar (:4180, --allowed-group=media-users, --upstream=http://127.0.0.1:<uiport>); the gateway-ingress CiliumNetworkPolicy allows the Gateway to reach ONLY :4180, so the bare UI is never served un-authenticated (fail-closed; R-IDP-02/06). The apps' own app/API ports stay in-Service for EAST-WEST callers (arr -> qbittorrent:8080 / jackett:9117) via the existing reciprocal-ingress allows. - keycloak: confidential jackett/qbittorrent/radarr/sonarr clients (realm-import + realm-clients mirror), provisioned live via kcadm (groups membership mapper). - secrets: four SOPS/age <app>-oauth2-proxy-credentials, wired into apps/wave-0 and apps/wave-1 by the renderer's sidecar-creds wave rule. - ci: remove the temporary canary deferral (all five goldens now reproduce). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
72d4720d58 |
feat(auth): gate pihole UI with per-app oauth2-proxy sidecar (canary)
Close the un-gated-UI security finding for forward-auth apps (R-IDP-02) by rendering the Option-B per-app oauth2-proxy reverse-proxy sidecar that crafty proves working. catalog/render.py now emits, for any forward-auth app with an exposed UI: the oauth2-proxy sidecar (upstream http://127.0.0.1:<uiport>, :4180, --allowed-group per auth.group), an authproxy :4180 Service port, an HTTPRoute backendRef -> :4180, and a gateway-ingress CiliumNetworkPolicy that allows the Gateway to reach ONLY :4180 (fail-closed: the bare UI is never served un-authenticated). oidc/native/none apps render byte-identical. This is the CANARY: only pihole's golden carries the sidecar and is verified fail-closed in-cluster first. jackett/qbittorrent/radarr/sonarr are deferred to the rollout commit (temporarily listed in ci/render-golden.sh) so their UIs keep serving exactly as today until their clients+secrets land. - keycloak: confidential pihole client (realm-import + realm-clients mirror), provisioned live via kcadm (groups membership mapper, bare names). - secret: SOPS/age pihole-oauth2-proxy-credentials (client-id/secret/cookie), wired into apps/wave-0 via the renderer's new sidecar-creds wave rule. - L1 render goldens regenerated for the sidecar (forward-auth fixtures). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
68690e5019 |
fix(net): allow SSO apps egress to keycloak backend (Cilium 1.20 CVE-2025-30162)
App SSO broke cluster-wide after the Cilium 1.16.5->1.20.1 upgrade: the CVE-2025-30162 fix (in
>=1.16.8/1.17.2) makes the shared Gateway's L7 proxy re-evaluate a pod's egress policy against
Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) when it hairpins to keycloak.home.
'toEntities: world' matches the Gateway VIP at L3/L4 but not the resolved keycloak identity at L7, so
OIDC discovery/token 403'd ('Access denied') for every app with a NetworkPolicy (gitea/nextcloud/
semaphore/headlamp/crafty...); Grafana survived only because monitoring has no NetworkPolicy.
Fix: explicit toEndpoints egress allow to the keycloak backend for SSO apps.
- catalog/render.py: emit it for auth.mode in {oidc,forward-auth}; re-rendered 8 app goldens.
- platform/headlamp + platform/crafty: hand-written policies get the same allow.
Split-horizon design unchanged. Verified live: headlamp-ns -> keycloak discovery 403 -> 200.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
d1248e6778 |
feat(backup): wire Velero + CNPG Barman to the live MinIO S3 target
Author the GitOps side of off-cluster backups against the on-Proxmox MinIO (http://192.168.1.9:9000, path-style; buckets velero + cnpg). Velero (R-STATE-03): - HelmRelease (chart 8.x / Velero 1.15) + velero-plugin-for-aws init container + node-agent kopia File System Backup (deployNodeAgent, uploaderType=kopia, snapshotsEnabled=false). BSL default -> bucket velero, s3ForcePathStyle. - Two daily Schedules: `daily` backs up the whole cluster's resources+PVC data (defaultVolumesToFsBackup opt-out) EXCEPT monitoring+unifi; `daily-resources-only` captures monitoring+unifi RESOURCES only (Prometheus TSDB, Loki 72h logs, and the ~7.4GiB UniFi VM disk are reconstructable — excluded from volume backup). - velero-cloud-credentials SOPS Secret (cloud INI key, ns velero). - Own non-gating Flux tier clusters/base/backup.yaml (dependsOn platform-config, wait:false) + clusters/dev SOPS decryption patch — mirrors headlamp. CNPG native Barman PITR (R-IDP-05 / R-STATE-03): - keycloak-db, nextcloud-db, semaphore-db gain spec.backup.barmanObjectStore (s3://cnpg/<cluster>, endpointURL MinIO, gzip WAL+data, 30d retention) + a daily ScheduledBackup (6-field cron, immediate) each. - Per-namespace scoped-key SOPS Secrets (<db>-backup-credentials), same-ns only. - render.py: emit-waves now recognizes <svc>-db-backup-credentials.sops.yaml as a DB-wave sidecar (+ self-test); wave-0 kustomization regenerated. Creds mirror platform/backup/minio-credentials.sops.yaml (scoped key k3s-backup), SOPS round-trip verified with the cluster age key. gate: green. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
cde2cd138f |
plex: add env vars (PLEX_UID/GID, TZ, ALLOWED_NETWORKS); homepage: rename Overseerr->Seerr
Plex Deployment had no Plex env vars. Add via the ServiceDefinition env list (rendered verbatim by catalog/render.py): - PLEX_UID/PLEX_GID=1000 (image chowns /config to this and drops; matches the Longhorn ownership convention seerr/semaphore use) - TZ=Etc/UTC (no other service sets TZ repo-wide) - ALLOWED_NETWORKS=10.0.0.0/8,192.168.0.0/16 so in-cluster clients (Seerr -> plex:32400) are trusted as LAN (pod CIDR 10.0.0.0/16 + svc 10.43.0.0/16 + LAN) PLEX_CLAIM intentionally not set (4-min TTL is unusable via git; one-time live claim persists on /config). ADVERTISE_IP deferred: the only candidate host resolves via split-horizon DNS to the internal Gateway VIP and its remote reachability is unverified — a wrong value breaks remote access, so omitted per prefer-safety, documented as a follow-up in catalog/plex.yaml. Homepage still labeled seerr as "Overseerr"; renamed the tile to "Seerr" (icon mdi-play-box-multiple keeps — not overseerr-specific). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
fe0ee698c4 |
fix(apps): reconcile goldens to one canonical render + enforce render==golden gate
The committed apps/ goldens were a MIX of two render flag sets: 8 services were rendered with `--no-image-automation` (no image CRs) while gitea and nextcloud carried stale ImageRepository/ImagePolicy CRs from a render WITHOUT the flag. Neither single command reproduced all goldens, and the gate did not catch it. Part A — canonical render: python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation `--no-image-automation` is canonical: image automation is authored-but-DISABLED (DECISIONS-LOG; image-reflector/-automation controllers not installed), and every app was onboarded via this flag. Re-render drops the ImageRepository + ImagePolicy objects from gitea + nextcloud only (52 line deletions, purely the trailing image-automation blocks). No functional content changed — the nextcloud postStart hardening and gitea OIDC hook are untouched. Part B — enforcement: ci/render-golden.sh renders every onboarded catalog service with the canonical command into a temp dir and FAILS on ANY diff vs committed apps/ (missing golden or content drift), across all 10 services (not just the 3 L1 spot-checks). Not-onboarded catalog entries (keycloak, vm-demo) are excluded per DESIGN-waves. Wired as a required no-cluster stage in ci/gate.sh, ci/Makefile, and both the live (.gitea/workflows/gate.yml) and dormant (merge-gate.yaml) pipelines. Proof: tampering a golden makes the stage exit 1 (blocks merge); reverting restores exit 0. `make -C ci gate` green in the devcontainer. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
7f86e1099b |
fix(nextcloud-sso): register post-logout redirect URI + harden admin promotion
Two SSO follow-ups the operator hit after Nextcloud OIDC login worked. 1. Sign-out "redirect failed". On logout user_oidc redirects the browser to Keycloak's end_session_endpoint with post_logout_redirect_uri=<Nextcloud base '/'> and client_id=nextcloud but NO id_token_hint (provider sendIdTokenHint=0). Keycloak then validates that redirect against the client's post.logout.redirect.uris attribute, which was empty -> "Invalid redirect uri" -> the operator's "redirect failed". Register https://nextcloud.home.keithjackson.dev/* on the nextcloud client (attributes.post.logout.redirect.uris). Applied live via kcadm and captured here so git stays authoritative (a RealmImport no-ops on the existing realm). 2. postStart step (d) mirrors platform-admins -> built-in admin group, but the previous single best-effort pass ran occ group:adduser once with output swallowed and no retry. At early boot (container Started ~9s after create) that call can transiently fail (DB / maintenance race), silently dropping the promotion until the next restart. Harden step (d) with a verify-and-retry loop (like step (c)): re-read group:list and re-add any platform-admins member still missing from admin until verified (bounded ~30s), no-op for existing members. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85
|
01f3dd0044 | Merge fix/nextcloud-oidc-reach: allow_local_remote_servers for private-IP IdP + platform-admins→admin (#2,#5) | ||
|
jakman85andClaude Opus 4.8
|
7f92adfac6 |
fix(nextcloud): allow SSRF to private IdP + promote platform-admins to admin
Task 1 — browser SSO "Could not reach OpenID Connect provider" (confirmed root cause, not guessed). keycloak.home.keithjackson.dev resolves split-horizon to the PRIVATE Cilium gateway VIP (192.168.1.53). Nextcloud's SSRF guard refuses its own server-side HTTP requests to RFC1918 addresses, so user_oidc's discovery/JWKS/token fetch AT LOGIN throws OCP\Http\Client\LocalServerException "violates local access rules" -> surfaced to the user as "Could not reach the provider". Proven from the live pod: nextcloud.log carries the exact LocalServerException from DiscoveryService::obtainDiscovery with Guzzle option allow_local_address:false; a Nextcloud-context IClientService->get() of the discovery URL returns LocalServerException with the flag false and HTTP 200 with it true (A/B toggled). The postStart (b) curl check passed regardless because it uses system libcurl, not Nextcloud's guard — which is why the provider registered yet login still failed. Fix (reproducible, survives pod rolls): the postStart hook now runs `occ config:system:set allow_local_remote_servers --value=true --type=boolean` right after the install/maintenance wait, on every start. Task 2 — promote Keycloak `platform-admins` members to Nextcloud admins (#5). user_oidc 8.11 has NO native group->admin mapping (occ user_oidc:provider offers group PROVISIONING and a login WHITELIST only), so the hook mirrors the provisioned `platform-admins` membership into the built-in `admin` group idempotently (occ group:list --output=json parsed via php argv; occ group:adduser admin per member). A user is promotable after their first OIDC login provisions them + the next pod start (or one manual `occ group:adduser admin <uid>`). The LOCAL admin stays as break-glass (docs/15). apps/wave-1/nextcloud.yaml re-rendered from catalog (single-file render; byte-diff confined to the embedded hook). Rendered hook passes `sh -n`; parse step validated in-pod. `make -C ci gate` green (lint/l0/l1; conformance + spec-immutability skip: no cluster / spec repo unmounted). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d835a27875 |
fix(plex): mount config PVC at /config (image's real config dir)
The plexinc/pms-docker image writes its identity/claim token and the Library metadata DB under /config, but the config PVC was mounted at the renderer default /data/config. Plex therefore wrote all state to the container's EPHEMERAL layer: the server stayed UNCLAIMED (servers.plex.tv returns 401 with an empty X-Plex-Token) and lost everything on restart, which is why seerr's "add Plex" failed even with SSL off. Set the config volume mountPath to /config in catalog/plex.yaml and re-render apps/wave-0/plex.yaml (only the mountPath line changes; all other services byte-identical). No fsGroup/securityContext needed: verified in-cluster the image inits as root (s6-svscan PID 1), chowns /config to the plex user, then drops privileges (the "Plex Media Server" process runs as plex) — root can chown the mounted PVC regardless of fsGroup. PLEX_CLAIM auto-claim is intentionally NOT wired: the env secretKeyRef schema has no `optional`, so a placeholder/absent claim Secret would crashloop the pod. The claim is an operator step (UI sign-in after the roll, or add a PLEX_CLAIM env from a fresh https://plex.tv/claim token). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d34c639269 |
fix(oidc): make gitea + nextcloud OIDC login-source registration persist
The Keycloak OIDC postStart registration for gitea and nextcloud ran but never persisted, leaving `gitea admin auth list` empty and nextcloud's user_oidc with "No providers configured" despite the apps installing/ enabling and the client secret being present. Two distinct root causes, confirmed by running the registration manually against the live warm pods: - nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs `su -c "php occ $*"`, which word-split `--scope="openid email profile"` back into three tokens, so occ saw `email profile` as extra positional args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider call as ONE `su -c` with the scope single-quoted so www-data's shell keeps it a single argument; keep the OCC() wrapper only for single-token subcommands. - gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once the app/network is warm (verified live), but fired too early at postStart and its failure was swallowed (`|| echo ... >&2`, exit 0). Hardening applied to both catalog sources (rendered into apps/wave-*): - wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200; - nextcloud also waits for `installed: true` AND `maintenance: false`; - register idempotently (create-or-update) then VERIFY the source/provider actually persisted, retrying the whole cycle (guards exit-0-but-nothing- stuck); - on give-up, log LOUDLY to stderr AND a marker file on the persistent volume (/data, /var/www/html) instead of swallowing — next failure is visible; - never crashloop: missing-secret guard retained, break-glass local admin stays usable. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85
|
76ba88a51e | Merge feature/oidc-sso-apps: native Keycloak OIDC SSO for gitea, nextcloud, semaphore | ||
|
jakman85andClaude Opus 4.8
|
f163026954 |
feat(nextcloud,semaphore): wire native Keycloak OIDC SSO (break-glass admins retained)
Extends the SSO work to the two remaining `auth: oidc` apps still on local-admin
logins, each as a native confidential client of the Keycloak `homelab` realm with
its local admin kept as break-glass (docs/15 availability caveat; R-IDP-02/06).
Realm (source of truth): add the `nextcloud` and `semaphore` confidential clients
to platform/keycloak/keycloak-realm-import.yaml (modeled on grafana/gitea) + their
NEXTCLOUD_CLIENT_SECRET / SEMAPHORE_CLIENT_SECRET placeholders. As before, a
KeycloakRealmImport no-ops on an already-existing realm, so both live clients were
ALSO created via kcadm (nextcloud e95049ed-ca10-4085-97d9-4bbef495fe61, semaphore
1835bfe9-b125-4026-8f69-33557519cf4a); the YAML keeps git authoritative.
Secrets (SOPS/age): add nextcloud-client-secret + semaphore-client-secret to
keycloak-realm-clients.sops.yaml (keycloak ns). Mirror the nextcloud secret as
oidc-client-secret into nextcloud-bootstrap (nextcloud ns). For semaphore, the
whole SEMAPHORE_OIDC_PROVIDERS JSON (which embeds the client secret) is held
encrypted as oidc-providers-json in semaphore-bootstrap (semaphore ns). All values
verified == the live clients; existing keys untouched; no plaintext in git.
App config (catalog source + re-render):
- nextcloud: add NEXTCLOUD_OIDC_CLIENT_SECRET env + an idempotent postStart hook
that installs/enables user_oidc and runs `occ user_oidc:provider keycloak`
(create-or-update; secret via --clientsecret-env so it never hits argv; occ as
www-data; group provisioning on the `groups` claim). Syntax verified against the
running user_oidc 8.11.0 on Nextcloud 31.0.14; su-to-www-data env passthrough
verified. Missing-secret/not-installed guards prevent crashloops.
- semaphore: add SEMAPHORE_OIDC_PROVIDERS env sourced from the encrypted
oidc-providers-json (provider_url=realm issuer/auto-discovery, redirect
/api/auth/oidc/keycloak/redirect). Mechanism verified against semaphore v2.19.10
source (util ConfigType oidc_providers env tag + OidcProvider struct + router
redirect path); no kubelet service-link env collision.
apps/wave-1/{nextcloud,semaphore}.yaml re-rendered via
`render.py --no-image-automation` (diffs are env/lifecycle-only).
Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run; live
homelab clients now include gitea/nextcloud/semaphore; static gate (lint/l0/l1)
green. Browser SSO login for each still needs a human check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
87388712f0 |
feat(gitea): wire native Keycloak OIDC SSO (break-glass admin retained)
Wires the bootstrapping-critical Gitea git host to the Keycloak `homelab` realm as a native confidential OIDC client, moving it off the local-admin-only login while keeping the local admin as the break-glass path (docs/15 availability caveat; R-IDP-02/06). Realm (source of truth): add the `gitea` confidential client to platform/keycloak/keycloak-realm-import.yaml (modeled on grafana) + its GITEA_CLIENT_SECRET placeholder sourced from the keycloak-realm-clients Secret. NOTE: a KeycloakRealmImport no-ops on an already-existing realm, so the live client was ALSO created via kcadm (id 70a6ca60-317a-42ce-8331-a60e380838ee); the YAML keeps git authoritative for a rebuild. Secrets (SOPS/age, never plaintext): add gitea-client-secret to platform/keycloak/keycloak-realm-clients.sops.yaml (keycloak ns) and mirror the same value as oidc-client-secret into apps/wave-0/gitea-bootstrap-secret.sops.yaml (gitea ns) so the app can read it in-namespace. Both == the live client secret. App config (catalog source + re-render): catalog/gitea.yaml gains the GITEA_OIDC_CLIENT_SECRET env (from the mirrored Secret), [oauth2_client] auto-registration/account-linking app.ini env, and an idempotent postStart step that registers the "keycloak" OAuth2 login source (auto-discovery; groups claim; platform-admins -> Gitea admin). apps/wave-0/gitea.yaml re-rendered via `render.py --no-image-automation`. Verified: realm import Done=True/HasErrors=False and platform-config Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run; static gate (lint/l0/l1) green. Browser SSO login still needs a human check. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
7b8777c1e1 |
fix(seerr): add fsGroup podSecurityContext to resolve non-root PVC CrashLoop
Seerr v3.4.1 (ghcr.io/seerr-team/seerr) runs as the non-root user node:node (UID/GID 1000), unlike the old root-running sctx/overseerr image. Its root-owned Longhorn RWO config PVC at /app/config was not writable by UID 1000, so the pod CrashLoopBackOff'd with: Error: EACCES: permission denied, mkdir '/app/config/logs/' Add a podSecurityContext with fsGroup=1000 (plus runAsUser/runAsGroup/runAsNonRoot) to the seerr ServiceDefinition — the same schema 1.3.0 mechanism semaphore uses. Longhorn's CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType applies fsGroup to this RWO filesystem volume, group-owning the PVC to GID 1000 so the process can write. Re-rendered apps/wave-2/seerr.yaml from the catalog source. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
8b0dcbf33b |
fix(seerr): move to maintained Seerr image and correct config mountPath
Two operator-flagged defects in the seerr ServiceDefinition, fixed at source (catalog/seerr.yaml) and re-rendered into apps/wave-2/seerr.yaml: - Image: sctx/overseerr:latest (original Overseerr, unmaintained) -> ghcr.io/seerr-team/seerr:v3.4.1. Overseerr and Jellyseerr have merged into the actively maintained Seerr project (seerr-team/seerr, docs.seerr.dev); ghcr.io/seerr-team/seerr is the officially recommended image. Pinned to the current stable tag v3.4.1 per the arr-stack convention (sonarr/radarr/ qbittorrent/gitea pin explicit tags; R-UPD-01). update.range bumped ">=1.33.0" -> ">=3.4.0" to track the Seerr 3.x line. Plex-native (native auth) support is retained. - Config mountPath: /data/config -> /app/config. Overseerr/Jellyseerr/Seerr store settings + request/cache DB at /app/config; the previous /data/config left the config PVC effectively unused (config not persisted across restarts). Set via the schema-supported storage[].mountPath field. Dependencies (sonarr:8989, radarr:7878, plex:32400), NetworkPolicies, public exposure, and native auth are unchanged. Render == golden verified; lint, schema, and L1 render checks green. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
12c40d8c9c |
docs: reconcile the documentation with the deployed reality
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.
Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.
README.md
- Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
- Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
- Real conformance number (36/51, 70.6%) and how to regenerate it.
- Devcontainer workflow; note that the gate was red and why.
infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
the toolchain, off-repo state that must not be lost, and the fact that Flux's
git source is an app inside the cluster (`git push origin main` IS the deploy).
infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
rationale are still worth keeping; the libvirt execution steps are not.
infra/README.md — Proxmox, plus a map of the subtree and where the cluster
actually lives (it is not on this workstation).
infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
talks to an HTTP API. Everything runs in the container now. Documents the full
CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
and the read-only spec repo especially — without the latter, spec-immutability
silently skips instead of verifying).
infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
against the running nodes), pfSense NAT marked done, and a warning that
terraform.tfstate is absent on a fresh workstation — an apply with empty state
would try to recreate the three live nodes.
infra/tofu/variables.tf — section comment disagreed with its own defaults.
infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
domains that do not exist; behaviour unchanged, but it no longer lies.
apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
since been dropped. Now: live status, 40Gi, resolved open questions marked
resolved, and the two constraints the next person will otherwise rediscover —
KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
postBuild.substituteFrom (envsubst corrupts the cloud-init).
DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
the Flux source cutover, CI going live in-cluster, ten app onboardings and the
renderer bugs they exposed, the wave split (verified: DBs adopted, not
recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
VM, and kubectl SSO. Plus verified live state and the carried-forward open list.
Two corrections to the record while verifying:
- kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
[oidc:platform-admins]), so the rolling restart
|
||
|
jakman85andClaude Opus 5
|
4cf9572c58 |
fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
The hand-authored UniFi VM manifests used column-aligned inline flow mappings
(`{ name: inform, port: 8080, protocol: TCP }`), which yamllint's `commas`
rule rejects as "too many spaces after comma". That made `make -C ci lint` FAIL,
so the merge gate has been BLOCKED on `main` since the UniFi commits landed —
every other static stage was green.
Collapse each run of spaces after a comma to one, and re-align the trailing
comments so the port tables stay readable. Whitespace only:
`kubectl kustomize apps/vms` renders byte-for-byte identically before and after.
The lint config is left untouched — relaxing the rule to make failing files pass
is moving the goalposts, not fixing the code.
Also refresh these files' stale `STAGING / DRAFT` headers: the tree is live,
reconciled by the non-gating `apps-vms` Flux Kustomization.
Gate after this change: spec-immutability / lint / l0 / l1 all green,
MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
ab592028f6 |
fix(vms): actually apply inline userData + drop secretGenerator (amend 760084e)
|
||
|
jakman85andClaude Opus 4.8
|
760084ea8c |
fix(vms): revert UniFi cloud-init to minimal inline userData (1112B)
userDataSecretRef tripped a Flux server-side-apply schema quirk on the KubeVirt
VirtualMachine CRD ("field not declared in schema"). Go back to inline userData,
trimmed to 1112 bytes (< the 2048 limit) by dropping the break-glass SSH key —
no longer needed for debugging now that the postBuild cloud-init corruption is
fixed. Keeps qemu-guest-agent (AgentConnected proves cloud-init ran) and the
public-API UOS resolver + non-interactive install. Removes the secretGenerator +
unifi-cloudinit.yaml. (Add SSH back via userDataSecretRef only if the SSA quirk is
worked around and guest access is needed.)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
e99e3d1666 |
fix(vms): deliver UniFi cloud-init via Secret (userDataSecretRef) — 2048B limit
Adding the SSH key pushed the inline cloudInitNoCloud.userData over KubeVirt's
2048-byte limit ("userdata exceeds 2048 byte limit. Should use UserDataSecretRef").
Move the cloud-init into apps/vms/unifi/unifi-cloudinit.yaml and wrap it as the
`unifi-cloudinit` Secret via a Kustomize secretGenerator (disableNameSuffixHash
so the VM can reference it by fixed name); VM now uses userDataSecretRef. Content
is non-sensitive (public key + public installer script), so not SOPS-encrypted.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
d5f8212415 |
feat(vms): add SSH key + qemu-guest-agent to UniFi VM for ops/debugging
The first UOS bring-up left no way into the guest (cloud-init set no credentials), so the in-guest UOS install couldn't be observed. Add: - ssh_authorized_keys (public key, safe to commit) on the default ubuntu user - qemu-guest-agent (KubeVirt AgentConnected + virtctl guest access) - tee the install-uos output to the serial console + /var/log/uos-install.log, and record the installer exit code, so the install is observable. Co-Authored-By: Claude Opus 4.8 <[email protected]> |