dce20b7be6de436aec4bcd2b482bfa689a656a31
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
1677dff568 |
fix(runbooks): Plex hw transcoding WORKS — the dashboard was never the proof
Hardware transcoding is confirmed working on prod (2026-10-02): VAAPI for both decode and encode on /dev/dri/renderD128, with zero-copy, measured in `Plex Media Server.log` in the running pod. The device-plugin mechanism from #37 works end to end. `runbooks/plex-quicksync.md` said the Plex dashboard showing `Transcode (hw)` was "the *only* proof available, and nothing else will ever tell you". That is false: Dashboard -> Activity showed nothing while the feature was fully working, so the page's authoritative check was silent on a healthy system and sent the operator hunting a fault that did not exist. Three inference errors corrected, two kept as named false leads: 1. The LOG is the primary check, not the dashboard. The grep ships in its REDACTING form -- that log carries API URLs with `X-Plex-Token` (CLAUDE.md §6). Also recorded: `Plex Transcoder Statistics.log` is the WRONG file (XML <Variant/> records, no ffmpeg output), and why `grep -a` is required. The dashboard is kept but demoted to best-effort, with its silence stated to prove nothing. 2. An fd snapshot is not evidence. A /proc/*/fd sweep found no holder of renderD128 while hardware transcoding was active -- sessions are short and the scan landed in a gap. The reliable process-level check is the transcoder's argv (-hwaccel, -hwaccel_device, vaapi, renderD128). 3. A missing preference key means nothing. `HardwareAcceleratedCodecs` was absent from all 27 attributes with the toggle on and working, while `HardwareDevicePath` was written and the log said "hardware transcoding: enabled". Only `="0"` is conclusive; the log settles it. Also stale and now fixed: GID 992 is a hostPath-era number. Under the plugin the injected device is root:root, so the image's hook resolves GID 0 and adds `plex` to the root group (`Groups: 0 100 1000`, `groups=1000(plex),0(root),100(users)`). Marked as an inherent property of the plugin's DeviceSpec, not a defect, and explicitly not a reason to reintroduce supplementalGroups (still a no-op -- s6-setuidgid rebuilds the group list from /etc/group). Added a per-title fallback entry (UHD 630 has no AV1 decode), labelled a HYPOTHESIS: the title that fell back was never checked. `runbooks/README.md`'s index entry was stale the same two ways and is corrected. Unchanged on purpose: the "STOP HERE IF PLEX IS Pending" section, the escape hatch, and the no-alert-is-possible reasoning (no metricsEndpoint, no ServiceMonitor, nothing a rule could evaluate). Documentation only -- no catalog/, schemas/, apps/ or platform/ change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
bce101d03f |
feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it
PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
ba79a1b59b |
Merge remote-tracking branch 'origin/main' into feat/oauth2-proxy-auth-failure-visible
# Conflicts: # runbooks/kustomization.yaml |
||
|
jakman85andClaude Opus 5
|
0fee584e3a |
feat(plex): opt into hardware.gpu: intel-quicksync, and write the runbook for the step GitOps cannot reach
Plex has been transcoding in software on hardware that can do it in the iGPU. This opts its ServiceDefinition into the `hardware` capability from #27 and re-renders. catalog/plex.yaml gains four lines of YAML and the comment that explains them: hardware: gpu: intel-quicksync which renders exactly two things into apps/wave-0/plex.yaml and nothing else — a `dri` hostPath volume (/dev/dri, type: DirectoryOrCreate) and a read-write /dev/dri volumeMount on the app container. Whole-tree render diff: 6 added lines, one file. The comment records what is not obvious from the four lines: the device exists on ALL THREE prod nodes with i915 bound (so no nodeSelector is rendered and none is wanted); why `supplementalGroups` would be INERT rather than a hardening (the image's cont-init hook adds the plex user to the device's group inside the container, and s6-setuidgid then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied); and why `DirectoryOrCreate` is load-bearing (apps-wave-0 is wait:true with apps-wave-1 depending on it, so a failed mount would stall the whole app tier for a feature that is not). THE FEATURE IS INERT UNTIL A HUMAN CLICKS A BOX, and that is the part this commit is really about. Mounting the device does not enable hardware transcoding; Plex's `HardwareAcceleratedCodecs` preference does, and it is ABSENT from Preferences.xml — the toggle has never been on. It cannot be automated: a bounded grep across the running image's entire init surface (/etc/cont-init.d, /etc/services.d, /etc/fix-attrs.d, /plex-common.sh, /usr/local/bin) found ZERO PLEX_PREFERENCE* support, and the six setPref keys the image does write are all unrelated. So runbooks/plex-quicksync.md is the procedure, registered in the configMapGenerator (a runbook in git with no generator entry is reviewed, merged, and still a 404) and indexed in runbooks/README.md. It covers the GitOps/manual boundary, the toggle and the real preference keys (including the widely-cited one that does not exist and the upstream misspelling nobody should "fix"), verification via `Transcode (hw)` on the Plex dashboard — stated plainly as the ONLY proof, because Plex renders no ServiceMonitor and an alert could never fire — the device-cgroup failure to expect first with its Intel-device-plugin fallback, and the floating-`:latest` caveat. Stacked on #27 (feat/sd-hardware-capability), which is open and unmerged; that branch's commit is in this range by design and this must merge after it. NOT VERIFIED, and the handoff says so at length: hardware transcoding has not been observed working. Whether containerd's device cgroup lets a non-privileged pod open() a bind-mounted char device is an open question that only deploying settles, and the Plex-side toggle was still unset at the time of writing. A green gate proves the manifest renders; it proves nothing about transcoding. Gate (pinned toolbox, worktree): rc 3 — UNDETERMINED, not green. Passed: decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped (the known local ceiling): spec-immutability (no sibling spec repo), gitea-settings (no SETTINGS_API_TOKEN), conformance (CLUSTER_AVAILABLE unset, so L2/L3 did not run). No fourth skip, nothing failed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6a6fea8151 |
docs(auth): root-cause jackett's session loss; add the oauth2-proxy-session runbook
Reported as "GET works, PUT/POST to add indexers fail, reloading never helps,
and I am never re-prompted". Diagnosis lands here; the MANIFEST FIX DOES NOT,
deliberately -- see the blocker below.
ROOT CAUSE, measured from two pods' logs interleaved to the second:
15:31:06 jackett [AuthSuccess]
15:31:29 radarr "cookie signature not valid, removing session"
15:31:32 jackett "No valid authentication in request" <- 3s later
Six oauth2-proxy instances (seven pods) emit the DEFAULT cookie name
`_oauth2_proxy` on `.home.keithjackson.dev` holding seven DIFFERENT
cookie-secrets (compared in-cluster by sha256; all distinct). `removing
session` is ClearSession, which expires the cookie on radarr's own zone-wide
--cookie-domain, so jackett's session is DELETED by a process in another
namespace, 26s after a clean login whose `expires` was 5 minutes out. 8 such
lines in radarr's log, 1 in jackett's, 0 elsewhere. It was never expiry and it
was never Jackett -- and it is why reloading does not help: the operator's
Radarr tab re-clears the cookie seconds after each new login.
Two contributing mechanisms, also measured:
* session lifetime == Keycloak access-token lifetime, 300.000s exactly, with
refresh_token:true on the same line and --cookie-refresh unset (default 0).
* XHRs answered 302 not 401: oauth2-proxy's AJAX guard needs an exact
`Accept: application/json`, which Jackett's jQuery does not send. On /api/
paths: 33 x 302 vs 2 x 401. A browser following a 302 re-issues a POST as a
GET and DROPS THE BODY -- 4 of 6 POSTs never reached Jackett at all.
WHY NO MANIFEST FIX HERE. apps/wave-0/jackett.yaml is a GENERATED GOLDEN. The
sidecar args come from one shared function, _oauth2_sidecar_container() in
catalog/render.py; catalog/jackett.yaml declares only auth.mode/auth.groups and
the auth schema is additionalProperties:false, so there is NO per-app override
hook. Hand-editing the rendered file is rejected by the gate exactly as
designed ("GOLDEN DRIFT ... render-golden: FAIL"). Changing the renderer
instead rewrites all five rendered apps -- measured:
CHANGED: apps/wave-0/{jackett,pihole,qbittorrent}.yaml
CHANGED: apps/wave-1/{radarr,sonarr}.yaml
and render-golden requires all five be regenerated in the same commit. Four of
them are owned by fix/oauth2-proxy-cookie-collision-fleet right now. The cookie
fix is ONE change, not five; it needs re-cutting as a single renderer commit
rather than two branches rewriting the same generated block.
Validated the intended argument list against the pinned image and the real
issuer before stopping: it printed back `name:_oauth2_proxy_jackett ...
domains:jackett.home.keithjackson.dev ... refresh:after 2m30s` and
`API route - Path: ^/api/`. Two corrections fell out of that: --api-route is
SINGULAR (--api-routes does not exist and the proxy refuses to start), and
--session-cookie-minimal is mutually exclusive with --cookie-refresh
("cookie_refresh > 0 requires oauth tokens in sessions"), so dropping
--pass-access-token does NOT shrink the cookie -- it is a least-privilege
change, not a size one.
Adds runbooks/oauth2-proxy-session.md: the four sidecar log lines that tell the
auth layers apart (including the upstream `-` that means the proxy answered a
request the app never saw), the cross-log collision diagnosis, and the
generated-golden warning with its blast radius. Registered in the
configMapGenerator AND runbooks/README.md so the URL resolves.
Jackett is NOT fixed by this commit and will keep losing its session to Radarr
until the renderer change ships.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
fc7ec01628 |
feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics
Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.
The reason nothing alerted turns out not to be a missing rule. Measured on prod:
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.
So alert on the condition that made the incident invisible, from families that ARE
scraped:
OAuth2ProxyGateUnobserved
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
unless
count by (namespace) (up{container="oauth2-proxy"})
kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.
Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
check from the app's own login from an XHR that silently got HTML, with the
2026-09-30 jackett trace as the worked example and the Loki queries to use while
there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.
Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
ea0c0632d2 |
fix(crafty): the Keycloak gate works -- Crafty's own login was the wall
Reported: "crafty does not sign in after signing into keycloak, and there is
no way to set the service up without credentials."
MEASURED, not inferred. The sidecar log settles which layer refused:
GET / -> 302 to Keycloak
[AuthSuccess] ... groups:[platform-admins] <- oauth2-proxy is FINE
GET /oauth2/callback -> 302
GET /login?next=/panel/dashboard -> 200 <- Crafty's OWN login page
POST /api/v2/auth/login/ -> 401 <- Crafty refused
So the Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume
the sidecar's identity headers, so it shows its own local login to an already
authenticated user. That second factor is intended. What was NOT intended is
that nothing documented where its password comes from, which is exactly why
setup was impossible: Crafty generates it ONCE on first boot (users table
empty -> DatabaseBuilder.default_settings()) into /crafty/app/config/
default-creds.txt, mode 0600, on the PVC's `config` subPath. Never in git;
a fresh PVC generates a new one. The README asserted the double login was
"expected, not a bug" and then left the reader with no way forward.
Verified in-cluster that the existing credential is still usable, so the
service can be set up without touching the DB: users.last_update equals
users.created for `admin` (09/05/2026, 08:10:04), i.e. the password has never
been changed, and admin's last_login is 09/14 from 127.0.0.1 -- the API login
that created the "homelab" server. default-creds.txt is therefore live.
ALSO FIXED -- a real, separate defect found while proving the above, which
produces the reported symptom intermittently and would have been blamed on
Keycloak forever: this sidecar and the SHARED forward-auth oauth2-proxy both
emitted oauth2-proxy's DEFAULT cookie names (_oauth2_proxy,
_oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding DIFFERENT
cookie-secrets (compared the two live Secrets: they differ, so neither can
decrypt the other's cookie). One cookie pair, two proxies: whichever
authenticated last clobbered the other's session; the loser logged "No valid
authentication in request", bounced to Keycloak, and -- SSO session still live
-- was redirected straight back, clobbering it again. Fixed on this side only
(renaming the shared proxy's cookie would sign out every other gated app):
a unique --cookie-name plus a host-scoped --cookie-domain, since a per-app
reverse proxy has no reason to broadcast its session zone-wide.
--whitelist-domain stays zone-wide: it governs redirect targets, not cookies.
Two traps measured rather than assumed, both now documented:
- The 3-strikes cooldown CANNOT fire behind this sidecar. Crafty records
attempts under get_remote_ip() (honours X-Forwarded-For -- the live
tracker key is the chain "192.168.2.132, 10.0.0.123") but checks the
cooldown under request.remote_ip, which is always 127.0.0.1 here. The
halves key differently, so get_recent_attempts() always reads empty.
Good (no lockout risk) and bad (that throttle is not a control we have).
- Crafty's `set_passwd` console command is unreachable in k8s: it calls
getpass.getpass() and needs a TTY; the container runs `main.py -d -i`.
Recovery has to go through the DB, and argon2 is installed ONLY in
/crafty/.venv/bin/python -- the system python3 lacks it.
Deliberately NOT done: an initContainer seeding default.json from a SOPS
Secret to make a rebuilt Crafty's password deterministic. Crafty does support
that hook (find_default_password reads app/config/default.json, min 8 chars),
but the image entrypoint pulls its default config ONLY when app/config is
EMPTY -- so writing a file there pre-boot suppresses the whole config
bootstrap. Buying reproducibility means co-owning image-internal bootstrap
logic that a 4.11 bump could silently change, to replace a one-command
retrieval. Recorded in DECISIONS-LOG as considered and rejected.
New runbook runbooks/crafty-admin-credential.md: tell the two auth layers
apart from the sidecar log, retrieve and rotate the bootstrap credential,
and reset a lost one via argon2 -- generating the password in-pod into a 0600
file rather than passing it in, because dev.sh does not forward stdin (a
`kubectl exec -i` through it hangs; verified). Registered in the configMapGenerator
so the URL actually resolves, not just the file exists.
Gate: toolbox ./ci/gate.sh -- lint/l0/render-golden/l1 green, spec-immutability
green. rc 3 overall: gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and L2/L3
NOT RUN (CLUSTER_AVAILABLE unset). No runtime browser round-trip performed --
the sign-in fix is UNVERIFIED end-to-end and needs the human test in the README.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
761e25cc3f |
docs(runbooks): two more uncited "4-minute" claims the first pass missed
A systematic grep for the TTL assertion found it still in runbooks/README.md and in runbooks/kustomization.yaml, outside the page being rewritten. Same correction: the countdown on plex.tv/claim is the authority, not a number in our tree. README.md also now records that the server is CLAIMED and points at claim-ship.sh with the measured ~2m45s loop, rather than describing a push-based sequence that no longer exists. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8e017fad91 |
docs(runbooks): the PR flow, not git push origin main -- that instruction is now blocked
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
4fd75ccdaa |
feat(plex): wire PLEX_CLAIM from a SOPS Secret so the server can be claimed
Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving PLEX_CLAIM unwired; the operator has decided to override it. This implements the wiring in the way that respects the half of that argument which is still true. WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers. So the committed value is a documented PLACEHOLDER and the live token is injected by the operator inside that window, with the reconcile FORCED rather than waited for. runbooks/plex-claim.md is that timed procedure. WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret would crashloop the pod". Half of that is wrong, and the difference is the whole design. Established from plexinc/pms-docker's own first-run script (branch `master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked byte-identical against the same file inside the running :latest image: * the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the PlexOnlineToken pref in /config being empty. Once the server is claimed the block is skipped entirely, so the env is a NO-OP on every later start whatever it holds ("If server is already logged in, this parameter is ignored" -- upstream README). * with an invalid/expired token on a still-unclaimed /config, plex.tv answers 403 with an empty body; curl's status is never checked, the token sed yields empty, the success branch at line 120 does not fire, there is no else branch and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up UNCLAIMED AND SERVING. It does not crashloop. * the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even if a cont-init script failed), so even a non-zero exit would not halt startup. An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it is CreateContainerConfigError), which is why the Secret is committed, encrypted and listed in the wave kustomization rather than hand-created at claim time. NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had to tolerate a missing Secret or could not start without a VALID token. Neither holds: the Secret always exists in the Flux graph, and an invalid token is non-fatal. schemas/servicedefinition.schema.json is untouched, and so are catalog/render.py's env emission and the renderer contract. Changes: * apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex, key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header carries NO dollar-brace expressions on purpose: kustomize preserves body comments into the stream envsubst reads (per conformance/tests/l0-static/flux-postbuild-substitution), so quoting the upstream shell conditions verbatim would have been a live substitution hazard on apps-wave-0, not prose. * catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the "deliberately NOT set" block rewritten around the TTL + one-shot design, and the stale crashloop claim in the /config storage comment corrected. * apps/wave-0/plex.yaml -- re-rendered with the canonical command (`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden under conformance/tests/l1-rendering covers plex, so this is the only rendered surface that moves. * catalog/render.py -- plan_apps_waves places `<svc>-claim-secret.sops.yaml` in the app's OWN wave, like the bootstrap secret, plus a selftest assertion. Without this, the next `--emit-waves` would have silently dropped the Secret from apps/wave-0/kustomization.yaml and put the pod into CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one new line was applied without re-emitting the rest. * runbooks/plex-claim.md + its kustomization.yaml generator entry + README index -- the operator procedure that fits in 4 minutes, including the single forced-reconcile command and the mandatory pod roll (a changed Secret does NOT restart anything: PLEX_CLAIM is read only at container start and the pod template is byte-identical across this change). Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02 (ServiceDefinition onboarding contract). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a3f4ddb971 |
merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
4687521d65 |
fix(monitoring): delete the Longhorn rule that measured allocation and called it capacity
LonghornVolumeCapacityHigh went pending within minutes of the Longhorn scrape
landing today. Re-measured on prod: it read 0.9731 on the Prometheus TSDB volume
(actual_size 52243681280 / capacity 53687091200) while the kubelet reported the
SAME filesystem 20.8% used. actual_size is head + pinned snapshots, and one
user_created=false system snapshot from prod-1's replica rebuild holds 40510484480
bytes (37.7 GiB) of it; Longhorn returns no freed blocks without discard/TRIM, and
retentionSize 45GiB on a 50Gi volume targets 90% by design. The ratio is pinned
near 1.0 and cannot fall back under 0.85 — a permanently-firing alert, the same
shape as CNPGNoRecoverabilityPoint.
DELETED, not re-thresholded. Raising the threshold would tune away a true
statement about allocation. Redundancy proven, not asserted: of 24 Longhorn
volumes on prod, 23 appear in kubelet_volume_stats_inodes > 0 and so fall inside
MonitoringVolumeFillingUp (namespace="monitoring") or PersistentVolumeFillingUp /
PersistentVolumeAlmostFull. The 24th is unifi/unifi-rootdisk, volumeMode Block —
no filesystem for anyone to measure, and the deleted rule was not covering it
either.
AND A SECOND INERTNESS the scrape did not fix. longhorn_volume_robustness on
longhorn 1.12.1 is one-hot over a `state` label with values 0/1, not the 0-3 enum
the rules assumed:
count(longhorn_volume_robustness) => 96 (24 volumes x 4 states)
count by (state) (longhorn_volume_robustness == 1) => {state="healthy"} 24
count(longhorn_volume_robustness == 2) => (empty)
count(longhorn_volume_robustness == 3) => (empty)
So a FAULTED volume would still have alerted nobody. Re-keyed to the label; the
"44 series / 72 unknown(0, detached)" note in platform/longhorn/helmrelease.yaml
was the same misreading and is corrected in place.
The allocation question moved to where allocation is a budget. LonghornNodeStorageHigh
fired at 0.85, but storage-minimal-available-percentage is 25 on this cluster, so a
disk leaves the scheduling pool at 75% used — the old threshold could only fire
after scheduling had already been refused. Replaced by:
LonghornDiskAllocationHigh (longhorn_disk_usage_bytes / longhorn_disk_capacity_bytes) > 0.70, 30m
LonghornDiskUnschedulable longhorn_disk_status{condition="schedulable"} == 0, 15m
Per DISK, not per node: longhorn_node_storage_usage_bytes sums a node's disks, so
one full disk could hide behind a half-empty sibling. Measured prod usage
0.2254 / 0.2306 / 0.2369 on prod-1/2/3.
PROVEN NEITHER PERMANENTLY FIRING NOR INERT. platform/monitoring/tests/longhorn.test.yaml
adds 6 cases pinning every rule quiet on today's measured prod state and firing on
a constructed bad state, with every `for:` asserted, plus regression assertions that
the old `== 2`/`== 3` forms match nothing and that per-disk 0.72 is visible where the
node-summed 0.475 is not. Mutation-checked 11 ways; each turns the suite RED. Live on
prod: all four new expressions return empty, while the sum of their selector
cardinalities is 54. The L0 UNTESTED_ALERTS register shrinks by seven (7/56 -> 14/58
alert rules unit-tested), which also brings MonitoringVolumeFillingUp,
PersistentVolumeFillingUp and PersistentVolumeAlmostFull under test since the
redundancy claim rests on them.
REPORTED, NOT CHANGED (DECISIONS-LOG, owner decisions): retentionSize 45GiB on a
50Gi volume is too tight — recommend 36GiB, with reasoning; and discard/TRIM on the
longhorn StorageClass is feasible but not advisable now — remove-snapshots-during-
filesystem-trim reads false on prod, so trim would not have reclaimed the 37.7 GiB
anyway, and continuous discard on nodes where the etcd WAL shares the spindle points
at this cluster's known failure mode.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
bce8778096 |
docs(runbooks): forward-auth IS enforced — correct the posture note
The serving route's comment claimed that because no Gateway ext-authz attachment exists, every forward-auth route is 'in practice reachable unauthenticated today'. That is wrong, and wrong in a dangerous direction for a document an operator reads during an incident. Enforcement here is PER-POD, not gateway-level: each of the five services declaring auth: forward-auth (pihole, jackett, qbittorrent, radarr, sonarr) publishes both its app port and 4180 on its Service, and its HTTPRoute targets :4180 — the oauth2-proxy sidecar. Verified on prod. The app port remains reachable in-cluster on purpose, which is how sonarr/radarr drive qBittorrent's API, and Cilium default-deny governs that path. The public-by-design decision for this route is unchanged and still correct; what changes is that it is genuinely the exception rather than one open route among many. Headlamp's exclusion is likewise deliberate (native OIDC with per-user id_token forwarding and no ServiceAccount cluster rights). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b7341b329b |
fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
44983095e1 |
feat(runbooks): make the 42 alert runbook_url annotations resolve
Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 <[email protected]> |