0dd57fbaf0028e183c8d2d050b95e40a35ddfec6
18
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jakman85
|
ee196c7bcd |
Merge pull request 'feat(upgrades): automate node reboots (kured) and wire Renovate, which had never run' (#73) from feat/automated-upgrades into main
Reviewed-on: #73 Reviewed-by: Keith Jackson <[email protected]> |
||
 jakman85andClaude Opus 5
|
a2440a3125 |
feat(upgrades): automated OS reboots (kured) + security-only unattended-upgrades, and give Renovate a runner
Two halves of one repeatable upgrade process. Neither works alone. (a) OS upgrades and reboots platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml: that tier gates platform-config and every apps wave, so a wedged reboot daemon there could stall every application. A component added to make maintenance safer must not be able to cause an outage. infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other open branches, and this is a different lifecycle), enabled for prod in group_vars. The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's rejection of automating away the gate between nodes: u-u is security-only with the kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then does only the mechanical half, including the ordering rule node_baseline has to shout. Quorum: concurrent reboots are structurally impossible, not merely unlikely. concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency), and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node holds it forever and nothing cascades. Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait, serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m (unconditional) plus kured's Prometheus alert gate (semantic). alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and under the chart default kured would have been blocked on day one by the five alerts measured firing with no end in sight, and never rebooted anything. Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it. Asserted against apt-config dump, not against the file just written. 6.12 kernels are still installed and bootable on all three nodes, and nothing was held. Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING could create the sentinel. kured alone would have idled forever at 1/1 Running. platform/kured/alerts.yaml — three rules over kured's only metric, with a companion promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md. Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this cluster's 7d Prometheus retention. (b) Image/dependency automation Renovate was aspirational: renovate.json present and complete, with no workflow, no in-cluster workload, and its docker+kubernetes managers disabled — so every plain image: tag was maintained by nothing while seven files claimed otherwise. .gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a skipped Gitea job reports success and that is how this stayed inert. Renovate now owns both halves; Flux image-automation is formally closed: it pushes commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here anyway — main is protected user_can_push:false and the flux-system GitRepository has no secretRef. Its controllers are installed and have idled 43 days with zero CRs. conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing; tightens the check for this component (requires service.create+metrics.create, both chart-default false, plus a companion NetworkPolicy requirement). Required because assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is shrink-only. Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author approving review; /approve-override deliberately NOT posted. NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role never executed, and whether CNPG switches its primary on a drain-induced eviction (the most likely way the first real reboot fails). Full list in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
2f2e98aa19 |
feat(shutdown): configure graceful node shutdown, and never scale a Delete-policy StatefulSet
All three nodes had to be powered off to move them onto a UPS, and there was NO procedure: infra/cluster.sh stop is virsh-based and only ever applied to the decommissioned dev cluster. The sequence was improvised live. Two defects came out of doing it. 1. GRACEFUL NODE SHUTDOWN WAS UNCONFIGURED ON EVERY PROD NODE. No shutdownGracePeriod anywhere in infra/ansible/, so `shutdown -h` stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood -- 9 CNPG instances mid-checkpoint, the UniFi VM, 25 attached Longhorn volumes. roles/k3s_server now sets kubelet-arg shutdown-grace-period=90s and shutdown-grace-period-critical-pods=20s. The half that silently defeats the other half: this is enforced by LOGIND, not the kubelet. The kubelet holds an inhibitor DELAY lock and systemd waits at most InhibitDelayMaxSec -- DEFAULT 5 SECONDS -- then proceeds regardless, truncating the drain with no error. The role ships a logind.conf.d drop-in (120) and ASSERTS cap >= grace period, because the failure is invisible. 2. SCALING A StatefulSet TO 0 DELETED ITS PVC AND VOLUME. monitoring/loki carries persistentVolumeClaimRetentionPolicy whenScaled: Delete, so the quiesce destroyed storage-loki-0 outright. Loki stores type: filesystem, so that PVC was the only copy; 72h retention bounded the loss, the mechanism does not. Symptom was the volume count going 25 -> 24, first misread as a miscount. Repaired: scale back to the declared 1 recreated the PVC and loki-0 is 2/2 Running. New infra/maintenance/quiesce.sh (quiesce|resume|status) encodes the sequence that worked and REFUSES to scale any whenScaled: Delete StatefulSet -- it stays running, its volume hard-detaches, and it rebuilds on restart. A rebuild beats a deletion. Discovery is computed from the live cluster, so a newly onboarded service needs no edit here. runbooks/cluster-maintenance-shutdown.md is the operator page: order in both directions, TrueNAS-after-nodes (hard NFS mount wedges kubelet), the un-hibernate-Postgres step Flux will NOT do for you, and a "looks broken but is fine" section separating the 4/19 Flux cascade, the degraded-and-rebuilding volume and the self-recovering keycloak-operator from the one thing that needs a human (a stuck oauth2-proxy replica). THE KUBELET CHANGE HAS NEVER BEEN APPLIED OR TESTED -- it needs a rolling k3s restart, which is an operator action. Review config.yaml's kubelet-arg before applying: a bad config.yaml is how k3s fails to come up. See the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0569d47d49 |
Merge branch 'main' into feat/pm-agent-issue-to-pr
One conflict, in clusters/base/kustomization.yaml: PR #55 (vpn-ingress) and this branch both appended a resource line to the same tail. Resolved by keeping BOTH, with main's vpn-ingress first — it is a platform-config-tier leaf, while pm-agent dependsOn apps-wave-0, which is later. Nothing else conflicted. runbooks/kustomization.yaml auto-merged and carries both the vpn-remote-access and pm-agent entries; the handoff entries are separate files and had nothing to merge, which is the whole point of decisions/ being a directory. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f92707f38c |
feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers)
Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/** , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c5b5b8672a |
feat(pm-agent): an in-cluster AI Program Manager that turns Gitea issues into pull requests
A service that reads this repository's Gitea issues, decides what to do about each thread as the Program Manager persona, and runs one worker agent per assignment that authors a change, runs the merge gate and opens a pull request. It reacts to issue creation and to comments; the assignment and all correspondence travel through the issue thread and the PR. It authenticates with an Anthropic subscription token. It never pushes to main and never merges. Three choices carry the design, with alternatives recorded in docs/plans/: - it POLLS rather than receiving webhooks (Gitea's webhook SSRF guard, lost deliveries, and no inbound reach needed at all); - it keeps NO state: the trigger is "the newest thing in this thread was not written by me", so the issue thread is the state — no PVC, no backup obligation; - the model holds NO tools: its reply is one JSON object this code validates, so nothing in the path executes text. The boundary is platform/kyverno/admission/pm-agent-job-shape.yaml (Enforce, failurePolicy: Fail), not the RBAC: Kubernetes places no `use` check on serviceAccountName, so "may create Jobs" is "may run arbitrary code as the namespace's most privileged identity". Six rules admit only the committed shape, and the worker identity is bound to nothing with no token mounted, so there is nothing to escalate to. 28/28 offline Kyverno assertions. Nothing has run: the image has never been built or pushed, and both the image digest and the two credentials are deliberate placeholders. See the handoff entry's "What I did NOT verify". Refs CLAUDE.md §1/§3/§6, R-SEC-02, R-GITOPS-02. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
ba79a1b59b |
Merge remote-tracking branch 'origin/main' into feat/oauth2-proxy-auth-failure-visible
# Conflicts: # runbooks/kustomization.yaml |
||
|
jakman85andClaude Opus 5
|
0fee584e3a |
feat(plex): opt into hardware.gpu: intel-quicksync, and write the runbook for the step GitOps cannot reach
Plex has been transcoding in software on hardware that can do it in the iGPU. This opts its ServiceDefinition into the `hardware` capability from #27 and re-renders. catalog/plex.yaml gains four lines of YAML and the comment that explains them: hardware: gpu: intel-quicksync which renders exactly two things into apps/wave-0/plex.yaml and nothing else — a `dri` hostPath volume (/dev/dri, type: DirectoryOrCreate) and a read-write /dev/dri volumeMount on the app container. Whole-tree render diff: 6 added lines, one file. The comment records what is not obvious from the four lines: the device exists on ALL THREE prod nodes with i915 bound (so no nodeSelector is rendered and none is wanted); why `supplementalGroups` would be INERT rather than a hardening (the image's cont-init hook adds the plex user to the device's group inside the container, and s6-setuidgid then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied); and why `DirectoryOrCreate` is load-bearing (apps-wave-0 is wait:true with apps-wave-1 depending on it, so a failed mount would stall the whole app tier for a feature that is not). THE FEATURE IS INERT UNTIL A HUMAN CLICKS A BOX, and that is the part this commit is really about. Mounting the device does not enable hardware transcoding; Plex's `HardwareAcceleratedCodecs` preference does, and it is ABSENT from Preferences.xml — the toggle has never been on. It cannot be automated: a bounded grep across the running image's entire init surface (/etc/cont-init.d, /etc/services.d, /etc/fix-attrs.d, /plex-common.sh, /usr/local/bin) found ZERO PLEX_PREFERENCE* support, and the six setPref keys the image does write are all unrelated. So runbooks/plex-quicksync.md is the procedure, registered in the configMapGenerator (a runbook in git with no generator entry is reviewed, merged, and still a 404) and indexed in runbooks/README.md. It covers the GitOps/manual boundary, the toggle and the real preference keys (including the widely-cited one that does not exist and the upstream misspelling nobody should "fix"), verification via `Transcode (hw)` on the Plex dashboard — stated plainly as the ONLY proof, because Plex renders no ServiceMonitor and an alert could never fire — the device-cgroup failure to expect first with its Intel-device-plugin fallback, and the floating-`:latest` caveat. Stacked on #27 (feat/sd-hardware-capability), which is open and unmerged; that branch's commit is in this range by design and this must merge after it. NOT VERIFIED, and the handoff says so at length: hardware transcoding has not been observed working. Whether containerd's device cgroup lets a non-privileged pod open() a bind-mounted char device is an open question that only deploying settles, and the Plex-side toggle was still unset at the time of writing. A green gate proves the manifest renders; it proves nothing about transcoding. Gate (pinned toolbox, worktree): rc 3 — UNDETERMINED, not green. Passed: decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped (the known local ceiling): spec-immutability (no sibling spec repo), gitea-settings (no SETTINGS_API_TOKEN), conformance (CLUSTER_AVAILABLE unset, so L2/L3 did not run). No fourth skip, nothing failed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6a6fea8151 |
docs(auth): root-cause jackett's session loss; add the oauth2-proxy-session runbook
Reported as "GET works, PUT/POST to add indexers fail, reloading never helps,
and I am never re-prompted". Diagnosis lands here; the MANIFEST FIX DOES NOT,
deliberately -- see the blocker below.
ROOT CAUSE, measured from two pods' logs interleaved to the second:
15:31:06 jackett [AuthSuccess]
15:31:29 radarr "cookie signature not valid, removing session"
15:31:32 jackett "No valid authentication in request" <- 3s later
Six oauth2-proxy instances (seven pods) emit the DEFAULT cookie name
`_oauth2_proxy` on `.home.keithjackson.dev` holding seven DIFFERENT
cookie-secrets (compared in-cluster by sha256; all distinct). `removing
session` is ClearSession, which expires the cookie on radarr's own zone-wide
--cookie-domain, so jackett's session is DELETED by a process in another
namespace, 26s after a clean login whose `expires` was 5 minutes out. 8 such
lines in radarr's log, 1 in jackett's, 0 elsewhere. It was never expiry and it
was never Jackett -- and it is why reloading does not help: the operator's
Radarr tab re-clears the cookie seconds after each new login.
Two contributing mechanisms, also measured:
* session lifetime == Keycloak access-token lifetime, 300.000s exactly, with
refresh_token:true on the same line and --cookie-refresh unset (default 0).
* XHRs answered 302 not 401: oauth2-proxy's AJAX guard needs an exact
`Accept: application/json`, which Jackett's jQuery does not send. On /api/
paths: 33 x 302 vs 2 x 401. A browser following a 302 re-issues a POST as a
GET and DROPS THE BODY -- 4 of 6 POSTs never reached Jackett at all.
WHY NO MANIFEST FIX HERE. apps/wave-0/jackett.yaml is a GENERATED GOLDEN. The
sidecar args come from one shared function, _oauth2_sidecar_container() in
catalog/render.py; catalog/jackett.yaml declares only auth.mode/auth.groups and
the auth schema is additionalProperties:false, so there is NO per-app override
hook. Hand-editing the rendered file is rejected by the gate exactly as
designed ("GOLDEN DRIFT ... render-golden: FAIL"). Changing the renderer
instead rewrites all five rendered apps -- measured:
CHANGED: apps/wave-0/{jackett,pihole,qbittorrent}.yaml
CHANGED: apps/wave-1/{radarr,sonarr}.yaml
and render-golden requires all five be regenerated in the same commit. Four of
them are owned by fix/oauth2-proxy-cookie-collision-fleet right now. The cookie
fix is ONE change, not five; it needs re-cutting as a single renderer commit
rather than two branches rewriting the same generated block.
Validated the intended argument list against the pinned image and the real
issuer before stopping: it printed back `name:_oauth2_proxy_jackett ...
domains:jackett.home.keithjackson.dev ... refresh:after 2m30s` and
`API route - Path: ^/api/`. Two corrections fell out of that: --api-route is
SINGULAR (--api-routes does not exist and the proxy refuses to start), and
--session-cookie-minimal is mutually exclusive with --cookie-refresh
("cookie_refresh > 0 requires oauth tokens in sessions"), so dropping
--pass-access-token does NOT shrink the cookie -- it is a least-privilege
change, not a size one.
Adds runbooks/oauth2-proxy-session.md: the four sidecar log lines that tell the
auth layers apart (including the upstream `-` that means the proxy answered a
request the app never saw), the cross-log collision diagnosis, and the
generated-golden warning with its blast radius. Registered in the
configMapGenerator AND runbooks/README.md so the URL resolves.
Jackett is NOT fixed by this commit and will keep losing its session to Radarr
until the renderer change ships.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
fc7ec01628 |
feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics
Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.
The reason nothing alerted turns out not to be a missing rule. Measured on prod:
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.
So alert on the condition that made the incident invisible, from families that ARE
scraped:
OAuth2ProxyGateUnobserved
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
unless
count by (namespace) (up{container="oauth2-proxy"})
kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.
Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
check from the app's own login from an XHR that silently got HTML, with the
2026-09-30 jackett trace as the worked example and the Loki queries to use while
there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.
Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
55b232e2ce |
feat(obs): notify on Flux deploy start and completion
The operator got no mail when Flux began applying a git change and none when it finished. Measured cause: notification-controller v1.4.0 is running and its CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the cluster — nothing in git ever created any. The obvious fix does not work. Neither Flux event can be filtered into a deploy signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16 Kustomizations, message byte-identical whether the apply changed anything), and Progressing is worse (five Kustomizations re-apply the same object on every tick — event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one). The applied revision is the only signal that moves when and only when git moves, and it lives in object status alone. So export it via the kube-state-metrics custom-resource-state path that already serves the Velero Schedule metrics — kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and ask the question in PromQL: FluxRolloutInProgress (severity: info, no `for:`) fires the moment a non-suspended Kustomization's applied revision is not the revision the GitRepository fetched from main, and CLEARS when all have caught up. Fired mail = a deploy started; RESOLVED mail = it completed. The completion half is send_resolved, not a second rule. FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the escalation for a rollout that started and never converged. Two guards, each with a test whose only job is to fail if the clause is removed: `and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes `unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert forever, the permanently-firing mode this repo shipped once already — and the suspended-Kustomization exclusion, without which a suspended tier holds the rollout open and the "applied" mail never arrives. A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because the chart sets no checksum over the custom-resource-state ConfigMap, so a config-only edit would leave KSM on the old collector set with the new alerts silently inert. Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/ spec-immutability green in the pinned toolbox. NOT verified: no mail has been sent — the metrics do not exist until this merges. Full gap list, and the deliberately-cut notification-controller half, in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
ea0c0632d2 |
fix(crafty): the Keycloak gate works -- Crafty's own login was the wall
Reported: "crafty does not sign in after signing into keycloak, and there is
no way to set the service up without credentials."
MEASURED, not inferred. The sidecar log settles which layer refused:
GET / -> 302 to Keycloak
[AuthSuccess] ... groups:[platform-admins] <- oauth2-proxy is FINE
GET /oauth2/callback -> 302
GET /login?next=/panel/dashboard -> 200 <- Crafty's OWN login page
POST /api/v2/auth/login/ -> 401 <- Crafty refused
So the Keycloak gate was never broken. Crafty 4 has no OIDC and cannot consume
the sidecar's identity headers, so it shows its own local login to an already
authenticated user. That second factor is intended. What was NOT intended is
that nothing documented where its password comes from, which is exactly why
setup was impossible: Crafty generates it ONCE on first boot (users table
empty -> DatabaseBuilder.default_settings()) into /crafty/app/config/
default-creds.txt, mode 0600, on the PVC's `config` subPath. Never in git;
a fresh PVC generates a new one. The README asserted the double login was
"expected, not a bug" and then left the reader with no way forward.
Verified in-cluster that the existing credential is still usable, so the
service can be set up without touching the DB: users.last_update equals
users.created for `admin` (09/05/2026, 08:10:04), i.e. the password has never
been changed, and admin's last_login is 09/14 from 127.0.0.1 -- the API login
that created the "homelab" server. default-creds.txt is therefore live.
ALSO FIXED -- a real, separate defect found while proving the above, which
produces the reported symptom intermittently and would have been blamed on
Keycloak forever: this sidecar and the SHARED forward-auth oauth2-proxy both
emitted oauth2-proxy's DEFAULT cookie names (_oauth2_proxy,
_oauth2_proxy_csrf) scoped to .home.keithjackson.dev, while holding DIFFERENT
cookie-secrets (compared the two live Secrets: they differ, so neither can
decrypt the other's cookie). One cookie pair, two proxies: whichever
authenticated last clobbered the other's session; the loser logged "No valid
authentication in request", bounced to Keycloak, and -- SSO session still live
-- was redirected straight back, clobbering it again. Fixed on this side only
(renaming the shared proxy's cookie would sign out every other gated app):
a unique --cookie-name plus a host-scoped --cookie-domain, since a per-app
reverse proxy has no reason to broadcast its session zone-wide.
--whitelist-domain stays zone-wide: it governs redirect targets, not cookies.
Two traps measured rather than assumed, both now documented:
- The 3-strikes cooldown CANNOT fire behind this sidecar. Crafty records
attempts under get_remote_ip() (honours X-Forwarded-For -- the live
tracker key is the chain "192.168.2.132, 10.0.0.123") but checks the
cooldown under request.remote_ip, which is always 127.0.0.1 here. The
halves key differently, so get_recent_attempts() always reads empty.
Good (no lockout risk) and bad (that throttle is not a control we have).
- Crafty's `set_passwd` console command is unreachable in k8s: it calls
getpass.getpass() and needs a TTY; the container runs `main.py -d -i`.
Recovery has to go through the DB, and argon2 is installed ONLY in
/crafty/.venv/bin/python -- the system python3 lacks it.
Deliberately NOT done: an initContainer seeding default.json from a SOPS
Secret to make a rebuilt Crafty's password deterministic. Crafty does support
that hook (find_default_password reads app/config/default.json, min 8 chars),
but the image entrypoint pulls its default config ONLY when app/config is
EMPTY -- so writing a file there pre-boot suppresses the whole config
bootstrap. Buying reproducibility means co-owning image-internal bootstrap
logic that a 4.11 bump could silently change, to replace a one-command
retrieval. Recorded in DECISIONS-LOG as considered and rejected.
New runbook runbooks/crafty-admin-credential.md: tell the two auth layers
apart from the sidecar log, retrieve and rotate the bootstrap credential,
and reset a lost one via argon2 -- generating the password in-pod into a 0600
file rather than passing it in, because dev.sh does not forward stdin (a
`kubectl exec -i` through it hangs; verified). Registered in the configMapGenerator
so the URL actually resolves, not just the file exists.
Gate: toolbox ./ci/gate.sh -- lint/l0/render-golden/l1 green, spec-immutability
green. rc 3 overall: gitea-settings SKIPPED (no SETTINGS_API_TOKEN) and L2/L3
NOT RUN (CLUSTER_AVAILABLE unset). No runtime browser round-trip performed --
the sign-in fix is UNVERIFIED end-to-end and needs the human test in the README.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
761e25cc3f |
docs(runbooks): two more uncited "4-minute" claims the first pass missed
A systematic grep for the TTL assertion found it still in runbooks/README.md and in runbooks/kustomization.yaml, outside the page being rewritten. Same correction: the countdown on plex.tv/claim is the authority, not a number in our tree. README.md also now records that the server is CLAIMED and points at claim-ship.sh with the measured ~2m45s loop, rather than describing a push-based sequence that no longer exists. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
4fd75ccdaa |
feat(plex): wire PLEX_CLAIM from a SOPS Secret so the server can be claimed
Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving PLEX_CLAIM unwired; the operator has decided to override it. This implements the wiring in the way that respects the half of that argument which is still true. WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers. So the committed value is a documented PLACEHOLDER and the live token is injected by the operator inside that window, with the reconcile FORCED rather than waited for. runbooks/plex-claim.md is that timed procedure. WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret would crashloop the pod". Half of that is wrong, and the difference is the whole design. Established from plexinc/pms-docker's own first-run script (branch `master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked byte-identical against the same file inside the running :latest image: * the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the PlexOnlineToken pref in /config being empty. Once the server is claimed the block is skipped entirely, so the env is a NO-OP on every later start whatever it holds ("If server is already logged in, this parameter is ignored" -- upstream README). * with an invalid/expired token on a still-unclaimed /config, plex.tv answers 403 with an empty body; curl's status is never checked, the token sed yields empty, the success branch at line 120 does not fire, there is no else branch and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up UNCLAIMED AND SERVING. It does not crashloop. * the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even if a cont-init script failed), so even a non-zero exit would not halt startup. An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it is CreateContainerConfigError), which is why the Secret is committed, encrypted and listed in the wave kustomization rather than hand-created at claim time. NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had to tolerate a missing Secret or could not start without a VALID token. Neither holds: the Secret always exists in the Flux graph, and an invalid token is non-fatal. schemas/servicedefinition.schema.json is untouched, and so are catalog/render.py's env emission and the renderer contract. Changes: * apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex, key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header carries NO dollar-brace expressions on purpose: kustomize preserves body comments into the stream envsubst reads (per conformance/tests/l0-static/flux-postbuild-substitution), so quoting the upstream shell conditions verbatim would have been a live substitution hazard on apps-wave-0, not prose. * catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the "deliberately NOT set" block rewritten around the TTL + one-shot design, and the stale crashloop claim in the /config storage comment corrected. * apps/wave-0/plex.yaml -- re-rendered with the canonical command (`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden under conformance/tests/l1-rendering covers plex, so this is the only rendered surface that moves. * catalog/render.py -- plan_apps_waves places `<svc>-claim-secret.sops.yaml` in the app's OWN wave, like the bootstrap secret, plus a selftest assertion. Without this, the next `--emit-waves` would have silently dropped the Secret from apps/wave-0/kustomization.yaml and put the pod into CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one new line was applied without re-emitting the rest. * runbooks/plex-claim.md + its kustomization.yaml generator entry + README index -- the operator procedure that fits in 4 minutes, including the single forced-reconcile command and the mandatory pod roll (a changed Secret does NOT restart anything: PLEX_CLAIM is read only at container start and the pod template is byte-identical across this change). Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02 (ServiceDefinition onboarding contract). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a3f4ddb971 |
merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
bc106caac1 |
docs(runbooks): node-memory-swap — replacing k3s-prod-1's mismatched DIMM pair
The operator has a matched 32 GB pair to install. k3s-prod-1 is the node that needs it: DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (16 GB, DDR4-3200, 1Rx8) and DIMM3 a 16ATF2G64HZ-2G6E1 (16 GB, DDR4-2667, 2Rx8), so capacity is correct at 32 GB but the pair runs at the slower 2667. Nothing is faulty. k3s-prod-3 is a matched Crucial pair. The procedure exists because this node is the awkward one to take down: it held 24 Longhorn replicas, it is one of three etcd members (so the drain leaves quorum at 2 of 3 with no further margin), and its rebuilds are the load that starved etcd once already. So the runbook front-loads the checks that decide whether to start at all, and treats a BLOCKED drain as the safety property working rather than an obstacle to force past. Two things it deliberately does not assert: - node-drain-policy is not pinned in platform/longhorn/helmrelease.yaml, so the runbook has the operator READ the live Setting and set it before draining if it is not block-if-contains-last-replica, instead of assuming the default. - Configured Memory Speed, not Speed, is named as the field that proves the swap worked — a matched pair being clocked down still reports the rated Speed. It also flags the capacity decision as a decision: 2x16 GB keeps the three nodes symmetric as PROD-PROVISIONING.md §2 assumes, while 2x32 GB makes node 1 the asymmetric one on memory as well as disk. Recorded caveats: the node-health artifacts de-duplicate DIMM part numbers, so they can prove "two different modules" but cannot distinguish one module from two identical ones — verification therefore collects dmidecode live. And node 2's inventory is stale and self-contradictory (documented as the only single-DIMM node at 16 GB, yet reporting ~31.1 GiB), so it needs re-inventory before anyone claims three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was an EFI memattr kernel bug on node 2. Wired into the configMapGenerator so it is served at /runbooks/node-memory-swap rather than existing only in git. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b7341b329b |
fix(runbooks): declare whole-repo scope, drop the bespoke l0 stage, +cnpg-primary-spread
Three corrections after rebasing onto main. 1. THE GUARD CAUGHT A REAL ONE ON ITS FIRST CONTACT WITH main. main had gained `CNPGPrimariesCoLocated` (platform/cnpg/alerts.yaml) pointing at /runbooks/cnpg-primary-spread, with no runbook. The check failed the rebase exactly as designed — 46 annotations, 35 names, 34 present — naming both the missing file AND the missing configMapGenerator entry. runbooks/cnpg-primary-spread.md now exists: what co-located primaries actually cost here (blast radius, not an outage, because instances: 3 + required anti-affinity), why the drift is uncorrectable-by-declaration (CNPG has no primary-placement field), the readiness gate that keeps the rule quiet during a real node outage, and the ~9s-per-database switchover rebalance, one cluster at a time. The 2026-09-07 outage in which all three primaries sat on one node is the reason the alert exists at all. 2. WHOLE-REPO SCOPE, NOT A HAND-WRITTEN STAGE. main's refactor replaced ci/l0.sh's per-check "UNCONDITIONAL" stages with a declaration: `conformance.home.k3s/scope: "whole-repo"`, unioned into every gate run by Stage 3b's `--include-scope`. My +38-line Stage 3f is removed (ci/l0.sh is now byte-identical to main) and the annotation is declared instead. Assertion (7) of the check was rewritten to match: it no longer asserts its own path appears in ci/l0.sh — it asserts the test DECLARES the scope and that ci/l0.sh still passes --include-scope, since neither half alone makes the check load-bearing. That matters here because ci/touched-requirements.sh maps platform/monitoring/* to "R-OBS-01 R-OBS-02", not R-OBS-04, and maps platform/kubevirt/* and runbooks/* to nothing at all. 3. A FACTUAL CORRECTION IN ALL 35 RUNBOOKS. Every page claimed `dev.sh` defaults its current-context to k3s-dev. main's `9b4744d fix(devcontainer): dev.sh defaults to prod` made that false, and a runbook that misstates which cluster a command talks to is worse than one that says nothing. The pages now say what is durably true: the default has already changed once, so naming `--context k3s-prod` is what makes a command mean the same thing next month, and the per-invocation banner is there to be read. The check's own failure message was corrected the same way. Re-verified after all three: check green at 374 assertions (46 annotations, 35/35 runbooks present, 35/35 projected), re-applied to prod, and re-probed in-cluster through the Gateway — node-down, keycloak-down, keycloak-client-secrets, cnpg-archiving, longhorn-faulted, vpn-down, node-io and cnpg-primary-spread all HTTP 200 with text/plain; charset=utf-8 and 8.9-12.1 KB of real body; /runbooks/ index 200; an unknown name 404s with a pointer rather than a blank page; and homepage still answers the apex `/`, confirming Gateway API longest-path precedence. Full merge gate: PASS (spec-immutability, lint, l0, render-golden, l1 all green; prom-rules SKIPPED for want of promtool in the image; L2/L3 SKIPPED, no cluster). Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
44983095e1 |
feat(runbooks): make the 42 alert runbook_url annotations resolve
Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 <[email protected]> |