jakman85andClaude Opus 5
6a6fea8151
Reported as "GET works, PUT/POST to add indexers fail, reloading never helps,
and I am never re-prompted". Diagnosis lands here; the MANIFEST FIX DOES NOT,
deliberately -- see the blocker below.
ROOT CAUSE, measured from two pods' logs interleaved to the second:
15:31:06 jackett [AuthSuccess]
15:31:29 radarr "cookie signature not valid, removing session"
15:31:32 jackett "No valid authentication in request" <- 3s later
Six oauth2-proxy instances (seven pods) emit the DEFAULT cookie name
`_oauth2_proxy` on `.home.keithjackson.dev` holding seven DIFFERENT
cookie-secrets (compared in-cluster by sha256; all distinct). `removing
session` is ClearSession, which expires the cookie on radarr's own zone-wide
--cookie-domain, so jackett's session is DELETED by a process in another
namespace, 26s after a clean login whose `expires` was 5 minutes out. 8 such
lines in radarr's log, 1 in jackett's, 0 elsewhere. It was never expiry and it
was never Jackett -- and it is why reloading does not help: the operator's
Radarr tab re-clears the cookie seconds after each new login.
Two contributing mechanisms, also measured:
* session lifetime == Keycloak access-token lifetime, 300.000s exactly, with
refresh_token:true on the same line and --cookie-refresh unset (default 0).
* XHRs answered 302 not 401: oauth2-proxy's AJAX guard needs an exact
`Accept: application/json`, which Jackett's jQuery does not send. On /api/
paths: 33 x 302 vs 2 x 401. A browser following a 302 re-issues a POST as a
GET and DROPS THE BODY -- 4 of 6 POSTs never reached Jackett at all.
WHY NO MANIFEST FIX HERE. apps/wave-0/jackett.yaml is a GENERATED GOLDEN. The
sidecar args come from one shared function, _oauth2_sidecar_container() in
catalog/render.py; catalog/jackett.yaml declares only auth.mode/auth.groups and
the auth schema is additionalProperties:false, so there is NO per-app override
hook. Hand-editing the rendered file is rejected by the gate exactly as
designed ("GOLDEN DRIFT ... render-golden: FAIL"). Changing the renderer
instead rewrites all five rendered apps -- measured:
CHANGED: apps/wave-0/{jackett,pihole,qbittorrent}.yaml
CHANGED: apps/wave-1/{radarr,sonarr}.yaml
and render-golden requires all five be regenerated in the same commit. Four of
them are owned by fix/oauth2-proxy-cookie-collision-fleet right now. The cookie
fix is ONE change, not five; it needs re-cutting as a single renderer commit
rather than two branches rewriting the same generated block.
Validated the intended argument list against the pinned image and the real
issuer before stopping: it printed back `name:_oauth2_proxy_jackett ...
domains:jackett.home.keithjackson.dev ... refresh:after 2m30s` and
`API route - Path: ^/api/`. Two corrections fell out of that: --api-route is
SINGULAR (--api-routes does not exist and the proxy refuses to start), and
--session-cookie-minimal is mutually exclusive with --cookie-refresh
("cookie_refresh > 0 requires oauth tokens in sessions"), so dropping
--pass-access-token does NOT shrink the cookie -- it is a least-privilege
change, not a size one.
Adds runbooks/oauth2-proxy-session.md: the four sidecar log lines that tell the
auth layers apart (including the upstream `-` that means the proxy answered a
request the app never saw), the cross-log collision diagnosis, and the
generated-golden warning with its blast radius. Registered in the
configMapGenerator AND runbooks/README.md so the URL resolves.
Jackett is NOT fixed by this commit and will keep losing its session to Radarr
until the renderer change ships.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
103 lines
5.7 KiB
YAML
103 lines
5.7 KiB
YAML
---
|
|
# Runbooks tier — reconciled by its OWN non-gating post-platform Flux Kustomization
|
|
# (clusters/base/runbooks.yaml, dependsOn platform-config, wait:false). Same placement and rationale as
|
|
# platform/homepage: a leaf web server must never sit in the gating `platform` tier, where a UI's
|
|
# unhealth once stalled the whole chain (DECISIONS-LOG: "Headlamp gating stall").
|
|
#
|
|
# WHY THE MANIFESTS LIVE HERE, ALONGSIDE THE DOCS, RATHER THAN IN platform/runbooks/.
|
|
# The markdown is the source of truth and the task is to serve it verbatim, which means the ConfigMap must
|
|
# be GENERATED from these files — there must not be a second hand-maintained copy of 34 documents that can
|
|
# drift from the originals. `configMapGenerator` does that, and its `files:` paths are resolved relative to
|
|
# THIS directory. A `platform/runbooks/kustomization.yaml` would therefore need `../../runbooks/<name>.md`,
|
|
# which depends on kustomize load restrictions being relaxed — true for Flux's kustomize-controller,
|
|
# refused by a plain `kustomize build`, and so a divergence between what the gate can verify and what the
|
|
# cluster does. Making this directory the Flux path removes the question entirely: no `../`, identical
|
|
# behaviour locally and in-cluster, one copy of every document. The serving manifests are tucked under
|
|
# `serving/` so `runbooks/*.md` stays exactly the flat set of pages the URLs name.
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
namespace: runbooks
|
|
resources:
|
|
- serving/namespace.yaml
|
|
- serving/nginx-config.yaml
|
|
- serving/deployment.yaml
|
|
- serving/service.yaml
|
|
# default-deny baseline + Cilium gateway-ingress allow (DNS-only egress from the baseline).
|
|
- serving/networkpolicy.yaml
|
|
- serving/httproute.yaml # apex `/runbooks` prefix (binds the Gateway `https-apex` listener)
|
|
|
|
configMapGenerator:
|
|
# ONE ConfigMap holding every runbook, keyed by the EXACT name the alert annotations use.
|
|
#
|
|
# THE KEYS ARE EXTENSIONLESS ON PURPOSE. The annotations are `https://${BASE_DOMAIN}/runbooks/<name>`
|
|
# with NO `.md`, and this repo must not change them (they are spread across four alerts.yaml files that
|
|
# other work is editing). `<key>=<file>` maps `node-down.md` on disk to the key `node-down`, the volume
|
|
# mount turns each key into a file under the nginx root, and `/runbooks/node-down` becomes a plain static
|
|
# lookup — no rewrite rule, nothing to get subtly wrong at 3am. Drop the `<key>=` and every key would
|
|
# carry `.md`, so every alert link would 404 while this file still looked correct.
|
|
#
|
|
# The generated name carries kustomize's content hash, so editing any runbook rolls the server
|
|
# automatically and a stale projection is impossible.
|
|
#
|
|
# ADDING A RUNBOOK: add the file AND its line here. The L0 check
|
|
# conformance/tests/l0-static/alert-runbooks-resolvable asserts both halves, because a file in git with
|
|
# no generator entry is reviewed, merged, apparently done — and still a 404 in the cluster.
|
|
- name: runbooks-content
|
|
options:
|
|
labels:
|
|
app.kubernetes.io/name: runbooks
|
|
files:
|
|
# The index, served at /runbooks/ (nginx `try_files /runbooks/README`).
|
|
- README=README.md
|
|
- backup-failure=backup-failure.md
|
|
- backup-missed=backup-missed.md
|
|
- backup-partial=backup-partial.md
|
|
- cdi-import-stuck=cdi-import-stuck.md
|
|
- cert-expiry=cert-expiry.md
|
|
- cert-issuance=cert-issuance.md
|
|
- cnpg-archiving=cnpg-archiving.md
|
|
- cnpg-backup-missed=cnpg-backup-missed.md
|
|
- cnpg-exporter=cnpg-exporter.md
|
|
- cnpg-primary-spread=cnpg-primary-spread.md
|
|
- cnpg-wal-growth=cnpg-wal-growth.md
|
|
# Operator procedure, not alert-linked (Crafty's local login is a second factor behind
|
|
# the Keycloak gate and its bootstrap password lives only on the PVC).
|
|
- crafty-admin-credential=crafty-admin-credential.md
|
|
- denied-flows=denied-flows.md
|
|
- flux-reconcile=flux-reconcile.md
|
|
- flux-rollout=flux-rollout.md
|
|
- flux-suspended=flux-suspended.md
|
|
- image-automation=image-automation.md
|
|
- image-policy-stalled=image-policy-stalled.md
|
|
- keycloak-client-secrets=keycloak-client-secrets.md
|
|
- keycloak-db=keycloak-db.md
|
|
- keycloak-down=keycloak-down.md
|
|
- kubevirt-control-plane=kubevirt-control-plane.md
|
|
- kubevirt-crashloop=kubevirt-crashloop.md
|
|
- kubevirt-launch-failed=kubevirt-launch-failed.md
|
|
- longhorn-capacity=longhorn-capacity.md
|
|
- longhorn-degraded=longhorn-degraded.md
|
|
- longhorn-faulted=longhorn-faulted.md
|
|
- monitoring-capacity=monitoring-capacity.md
|
|
- node-cpu=node-cpu.md
|
|
- node-disk=node-disk.md
|
|
- node-down=node-down.md
|
|
- node-io=node-io.md
|
|
- node-memory=node-memory.md
|
|
- node-memory-swap=node-memory-swap.md
|
|
# Operator procedure, not alert-linked. Reading the per-app oauth2-proxy sidecar log to tell
|
|
# Keycloak, the sidecar and the app's own login apart. Cookie collision between two sidecars
|
|
# presents as random session loss in ONE app's log and is only legible with both side by side,
|
|
# so the page has to be reachable at 3am without going looking for it in git.
|
|
- oauth2-proxy-session=oauth2-proxy-session.md
|
|
# Operator procedure, not an alert page (no alert links to it — the
|
|
# alert-runbooks-resolvable check treats that as advisory). It is served
|
|
# because the plex.tv claim window is short and is not a thing to go looking
|
|
# for in git: /runbooks/plex-claim has to resolve the moment it is needed.
|
|
- plex-claim=plex-claim.md
|
|
- pvc-capacity=pvc-capacity.md
|
|
- targets-down=targets-down.md
|
|
- update-rollback=update-rollback.md
|
|
- vpn-down=vpn-down.md
|
|
- vpn-egress-stale=vpn-egress-stale.md
|