Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
616a4974f3 | ||
|
|
0c7e8dd3ba | ||
|
|
8c12a1e525 | ||
|
|
8214642d87 | ||
|
|
4e1b169256 | ||
|
|
d44490736c |
No files matched your search
@@ -36,6 +36,28 @@ sensitive path, so the policy cannot be widened or narrowed without a human.
|
||||
|
||||
That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
|
||||
missing credential is a lock-out, here a skip would be a bypass.
|
||||
|
||||
**The administrator override (2026-10-01).** The rule above was unsatisfiable for the one human
|
||||
who runs this repo: Gitea refuses to let an author approve their own PR, and `jakman85` is the
|
||||
only collaborator — so a sensitive-path PR the operator opened could never be merged, *including
|
||||
the PR that would fix that.* An administrator may therefore waive the review by commenting on
|
||||
the PR:
|
||||
|
||||
/approve-override <reason>
|
||||
|
||||
The directive must **start a line**, the **reason is mandatory**, the commenter's `admin`/`owner`
|
||||
permission is re-checked **live** when the gate runs, and the comment must be **newer than the
|
||||
head commit** — push again and you must override again. The overrider, the timestamp and the
|
||||
reason are printed in the gate log and the job summary. For an administrator, "a change to the
|
||||
gate cannot be gated by the gate" is therefore **advisory, not structural**; what is kept is an
|
||||
attributed record of each waiver. Note that the agent account `gitea-admin` also holds `owner`
|
||||
here, so an agent *can* clear this on its own PR — **do not**: posting that directive to unblock
|
||||
your own agent work is exactly the self-approval the stage exists to prevent. Get the operator
|
||||
to post it, or move the change out of the sensitive paths.
|
||||
|
||||
**`required_approvals` on `main` is `0`**, not 1 — the gate is the approver. That drops the
|
||||
approval *count* only: `block_admin_merge_override` and `enable_status_check` both remain `true`,
|
||||
so **a red gate is unmergeable by everyone, administrators included.**
|
||||
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
|
||||
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
|
||||
|
||||
@@ -53,12 +75,19 @@ Verified against the live Gitea on 2026-09-29:
|
||||
| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
|
||||
| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
|
||||
| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
|
||||
| Approval required before merge | ✅ `required_approvals: 1` |
|
||||
| Gate cannot be overridden, by anyone | ✅ `block_admin_merge_override: true` |
|
||||
| Human approval required before merge | ⬜ **No — `required_approvals: 0` since 2026-10-01.** The gate is the approver; sensitive paths are gated by `ci/approval-policy.sh` instead. See §1 |
|
||||
|
||||
All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
|
||||
The first three were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
|
||||
true. They are true now because each was measured, and the push rejection was provoked rather
|
||||
than assumed. **Re-measure before relying on any of them** — see §2b.
|
||||
|
||||
The fourth row deliberately changed on 2026-10-01. `required_approvals: 1` was a **lock-out**, not
|
||||
a protection: with a single collaborator and Gitea refusing author self-approval, nothing could
|
||||
satisfy it (measured on PR #35 — green gate, zero reviews, unmergeable). It also contradicted the
|
||||
model already written into `ci/approval-policy.sh` and `ci/gate.sh`. Rationale and the live-change
|
||||
script: `infra/gitea/allow-admin-self-merge.sh`.
|
||||
|
||||
Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
|
||||
CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
|
||||
|
||||
|
||||
Executable
+139
@@ -0,0 +1,139 @@
|
||||
#!/usr/bin/env python3
|
||||
"""ci/approval-override.py — select the admin-override comment on a PR, if there is a valid one.
|
||||
|
||||
This script decides NOTHING about authority. It parses a PR's comment list and returns the
|
||||
single override *candidate* that ci/approval-policy.sh should then try to authorise against
|
||||
the live Gitea permission API. Keeping the two apart is the point:
|
||||
|
||||
* parsing is pure, offline, and therefore TESTABLE (ci/tests/approval-override.test.sh);
|
||||
* authorisation needs the API and must stay in the stage that fails closed.
|
||||
|
||||
A candidate is a comment whose body contains, AT THE START OF A LINE, the directive followed
|
||||
by a non-empty reason:
|
||||
|
||||
/approve-override the gate cannot gate itself and I am the only human reviewer
|
||||
|
||||
WHY EACH RULE IS HERE
|
||||
|
||||
* START OF A LINE, not "anywhere in the body". Otherwise quoting someone else's override
|
||||
("you could just /approve-override this") re-arms it. Markdown quotes are stripped of a
|
||||
leading '>' before matching for the same reason -- a quoted directive must NOT count, so
|
||||
lines beginning with '>' are DROPPED, not unquoted.
|
||||
* A REASON IS MANDATORY. A bare directive records that a review was waived but not why,
|
||||
which is the difference between an audit trail and a rubber stamp.
|
||||
* STALE OVERRIDES ARE DISCARDED. An override earned at commit A must not silently cover
|
||||
commit B. Any candidate created at or before the head commit's timestamp is dropped --
|
||||
the same semantics Gitea's own `dismiss_stale_approvals` applies to reviews. Push new
|
||||
commits and you must re-override.
|
||||
* THE LATEST CANDIDATE WINS, so a second override comment supersedes the first rather than
|
||||
both being live.
|
||||
|
||||
Output: one TSV line (login, created_at, reason) on stdout, or nothing at all when there is no
|
||||
valid candidate. Never exits non-zero for "no override" -- absence is a normal answer, and the
|
||||
caller must not be able to confuse it with a parse crash.
|
||||
|
||||
Input, all via env so nothing lands in `ps`:
|
||||
COMMENTS_JSON the Gitea /issues/{n}/comments array
|
||||
HEAD_TIME ISO-8601 timestamp of the PR head commit
|
||||
DIRECTIVE the override directive, e.g. /approve-override
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
def parse_ts(value):
|
||||
"""Gitea emits RFC3339. Return an aware datetime, or None if it is unparseable.
|
||||
|
||||
Unparseable is NOT 'treat as fresh': the caller drops such a candidate, because a
|
||||
timestamp we cannot read is a staleness check we cannot perform.
|
||||
"""
|
||||
if not value:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
# Python < 3.11 cannot read a trailing 'Z' in fromisoformat.
|
||||
if text.endswith(("Z", "z")):
|
||||
text = text[:-1] + "+00:00"
|
||||
try:
|
||||
parsed = datetime.fromisoformat(text)
|
||||
except ValueError:
|
||||
return None
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||
return parsed
|
||||
|
||||
|
||||
def extract_reason(body, directive):
|
||||
"""Return the reason from the first line that STARTS with the directive, else None."""
|
||||
pattern = re.compile(
|
||||
r"^[ \t]*" + re.escape(directive) + r"(?=[ \t:]|$)[ \t:]*(.*)$"
|
||||
)
|
||||
for raw in (body or "").splitlines():
|
||||
# A quoted line is somebody repeating an override, not casting one.
|
||||
if raw.lstrip().startswith(">"):
|
||||
continue
|
||||
match = pattern.match(raw)
|
||||
if match:
|
||||
reason = match.group(1).strip()
|
||||
return reason or None
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
directive = os.environ.get("DIRECTIVE", "/approve-override").strip()
|
||||
head_time = parse_ts(os.environ.get("HEAD_TIME", ""))
|
||||
|
||||
try:
|
||||
comments = json.loads(os.environ.get("COMMENTS_JSON") or "[]")
|
||||
except (ValueError, TypeError):
|
||||
# The caller validated this JSON already; if it is broken here, say so loudly on
|
||||
# stderr and emit no candidate. Silence on stdout means "no override".
|
||||
print("approval-override: COMMENTS_JSON is not valid JSON", file=sys.stderr)
|
||||
return 0
|
||||
if not isinstance(comments, list):
|
||||
print("approval-override: COMMENTS_JSON is not a list", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
candidates = []
|
||||
for comment in comments:
|
||||
if not isinstance(comment, dict):
|
||||
continue
|
||||
login = ((comment.get("user") or {}).get("login") or "").strip()
|
||||
reason = extract_reason(comment.get("body"), directive)
|
||||
if not login or not reason:
|
||||
continue
|
||||
created_raw = comment.get("created_at") or comment.get("updated_at") or ""
|
||||
created = parse_ts(created_raw)
|
||||
if created is None:
|
||||
print(
|
||||
"approval-override: dropping override by '%s' — unreadable timestamp %r"
|
||||
% (login, created_raw),
|
||||
file=sys.stderr,
|
||||
)
|
||||
continue
|
||||
# An override must post-date the commit it covers.
|
||||
if head_time is not None and created <= head_time:
|
||||
print(
|
||||
"approval-override: dropping STALE override by '%s' (%s) — not newer than "
|
||||
"head commit (%s). Re-comment to override the current commit."
|
||||
% (login, created.isoformat(), head_time.isoformat()),
|
||||
file=sys.stderr,
|
||||
)
|
||||
continue
|
||||
candidates.append((created, login, reason, created_raw))
|
||||
|
||||
if not candidates:
|
||||
return 0
|
||||
|
||||
# Latest wins: a later override supersedes an earlier one.
|
||||
created, login, reason, created_raw = max(candidates, key=lambda c: c[0])
|
||||
# Keep the reason on one line — this becomes a TSV field.
|
||||
reason = " ".join(reason.split())
|
||||
print("%s\t%s\t%s" % (login, created_raw, reason))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+136
-5
@@ -25,10 +25,45 @@
|
||||
# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
|
||||
# BYPASS, not a gap. So: cannot determine -> BLOCK.
|
||||
#
|
||||
# THE ADMIN OVERRIDE (added 2026-10-01), and the lock-out it clears.
|
||||
#
|
||||
# The rule above was UNSATISFIABLE for the one human who runs this repo. Gitea refuses to let a
|
||||
# pull request's author approve their own PR, and `jakman85` is the only collaborator — so "an
|
||||
# APPROVED review from someone who is not its author" is a condition the sole administrator can
|
||||
# never meet. Every sensitive-path PR they opened was permanently unmergeable, INCLUDING the PR
|
||||
# that would fix it. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, blocked.
|
||||
#
|
||||
# So an administrator may waive the review by COMMENTING on the PR:
|
||||
#
|
||||
# /approve-override <reason>
|
||||
#
|
||||
# Authority comes from the LIVE Gitea permission API, never from the comment — the commenter must
|
||||
# hold a permission listed in `.override.authorized_permissions` (admin/owner) at the moment the
|
||||
# gate runs, so an old comment from a since-demoted account is worth nothing. The directive must
|
||||
# start a line, a reason is mandatory, and the comment must POST-DATE the head commit: an override
|
||||
# earned at commit A does not cover commit B. Choosing the candidate comment is the job of the
|
||||
# pure, offline, unit-tested ci/approval-override.py; AUTHORISING it is the job of this stage,
|
||||
# which fails closed.
|
||||
#
|
||||
# WHAT THIS COSTS, STATED PLAINLY. For an administrator, "a change to the gate cannot be gated by
|
||||
# the gate" stops being structural and becomes advisory. That is the unavoidable consequence of
|
||||
# there being one human; the alternative is a repository its owner cannot change. What is bought
|
||||
# back is a RECORD — the waiver is an explicit act, attributed to an account, carrying a stated
|
||||
# reason, in the PR history and in this stage's output. A silent bypass keyed on "is the author an
|
||||
# admin?" would have been less code and is exactly the fail-open shape this repo keeps finding.
|
||||
#
|
||||
# AND THE PART THAT IS NOT COMFORTABLE. Authorisation is by permission ALONE, with no allowlist of
|
||||
# human accounts (operator decision, 2026-10-01, taken with this consequence on the table). The
|
||||
# AGENT account `gitea-admin` also reports permission `owner` here, so an agent can clear the
|
||||
# sensitive-path review on its own PR by posting the directive — and agents author most PRs in
|
||||
# this repo, so that is the common case, not an edge case. To narrow it without touching this
|
||||
# logic, add an `override_admins` list to infra/gitea/approval-policy.json: the check below
|
||||
# already requires membership whenever that key is present.
|
||||
#
|
||||
# EXIT CODES
|
||||
# 0 no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
|
||||
# non-author APPROVED review is present
|
||||
# 1 sensitive paths touched without such a review, OR the policy could not be evaluated
|
||||
# 0 no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and
|
||||
# either a non-author APPROVED review or a valid admin override is present
|
||||
# 1 sensitive paths touched with neither, OR the policy could not be evaluated
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
@@ -65,7 +100,11 @@ api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
|
||||
FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
|
||||
PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
|
||||
REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
|
||||
for v in FILES_JSON PR_JSON REVIEWS_JSON; do
|
||||
# PR conversation comments live under the ISSUES endpoint. `/pulls/{n}/comments` returns review
|
||||
# (diff) comments ONLY, so an override posted in the normal comment box would never appear in it.
|
||||
# Verified against this Gitea (1.24.3) rather than assumed.
|
||||
COMMENTS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/issues/${PR}/comments?limit=1000" || true)"
|
||||
for v in FILES_JSON PR_JSON REVIEWS_JSON COMMENTS_JSON; do
|
||||
jq -e . >/dev/null 2>&1 <<<"${!v}" || {
|
||||
err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
|
||||
summary "- approval-policy: **BLOCKED — Gitea unreadable**"
|
||||
@@ -107,11 +146,103 @@ if [ -n "$APPROVER" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────
|
||||
# No non-author approval exists. Before refusing, look for an administrator's explicit
|
||||
# override. See the header for why this hatch exists and what it costs.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────
|
||||
OVERRIDE_ENABLED="$(jq -r '.override.enabled // false' "$DECL")"
|
||||
DIRECTIVE="$(jq -r '.override.comment_directive // "/approve-override"' "$DECL")"
|
||||
HEAD_SHA="$(jq -r '.head.sha // empty' <<<"$PR_JSON")"
|
||||
|
||||
if [ "$OVERRIDE_ENABLED" = "true" ]; then
|
||||
# The head commit's timestamp anchors staleness. Read it from GITEA, not from the local
|
||||
# tree: CI checks out a merge commit, whose date is not the head commit's date.
|
||||
HEAD_TIME=""
|
||||
if [ -n "$HEAD_SHA" ]; then
|
||||
HEAD_COMMIT_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/git/commits/${HEAD_SHA}" || true)"
|
||||
HEAD_TIME="$(jq -r '.commit.committer.date // .created // empty' <<<"${HEAD_COMMIT_JSON:-null}" 2>/dev/null || true)"
|
||||
fi
|
||||
if [ -z "$HEAD_TIME" ]; then
|
||||
err "could not read the head commit's timestamp for PR #${PR} — BLOCKING."
|
||||
err " Without it an override cannot be aged, and an unaged override is one that silently"
|
||||
err " covers commits nobody waived review on."
|
||||
summary "- approval-policy: **BLOCKED — head commit timestamp unreadable** (cannot age an override)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Pure, offline, unit-tested selection: which comment, if any, is a live override?
|
||||
OVERRIDE_TSV="$(COMMENTS_JSON="$COMMENTS_JSON" HEAD_TIME="$HEAD_TIME" DIRECTIVE="$DIRECTIVE" \
|
||||
python3 "${CI_DIR}/approval-override.py")" || {
|
||||
err "ci/approval-override.py failed — BLOCKING rather than assuming there is no override."
|
||||
summary "- approval-policy: **BLOCKED — override parser failed**"
|
||||
exit 1; }
|
||||
|
||||
if [ -n "$OVERRIDE_TSV" ]; then
|
||||
OV_USER="$(cut -f1 <<<"$OVERRIDE_TSV")"
|
||||
OV_WHEN="$(cut -f2 <<<"$OVERRIDE_TSV")"
|
||||
OV_WHY="$(cut -f3- <<<"$OVERRIDE_TSV")"
|
||||
|
||||
# AUTHORITY IS MEASURED LIVE, never inferred from the comment or from a name in git.
|
||||
PERM_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/collaborators/${OV_USER}/permission" || true)"
|
||||
OV_PERM="$(jq -r '.permission // empty' <<<"${PERM_JSON:-null}" 2>/dev/null || true)"
|
||||
if [ -z "$OV_PERM" ]; then
|
||||
err "'${OV_USER}' posted ${DIRECTIVE}, but their permission on ${GITEA_REPO} could not be"
|
||||
err " read — BLOCKING. An override whose authority is unverified is not an override."
|
||||
summary "- approval-policy: **BLOCKED — cannot verify \`${OV_USER}\`'s permission**"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Authorised when the LIVE permission is one of the declared ones and — only if an
|
||||
# `override_admins` allowlist exists at all — the account is on it. No allowlist means
|
||||
# permission alone, which is the operator's standing decision; adding the key narrows
|
||||
# this with no change to any code.
|
||||
AUTHORISED="$(DECL="$DECL" OV_PERM="$OV_PERM" OV_USER="$OV_USER" python3 -c '
|
||||
import json, os
|
||||
d = json.load(open(os.environ["DECL"])).get("override", {})
|
||||
perms = d.get("authorized_permissions") or ["admin", "owner"]
|
||||
allow = d.get("override_admins")
|
||||
ok_perm = os.environ["OV_PERM"] in perms
|
||||
ok_list = True if allow is None else os.environ["OV_USER"] in allow
|
||||
print("yes" if (ok_perm and ok_list) else "no")
|
||||
')"
|
||||
|
||||
if [ "$AUTHORISED" = "yes" ]; then
|
||||
if [ "$OV_USER" = "$AUTHOR" ]; then
|
||||
warn "ADMIN OVERRIDE — SELF-WAIVED. '${OV_USER}' authored this PR and waived the"
|
||||
warn " non-author review requirement on their own change. Permitted (permission"
|
||||
warn " '${OV_PERM}'), deliberate, and shouted about here precisely because it is not free."
|
||||
else
|
||||
warn "ADMIN OVERRIDE by '${OV_USER}' (permission '${OV_PERM}'); author is '${AUTHOR}'."
|
||||
fi
|
||||
warn " when: ${OV_WHEN} (head commit ${HEAD_SHA:0:8}, ${HEAD_TIME})"
|
||||
warn " reason: ${OV_WHY}"
|
||||
warn " review waived on: $(printf '%s ' $SENSITIVE)"
|
||||
ok "sensitive paths cleared by administrator override — this stage does not block."
|
||||
SELFNOTE=""; [ "$OV_USER" = "$AUTHOR" ] && SELFNOTE=", **self-waived as author**"
|
||||
summary "- approval-policy: **OVERRIDDEN by \`${OV_USER}\`** (permission \`${OV_PERM}\`${SELFNOTE})"
|
||||
summary " - reason: ${OV_WHY}"
|
||||
summary " - review waived on: $(printf '\`%s\` ' $SENSITIVE)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
err "'${OV_USER}' posted ${DIRECTIVE} but is NOT authorised to override: live permission is"
|
||||
err " '${OV_PERM}'. The authorised set is declared in infra/gitea/approval-policy.json."
|
||||
summary "- approval-policy: **BLOCKED — \`${OV_USER}\` may not override** (permission \`${OV_PERM}\`)"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
|
||||
err " review from anyone other than its author ('${AUTHOR}')."
|
||||
err " A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
|
||||
err " honestly approve for itself. Get a human review, or move the change out of:"
|
||||
printf ' %s\n' $SENSITIVE >&2
|
||||
err " The list lives in infra/gitea/approval-policy.json, with a reason per entry."
|
||||
summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
|
||||
if [ "$OVERRIDE_ENABLED" = "true" ]; then
|
||||
err " Or, if you hold admin/owner on this repository, comment on the PR:"
|
||||
err " ${DIRECTIVE} <reason>"
|
||||
err " The reason is mandatory, the directive must START a line, and the comment must be"
|
||||
err " NEWER than head commit ${HEAD_SHA:0:8} — push again and you must override again."
|
||||
fi
|
||||
summary "- approval-policy: **BLOCKED — needs a non-author approval, or an admin \`${DIRECTIVE}\` comment**"
|
||||
exit 1
|
||||
+54
-1
@@ -1,5 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/render-golden.sh — Stage: render==golden reproducibility gate.
|
||||
# ci/render-golden.sh — Stage: the renderer is trustworthy.
|
||||
# (1) catalog/render.py passes its OWN output-contract self-test, and
|
||||
# (2) every committed apps/ golden reproduces byte-for-byte from the canonical render.
|
||||
# Two independent assertions about one subject, catalog/render.py. (1) was added
|
||||
# 2026-10-01 because nothing in ci/ or .gitea/ had ever invoked `--selftest`; see the
|
||||
# block above the invocation for why it lives here and not in a conformance check.
|
||||
#
|
||||
# WHY: the committed apps/ tree is a set of GOLDENS produced by ONE canonical
|
||||
# invocation of catalog/render.py. If a golden is hand-edited, or a catalog
|
||||
@@ -54,6 +59,54 @@ if [ ! -f "$RENDER_PY" ]; then
|
||||
summary "- render-golden: **SKIPPED** (renderer missing)"
|
||||
exit 3
|
||||
fi
|
||||
# --- The renderer's OWN self-test, before anything is compared ---------------
|
||||
# `catalog/render.py --selftest` is ~106 executed assertions on the renderer's output
|
||||
# contract (default-deny covers both directions, the Kyverno namespace hook label, a valid
|
||||
# auth-mode, the oauth2-proxy sidecar's cookie-name/-domain/-refresh flags, the
|
||||
# placeholder-substitution guard, the observability refusals...). Until now NOTHING in ci/
|
||||
# or .gitea/ invoked it — measured by grep on 2026-09-30 and recorded as the top follow-up
|
||||
# of decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md — so every one of those
|
||||
# assertions was dev-time only and a merge gate could not tell a sound renderer from a
|
||||
# gutted one.
|
||||
#
|
||||
# WHY HERE, AND NOT IN A CONFORMANCE CHECK. The lead was
|
||||
# conformance/tests/l0-static/auditability-proofs-producible/check.sh, which drives
|
||||
# `conformance/lib/{record_run,status}.py --selftest` "because nothing else drives them".
|
||||
# That loop is NOT the right hook: that check serves R-SEC-03 and its subject is the
|
||||
# EVIDENCE CHAIN (can a forged or stale run become a proof?). The renderer is not part of
|
||||
# that chain, and hanging it there would make the check's name and its requirement mapping
|
||||
# a lie. This stage's subject already IS catalog/render.py — it exists to assert that the
|
||||
# renderer's output is trustworthy — so the renderer's own contract test belongs in it.
|
||||
#
|
||||
# It runs BEFORE the catalog/empty-catalog exit-3 below on purpose: the self-test is
|
||||
# deterministic and catalog-independent (it renders synthetic ServiceDefinitions in
|
||||
# memory), so an empty catalog must not be able to skip it.
|
||||
#
|
||||
# render-golden and the self-test are INDEPENDENT assertions, which is why both are needed:
|
||||
# render-golden catches golden-vs-renderer drift but is satisfied by ANY renderer whose
|
||||
# output was re-rendered, including one that just lost a contract invariant and had all ten
|
||||
# goldens regenerated in the same commit. The self-test catches exactly that.
|
||||
#
|
||||
# FAILURE IS A HARD FAIL, NOT A SKIP: the renderer is source in this repo, so "could not
|
||||
# run its self-test" has no legitimate environmental cause once python3 and render.py are
|
||||
# both present (both asserted above). ci/gate.sh maps this stage's rc 1 to BLOCKED.
|
||||
log "Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)"
|
||||
selftest_out="$(mktemp)"
|
||||
if "$PYTHON" "$RENDER_PY" --selftest >"$selftest_out" 2>&1; then
|
||||
ok "render.py --selftest passed"
|
||||
summary "- render-selftest: **passed** (\`catalog/render.py --selftest\`)"
|
||||
else
|
||||
err "catalog/render.py --selftest FAILED — the renderer violates its own output contract."
|
||||
err " Every golden under apps/ is produced by this renderer, so the goldens below cannot"
|
||||
err " be trusted even if they reproduce byte-for-byte. Reproduce with:"
|
||||
err " ${PYTHON} catalog/render.py --selftest"
|
||||
sed 's/^/ /' "$selftest_out" >&2 || true
|
||||
summary "- render-selftest: **FAILED** (\`catalog/render.py --selftest\` — renderer contract violated)"
|
||||
rm -f "$selftest_out"
|
||||
exit 1
|
||||
fi
|
||||
rm -f "$selftest_out"
|
||||
|
||||
if [ ! -d "$CATALOG_DIR" ] || ! compgen -G "${CATALOG_DIR}/*.yaml" >/dev/null 2>&1; then
|
||||
skip "no catalog ServiceDefinitions under '${CATALOG_DIR}' — nothing to render."
|
||||
summary "- render-golden: **SKIPPED** (empty catalog)"
|
||||
|
||||
Executable
+145
@@ -0,0 +1,145 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/approval-override.test.sh — proves ci/approval-override.py selects the override it
|
||||
# claims to select, and REJECTS the ones it claims to reject.
|
||||
#
|
||||
# This parser decides whether a human review requirement gets waived. Every rule in it exists
|
||||
# because the obvious implementation ("does the body contain the directive?") is wrong in a way
|
||||
# that is invisible until abused: a quoted directive would re-arm an override, a bare directive
|
||||
# would waive review with no stated reason, and an override posted against commit A would go on
|
||||
# covering commit B forever. A guard is worth nothing until somebody has watched it refuse, which
|
||||
# is the same reasoning as ci/tests/decisions-log.test.sh one layer up.
|
||||
#
|
||||
# The parser is pure and offline BY DESIGN — authorisation (the live permission lookup) lives in
|
||||
# ci/approval-policy.sh, which cannot be tested without Gitea. That split is what makes this file
|
||||
# possible, and it is also why a green run here does NOT prove the stage authorises correctly.
|
||||
set -uo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SUT="${CI_DIR}/approval-override.py"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
||||
|
||||
HEAD_TIME="2026-10-01T12:00:00Z"
|
||||
AFTER="2026-10-01T12:30:00Z"
|
||||
BEFORE="2026-10-01T11:30:00Z"
|
||||
|
||||
# run <comments-json> -> stdout of the parser (stderr is diagnostic only)
|
||||
run() {
|
||||
COMMENTS_JSON="$1" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
|
||||
python3 "$SUT" 2>/dev/null
|
||||
}
|
||||
|
||||
# comment <login> <created_at> <body>
|
||||
comment() {
|
||||
python3 -c '
|
||||
import json, sys
|
||||
print(json.dumps([{"user": {"login": sys.argv[1]}, "created_at": sys.argv[2], "body": sys.argv[3]}]))
|
||||
' "$1" "$2" "$3"
|
||||
}
|
||||
|
||||
expect_none() {
|
||||
local label="$1" out
|
||||
out="$(run "$2")"
|
||||
if [ -z "$out" ]; then ok "$label"; else bad "$label (got: ${out})"; fi
|
||||
}
|
||||
|
||||
expect_user() {
|
||||
local label="$1" json="$2" want="$3" out got
|
||||
out="$(run "$json")"
|
||||
got="$(cut -f1 <<<"$out")"
|
||||
if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted ${want}, got '${got}')"; fi
|
||||
}
|
||||
|
||||
expect_reason() {
|
||||
local label="$1" json="$2" want="$3" out got
|
||||
out="$(run "$json")"
|
||||
got="$(cut -f3- <<<"$out")"
|
||||
if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted '${want}', got '${got}')"; fi
|
||||
}
|
||||
|
||||
echo "== ci/approval-override.py"
|
||||
|
||||
# --- the happy path -------------------------------------------------------------------
|
||||
expect_user "a directive with a reason, posted after the head commit, is selected" \
|
||||
"$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" jakman85
|
||||
|
||||
expect_reason "the reason is returned verbatim" \
|
||||
"$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" \
|
||||
'sole admin, gate is green'
|
||||
|
||||
# --- absence is a normal answer, not an error -----------------------------------------
|
||||
expect_none "no comments at all" '[]'
|
||||
expect_none "an ordinary comment" "$(comment jakman85 "$AFTER" 'looks good to me')"
|
||||
|
||||
# --- a reason is MANDATORY ------------------------------------------------------------
|
||||
expect_none "a bare directive (no reason) is refused" \
|
||||
"$(comment jakman85 "$AFTER" '/approve-override')"
|
||||
expect_none "a directive followed by only whitespace is refused" \
|
||||
"$(comment jakman85 "$AFTER" '/approve-override ')"
|
||||
|
||||
# --- the directive must START a line --------------------------------------------------
|
||||
expect_none "the directive mid-sentence does not count" \
|
||||
"$(comment jakman85 "$AFTER" 'you could just /approve-override this one')"
|
||||
expect_none "a QUOTED directive does not re-arm an override" \
|
||||
"$(comment jakman85 "$AFTER" '> /approve-override quoting someone else')"
|
||||
expect_user "leading whitespace before the directive is fine" \
|
||||
"$(comment jakman85 "$AFTER" ' /approve-override indented but mine')" jakman85
|
||||
expect_user "the directive on a later line of a longer comment counts" \
|
||||
"$(comment jakman85 "$AFTER" 'some preamble
|
||||
/approve-override the real reason here')" jakman85
|
||||
|
||||
# --- staleness: an override must post-date the commit it waives review on -------------
|
||||
expect_none "an override OLDER than the head commit is dropped as stale" \
|
||||
"$(comment jakman85 "$BEFORE" '/approve-override stale, predates the push')"
|
||||
expect_none "an override exactly AT the head commit time is dropped (strictly newer)" \
|
||||
"$(comment jakman85 "$HEAD_TIME" '/approve-override same instant')"
|
||||
|
||||
# --- malformed input is never a free pass --------------------------------------------
|
||||
expect_none "an unreadable timestamp drops the candidate" \
|
||||
"$(comment jakman85 'not-a-date' '/approve-override bad clock')"
|
||||
expect_none "a comment with no author is dropped" \
|
||||
'[{"user": {}, "created_at": "2026-10-01T12:30:00Z", "body": "/approve-override no login"}]'
|
||||
expect_none "invalid JSON yields no override (and never a traceback into stdout)" \
|
||||
'this is not json'
|
||||
expect_none "a JSON object instead of a list yields no override" \
|
||||
'{"user": "nope"}'
|
||||
|
||||
# --- latest wins ----------------------------------------------------------------------
|
||||
TWO='[
|
||||
{"user": {"login": "early"}, "created_at": "2026-10-01T12:10:00Z", "body": "/approve-override first"},
|
||||
{"user": {"login": "latest"}, "created_at": "2026-10-01T12:40:00Z", "body": "/approve-override second"}
|
||||
]'
|
||||
expect_user "the LATEST override supersedes an earlier one" "$TWO" latest
|
||||
expect_reason "and brings its own reason" "$TWO" second
|
||||
|
||||
# --- offsets, not just Z --------------------------------------------------------------
|
||||
# Gitea emits RFC3339 WITH AN OFFSET (measured: 2026-10-01T23:09:08-05:00), not a trailing Z.
|
||||
# A parser that only handles 'Z' silently drops every real override, so this case is load-bearing.
|
||||
expect_user "an RFC3339 offset timestamp (-05:00), as Gitea actually emits, parses" \
|
||||
'[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T08:30:00-05:00", "body": "/approve-override offset form"}]' \
|
||||
jakman85
|
||||
expect_none "and an offset timestamp that is genuinely earlier is still stale" \
|
||||
'[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T06:00:00-05:00", "body": "/approve-override offset, stale"}]'
|
||||
|
||||
# --- the directive is configurable ----------------------------------------------------
|
||||
out="$(run "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')" "$HEAD_TIME" '/admin-waive')"
|
||||
if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
|
||||
ok "a custom directive from the declaration is honoured"
|
||||
else
|
||||
bad "a custom directive from the declaration is honoured (got '${out}')"
|
||||
fi
|
||||
expect_none "the default directive does not match a custom-directive comment" \
|
||||
"$(comment jakman85 "$AFTER" '/admin-waive because policy says so')"
|
||||
|
||||
# --- no staleness anchor: do not invent one -------------------------------------------
|
||||
out="$(COMMENTS_JSON="$(comment jakman85 "$BEFORE" '/approve-override no anchor')" \
|
||||
HEAD_TIME="" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
||||
if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
|
||||
ok "with no HEAD_TIME the staleness test is skipped, not faked (caller blocks instead)"
|
||||
else
|
||||
bad "with no HEAD_TIME the candidate should still be returned (got '${out}')"
|
||||
fi
|
||||
|
||||
printf '\n %d passed, %d failed\n' "$PASS" "$FAIL"
|
||||
[ "$FAIL" -eq 0 ] || exit 1
|
||||
@@ -42,7 +42,16 @@ cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env
|
||||
# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
|
||||
# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
|
||||
# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
|
||||
for stage in spec-immutability decisions-log lint l0 render-golden l1 gitea-settings conformance; do
|
||||
#
|
||||
# AND IT HAPPENED AGAIN, with `approval-policy` (fixed 2026-10-01). Same shape, same cost: the
|
||||
# stage was added to gate.sh by PR #17, never added here, so 15 of 50 assertions failed on `main`
|
||||
# — measured on 0c7e8dd before touching anything, to be sure it was not the change under test.
|
||||
# The note above was written about the previous instance and did not prevent the next one, which
|
||||
# says the comment is not the fix. THE FIX IS TO DERIVE THE LIST: it now comes from gate.sh's own
|
||||
# `run_stage` calls, so a stage added there can no longer be silently missing here.
|
||||
mapfile -t STAGES < <(sed -n 's/^run_stage[[:space:]]\+\([a-z0-9-]\+\).*/\1/p' "${REAL_CI_DIR}/gate.sh")
|
||||
[ "${#STAGES[@]}" -ge 8 ] || { echo "could not parse stage names out of gate.sh (got ${#STAGES[@]})" >&2; exit 1; }
|
||||
for stage in "${STAGES[@]}"; do
|
||||
cat > "${CI}/${stage}.sh" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
# Flux Kustomization: Intel GPU device plugin — reconciles ./platform/intel-gpu-plugin.
|
||||
# Makes the three prod nodes advertise `gpu.intel.com/i915`, which is the resource a later PR will
|
||||
# have `hardware.gpu: intel-quicksync` request in place of today's non-working /dev/dri hostPath.
|
||||
#
|
||||
# WHY ITS OWN KUSTOMIZATION, AND NOT ONE LINE IN platform/kustomization.yaml
|
||||
#
|
||||
# The cheap option was a line in the shared `platform` tier plus a new platform/ directory, and
|
||||
# neither path is sensitive in infra/gitea/approval-policy.json, so it would have needed no human
|
||||
# reviewer on the policy's terms. It is still the wrong place, for the reason the repo already
|
||||
# wrote down in platform/kustomization.yaml when Cilium was split out of that tier:
|
||||
#
|
||||
# * `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot tell
|
||||
# you "the GPU plugin is wedged" from "longhorn is slow".
|
||||
# * The only emergency valve would be `flux suspend kustomization platform` — freezing storage,
|
||||
# TLS, DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device
|
||||
# plugin.
|
||||
# * The tier's 10m interval / 5m timeout are explicitly called out there as WRONG for a 3-node
|
||||
# DaemonSet rollout. This is a 3-node DaemonSet rollout.
|
||||
#
|
||||
# And the consequence is worse here than for an average component, because of what sits downstream:
|
||||
# `platform` gates platform-config, which gates apps-wave-0 (wait:true), which gates apps-wave-1.
|
||||
# A plugin DaemonSet that never went healthy inside that tier would stall every app on "dependency
|
||||
# not ready" — i.e. a component added to fix a SILENT NO-OP (software transcoding) would have
|
||||
# acquired the ability to cause a REAL OUTAGE. That trade is the whole argument.
|
||||
#
|
||||
# The usual objection to this placement is the approval cost, and it does not apply: Gitea enforces
|
||||
# `required_approvals: 1` with `block_admin_merge_override: true` on EVERY PR here regardless of
|
||||
# path, and authors cannot self-approve. So this change needed the operator's approval either way;
|
||||
# touching clusters/** costs nothing extra and buys a blast radius of exactly one component.
|
||||
#
|
||||
# dependsOn: [cilium], WHICH IS THE MINIMUM AND ALSO THE EARLIEST
|
||||
#
|
||||
# Not platform-config (what headlamp/crafty/homepage/oauth2-proxy use) — those are leaf UIs that
|
||||
# genuinely need Keycloak + wildcard TLS + split-horizon DNS + Gateway. This plugin needs none of
|
||||
# it. It never uses the network and never talks to the API server; it talks to the local kubelet
|
||||
# over a unix socket. Its only real prerequisite is a working CNI, so its pod can get a sandbox at
|
||||
# all, and `cilium` already depends on `flux-system` (the root Kustomization that applies the
|
||||
# cluster-vars ConfigMap this one substitutes from), so that edge is transitively satisfied too.
|
||||
#
|
||||
# Depending on `cilium` rather than `platform` also gets the resource advertised EARLIEST — this
|
||||
# reconciles in parallel with the `platform` tier instead of behind all eleven of its components.
|
||||
# That matters on a cold start once the consumer PR lands: a Plex pod requesting a resource no node
|
||||
# advertises is Unschedulable, and apps-wave-0 is wait:true. Reconciling early shrinks that window
|
||||
# instead of widening it. It is not a GUARANTEE — see the ordering note below, which is a real
|
||||
# constraint on the next PR and not a formality.
|
||||
#
|
||||
# Kyverno (in `platform`) is deliberately NOT a dependency: every policy that could match this
|
||||
# workload is scoped to namespaces labelled app-namespace=true, and this namespace is labelled
|
||||
# "false" (platform/intel-gpu-plugin/namespace.yaml). So there is no admission race to lose.
|
||||
#
|
||||
# wait: true — A HEALTH SIGNAL, NOT A GATE. This is the one place this component departs from the
|
||||
# headlamp/crafty leaf template, and the departure is deliberate. Those run wait:false because a
|
||||
# leaf UI can be legitimately not-Ready for a long time during a cold start while it waits on
|
||||
# Keycloak/DNS/TLS, so its health is not a useful signal. This DaemonSet has no such dependency: it
|
||||
# reads sysfs and opens a socket, so on healthy nodes it should be Ready in seconds, and
|
||||
# not-Ready genuinely means BROKEN. wait:true turns `flux get kustomization intel-gpu-plugin` into
|
||||
# a one-line answer to "did the plugin come up on all three nodes", which is exactly the question
|
||||
# the operator must answer before the consumer PR may merge.
|
||||
#
|
||||
# NOTHING IN THE GRAPH dependsOn THIS, so wait:true gates nothing — the blast radius of an unhealthy
|
||||
# plugin is this Kustomization's own Ready condition and nothing else. Anyone later adding
|
||||
# `dependsOn: [intel-gpu-plugin]` is choosing to make plugin health gate their thing; for the app
|
||||
# waves that would hand a device plugin the power to stall every app, which is the thing this file
|
||||
# exists to prevent. Think before adding that edge.
|
||||
#
|
||||
# ORDERING — DO NOT REORDER THE STACK. The renderer change (catalog/render.py emitting
|
||||
# `gpu.intel.com/i915: 1` for `hardware.gpu: intel-quicksync` instead of the hostPath) must NOT
|
||||
# land until this plugin is merged AND confirmed advertising the resource on all three nodes. If
|
||||
# the renderer requests a resource no node advertises, the Plex pod is Unschedulable; apps-wave-0 is
|
||||
# wait:true and apps-wave-1 dependsOn it, so that stalls the whole app tier — turning today's silent
|
||||
# no-op into a real outage. This PR's only job is to make the resource exist.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: intel-gpu-plugin
|
||||
namespace: flux-system
|
||||
spec:
|
||||
# Static vendored manifests with nothing to re-render; kustomize-controller is source-triggered,
|
||||
# so this is only the drift-correction floor.
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
# 10m, not the 5m the `platform` tier uses: a 3-node DaemonSet rollout under `wait: true` can
|
||||
# exceed 5m on a cold start (image pull on three nodes), and the Cilium split-out comment calls
|
||||
# out that exact mismatch as one of its reasons for leaving the shared tier.
|
||||
timeout: 10m
|
||||
prune: true
|
||||
# See the header: a health SIGNAL for a component nothing depends on, not a gate.
|
||||
wait: true
|
||||
dependsOn:
|
||||
- name: cilium # the CNI — the only real prerequisite; transitively gives us flux-system too
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/intel-gpu-plugin
|
||||
# Retained for parity with every other tier / future overlay knobs. This component references no
|
||||
# ${VARS} today, and its manifests contain no `$` literals for envsubst to eat (unlike
|
||||
# clusters/base/runbooks.yaml, which omits substitution for exactly that reason).
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# No SOPS decryption patch: this component carries no Secret, so clusters/dev adds nothing for it
|
||||
# (same as homepage/runbooks, unlike headlamp/crafty/oauth2-proxy/backup).
|
||||
@@ -12,6 +12,15 @@ resources:
|
||||
# interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
|
||||
# freeze the other eleven platform components. See clusters/base/cilium.yaml.
|
||||
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
||||
# Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
|
||||
# platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
|
||||
# that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
|
||||
# components AND would stall platform-config + every apps wave on "dependency not ready" — so a
|
||||
# component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
|
||||
# wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
|
||||
# It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
|
||||
# strictly later. See clusters/base/intel-gpu-plugin.yaml.
|
||||
- intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
|
||||
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
||||
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
||||
# SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
---
|
||||
date: 2026-10-01
|
||||
branch: fix/admin-approval-override
|
||||
title: Administrators can merge their own green PR — required_approvals 1 → 0, plus an attributed /approve-override for sensitive paths
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**The problem, measured first.** `required_approvals: 1` on `main` was a lock-out, not a
|
||||
protection. Gitea refuses to let a pull request's author approve their own PR, and `jakman85` is
|
||||
the **only** collaborator (permission `owner`). So no account in existence could approve a PR the
|
||||
operator opened. Measured on the live API, 2026-10-01:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| PR #35 (`bugfix/arr-to-plex`, author `jakman85`) | `gate / gate (pull_request)` = **success, 1m41s** |
|
||||
| its reviews | `[]` |
|
||||
| live `required_approvals` | `1` |
|
||||
| live `block_admin_merge_override` | `true` |
|
||||
|
||||
Green gate, zero reviews, no path to a merge — and the same was true of any PR that would have
|
||||
fixed it.
|
||||
|
||||
It was also never the design this repo wrote down. `ci/approval-policy.sh`'s own header and
|
||||
`ci/gate.sh:194` both already described the model as *"`required_approvals` drops to 0 and the
|
||||
GATE decides"*, and `infra/gitea/require-gate.sh`'s closing note named this exact failure and
|
||||
prescribed this exact remedy. `decisions/2026-09-30-flux-deploy-notification-inhibited.md` had
|
||||
flagged the 1-vs-0 disagreement as unresolved and not that session's to settle. This settles it
|
||||
at 0.
|
||||
|
||||
**Two layers were blocking, and both are addressed.**
|
||||
|
||||
*Layer 1 — Gitea branch protection.* `required_approvals: 1 → 0` in
|
||||
`infra/gitea/expected-branch-protection.json`, with the rationale in the file, plus
|
||||
`infra/gitea/allow-admin-self-merge.sh` to apply it to the live rule (it reads the target **from
|
||||
the declaration**, so the script cannot drift from what the gate checks). `require-gate.sh` was
|
||||
changed to 0 as well — left at 1 it would silently re-impose the lock-out on its next run.
|
||||
|
||||
The narrow choice matters: the *other* way to unblock an owner is
|
||||
`block_admin_merge_override: false`, and that would have re-opened precisely what
|
||||
`infra/gitea/block-admin-override.sh` was written to close (PRs #5–#9 merged with zero approvals
|
||||
while the settings read as enforced). This change drops the approval **count** only.
|
||||
`block_admin_merge_override` and `enable_status_check` both stay `true`, and the script asserts
|
||||
both afterwards and fails loudly if the PATCH disturbed either. **A red gate remains unmergeable
|
||||
by everyone, administrators included.**
|
||||
|
||||
*Layer 2 — `ci/approval-policy.sh`.* Sensitive paths (`ci/**`, `infra/gitea/**`, `CLAUDE.md`,
|
||||
kyverno, rbac, sops, Flux wiring) still require a non-author `APPROVED` review, which the sole
|
||||
administrator still cannot obtain. An administrator may now waive it by commenting on the PR:
|
||||
|
||||
/approve-override <reason>
|
||||
|
||||
- the directive must **start a line** (a quoted or mid-sentence mention does not count, or
|
||||
repeating someone else's override would re-arm it);
|
||||
- a **reason is mandatory** — a bare directive records that review was waived but not why;
|
||||
- the commenter's permission is **re-checked live** against
|
||||
`/collaborators/{login}/permission` when the gate runs, so an old comment from a demoted
|
||||
account carries nothing;
|
||||
- it is **invalidated by a new commit** (the comment must post-date the head commit), mirroring
|
||||
Gitea's own `dismiss_stale_approvals`;
|
||||
- the overrider, timestamp and reason are printed in the gate log and the job summary, and a
|
||||
self-waiver by the PR's own author is called out as such.
|
||||
|
||||
Selection of the candidate comment is `ci/approval-override.py` — pure, offline, no authority of
|
||||
its own — so it could be unit-tested; authorisation stays in the fail-closed stage.
|
||||
`ci/tests/approval-override.test.sh` adds **23 assertions**, all passing, each constructing a
|
||||
case the parser must refuse. Every new failure mode blocks: unreadable head-commit timestamp,
|
||||
unreadable permission, parser error.
|
||||
|
||||
**The cost, stated rather than buried.** For an administrator, *"a change to the gate cannot be
|
||||
gated by the gate"* is now **advisory, not structural**. That is unavoidable with one human — the
|
||||
alternative is a repository its owner cannot change — and what is bought back is an attributed,
|
||||
reasoned, commit-scoped record of each waiver instead of a silent bypass.
|
||||
|
||||
**The uncomfortable part, accepted with eyes open (operator decision).** Authorisation is by live
|
||||
permission **alone**, with no allowlist of human accounts. The **agent account `gitea-admin` also
|
||||
reports permission `owner`** on this repo (measured), so an agent *can* clear the sensitive-path
|
||||
review on its own PR — and agents author most PRs here, so that is the common case, not an edge
|
||||
case. I raised this before implementing; the operator chose permission-only. It is documented in
|
||||
three places rather than left to be discovered: `approval-policy.json`
|
||||
(`_known_consequence_agent_accounts`), the stage's header, and `CLAUDE.md`, which now tells
|
||||
agents plainly **not** to use it on their own work. The narrowing path needs no code change —
|
||||
add an `override_admins` list to `approval-policy.json` and the stage already requires membership
|
||||
in addition to the permission check.
|
||||
|
||||
Also updated: `CLAUDE.md` §1 and the §2 enforcement table (the "approval required" row is now
|
||||
explicitly ⬜ with the reason, rather than a stale ✅), and `infra/gitea/README.md` gains §6 and
|
||||
marks its historical step 4 as reversed — do not re-apply.
|
||||
|
||||
**One adjacent fix, deliberately in scope: `ci/tests/gate-decision.test.sh` was already broken.**
|
||||
It hand-maintains a list of stub stage scripts that "must name EVERY stage `gate.sh` runs", and it
|
||||
did not name `approval-policy` — added to `gate.sh` by PR #17 and never added here. `run_stage`
|
||||
therefore got exit 127 from a missing file and recorded a FAILED stage in every case, so **15 of
|
||||
50 assertions failed**, including every one expecting a PASS/UNDETERMINED headline. Measured on
|
||||
pristine `origin/main` (`0c7e8dd`) before touching it, so the attribution is not a guess.
|
||||
|
||||
I fixed it because `make -C ci test` is the *only* runner for the suite this change adds — leaving
|
||||
it red would have made the new tests unverifiable in practice. The file's own comment already
|
||||
narrated this exact failure for `gitea-settings` and did not prevent the repeat, which says the
|
||||
comment was not the fix. So the stub list is now **derived from `gate.sh`'s own `run_stage` lines**
|
||||
and asserts it parsed at least 8 of them: a stage added to the gate can no longer be silently
|
||||
missing from its test. `make -C ci test` goes from **FAILED to all suites passed**.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **The lock-out is real, and PR #35 is the proof.** Queried live: head sha
|
||||
`8a094c46`, statuses `gate / gate (pull_request) = success :: Successful in 1m41s`,
|
||||
`/pulls/35/reviews` → `[]`, `/branch_protections/main` → `required_approvals: 1`. I read the
|
||||
**rule object**, not `/branches/main`, per §2b.
|
||||
- **`jakman85` is the only collaborator**: `/collaborators` → one entry. Permission `owner`.
|
||||
- **`gitea-admin` also reports `owner`** — this is why the agent-account consequence above is a
|
||||
statement of fact and not a hypothetical.
|
||||
- **PR #35 touches no sensitive paths** (`apps/wave-0/plex.yaml`, `apps/wave-1/{radarr,sonarr}.yaml`,
|
||||
`catalog/{radarr,sonarr}.yaml`, its decisions entry) — so `ci/approval-policy.sh` was never its
|
||||
blocker. Layer 1 alone unblocks it.
|
||||
- **API shapes verified before depending on them**, rather than assumed: `/git/commits/{sha}`
|
||||
returns `.commit.committer.date` as RFC3339 **with an offset** (`2026-10-01T23:09:08-05:00`),
|
||||
not a trailing `Z` — a parser handling only `Z` would have silently dropped every real
|
||||
override, so there is a test for the offset form. PR conversation comments are under
|
||||
`/issues/{n}/comments`; `/pulls/{n}/comments` returns diff comments only and would never have
|
||||
seen an override.
|
||||
- **`ci/tests/approval-override.test.sh`: 23 passed, 0 failed.** Covers the happy path, bare
|
||||
directive, mid-sentence and `>`-quoted directives, stale and equal-timestamp overrides,
|
||||
unparseable timestamps, missing login, invalid JSON, non-list JSON, latest-wins, offset
|
||||
timestamps, and a configurable directive.
|
||||
- `bash -n` clean on `ci/approval-policy.sh`, `infra/gitea/allow-admin-self-merge.sh`,
|
||||
`infra/gitea/require-gate.sh`; `jq -e .` clean on both JSON declarations.
|
||||
- **`gate-decision.test.sh`'s 15 failures are pre-existing, measured not assumed.** Extracted
|
||||
`origin/main`'s `ci/` at `0c7e8dd` into a throwaway directory and ran the suite from there:
|
||||
**FAIL, 15 failed / 35 passed**. After deriving the stub list from `gate.sh`: **PASS, 50
|
||||
assertions**.
|
||||
- **`make -C ci test` in the pinned toolbox: `ci/tests: all suites passed`, rc 0** — approval-override
|
||||
23, gate-decision 50, touched-requirements-base 31, test-attribution 15, touched-requirements 26,
|
||||
decisions-log pass. It was rc 2 (`ci/tests: FAILED`) before the stub-list fix.
|
||||
- **Toolbox merge gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`), run on the
|
||||
rebased branch: **rc 3, `MERGE GATE: UNDETERMINED`**. Green stages: spec-immutability,
|
||||
decisions-log, lint, l0 (23 offline cases, 0 failures), render-golden including the new
|
||||
`catalog/render.py --selftest`, l1 (3 cases), approval-policy (N/A offline). Skipped:
|
||||
`gitea-settings` (no `SETTINGS_API_TOKEN` on this host) and `conformance` (no cluster).
|
||||
**This is NOT a green run and I am not reporting it as one** — rc 3 is UNDETERMINED per the
|
||||
CLAUDE.md §3 contract.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The live Gitea rule still says `required_approvals: 1`. I did not change it.** Mutating live
|
||||
platform settings is an operator action; `infra/gitea/allow-admin-self-merge.sh` is written,
|
||||
syntax-checked, and **unrun**. The declaration and the live rule must move together.
|
||||
*Now measured, not predicted:* PR #38's CI run (run 202, job 0) fails at **1m44s** with
|
||||
`required_approvals: DECLARED 0 but LIVE is 1` → `gitea-settings: FAIL`. Correct behaviour, and
|
||||
the PR stays red until the script is run.
|
||||
- **The override has never been ACCEPTED by a real run** — only offered. Measured in the same CI
|
||||
log: the stage reached `approval-policy: FAIL` on the sensitive paths and printed the *new*
|
||||
code's guidance, `Or, if you hold admin/owner on this repository, comment on the PR:
|
||||
/approve-override <reason> ... NEWER than head commit 7b9f733c`. So the new stage **is** what
|
||||
runs on this PR (self-bootstrapping confirmed by measurement, not by reasoning), and the
|
||||
authorising branch — comment parse → live permission lookup → exit 0 — is still **unexercised**.
|
||||
Posting the comment on PR #38 is its first real test. If it does not work, the recovery is to
|
||||
PATCH `required_approvals` and read run 202's successor log.
|
||||
- **I have not seen the gate still REFUSE a merge** now that approvals are 0. The claim that
|
||||
`block_admin_merge_override: true` keeps a red gate unmergeable rests on Gitea's documented
|
||||
behaviour and on the 2026-09-29 session, **not** on a provoked refusal in this one. That probe
|
||||
is written out at the foot of `allow-admin-self-merge.sh` and should be done, exactly as the
|
||||
push rejection was provoked rather than assumed.
|
||||
- **Clock skew in the staleness check is unhandled.** Staleness compares the comment's
|
||||
server-side `created_at` against the head commit's *committer* date, which comes from the
|
||||
committer's machine. A commit dated in the future would reject legitimate overrides until real
|
||||
time catches up (fail-closed, so loud, but annoying). Gitea's own `dismiss_stale_approvals`
|
||||
compares SHAs instead; a comment cannot. Not hit here — PR #35's commit date is ~21 min behind
|
||||
server time — and not fixed.
|
||||
- **`ci/tests/approval-override.test.sh` does not run in the merge gate.** `ci/gate.sh` runs no
|
||||
`ci/tests/*`; they run only via `make -C ci test`. So a regression in the parser would not be
|
||||
caught by CI. PR #33 (merged while this branch was being written, and rebased onto here) fixed
|
||||
the *same class* of gap for `catalog/render.py --selftest` by invoking it from
|
||||
`ci/render-golden.sh` — it did **not** wire `ci/tests/*` in, which I confirmed after rebasing
|
||||
rather than assuming. Wiring them is a one-line stage and I did not widen scope to it.
|
||||
- **Authorisation itself is untested code.** `ci/approval-override.py` is unit-tested; the live
|
||||
permission lookup, the `override_admins` allowlist branch, and every new BLOCK path in
|
||||
`ci/approval-policy.sh` need Gitea and have never been exercised. The allowlist branch in
|
||||
particular is implemented but **unused** — `override_admins` is absent by design.
|
||||
- **The other open PRs will go red** once the live value changes, until they are rebased onto the
|
||||
merged declaration: as of writing those are **#29** and **#37** (both `gitea-admin`), plus
|
||||
**#35**. I did not rebase them. Merging them first avoids it entirely; the script's header gives
|
||||
the order. PRs #33, #34 and #36 merged while this branch was being written — this branch is
|
||||
rebased onto `0c7e8dd`, so the list was re-measured rather than carried forward.
|
||||
- No cluster state was read or changed; nothing here touches Flux, Kyverno or any workload.
|
||||
@@ -0,0 +1,198 @@
|
||||
---
|
||||
date: 2026-10-01
|
||||
branch: feat/intel-gpu-device-plugin
|
||||
title: Intel GPU device plugin, in its own Kustomization — the resource exists; nothing has run yet
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Deployed the **Intel Device Plugin for Kubernetes (GPU)**, standalone DaemonSet, pinned at
|
||||
**v0.37.0**, so the prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the
|
||||
Plex hardware-transcoding blocker: the `/dev/dri` hostPath merged on `main` @ `cdf3e14` makes the
|
||||
device node *visible* and not *openable* (`EPERM` from the device cgroup at both uid 0 and uid
|
||||
1000; the container's runc `config.json` carries one deny-all rule and `linux.devices: None`). A
|
||||
hostPath is a filesystem operation and adds no cgroup allow rule. A device plugin's `Allocate`
|
||||
response is what makes kubelet populate `linux.devices` *and* the matching cgroup allow rule, and
|
||||
it is the only non-privileged way to get there — `privileged: true` is blocked by Kyverno
|
||||
`restrict-host-namespaces` under Enforce in app namespaces, and `plex` is labelled
|
||||
`app-namespace=true`.
|
||||
|
||||
New files:
|
||||
|
||||
- `platform/intel-gpu-plugin/intel-gpu-plugin.yaml` — the DaemonSet, **vendored** as
|
||||
`kubectl kustomize` output of upstream `deployments/gpu_plugin/base` at `v0.37.0`, under a
|
||||
provenance header carrying the exact re-vendor command and both sha256s (generated body
|
||||
`80ebd070…`, raw upstream source `c8773918…`). Generated rather than byte-copied because
|
||||
upstream mixes YAML sequence-indentation styles within the one document and `ci/yamllint.yaml`
|
||||
sets `indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate.
|
||||
That is not a guess: the first gate run of this branch returned **rc 1 BLOCKED** on exactly that
|
||||
error, and regenerating through kustomize is the fix (and is also the re-vendor idiom
|
||||
`platform/external-snapshotter` documents). Vendored at all, rather than referenced as a remote
|
||||
kustomize base, for the other reason `external-snapshotter` records: a remote base is
|
||||
`git fetch`ed on every reconcile and those fetches have flaked a whole tier.
|
||||
- `platform/intel-gpu-plugin/namespace.yaml` — a PLATFORM namespace, explicitly
|
||||
`homelab.keithjackson.dev/app-namespace: "false"` (following `vpn-egress`/`kyverno`).
|
||||
- `platform/intel-gpu-plugin/kustomization.yaml` — namespace scoping plus the image pin.
|
||||
- `platform/intel-gpu-plugin/README.md` — the measurement, the placement argument, the
|
||||
non-choices, and the post-merge verification sequence.
|
||||
- `clusters/base/intel-gpu-plugin.yaml` — **its own Flux Kustomization**: `dependsOn: [cilium]`,
|
||||
`wait: true`, `interval: 10m`, `timeout: 10m`, `prune: true`.
|
||||
- One wired line (plus its rationale comment) in `clusters/base/kustomization.yaml`.
|
||||
|
||||
**Placement: its own Kustomization, not a line in the shared `platform` tier.** The shared tier was
|
||||
cheaper and needed no human reviewer on `approval-policy.json`'s terms, but `platform` is
|
||||
`wait: true` over eleven-plus components and the repo already argued this case when Cilium was
|
||||
split out: its Ready condition cannot distinguish "the plugin is wedged" from "longhorn is slow",
|
||||
the only emergency valve is freezing the entire tier, and the tier's 10m/5m interval/timeout are
|
||||
called out there as *wrong for a 3-node DaemonSet rollout* — which is exactly what this is. The
|
||||
consequence is worse than average here, because `platform` gates `platform-config` gates
|
||||
`apps-wave-0` (`wait: true`) gates `apps-wave-1`: a plugin that never went healthy inside that tier
|
||||
would stall every app, so a component added to fix a **silent no-op** would have gained the ability
|
||||
to cause a **real outage**. The approval cost of touching `clusters/**` is nil, because
|
||||
`required_approvals: 1` with `block_admin_merge_override: true` applies to every PR here regardless
|
||||
of path and authors cannot self-approve.
|
||||
|
||||
`dependsOn: [cilium]` is the minimum *and* the earliest: the plugin never uses the network and
|
||||
never talks to the API server (it talks to the local kubelet over a unix socket), so a working CNI
|
||||
is its only real prerequisite, and `cilium` already depends on `flux-system`, which applies the
|
||||
`cluster-vars` ConfigMap this Kustomization substitutes from. Depending on `cilium` rather than
|
||||
`platform` also means it reconciles *in parallel with* the platform tier rather than behind all
|
||||
eleven of its components, which shrinks the cold-start window in which a node has not yet
|
||||
advertised the resource.
|
||||
|
||||
`wait: true` is a deliberate departure from the `headlamp`/`crafty` leaf template. Those use
|
||||
`wait: false` because a leaf UI can be legitimately not-Ready for a long time during a cold start,
|
||||
making its health a useless signal. This DaemonSet reads sysfs and opens a socket, so not-Ready
|
||||
genuinely means broken — and since **nothing in the graph `dependsOn` it** (verified against the
|
||||
rendered graph), `wait: true` gates nothing while turning
|
||||
`flux get kustomization intel-gpu-plugin` into a one-line answer to the question that must be
|
||||
answered before the consumer PR may merge.
|
||||
|
||||
Deliberate non-choices, each argued in the README: standalone DaemonSet rather than the operator
|
||||
(no CRD, no controller-manager, no cert-manager webhook, no second namespace to keep healthy);
|
||||
`--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler
|
||||
believes — there is no GPU-side partitioning behind it, so it converts a loud "Unschedulable" into
|
||||
a quiet "transcodes are slow"); `--health-management` left OFF, because it is the one flag that
|
||||
*would* open a device and would reintroduce the exact `EPERM` this fixes; no NFD overlay (all three
|
||||
nodes are identical); no `supplementalGroups` anywhere, because the `pms-docker` cont-init hook
|
||||
already adds the `plex` user to each device's group and that works identically on a
|
||||
plugin-injected device.
|
||||
|
||||
**Scope held deliberately:** `catalog/render.py`, `catalog/plex.yaml` and `apps/wave-0/plex.yaml`
|
||||
are untouched. Switching Plex from the hostPath to a `gpu.intel.com/i915: 1` request is a strictly
|
||||
later PR, and **it must not land until this plugin is merged AND confirmed advertising the resource
|
||||
on all three nodes** — a request for a resource no node advertises makes the Plex pod
|
||||
Unschedulable, and `apps-wave-0` is `wait: true` with `apps-wave-1` depending on it, so that would
|
||||
stall the whole app tier and turn today's silent no-op into a real outage.
|
||||
|
||||
## What I verified
|
||||
|
||||
Mechanism, read from the v0.37.0 source rather than assumed:
|
||||
|
||||
- **The plugin only lists devices; it never opens one.** The scan path uses `os.ReadDir`,
|
||||
`os.ReadFile`, `os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. `grep` for
|
||||
`os.Open|OpenFile|ioctl|syscall|unix\.` across `cmd/gpu_plugin/gpu_plugin.go` (908 lines) returns
|
||||
**nothing**. Listing and `stat()` are what demonstrably work today, so the plugin needs no cgroup
|
||||
grant and no extra privilege — it does not need more than the upstream manifest gives it.
|
||||
- **The fix mechanism is real:** `createDeviceSpecsFromDrmFiles` emits
|
||||
`pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, which is the
|
||||
thing kubelet turns into `linux.devices` plus a cgroup allow rule.
|
||||
- **The resource name is `gpu.intel.com/i915` by derivation, not by hope:** `scan()` calls
|
||||
`devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the kernel driver name, and these
|
||||
nodes have `i915` bound.
|
||||
- `--shared-dev-num` default is `1` (`flag.IntVar(..., 1, ...)`); `--health-management` default is
|
||||
`false` and `setupLevelZeroService` returns early unless it is set.
|
||||
- Missing `/dev/dri/by-path` is non-fatal (`newDevicePlugin` warns, sets `bypathFound = false`).
|
||||
|
||||
Image pin, verified against the registry:
|
||||
|
||||
- `docker.io/intel/intel-gpu-plugin:0.37.0` exists; manifest digest
|
||||
`sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1` (single amd64
|
||||
manifest, not an index), fetched from `registry-1.docker.io` on 2026-10-01. `v0.37.0` is the
|
||||
latest upstream release (2026-09-16) and is not a prerelease.
|
||||
- The Docker Hub tag list carries `0.35.0 / 0.36.0 / 0.37.0` — **image tags have no `v` prefix**
|
||||
while GitHub release tags do.
|
||||
|
||||
Policy scope, read from the policies rather than assumed:
|
||||
|
||||
- All four app-scoped Kyverno policies (`restrict-host-namespaces`, `require-egress-class-label`,
|
||||
`require-auth-mode-label`, `require-default-deny-baseline`) are gated on
|
||||
`namespaceSelector`/`selector` `matchLabels: homelab.keithjackson.dev/app-namespace: "true"`.
|
||||
The new namespace is labelled `"false"`, so none of them match it.
|
||||
- Measured on prod: `plex` **is** `app-namespace=true`, confirming `privileged: true` is genuinely
|
||||
unavailable there; `kyverno` and `vpn-egress` are the two namespaces carrying the label as
|
||||
`"false"`; the other platform namespaces leave it unset. No `intel*` namespace exists, so no
|
||||
collision.
|
||||
|
||||
Cluster before-state (read-only, `kubectl` in the pinned toolbox with the prod kubeconfig
|
||||
bind-mounted read-only — never copied into the worktree):
|
||||
|
||||
- Three nodes `k3s-prod-{1,2,3}`, each `v1.36.4+k3s1`, Debian 13 trixie, `containerd://2.3.4-k3s1.36`.
|
||||
- Every extended resource currently advertised is `devices.kubevirt.io/{kvm,sev,tun,vhost-net}` on
|
||||
all three nodes. **No node advertises `gpu.intel.com/*`** — queried explicitly, returned
|
||||
"NONE". That is the before-state this change is meant to move.
|
||||
- Those `devices.kubevirt.io/*` resources are themselves advertised by a device plugin, which is
|
||||
measured evidence that the device-plugin API and the default kubelet socket path
|
||||
(`/var/lib/kubelet/device-plugins`) work on these nodes.
|
||||
|
||||
Offline rendering (the main proof available to a PR):
|
||||
|
||||
- `kubectl kustomize platform/intel-gpu-plugin` renders two objects: `Namespace/intel-gpu-plugin`
|
||||
carrying `app-namespace=false`, and `DaemonSet/intel-gpu-plugin` in namespace
|
||||
`intel-gpu-plugin`, with the image resolved to `docker.io/intel/intel-gpu-plugin:0.37.0`.
|
||||
- `kubectl kustomize clusters/base` renders, and the new Flux Kustomization appears as
|
||||
`intel-gpu-plugin ./platform/intel-gpu-plugin wait=true dependsOn=cilium`. Both overlays
|
||||
(`clusters/prod`, `clusters/dev`) also render and carry it.
|
||||
- Across the whole rendered graph, **no Kustomization lists `intel-gpu-plugin` in `dependsOn`** —
|
||||
so `wait: true` gates nothing.
|
||||
- The vendored DaemonSet is a **formatting-only** transform of upstream, proven rather than
|
||||
asserted: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file `diff`
|
||||
clean, so passing it through kustomize changed no content. The committed body's sha256 matches
|
||||
the one recorded in its own provenance header (`80ebd070…`), so the file has not drifted from
|
||||
what was generated.
|
||||
|
||||
Gate (pinned toolbox, from this worktree, with the `ENGINE` shim bind-mounting the main checkout's
|
||||
gitdir and `DEV_KUBECONFIG=/dev/null`): see the PR description for the per-stage table. **`rc 3`
|
||||
(UNDETERMINED), with exactly the three expected skips** — `spec-immutability` (sibling spec repo is
|
||||
not beside a worktree), `gitea-settings` (no CI token), `conformance` (no cluster in CI). Every
|
||||
other stage passed. **rc 3 is not green** and is not reported as such.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
**Nothing about this component's runtime behaviour. It has never run.**
|
||||
|
||||
- **The plugin has never executed.** No pod has been scheduled, no image has been pulled, no scan
|
||||
has happened. It cannot run until the operator merges this and Flux reconciles — `kubectl apply`
|
||||
and `flux reconcile` are operator actions (CLAUDE.md §5), so this is expected, not an oversight.
|
||||
- **The resource has never been advertised.** `gpu.intel.com/i915` does not exist on any node and
|
||||
I have not seen it exist. The claim that it *will* be `i915` rather than `xe` is derived from
|
||||
source plus the established fact that `i915` is bound — it is a derivation, not an observation.
|
||||
- **The DaemonSet has never been admitted.** That the Kyverno policies do not match this namespace
|
||||
is read off the policy manifests; no admission request has actually been evaluated. Nor has the
|
||||
`kubectl kustomize` output ever been server-side dry-run against the API — rendering is not
|
||||
admission.
|
||||
- **The `open()` test has NOT been re-run.** The `EPERM` is exactly as it was. Nothing in this PR
|
||||
touches Plex, so Plex is still software-transcoding and will continue to until the later renderer
|
||||
PR lands. The decisive evidence is an `open()` as **uid 1000 through `s6-setuidgid plex`**;
|
||||
passing as uid 0 is weaker and must not be accepted as proof.
|
||||
- **A green gate proves the manifests render, nothing more.** And this gate was not green: it was
|
||||
`rc 3`. Note also that the Gitea `gate` job runs `./ci/gate.sh || [ "$?" -eq 3 ]`
|
||||
(`.gitea/workflows/gate.yml:138`), so a green tick on this PR cannot distinguish rc 0 from rc 3,
|
||||
and `vars.CLUSTER_AVAILABLE` is unset so no L2/L3 conformance runs in CI.
|
||||
- **`seLinuxOptions: type: container_device_plugin_t`** is carried verbatim from upstream. These
|
||||
nodes are Debian 13 and I did not measure whether SELinux is enabled or whether that type exists
|
||||
there. Expected to be inert (runc skips labelling when SELinux is disabled), but unverified — if
|
||||
the pods fail to start with an SELinux error, that line is the first suspect.
|
||||
- **Renovate has not been run** against this pin. The inline-marker idiom is copied from
|
||||
`platform/kubevirt` / `platform/external-snapshotter`, but whether Renovate normalises the
|
||||
`v`-prefix mismatch (release `v0.37.0` vs image `0.37.0`) for this dep is unknown. If it proposes
|
||||
`newTag: v0.38.0`, the fix is an `extractVersion` packageRule in `renovate.json`.
|
||||
- **Resource limits unexamined under load.** The upstream 115Mi/100m limits are taken as-is; I have
|
||||
no measurement of this plugin's footprint on these nodes. Worth a glance given
|
||||
`keycloak-operator` is OOMing at 91% of its limit elsewhere in this cluster.
|
||||
- **`clusters/dev` renders it, and there is no dev cluster** (the operator confirmed the
|
||||
`k3s-dev-*` nodes are gone). That is inert rather than wrong — the unified graph (R-ENV-02) has
|
||||
both overlays render the same base — but it is not a thing I could test.
|
||||
- **Branch protection / required-checks state not re-measured** in this session. CLAUDE.md §2b
|
||||
says re-measure before relying on it; I did not, so treat the approval/gating claims in §2 as
|
||||
unconfirmed-by-me.
|
||||
@@ -0,0 +1,205 @@
|
||||
---
|
||||
date: 2026-10-01
|
||||
branch: fix/render-selftest-runs-in-gate
|
||||
title: catalog/render.py's 106 assertions were dev-time only; the gate now runs them and blocks on them
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Single subject, one file: **`ci/render-golden.sh` now invokes `catalog/render.py --selftest`**
|
||||
and exits 1 when it fails. Nothing else. No renderer change, no new conformance test, no new
|
||||
gate stage.
|
||||
|
||||
This closes the top follow-up of `decisions/2026-09-30-oauth2-proxy-cookie-collision-fleet.md`:
|
||||
|
||||
> Nothing in `ci/` or `.gitea/` runs `catalog/render.py --selftest` — I grepped for it. So the
|
||||
> twenty-odd assertions in that selftest, including the six I just added, are dev-time only: a
|
||||
> future edit that drops either flag *and* re-renders the goldens would pass the entire merge
|
||||
> gate.
|
||||
|
||||
Re-measured here and confirmed: before this commit, `grep -rl -- --selftest` over `ci/` and
|
||||
`.gitea/` returned **nothing** for the renderer. The selftest executes **106** assertions
|
||||
(`check()` invocations counted at runtime via `sys.settrace`, without editing `render.py`; 109
|
||||
static call sites, three of them on failure-only branches). All 106 were unexercised by the
|
||||
merge gate.
|
||||
|
||||
### The lead was WRONG, and here is why
|
||||
|
||||
The brief named `conformance/tests/l0-static/auditability-proofs-producible/check.sh:291` as the
|
||||
one-line fix. That line is real and it does drive selftests:
|
||||
|
||||
for tool in ("record_run.py", "status.py"):
|
||||
r = run([sys.executable, str(LIB / tool), "--selftest"])
|
||||
|
||||
But it is **the wrong hook**. That loop is section (d) of a check that serves **R-SEC-03**, and
|
||||
its subject is the *evidence chain* — can a self-skipped run, or a stale ledger row, become an
|
||||
audit proof? `record_run.py` and `status.py` are **in** that chain; `catalog/render.py` is not.
|
||||
Hanging the renderer there would have made the check's name, its `intent` annotation and its
|
||||
requirement mapping (`R-SEC-03`) all untrue, and a future decision about the renderer could then
|
||||
only be expressed by weakening a *security-auditability* check. That is the precise coupling the
|
||||
cookie-collision entry refused for its own check, and it applies here.
|
||||
|
||||
The second option that entry offered — a new scope-declaring L0 test — is the supported mechanism
|
||||
for a **whole-repo invariant** (`ci/l0.sh` Stage 3b's `--include-scope whole-repo` union, ratcheted
|
||||
by `tests/l0-static/whole-repo-invariants-wired`). It is the right shape for an invariant
|
||||
quantified over the tracked tree. It is the wrong shape for *"a script in this repo must pass its
|
||||
own unit test"*: it would need to declare requirements it does not serve, and it would be two new
|
||||
files of several hundred lines by this tree's conventions for what is a six-line invocation.
|
||||
|
||||
**The honest home is `ci/render-golden.sh`** — the gate stage whose subject *already is*
|
||||
`catalog/render.py`. It already resolves `$PYTHON` and `$RENDER_PY`, already reports rc 3 when
|
||||
either is missing, already exits 1 on renderer error, and `ci/gate.sh` already maps it through
|
||||
`for s in lint l0 render-golden l1` where a FAIL is BLOCKED and a SKIP is UNDETERMINED. The
|
||||
stage's two assertions are independent and both are needed:
|
||||
|
||||
- `render-golden` catches golden-vs-renderer drift, but is satisfied by **any** renderer whose
|
||||
output was re-rendered — including one that just lost a contract invariant and had all ten
|
||||
goldens regenerated in the same commit. That is exactly the hole named in the quote above.
|
||||
- `--selftest` catches that, and is blind to golden drift.
|
||||
|
||||
The file header and the stage name were updated so neither is a lie: the stage is now "the
|
||||
renderer is trustworthy", with two numbered assertions, and the step summary emits its own
|
||||
`- render-selftest:` line so the gate output still NAMES what ran.
|
||||
|
||||
The selftest runs **before** the empty-catalog `exit 3` on purpose: it renders synthetic
|
||||
ServiceDefinitions in memory and is catalog-independent, so an empty catalog must not be able to
|
||||
skip it.
|
||||
|
||||
`catalog/render.py` itself was **not touched** (owned by a concurrent branch this session), and
|
||||
neither was any file under `conformance/`.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **The selftest runs and passes: rc 0, `render.py selftest: PASS`, 106 assertions executed.**
|
||||
Counted at runtime, not from the source: a `sys.settrace` hook counting `check()` calls, in a
|
||||
throwaway script outside the repo, with no edit to `render.py`.
|
||||
- **The stage passes with it wired in**, host run of the single stage:
|
||||
|
||||
[gate] Stage: render==golden reproducibility (canonical render vs committed apps/)
|
||||
[gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
|
||||
[ ok ] render.py --selftest passed
|
||||
[ ok ] render==golden: all 10 onboarded service golden(s) reproduce from the canonical render
|
||||
|
||||
rc **0**. The golden comparison still reports all 10 goldens, so the new invocation did not
|
||||
displace what the stage already asserted.
|
||||
- **IT CAN FAIL — provoked twice, against a mutated COPY of the renderer, never against the
|
||||
tree.** `ci/render-golden.sh` already honours `RENDER_PY`, so each mutant lived in
|
||||
`/tmp/claude-1000/desmond-selftest/` and the repo's `catalog/render.py` was never modified —
|
||||
which matters, because that file belongs to a concurrent branch this session.
|
||||
- **Mutation 1 — the regression this selftest was written for.** Deleted the
|
||||
`f"--cookie-name=_oauth2_proxy_{s['name']}"` emission from `_oauth2_sidecar_container()`,
|
||||
i.e. reverted the fleet-wide cookie-collision fix of PR #28. Stage rc **1**. Note the shape
|
||||
of the failure, because it is a real (minor) weakness and is listed as a follow-up: the
|
||||
selftest died with an uncaught `StopIteration` out of `next(a for a in container["args"] if
|
||||
a.startswith("--cookie-name="))` at `render.py:2641`, rather than printing its collected
|
||||
`SELFTEST FAIL:` lines. It is correctly non-zero and the stage correctly blocks, but the
|
||||
operator gets a traceback instead of the assertion message.
|
||||
- **Mutation 2 — a clean, named assertion failure.** `OAUTH2_COOKIE_REFRESH = "2m30s"` ->
|
||||
`"10m"`, widening the refresh past the measured 300s access-token lifetime. Stage rc **1**:
|
||||
|
||||
[fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
|
||||
[fail] Every golden under apps/ is produced by this renderer, so the goldens below cannot
|
||||
[fail] be trusted even if they reproduce byte-for-byte. Reproduce with:
|
||||
[fail] python3 catalog/render.py --selftest
|
||||
SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). A
|
||||
refresh at or past the access-token lifetime never renews in time, which is the same
|
||||
logout it was added to stop
|
||||
|
||||
This is the mutation that proves the wiring's value most directly: mutation 2 would **not**
|
||||
be caught by `render-golden`'s diff if the goldens were regenerated in the same commit, and
|
||||
it is one of the two flags the cookie-collision entry explicitly recorded as **ungated**.
|
||||
- **THE NON-ZERO PROPAGATES ALL THE WAY TO `gate.sh`'s rc — provoked, in the pinned toolbox, not
|
||||
reasoned about.** `RENDER_PY=/desmond-selftest/render-mutant2.py ./ci/gate.sh`:
|
||||
|
||||
[ ok ] stage spec-immutability: pass
|
||||
[ ok ] stage decisions-log: pass
|
||||
[ ok ] stage lint: pass
|
||||
[ ok ] stage l0: pass
|
||||
[gate] Stage: catalog/render.py --selftest (the renderer's own output-contract assertions)
|
||||
[fail] catalog/render.py --selftest FAILED — the renderer violates its own output contract.
|
||||
SELFTEST FAIL: --cookie-refresh=10m parses to 600.0s, which is not inside (0, 300s). ...
|
||||
[fail] stage render-golden: FAIL
|
||||
[ ok ] stage l1: pass
|
||||
...
|
||||
[fail] render-golden: FAILED -> blocks merge
|
||||
[fail] MERGE GATE: BLOCKED
|
||||
|
||||
`gate.sh` rc **1**. Three links in the chain, each measured: selftest rc 1 -> stage rc 1 ->
|
||||
gate rc 1 / `MERGE GATE: BLOCKED`. This mattered enough to provoke rather than assume: this
|
||||
repo has a documented habit of exit codes getting flattened, and
|
||||
`.gitea/workflows/gate.yml:138` runs `./ci/gate.sh || [ "$?" -eq 3 ]`, which tolerates rc 3 —
|
||||
so a stage that degraded a failure to a skip would be swallowed by CI. rc 1 is not tolerated
|
||||
there, and rc 1 is what this produces.
|
||||
- **`spec-immutability` genuinely RAN and passed, it did not skip.** The two-mount worktree shim
|
||||
plus `-e SPEC_REPO=/home-network-k3s` (the fix recorded in the 2026-09-30 cookie entry) works:
|
||||
`stage spec-immutability: pass`, not SKIPPED. R-TEST-06 is therefore actually covered by these
|
||||
runs.
|
||||
- **No collision with PR #29.** The brief warned that `feat/oauth2-proxy-auth-failure-visible`
|
||||
had added checks under `conformance/tests/l0-static/`. Measured —
|
||||
`git diff origin/main...origin/feat/oauth2-proxy-auth-failure-visible --name-only` — it touches
|
||||
**no `conformance/` path at all**: `decisions/`, `platform/oauth2-proxy/` (a promtool test, an
|
||||
alert rule, its kustomization), `runbooks/`. Disjoint from this branch either way, since this
|
||||
branch touches only `ci/render-golden.sh` and this file.
|
||||
- **Priya's files were not touched.** `catalog/render.py`, `schemas/servicedefinition.schema.json`,
|
||||
`apps/wave-*` and `conformance/tests/l1-rendering/` are all untouched by this branch — confirmed
|
||||
by `git diff --name-only`, which lists exactly two paths.
|
||||
- **`catalog/render.py --selftest` was the LAST unwired selftest in the tree.** Measured:
|
||||
`grep -rl -- --selftest` across the repo finds selftest-capable artifacts in exactly four
|
||||
places, and after this commit every one is driven by the gate — `conformance/lib/record_run.py`
|
||||
and `conformance/lib/status.py` by `auditability-proofs-producible/check.sh:291`, the Keycloak
|
||||
client-secret reconciler (embedded in `platform/keycloak/client-secret-reconciler.yaml`) by
|
||||
`keycloak-client-secret-reconciler/check.sh:633`, and `catalog/render.py` by this change.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Local gate: rc 3 (UNDETERMINED), NOT 0. This is not green and must not be reported as
|
||||
green.** Measured in the pinned toolbox on `eeb6211`:
|
||||
|
||||
[ ok ] spec-immutability: green
|
||||
[ ok ] approval-policy: green
|
||||
[ ok ] decisions-log: green
|
||||
[ ok ] lint: green
|
||||
[ ok ] l0: green
|
||||
[ ok ] render-golden: green
|
||||
[ ok ] l1: green
|
||||
[fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
|
||||
[fail] Stages that did not run: gitea-settings
|
||||
|
||||
Nothing FAILED. The single cause is `gitea-settings` **SKIPPED** — `SETTINGS_API_TOKEN` is not
|
||||
set in my environment, so branch protection and runner registration were not read; CI has that
|
||||
token and the stage is expected to run there. **L2/L3 `conformance`** also skipped
|
||||
(`CLUSTER_AVAILABLE` unset) but is a *declared* gap that `gate.sh` treats as not-applicable
|
||||
rather than undetermined, so it is not what produced the 3. Touched requirements
|
||||
**R-GITOPS-01 R-TEST-02 R-TEST-03 R-TEST-06** are therefore **UNVERIFIED at L2/L3**, all four.
|
||||
The stage this branch changes, `render-golden`, is green **and** was independently proven able
|
||||
to turn the whole gate red (above).
|
||||
- **`approval-policy` passed LOCALLY only because there is no pull_request event**
|
||||
(`not a pull_request event (event='none') — approval policy N/A`). That is not evidence that it
|
||||
passes. This diff touches `ci/**`, a sensitive path, so in CI the stage is expected to **FAIL**
|
||||
until a non-author approving review exists. That is §1 working, fail-closed, not a defect.
|
||||
- **The selftest's own quality is unchanged and ungated.** I wired its *invocation*; I did not
|
||||
review its 106 assertions, and nothing ratchets them. Deleting the body of `_selftest()` would
|
||||
still pass this stage, because an empty selftest returns 0. The repo already has the right
|
||||
answer for that — `conformance/tests/l0-static/keycloak-client-secret-reconciler/mutations.py`
|
||||
injects regressions one at a time and requires `--selftest` to catch every one — and applying
|
||||
that pattern to `catalog/render.py` is the follow-up, deliberately not done here: it is a
|
||||
second subject, it touches `conformance/`, and it would need to live beside the renderer whose
|
||||
file another branch owns this session. **Until it exists, "the selftest runs" is proven and
|
||||
"the selftest is falsifiable" rests on my two hand-run mutations, which are not in the gate.**
|
||||
- **Mutation 1's traceback is not fixed.** `render.py:2641`'s `next(...)` has no default, so the
|
||||
absence of `--cookie-name` raises `StopIteration` and the selftest's collected `SELFTEST FAIL:`
|
||||
lines are never printed. Correct verdict, poor diagnostics, and it suppresses the other
|
||||
assertions that mutation also broke. The one-line fix (`next(..., None)` plus a `check()`) is in
|
||||
`catalog/render.py`, which is another branch's file this session, so it is a follow-up and not
|
||||
a thing I touched.
|
||||
- **Nothing about the live cluster.** No `kubectl` of any kind was run on this branch — not even
|
||||
read-only; this change is offline CI wiring and needed none. No `apply`/`edit`/`patch`, no
|
||||
`flux reconcile`. Merging this deploys nothing: it changes only what the gate refuses.
|
||||
- **CI's own run of this branch.** Quoted in the PR body from the commit's `/statuses`, with
|
||||
descriptions read rather than colours. A *skipped* Gitea job reports `success` with
|
||||
"Has been skipped" and certifies nothing.
|
||||
- **`vars.CLUSTER_AVAILABLE` is still unset in CI**, so even a fully green PR proves the L0/L1
|
||||
offline subset and nothing about runtime behaviour.
|
||||
- **I did not re-measure branch protection, required approvals, or the runner.** Those are
|
||||
CLAUDE.md §2 claims and §2b says re-measure before relying on them; this branch relies on none
|
||||
of them beyond the PR flow itself, and I read the gate's own output rather than the API.
|
||||
+52
-1
@@ -52,7 +52,7 @@ All four steps are complete and each was verified, not assumed:
|
||||
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
|
||||
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
|
||||
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
|
||||
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
|
||||
| 4. Collaborator + approvals | ✅ `jakman85` is owner. `required_approvals` was raised to 1 here and **lowered back to 0 on 2026-10-01** — see §6, it was a lock-out |
|
||||
|
||||
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
|
||||
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
|
||||
@@ -124,6 +124,55 @@ the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS
|
||||
**Changing the declaration is how you change policy.** If the live state is what you now want,
|
||||
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
|
||||
|
||||
### 6. `allow-admin-self-merge.sh` — undoing the lock-out step 4 created (2026-10-01)
|
||||
|
||||
Step 4 above raised `required_approvals` to 1. That was a **lock-out, not a protection**, and the
|
||||
note at the bottom of `require-gate.sh` had already said why: *Gitea will not let an author approve
|
||||
their own PR.* With `jakman85` the only collaborator, there was no account in existence that could
|
||||
approve a PR `jakman85` opened.
|
||||
|
||||
Measured 2026-10-01 — **PR #35**: `gate / gate (pull_request)` **success in 1m41s**, reviews `[]`,
|
||||
`required_approvals: 1`, `block_admin_merge_override: true`. Unmergeable, permanently, with no
|
||||
action available to its author. The same was true of any PR fixing it.
|
||||
|
||||
It was also never the documented design. `ci/approval-policy.sh`'s header and `ci/gate.sh:194`
|
||||
both already describe the model as *"`required_approvals` drops to 0 and the GATE decides"*, and
|
||||
`decisions/2026-09-30-flux-deploy-notification-inhibited.md` flagged the 1-vs-0 disagreement as
|
||||
unresolved and not that session's to settle. So this is a correction toward the stated intent, not
|
||||
a weakening away from it.
|
||||
|
||||
./infra/gitea/allow-admin-self-merge.sh # required_approvals -> 0 (read from the declaration)
|
||||
|
||||
**What stays switched on, and why that is the point.** The other way to unblock an owner is
|
||||
`block_admin_merge_override: false`, and it would have been the wrong one — that re-opens exactly
|
||||
what §5 above closed, letting the one account that merges bypass the gate itself. This script
|
||||
touches the approval **count** only and *verifies* afterwards that `block_admin_merge_override` and
|
||||
`enable_status_check` are both still `true`, failing loudly if the PATCH disturbed either:
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| `required_approvals` | 1 | **0** |
|
||||
| `block_admin_merge_override` | true | true (asserted) |
|
||||
| `enable_status_check` | true | true (asserted) |
|
||||
|
||||
So a **red gate is still unmergeable by everyone, administrators included**. What changed is that
|
||||
a *green* one no longer needs a signature that cannot exist.
|
||||
|
||||
**Security-relevant changes are still reviewed** — by `ci/approval-policy.sh` against
|
||||
`approval-policy.json` `sensitive_paths`, not by this field. An administrator waives that review
|
||||
only by posting `/approve-override <reason>` on the PR: reason mandatory, permission re-checked
|
||||
live, invalidated by a new commit, and recorded by account in the gate summary. Note that the
|
||||
**agent account `gitea-admin` also reports permission `owner`**, so it can clear its own PRs that
|
||||
way; that is an accepted consequence of authorising by permission alone, and the route to
|
||||
narrowing it (an `override_admins` allowlist, already honoured by the code) is written down in
|
||||
`approval-policy.json`.
|
||||
|
||||
**Sequencing.** `ci/gitea-settings.sh` compares live against the declaration *from the branch under
|
||||
test*, and `REQUIRE_GITEA_SETTINGS=true`. So merge any other open PRs **before** running this, then
|
||||
run it, then re-run the gate on the branch carrying the new declaration. Any PR left open across
|
||||
the change declares 1 against a live 0 and goes **red until rebased** — the detector working, not
|
||||
a fault. The script's header spells the order out.
|
||||
|
||||
## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential
|
||||
|
||||
Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
|
||||
@@ -204,6 +253,8 @@ it. The name matters — see the note under the behaviour table above.)
|
||||
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
|
||||
3. **Only then set `enable_status_check: true`** with that exact context.
|
||||
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
|
||||
— **Step 4 was REVERSED on 2026-10-01 (§6).** It assumed a reviewer distinct from the author
|
||||
would exist. One never did, so this step produced a lock-out. Do not re-apply it.
|
||||
|
||||
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
|
||||
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
|
||||
|
||||
Executable
+105
@@ -0,0 +1,105 @@
|
||||
#!/usr/bin/env bash
|
||||
# Let the sole administrator merge their own green PR: required_approvals 1 -> 0 on `main`.
|
||||
#
|
||||
# WHY. `required_approvals: 1` plus Gitea's refusal to let an author approve their own pull
|
||||
# request plus exactly ONE collaborator (`jakman85`, permission `owner`) is a lock-out, not a
|
||||
# policy. Every PR the operator opened was permanently unmergeable, including the PR that would
|
||||
# fix it. Measured 2026-10-01: PR #35 — `gate / gate (pull_request)` success in 1m41s, zero
|
||||
# reviews, `required_approvals: 1`, no account in existence able to satisfy it.
|
||||
#
|
||||
# The approvals field was also never what the design said it was. `ci/approval-policy.sh`'s
|
||||
# header and `ci/gate.sh:194` both already described the model as "required_approvals: 0 and the
|
||||
# GATE decides", and `require-gate.sh`'s own closing note named this exact failure and this exact
|
||||
# remedy. decisions/2026-09-30-flux-deploy-notification-inhibited.md flagged the 1-vs-0
|
||||
# disagreement as unresolved. This resolves it at 0.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* DO, and the distinction is the whole point.
|
||||
#
|
||||
# * It does NOT touch `block_admin_merge_override`, which stays TRUE. The required status check
|
||||
# still binds ABSOLUTELY — a red gate is unmergeable by everyone, administrators included.
|
||||
# Setting `block_admin_merge_override: false` would have been the other way to unblock an
|
||||
# owner, and it would have re-opened precisely what block-admin-override.sh was written to
|
||||
# close: PRs #5-#9 merged with zero approvals while the settings read as enforced.
|
||||
# * It does NOT stop reviewing security-relevant changes. Those are gated by
|
||||
# ci/approval-policy.sh against infra/gitea/approval-policy.json `sensitive_paths`, which an
|
||||
# administrator can waive only with an explicit, attributed `/approve-override <reason>`
|
||||
# comment on the PR.
|
||||
#
|
||||
# ORDER MATTERS — READ THIS BEFORE RUNNING IT.
|
||||
#
|
||||
# ci/gitea-settings.sh fails the merge gate on ANY divergence between the live rule and
|
||||
# infra/gitea/expected-branch-protection.json, and `gate.yml` sets REQUIRE_GITEA_SETTINGS=true.
|
||||
# The declaration in git now says 0. So between running this and merging that declaration, the
|
||||
# live value and the declared value must be compared from a branch that CARRIES the new
|
||||
# declaration — otherwise the stage goes red, correctly, for drift.
|
||||
#
|
||||
# 1. MERGE any other open PRs FIRST, while approvals still work for them (their branches
|
||||
# declare 1, and a PR you did not author can still be approved normally). Skipping this
|
||||
# only costs a rebase later; it does not break anything.
|
||||
# 2. Run this script.
|
||||
# 3. Re-run the gate on the branch that carries the new declaration. `gitea-settings` goes
|
||||
# green because live 0 == declared 0.
|
||||
# 4. Merge that branch.
|
||||
# 5. Any PR still open from before step 2 now declares 1 against a live 0 and will be RED
|
||||
# until it is rebased onto the new main. That is the detector doing its job, not a fault.
|
||||
#
|
||||
# Recoverable: `required_approvals` can be PATCHed back to 1 by an owner at any time. If you do,
|
||||
# change expected-branch-protection.json in the same breath or every PR goes red on drift.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
DECL="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/expected-branch-protection.json"
|
||||
|
||||
command -v jq >/dev/null || { echo "jq is required" >&2; exit 1; }
|
||||
[ -f "$DECL" ] || { echo "declaration missing: ${DECL}" >&2; exit 1; }
|
||||
|
||||
# Take the target from the DECLARATION, not from a literal here. If the two could disagree,
|
||||
# this script would be one more thing re-asserting a value nothing checks — which is the
|
||||
# failure mode the whole of infra/gitea/ exists to stop.
|
||||
WANT="$(jq -r '.protection.required_approvals' "$DECL")"
|
||||
[[ "$WANT" =~ ^[0-9]+$ ]] || { echo "declaration has no numeric required_approvals" >&2; exit 1; }
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
|
||||
echo "authenticating as '${U}' (secret not shown)"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
|
||||
|
||||
# Read the RULE OBJECT, never /branches/main: the branch view does not expose
|
||||
# block_admin_merge_override and showed all-green for hours while the rule was overridable.
|
||||
echo "==> before"
|
||||
api "$B" | jq -r '{required_approvals, block_admin_merge_override, enable_status_check, status_check_contexts}'
|
||||
|
||||
echo "==> setting required_approvals = ${WANT} (declared in expected-branch-protection.json)"
|
||||
api -X PATCH -H 'Content-Type: application/json' \
|
||||
-d "{\"required_approvals\": ${WANT}}" "$B" >/dev/null
|
||||
|
||||
echo "==> after"
|
||||
AFTER_JSON="$(api "$B")"
|
||||
jq -r '{required_approvals, block_admin_merge_override, enable_status_check, status_check_contexts}' <<<"$AFTER_JSON"
|
||||
|
||||
GOT="$(jq -r '.required_approvals' <<<"$AFTER_JSON")"
|
||||
ADMIN_BLOCK="$(jq -r '.block_admin_merge_override' <<<"$AFTER_JSON")"
|
||||
STATUS="$(jq -r '.enable_status_check' <<<"$AFTER_JSON")"
|
||||
|
||||
fail=0
|
||||
[ "$GOT" = "$WANT" ] || { echo "PATCH did not take: required_approvals is ${GOT}, wanted ${WANT}" >&2; fail=1; }
|
||||
# These two are what keeps this change narrow. If the PATCH disturbed either, the gate has
|
||||
# stopped binding and that must be loud, not discovered later.
|
||||
[ "$ADMIN_BLOCK" = true ] || { echo "block_admin_merge_override is ${ADMIN_BLOCK}, MUST be true — the gate is now bypassable" >&2; fail=1; }
|
||||
[ "$STATUS" = true ] || { echo "enable_status_check is ${STATUS}, MUST be true — the gate is no longer required" >&2; fail=1; }
|
||||
[ "$fail" -eq 0 ] || { echo "do NOT record this as applied." >&2; exit 1; }
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
NOW PROVE IT -- the setting is not the evidence. Two things to see, not assume:
|
||||
|
||||
1. An APPROVAL IS NO LONGER NEEDED. On your own PR with a green gate, the merge button is
|
||||
live with zero reviews on it.
|
||||
2. THE GATE STILL BINDS. On a PR whose gate is RED or still pending, Gitea must REFUSE the
|
||||
merge -- for you, as owner. If it lets you through, block_admin_merge_override is not
|
||||
doing its job and this change has traded a lock-out for a bypass.
|
||||
|
||||
Then confirm the detector agrees: run ci/gate.sh from a branch carrying the new
|
||||
expected-branch-protection.json and check that `gitea-settings` is green rather than DRIFT.
|
||||
NOTE
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
|
||||
"_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author. Everything else merges on a green gate alone.",
|
||||
"_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author, OR an admin override (see .override). Everything else merges on a green gate alone.",
|
||||
"_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
|
||||
"sensitive_paths": [
|
||||
"ci/**",
|
||||
@@ -24,5 +24,17 @@
|
||||
"platform/kyverno/**": "the admission policy engine",
|
||||
".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
|
||||
"CLAUDE.md": "the delivery contract agents follow"
|
||||
},
|
||||
"override": {
|
||||
"_why": "This repository has ONE human. Gitea refuses to let an author approve their own pull request, so 'a non-author APPROVED review' is a requirement the sole administrator can never satisfy -- every sensitive-path PR they open was permanently unmergeable, including the PR that would fix it. Measured 2026-10-01: PR #35 green gate, zero reviews, unmergeable. An escape hatch that the administrator can actually reach is therefore REQUIRED for the repository to function; the design question is only whether it leaves a record. This one does.",
|
||||
"enabled": true,
|
||||
"comment_directive": "/approve-override",
|
||||
"requires_reason": true,
|
||||
"authorized_permissions": ["admin", "owner"],
|
||||
"invalidated_by_new_commits": true,
|
||||
"_how": "Comment on the PR with the directive at the START of a line, followed by a reason: '/approve-override <why>'. ci/approval-policy.sh resolves the commenter's LIVE repo permission via /collaborators/{login}/permission and accepts the override only if it is in authorized_permissions. The overrider, the timestamp and the reason are printed in the gate log and the job summary.",
|
||||
"_self_override_is_allowed": "The author may override their own PR if they hold an authorized permission. That is the entire point -- with one human there is no one else -- but it means the 'a change to the gate cannot be gated by the gate' protection is, for an administrator, advisory rather than structural. It is traded for an auditable record: the waiver is an explicit act, attributed, with a stated reason, in the PR's comment history and the gate output.",
|
||||
"_known_consequence_agent_accounts": "ACCEPTED, EYES OPEN (operator decision, 2026-10-01). Authorisation is by live permission ALONE, with no allowlist of human accounts. The agent account `gitea-admin` reports permission `owner` on this repo (measured 2026-10-01), so an AGENT can clear the sensitive-path review on its own PR by posting the directive. Agents author most PRs here, so this is the broad case, not the edge case. To narrow it later without changing any logic, add an `override_admins` allowlist here and require membership in addition to the permission check -- ci/approval-policy.sh reads both from this file.",
|
||||
"_staleness": "An override is dropped unless its comment is NEWER than the PR head commit, mirroring Gitea's own dismiss_stale_approvals. An override earned at commit A therefore does not silently cover commit B; push again and you must override again."
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,9 @@
|
||||
{
|
||||
"_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
|
||||
"_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
|
||||
"_amended": "2026-10-01: required_approvals 1 -> 0. See _why_required_approvals_is_zero.",
|
||||
"_why_required_approvals_is_zero": "The GATE is the approver (operator intent, 2026-09-29). required_approvals: 1 was never consistent with that intent: ci/approval-policy.sh's header and ci/gate.sh:194 both already described the model as 'required_approvals: 0 and the GATE decides', and infra/gitea/require-gate.sh's own closing note named the consequence of 1 -- 'Gitea will not let an author approve their own PR'. With exactly ONE collaborator (jakman85, owner), that made every PR the operator opened permanently unmergeable. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, no path to a merge. decisions/2026-09-30-flux-deploy-notification-inhibited.md had already flagged the 1-vs-0 disagreement as unresolved; this settles it at 0.",
|
||||
"_what_still_holds_at_zero": "This drops the APPROVAL COUNT only. block_admin_merge_override stays TRUE and enable_status_check stays TRUE, so the required gate context still binds absolutely -- nobody, administrator included, can merge a red gate. That distinction is the whole reason this is the chosen fix rather than block_admin_merge_override: false, which would have handed admins a bypass over the gate itself and re-opened what infra/gitea/block-admin-override.sh was written to close (PRs #5-#9, merged with zero approvals). Architecturally and security-relevant paths are still reviewed, by ci/approval-policy.sh rather than by this field.",
|
||||
"branch": "main",
|
||||
"runners_online_min": 1,
|
||||
"protection": {
|
||||
@@ -11,7 +14,7 @@
|
||||
"status_check_contexts": [
|
||||
"gate / gate (pull_request)"
|
||||
],
|
||||
"required_approvals": 1,
|
||||
"required_approvals": 0,
|
||||
"enable_approvals_whitelist": false,
|
||||
"block_admin_merge_override": true,
|
||||
"block_on_rejected_reviews": true,
|
||||
|
||||
@@ -34,7 +34,7 @@ jq -n '{
|
||||
enable_push: false,
|
||||
enable_status_check: true,
|
||||
status_check_contexts: ["gate / gate (pull_request)"],
|
||||
required_approvals: 1,
|
||||
required_approvals: 0,
|
||||
block_on_rejected_reviews: true,
|
||||
dismiss_stale_approvals: true,
|
||||
block_on_outdated_branch: false,
|
||||
@@ -48,11 +48,21 @@ api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
|
||||
* Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
|
||||
* PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
|
||||
agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
|
||||
and rely on the status check alone.
|
||||
NOTE ON required_approvals: 0 -- CHANGED FROM 1 ON 2026-10-01, and this script is the reason
|
||||
the change had to land here too: re-running it with 1 would silently re-impose the lock-out.
|
||||
|
||||
The note that used to sit here called the problem and its own fix: "Gitea will not let an
|
||||
author approve their own PR ... if that is awkward ... set required_approvals back to 0 and
|
||||
rely on the status check alone." It was awkward. With jakman85 the ONLY collaborator, every PR
|
||||
the operator opened was unmergeable -- measured on PR #35, gate green in 1m41s, zero reviews,
|
||||
nothing that could ever satisfy the rule. So: 0, and the status check alone, as written.
|
||||
|
||||
What did NOT change, deliberately:
|
||||
* block_admin_merge_override stays TRUE. This drops the approval COUNT; it does not hand
|
||||
anyone a bypass over the gate. A red gate is still unmergeable by everybody.
|
||||
* Architecturally/security-relevant paths are still human-reviewed -- by ci/approval-policy.sh
|
||||
(sensitive_paths in infra/gitea/approval-policy.json), which an administrator can waive
|
||||
only by posting an explicit, attributed `/approve-override <reason>` comment.
|
||||
|
||||
Then prove it, rather than trusting the API:
|
||||
git commit --allow-empty -m probe && git push origin main # EXPECT: rejected
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
# platform/intel-gpu-plugin
|
||||
|
||||
**Intel Device Plugin for Kubernetes (GPU)** — the standalone GPU plugin DaemonSet from
|
||||
[`intel/intel-device-plugins-for-kubernetes`](https://github.com/intel/intel-device-plugins-for-kubernetes),
|
||||
pinned at **v0.37.0**. It makes every node advertise **`gpu.intel.com/i915`** so a workload can
|
||||
*request* a GPU instead of hoping a `hostPath` is enough.
|
||||
|
||||
- **Serves:** hardware transcoding for Plex (`hardware.gpu: intel-quicksync`), and any later
|
||||
consumer of an Intel iGPU.
|
||||
- **Tier:** its **own** Flux Kustomization
|
||||
([`clusters/base/intel-gpu-plugin.yaml`](../../clusters/base/intel-gpu-plugin.yaml),
|
||||
`dependsOn: cilium`, `wait: true`) — **not** a line in `platform/kustomization.yaml`. See
|
||||
*Placement* below.
|
||||
- **Namespace:** `intel-gpu-plugin` — a PLATFORM namespace, labelled
|
||||
`homelab.keithjackson.dev/app-namespace: "false"`, so the app-scoped Kyverno policies
|
||||
(default-deny generation, auth-mode, egress-class, host-namespaces) do not apply.
|
||||
- **No RBAC, no ServiceAccount, no CRDs, no webhook, no API-server access.** A device plugin talks
|
||||
to the *local kubelet* over a unix socket.
|
||||
|
||||
## What this component owns
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `namespace.yaml` | `intel-gpu-plugin` ns, explicitly `app-namespace=false` |
|
||||
| `intel-gpu-plugin.yaml` | the DaemonSet, **vendored** (machine-generated) from upstream v0.37.0 |
|
||||
| `kustomization.yaml` | namespace scoping + the pinned image (`images:` transformer) |
|
||||
|
||||
## The problem this fixes
|
||||
|
||||
Plex was given a `/dev/dri` hostPath (merged on `main` @ `cdf3e14`). The device is **visible** in
|
||||
the container and **cannot be opened**. Measured in the running pod on `k3s-prod-2`:
|
||||
|
||||
```
|
||||
uid 0, O_RDWR -> EPERM "Operation not permitted"
|
||||
uid 1000, O_RDWR -> EPERM "Operation not permitted"
|
||||
```
|
||||
|
||||
`EPERM` from the **device cgroup**, not `EACCES` from file permissions — proven three ways:
|
||||
|
||||
1. root on the **host** opens the same device with `rc=0`;
|
||||
2. the node is unified `cgroup2fs`;
|
||||
3. the container's runc `config.json` reads
|
||||
`linux.resources.devices: [{'allow': False, 'access': 'rwm'}]` and `linux.devices: None` —
|
||||
one deny-all rule, no allow entry for major 226, and the runtime was never asked to grant a
|
||||
device.
|
||||
|
||||
A hostPath is a **filesystem** operation. It makes the device *node* appear and adds no cgroup
|
||||
allow rule, because Kubernetes cannot infer one from a volume.
|
||||
|
||||
**A device plugin is the mechanism that closes exactly this gap.** Verified in the v0.37.0 source
|
||||
(`cmd/gpu_plugin/gpu_plugin.go`, `createDeviceSpecsFromDrmFiles`), the `Allocate` response carries
|
||||
per DRM node:
|
||||
|
||||
```go
|
||||
specs = append(specs, pluginapi.DeviceSpec{
|
||||
HostPath: devPath, ContainerPath: devPath, Permissions: "rw",
|
||||
})
|
||||
```
|
||||
|
||||
and it is that `DeviceSpec` which makes kubelet populate `linux.devices` **and** the matching
|
||||
cgroup allow rule. It is the only **non-privileged** way to get there. `privileged: true` would
|
||||
also work and is not available: `platform/kyverno/policies/restrict-host-namespaces.yaml` blocks it
|
||||
under Enforce in app namespaces, and `plex` is labelled `app-namespace=true` (measured
|
||||
2026-10-01). Correctly so.
|
||||
|
||||
## Placement — why its own Kustomization
|
||||
|
||||
The cheap option was one line in the shared `platform` tier. Neither that file nor a new
|
||||
`platform/<name>/` directory is a sensitive path in `infra/gitea/approval-policy.json`, so it would
|
||||
have needed no human reviewer *on the policy's terms*. It is still the wrong place, and the repo
|
||||
already argued the case when Cilium was split out (read the comment in
|
||||
`platform/kustomization.yaml`):
|
||||
|
||||
* `platform` is `wait: true` over eleven-plus components, so its Ready condition cannot distinguish
|
||||
"the GPU plugin is wedged" from "longhorn is slow".
|
||||
* The only emergency valve would be `flux suspend kustomization platform` — freezing storage, TLS,
|
||||
DNS, ingress, monitoring, Keycloak and KubeVirt in order to stop touching a device plugin.
|
||||
* The tier's 10m/5m interval/timeout are called out there as **wrong for a 3-node DaemonSet
|
||||
rollout**. This is a 3-node DaemonSet rollout.
|
||||
|
||||
The consequence is worse than average here because of what sits downstream: `platform` gates
|
||||
`platform-config`, which gates `apps-wave-0` (`wait: true`), which gates `apps-wave-1`. A plugin
|
||||
that never went healthy inside that tier would stall **every app** on "dependency not ready" — so a
|
||||
component added to fix a *silent no-op* (software transcoding) would have acquired the ability to
|
||||
cause a *real outage*.
|
||||
|
||||
The usual objection — that `clusters/**` needs a human approval — costs nothing: Gitea enforces
|
||||
`required_approvals: 1` with `block_admin_merge_override: true` on **every** PR here regardless of
|
||||
path, and authors cannot self-approve. The approval was required either way.
|
||||
|
||||
### `wait: true` is a health signal, not a gate
|
||||
|
||||
This departs from the `headlamp`/`crafty` leaf template, deliberately. Those run `wait: false`
|
||||
because a leaf UI can be legitimately not-Ready for a long time during a cold start while it waits
|
||||
on Keycloak/DNS/TLS, making its health useless as a signal. This DaemonSet has no such dependency —
|
||||
it reads sysfs and opens a socket, so on healthy nodes it is Ready in seconds and **not-Ready
|
||||
genuinely means broken**. `wait: true` makes `flux get kustomization intel-gpu-plugin` a one-line
|
||||
answer to "did it come up on all three nodes", which is precisely the question that must be
|
||||
answered before the consumer PR may merge.
|
||||
|
||||
Nothing in the graph `dependsOn` this, so `wait: true` gates nothing. **Think hard before adding
|
||||
`dependsOn: [intel-gpu-plugin]`** anywhere — for the app waves that would hand a device plugin the
|
||||
power to stall every app, which is the thing this placement exists to prevent.
|
||||
|
||||
## Deliberate non-choices
|
||||
|
||||
**Standalone DaemonSet, not the operator.** The operator adds a CRD (`GpuDevicePlugin`), a
|
||||
controller-manager Deployment, cert-manager-backed admission webhooks and a second namespace —
|
||||
three more things that can be unhealthy — to buy declarative management of a DaemonSet that is
|
||||
already declared in git. For three identical nodes and one consumer that is strictly more blast
|
||||
radius for no gain.
|
||||
|
||||
**It lists devices; it never opens them** — worth being certain about, since *opening* is exactly
|
||||
what fails today. Verified in v0.37.0 source: the scan path uses only `os.ReadDir`, `os.ReadFile`,
|
||||
`os.Stat` and `os.Readlink` over `/sys/class/drm` and `/dev/dri`. There is no `os.Open`, no
|
||||
`OpenFile`, no `ioctl` and no raw syscall against a DRM node anywhere in
|
||||
`cmd/gpu_plugin/gpu_plugin.go`. Listing and `stat()` demonstrably work today — that is how the Plex
|
||||
probe found the device before failing to open it — so the plugin needs no cgroup grant of its own
|
||||
and no privilege. Accordingly `/dev/dri` and `/sys/class/drm` are mounted **readOnly**.
|
||||
|
||||
* **`--health-management` stays OFF** (upstream default). It is the one flag that *would* need to
|
||||
open a device: it starts a Level-Zero sidecar (`setupLevelZeroService` returns early unless the
|
||||
flag is set). Enabling it would reintroduce the exact `EPERM` this component exists to fix.
|
||||
* `/dev/dri/by-path` being absent is non-fatal — `newDevicePlugin` warns and sets
|
||||
`bypathFound = false`.
|
||||
|
||||
**`--shared-dev-num` left at the upstream default of 1** (i.e. not set). The flag multiplies each
|
||||
card into N schedulable device IDs, letting N containers share one GPU. Not raised, because:
|
||||
|
||||
* Plex is the only consumer today and wants one device;
|
||||
* the count is a **lie the scheduler believes** — there is no GPU-side partitioning or quota behind
|
||||
it, so N containers get the same `renderD128` and contend for the same two media engines
|
||||
(`vcs0`/`vecs0`). Raising it converts "Unschedulable", which is loud and obvious, into
|
||||
"transcodes are mysteriously slow", which is neither;
|
||||
* it is a one-line, zero-migration change on the day a second consumer exists — and *then* the
|
||||
right number will be known rather than guessed.
|
||||
|
||||
**No NFD overlay.** All three nodes are identical (HP EliteDesk 800 G5 Mini, i7-9700T, UHD Graphics
|
||||
630, PCI `8086:3e98`, `i915` bound), so a nodeSelector on a GPU feature label would gate on
|
||||
something true everywhere. If a GPU-less node ever joins, add upstream's `nfd` overlay *then*
|
||||
rather than relying on the plugin's behaviour on such a node.
|
||||
|
||||
**No `supplementalGroups`, anywhere.** The `pms-docker` image's own cont-init hook stats each
|
||||
`/dev/dri` char device and adds the `plex` user to its group — measured live as
|
||||
`Groups: 44 100 992 1000` on the running Plex process, where GID 992 is the host `render` group.
|
||||
That mechanism is image-side and works identically on a plugin-injected device.
|
||||
|
||||
## Resource name
|
||||
|
||||
`gpu.intel.com/i915`. Not a guess: `scan()` calls `devTree.AddDevice(devProps.driver(), ...)`, so
|
||||
the suffix is the **kernel driver name**. These nodes have `i915` bound (not `xe`), so the resource
|
||||
is `i915`. A future kernel moving CoffeeLake to `xe` would silently rename it to `gpu.intel.com/xe`
|
||||
and a consumer's request would stop matching — worth knowing before that kernel arrives.
|
||||
|
||||
## Versioning
|
||||
|
||||
Pinned to **v0.37.0** via the `images:` transformer in `kustomization.yaml`; the DaemonSet itself is
|
||||
vendored as `kubectl kustomize` output of the upstream base (see its header for the exact re-vendor
|
||||
command and both sha256s). It is *generated* rather than a byte copy because upstream mixes YAML
|
||||
sequence-indentation styles within the one document and `ci/yamllint.yaml` sets
|
||||
`indent-sequences: consistent` — a verbatim copy is a hard lint error that blocks the gate
|
||||
(measured: it did). `kubectl kustomize` normalises the formatting, and the result was proven to be
|
||||
a formatting-only change: `yq -P 'sort_keys(..)'` of the generated file and of the raw upstream file
|
||||
diff clean. This is also the re-vendor idiom `platform/external-snapshotter` documents.
|
||||
|
||||
Vendored rather than referenced as a remote kustomize base for the same reason as
|
||||
`platform/external-snapshotter`: a remote base is `git fetch`ed on **every** reconcile, those
|
||||
fetches intermittently exceeded kustomize-controller's 27s timeout, and transient GitHub slowness
|
||||
then rendered as "kustomize build failed". A local file makes the build offline and deterministic —
|
||||
which also means the CI gate's `kustomize build` proves the manifests rather than proving GitHub
|
||||
was up.
|
||||
|
||||
Renovate tracks it through the inline-marker idiom (`# renovate: datasource=github-releases
|
||||
depName=...` immediately above the version literal), because `renovate.json` sets
|
||||
`"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}` — an image reference in a
|
||||
plain manifest is tracked by nothing. **Caveat:** upstream *release* tags carry a `v` (`v0.37.0`)
|
||||
while *image* tags do not (`0.37.0`). Whether Renovate normalises that for this dep is unverified;
|
||||
if it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
|
||||
`renovate.json`, not a hand-edit.
|
||||
|
||||
## Ordering — do not reorder the stack
|
||||
|
||||
The renderer change (`catalog/render.py` emitting `gpu.intel.com/i915: 1` for
|
||||
`hardware.gpu: intel-quicksync` instead of the hostPath) **must not land until this plugin is
|
||||
merged AND confirmed advertising the resource on all three nodes.** If the renderer requests a
|
||||
resource no node advertises, the Plex pod is **Unschedulable**; `apps-wave-0` is `wait: true` and
|
||||
`apps-wave-1` `dependsOn` it, so that stalls the whole app tier — turning today's silent no-op into
|
||||
a real outage.
|
||||
|
||||
## Post-merge verification
|
||||
|
||||
Nothing about this component's runtime behaviour can be verified from a PR: it does not exist until
|
||||
the operator merges and Flux reconciles. Afterwards:
|
||||
|
||||
```bash
|
||||
D=./infra/devcontainer/dev.sh
|
||||
|
||||
# 1. Flux applied it, and (wait:true) the DaemonSet went healthy.
|
||||
$D run bash -c 'flux get kustomization intel-gpu-plugin'
|
||||
$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin get ds intel-gpu-plugin'
|
||||
# want: DESIRED=3 CURRENT=3 READY=3 AVAILABLE=3
|
||||
|
||||
# 2. The resource is advertised on ALL THREE nodes. This is the gate on the renderer PR.
|
||||
$D run bash -c 'kubectl --context k3s-prod get nodes -o json \
|
||||
| jq -r ".items[] | \"\(.metadata.name) i915=\(.status.allocatable[\"gpu.intel.com/i915\"] // \"ABSENT\")\""'
|
||||
# want: three lines, each i915=1 (before this change: ABSENT on all three)
|
||||
|
||||
# 3. If a node shows ABSENT, read that node's plugin logs.
|
||||
$D run bash -c 'kubectl --context k3s-prod -n intel-gpu-plugin logs -l app=intel-gpu-plugin --tail=50 --prefix'
|
||||
```
|
||||
|
||||
### The decisive test
|
||||
|
||||
Resource-advertised is necessary but **not** sufficient — it proves the plugin registered, not that
|
||||
the cgroup allow rule reaches Plex. The decisive test is re-running the `open()` probe **as uid 1000
|
||||
through `s6-setuidgid plex`** inside the Plex pod, *after* the renderer PR has switched Plex to
|
||||
request the resource:
|
||||
|
||||
```bash
|
||||
$D run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||
s6-setuidgid plex python3 -c "import os; os.close(os.open(\"/dev/dri/renderD128\", os.O_RDWR)); print(\"rc=0 OK\")"'
|
||||
```
|
||||
|
||||
**Passing as uid 0 is weaker evidence and must not be accepted as proof** — Plex transcodes as uid
|
||||
1000, and the two differ in group membership, which is the half the image's cont-init hook supplies.
|
||||
@@ -0,0 +1,117 @@
|
||||
---
|
||||
# VENDORED, MACHINE-GENERATED, from intel/intel-device-plugins-for-kubernetes @ v0.37.0:
|
||||
# deployments/gpu_plugin/base
|
||||
#
|
||||
# RE-VENDOR on a version bump (and bump `newTag` in kustomization.yaml to match):
|
||||
# kubectl kustomize \
|
||||
# "https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/gpu_plugin/base?ref=vX.Y.Z" \
|
||||
# > platform/intel-gpu-plugin/intel-gpu-plugin.yaml
|
||||
# # then re-add this header, and READ THE DIFF — upstream has changed the securityContext
|
||||
# # and the volume set between minor releases before.
|
||||
#
|
||||
# Generated 2026-10-01 at v0.37.0; sha256 of the body below (header excluded):
|
||||
# 80ebd070f983f3116f4004f719d9c872220997ec766b4ab7e7290585ac8ef842
|
||||
# The raw upstream source file reviewed to produce it
|
||||
# (deployments/gpu_plugin/base/intel-gpu-plugin.yaml) was sha256
|
||||
# c8773918c2370e711d4ba84120a9971a290bf75ecf81dc520782bcccd7ef24b6
|
||||
# and the two were proven EQUIVALENT, not merely similar: `yq -P 'sort_keys(..)'` of each diffs
|
||||
# clean, so passing it through kustomize changed formatting only and no content.
|
||||
#
|
||||
# WHY GENERATED RATHER THAN A BYTE COPY of the upstream file: upstream mixes YAML sequence
|
||||
# indentation styles within the one document (`containers:` non-indented, `env:` indented), and
|
||||
# ci/yamllint.yaml sets `indentation: indent-sequences: consistent`, so a verbatim copy is a hard
|
||||
# lint ERROR and blocks the merge gate — measured, this file failed exactly that way first time
|
||||
# round. `kubectl kustomize` normalises the output, which is also precisely the re-vendor idiom
|
||||
# platform/external-snapshotter documents for the same reason.
|
||||
#
|
||||
# WHY VENDORED AT ALL rather than a remote kustomize base (`...gpu_plugin/base?ref=v0.37.0` left in
|
||||
# `resources:`): the same reason platform/external-snapshotter is vendored. A remote base is
|
||||
# `git fetch`ed by Flux's kustomize-controller on EVERY reconcile, those fetches intermittently
|
||||
# exceeded the controller's 27s timeout, and a transient GitHub slowness then rendered as
|
||||
# "kustomize build failed" on the whole tier. A local file makes the build OFFLINE and
|
||||
# deterministic — which also means `kustomize build` in the CI gate proves the real thing instead
|
||||
# of proving GitHub was up. (platform/kubevirt still carries the remote-base pattern and its latent
|
||||
# fragility; not this component's problem.)
|
||||
#
|
||||
# DO NOT HAND-EDIT THE BODY. Every local change belongs in kustomization.yaml as a transformer or
|
||||
# patch, so the next re-vendor stays a mechanical regenerate-and-diff instead of a three-way merge.
|
||||
# The image tag below is re-pinned (and fully qualified) by the `images:` transformer there. The
|
||||
# rationale for each thing deliberately NOT changed — no privileged, no supplementalGroups, no
|
||||
# -shared-dev-num, no NFD overlay, no health-management — is in README.md.
|
||||
apiVersion: apps/v1
|
||||
kind: DaemonSet
|
||||
metadata:
|
||||
labels:
|
||||
app: intel-gpu-plugin
|
||||
name: intel-gpu-plugin
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: intel-gpu-plugin
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: intel-gpu-plugin
|
||||
spec:
|
||||
containers:
|
||||
- env:
|
||||
- name: NODE_NAME
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: spec.nodeName
|
||||
- name: HOST_IP
|
||||
valueFrom:
|
||||
fieldRef:
|
||||
fieldPath: status.hostIP
|
||||
image: intel/intel-gpu-plugin:0.37.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
name: intel-gpu-plugin
|
||||
resources:
|
||||
limits:
|
||||
cpu: 100m
|
||||
memory: 115Mi
|
||||
requests:
|
||||
cpu: 40m
|
||||
memory: 45Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
readOnlyRootFilesystem: true
|
||||
seLinuxOptions:
|
||||
type: container_device_plugin_t
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
volumeMounts:
|
||||
- mountPath: /dev/dri
|
||||
name: devfs
|
||||
readOnly: true
|
||||
- mountPath: /sys/class/drm
|
||||
name: sysfsdrm
|
||||
readOnly: true
|
||||
- mountPath: /var/lib/kubelet/device-plugins
|
||||
name: kubeletsockets
|
||||
- mountPath: /var/run/cdi
|
||||
name: cdipath
|
||||
nodeSelector:
|
||||
kubernetes.io/arch: amd64
|
||||
volumes:
|
||||
- hostPath:
|
||||
path: /dev/dri
|
||||
name: devfs
|
||||
- hostPath:
|
||||
path: /sys/class/drm
|
||||
name: sysfsdrm
|
||||
- hostPath:
|
||||
path: /var/lib/kubelet/device-plugins
|
||||
name: kubeletsockets
|
||||
- hostPath:
|
||||
path: /var/run/cdi
|
||||
type: DirectoryOrCreate
|
||||
name: cdipath
|
||||
updateStrategy:
|
||||
rollingUpdate:
|
||||
maxSurge: 0
|
||||
maxUnavailable: 1
|
||||
type: RollingUpdate
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
# Intel GPU device plugin for Kubernetes — makes the nodes advertise `gpu.intel.com/i915`.
|
||||
#
|
||||
# WHAT THIS FIXES, and why nothing cheaper works. Plex already has a `/dev/dri` hostPath mount
|
||||
# (merged on main @ cdf3e14). The device node is VISIBLE in the container and cannot be OPENED.
|
||||
# Measured in the running pod on k3s-prod-2:
|
||||
# uid 0, O_RDWR -> EPERM "Operation not permitted"
|
||||
# uid 1000, O_RDWR -> EPERM "Operation not permitted"
|
||||
# EPERM from the DEVICE CGROUP, not EACCES from file permissions — proven three ways: root on the
|
||||
# HOST opens the same device rc=0; the node is unified cgroup2fs; and the container's runc
|
||||
# config.json carries one deny-all rule and nothing else:
|
||||
# linux.resources.devices: [{'allow': False, 'access': 'rwm'}]
|
||||
# linux.devices: None
|
||||
# No allow entry for major 226, and the runtime was never asked to grant a device. A hostPath is a
|
||||
# FILESYSTEM operation: it makes the device node appear, and adds no cgroup allow rule, because
|
||||
# Kubernetes cannot infer one from a volume.
|
||||
#
|
||||
# A device plugin is the mechanism that closes exactly that gap. Verified by reading the v0.37.0
|
||||
# source (cmd/gpu_plugin/gpu_plugin.go, createDeviceSpecsFromDrmFiles): the Allocate response
|
||||
# carries `pluginapi.DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node, and it
|
||||
# is that DeviceSpec which makes kubelet populate `linux.devices` AND the matching cgroup allow
|
||||
# rule. It is the only NON-PRIVILEGED way to get there. `privileged: true` would also work and is
|
||||
# not available: platform/kyverno/policies/restrict-host-namespaces.yaml blocks it under Enforce in
|
||||
# app namespaces, and `plex` is labelled app-namespace=true (measured 2026-10-01). Correctly so.
|
||||
#
|
||||
# STANDALONE DAEMONSET, NOT THE OPERATOR. Upstream ships both. The operator adds a CRD
|
||||
# (GpuDevicePlugin), a controller-manager Deployment, cert-manager-backed admission webhooks and a
|
||||
# second namespace — i.e. three more things that can be unhealthy — to buy declarative management
|
||||
# of a DaemonSet we are declaring in git anyway. For three identical nodes and one consumer that is
|
||||
# strictly more blast radius for no gain. The DaemonSet has no CRDs, no controller, no webhook, and
|
||||
# no API-server access at all: a device plugin talks to the LOCAL kubelet over a unix socket, which
|
||||
# is why there is no ServiceAccount or RBAC here (and therefore no `**/rbac.yaml` sensitive path).
|
||||
#
|
||||
# LISTING, NOT OPENING — the thing worth being sure about, since opening is precisely what fails
|
||||
# today. Verified in v0.37.0 source: the scan path uses only os.ReadDir / os.ReadFile / os.Stat /
|
||||
# os.Readlink over /sys/class/drm and /dev/dri. There is no os.Open, no OpenFile, no ioctl and no
|
||||
# syscall of any kind against a DRM node anywhere in cmd/gpu_plugin/gpu_plugin.go. Listing and
|
||||
# stat() demonstrably work today (that is how the Plex probe found the device before failing to
|
||||
# open it), so the plugin needs no cgroup grant of its own and no privilege.
|
||||
# * `--health-management` is left OFF (upstream default). It is the one flag that WOULD need to
|
||||
# open a device: it starts a Level-Zero sidecar (setupLevelZeroService returns early unless the
|
||||
# flag is set). Turning it on would reintroduce the exact EPERM this component exists to fix.
|
||||
# * /dev/dri and /sys/class/drm are mounted readOnly, consistent with list-and-stat only.
|
||||
# * /dev/dri/by-path absence is non-fatal (newDevicePlugin warns and sets bypathFound=false).
|
||||
#
|
||||
# RESOURCE NAME: `gpu.intel.com/i915`. Not a guess — `scan()` does
|
||||
# `devTree.AddDevice(devProps.driver(), ...)`, so the suffix is the KERNEL DRIVER name. These nodes
|
||||
# have i915 bound (not xe) on UHD Graphics 630 / PCI 8086:3e98, so the resource is i915. A future
|
||||
# kernel moving CoffeeLake to xe would silently rename it to `gpu.intel.com/xe` and the consumer's
|
||||
# request would stop matching — worth knowing before that kernel arrives.
|
||||
#
|
||||
# `--shared-dev-num` LEFT AT THE UPSTREAM DEFAULT OF 1 (so: not set). The flag multiplies each
|
||||
# card into N schedulable device IDs (`for i := 0; i < sharedDevNum; i++`), letting N containers
|
||||
# share one GPU. Deliberately not raised:
|
||||
# * Plex is the only consumer and wants one device.
|
||||
# * The count is a LIE the scheduler believes. There is no GPU-side partitioning or quota behind
|
||||
# it — N containers get the same renderD128 and contend for the same two media engines
|
||||
# (vcs0/vecs0). Raising it converts "Unschedulable", which is loud and obvious, into
|
||||
# "transcodes are mysteriously slow", which is neither.
|
||||
# * It is a one-line, zero-migration change on the day a second consumer actually exists, and at
|
||||
# that point the right number will be known rather than guessed.
|
||||
# No NFD overlay either: all three nodes are identical (established cluster fact), so a nodeSelector
|
||||
# on a GPU feature label would gate on something that is true everywhere. If a GPU-less node ever
|
||||
# joins, add upstream's nfd overlay THEN rather than relying on the plugin's behaviour there.
|
||||
#
|
||||
# NO supplementalGroups, anywhere, on purpose. The pms-docker image's own cont-init hook stats each
|
||||
# /dev/dri char device and adds the `plex` user to its group — measured live as `Groups: 44 100 992
|
||||
# 1000` on the running Plex process, where GID 992 is the host `render` group. That mechanism is
|
||||
# image-side and works identically on a plugin-injected device, so adding a supplementalGroup here
|
||||
# or on the consumer would duplicate a thing that already works.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
namespace: intel-gpu-plugin
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- intel-gpu-plugin.yaml
|
||||
images:
|
||||
# PINNED, never `latest`. Upstream release v0.37.0 (2026-09-16); verified present on Docker Hub
|
||||
# 2026-10-01, manifest digest sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1
|
||||
# (single amd64 manifest, not an index). Recorded here rather than pinned by digest because this
|
||||
# repo pins upstream platform components by VERSION TAG (platform/kubevirt, external-snapshotter);
|
||||
# digest pinning is reserved for the conformance runner, where it is a security boundary.
|
||||
# `newName` fully qualifies the registry: upstream writes the bare `intel/intel-gpu-plugin`, which
|
||||
# relies on containerd's docker.io default rather than saying so.
|
||||
#
|
||||
# RENOVATE: the inline-marker idiom (customManagers in renovate.json), not the docker datasource —
|
||||
# renovate.json sets `"docker": {"enabled": false}` and `"kubernetes": {"enabled": false}`, so an
|
||||
# image reference in a plain manifest is tracked by NOTHING. The marker must sit on the line
|
||||
# IMMEDIATELY above the version literal for the regex to match, which is why it is down here on
|
||||
# `newTag` and not up in this header.
|
||||
# CAVEAT, stated because it will bite someone: the GitHub release tags carry a `v`
|
||||
# (`v0.37.0`) and the IMAGE tags do not (`0.37.0` — confirmed against the Docker Hub tag list).
|
||||
# Whether Renovate normalises that for this dep is UNVERIFIED here (no Renovate run available).
|
||||
# If it ever proposes `newTag: v0.38.0`, the fix is an `extractVersion` packageRule in
|
||||
# renovate.json, not a hand-edit of this line.
|
||||
- name: intel/intel-gpu-plugin
|
||||
newName: docker.io/intel/intel-gpu-plugin
|
||||
# renovate: datasource=github-releases depName=intel/intel-device-plugins-for-kubernetes
|
||||
newTag: "0.37.0"
|
||||
@@ -0,0 +1,34 @@
|
||||
---
|
||||
# intel-gpu-plugin — namespace for the Intel GPU device plugin DaemonSet.
|
||||
#
|
||||
# PLATFORM INFRA, NOT AN APPLICATION NAMESPACE, and that distinction is load-bearing here rather
|
||||
# than cosmetic. Every Kyverno policy that could object to this DaemonSet is scoped by a
|
||||
# `namespaceSelector: matchLabels: homelab.keithjackson.dev/app-namespace: "true"` — verified by
|
||||
# reading the policies, not assumed:
|
||||
# * platform/kyverno/policies/restrict-host-namespaces.yaml (hostNetwork/PID/IPC + privileged)
|
||||
# * platform/kyverno/policies/require-egress-class-label.yaml
|
||||
# * platform/kyverno/policies/require-auth-mode-label.yaml
|
||||
# * platform/kyverno/policies/require-default-deny-baseline.yaml (a *generate* rule, so an
|
||||
# app-namespace also gets a default-deny NetworkPolicy minted into it)
|
||||
# Labelling this namespace "false" therefore puts it out of scope for all four. That is the same
|
||||
# mechanism restrict-host-namespaces' own header describes: "platform components that do [need host
|
||||
# access] live in platform namespaces that do NOT carry the app-namespace label".
|
||||
#
|
||||
# Measured on prod 2026-10-01: `kyverno` and `vpn-egress` are the two namespaces that carry the
|
||||
# label explicitly as "false"; the rest of the platform namespaces leave it unset. Both are out of
|
||||
# scope identically — an unset label does not match `"true"`. We set it EXPLICITLY, following
|
||||
# vpn-egress, because "someone chose this" and "nobody thought about it" should not look the same
|
||||
# in git, and because this component's correctness depends on the answer.
|
||||
#
|
||||
# NOTE this plugin does NOT actually need any of the things those policies forbid: it sets no
|
||||
# hostNetwork/hostPID/hostIPC and is not privileged (see intel-gpu-plugin.yaml). The namespace
|
||||
# placement is not a workaround for a policy — there is nothing to work around. It is here because
|
||||
# a node-level platform DaemonSet is not an application, and because the app-namespace fabric
|
||||
# (default-deny NetworkPolicy, egress-class, auth-mode) describes obligations that are meaningless
|
||||
# for a process that talks only to the local kubelet over a unix socket and never uses the network.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: intel-gpu-plugin
|
||||
labels:
|
||||
homelab.keithjackson.dev/app-namespace: "false"
|
||||
Reference in new issue
Block a user