Compare commits
15
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
788eaeae62 | ||
|
|
576e1abb19 | ||
|
|
a1e808e697 | ||
|
|
bcb8931b4f | ||
|
|
0d7f94098b | ||
|
|
5ffd0c98bc | ||
|
|
f0994c11a0 | ||
|
|
b52c8d1af6 | ||
|
|
1677dff568 | ||
|
|
8ccfdab180 | ||
|
|
948875aad3 | ||
|
|
8a094c46f7 | ||
|
|
58f02fe8b1 | ||
|
|
9048c2f5e8 | ||
|
|
753c48be24 |
No files matched your search
@@ -36,6 +36,36 @@ sensitive path, so the policy cannot be widened or narrowed without a human.
|
||||
|
||||
That stage is **fail-closed** — if it cannot evaluate, it blocks. Unlike `gitea-settings`, where a
|
||||
missing credential is a lock-out, here a skip would be a bypass.
|
||||
|
||||
**The administrator override (2026-10-01).** The rule above was unsatisfiable for the one human
|
||||
who runs this repo: Gitea refuses to let an author approve their own PR, and `jakman85` is the
|
||||
only collaborator — so a sensitive-path PR the operator opened could never be merged, *including
|
||||
the PR that would fix that.* An administrator may therefore waive the review by posting this on
|
||||
the PR:
|
||||
|
||||
/approve-override <reason>
|
||||
|
||||
**Post it as a PR *review* of type "Comment", not in the plain comment box.** CI's
|
||||
`SETTINGS_API_TOKEN` is minted `[read:repository, read:admin]`, and `/issues/{n}/comments` sits
|
||||
behind Gitea's *issue* scope — so an override typed into the comment box is **invisible to CI**,
|
||||
and the stage reports it as a missing approval rather than as a permission problem (measured on
|
||||
PR #38, 2026-10-02). `/pulls/{n}/reviews` is readable, so a review body works today. Both sources
|
||||
are accepted and the gate names which it used; the comment box starts working once
|
||||
`infra/gitea/set-ci-token.sh` is re-run to pick up `read:issue`.
|
||||
|
||||
The directive must **start a line**, the **reason is mandatory**, the poster's `admin`/`owner`
|
||||
permission is re-checked **live** when the gate runs, and it must be **newer than the
|
||||
head commit** — push again and you must override again. The overrider, the timestamp and the
|
||||
reason are printed in the gate log and the job summary. For an administrator, "a change to the
|
||||
gate cannot be gated by the gate" is therefore **advisory, not structural**; what is kept is an
|
||||
attributed record of each waiver. Note that the agent account `gitea-admin` also holds `owner`
|
||||
here, so an agent *can* clear this on its own PR — **do not**: posting that directive to unblock
|
||||
your own agent work is exactly the self-approval the stage exists to prevent. Get the operator
|
||||
to post it, or move the change out of the sensitive paths.
|
||||
|
||||
**`required_approvals` on `main` is `0`**, not 1 — the gate is the approver. That drops the
|
||||
approval *count* only: `block_admin_merge_override` and `enable_status_check` both remain `true`,
|
||||
so **a red gate is unmergeable by everyone, administrators included.**
|
||||
- **Do not `kubectl apply` / `edit` / `patch` a durable change.** Flux reconciles `main`;
|
||||
anything applied by hand is drift that the next reconcile reverts or, worse, adopts.
|
||||
|
||||
@@ -53,12 +83,19 @@ Verified against the live Gitea on 2026-09-29:
|
||||
| `main` protected, direct push rejected | ✅ `protected: true`, `user_can_push: false` — **verified by a real rejected push**, not by the API's word |
|
||||
| PR blocked until the gate passes | ✅ `enable_status_check: true`, context `gate / gate (pull_request)` |
|
||||
| Gate runs on every PR | ✅ runner `incluster-dind` online; ~1m35s |
|
||||
| Approval required before merge | ✅ `required_approvals: 1` |
|
||||
| Gate cannot be overridden, by anyone | ✅ `block_admin_merge_override: true` |
|
||||
| Human approval required before merge | ⬜ **No — `required_approvals: 0` since 2026-10-01.** The gate is the approver; sensitive paths are gated by `ci/approval-policy.sh` instead. See §1 |
|
||||
|
||||
All four were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
|
||||
The first three were FALSE as recently as 2026-09-29, while this repo's own log asserted they were
|
||||
true. They are true now because each was measured, and the push rejection was provoked rather
|
||||
than assumed. **Re-measure before relying on any of them** — see §2b.
|
||||
|
||||
The fourth row deliberately changed on 2026-10-01. `required_approvals: 1` was a **lock-out**, not
|
||||
a protection: with a single collaborator and Gitea refusing author self-approval, nothing could
|
||||
satisfy it (measured on PR #35 — green gate, zero reviews, unmergeable). It also contradicted the
|
||||
model already written into `ci/approval-policy.sh` and `ci/gate.sh`. Rationale and the live-change
|
||||
script: `infra/gitea/allow-admin-self-merge.sh`.
|
||||
|
||||
Still not covered: `vars.CLUSTER_AVAILABLE` is unset, so **L2/L3 conformance does not run in
|
||||
CI**. A green PR therefore proves the L0/L1 offline subset, not runtime behaviour.
|
||||
|
||||
|
||||
@@ -170,6 +170,28 @@ spec:
|
||||
port: 32400
|
||||
targetPort: 32400
|
||||
---
|
||||
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-radarr-32400
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: plex-allow-radarr-32400
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: plex
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
app.kubernetes.io/name: radarr
|
||||
k8s:io.kubernetes.pod.namespace: radarr
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '32400'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-seerr-32400
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
@@ -192,6 +214,28 @@ spec:
|
||||
- port: '32400'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): plex-allow-sonarr-32400
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: plex-allow-sonarr-32400
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: plex
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
k8s:io.kubernetes.pod.namespace: sonarr
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '32400'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): plex-allow-gateway-ingress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -283,6 +283,28 @@ spec:
|
||||
- port: '9117'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy egress allow (R-NET-02): radarr-allow-plex-32400
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: radarr-allow-plex-32400
|
||||
namespace: radarr
|
||||
labels:
|
||||
app.kubernetes.io/name: radarr
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: radarr
|
||||
egress:
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
app.kubernetes.io/name: plex
|
||||
k8s:io.kubernetes.pod.namespace: plex
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '32400'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): radarr-allow-seerr-7878
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -281,6 +281,28 @@ spec:
|
||||
- port: '9117'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy egress allow (R-NET-02): sonarr-allow-plex-32400
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: sonarr-allow-plex-32400
|
||||
namespace: sonarr
|
||||
labels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
egress:
|
||||
- toEndpoints:
|
||||
- matchLabels:
|
||||
app.kubernetes.io/name: plex
|
||||
k8s:io.kubernetes.pod.namespace: plex
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '32400'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): sonarr-allow-seerr-8989
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -100,6 +100,9 @@ dependencies:
|
||||
- to: jackett # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
|
||||
port: 9117
|
||||
protocol: tcp
|
||||
- to: plex
|
||||
port: 32400 # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: radarr.${BASE_DOMAIN}
|
||||
port: 7878 # Radarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
||||
|
||||
@@ -100,6 +100,9 @@ dependencies:
|
||||
- to: jackett # arr → indexer proxy (docs/13 dependency graph; R-NET-02 explicit allow)
|
||||
port: 9117
|
||||
protocol: tcp
|
||||
- to: plex # arr -> plex: required since library is on NFS which does not provide file CRUD notifications
|
||||
port: 32400
|
||||
protocol: tcp
|
||||
ingressRoutes:
|
||||
- host: sonarr.${BASE_DOMAIN}
|
||||
port: 8989 # Sonarr web/API; TLS terminates at the shared Gateway (docs/08, docs/09)
|
||||
|
||||
Executable
+194
@@ -0,0 +1,194 @@
|
||||
#!/usr/bin/env python3
|
||||
"""ci/approval-override.py — select the admin-override on a PR, if there is a valid one.
|
||||
|
||||
This script decides NOTHING about authority. It parses a PR's comments and reviews and returns
|
||||
the single override *candidate* that ci/approval-policy.sh should then try to authorise against
|
||||
the live Gitea permission API. Keeping the two apart is the point:
|
||||
|
||||
* parsing is pure, offline, and therefore TESTABLE (ci/tests/approval-override.test.sh);
|
||||
* authorisation needs the API and must stay in the stage that fails closed.
|
||||
|
||||
A candidate is a comment OR a review whose body contains, AT THE START OF A LINE, the directive
|
||||
followed by a non-empty reason:
|
||||
|
||||
/approve-override the gate cannot gate itself and I am the only human reviewer
|
||||
|
||||
TWO SOURCES, AND WHY THAT IS NOT BELT-AND-BRACES. Measured 2026-10-02 on PR #38: the override was
|
||||
posted as an ordinary PR comment, was well-formed, was newer than the head commit — and the stage
|
||||
still blocked, because CI could not READ it. `SETTINGS_API_TOKEN` is minted with scopes
|
||||
`[read:repository, read:admin]`, and Gitea puts /issues/{n}/comments behind the *issue* scope, so
|
||||
the request came back as an error OBJECT rather than an array. `/pulls/{n}/reviews` is under
|
||||
read:repository and was readable in the same run. So:
|
||||
|
||||
* a REVIEW body works with the token CI has today (leave a review of type "Comment");
|
||||
* a COMMENT body works once the token carries read:issue (see infra/gitea/set-ci-token.sh).
|
||||
|
||||
Accepting both means the hatch is not hostage to one API scope. A source that cannot be read
|
||||
contributes no candidates and is reported on stderr; it can only cause a false BLOCK, never a
|
||||
false pass, which is why degrading it is safe and blocking on it was not necessary.
|
||||
|
||||
WHY EACH RULE IS HERE
|
||||
|
||||
* START OF A LINE, not "anywhere in the body". Otherwise quoting someone else's override
|
||||
("you could just /approve-override this") re-arms it. Lines beginning with '>' are DROPPED
|
||||
rather than unquoted, for exactly that reason.
|
||||
* A REASON IS MANDATORY. A bare directive records that a review was waived but not why, which
|
||||
is the difference between an audit trail and a rubber stamp.
|
||||
* STALE OVERRIDES ARE DISCARDED. An override earned at commit A must not silently cover commit
|
||||
B. Any candidate created at or before the head commit's timestamp is dropped — the same
|
||||
semantics Gitea's own `dismiss_stale_approvals` applies to approvals. Push new commits and
|
||||
you must re-override.
|
||||
* A PENDING REVIEW IS NOT SUBMITTED. It is a private draft visible only to its author, so it is
|
||||
not an act of anything and is dropped.
|
||||
* THE LATEST CANDIDATE WINS, across both sources, so a later override supersedes an earlier one.
|
||||
|
||||
Output: one TSV line (login, timestamp, source, reason) on stdout, or nothing at all when there is
|
||||
no valid candidate. Never exits non-zero for "no override" — absence is a normal answer and the
|
||||
caller must not be able to confuse it with a parse crash.
|
||||
|
||||
Input, all via env so nothing lands in `ps`:
|
||||
COMMENTS_JSON the Gitea /issues/{n}/comments array (or an error object; or absent)
|
||||
REVIEWS_JSON the Gitea /pulls/{n}/reviews array (or an error object; or absent)
|
||||
HEAD_TIME ISO-8601 timestamp of the PR head commit
|
||||
DIRECTIVE the override directive, e.g. /approve-override
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
|
||||
def parse_ts(value):
|
||||
"""Gitea emits RFC3339. Return an aware datetime, or None if it is unparseable.
|
||||
|
||||
Unparseable is NOT 'treat as fresh': the caller drops such a candidate, because a
|
||||
timestamp we cannot read is a staleness check we cannot perform.
|
||||
"""
|
||||
if not value:
|
||||
return None
|
||||
text = str(value).strip()
|
||||
# Python < 3.11 cannot read a trailing 'Z' in fromisoformat.
|
||||
if text.endswith(("Z", "z")):
|
||||
text = text[:-1] + "+00:00"
|
||||
try:
|
||||
parsed = datetime.fromisoformat(text)
|
||||
except ValueError:
|
||||
return None
|
||||
if parsed.tzinfo is None:
|
||||
parsed = parsed.replace(tzinfo=timezone.utc)
|
||||
return parsed
|
||||
|
||||
|
||||
def extract_reason(body, directive):
|
||||
"""Return the reason from the first line that STARTS with the directive, else None."""
|
||||
pattern = re.compile(
|
||||
r"^[ \t]*" + re.escape(directive) + r"(?=[ \t:]|$)[ \t:]*(.*)$"
|
||||
)
|
||||
for raw in (body or "").splitlines():
|
||||
# A quoted line is somebody repeating an override, not casting one.
|
||||
if raw.lstrip().startswith(">"):
|
||||
continue
|
||||
match = pattern.match(raw)
|
||||
if match:
|
||||
reason = match.group(1).strip()
|
||||
return reason or None
|
||||
return None
|
||||
|
||||
|
||||
def load_source(env_name, label):
|
||||
"""Return a list of items for one source, or [] with a stderr note if it is unusable."""
|
||||
raw = os.environ.get(env_name)
|
||||
if not raw or not raw.strip():
|
||||
return []
|
||||
try:
|
||||
parsed = json.loads(raw)
|
||||
except (ValueError, TypeError):
|
||||
print(
|
||||
"approval-override: %s is not valid JSON — ignoring that source" % label,
|
||||
file=sys.stderr,
|
||||
)
|
||||
return []
|
||||
if isinstance(parsed, list):
|
||||
return parsed
|
||||
# An error object, typically {"message": "token does not have ... scope"}. Say which,
|
||||
# because "no override found" and "could not read the overrides" look identical from
|
||||
# the outside and the first wasted a real debugging session.
|
||||
detail = ""
|
||||
if isinstance(parsed, dict):
|
||||
detail = str(parsed.get("message") or parsed.get("errors") or "")[:200]
|
||||
print(
|
||||
"approval-override: %s is not a list (Gitea returned %s%s) — ignoring that source"
|
||||
% (label, type(parsed).__name__, ": " + detail if detail else ""),
|
||||
file=sys.stderr,
|
||||
)
|
||||
return []
|
||||
|
||||
|
||||
def candidates_from(items, source, directive, head_time):
|
||||
"""Yield (when, login, reason, raw_when, source) for every valid override in one source."""
|
||||
found = []
|
||||
for item in items:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
login = ((item.get("user") or {}).get("login") or "").strip()
|
||||
reason = extract_reason(item.get("body"), directive)
|
||||
if not login or not reason:
|
||||
continue
|
||||
# A PENDING review is an unsubmitted private draft — not an act of anything.
|
||||
state = str(item.get("state") or "").upper()
|
||||
if source == "review" and state == "PENDING":
|
||||
print(
|
||||
"approval-override: dropping PENDING (unsubmitted) review by '%s'" % login,
|
||||
file=sys.stderr,
|
||||
)
|
||||
continue
|
||||
raw_when = (
|
||||
item.get("submitted_at") or item.get("created_at") or item.get("updated_at") or ""
|
||||
)
|
||||
when = parse_ts(raw_when)
|
||||
if when is None:
|
||||
print(
|
||||
"approval-override: dropping %s override by '%s' — unreadable timestamp %r"
|
||||
% (source, login, raw_when),
|
||||
file=sys.stderr,
|
||||
)
|
||||
continue
|
||||
# An override must post-date the commit it covers.
|
||||
if head_time is not None and when <= head_time:
|
||||
print(
|
||||
"approval-override: dropping STALE %s override by '%s' (%s) — not newer than "
|
||||
"head commit (%s). Re-post it to override the current commit."
|
||||
% (source, login, when.isoformat(), head_time.isoformat()),
|
||||
file=sys.stderr,
|
||||
)
|
||||
continue
|
||||
found.append((when, login, reason, raw_when, source))
|
||||
return found
|
||||
|
||||
|
||||
def main():
|
||||
directive = os.environ.get("DIRECTIVE", "/approve-override").strip()
|
||||
head_time = parse_ts(os.environ.get("HEAD_TIME", ""))
|
||||
|
||||
found = []
|
||||
found += candidates_from(
|
||||
load_source("COMMENTS_JSON", "COMMENTS_JSON"), "comment", directive, head_time
|
||||
)
|
||||
found += candidates_from(
|
||||
load_source("REVIEWS_JSON", "REVIEWS_JSON"), "review", directive, head_time
|
||||
)
|
||||
|
||||
if not found:
|
||||
return 0
|
||||
|
||||
# Latest wins, across both sources: a later override supersedes an earlier one.
|
||||
when, login, reason, raw_when, source = max(found, key=lambda c: c[0])
|
||||
# Keep the reason on one line — this becomes a TSV field.
|
||||
reason = " ".join(reason.split())
|
||||
print("%s\t%s\t%s\t%s" % (login, raw_when, source, reason))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
+161
-4
@@ -25,10 +25,45 @@
|
||||
# true: if this cannot evaluate, an unreviewed change to the gate itself could merge. A skip is a
|
||||
# BYPASS, not a gap. So: cannot determine -> BLOCK.
|
||||
#
|
||||
# THE ADMIN OVERRIDE (added 2026-10-01), and the lock-out it clears.
|
||||
#
|
||||
# The rule above was UNSATISFIABLE for the one human who runs this repo. Gitea refuses to let a
|
||||
# pull request's author approve their own PR, and `jakman85` is the only collaborator — so "an
|
||||
# APPROVED review from someone who is not its author" is a condition the sole administrator can
|
||||
# never meet. Every sensitive-path PR they opened was permanently unmergeable, INCLUDING the PR
|
||||
# that would fix it. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, blocked.
|
||||
#
|
||||
# So an administrator may waive the review by COMMENTING on the PR:
|
||||
#
|
||||
# /approve-override <reason>
|
||||
#
|
||||
# Authority comes from the LIVE Gitea permission API, never from the comment — the commenter must
|
||||
# hold a permission listed in `.override.authorized_permissions` (admin/owner) at the moment the
|
||||
# gate runs, so an old comment from a since-demoted account is worth nothing. The directive must
|
||||
# start a line, a reason is mandatory, and the comment must POST-DATE the head commit: an override
|
||||
# earned at commit A does not cover commit B. Choosing the candidate comment is the job of the
|
||||
# pure, offline, unit-tested ci/approval-override.py; AUTHORISING it is the job of this stage,
|
||||
# which fails closed.
|
||||
#
|
||||
# WHAT THIS COSTS, STATED PLAINLY. For an administrator, "a change to the gate cannot be gated by
|
||||
# the gate" stops being structural and becomes advisory. That is the unavoidable consequence of
|
||||
# there being one human; the alternative is a repository its owner cannot change. What is bought
|
||||
# back is a RECORD — the waiver is an explicit act, attributed to an account, carrying a stated
|
||||
# reason, in the PR history and in this stage's output. A silent bypass keyed on "is the author an
|
||||
# admin?" would have been less code and is exactly the fail-open shape this repo keeps finding.
|
||||
#
|
||||
# AND THE PART THAT IS NOT COMFORTABLE. Authorisation is by permission ALONE, with no allowlist of
|
||||
# human accounts (operator decision, 2026-10-01, taken with this consequence on the table). The
|
||||
# AGENT account `gitea-admin` also reports permission `owner` here, so an agent can clear the
|
||||
# sensitive-path review on its own PR by posting the directive — and agents author most PRs in
|
||||
# this repo, so that is the common case, not an edge case. To narrow it without touching this
|
||||
# logic, add an `override_admins` list to infra/gitea/approval-policy.json: the check below
|
||||
# already requires membership whenever that key is present.
|
||||
#
|
||||
# EXIT CODES
|
||||
# 0 no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and a
|
||||
# non-author APPROVED review is present
|
||||
# 1 sensitive paths touched without such a review, OR the policy could not be evaluated
|
||||
# 0 no sensitive paths touched (gate alone is sufficient), OR sensitive paths touched and
|
||||
# either a non-author APPROVED review or a valid admin override is present
|
||||
# 1 sensitive paths touched with neither, OR the policy could not be evaluated
|
||||
set -euo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${CI_DIR}/.." && pwd)"
|
||||
@@ -65,6 +100,17 @@ api() { curl -sS --max-time 20 -H "Authorization: token ${TOKEN}" "$@"; }
|
||||
FILES_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/files?limit=1000" || true)"
|
||||
PR_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}" || true)"
|
||||
REVIEWS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/pulls/${PR}/reviews" || true)"
|
||||
# PR conversation comments live under the ISSUES endpoint. `/pulls/{n}/comments` returns review
|
||||
# (diff) comments ONLY, so an override posted in the normal comment box would never appear in it.
|
||||
# Verified against this Gitea (1.24.3) rather than assumed.
|
||||
#
|
||||
# This one is NOT in the must-parse loop below, and that is deliberate. /issues/{n}/comments sits
|
||||
# behind Gitea's *issue* token scope, which `SETTINGS_API_TOKEN` does not carry (it is minted
|
||||
# `[read:repository, read:admin]`) — so it returns an error OBJECT, not an array. Blocking on that
|
||||
# turned "CI cannot read your override" into a message that said "you did not post one", and cost a
|
||||
# real debugging session on PR #38. It is handled explicitly, and loudly, further down: an
|
||||
# unreadable source can only cause a false BLOCK, never a false pass.
|
||||
COMMENTS_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/issues/${PR}/comments?limit=1000" || true)"
|
||||
for v in FILES_JSON PR_JSON REVIEWS_JSON; do
|
||||
jq -e . >/dev/null 2>&1 <<<"${!v}" || {
|
||||
err "could not read ${v} from Gitea for PR #${PR} — BLOCKING rather than assuming."
|
||||
@@ -107,11 +153,122 @@ if [ -n "$APPROVER" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────
|
||||
# No non-author approval exists. Before refusing, look for an administrator's explicit
|
||||
# override. See the header for why this hatch exists and what it costs.
|
||||
# ─────────────────────────────────────────────────────────────────────────────────────────
|
||||
OVERRIDE_ENABLED="$(jq -r '.override.enabled // false' "$DECL")"
|
||||
DIRECTIVE="$(jq -r '.override.comment_directive // "/approve-override"' "$DECL")"
|
||||
HEAD_SHA="$(jq -r '.head.sha // empty' <<<"$PR_JSON")"
|
||||
|
||||
if [ "$OVERRIDE_ENABLED" = "true" ]; then
|
||||
# The head commit's timestamp anchors staleness. Read it from GITEA, not from the local
|
||||
# tree: CI checks out a merge commit, whose date is not the head commit's date.
|
||||
HEAD_TIME=""
|
||||
if [ -n "$HEAD_SHA" ]; then
|
||||
HEAD_COMMIT_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/git/commits/${HEAD_SHA}" || true)"
|
||||
HEAD_TIME="$(jq -r '.commit.committer.date // .created // empty' <<<"${HEAD_COMMIT_JSON:-null}" 2>/dev/null || true)"
|
||||
fi
|
||||
if [ -z "$HEAD_TIME" ]; then
|
||||
err "could not read the head commit's timestamp for PR #${PR} — BLOCKING."
|
||||
err " Without it an override cannot be aged, and an unaged override is one that silently"
|
||||
err " covers commits nobody waived review on."
|
||||
summary "- approval-policy: **BLOCKED — head commit timestamp unreadable** (cannot age an override)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Is the comments source actually readable? Say so plainly if not — a scope problem and "you
|
||||
# never posted an override" are indistinguishable from the outside, and conflating them is what
|
||||
# made PR #38 look like a code bug.
|
||||
if ! jq -e 'type == "array"' >/dev/null 2>&1 <<<"${COMMENTS_JSON:-}"; then
|
||||
SCOPE_MSG="$(jq -r '.message // empty' <<<"${COMMENTS_JSON:-null}" 2>/dev/null || true)"
|
||||
warn "cannot read PR COMMENTS from Gitea${SCOPE_MSG:+ — ${SCOPE_MSG}}."
|
||||
warn " /issues/${PR}/comments needs the *issue* token scope; SETTINGS_API_TOKEN is minted"
|
||||
warn " [read:repository, read:admin] (infra/gitea/set-ci-token.sh). An override posted as an"
|
||||
warn " ordinary PR COMMENT is therefore invisible to CI — post it as a REVIEW of type"
|
||||
warn " 'Comment' instead, which /pulls/${PR}/reviews exposes under read:repository."
|
||||
summary "- approval-policy: note — PR comments unreadable (token lacks the issue scope); use a **review** to override"
|
||||
COMMENTS_JSON='[]'
|
||||
fi
|
||||
|
||||
# Pure, offline, unit-tested selection: which comment or review, if any, is a live override?
|
||||
OVERRIDE_TSV="$(COMMENTS_JSON="$COMMENTS_JSON" REVIEWS_JSON="$REVIEWS_JSON" \
|
||||
HEAD_TIME="$HEAD_TIME" DIRECTIVE="$DIRECTIVE" \
|
||||
python3 "${CI_DIR}/approval-override.py")" || {
|
||||
err "ci/approval-override.py failed — BLOCKING rather than assuming there is no override."
|
||||
summary "- approval-policy: **BLOCKED — override parser failed**"
|
||||
exit 1; }
|
||||
|
||||
if [ -n "$OVERRIDE_TSV" ]; then
|
||||
OV_USER="$(cut -f1 <<<"$OVERRIDE_TSV")"
|
||||
OV_WHEN="$(cut -f2 <<<"$OVERRIDE_TSV")"
|
||||
OV_SRC="$(cut -f3 <<<"$OVERRIDE_TSV")"
|
||||
OV_WHY="$(cut -f4- <<<"$OVERRIDE_TSV")"
|
||||
|
||||
# AUTHORITY IS MEASURED LIVE, never inferred from the comment or from a name in git.
|
||||
PERM_JSON="$(api "${GITEA_URL}/api/v1/repos/${GITEA_REPO}/collaborators/${OV_USER}/permission" || true)"
|
||||
OV_PERM="$(jq -r '.permission // empty' <<<"${PERM_JSON:-null}" 2>/dev/null || true)"
|
||||
if [ -z "$OV_PERM" ]; then
|
||||
err "'${OV_USER}' posted ${DIRECTIVE}, but their permission on ${GITEA_REPO} could not be"
|
||||
err " read — BLOCKING. An override whose authority is unverified is not an override."
|
||||
summary "- approval-policy: **BLOCKED — cannot verify \`${OV_USER}\`'s permission**"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Authorised when the LIVE permission is one of the declared ones and — only if an
|
||||
# `override_admins` allowlist exists at all — the account is on it. No allowlist means
|
||||
# permission alone, which is the operator's standing decision; adding the key narrows
|
||||
# this with no change to any code.
|
||||
AUTHORISED="$(DECL="$DECL" OV_PERM="$OV_PERM" OV_USER="$OV_USER" python3 -c '
|
||||
import json, os
|
||||
d = json.load(open(os.environ["DECL"])).get("override", {})
|
||||
perms = d.get("authorized_permissions") or ["admin", "owner"]
|
||||
allow = d.get("override_admins")
|
||||
ok_perm = os.environ["OV_PERM"] in perms
|
||||
ok_list = True if allow is None else os.environ["OV_USER"] in allow
|
||||
print("yes" if (ok_perm and ok_list) else "no")
|
||||
')"
|
||||
|
||||
if [ "$AUTHORISED" = "yes" ]; then
|
||||
if [ "$OV_USER" = "$AUTHOR" ]; then
|
||||
warn "ADMIN OVERRIDE — SELF-WAIVED. '${OV_USER}' authored this PR and waived the"
|
||||
warn " non-author review requirement on their own change. Permitted (permission"
|
||||
warn " '${OV_PERM}'), deliberate, and shouted about here precisely because it is not free."
|
||||
else
|
||||
warn "ADMIN OVERRIDE by '${OV_USER}' (permission '${OV_PERM}'); author is '${AUTHOR}'."
|
||||
fi
|
||||
warn " via: PR ${OV_SRC}"
|
||||
warn " when: ${OV_WHEN} (head commit ${HEAD_SHA:0:8}, ${HEAD_TIME})"
|
||||
warn " reason: ${OV_WHY}"
|
||||
warn " review waived on: $(printf '%s ' $SENSITIVE)"
|
||||
ok "sensitive paths cleared by administrator override — this stage does not block."
|
||||
SELFNOTE=""; [ "$OV_USER" = "$AUTHOR" ] && SELFNOTE=", **self-waived as author**"
|
||||
summary "- approval-policy: **OVERRIDDEN by \`${OV_USER}\`** (permission \`${OV_PERM}\`, via PR ${OV_SRC}${SELFNOTE})"
|
||||
summary " - reason: ${OV_WHY}"
|
||||
summary " - review waived on: $(printf '\`%s\` ' $SENSITIVE)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
err "'${OV_USER}' posted ${DIRECTIVE} but is NOT authorised to override: live permission is"
|
||||
err " '${OV_PERM}'. The authorised set is declared in infra/gitea/approval-policy.json."
|
||||
summary "- approval-policy: **BLOCKED — \`${OV_USER}\` may not override** (permission \`${OV_PERM}\`)"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
err "This PR changes the pipeline or the security posture it rests on, and has NO approving"
|
||||
err " review from anyone other than its author ('${AUTHOR}')."
|
||||
err " A change to the gate cannot be gated by the gate — that is the one thing CI cannot"
|
||||
err " honestly approve for itself. Get a human review, or move the change out of:"
|
||||
printf ' %s\n' $SENSITIVE >&2
|
||||
err " The list lives in infra/gitea/approval-policy.json, with a reason per entry."
|
||||
summary "- approval-policy: **BLOCKED — sensitive paths need a non-author approval**"
|
||||
if [ "$OVERRIDE_ENABLED" = "true" ]; then
|
||||
err " Or, if you hold admin/owner on this repository, post this on the PR:"
|
||||
err " ${DIRECTIVE} <reason>"
|
||||
err " as a REVIEW of type 'Comment' (not the plain comment box — CI's token cannot read"
|
||||
err " /issues/${PR}/comments without the issue scope; see infra/gitea/set-ci-token.sh)."
|
||||
err " The reason is mandatory, the directive must START a line, and it must be NEWER than"
|
||||
err " head commit ${HEAD_SHA:0:8} — push again and you must override again."
|
||||
fi
|
||||
summary "- approval-policy: **BLOCKED — needs a non-author approval, or an admin \`${DIRECTIVE}\` comment**"
|
||||
exit 1
|
||||
Executable
+232
@@ -0,0 +1,232 @@
|
||||
#!/usr/bin/env bash
|
||||
# ci/tests/approval-override.test.sh — proves ci/approval-override.py selects the override it
|
||||
# claims to select, and REJECTS the ones it claims to reject.
|
||||
#
|
||||
# This parser decides whether a human review requirement gets waived. Every rule in it exists
|
||||
# because the obvious implementation ("does the body contain the directive?") is wrong in a way
|
||||
# that is invisible until abused: a quoted directive would re-arm an override, a bare directive
|
||||
# would waive review with no stated reason, and an override posted against commit A would go on
|
||||
# covering commit B forever. A guard is worth nothing until somebody has watched it refuse, which
|
||||
# is the same reasoning as ci/tests/decisions-log.test.sh one layer up.
|
||||
#
|
||||
# The parser is pure and offline BY DESIGN — authorisation (the live permission lookup) lives in
|
||||
# ci/approval-policy.sh, which cannot be tested without Gitea. That split is what makes this file
|
||||
# possible, and it is also why a green run here does NOT prove the stage authorises correctly.
|
||||
set -uo pipefail
|
||||
CI_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
SUT="${CI_DIR}/approval-override.py"
|
||||
|
||||
PASS=0 FAIL=0
|
||||
ok() { printf ' ok %s\n' "$*"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$*"; FAIL=$((FAIL+1)); }
|
||||
|
||||
HEAD_TIME="2026-10-01T12:00:00Z"
|
||||
AFTER="2026-10-01T12:30:00Z"
|
||||
BEFORE="2026-10-01T11:30:00Z"
|
||||
|
||||
# run <comments-json> -> stdout of the parser (stderr is diagnostic only)
|
||||
run() {
|
||||
COMMENTS_JSON="$1" REVIEWS_JSON="" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
|
||||
python3 "$SUT" 2>/dev/null
|
||||
}
|
||||
|
||||
# run_rev <reviews-json> — the same, but through the REVIEWS source.
|
||||
run_rev() {
|
||||
COMMENTS_JSON="" REVIEWS_JSON="$1" HEAD_TIME="${2:-$HEAD_TIME}" DIRECTIVE="${3:-/approve-override}" \
|
||||
python3 "$SUT" 2>/dev/null
|
||||
}
|
||||
|
||||
# comment <login> <created_at> <body>
|
||||
comment() {
|
||||
python3 -c '
|
||||
import json, sys
|
||||
print(json.dumps([{"user": {"login": sys.argv[1]}, "created_at": sys.argv[2], "body": sys.argv[3]}]))
|
||||
' "$1" "$2" "$3"
|
||||
}
|
||||
|
||||
expect_none() {
|
||||
local label="$1" out
|
||||
out="$(run "$2")"
|
||||
if [ -z "$out" ]; then ok "$label"; else bad "$label (got: ${out})"; fi
|
||||
}
|
||||
|
||||
expect_user() {
|
||||
local label="$1" json="$2" want="$3" out got
|
||||
out="$(run "$json")"
|
||||
got="$(cut -f1 <<<"$out")"
|
||||
if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted ${want}, got '${got}')"; fi
|
||||
}
|
||||
|
||||
expect_reason() {
|
||||
local label="$1" json="$2" want="$3" out got
|
||||
out="$(run "$json")"
|
||||
got="$(cut -f4- <<<"$out")"
|
||||
if [ "$got" = "$want" ]; then ok "$label"; else bad "$label (wanted '${want}', got '${got}')"; fi
|
||||
}
|
||||
|
||||
# check <label> <actual> <wanted>
|
||||
check() {
|
||||
if [ "$2" = "$3" ]; then ok "$1"; else bad "$1 (wanted '${3}', got '${2}')"; fi
|
||||
}
|
||||
|
||||
# review <login> <submitted_at> <state> <body>
|
||||
review() {
|
||||
python3 -c '
|
||||
import json, sys
|
||||
print(json.dumps([{"user": {"login": sys.argv[1]}, "submitted_at": sys.argv[2],
|
||||
"state": sys.argv[3], "body": sys.argv[4]}]))
|
||||
' "$1" "$2" "$3" "$4"
|
||||
}
|
||||
|
||||
echo "== ci/approval-override.py"
|
||||
|
||||
# --- the happy path -------------------------------------------------------------------
|
||||
expect_user "a directive with a reason, posted after the head commit, is selected" \
|
||||
"$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" jakman85
|
||||
|
||||
expect_reason "the reason is returned verbatim" \
|
||||
"$(comment jakman85 "$AFTER" '/approve-override sole admin, gate is green')" \
|
||||
'sole admin, gate is green'
|
||||
|
||||
# --- absence is a normal answer, not an error -----------------------------------------
|
||||
expect_none "no comments at all" '[]'
|
||||
expect_none "an ordinary comment" "$(comment jakman85 "$AFTER" 'looks good to me')"
|
||||
|
||||
# --- a reason is MANDATORY ------------------------------------------------------------
|
||||
expect_none "a bare directive (no reason) is refused" \
|
||||
"$(comment jakman85 "$AFTER" '/approve-override')"
|
||||
expect_none "a directive followed by only whitespace is refused" \
|
||||
"$(comment jakman85 "$AFTER" '/approve-override ')"
|
||||
|
||||
# --- the directive must START a line --------------------------------------------------
|
||||
expect_none "the directive mid-sentence does not count" \
|
||||
"$(comment jakman85 "$AFTER" 'you could just /approve-override this one')"
|
||||
expect_none "a QUOTED directive does not re-arm an override" \
|
||||
"$(comment jakman85 "$AFTER" '> /approve-override quoting someone else')"
|
||||
expect_user "leading whitespace before the directive is fine" \
|
||||
"$(comment jakman85 "$AFTER" ' /approve-override indented but mine')" jakman85
|
||||
expect_user "the directive on a later line of a longer comment counts" \
|
||||
"$(comment jakman85 "$AFTER" 'some preamble
|
||||
/approve-override the real reason here')" jakman85
|
||||
|
||||
# --- staleness: an override must post-date the commit it waives review on -------------
|
||||
expect_none "an override OLDER than the head commit is dropped as stale" \
|
||||
"$(comment jakman85 "$BEFORE" '/approve-override stale, predates the push')"
|
||||
expect_none "an override exactly AT the head commit time is dropped (strictly newer)" \
|
||||
"$(comment jakman85 "$HEAD_TIME" '/approve-override same instant')"
|
||||
|
||||
# --- malformed input is never a free pass --------------------------------------------
|
||||
expect_none "an unreadable timestamp drops the candidate" \
|
||||
"$(comment jakman85 'not-a-date' '/approve-override bad clock')"
|
||||
expect_none "a comment with no author is dropped" \
|
||||
'[{"user": {}, "created_at": "2026-10-01T12:30:00Z", "body": "/approve-override no login"}]'
|
||||
expect_none "invalid JSON yields no override (and never a traceback into stdout)" \
|
||||
'this is not json'
|
||||
expect_none "a JSON object instead of a list yields no override" \
|
||||
'{"user": "nope"}'
|
||||
|
||||
# --- latest wins ----------------------------------------------------------------------
|
||||
TWO='[
|
||||
{"user": {"login": "early"}, "created_at": "2026-10-01T12:10:00Z", "body": "/approve-override first"},
|
||||
{"user": {"login": "latest"}, "created_at": "2026-10-01T12:40:00Z", "body": "/approve-override second"}
|
||||
]'
|
||||
expect_user "the LATEST override supersedes an earlier one" "$TWO" latest
|
||||
expect_reason "and brings its own reason" "$TWO" second
|
||||
|
||||
# --- offsets, not just Z --------------------------------------------------------------
|
||||
# Gitea emits RFC3339 WITH AN OFFSET (measured: 2026-10-01T23:09:08-05:00), not a trailing Z.
|
||||
# A parser that only handles 'Z' silently drops every real override, so this case is load-bearing.
|
||||
expect_user "an RFC3339 offset timestamp (-05:00), as Gitea actually emits, parses" \
|
||||
'[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T08:30:00-05:00", "body": "/approve-override offset form"}]' \
|
||||
jakman85
|
||||
expect_none "and an offset timestamp that is genuinely earlier is still stale" \
|
||||
'[{"user": {"login": "jakman85"}, "created_at": "2026-10-01T06:00:00-05:00", "body": "/approve-override offset, stale"}]'
|
||||
|
||||
# --- the directive is configurable ----------------------------------------------------
|
||||
out="$(run "$(comment jakman85 "$AFTER" '/admin-waive because policy says so')" "$HEAD_TIME" '/admin-waive')"
|
||||
if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
|
||||
ok "a custom directive from the declaration is honoured"
|
||||
else
|
||||
bad "a custom directive from the declaration is honoured (got '${out}')"
|
||||
fi
|
||||
expect_none "the default directive does not match a custom-directive comment" \
|
||||
"$(comment jakman85 "$AFTER" '/admin-waive because policy says so')"
|
||||
|
||||
# --- no staleness anchor: do not invent one -------------------------------------------
|
||||
out="$(COMMENTS_JSON="$(comment jakman85 "$BEFORE" '/approve-override no anchor')" \
|
||||
HEAD_TIME="" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
||||
if [ "$(cut -f1 <<<"$out")" = "jakman85" ]; then
|
||||
ok "with no HEAD_TIME the staleness test is skipped, not faked (caller blocks instead)"
|
||||
else
|
||||
bad "with no HEAD_TIME the candidate should still be returned (got '${out}')"
|
||||
fi
|
||||
|
||||
# --- the REVIEWS source ---------------------------------------------------------------
|
||||
# Load-bearing, not redundant. Measured on PR #38: SETTINGS_API_TOKEN is minted
|
||||
# [read:repository, read:admin], /issues/{n}/comments needs the ISSUE scope, so a well-formed
|
||||
# override in the comment box came back as an error object and was invisible. /pulls/{n}/reviews
|
||||
# IS readable with that token, so a review body is the route that works today.
|
||||
check "a review body carries an override" \
|
||||
"$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
|
||||
jakman85
|
||||
check "the source is reported as 'review'" \
|
||||
"$(cut -f3 <<<"$(run_rev "$(review jakman85 "$AFTER" COMMENT '/approve-override via a review')")")" \
|
||||
review
|
||||
check "the source is reported as 'comment' for a comment" \
|
||||
"$(cut -f3 <<<"$(run "$(comment jakman85 "$AFTER" '/approve-override via a comment')")")" \
|
||||
comment
|
||||
|
||||
out="$(run_rev "$(review jakman85 "$BEFORE" COMMENT '/approve-override stale review')")"
|
||||
check "a stale review override is dropped too" "${out:-<none>}" "<none>"
|
||||
|
||||
out="$(run_rev "$(review jakman85 "$AFTER" PENDING '/approve-override unsubmitted draft')")"
|
||||
check "a PENDING (unsubmitted) review is NOT an act of anything" "${out:-<none>}" "<none>"
|
||||
|
||||
check "an APPROVED review carrying the directive still counts as an override" \
|
||||
"$(cut -f1 <<<"$(run_rev "$(review jakman85 "$AFTER" APPROVED '/approve-override belt and braces')")")" \
|
||||
jakman85
|
||||
|
||||
out="$(run_rev "$(review jakman85 "$AFTER" COMMENT 'ordinary review prose, no directive')")"
|
||||
check "an ordinary review body is not an override" "${out:-<none>}" "<none>"
|
||||
|
||||
# --- an unreadable source degrades to 'no candidates', never to a pass ----------------
|
||||
# The real-world shape: Gitea answers with {"message": "token does not have ... scope"}.
|
||||
ERR_OBJ='{"message":"token does not have at least one of required scope(s): [read:issue]"}'
|
||||
out="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
|
||||
DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
||||
check "an error OBJECT from the comments API yields no override" "${out:-<none>}" "<none>"
|
||||
|
||||
out="$(COMMENTS_JSON="$ERR_OBJ" \
|
||||
REVIEWS_JSON="$(review jakman85 "$AFTER" COMMENT '/approve-override reviews still work')" \
|
||||
HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
||||
check "an unreadable COMMENTS source does not suppress a valid REVIEW override" \
|
||||
"$(cut -f1 <<<"$out")" jakman85
|
||||
|
||||
# and it must SAY so, or a scope problem reads as 'you never posted one' (it did, on PR #38)
|
||||
errout="$(COMMENTS_JSON="$ERR_OBJ" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
|
||||
DIRECTIVE=/approve-override python3 "$SUT" 2>&1 >/dev/null)"
|
||||
case "$errout" in
|
||||
*"not a list"*read:issue*) ok "the diagnostic names the scope Gitea complained about" ;;
|
||||
*) bad "the diagnostic should quote Gitea's message (got: ${errout})" ;;
|
||||
esac
|
||||
|
||||
# --- latest wins ACROSS the two sources ----------------------------------------------
|
||||
out="$(COMMENTS_JSON="$(comment early "2026-10-01T12:10:00Z" '/approve-override from a comment')" \
|
||||
REVIEWS_JSON="$(review later "2026-10-01T12:50:00Z" COMMENT '/approve-override from a review')" \
|
||||
HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
||||
check "the latest override wins across both sources (review newer)" "$(cut -f1 <<<"$out")" later
|
||||
check "and its source is labelled" "$(cut -f3 <<<"$out")" review
|
||||
|
||||
out="$(COMMENTS_JSON="$(comment later "2026-10-01T12:50:00Z" '/approve-override from a comment')" \
|
||||
REVIEWS_JSON="$(review early "2026-10-01T12:10:00Z" COMMENT '/approve-override from a review')" \
|
||||
HEAD_TIME="$HEAD_TIME" DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
||||
check "the latest override wins across both sources (comment newer)" "$(cut -f1 <<<"$out")" later
|
||||
check "and that source is labelled too" "$(cut -f3 <<<"$out")" comment
|
||||
|
||||
# --- both sources absent entirely -----------------------------------------------------
|
||||
out="$(COMMENTS_JSON="" REVIEWS_JSON="" HEAD_TIME="$HEAD_TIME" \
|
||||
DIRECTIVE=/approve-override python3 "$SUT" 2>/dev/null)"
|
||||
check "no sources at all yields no override, and no crash" "${out:-<none>}" "<none>"
|
||||
|
||||
printf '\n %d passed, %d failed\n' "$PASS" "$FAIL"
|
||||
[ "$FAIL" -eq 0 ] || exit 1
|
||||
@@ -42,7 +42,16 @@ cp "${REAL_CI_DIR}/gate.sh" "${REAL_CI_DIR}/lib.sh" "${REAL_CI_DIR}/versions.env
|
||||
# 15 of 47 assertions failed on `main` for that reason alone, including every one that expects a
|
||||
# PASS/UNDETERMINED/NOT-APPLICABLE headline. The suite was measuring the missing stub, not the
|
||||
# decision logic. `decisions-log` is added here in the same change that adds it to gate.sh.
|
||||
for stage in spec-immutability decisions-log lint l0 render-golden l1 gitea-settings conformance; do
|
||||
#
|
||||
# AND IT HAPPENED AGAIN, with `approval-policy` (fixed 2026-10-01). Same shape, same cost: the
|
||||
# stage was added to gate.sh by PR #17, never added here, so 15 of 50 assertions failed on `main`
|
||||
# — measured on 0c7e8dd before touching anything, to be sure it was not the change under test.
|
||||
# The note above was written about the previous instance and did not prevent the next one, which
|
||||
# says the comment is not the fix. THE FIX IS TO DERIVE THE LIST: it now comes from gate.sh's own
|
||||
# `run_stage` calls, so a stage added there can no longer be silently missing here.
|
||||
mapfile -t STAGES < <(sed -n 's/^run_stage[[:space:]]\+\([a-z0-9-]\+\).*/\1/p' "${REAL_CI_DIR}/gate.sh")
|
||||
[ "${#STAGES[@]}" -ge 8 ] || { echo "could not parse stage names out of gate.sh (got ${#STAGES[@]})" >&2; exit 1; }
|
||||
for stage in "${STAGES[@]}"; do
|
||||
cat > "${CI}/${stage}.sh" <<'STUB'
|
||||
#!/usr/bin/env bash
|
||||
n="$(basename "$0" .sh)"; v="STAGE_RC_${n//-/_}"
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
---
|
||||
date: 2026-10-01
|
||||
branch: fix/admin-approval-override
|
||||
title: Administrators can merge their own green PR — required_approvals 1 → 0, plus an attributed /approve-override for sensitive paths
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**The problem, measured first.** `required_approvals: 1` on `main` was a lock-out, not a
|
||||
protection. Gitea refuses to let a pull request's author approve their own PR, and `jakman85` is
|
||||
the **only** collaborator (permission `owner`). So no account in existence could approve a PR the
|
||||
operator opened. Measured on the live API, 2026-10-01:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| PR #35 (`bugfix/arr-to-plex`, author `jakman85`) | `gate / gate (pull_request)` = **success, 1m41s** |
|
||||
| its reviews | `[]` |
|
||||
| live `required_approvals` | `1` |
|
||||
| live `block_admin_merge_override` | `true` |
|
||||
|
||||
Green gate, zero reviews, no path to a merge — and the same was true of any PR that would have
|
||||
fixed it.
|
||||
|
||||
It was also never the design this repo wrote down. `ci/approval-policy.sh`'s own header and
|
||||
`ci/gate.sh:194` both already described the model as *"`required_approvals` drops to 0 and the
|
||||
GATE decides"*, and `infra/gitea/require-gate.sh`'s closing note named this exact failure and
|
||||
prescribed this exact remedy. `decisions/2026-09-30-flux-deploy-notification-inhibited.md` had
|
||||
flagged the 1-vs-0 disagreement as unresolved and not that session's to settle. This settles it
|
||||
at 0.
|
||||
|
||||
**Two layers were blocking, and both are addressed.**
|
||||
|
||||
*Layer 1 — Gitea branch protection.* `required_approvals: 1 → 0` in
|
||||
`infra/gitea/expected-branch-protection.json`, with the rationale in the file, plus
|
||||
`infra/gitea/allow-admin-self-merge.sh` to apply it to the live rule (it reads the target **from
|
||||
the declaration**, so the script cannot drift from what the gate checks). `require-gate.sh` was
|
||||
changed to 0 as well — left at 1 it would silently re-impose the lock-out on its next run.
|
||||
|
||||
The narrow choice matters: the *other* way to unblock an owner is
|
||||
`block_admin_merge_override: false`, and that would have re-opened precisely what
|
||||
`infra/gitea/block-admin-override.sh` was written to close (PRs #5–#9 merged with zero approvals
|
||||
while the settings read as enforced). This change drops the approval **count** only.
|
||||
`block_admin_merge_override` and `enable_status_check` both stay `true`, and the script asserts
|
||||
both afterwards and fails loudly if the PATCH disturbed either. **A red gate remains unmergeable
|
||||
by everyone, administrators included.**
|
||||
|
||||
*Layer 2 — `ci/approval-policy.sh`.* Sensitive paths (`ci/**`, `infra/gitea/**`, `CLAUDE.md`,
|
||||
kyverno, rbac, sops, Flux wiring) still require a non-author `APPROVED` review, which the sole
|
||||
administrator still cannot obtain. An administrator may now waive it by commenting on the PR:
|
||||
|
||||
/approve-override <reason>
|
||||
|
||||
- the directive must **start a line** (a quoted or mid-sentence mention does not count, or
|
||||
repeating someone else's override would re-arm it);
|
||||
- a **reason is mandatory** — a bare directive records that review was waived but not why;
|
||||
- the commenter's permission is **re-checked live** against
|
||||
`/collaborators/{login}/permission` when the gate runs, so an old comment from a demoted
|
||||
account carries nothing;
|
||||
- it is **invalidated by a new commit** (the comment must post-date the head commit), mirroring
|
||||
Gitea's own `dismiss_stale_approvals`;
|
||||
- the overrider, timestamp and reason are printed in the gate log and the job summary, and a
|
||||
self-waiver by the PR's own author is called out as such.
|
||||
|
||||
Selection of the candidate comment is `ci/approval-override.py` — pure, offline, no authority of
|
||||
its own — so it could be unit-tested; authorisation stays in the fail-closed stage.
|
||||
`ci/tests/approval-override.test.sh` adds **23 assertions**, all passing, each constructing a
|
||||
case the parser must refuse. Every new failure mode blocks: unreadable head-commit timestamp,
|
||||
unreadable permission, parser error.
|
||||
|
||||
**The cost, stated rather than buried.** For an administrator, *"a change to the gate cannot be
|
||||
gated by the gate"* is now **advisory, not structural**. That is unavoidable with one human — the
|
||||
alternative is a repository its owner cannot change — and what is bought back is an attributed,
|
||||
reasoned, commit-scoped record of each waiver instead of a silent bypass.
|
||||
|
||||
**The uncomfortable part, accepted with eyes open (operator decision).** Authorisation is by live
|
||||
permission **alone**, with no allowlist of human accounts. The **agent account `gitea-admin` also
|
||||
reports permission `owner`** on this repo (measured), so an agent *can* clear the sensitive-path
|
||||
review on its own PR — and agents author most PRs here, so that is the common case, not an edge
|
||||
case. I raised this before implementing; the operator chose permission-only. It is documented in
|
||||
three places rather than left to be discovered: `approval-policy.json`
|
||||
(`_known_consequence_agent_accounts`), the stage's header, and `CLAUDE.md`, which now tells
|
||||
agents plainly **not** to use it on their own work. The narrowing path needs no code change —
|
||||
add an `override_admins` list to `approval-policy.json` and the stage already requires membership
|
||||
in addition to the permission check.
|
||||
|
||||
Also updated: `CLAUDE.md` §1 and the §2 enforcement table (the "approval required" row is now
|
||||
explicitly ⬜ with the reason, rather than a stale ✅), and `infra/gitea/README.md` gains §6 and
|
||||
marks its historical step 4 as reversed — do not re-apply.
|
||||
|
||||
**Second pass, 2026-10-02 — the override did not work the first time, and why.** The operator ran
|
||||
the live PATCH (`required_approvals: 0` confirmed live, with `block_admin_merge_override` and
|
||||
`enable_status_check` still `true`), merged PR #35, and posted `/approve-override Need to ensure
|
||||
approval override work as expected` on PR #38. The gate re-ran (run 203) and **still blocked.**
|
||||
|
||||
The comment was not at fault: posted by `jakman85`, directive at the start of the line, reason
|
||||
present, and 29 minutes **newer** than head commit `616a497`, so it cleared the staleness rule.
|
||||
What failed was that **CI could not read it.** Run 203's log:
|
||||
|
||||
approval-override: COMMENTS_JSON is not a list
|
||||
|
||||
`SETTINGS_API_TOKEN` is minted `["read:repository", "read:admin"]` by
|
||||
`infra/gitea/set-ci-token.sh`, and Gitea puts `/issues/{n}/comments` behind the **issue** scope. So
|
||||
Gitea returned an error *object*, `jq -e .` accepted it as valid JSON, and the stage fell through
|
||||
to "no approving review" — **a permission failure wearing the costume of a missing override.**
|
||||
`/pulls/{n}/files` and `/pulls/{n}/reviews` are under `read:repository`, which is exactly why every
|
||||
other part of the same stage worked and this one did not.
|
||||
|
||||
Three fixes, and the first is the one that mattered most:
|
||||
|
||||
1. **The diagnosis is no longer misleading.** `COMMENTS_JSON` is out of the must-parse-or-block
|
||||
loop and is explicitly checked for being an *array*; when it is not, the stage prints Gitea's
|
||||
own `.message`, names the missing scope, and says where to post the override instead. An
|
||||
unreadable source can only cause a false BLOCK, never a false pass, so degrading it to a loud
|
||||
warning is safe — and blocking on it was never the honest behaviour.
|
||||
2. **The override is accepted from a PR REVIEW body as well as a comment**, so the hatch is not
|
||||
hostage to one API scope. A review of type "Comment" works on the token CI has *today*. The
|
||||
gate names which source it used. `PENDING` (unsubmitted, author-only draft) reviews are dropped.
|
||||
3. **`set-ci-token.sh` now requests `read:issue`**, which makes the ordinary comment box work —
|
||||
the thing anybody reaches for first. It takes effect only on a **re-run**: an existing token's
|
||||
scopes cannot be widened in place.
|
||||
|
||||
`ci/tests/approval-override.test.sh` grows from 23 to **38 assertions**, covering the reviews
|
||||
source, `PENDING` reviews, an error *object* from the comments API, "an unreadable comments source
|
||||
must not suppress a valid review override", latest-wins across both sources, and that the
|
||||
diagnostic actually quotes Gitea's message.
|
||||
|
||||
**One adjacent fix, deliberately in scope: `ci/tests/gate-decision.test.sh` was already broken.**
|
||||
It hand-maintains a list of stub stage scripts that "must name EVERY stage `gate.sh` runs", and it
|
||||
did not name `approval-policy` — added to `gate.sh` by PR #17 and never added here. `run_stage`
|
||||
therefore got exit 127 from a missing file and recorded a FAILED stage in every case, so **15 of
|
||||
50 assertions failed**, including every one expecting a PASS/UNDETERMINED headline. Measured on
|
||||
pristine `origin/main` (`0c7e8dd`) before touching it, so the attribution is not a guess.
|
||||
|
||||
I fixed it because `make -C ci test` is the *only* runner for the suite this change adds — leaving
|
||||
it red would have made the new tests unverifiable in practice. The file's own comment already
|
||||
narrated this exact failure for `gitea-settings` and did not prevent the repeat, which says the
|
||||
comment was not the fix. So the stub list is now **derived from `gate.sh`'s own `run_stage` lines**
|
||||
and asserts it parsed at least 8 of them: a stage added to the gate can no longer be silently
|
||||
missing from its test. `make -C ci test` goes from **FAILED to all suites passed**.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **The lock-out is real, and PR #35 is the proof.** Queried live: head sha
|
||||
`8a094c46`, statuses `gate / gate (pull_request) = success :: Successful in 1m41s`,
|
||||
`/pulls/35/reviews` → `[]`, `/branch_protections/main` → `required_approvals: 1`. I read the
|
||||
**rule object**, not `/branches/main`, per §2b.
|
||||
- **`jakman85` is the only collaborator**: `/collaborators` → one entry. Permission `owner`.
|
||||
- **`gitea-admin` also reports `owner`** — this is why the agent-account consequence above is a
|
||||
statement of fact and not a hypothetical.
|
||||
- **PR #35 touches no sensitive paths** (`apps/wave-0/plex.yaml`, `apps/wave-1/{radarr,sonarr}.yaml`,
|
||||
`catalog/{radarr,sonarr}.yaml`, its decisions entry) — so `ci/approval-policy.sh` was never its
|
||||
blocker. Layer 1 alone unblocks it.
|
||||
- **API shapes verified before depending on them**, rather than assumed: `/git/commits/{sha}`
|
||||
returns `.commit.committer.date` as RFC3339 **with an offset** (`2026-10-01T23:09:08-05:00`),
|
||||
not a trailing `Z` — a parser handling only `Z` would have silently dropped every real
|
||||
override, so there is a test for the offset form. PR conversation comments are under
|
||||
`/issues/{n}/comments`; `/pulls/{n}/comments` returns diff comments only and would never have
|
||||
seen an override.
|
||||
- **`ci/tests/approval-override.test.sh`: 23 passed, 0 failed.** Covers the happy path, bare
|
||||
directive, mid-sentence and `>`-quoted directives, stale and equal-timestamp overrides,
|
||||
unparseable timestamps, missing login, invalid JSON, non-list JSON, latest-wins, offset
|
||||
timestamps, and a configurable directive.
|
||||
- `bash -n` clean on `ci/approval-policy.sh`, `infra/gitea/allow-admin-self-merge.sh`,
|
||||
`infra/gitea/require-gate.sh`; `jq -e .` clean on both JSON declarations.
|
||||
- **`gate-decision.test.sh`'s 15 failures are pre-existing, measured not assumed.** Extracted
|
||||
`origin/main`'s `ci/` at `0c7e8dd` into a throwaway directory and ran the suite from there:
|
||||
**FAIL, 15 failed / 35 passed**. After deriving the stub list from `gate.sh`: **PASS, 50
|
||||
assertions**.
|
||||
- **`make -C ci test` in the pinned toolbox: `ci/tests: all suites passed`, rc 0** — approval-override
|
||||
**38**, gate-decision 50, touched-requirements-base 31, test-attribution 15, touched-requirements 26,
|
||||
decisions-log pass. It was rc 2 (`ci/tests: FAILED`) before the stub-list fix.
|
||||
- **The live flip worked, and the drift detector proved it.** Run 202 (before): `gitea-settings:
|
||||
FAIL`, `required_approvals: DECLARED 0 but LIVE is 1`. Run 203 (after the operator ran the
|
||||
PATCH): `gitea-settings: pass`, with `block_admin_merge_override = true` among the keys it
|
||||
checked. So the narrow change held — the approval count moved and the gate's teeth did not.
|
||||
- **The first override attempt failed for a measured reason, not a guessed one.** Run 203 log:
|
||||
`approval-override: COMMENTS_JSON is not a list`. Token scopes read from `set-ci-token.sh`:
|
||||
`[read:repository, read:admin]`. The same run successfully read `/pulls/38` and
|
||||
`/pulls/38/files` (author and sensitive paths both resolved) and got past the reviews
|
||||
validation, which is how I know `/pulls/{n}/reviews` IS readable with that token — from
|
||||
behaviour, not from documentation.
|
||||
- **The comment itself was valid.** `created_at` 2026-10-02T05:31:24Z vs head commit `616a497`
|
||||
committer date 2026-10-02T00:02:20-05:00 → the comment is 29m04s newer, so it passed the
|
||||
staleness rule. Ruling the comment out is what left the token scope as the only candidate.
|
||||
- **`?limit=1000` is not the culprit.** Queried `/issues/38/comments` with `limit=1000`, `limit=50`
|
||||
and no limit using my own credentials: `type=array` in all three. So the difference is the
|
||||
credential, not the query.
|
||||
- **The root cause is confirmed by Gitea, in its own words.** Run 206 (the fix in place):
|
||||
`[warn] cannot read PR COMMENTS from Gitea — token does not have at least one of required
|
||||
scope(s), required=[read:issue], token scope=read:admin,read:repository.` followed by the
|
||||
instruction to post the override as a review. That is the diagnosis the first attempt should
|
||||
have produced, and `gitea-settings: pass` in the same run. The stage still ends `FAIL`, correctly
|
||||
— there is no valid override on the PR yet.
|
||||
- **Toolbox merge gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`), run on the
|
||||
rebased branch: **rc 3, `MERGE GATE: UNDETERMINED`**. Green stages: spec-immutability,
|
||||
decisions-log, lint, l0 (23 offline cases, 0 failures), render-golden including the new
|
||||
`catalog/render.py --selftest`, l1 (3 cases), approval-policy (N/A offline). Skipped:
|
||||
`gitea-settings` (no `SETTINGS_API_TOKEN` on this host) and `conformance` (no cluster).
|
||||
**This is NOT a green run and I am not reporting it as one** — rc 3 is UNDETERMINED per the
|
||||
CLAUDE.md §3 contract.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The override has still never been ACCEPTED by a real run.** This is the second attempt at
|
||||
this bullet, which is itself the point: the first pass called the hatch unverified, and the
|
||||
first real test found a defect that no amount of local testing would have (an API scope CI has
|
||||
and I did not). What is now measured is that the parse → *reject* path works end to end and
|
||||
diagnoses itself. The **accepting** branch — directive found → live permission lookup → exit 0 —
|
||||
has never executed. Posting the override as a **review** on PR #38 is its first real test, and
|
||||
I would not be surprised by a third round.
|
||||
- **`read:issue` is the right scope name — Gitea said so itself, so this is no longer a guess.**
|
||||
Run 206 printed Gitea's own message: `token does not have at least one of required scope(s),
|
||||
required=[read:issue], token scope=read:admin,read:repository`. That both confirms the root
|
||||
cause and names the string, which matters because this repo has been burned by a token *name*
|
||||
that could not exist (`GITEA_SETTINGS_TOKEN`). What is **still** unverified is that the token
|
||||
*mint* accepts it — `POST /users/{u}/tokens` with that scope has not been run. The review route
|
||||
does not depend on it.
|
||||
- **I did not verify that Gitea lets a PR's own author submit a `COMMENT`-type review.** Gitea
|
||||
blocks author self-*approval*; "Comment" is expected to be allowed, and that expectation is now
|
||||
load-bearing for the recommended route. It is one click to find out, and I deliberately did not
|
||||
test it by posting an override myself: doing so as `gitea-admin` is precisely the agent
|
||||
self-override that `CLAUDE.md` now forbids.
|
||||
- **The whole re-run mechanism is manual.** `gate.yml` triggers on `pull_request` and `push: main`
|
||||
only — there is no `issue_comment` trigger, so posting an override does **not** re-run the gate.
|
||||
Re-run the job from the Actions UI, or close and reopen the PR (which fires `pull_request` and
|
||||
leaves the head SHA alone). **Do not push an empty commit to re-trigger it:** that moves the head
|
||||
commit and makes the override stale by the rule above, so it would have to be posted again. A
|
||||
comment trigger was considered and not built — it would need to check out PR head code while the
|
||||
job holds `SETTINGS_API_TOKEN`, which is the classic pwn-request shape and not worth it here.
|
||||
- **I have not seen the gate still REFUSE a merge** now that approvals are 0. The claim that
|
||||
`block_admin_merge_override: true` keeps a red gate unmergeable rests on Gitea's documented
|
||||
behaviour and on the 2026-09-29 session, **not** on a provoked refusal in this one. That probe
|
||||
is written out at the foot of `allow-admin-self-merge.sh` and should be done, exactly as the
|
||||
push rejection was provoked rather than assumed.
|
||||
- **Clock skew in the staleness check is unhandled.** Staleness compares the comment's
|
||||
server-side `created_at` against the head commit's *committer* date, which comes from the
|
||||
committer's machine. A commit dated in the future would reject legitimate overrides until real
|
||||
time catches up (fail-closed, so loud, but annoying). Gitea's own `dismiss_stale_approvals`
|
||||
compares SHAs instead; a comment cannot. Not hit here — PR #35's commit date is ~21 min behind
|
||||
server time — and not fixed.
|
||||
- **`ci/tests/approval-override.test.sh` does not run in the merge gate.** `ci/gate.sh` runs no
|
||||
`ci/tests/*`; they run only via `make -C ci test`. So a regression in the parser would not be
|
||||
caught by CI. PR #33 (merged while this branch was being written, and rebased onto here) fixed
|
||||
the *same class* of gap for `catalog/render.py --selftest` by invoking it from
|
||||
`ci/render-golden.sh` — it did **not** wire `ci/tests/*` in, which I confirmed after rebasing
|
||||
rather than assuming. Wiring them is a one-line stage and I did not widen scope to it.
|
||||
- **Authorisation itself is untested code.** `ci/approval-override.py` is unit-tested; the live
|
||||
permission lookup, the `override_admins` allowlist branch, and every new BLOCK path in
|
||||
`ci/approval-policy.sh` need Gitea and have never been exercised. The allowlist branch in
|
||||
particular is implemented but **unused** — `override_admins` is absent by design.
|
||||
- **The other open PRs will go red** once the live value changes, until they are rebased onto the
|
||||
merged declaration: as of writing those are **#29** and **#37** (both `gitea-admin`), plus
|
||||
**#35**. I did not rebase them. Merging them first avoids it entirely; the script's header gives
|
||||
the order. PRs #33, #34 and #36 merged while this branch was being written — this branch is
|
||||
rebased onto `0c7e8dd`, so the list was re-measured rather than carried forward.
|
||||
- No cluster state was read or changed; nothing here touches Flux, Kyverno or any workload.
|
||||
@@ -0,0 +1,120 @@
|
||||
---
|
||||
date: 2026-10-01
|
||||
branch: bugfix/arr-to-plex
|
||||
title: Radarr and Sonarr may now reach Plex on 32400 — the network path only, not the Plex connection itself
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Radarr and Sonarr had no route to Plex. The media library lives on NFS
|
||||
(`/mnt/Storage/plex`, the ZFS dataset `Storage/plex` on `${BULK_NFS_SERVER}`), and NFS
|
||||
does not deliver filesystem CRUD notifications, so Plex never learns that an arr has
|
||||
written a new file. Without a push from the arrs, the library only reflects new media
|
||||
when somebody scans it by hand. That is the bug this branch addresses.
|
||||
|
||||
The fix is one `dependencies:` entry per app, added to the catalog — the source of truth:
|
||||
|
||||
- to: plex
|
||||
port: 32400
|
||||
protocol: tcp
|
||||
|
||||
in both `catalog/radarr.yaml` and `catalog/sonarr.yaml` (these two *are* the arr suite
|
||||
here; `jackett`, `qbittorrent` and `seerr` are different roles and are untouched).
|
||||
|
||||
The renderer expands each entry into a **pair** of Cilium policies, which is why a
|
||||
three-line catalog change produces 88 lines under `apps/`:
|
||||
|
||||
| Side | Policy | File |
|
||||
|---|---|---|
|
||||
| egress | `radarr-allow-plex-32400` | `apps/wave-1/radarr.yaml` |
|
||||
| egress | `sonarr-allow-plex-32400` | `apps/wave-1/sonarr.yaml` |
|
||||
| reciprocal ingress | `plex-allow-radarr-32400` | `apps/wave-0/plex.yaml` |
|
||||
| reciprocal ingress | `plex-allow-sonarr-32400` | `apps/wave-0/plex.yaml` |
|
||||
|
||||
Three commits, and the middle two are worth recording because both were self-inflicted
|
||||
gate failures rather than design changes:
|
||||
|
||||
- `753c48b` — the catalog edit alone, with `apps/` left unrendered.
|
||||
- `9048c2f` — re-rendered `apps/` from the catalog, which `render==golden` requires.
|
||||
- `58f02fe` — removed 20 trailing spaces on `catalog/radarr.yaml:103`, left behind as
|
||||
comment-alignment padding when the comment moved down to the `port:` line. Sonarr's
|
||||
equivalent line kept its comment in place, which is why only radarr tripped.
|
||||
|
||||
## What I verified
|
||||
|
||||
**The lint failure, root-caused.** Gate run 193 on `9048c2f` failed with
|
||||
`[fail] lint: FAILED -> blocks merge`, and lint was the only failing stage. yamllint
|
||||
emitted 419 findings, but **418 were `line-length`**, which `ci/yamllint.yaml` pins at
|
||||
`level: warning` and which therefore cannot change the exit code. The single error-level
|
||||
finding in the entire repository was:
|
||||
|
||||
catalog/radarr.yaml
|
||||
103:13 error trailing spaces (trailing-spaces)
|
||||
|
||||
`trailing-spaces` inherits the default severity (error), so that one line returned rc 1.
|
||||
After the fix, `git grep -nIE ' +$' -- '*.yaml' '*.yml'` matches nothing repo-wide.
|
||||
|
||||
**Gate run 195 on `58f02fe`** (read from the job log, not from the status tick):
|
||||
|
||||
[ ok ] approval-policy: green [ ok ] l0: green
|
||||
[ ok ] gitea-settings: green [ ok ] render-golden: green
|
||||
[ ok ] decisions-log: green [ ok ] l1: green
|
||||
[ ok ] lint: green
|
||||
[fail] MERGE GATE: UNDETERMINED — greenness was NOT ESTABLISHED.
|
||||
[fail] Stages that did not run: spec-immutability
|
||||
|
||||
The Gitea check reports `success` / "Successful in 1m41s", but the gate's own verdict is
|
||||
**UNDETERMINED (rc 3)**, not PASS. That tick is `gate.yml` tolerating exit 3 for the one
|
||||
stage it documents as unverifiable on this runner (the spec repo is not checked out
|
||||
there). Nothing else produced a 3.
|
||||
|
||||
**`render==golden` green** on `58f02fe`, so `apps/` is a faithful canonical render of
|
||||
`catalog/` — the four generated policies are not hand-written drift.
|
||||
|
||||
**DNS egress was already open** and is not a new gap: `apps/wave-1/radarr.yaml` carries
|
||||
an existing allow for port 53 UDP+TCP, so resolving the Plex service name does not need
|
||||
anything added here.
|
||||
|
||||
**Live branch protection on `main`**, measured against
|
||||
`/api/v1/repos/Home-Network/home-network-k3s-impl/branches/main` rather than assumed:
|
||||
`protected: true`, `user_can_push: false`, `enable_status_check: true`,
|
||||
`status_check_contexts: ['gate / gate (pull_request)']`, `required_approvals: 1`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
**This change does not accomplish what the PR set out to do.** It opens the network path
|
||||
and nothing else. Triggering a library update also requires each arr's
|
||||
*Settings → Connect → Plex Media Server* entry plus a Plex token, and **none of that
|
||||
exists anywhere in `catalog/` or `apps/`** — I grepped both for `plex`, `token` and
|
||||
`connect` and found only NFS paths, a uid and a timezone. Radarr's and Sonarr's config
|
||||
lives on a 5Gi Longhorn PVC mounted at `/config`, so that connection is UI-only state
|
||||
that exists solely on the PVC and never in git, in the same way Crafty's local login
|
||||
does. **Until somebody adds it in each app's UI, Plex still will not see library
|
||||
updates, and this branch will look like it did nothing.** That follow-up is not tracked
|
||||
by any file in this PR.
|
||||
|
||||
**L2/L3 conformance never ran.** `CLUSTER_AVAILABLE` is unset on the runner, so the
|
||||
eight requirements this diff touches — `R-NET-01`, `R-NET-05`, `R-NET-06`, `R-NET-07`,
|
||||
`R-IDP-02`, `R-IDP-06`, `R-GITOPS-02`, `R-PLAT-02` — are **UNVERIFIED**. That gap is
|
||||
unusually load-bearing here: the entire substance of this change is four
|
||||
CiliumNetworkPolicies, and no test has confirmed that traffic from the `radarr` or
|
||||
`sonarr` namespaces actually reaches `plex:32400`. The green gate proves the offline
|
||||
render subset only.
|
||||
|
||||
**`spec-immutability` did not run** (`R-TEST-06` not verified), for the documented
|
||||
reason — the sibling spec repo is absent from the runner.
|
||||
|
||||
**Nothing was checked on the cluster.** I ran no `kubectl`, so I did not confirm that
|
||||
the `plex` Service exposes 32400 as the arrs would address it, that the pod labels the
|
||||
policies select (`app.kubernetes.io/name`, `k8s:io.kubernetes.pod.namespace`) match the
|
||||
running pods, or that Cilium accepted the four new objects. Flux has not applied any of
|
||||
this: the branch is unmerged, so `.status.lastAppliedRevision` does not contain it.
|
||||
|
||||
**I did not verify Plex's own side** — whether its library-update endpoint on 32400 is
|
||||
the right target for the arrs' notification, or whether Plex requires
|
||||
"Update my library automatically" enabled to act on one.
|
||||
|
||||
**I am not the approver and did not approve this PR.** `required_approvals: 1` is live,
|
||||
and the only credential on the authoring workstation belongs to the PR's author
|
||||
(`jakman85`), whose own review Gitea does not count. Per CLAUDE.md §1 the merge is a
|
||||
human action regardless.
|
||||
@@ -0,0 +1,149 @@
|
||||
---
|
||||
date: 2026-10-02
|
||||
branch: fix/plex-quicksync-verification
|
||||
title: Hardware transcoding works — and `runbooks/plex-quicksync.md`'s verification section was wrong
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**Hardware transcoding is confirmed working on prod.** Measured 2026-10-02 from
|
||||
`Plex Media Server.log` in the running pod: `vaapi` on `/dev/dri/renderD128`
|
||||
(`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`,
|
||||
`zero-copy support present`, `using zero-copy transcoding`, and
|
||||
`final decoder: vaapi, final encoder: vaapi`. VAAPI for **both** directions, zero-copy, so frames
|
||||
never round-trip through the CPU. The device-plugin mechanism from PR #37 works end to end.
|
||||
|
||||
**And `runbooks/plex-quicksync.md`'s verification section was wrong, which is the correction this
|
||||
branch exists to publish.** The page said the Plex dashboard showing `Transcode (hw)` was "the *only*
|
||||
proof available, and nothing else will ever tell you". The operator had hardware transcoding fully
|
||||
working while **Dashboard → Activity showed nothing of the sort** — so the page's authoritative check
|
||||
was silent on a working system, and sent someone looking for a fault that did not exist. Three
|
||||
inference errors are now corrected on the page, two of them recorded as named false leads so the next
|
||||
reader does not re-walk them:
|
||||
|
||||
1. **The dashboard is not proof.** Replaced as the primary check by Plex's own log — the grep is in
|
||||
the page, in its **redacting** form, because `Plex Media Server.log` carries API URLs containing
|
||||
`X-Plex-Token` (CLAUDE.md §6; a credential has been leaked into a transcript here before). The
|
||||
page also now says explicitly that **`Plex Transcoder Statistics.log` is the wrong file** — it
|
||||
holds XML `<Variant .../>` records, not ffmpeg output, and the name invites the mistake — and why
|
||||
`grep -a` is required. The dashboard is **kept but demoted** to a secondary, best-effort signal,
|
||||
with its absence stated to prove nothing.
|
||||
2. **An fd snapshot is not evidence.** A `/proc/*/fd` sweep inside the pod found no holder of
|
||||
`renderD128` while hardware transcoding was active; transcode sessions are short and the scan
|
||||
landed in a gap. Recorded as a documented false lead, with the reliable process-level check in its
|
||||
place: the transcoder's **argv** (`-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128`), found via
|
||||
`ps -eo pid,user,comm | grep -i transcoder`.
|
||||
3. **A missing preference key does not mean the setting is off.** The page told readers to treat an
|
||||
absent `HardwareAcceleratedCodecs` in `Preferences.xml` as "never enabled". With hardware
|
||||
acceleration demonstrably on and working, that key was **absent from all 27 attributes** while
|
||||
`HardwareDevicePath` **was** written and the log said `hardware transcoding: enabled`. First-command
|
||||
#4 now reads both keys and states plainly that `HardwareDevicePath` present is a positive signal,
|
||||
absence of `HardwareAcceleratedCodecs` means nothing either way, only `="0"` is conclusive, and the
|
||||
log settles it.
|
||||
|
||||
Also on the page:
|
||||
|
||||
- **GID `992` is no longer presented as a number to expect.** Under the plugin the injected device
|
||||
arrives `root:root`, so the image's `cont-init.d` hook resolves GID **`0`** and adds `plex` to the
|
||||
**root group** — measured `Groups: 0 100 1000`, `id plex` → `groups=1000(plex),0(root),100(users)`.
|
||||
992 was the hostPath-era host `render` GID. The page now says so, marks the root-group membership as
|
||||
an inherent property of the plugin's `DeviceSpec` rather than a defect, and repeats that it is
|
||||
**not** a reason to reintroduce `supplementalGroups` (still a no-op: `s6-setuidgid` rebuilds the
|
||||
group list from `/etc/group`). The stale `Groups: 100 1000` claim in the escalation section is gone.
|
||||
- **A new troubleshooting entry for legitimate per-title fallbacks**, marked as a **hypothesis**.
|
||||
Between two fully hardware sessions, one logged `enabled, but no hardware decode accelerator found`
|
||||
and empty final decoder/encoder. UHD 630 decodes H.264, HEVC 8/10-bit and VP9 but **not AV1** and
|
||||
not 10-bit 4:2:2 HEVC, so the likely cause is the source file, not the platform — explicitly
|
||||
written as a lead to check first, **not** a confirmed diagnosis of that item.
|
||||
- The status is stated at the top of the page and in a rewritten provenance footer that separates the
|
||||
2026-10-02 confirmations from the superseded 2026-09-30/2026-10-01 hostPath-era readings.
|
||||
- `runbooks/README.md`'s index entry for the page was stale in the same two ways (it still described a
|
||||
`/dev/dri` hostPath and the dashboard-only proof) and is corrected to match.
|
||||
|
||||
Unchanged on purpose: the "STOP HERE IF PLEX IS `Pending`" section and the escape hatch (delete the
|
||||
`hardware:` block from `catalog/plex.yaml`, re-render) keep exactly their existing prominence — they
|
||||
are correct and they are the 2am material. The "no alert is possible" reasoning is also kept verbatim
|
||||
in substance: Plex declares no `metricsEndpoint`, renders no ServiceMonitor, and a fallback alert
|
||||
would be a rule evaluating against nothing.
|
||||
|
||||
**Documentation only.** No change to `catalog/`, `schemas/`, `apps/` or `platform/`.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **Gate, in the pinned toolbox from this worktree** (`DEV_KUBECONFIG=/dev/null`, `ENGINE` shim
|
||||
bind-mounting the main checkout's `.git`): **rc 3 UNDETERMINED**, diff base
|
||||
`origin/main @ 948875aad37d`, with exactly the three expected skips and **no failures**. Green:
|
||||
`decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy`. Skipped:
|
||||
`spec-immutability` (spec repo unavailable → R-TEST-06 not verified), `gitea-settings`
|
||||
(`SETTINGS_API_TOKEN` unset), `conformance` (`CLUSTER_AVAILABLE!=true` → L2/L3 not run).
|
||||
Touched requirements: **R-OBS-04**, reported `UNVERIFIED` for lack of a cluster. `approval-policy`
|
||||
passed as `N/A` locally (`event='none'`, not a `pull_request`) — it evaluates for real on the PR.
|
||||
**rc 3 is not green**, it is the calibrated worktree result, and it is reported here as rc 3.
|
||||
- **The gate was run three times, with the same rc 3 / three-skip result on each committed tip** —
|
||||
including the final one. Worth recording from the first attempt: run *before* committing, with
|
||||
`HEAD == origin/main`, the touched-requirement set is `NOT DETERMINABLE (on-base)` and the
|
||||
R-TEST-03 coverage gate cannot be applied at all. Commit before gating from a fresh worktree branch.
|
||||
- **`runbooks/kustomization.yaml` still carries `- plex-quicksync=plex-quicksync.md`** in the
|
||||
`configMapGenerator` `files:` list (line 110) — checked explicitly, because without it the page
|
||||
merges and still 404s. `README=README.md` is likewise present, which is what serves the corrected
|
||||
index entry.
|
||||
- **This change touches no `sensitive_paths`.** `infra/gitea/approval-policy.json` lists `ci/**`,
|
||||
`.gitea/**`, `conformance/**`, `clusters/**`, `bootstrap/**`, `infra/gitea/**`,
|
||||
`platform/kyverno/**`, `platform-config/kustomization.yaml`, `.sops.yaml`, `**/*.sops.yaml`,
|
||||
`**/rbac.yaml`, `CLAUDE.md`. `runbooks/**` and `decisions/**` are not among them, and the
|
||||
`approval-policy` stage passed.
|
||||
- **No YAML was added or edited**, so `indent-sequences: consistent` (which blocked a sibling branch
|
||||
today) is not in play here; `lint` passed over the full tracked YAML set regardless.
|
||||
- **`DECISIONS-LOG.md` is untouched** — this handoff is its own file, per CLAUDE.md §7.
|
||||
- **The PR's own CI run was read, not assumed** (PR #39, run 208, task 261, head `1677dff5688b`).
|
||||
**It is RED, and not because of this branch.** Every stage this change could affect is green —
|
||||
`decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` (`approval-policy` evaluated
|
||||
for real on the `pull_request` event and passed: no sensitive path touched). The single failure is
|
||||
**`gitea-settings`**, quoting the measured field:
|
||||
|
||||
[fail] required_approvals: DECLARED 1 but LIVE is 0
|
||||
[fail] Gitea has DRIFTED from infra/gitea/expected-branch-protection.json.
|
||||
[fail] MERGE GATE: BLOCKED
|
||||
|
||||
Every other declared setting matched (`block_admin_merge_override`, `block_on_rejected_reviews`,
|
||||
`dismiss_stale_approvals`, `enable_status_check`, `enable_push=false`,
|
||||
`status_check_contexts = ["gate / gate (pull_request)"]`, `runners online: 1`). So **live `main`
|
||||
currently requires ZERO approvals**, the declaration still says 1, and with
|
||||
`REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml` that drift **blocks every PR in
|
||||
the repo**, not just this one. Note the shape of it: the check that would notice approvals are
|
||||
unenforced is the check that is failing. Local branch `fix/admin-approval-override` (`616a497`,
|
||||
"fix(ci): let an administrator merge their own green PR") looks like the in-flight work that means
|
||||
to change the declaration. **I did not touch it** — `infra/gitea/**` is a sensitive path, needs a
|
||||
non-author approval, and is outside this branch's documentation-only scope. It is an operator
|
||||
decision: re-assert the protection, or change the declaration deliberately in its own PR.
|
||||
- The repository-wide grep for `992` to find what else still asserts it: remaining occurrences are in
|
||||
`catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md`,
|
||||
`platform/intel-gpu-plugin/README.md`, `platform/intel-gpu-plugin/kustomization.yaml`, and three
|
||||
older `decisions/` entries.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **I did not re-measure anything on the live cluster in this session.** Every platform number on the
|
||||
page — the VAAPI/zero-copy log lines, `RW-OPEN-OK` as uid 1000, the `gpu.intel.com/i915` limits and
|
||||
requests on the pod spec, `Groups: 0 100 1000`, `id plex`, the 27 `Preferences.xml` attributes, the
|
||||
empty `/proc/*/fd` sweep — came to me as measurements taken earlier on 2026-10-02 and was
|
||||
transcribed, not re-run. Per CLAUDE.md §2b, treat the page as reporting those readings rather than
|
||||
as proof of current state; re-run the primary check before relying on it.
|
||||
- **The per-title fallback cause is unverified and labelled as such.** The source codec of the title
|
||||
that logged `no hardware decode accelerator found` was never read. The AV1 / 10-bit-4:2:2-HEVC
|
||||
explanation is inferred from UHD 630's capability set alone.
|
||||
- **Whether Plex's dashboard *ever* shows `Transcode (hw)` on this deployment is unknown.** It was
|
||||
observed silent during working hardware transcoding; whether that is a client/version quirk, a
|
||||
reporting bug, or something about this setup was not investigated. The page therefore demotes it
|
||||
rather than explaining it.
|
||||
- **The commands as written in the page were not executed from the page.** They are the forms that
|
||||
were run, re-wrapped for `dev.sh`; in particular the `exec -it deploy/plex -- sh` route for the argv
|
||||
check is noted as possibly lacking a TTY under `dev.sh run` and was not exercised in that shape.
|
||||
- **Stale `992` copies outside `runbooks/` were left alone**, deliberately — scope was documentation
|
||||
only, and `catalog/plex.yaml`, `catalog/render.py`, `schemas/renderer-contract.md` and
|
||||
`platform/intel-gpu-plugin/` all still carry the hostPath-era `Groups: 44 100 992 1000` reading as
|
||||
if current. `render.py --selftest` asserts around that comment, so correcting it is a real change
|
||||
with gate consequences, not a doc edit. **That is open work.** Older `decisions/` entries also cite
|
||||
992; those are historical records and should stay as written.
|
||||
- **CI on the PR.** The branch is pushed and the PR opened; I did not watch the Gitea run report, so
|
||||
nothing here should be read as "CI green". The only gate result I can quote is the local rc 3 above.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
date: 2026-10-03
|
||||
branch: fix/node-keycloak-hosts-pin
|
||||
title: Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Follow-up to PR #40 (`decisions/2026-10-03-public-dns-wan-target.md`). Public DNS for
|
||||
`*.home.keithjackson.dev` now CNAMEs to the WAN IP. The prod nodes resolve through public DNS,
|
||||
and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer
|
||||
`keycloak.home.keithjackson.dev`.
|
||||
|
||||
- `roles/node_baseline`: a new `node_baseline_hosts_pins` list (default `[]`) and one
|
||||
`blockinfile` task (tags `dns`, `hosts`). It writes the pins into `/etc/hosts` as a single
|
||||
marked block. An empty list removes the block.
|
||||
- `group_vars/k3s_prod/vars.yml`: pins `keycloak.{{ k3s_base_domain }}` → `192.168.2.20`, which
|
||||
must stay equal to `GATEWAY_VIP` (parity comment added). It also replaces the stale "no
|
||||
/etc/hosts pin is needed" paragraph with the measured reason the pin is now required.
|
||||
|
||||
Dev is unaffected (empty default).
|
||||
|
||||
## What I verified
|
||||
|
||||
- After PR #40 reconciled (`platform` lastAppliedRevision `576e1ab`, Healthy=True), on
|
||||
k3s-prod-2 and k3s-prod-3: `getent hosts keycloak.home.keithjackson.dev` →
|
||||
`136.51.115.252 wan.home.keithjackson.dev`. `curl https://keycloak…/realms/homelab/.well-known/openid-configuration`
|
||||
→ exit 60, `ssl_verify_result=18` (the router's self-signed cert, not the Gateway).
|
||||
- `journalctl -u k3s` (last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds
|
||||
the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation.
|
||||
- Neither node has cloud-init (`/etc/cloud` absent), so nothing rewrites `/etc/hosts` at boot.
|
||||
- `--check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3`: each node gets exactly the
|
||||
three-line marked block `192.168.2.20 keycloak.home.keithjackson.dev`.
|
||||
- Outside-in, PR #40 works: check-host.net probes from 3 external nodes got `200 OK` from
|
||||
`https://plex.home.keithjackson.dev/identity` at `136.51.115.252`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **k3s-prod-1**: Ansible cannot reach it. SSH host key verification fails (`known_hosts:94`,
|
||||
ED25519 `SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE`), which is consistent with
|
||||
the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint
|
||||
before the pin can be applied there.
|
||||
- The playbook has not been applied for real. It is an operator run after merge:
|
||||
`ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`.
|
||||
- An end-to-end kubectl OIDC login after the pin (needs a browser).
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
date: 2026-10-03
|
||||
branch: feat/public-dns-wan-target
|
||||
title: Public DNS now points at the WAN — Gateway-published names CNAME to an in-cluster DDNS record
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator closed the open "public-target gap" recorded in
|
||||
`platform/externaldns/helmrelease.yaml`. The WAN NAT / port-forwarding (80/443 → Gateway VIP
|
||||
192.168.2.20) was done by the operator outside this repo.
|
||||
|
||||
Operator decisions (2026-10-03):
|
||||
|
||||
1. **Scope: every Gateway-published name goes public.** That covers the five `exposure: public`
|
||||
apps (plex, gitea, keycloak, nextcloud, seerr) and also the hand-authored admin UIs (auth,
|
||||
crafty, grafana, headlamp, the `home.keithjackson.dev` apex). `internal` services still publish
|
||||
nothing: their `controller: none` route annotation is checked before the target is read.
|
||||
2. **The WAN IP is kept current in-cluster**, not hardcoded.
|
||||
|
||||
Implementation:
|
||||
|
||||
- `platform/gateway/gateway.yaml`: `external-dns.alpha.kubernetes.io/target: wan.${BASE_DOMAIN}` on
|
||||
the shared Gateway. Every gateway-sourced record changes from `A 192.168.2.20` to
|
||||
`CNAME wan.home.keithjackson.dev`.
|
||||
- `platform/externaldns/ddns.yaml` (new): `favonia/cloudflare-ddns:1.17.1` Deployment in the
|
||||
`external-dns` namespace. It reuses `cloudflare-api-token` and keeps `wan.${BASE_DOMAIN}` an
|
||||
unproxied IPv4 A record at the WAN IP (detected via Cloudflare trace, checked every 5 min,
|
||||
`DELETE_ON_STOP=false`). external-dns has no TXT registry entry for it, so it never owns or
|
||||
deletes it.
|
||||
- The helmrelease comment block and the README now describe the decision instead of the gap.
|
||||
|
||||
Known consequence: **k3s nodes resolve through public DNS** (`node_baseline_dns_servers` = 8.8.8.8 /
|
||||
1.1.1.1). The kube-apiserver's OIDC issuer `keycloak.home.keithjackson.dev` therefore resolves to the
|
||||
WAN IP from the nodes, and kubectl SSO works only if the router hairpins WAN:443 back to the Gateway.
|
||||
A LAN-side probe of `136.51.115.252:443` on 2026-10-03 returned a **self-signed placeholder cert**,
|
||||
not the Gateway's Let's Encrypt wildcard. That suggests no hairpin. The cert-based admin kubeconfig
|
||||
is unaffected.
|
||||
|
||||
## What I verified
|
||||
|
||||
- Live prod, 2026-10-03, before the change: Gateway `gateway/shared` ADDRESS `192.168.2.20`.
|
||||
External-dns args include `--source=gateway-httproute --source=service --policy=sync
|
||||
--txt-owner-id=prod --domain-filter=keithjackson.dev`, and it logs "All records are already up to
|
||||
date".
|
||||
- Public answers at 1.1.1.1: plex/gitea/keycloak/grafana/auth/apex → `192.168.2.20`;
|
||||
`crafty-mc` → `.22` and `unifi` → `.21` (service source, not affected by this change); jackett →
|
||||
none (internal, correctly unpublished).
|
||||
- The authority (betty.ns.cloudflare.com) has no unprefixed TXT, AAAA or MX at `plex.` or the apex,
|
||||
so the A→CNAME switch has no co-existing record to collide with.
|
||||
- WAN IP from the workstation: `136.51.115.252`.
|
||||
- No CiliumNetworkPolicy or NetworkPolicy in `external-dns`. The Kyverno label and default-deny
|
||||
policies match only `app-namespace` namespaces. The vpn egress policy selects only
|
||||
`egress-class: vpn` pods.
|
||||
- Toolbox gate result: see the PR description (run after this commit).
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The outside-in path.** I have no vantage point outside the LAN. Whether
|
||||
`https://<svc>.home.keithjackson.dev` from the internet reaches the Gateway with the LE cert is
|
||||
UNVERIFIED.
|
||||
- **Hairpin / kubectl SSO after the flip.** Likely broken per the probe above. Fix candidates: a
|
||||
node `/etc/hosts` pin for `keycloak.${BASE_DOMAIN}` → Gateway VIP via `node_baseline`, or router
|
||||
NAT reflection.
|
||||
- **The record-type flip itself**, until Flux reconciles: whether Cloudflare accepts external-dns's
|
||||
create-CNAME in the same batch as the delete-A, or converges one interval later.
|
||||
- `favonia/cloudflare-ddns` under this pod securityContext. The README warns that
|
||||
`no-new-privileges` misbehaves on some Proxmox-style hosts. These are bare-metal nodes, but it is
|
||||
untested until the pod runs.
|
||||
- Whether the L4 service-source names (crafty-mc, unifi) should also go public. Not decided, not
|
||||
changed.
|
||||
- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh` header lines 39–40
|
||||
still describe the target as open. That is a sensitive path, so it is left for a follow-up.
|
||||
@@ -306,19 +306,27 @@ node_baseline_dns_servers:
|
||||
# k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
|
||||
# these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
|
||||
#
|
||||
# THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
|
||||
# authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
|
||||
# publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
|
||||
# kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
|
||||
# keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
|
||||
# PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
|
||||
# and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
|
||||
# THE PUBLIC ANSWER IS NOW THE WAN, SO THE ISSUER IS PINNED (2026-10-03). Since
|
||||
# platform/gateway/gateway.yaml set an external-dns `target`, every public name is a CNAME
|
||||
# to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the
|
||||
# LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
|
||||
# k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18). The apiserver would then
|
||||
# be unable to fetch its issuer's discovery/JWKS, so kubectl OIDC SSO would fail on the
|
||||
# next k3s restart or Keycloak key rotation. node_baseline_hosts_pins (below) pins the
|
||||
# issuer host to the Gateway VIP in /etc/hosts, which glibc consults before these
|
||||
# resolvers. (History: before the 2026-09-04 subdomain split the public answer was DEV's
|
||||
# VIP; after it, the public answer was prod's own VIP and no pin was needed.)
|
||||
#
|
||||
# STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
|
||||
# external-dns has actually published the record. Until then the issuer does not resolve
|
||||
# from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
|
||||
# path, and it always works. Ordering, not a design gap.
|
||||
# The cert-based admin kubeconfig never depends on any of this. It is the break-glass path.
|
||||
node_baseline_dns_search: []
|
||||
# /etc/hosts pins: node-process names that must reach the in-cluster Gateway directly rather
|
||||
# than via the public (WAN, non-hairpinned) answer. See "THE PUBLIC ANSWER IS NOW THE WAN" above.
|
||||
# [parity] the IP MUST equal GATEWAY_VIP in clusters/prod/kustomization.yaml (192.168.2.20),
|
||||
# and the name MUST follow k3s_oidc_issuer_url's host (roles/k3s_server/defaults/main.yml).
|
||||
node_baseline_hosts_pins:
|
||||
- ip: "192.168.2.20"
|
||||
names:
|
||||
- "keycloak.{{ k3s_base_domain }}"
|
||||
|
||||
# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
|
||||
# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
|
||||
|
||||
@@ -242,3 +242,9 @@ node_baseline_dns_search:
|
||||
node_baseline_dns_options:
|
||||
- timeout:2
|
||||
- attempts:2
|
||||
# Static /etc/hosts entries (tags: dns, hosts), as a list of {ip, names}. EMPTY by default:
|
||||
# dev needs none. Prod pins its OIDC issuer host to the Gateway VIP because the public
|
||||
# answer for it is the non-hairpinned WAN IP (inventory/group_vars/k3s_prod/vars.yml).
|
||||
# Written as ONE marked block, so removing an entry here removes it from the node, and
|
||||
# an empty list removes the block entirely.
|
||||
node_baseline_hosts_pins: []
|
||||
@@ -1090,3 +1090,22 @@
|
||||
{% endif %}
|
||||
when: node_baseline_dns_mechanism == 'resolv-conf-file'
|
||||
tags: [dns]
|
||||
|
||||
# /etc/hosts pins — tag `dns` (and `hosts` on its own). Some node processes must reach a
|
||||
# public hostname at the in-cluster Gateway, not at its public answer. On prod, the
|
||||
# kube-apiserver's OIDC issuer resolves publicly to the WAN IP, which the router does not
|
||||
# hairpin. glibc reads /etc/hosts before any nameserver (nsswitch `files dns`), so a pin wins
|
||||
# regardless of which resolver mechanism above is in effect. One marked block: entries
|
||||
# dropped from node_baseline_hosts_pins disappear from the node, and an empty list
|
||||
# removes the block.
|
||||
- name: Pin node-process hostnames in /etc/hosts
|
||||
ansible.builtin.blockinfile:
|
||||
path: /etc/hosts
|
||||
marker: "# {mark} node_baseline hosts pins (Ansible, tags: hosts)"
|
||||
block: |
|
||||
{% for pin in node_baseline_hosts_pins %}
|
||||
{{ pin.ip }} {{ pin.names | join(' ') }}
|
||||
{% endfor %}
|
||||
state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
|
||||
backup: true
|
||||
tags: [dns, hosts]
|
||||
+63
-1
@@ -52,7 +52,7 @@ All four steps are complete and each was verified, not assumed:
|
||||
| 1. Runner registered | ✅ `incluster-dind` online. Root cause of its absence was a stale `/data/.runner` identity; clearing the PVC forced re-registration |
|
||||
| 2. Gate observed green on a real PR | ✅ `gate / gate (pull_request)` ~1m35s |
|
||||
| 3. `enable_status_check: true` | ✅ context `gate / gate (pull_request)` — that one ONLY |
|
||||
| 4. Collaborator + approvals | ✅ `jakman85` is owner; `required_approvals: 1` |
|
||||
| 4. Collaborator + approvals | ✅ `jakman85` is owner. `required_approvals` was raised to 1 here and **lowered back to 0 on 2026-10-01** — see §6, it was a lock-out |
|
||||
|
||||
Direct push to `main` was **provoked and rejected** (`pre-receive hook declined`) as
|
||||
`gitea-admin`, a site admin — so push protection is not merely declared, it holds for the
|
||||
@@ -124,6 +124,66 @@ the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS
|
||||
**Changing the declaration is how you change policy.** If the live state is what you now want,
|
||||
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
|
||||
|
||||
### 6. `allow-admin-self-merge.sh` — undoing the lock-out step 4 created (2026-10-01)
|
||||
|
||||
Step 4 above raised `required_approvals` to 1. That was a **lock-out, not a protection**, and the
|
||||
note at the bottom of `require-gate.sh` had already said why: *Gitea will not let an author approve
|
||||
their own PR.* With `jakman85` the only collaborator, there was no account in existence that could
|
||||
approve a PR `jakman85` opened.
|
||||
|
||||
Measured 2026-10-01 — **PR #35**: `gate / gate (pull_request)` **success in 1m41s**, reviews `[]`,
|
||||
`required_approvals: 1`, `block_admin_merge_override: true`. Unmergeable, permanently, with no
|
||||
action available to its author. The same was true of any PR fixing it.
|
||||
|
||||
It was also never the documented design. `ci/approval-policy.sh`'s header and `ci/gate.sh:194`
|
||||
both already describe the model as *"`required_approvals` drops to 0 and the GATE decides"*, and
|
||||
`decisions/2026-09-30-flux-deploy-notification-inhibited.md` flagged the 1-vs-0 disagreement as
|
||||
unresolved and not that session's to settle. So this is a correction toward the stated intent, not
|
||||
a weakening away from it.
|
||||
|
||||
./infra/gitea/allow-admin-self-merge.sh # required_approvals -> 0 (read from the declaration)
|
||||
|
||||
**What stays switched on, and why that is the point.** The other way to unblock an owner is
|
||||
`block_admin_merge_override: false`, and it would have been the wrong one — that re-opens exactly
|
||||
what §5 above closed, letting the one account that merges bypass the gate itself. This script
|
||||
touches the approval **count** only and *verifies* afterwards that `block_admin_merge_override` and
|
||||
`enable_status_check` are both still `true`, failing loudly if the PATCH disturbed either:
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| `required_approvals` | 1 | **0** |
|
||||
| `block_admin_merge_override` | true | true (asserted) |
|
||||
| `enable_status_check` | true | true (asserted) |
|
||||
|
||||
So a **red gate is still unmergeable by everyone, administrators included**. What changed is that
|
||||
a *green* one no longer needs a signature that cannot exist.
|
||||
|
||||
**Security-relevant changes are still reviewed** — by `ci/approval-policy.sh` against
|
||||
`approval-policy.json` `sensitive_paths`, not by this field. An administrator waives that review
|
||||
only by posting `/approve-override <reason>` on the PR: reason mandatory, permission re-checked
|
||||
live, invalidated by a new commit, and recorded by account in the gate summary.
|
||||
|
||||
> **Post the override as a PR *review* (type "Comment"), not in the plain comment box** — until
|
||||
> the CI token is re-minted. `SETTINGS_API_TOKEN` is `[read:repository, read:admin]`, and
|
||||
> `/issues/{n}/comments` needs Gitea's *issue* scope, so Gitea answers with an error **object**
|
||||
> instead of an array and the override is invisible. Measured on PR #38 (2026-10-02): a correct,
|
||||
> fresh, well-formed override in the comment box produced `approval-policy: FAIL` reading as "no
|
||||
> approving review" — a permission failure disguised as a missing one. `/pulls/{n}/reviews` is
|
||||
> under `read:repository` and worked in the same run. The stage now reads **both** sources, names
|
||||
> which one it used, and says plainly when the comments API is unreadable. `set-ci-token.sh`
|
||||
> requests `read:issue` as of 2026-10-02, so **re-run it** and the comment box works too — token
|
||||
> scopes cannot be widened in place. Note that the
|
||||
**agent account `gitea-admin` also reports permission `owner`**, so it can clear its own PRs that
|
||||
way; that is an accepted consequence of authorising by permission alone, and the route to
|
||||
narrowing it (an `override_admins` allowlist, already honoured by the code) is written down in
|
||||
`approval-policy.json`.
|
||||
|
||||
**Sequencing.** `ci/gitea-settings.sh` compares live against the declaration *from the branch under
|
||||
test*, and `REQUIRE_GITEA_SETTINGS=true`. So merge any other open PRs **before** running this, then
|
||||
run it, then re-run the gate on the branch carrying the new declaration. Any PR left open across
|
||||
the change declares 1 against a live 0 and goes **red until rebased** — the detector working, not
|
||||
a fault. The script's header spells the order out.
|
||||
|
||||
## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential
|
||||
|
||||
Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
|
||||
@@ -204,6 +264,8 @@ it. The name matters — see the note under the behaviour table above.)
|
||||
2. **Watch the gate actually report on a real PR**, and note the context name it produced.
|
||||
3. **Only then set `enable_status_check: true`** with that exact context.
|
||||
4. Add the human reviewer as a repo collaborator, then raise `required_approvals` to 1.
|
||||
— **Step 4 was REVERSED on 2026-10-01 (§6).** It assumed a reviewer distinct from the author
|
||||
would exist. One never did, so this step produced a lock-out. Do not re-apply it.
|
||||
|
||||
> **Do not do step 3 before step 1.** A required status check with no runner leaves every
|
||||
> PR permanently pending — including the PR that would fix it. That half is measured: PR #1
|
||||
|
||||
Executable
+105
@@ -0,0 +1,105 @@
|
||||
#!/usr/bin/env bash
|
||||
# Let the sole administrator merge their own green PR: required_approvals 1 -> 0 on `main`.
|
||||
#
|
||||
# WHY. `required_approvals: 1` plus Gitea's refusal to let an author approve their own pull
|
||||
# request plus exactly ONE collaborator (`jakman85`, permission `owner`) is a lock-out, not a
|
||||
# policy. Every PR the operator opened was permanently unmergeable, including the PR that would
|
||||
# fix it. Measured 2026-10-01: PR #35 — `gate / gate (pull_request)` success in 1m41s, zero
|
||||
# reviews, `required_approvals: 1`, no account in existence able to satisfy it.
|
||||
#
|
||||
# The approvals field was also never what the design said it was. `ci/approval-policy.sh`'s
|
||||
# header and `ci/gate.sh:194` both already described the model as "required_approvals: 0 and the
|
||||
# GATE decides", and `require-gate.sh`'s own closing note named this exact failure and this exact
|
||||
# remedy. decisions/2026-09-30-flux-deploy-notification-inhibited.md flagged the 1-vs-0
|
||||
# disagreement as unresolved. This resolves it at 0.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* DO, and the distinction is the whole point.
|
||||
#
|
||||
# * It does NOT touch `block_admin_merge_override`, which stays TRUE. The required status check
|
||||
# still binds ABSOLUTELY — a red gate is unmergeable by everyone, administrators included.
|
||||
# Setting `block_admin_merge_override: false` would have been the other way to unblock an
|
||||
# owner, and it would have re-opened precisely what block-admin-override.sh was written to
|
||||
# close: PRs #5-#9 merged with zero approvals while the settings read as enforced.
|
||||
# * It does NOT stop reviewing security-relevant changes. Those are gated by
|
||||
# ci/approval-policy.sh against infra/gitea/approval-policy.json `sensitive_paths`, which an
|
||||
# administrator can waive only with an explicit, attributed `/approve-override <reason>`
|
||||
# comment on the PR.
|
||||
#
|
||||
# ORDER MATTERS — READ THIS BEFORE RUNNING IT.
|
||||
#
|
||||
# ci/gitea-settings.sh fails the merge gate on ANY divergence between the live rule and
|
||||
# infra/gitea/expected-branch-protection.json, and `gate.yml` sets REQUIRE_GITEA_SETTINGS=true.
|
||||
# The declaration in git now says 0. So between running this and merging that declaration, the
|
||||
# live value and the declared value must be compared from a branch that CARRIES the new
|
||||
# declaration — otherwise the stage goes red, correctly, for drift.
|
||||
#
|
||||
# 1. MERGE any other open PRs FIRST, while approvals still work for them (their branches
|
||||
# declare 1, and a PR you did not author can still be approved normally). Skipping this
|
||||
# only costs a rebase later; it does not break anything.
|
||||
# 2. Run this script.
|
||||
# 3. Re-run the gate on the branch that carries the new declaration. `gitea-settings` goes
|
||||
# green because live 0 == declared 0.
|
||||
# 4. Merge that branch.
|
||||
# 5. Any PR still open from before step 2 now declares 1 against a live 0 and will be RED
|
||||
# until it is rebased onto the new main. That is the detector doing its job, not a fault.
|
||||
#
|
||||
# Recoverable: `required_approvals` can be PATCHed back to 1 by an owner at any time. If you do,
|
||||
# change expected-branch-protection.json in the same breath or every PR goes red on drift.
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
DECL="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/expected-branch-protection.json"
|
||||
|
||||
command -v jq >/dev/null || { echo "jq is required" >&2; exit 1; }
|
||||
[ -f "$DECL" ] || { echo "declaration missing: ${DECL}" >&2; exit 1; }
|
||||
|
||||
# Take the target from the DECLARATION, not from a literal here. If the two could disagree,
|
||||
# this script would be one more thing re-asserting a value nothing checks — which is the
|
||||
# failure mode the whole of infra/gitea/ exists to stop.
|
||||
WANT="$(jq -r '.protection.required_approvals' "$DECL")"
|
||||
[[ "$WANT" =~ ^[0-9]+$ ]] || { echo "declaration has no numeric required_approvals" >&2; exit 1; }
|
||||
|
||||
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
|
||||
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
|
||||
[ -n "$U" ] && [ -n "$P" ] || { echo "no stored credential for $HOST" >&2; exit 1; }
|
||||
echo "authenticating as '${U}' (secret not shown)"
|
||||
api() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
B="https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branch_protections/main"
|
||||
|
||||
# Read the RULE OBJECT, never /branches/main: the branch view does not expose
|
||||
# block_admin_merge_override and showed all-green for hours while the rule was overridable.
|
||||
echo "==> before"
|
||||
api "$B" | jq -r '{required_approvals, block_admin_merge_override, enable_status_check, status_check_contexts}'
|
||||
|
||||
echo "==> setting required_approvals = ${WANT} (declared in expected-branch-protection.json)"
|
||||
api -X PATCH -H 'Content-Type: application/json' \
|
||||
-d "{\"required_approvals\": ${WANT}}" "$B" >/dev/null
|
||||
|
||||
echo "==> after"
|
||||
AFTER_JSON="$(api "$B")"
|
||||
jq -r '{required_approvals, block_admin_merge_override, enable_status_check, status_check_contexts}' <<<"$AFTER_JSON"
|
||||
|
||||
GOT="$(jq -r '.required_approvals' <<<"$AFTER_JSON")"
|
||||
ADMIN_BLOCK="$(jq -r '.block_admin_merge_override' <<<"$AFTER_JSON")"
|
||||
STATUS="$(jq -r '.enable_status_check' <<<"$AFTER_JSON")"
|
||||
|
||||
fail=0
|
||||
[ "$GOT" = "$WANT" ] || { echo "PATCH did not take: required_approvals is ${GOT}, wanted ${WANT}" >&2; fail=1; }
|
||||
# These two are what keeps this change narrow. If the PATCH disturbed either, the gate has
|
||||
# stopped binding and that must be loud, not discovered later.
|
||||
[ "$ADMIN_BLOCK" = true ] || { echo "block_admin_merge_override is ${ADMIN_BLOCK}, MUST be true — the gate is now bypassable" >&2; fail=1; }
|
||||
[ "$STATUS" = true ] || { echo "enable_status_check is ${STATUS}, MUST be true — the gate is no longer required" >&2; fail=1; }
|
||||
[ "$fail" -eq 0 ] || { echo "do NOT record this as applied." >&2; exit 1; }
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
NOW PROVE IT -- the setting is not the evidence. Two things to see, not assume:
|
||||
|
||||
1. An APPROVAL IS NO LONGER NEEDED. On your own PR with a green gate, the merge button is
|
||||
live with zero reviews on it.
|
||||
2. THE GATE STILL BINDS. On a PR whose gate is RED or still pending, Gitea must REFUSE the
|
||||
merge -- for you, as owner. If it lets you through, block_admin_merge_override is not
|
||||
doing its job and this change has traded a lock-out for a bypass.
|
||||
|
||||
Then confirm the detector agrees: run ci/gate.sh from a branch carrying the new
|
||||
expected-branch-protection.json and check that `gitea-settings` is green rather than DRIFT.
|
||||
NOTE
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"_intent": "The CI/CD pipeline is the approver. A human approval is required ONLY for changes that alter what the pipeline itself verifies, or the security posture it rests on -- because those are the changes CI cannot honestly gate: a PR that weakens the gate would be approved by the weakened gate.",
|
||||
"_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author. Everything else merges on a green gate alone.",
|
||||
"_rule": "If a PR touches any sensitive_paths glob, it needs an APPROVED review from someone who is not its author, OR an admin override (see .override). Everything else merges on a green gate alone.",
|
||||
"_on_failure_to_evaluate": "BLOCK. Unlike gitea-settings, a skip here is a bypass, not a gap.",
|
||||
"sensitive_paths": [
|
||||
"ci/**",
|
||||
@@ -24,5 +24,18 @@
|
||||
"platform/kyverno/**": "the admission policy engine",
|
||||
".sops.yaml, **/*.sops.yaml, **/rbac.yaml": "encryption recipients, secrets, and cluster authority",
|
||||
"CLAUDE.md": "the delivery contract agents follow"
|
||||
},
|
||||
"override": {
|
||||
"_why": "This repository has ONE human. Gitea refuses to let an author approve their own pull request, so 'a non-author APPROVED review' is a requirement the sole administrator can never satisfy -- every sensitive-path PR they open was permanently unmergeable, including the PR that would fix it. Measured 2026-10-01: PR #35 green gate, zero reviews, unmergeable. An escape hatch that the administrator can actually reach is therefore REQUIRED for the repository to function; the design question is only whether it leaves a record. This one does.",
|
||||
"enabled": true,
|
||||
"comment_directive": "/approve-override",
|
||||
"requires_reason": true,
|
||||
"authorized_permissions": ["admin", "owner"],
|
||||
"invalidated_by_new_commits": true,
|
||||
"_how": "Post the directive at the START of a line, followed by a reason: '/approve-override <why>'. ci/approval-policy.sh resolves the poster's LIVE repo permission via /collaborators/{login}/permission and accepts the override only if it is in authorized_permissions. The overrider, the source, the timestamp and the reason are printed in the gate log and the job summary.",
|
||||
"_where_to_post_it": "EITHER a PR REVIEW body (type 'Comment') OR the ordinary PR comment box. Prefer the REVIEW: measured 2026-10-02 on PR #38, a well-formed override in the comment box was INVISIBLE to CI, because /issues/{n}/comments sits behind Gitea's *issue* token scope and SETTINGS_API_TOKEN is minted [read:repository, read:admin]. Gitea returned an error object rather than an array, and the stage reported 'no approving review' -- a permission problem wearing the costume of a missing override. /pulls/{n}/reviews is under read:repository and was readable in the same run, so a review body is the route that works on today's token. infra/gitea/set-ci-token.sh now also requests read:issue, which makes the comment box work -- but only after that script is RE-RUN, since an existing token's scopes cannot be widened in place. The stage accepts both sources and names which one it used, so neither route is load-bearing alone.",
|
||||
"_self_override_is_allowed": "The author may override their own PR if they hold an authorized permission. That is the entire point -- with one human there is no one else -- but it means the 'a change to the gate cannot be gated by the gate' protection is, for an administrator, advisory rather than structural. It is traded for an auditable record: the waiver is an explicit act, attributed, with a stated reason, in the PR's comment history and the gate output.",
|
||||
"_known_consequence_agent_accounts": "ACCEPTED, EYES OPEN (operator decision, 2026-10-01). Authorisation is by live permission ALONE, with no allowlist of human accounts. The agent account `gitea-admin` reports permission `owner` on this repo (measured 2026-10-01), so an AGENT can clear the sensitive-path review on its own PR by posting the directive. Agents author most PRs here, so this is the broad case, not the edge case. To narrow it later without changing any logic, add an `override_admins` allowlist here and require membership in addition to the permission check -- ci/approval-policy.sh reads both from this file.",
|
||||
"_staleness": "An override is dropped unless its comment is NEWER than the PR head commit, mirroring Gitea's own dismiss_stale_approvals. An override earned at commit A therefore does not silently cover commit B; push again and you must override again."
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,9 @@
|
||||
{
|
||||
"_comment": "DECLARED desired state of Gitea branch protection on main. ci/gitea-settings.sh fails the merge gate when the LIVE Gitea diverges from this. Gitea settings live in Gitea database - invisible to Flux, unreadable by every other check - so this file plus that check are the only drift detection that exists for them.",
|
||||
"_generated": "2026-09-29 from the live rule, after the session that found protection OFF and zero runners for a month",
|
||||
"_amended": "2026-10-01: required_approvals 1 -> 0. See _why_required_approvals_is_zero.",
|
||||
"_why_required_approvals_is_zero": "The GATE is the approver (operator intent, 2026-09-29). required_approvals: 1 was never consistent with that intent: ci/approval-policy.sh's header and ci/gate.sh:194 both already described the model as 'required_approvals: 0 and the GATE decides', and infra/gitea/require-gate.sh's own closing note named the consequence of 1 -- 'Gitea will not let an author approve their own PR'. With exactly ONE collaborator (jakman85, owner), that made every PR the operator opened permanently unmergeable. Measured 2026-10-01: PR #35, gate green in 1m41s, zero reviews, no path to a merge. decisions/2026-09-30-flux-deploy-notification-inhibited.md had already flagged the 1-vs-0 disagreement as unresolved; this settles it at 0.",
|
||||
"_what_still_holds_at_zero": "This drops the APPROVAL COUNT only. block_admin_merge_override stays TRUE and enable_status_check stays TRUE, so the required gate context still binds absolutely -- nobody, administrator included, can merge a red gate. That distinction is the whole reason this is the chosen fix rather than block_admin_merge_override: false, which would have handed admins a bypass over the gate itself and re-opened what infra/gitea/block-admin-override.sh was written to close (PRs #5-#9, merged with zero approvals). Architecturally and security-relevant paths are still reviewed, by ci/approval-policy.sh rather than by this field.",
|
||||
"branch": "main",
|
||||
"runners_online_min": 1,
|
||||
"protection": {
|
||||
@@ -11,7 +14,7 @@
|
||||
"status_check_contexts": [
|
||||
"gate / gate (pull_request)"
|
||||
],
|
||||
"required_approvals": 1,
|
||||
"required_approvals": 0,
|
||||
"enable_approvals_whitelist": false,
|
||||
"block_admin_merge_override": true,
|
||||
"block_on_rejected_reviews": true,
|
||||
|
||||
@@ -34,7 +34,7 @@ jq -n '{
|
||||
enable_push: false,
|
||||
enable_status_check: true,
|
||||
status_check_contexts: ["gate / gate (pull_request)"],
|
||||
required_approvals: 1,
|
||||
required_approvals: 0,
|
||||
block_on_rejected_reviews: true,
|
||||
dismiss_stale_approvals: true,
|
||||
block_on_outdated_branch: false,
|
||||
@@ -48,11 +48,21 @@ api "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/branches/main" \
|
||||
|
||||
cat <<'NOTE'
|
||||
|
||||
NOTE ON required_approvals: 1 -- Gitea will not let an author approve their own PR.
|
||||
* Agent-opened PRs (author gitea-admin) are approved by jakman85. Fine.
|
||||
* PRs YOU open as jakman85 need gitea-admin to approve. If that is awkward, either open
|
||||
agent work as gitea-admin and review as jakman85, or set required_approvals back to 0
|
||||
and rely on the status check alone.
|
||||
NOTE ON required_approvals: 0 -- CHANGED FROM 1 ON 2026-10-01, and this script is the reason
|
||||
the change had to land here too: re-running it with 1 would silently re-impose the lock-out.
|
||||
|
||||
The note that used to sit here called the problem and its own fix: "Gitea will not let an
|
||||
author approve their own PR ... if that is awkward ... set required_approvals back to 0 and
|
||||
rely on the status check alone." It was awkward. With jakman85 the ONLY collaborator, every PR
|
||||
the operator opened was unmergeable -- measured on PR #35, gate green in 1m41s, zero reviews,
|
||||
nothing that could ever satisfy the rule. So: 0, and the status check alone, as written.
|
||||
|
||||
What did NOT change, deliberately:
|
||||
* block_admin_merge_override stays TRUE. This drops the approval COUNT; it does not hand
|
||||
anyone a bypass over the gate. A red gate is still unmergeable by everybody.
|
||||
* Architecturally/security-relevant paths are still human-reviewed -- by ci/approval-policy.sh
|
||||
(sensitive_paths in infra/gitea/approval-policy.json), which an administrator can waive
|
||||
only by posting an explicit, attributed `/approve-override <reason>` comment.
|
||||
|
||||
Then prove it, rather than trusting the API:
|
||||
git commit --allow-empty -m probe && git push origin main # EXPECT: rejected
|
||||
|
||||
@@ -5,9 +5,23 @@
|
||||
# The token value is NEVER printed. It goes from the token API into the secrets API and
|
||||
# nowhere else -- not to stdout, not to a file, not into an agent transcript.
|
||||
#
|
||||
# Scope: `read:repository` + `read:admin` are what the check needs (the rule object, and the
|
||||
# runner list). Gitea token scopes are coarse; grant the least that works on your version and
|
||||
# confirm the check goes green rather than assuming the scope name.
|
||||
# Scope: `read:repository` + `read:admin` are what ci/gitea-settings.sh needs (the rule object,
|
||||
# and the runner list). `read:issue` is for ci/approval-policy.sh, and it was MISSING — see below.
|
||||
# Gitea token scopes are coarse; grant the least that works on your version and confirm the check
|
||||
# goes green rather than assuming the scope name.
|
||||
#
|
||||
# WHY read:issue IS HERE (added 2026-10-02, measured on PR #38). ci/approval-policy.sh reads the
|
||||
# PR's comments to find an admin `/approve-override`. /issues/{n}/comments sits behind the *issue*
|
||||
# scope, which this script did not request — so Gitea answered with an error OBJECT instead of an
|
||||
# array, and a perfectly well-formed override posted in the PR comment box was invisible to CI.
|
||||
# The failure did not look like a permission problem from the outside; it looked like the override
|
||||
# had not been posted. `/pulls/{n}/files` and `/pulls/{n}/reviews` are under read:repository, which
|
||||
# is why every other part of the same stage worked and this one did not.
|
||||
#
|
||||
# The stage now accepts an override from a REVIEW body as well, so it still functions on a token
|
||||
# WITHOUT read:issue. This scope is what makes the ordinary comment box work too — the thing
|
||||
# anybody would reach for first. **Re-run this script to pick it up; an existing token's scopes
|
||||
# cannot be widened in place.**
|
||||
set -euo pipefail
|
||||
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
|
||||
# NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
|
||||
@@ -21,7 +35,7 @@ basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
|
||||
|
||||
echo "==> minting token '${NAME}' as ${U}"
|
||||
TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
|
||||
-d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\"]}" \
|
||||
-d "{\"name\":\"${NAME}\",\"scopes\":[\"read:repository\",\"read:admin\",\"read:issue\"]}" \
|
||||
"https://${HOST}/api/v1/users/${U}/tokens" | jq -r '.sha1 // empty')"
|
||||
[ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
|
||||
echo "==> got a ${#TOKEN}-character token (value not shown)"
|
||||
|
||||
@@ -20,6 +20,30 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
|
||||
its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
|
||||
Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.
|
||||
|
||||
## Public target + in-cluster DDNS (2026-10-03)
|
||||
|
||||
Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
|
||||
|
||||
```
|
||||
plex.home.keithjackson.dev CNAME wan.home.keithjackson.dev <- external-dns (Gateway `target` annotation)
|
||||
wan.home.keithjackson.dev A <current WAN IP> <- cloudflare-ddns (ddns.yaml), every 5m
|
||||
```
|
||||
|
||||
- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
|
||||
applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
|
||||
nothing.
|
||||
- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
|
||||
It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
|
||||
`egress-class: vpn`.
|
||||
- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
|
||||
(split-horizon) and never use the public path.
|
||||
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
|
||||
OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
|
||||
the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
|
||||
|
||||
Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
|
||||
`kubectl -n external-dns logs deploy/cloudflare-ddns`.
|
||||
|
||||
## Secrets
|
||||
|
||||
Cloudflare API token referenced **by name** (`cloudflare-api-token`, shared with cert-manager),
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
# In-cluster dynamic DNS for the home WAN (2026-10-03).
|
||||
#
|
||||
# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
|
||||
# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
|
||||
# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
|
||||
# moves all of them through this one record, with no Git change and no external-dns involvement.
|
||||
#
|
||||
# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
|
||||
# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
|
||||
# Conversely this updater manages only the names in DOMAINS and never touches external-dns's records.
|
||||
#
|
||||
# IP DETECTION. Default `cloudflare.trace`: an HTTPS request to Cloudflare's trace endpoint, which
|
||||
# reports the address the request arrived from. Correct here only because this pod is NOT in the
|
||||
# `vpn` egress class — the CiliumEgressGatewayPolicy (platform/vpn-egress) selects pods by
|
||||
# `homelab.keithjackson.dev/egress-class: vpn` alone, cluster-wide. Labelling this pod `vpn` would
|
||||
# publish the VPN exit IP instead of the WAN IP.
|
||||
#
|
||||
# TOKEN. Reuses this namespace's `cloudflare-api-token` (same Zone:DNS:Edit token as external-dns
|
||||
# and cert-manager) — no new secret.
|
||||
#
|
||||
# TIER NOTE. This lives in the gating `platform` Flux Kustomization (wait: true) next to
|
||||
# external-dns. If it ever crashloops, `platform` goes not-Ready and apps stop reconciling (no
|
||||
# outage). Kept here rather than in its own non-gating Kustomization to avoid a clusters/** change.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: cloudflare-ddns
|
||||
namespace: external-dns
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflare-ddns
|
||||
spec:
|
||||
replicas: 1 # exactly one writer for the record
|
||||
revisionHistoryLimit: 3
|
||||
strategy:
|
||||
type: Recreate # never two updaters at once
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: cloudflare-ddns
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflare-ddns
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: cloudflare-ddns
|
||||
image: favonia/cloudflare-ddns:1.17.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
env:
|
||||
- name: CLOUDFLARE_API_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: cloudflare-api-token
|
||||
key: api-token
|
||||
- name: DOMAINS
|
||||
value: "wan.${BASE_DOMAIN}"
|
||||
- name: IP6_PROVIDER
|
||||
value: "none" # IPv4 WAN only; leaves any AAAA alone
|
||||
- name: PROXIED
|
||||
value: "false" # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
|
||||
- name: UPDATE_CRON
|
||||
value: "@every 5m"
|
||||
- name: DELETE_ON_STOP
|
||||
value: "false" # a pod restart must not blank the record every CNAME depends on
|
||||
- name: RECORD_COMMENT
|
||||
value: "managed by k3s cloudflare-ddns (platform/externaldns/ddns.yaml)"
|
||||
- name: EMOJI
|
||||
value: "false"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 5m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
memory: 64Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
@@ -81,38 +81,29 @@ spec:
|
||||
# what this instance can see or own; per-resource exclusion only ever REMOVES an object from
|
||||
# this instance's view, so it cannot make prod capable of deleting a record it does not own.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
|
||||
# THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
|
||||
#
|
||||
# The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
|
||||
# that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
|
||||
# actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
|
||||
# this change, for two independent reasons:
|
||||
# Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
|
||||
# RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
|
||||
# fix, and both still hold:
|
||||
# * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
|
||||
# from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
||||
# gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
|
||||
# shared Gateway therefore moves EVERY gateway-sourced record at once.
|
||||
# * The home WAN IP is residential and can change, so a literal IP in Git would rot.
|
||||
#
|
||||
# 1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
|
||||
# (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
|
||||
# points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
|
||||
# certificate. Whether prod's public services belong behind it, behind a new forward, or
|
||||
# are not meant to be public at all is an operator decision. Breaking five services' DNS
|
||||
# on a guess is worse than leaving a known-wrong-but-stable state.
|
||||
# 2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
|
||||
# real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
|
||||
# GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
||||
# gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
|
||||
# no per-service target override while every public service shares the `shared` Gateway.
|
||||
# Annotating the Gateway would move EVERY gateway-sourced record, which after this change
|
||||
# still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
|
||||
# routable address for admin UIs. That is strictly worse than the status quo.
|
||||
#
|
||||
# SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
|
||||
# is answered, because a knob that cannot take effect invites someone to set it and expect one:
|
||||
# (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
|
||||
# port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
|
||||
# (b) HOW should they be separated from the non-public names that share the Gateway? Options,
|
||||
# roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
|
||||
# annotation, with only the public services' routes attached; or publishing the five names
|
||||
# from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
|
||||
# which restores per-service targeting without a second Gateway.
|
||||
# Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
|
||||
# The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
|
||||
# UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
|
||||
# (b) the address is kept current IN-CLUSTER. Implementation:
|
||||
# * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
|
||||
# each published name becomes a CNAME to that one hostname;
|
||||
# * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
|
||||
# record at the current WAN IP. That record has no external-dns TXT registry entry, so this
|
||||
# instance never owns or garbage-collects it.
|
||||
# Internal services are unchanged: their routes carry `controller: none` (above), which is
|
||||
# checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
|
||||
# LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
|
||||
# LB VIP. Whether those should go public is a separate, still-open question.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
|
||||
#
|
||||
|
||||
@@ -8,3 +8,5 @@ resources:
|
||||
# SOPS-encrypted Cloudflare token (env CF_API_TOKEN) — SAME token as cert-manager, second copy in
|
||||
# the external-dns namespace. Flux decrypts at reconcile.
|
||||
- cloudflare-api-token.sops.yaml
|
||||
# DDNS for `wan.${BASE_DOMAIN}` — the CNAME target every Gateway-published record points at.
|
||||
- ddns.yaml
|
||||
@@ -7,6 +7,23 @@ kind: Gateway
|
||||
metadata:
|
||||
name: shared
|
||||
namespace: gateway
|
||||
annotations:
|
||||
# PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
|
||||
# route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
|
||||
# unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
|
||||
# becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
|
||||
# pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
|
||||
#
|
||||
# Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
|
||||
# admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
|
||||
# `exposure: public` apps. `internal` services stay unpublished via their
|
||||
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
|
||||
# BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
|
||||
# from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
|
||||
#
|
||||
# LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
|
||||
# ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
|
||||
external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
|
||||
spec:
|
||||
gatewayClassName: cilium
|
||||
listeners:
|
||||
|
||||
+12
-8
@@ -153,14 +153,18 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
plex.tv/claim — so it is a timed sequence, scripted as `infra/plex/claim-ship.sh`:
|
||||
token → `sops` → PR → gate → merge → **forced** `flux reconcile` → pod roll (~2m45s measured).
|
||||
Includes why a changed Secret does not restart anything, and why an expired token is harmless.
|
||||
- **`plex-quicksync`** — hardware transcoding on the host iGPU. GitOps puts `/dev/dri` in the pod
|
||||
(`hardware.gpu: intel-quicksync`); **a human still has to tick Settings → Transcoder → "Use hardware
|
||||
acceleration when available"**, and no environment variable can do it (the image supports no
|
||||
`PLEX_PREFERENCE*`, verified). The page names the real preference key (`HardwareAcceleratedCodecs`,
|
||||
read out of the binary), says which widely-cited key **does not exist**, and is explicit that the
|
||||
**only** proof is `Transcode (hw)` on the Plex dashboard — Plex renders no ServiceMonitor, so no
|
||||
alert could ever fire. Also: the device-cgroup failure to expect first and its device-plugin
|
||||
fallback, and why a `:latest` pull can silently break it.
|
||||
- **`plex-quicksync`** — hardware transcoding on the host iGPU. **Confirmed working 2026-10-02**
|
||||
(VAAPI decode *and* encode, zero-copy). GitOps requests `gpu.intel.com/i915` on the pod
|
||||
(`hardware.gpu: intel-quicksync`, satisfied by the `intel-gpu-plugin` DaemonSet — a `/dev/dri`
|
||||
hostPath was measured not to work: device-cgroup `EPERM`); **a human still has to tick Settings →
|
||||
Transcoder → "Use hardware acceleration when available"**, and no environment variable can do it
|
||||
(the image supports no `PLEX_PREFERENCE*`, verified). **The proof is `Plex Media Server.log`'s
|
||||
`final decoder: / final encoder:` line, not the Plex dashboard** — the dashboard showed nothing
|
||||
while hardware transcoding was working, so its silence proves nothing; the page carries the
|
||||
redacting grep (`X-Plex-Token` lives in that log), the WRONG log file to avoid, and two more
|
||||
documented false leads (an empty `/proc/*/fd` scan, and a missing `HardwareAcceleratedCodecs` key).
|
||||
Plex renders no ServiceMonitor, so no alert could ever fire. Also: **`Pending` stalls the whole app
|
||||
tier** and its one-field escape hatch, and why a `:latest` pull can silently break it.
|
||||
- **`oauth2-proxy-session`** — a gated UI loads but its saves quietly do nothing, and you are never
|
||||
re-prompted to log in. Tells the three stacked auth layers apart from the sidecar's own access log
|
||||
(the four lines that matter, including the upstream `-` that means the proxy answered a request the
|
||||
|
||||
+194
-55
@@ -1,5 +1,10 @@
|
||||
# plex-quicksync — making Plex actually use Intel Quick Sync, and proving it did
|
||||
|
||||
**Status: hardware transcoding is CONFIRMED WORKING on prod as of 2026-10-02** — VAAPI for both
|
||||
decode *and* encode, zero-copy, measured in Plex's own server log (see "How to verify"). The manual
|
||||
toggle has been ticked. This page is therefore now mostly a **verification and regression**
|
||||
procedure, and its old verification section was **wrong** — see the correction at the top of
|
||||
"How to verify".
|
||||
**Alert:** none, and none is possible — see "How to verify" below. This is an **operator procedure**,
|
||||
not an incident page. It is indexed here because the feature it completes is **inert until a human
|
||||
clicks one button**, and a switched-off feature that looks shipped is exactly what this page exists
|
||||
@@ -33,7 +38,7 @@ is in git**:
|
||||
| Half | Who does it | State |
|
||||
|---|---|---|
|
||||
| The render device is inside the container **and openable** | **GitOps.** `hardware.gpu: intel-quicksync` in `catalog/plex.yaml` renders `resources.limits: {gpu.intel.com/i915: 1}` on the app container. The `intel-gpu-plugin` DaemonSet (`platform/intel-gpu-plugin/`) injects the device **and the matching device-cgroup allow rule**. Flux applies it. | Done by merging that change. |
|
||||
| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **NOT done.** `HardwareAcceleratedCodecs` was **absent** from `Preferences.xml` when this page was written — the toggle has never been on. |
|
||||
| Plex is *told* to use it | **The operator, by hand, in the Plex web UI.** | **Done, 2026-10-02**, and confirmed from the **log** rather than from a preference key. `HardwareAcceleratedCodecs` is **still absent** from `Preferences.xml` with hardware transcoding working, so its absence proves nothing — see first-command #4. |
|
||||
|
||||
**There is no GitOps path to the second half, and that is verified rather than assumed.** A bounded
|
||||
grep across the running image's entire init surface — `/etc/cont-init.d`, `/etc/services.d`,
|
||||
@@ -79,8 +84,8 @@ also the only thing that produces the cgroup allow rule. There is no graceful-de
|
||||
The one improvement is that the loss is no longer silent.
|
||||
|
||||
What this does **not** do, so nobody reads more into a green gate than is there: it does not enable
|
||||
hardware transcoding (that is the manual toggle below), a green gate only proves the **manifest
|
||||
renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
|
||||
hardware transcoding (that is the manual toggle below — done 2026-10-02), a green gate only proves
|
||||
the **manifest renders**, and it writes no `nodeSelector` — the resource itself constrains Plex to nodes the plugin
|
||||
advertises, which is the scheduler's doing and not a selector in this repo.
|
||||
|
||||
## Running the commands in this runbook
|
||||
@@ -172,17 +177,28 @@ cgroup rule that uid 0 could not buy its way past either. `RW-OPEN-DENIED` with
|
||||
visible is the **device-cgroup** failure — i.e. the plugin's allow rule did not arrive; see "Likely
|
||||
causes" #2.
|
||||
|
||||
**4. Is the toggle on?** Named key only — never the whole file:
|
||||
**4. Is the toggle on?** **You cannot answer this from `Preferences.xml` alone.** That mistake is on
|
||||
this page's record: it told readers to treat a missing `HardwareAcceleratedCodecs` as "never
|
||||
enabled", and on 2026-10-02 that key was **absent from all 27 attributes** while hardware
|
||||
transcoding was demonstrably working. Named keys only — never the whole file, it holds
|
||||
`PlexOnlineToken`:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||
grep -o "HardwareAcceleratedCodecs=\"[01]\"" \
|
||||
"/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo ABSENT'
|
||||
grep -o -e "HardwareAcceleratedCodecs=\"[01]\"" -e "HardwareDevicePath=\"[^\"]*\"" \
|
||||
"/config/Library/Application Support/Plex Media Server/Preferences.xml" || echo NEITHER-KEY'
|
||||
```
|
||||
|
||||
`ABSENT` ⇒ never enabled (the state this page was written in). `="0"` ⇒ explicitly off. `="1"` ⇒ on,
|
||||
and the remaining question is whether it *works* — go to "How to verify".
|
||||
| Reading | What it actually means |
|
||||
|---|---|
|
||||
| `HardwareDevicePath="…"` **present** | **Positive signal.** Plex writes this when hardware acceleration is configured; it was present on 2026-10-02 alongside a working VAAPI pipeline. |
|
||||
| `HardwareAcceleratedCodecs` **absent** | **Nothing. Neither on nor off.** Measured absent with the feature working (2026-10-02). Do not diagnose from it, and do not write it into the file to "fix" it. |
|
||||
| `HardwareAcceleratedCodecs="0"` | explicitly **off** — the only conclusive reading, and only in the negative. |
|
||||
| `HardwareAcceleratedCodecs="1"` | on. |
|
||||
|
||||
**The log settles it, this file does not.** Go to "How to verify"; the `hardware transcoding:
|
||||
enabled` and `final decoder/final encoder` lines are the verdict.
|
||||
|
||||
## The one manual step
|
||||
|
||||
@@ -191,8 +207,11 @@ Plex web UI → **Settings → Transcoder → "Use hardware acceleration when av
|
||||
|
||||
The underlying preference key is **`HardwareAcceleratedCodecs`**. That name was confirmed by
|
||||
extracting strings from the 23 MB `Plex Media Server` binary in the running container — **not** from
|
||||
recall and not from a forum post. Real neighbouring keys from the same binary, so you can recognise
|
||||
them and not invent them:
|
||||
recall and not from a forum post. **But do not expect to see it in `Preferences.xml` after ticking
|
||||
the box:** on 2026-10-02, with the toggle on and hardware transcoding working, the key was absent and
|
||||
`HardwareDevicePath` was what had been written. The key exists in the binary; Plex does not
|
||||
necessarily persist it. Real neighbouring keys from the same binary, so you can recognise them and
|
||||
not invent them:
|
||||
|
||||
| Key | What it is |
|
||||
|---|---|
|
||||
@@ -209,42 +228,121 @@ binary. Recorded explicitly so nobody spends an evening hunting for it or, worse
|
||||
Editing `Preferences.xml` by hand is **not** the procedure. It is app state on the replicated Longhorn
|
||||
tier, it is in no git commit, and the UI writes the key correctly. Use the UI.
|
||||
|
||||
## How to verify — and why there is no alert
|
||||
## How to verify — the log, not the dashboard
|
||||
|
||||
Start a playback that **forces a transcode** (a client that cannot direct-play the source, or pick a
|
||||
lower quality explicitly), then read Plex's own **Dashboard → Activity**:
|
||||
**Correction, 2026-10-02.** Until today this page said the Plex dashboard showing `Transcode (hw)`
|
||||
was "the *only* proof available, and nothing else will ever tell you". **That was false and it cost
|
||||
the operator real time:** hardware transcoding was fully working — VAAPI decode *and* encode, with
|
||||
zero-copy — while **Dashboard → Activity showed nothing of the sort**. A check that is silent while
|
||||
the feature works does not just fail to prove anything; it sends someone hunting a fault that does
|
||||
not exist. The authoritative check is **Plex's own server log**.
|
||||
|
||||
- **`Transcode (hw)`** on the session ⇒ hardware transcoding. **This is the proof.**
|
||||
- plain **`Transcode`** ⇒ software. The feature is not working, whatever the manifest says.
|
||||
### The primary check — run this one
|
||||
|
||||
**That dashboard is the *only* proof available, and nothing else will ever tell you.**
|
||||
`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
|
||||
there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open work
|
||||
to "alert on software-transcode fallback": with no metric behind it such a rule would be permanently
|
||||
inert, and this repo has already been burned at scale by rules that evaluated against nothing while
|
||||
looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence of"). If that
|
||||
gap is ever worth closing, the honest order is a real exporter first and a rule second.
|
||||
```
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||
grep -aiE "hardware transcoding: (final|using)" \
|
||||
"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log" \
|
||||
| sed -E "s/(X-Plex-Token|token)=[^&[:space:]]*/\1=REDACTED/g" | tail -5'
|
||||
```
|
||||
|
||||
Corroborating, not a substitute — the server log on the config PVC:
|
||||
| Line | Verdict |
|
||||
|---|---|
|
||||
| `hardware transcoding: final decoder: vaapi, final encoder: vaapi` | **PASS.** Hardware in **both** directions. This is the pass condition. |
|
||||
| `hardware transcoding: using zero-copy transcoding` | Best case — frames never round-trip through the CPU. Observed on prod 2026-10-02. |
|
||||
| `hardware transcoding: final decoder: , final encoder: ` (both empty) | It **fell back to software** for that session. |
|
||||
| no matching line at all | No transcode has been attempted since the log rolled — force one (below), then look again. Absence is not a failure. |
|
||||
|
||||
The 2026-10-02 reading on prod, in full, for the shape to recognise:
|
||||
|
||||
```
|
||||
Codecs: hardware transcoding: testing API vaapi for device '/dev/dri/renderD128'
|
||||
(Intel CoffeeLake-S GT2 [UHD Graphics 630])
|
||||
hardware transcoding: using hardware decode accelerator vaapi
|
||||
hardware transcoding: zero-copy support present
|
||||
hardware transcoding: using zero-copy transcoding
|
||||
hardware transcoding: final decoder: vaapi, final encoder: vaapi
|
||||
```
|
||||
|
||||
To make a line appear, start a playback that **forces a transcode**: a client that cannot
|
||||
direct-play the source, or an explicitly lower quality. Direct Play / Direct Stream never reaches
|
||||
the transcoder and logs nothing.
|
||||
|
||||
### Three things to get right about that command
|
||||
|
||||
1. **Always keep the `sed`.** `Plex Media Server.log` carries API URLs containing
|
||||
**`X-Plex-Token=…`**, a live plex.tv credential. Any output that might land in a ticket, a
|
||||
transcript or a chat window must be redacted — CLAUDE.md §6, and a credential has already been
|
||||
leaked into a transcript in this repo exactly this way. The redacting form above is the form to
|
||||
copy; do not "simplify" the pipe out of it.
|
||||
2. **`Plex Transcoder Statistics.log` is the WRONG FILE.** The name is irresistible and someone will
|
||||
reach for it. It holds XML `<Variant .../>` session records — **not** ffmpeg/VAAPI output. There
|
||||
is no decoder/encoder verdict anywhere in it. The file you want is `Plex Media Server.log`.
|
||||
3. `grep -a` is deliberate: the log carries occasional non-UTF-8 bytes, and without `-a` GNU grep
|
||||
reports "binary file matches" and prints nothing — which reads exactly like "no hardware
|
||||
transcoding".
|
||||
|
||||
### Secondary and unreliable: the Plex dashboard
|
||||
|
||||
**Dashboard → Activity** *may* label a transcoding session `Transcode (hw)`. If it does, it agrees
|
||||
with the log and you are done. **If it does not, that proves nothing at all** — measured 2026-10-02:
|
||||
hardware transcoding active and zero-copy, dashboard silent. Never conclude "software transcoding"
|
||||
from the dashboard and never open an investigation on its say-so. Kept here only because it is the
|
||||
fastest glance available; it is a convenience, **not evidence**.
|
||||
|
||||
### Process-level, while a transcode is in flight: the transcoder's argv
|
||||
|
||||
This shows Plex *requesting* hardware, as distinct from reporting it afterwards. Get a shell in the
|
||||
pod (if `-it` has no TTY under `dev.sh run`, wrap each line in `sh -c "…"` instead):
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex exec deploy/plex -- \
|
||||
sh -c "grep -i -m20 -e hardware -e vaapi -e hwdevice \
|
||||
\"/config/Library/Application Support/Plex Media Server/Logs/Plex Media Server.log\""'
|
||||
'kubectl --context k3s-prod -n plex exec -it deploy/plex -- sh'
|
||||
```
|
||||
|
||||
Treat those strings as advisory: they are Plex's, not this repo's, and they move between versions.
|
||||
Absence of a match is **not** evidence of software transcoding; a VA-API device open is good evidence
|
||||
of hardware. The dashboard remains the verdict.
|
||||
then, inside the pod:
|
||||
|
||||
```
|
||||
ps -eo pid,user,comm | grep -i transcoder
|
||||
tr '\0' ' ' < /proc/<transcoder-pid>/cmdline | grep -oE '(-hwaccel[a-z_]*|vaapi|renderD[0-9]+)'
|
||||
```
|
||||
|
||||
`-hwaccel`, `-hwaccel_device`, `vaapi`, `renderD128` present ⇒ Plex is asking for hardware. Keep the
|
||||
`grep -oE`: the raw argv contains media paths and session identifiers you do not want in a
|
||||
transcript. This only works while a session is actually transcoding.
|
||||
|
||||
### Documented false lead: scanning `/proc/*/fd` for a holder of `renderD128`
|
||||
|
||||
**It returns nothing even when hardware transcoding is working.** Measured 2026-10-02: a
|
||||
`/proc/*/fd` sweep inside the pod found **no** process holding `/dev/dri/renderD128`, while the log
|
||||
showed `final decoder: vaapi, final encoder: vaapi` for sessions either side of the scan. Transcode
|
||||
sessions are **short** and the scan landed in a gap between two of them. An empty fd scan is a
|
||||
**timing artefact, not a finding** — do not report it as evidence of software transcoding, and do
|
||||
not escalate on it. Use the argv check above (which has the same timing constraint but tells you
|
||||
something when it does catch a session), or just read the log, which is retrospective and therefore
|
||||
does not care about timing.
|
||||
|
||||
### Why there is no alert — this half was always right
|
||||
|
||||
`catalog/plex.yaml` declares no `metricsEndpoint`, so Plex renders **no ServiceMonitor at all** —
|
||||
there is no scrape, no transcode-mode metric, and therefore no alert that could fire. Do not open
|
||||
work to "alert on software-transcode fallback": with no metric behind it such a rule would be
|
||||
permanently inert, and this repo has already been burned at scale by rules that evaluated against
|
||||
nothing while looking like coverage (`runbooks/README.md`, "Alerts you should not trust the silence
|
||||
of"). If that gap is ever worth closing, the honest order is a real exporter first and a rule
|
||||
second.
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
When the dashboard still says plain `Transcode`:
|
||||
When the **log** says it fell back to software — `final decoder: , final encoder: ` with both
|
||||
fields empty. (Not when the dashboard is quiet: the dashboard's silence means nothing, see "How to
|
||||
verify".)
|
||||
|
||||
1. **The toggle is off.** Overwhelmingly the most likely answer, and the reason this page exists.
|
||||
First-command #4 returns `ABSENT` or `="0"`. Remedy: the manual step above. Check this **before**
|
||||
investigating anything technical.
|
||||
1. **The toggle is off.** Historically the most likely answer and the reason this page exists. It was
|
||||
ticked on 2026-10-02 and should stay ticked, but a `/config` restore or a stray UI save can unset
|
||||
it. Remedy: the manual step above. **Do not diagnose this from a missing
|
||||
`HardwareAcceleratedCodecs`** — see first-command #4; only `="0"` is conclusive, and the log line
|
||||
`hardware transcoding: enabled` is the real signal that the toggle took.
|
||||
|
||||
2. **The device cannot be `open()`ed — the plugin's cgroup allow rule did not arrive.** Symptom:
|
||||
first-command #3 shows `renderD128` present and correctly owned, and the open returns
|
||||
@@ -276,13 +374,27 @@ When the dashboard still says plain `Transcode`:
|
||||
4. **The image changed under you.** See "The floating-image caveat". A `:latest` pull can remove the
|
||||
mechanism this feature leans on with no commit in this repo.
|
||||
|
||||
5. **The session was never transcoding.** Direct Play / Direct Stream never touches the transcoder, so
|
||||
`Transcode (hw)` cannot appear. Confirm the session actually says `Transcode` before treating it as
|
||||
a hardware failure.
|
||||
5. **The session was never transcoding.** Direct Play / Direct Stream never reaches the transcoder,
|
||||
so no `hardware transcoding:` line is written for it at all. An *absent* line is this, not a
|
||||
failure — confirm a transcode actually happened before treating anything as a hardware fault.
|
||||
|
||||
6. **The source needs a codec Quick Sync on this generation cannot handle** — e.g. AV1 decode, which
|
||||
Coffee Lake's UHD 630 has no engine for. Plex falls back to software *per stream*. Symptom: some
|
||||
sessions show `(hw)` and others do not. Not a fault.
|
||||
6. **The *source* codec may be one this iGPU cannot decode — HYPOTHESIS, not a confirmed
|
||||
diagnosis.** The symptom that points here is fallback on **particular titles** while other
|
||||
sessions are hardware. Measured on prod 2026-10-02, *between* two fully hardware sessions, one
|
||||
session logged:
|
||||
|
||||
```
|
||||
hardware transcoding: enabled, but no hardware decode accelerator found
|
||||
hardware transcoding: final decoder: , final encoder:
|
||||
```
|
||||
|
||||
The platform was demonstrably working either side of it, so the likely explanation is the **file**,
|
||||
not the cluster: UHD 630's media engine decodes H.264, HEVC 8-bit and 10-bit, and VP9 — but **not
|
||||
AV1**, and not 10-bit 4:2:2 HEVC. **This was NOT verified against that particular title.** It is
|
||||
inferred from the hardware's capability set and is written here as the lead to check first, not as
|
||||
a finding. So: fallback on *all* files is a platform problem (causes 1-4); fallback on *some*
|
||||
files, check the source codec before suspecting the device, and expect occasional legitimate
|
||||
per-title fallbacks to be normal rather than a fault.
|
||||
|
||||
## Plex will not schedule
|
||||
|
||||
@@ -360,19 +472,30 @@ infra/devcontainer/dev.sh run bash -c \
|
||||
```
|
||||
|
||||
`45-plex-hw-transcode-and-connected-tuner` still present ⇒ the group mechanism is still the one
|
||||
described here. Gone or renamed ⇒ stop and re-measure. In particular the "no `supplementalGroups`
|
||||
needed" conclusion in `catalog/plex.yaml` and `schemas/renderer-contract.md` is only true while that
|
||||
hook exists.
|
||||
described here: the hook reads the injected device's **own** ownership, which under the device plugin
|
||||
is `root:root`, so it resolves GID `0` (see "When to escalate or stop"). Gone or renamed ⇒ stop and
|
||||
re-measure. In particular the "no `supplementalGroups` needed" conclusion in `catalog/plex.yaml` and
|
||||
`schemas/renderer-contract.md` is only true while that hook exists.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
- **Do not add `supplementalGroups` or a `podSecurityContext` to `catalog/plex.yaml`.** It would be
|
||||
**inert**, and it would mislead the next person. Measured in-container 2026-09-30: the image's
|
||||
`cont-init.d` hook stats each `/dev/dri` char device and `usermod -a -G`s the `plex` user into its
|
||||
group *inside* the container, and the service then execs via `s6-setuidgid plex`, which **replaces**
|
||||
the process group list from `/etc/group` rather than appending — `Plex Media Server` runs with
|
||||
`Groups: 100 1000`, root's gid `0` (which PID 1 does hold) gone. Anything Kubernetes supplies is
|
||||
**inert**, and it would mislead the next person. The image's `cont-init.d` hook stats each
|
||||
`/dev/dri` char device and `usermod -a -G`s the `plex` user into **that device's** group, inside
|
||||
the container; the service then execs via `s6-setuidgid plex`, which **rebuilds** the process group
|
||||
list from `/etc/group` rather than appending to it. Anything Kubernetes supplies is therefore
|
||||
discarded before Plex starts. `render.py --selftest` asserts that absence deliberately.
|
||||
- **GID `992` is a hostPath-era number — do not expect it, and do not read the root group as a
|
||||
defect.** Under the device plugin the injected `/dev/dri/renderD128` arrives **`root:root`**, so
|
||||
the hook resolves GID **`0`** and adds `plex` to the **root group**. Measured 2026-10-02:
|
||||
`Plex Media Server` runs with `Groups: 0 100 1000`, and `id plex` reports
|
||||
`groups=1000(plex),0(root),100(users)`. Under the old hostPath the device was `root:render`
|
||||
(GID 992 on these nodes) and the reading was `Groups: 44 100 992 1000`. **992 is now irrelevant**:
|
||||
if any page, comment or older handoff tells you to expect it, that text is stale and the system
|
||||
you are looking at is working. The root-group membership is an inherent property of the plugin's
|
||||
`DeviceSpec` — it injects the host device with its host ownership — **not** a defect, and **not** a
|
||||
reason to reintroduce `supplementalGroups`, which would still be a no-op for the reason in the
|
||||
bullet above.
|
||||
- **Do not reach for `privileged: true`.** Kyverno `restrict-host-namespaces` is Enforce and will
|
||||
reject it — and it would not help: `uid 0` failed the same `open()` in the 2026-09-30 measurement,
|
||||
because root does not bypass the cgroup device gate.
|
||||
@@ -390,7 +513,8 @@ hook exists.
|
||||
watch state (`policy: daily-offcluster`). The transcode toggle is one preference, not a reason to
|
||||
lose the DB.
|
||||
- **Stop and get the operator** if the device cannot be opened from the pod as uid 1000 (cause #2
|
||||
confirmed) *under the current mechanism*. The next step is a platform change in
|
||||
confirmed) *under the current mechanism* — note that this open **passed** on 2026-10-02
|
||||
(`RW-OPEN-OK` as uid 1000 via `s6-setuidgid plex`), so a failure now is a regression. The next step is a platform change in
|
||||
`platform/intel-gpu-plugin/`, with its own PR, not more poking at this workload.
|
||||
|
||||
## Known gaps, deliberately not fixed here
|
||||
@@ -413,10 +537,25 @@ Recorded so they are findable, and so nobody assumes they were overlooked:
|
||||
|
||||
---
|
||||
Source of truth: `runbooks/plex-quicksync.md`. Index: `/runbooks/`
|
||||
The node measurements (all three prod nodes) and the in-container image measurements cited here were
|
||||
taken **2026-09-30** against `plexinc/pms-docker` digest `sha256:e0ab27395614…`
|
||||
(`plexmediaserver 1.43.4.10903`). The `EPERM` open measurements, the `Groups: 44 100 992 1000`
|
||||
reading and the `gpu.intel.com/i915: 1` allocatable/DaemonSet readings were taken **2026-10-01**.
|
||||
Hardware transcoding has **still not been observed working**: the open test has not been re-run under
|
||||
the device-plugin mechanism (it cannot be, until that merges and Flux reconciles), and the Plex-side
|
||||
toggle is still unset.
|
||||
|
||||
**Hardware transcoding is CONFIRMED WORKING on prod, measured 2026-10-02.** From
|
||||
`Plex Media Server.log` in the running pod: API `vaapi` on `/dev/dri/renderD128`
|
||||
(`Intel CoffeeLake-S GT2 [UHD Graphics 630]`), `using hardware decode accelerator vaapi`,
|
||||
`zero-copy support present`, `using zero-copy transcoding`, and
|
||||
`final decoder: vaapi, final encoder: vaapi`. Also measured in the same pod: `RW-OPEN-OK` opening
|
||||
`/dev/dri/renderD128` as uid 1000 via `s6-setuidgid plex`; the pod spec carrying
|
||||
`{"limits":{"gpu.intel.com/i915":"1"},"requests":{"gpu.intel.com/i915":"1"}}`; `Groups: 0 100 1000`
|
||||
and `id plex` → `groups=1000(plex),0(root),100(users)`; and `HardwareDevicePath` present in
|
||||
`Preferences.xml` while `HardwareAcceleratedCodecs` was absent from all 27 attributes.
|
||||
|
||||
Earlier readings, kept for provenance and **superseded** where they conflict: the node measurements
|
||||
(all three prod nodes) and the in-container image measurements taken **2026-09-30** against
|
||||
`plexinc/pms-docker` digest `sha256:e0ab27395614…` (`plexmediaserver 1.43.4.10903`); the hostPath
|
||||
`EPERM` open results and the `Groups: 44 100 992 1000` / GID-992 group reading, **2026-10-01, under
|
||||
the hostPath mechanism that no longer exists**; and the `gpu.intel.com/i915: 1` allocatable and
|
||||
`intel-gpu-plugin` DaemonSet readings, **2026-10-01**.
|
||||
|
||||
What this page still does **not** establish: that any particular title's software fallback was caused
|
||||
by its source codec (cause 6 is an unverified hypothesis), and that Plex's dashboard *ever* labels a
|
||||
session `Transcode (hw)` on this deployment — it has not been observed doing so, which is exactly why
|
||||
the log is now the primary check.
|
||||
Reference in new issue
Block a user