95597bd688363744dcb6dd89b378ace6f2c7693a
855
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 jakman85
|
95597bd688 |
Merge pull request 'feat(pm-agent): an in-cluster AI Program Manager that turns Gitea issues into pull requests' (#57) from feat/pm-agent-issue-to-pr into main
Reviewed-on: #57 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
06ca9eeb88 |
Merge pull request 'fix(runbooks): vpn-remote-access step 1 named a nonexistent script; step 2 cited the wrong gitignore rule' (#58) from fix/vpn-runbook-ansible-command into main
Reviewed-on: #58 |
||
 jakman85andClaude Opus 5
|
18f741d744 |
fix(runbooks): vpn-remote-access step 1 named a nonexistent script; step 2 cited the wrong gitignore rule
Both reported by the operator on first use of the runbook merged in #55.
Step 1 said `cd infra/ansible && ./site.sh --tags system --limit k3s_prod`.
There is no site.sh anywhere in this repo. It was wrong four ways at once:
this repo drives Ansible with `ansible-playbook <playbook>` and never a
wrapper (playbooks/site.yml is the full bring-up chain, not this); the module
work is in playbooks/node-baseline.yml; targeting here is `-e k3s_hosts=...`,
not `--limit`; and Ansible exists only in the pinned toolbox, with ansible.cfg
read only from infra/ansible. Replaced with the real command plus why
`-e k3s_hosts=k3s_prod` matters (the default group is decommissioned dev) and
why `--tags system` must never be dropped (an untagged run also executes the
`disks` tasks, which assert on prod's deliberate placeholder devices).
Step 2 told the operator to stage the server key at /work/.cache/server.key
"inside the repo's gitignored cache". .cache/ is NOT gitignored -- the file was
ignored only incidentally by `*.key` (.gitignore:11). So the path was safe for
a reason I had not identified, and renaming it (server.pem, wg-server) would
have left a committable WireGuard private key in the checkout, which
no-plaintext-secrets does not recognise. The step now names the real rule,
marks the suffix as the load-bearing part, and gives a check-ignore
verification to run before trusting it.
Docs only -- no manifest and no behaviour change. Touches no sensitive path.
Also records the post-merge measurements for #55, whose own entry predates the
merge: Flux applied
|
||
|
jakman85andClaude Opus 5
|
867a82d8ad |
docs(pm-agent): correct the handoff — a STALE approval still satisfies the gate
I recorded that merging main in would dismiss the operator's approval and that approval-policy would therefore go red until they re-approved. Measured: wrong. dismiss_stale_approvals is true, so the review did go stale — but ci/approval-policy.sh selects on .state == "APPROVED" and a non-author login and reads nothing about staleness, and required_approvals is 0 so Gitea never required one either. CI run #257 on the merged head: "approved by 'jakman85' (author 'gitea-admin') — sensitive change is human-reviewed", approval-policy pass, gitea-settings pass, required check green, PR mergeable. The 'push again and you must override again' rule applies to the /approve-override directive, which IS staleness-checked against the head commit — not to a review. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
e4a9b02b00 |
docs(pm-agent): CLAUDE.md §0 is already satisfied in a worker — tell it not to make a worktree
PR #56 added §0 (work in your own worktree, never the shared checkout) while this branch was open. A worker agent reads CLAUDE.md first, and §0's instruction is to create a git worktree outside the shared checkout — which does not exist in a worker's container. It holds a fresh clone of its own, made for one assignment and thrown away with the pod, which is what §0 is actually asking for. A worker that obeyed §0 literally would 'git worktree add', edit somewhere the wrapper that started it does not look, and have its work committed by nobody while the issue got a 'changed nothing' report. worker-rules.md now says §0 is satisfied and not to do that. Also records in the handoff entry: the merge and its one conflict, and that this push dismisses the operator's approving review — measured, dismiss_stale_approvals=true — so the approval-policy stage needs them to re-approve. That is theirs to do, not an agent's. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0569d47d49 |
Merge branch 'main' into feat/pm-agent-issue-to-pr
One conflict, in clusters/base/kustomization.yaml: PR #55 (vpn-ingress) and this branch both appended a resource line to the same tail. Resolved by keeping BOTH, with main's vpn-ingress first — it is a platform-config-tier leaf, while pm-agent dependsOn apps-wave-0, which is later. Nothing else conflicted. runbooks/kustomization.yaml auto-merged and carries both the vpn-remote-access and pm-agent entries; the handoff entries are separate files and had nothing to merge, which is the whole point of decisions/ being a directory. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
3ee663dbcc |
Merge pull request 'docs(agents): one workspace per agent run — the shared checkout is off-limits' (#56) from docs/agents-work-in-their-own-workspace into main
Reviewed-on: #56 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
ad3f56eee8 |
Merge pull request 'feat(vpn-ingress): WireGuard remote access on udp/51820 — hostNetwork, because a pod identity would 403 every Gateway hostname' (#55) from feat/vpn-remote-access-wireguard into main
Reviewed-on: #55 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
005e80d38a |
docs(pm-agent): record the PR's own CI result, read from the log not the status tick
The PR gate ran and BLOCKED on approval-policy alone — the sensitive-path rule, with no non-author review — while every technical stage passed, gitea-settings included (the one stage a workstation cannot run). That is the gate working, and it is the one thing an agent must not clear for itself. Also recorded: the commit-status API reported 'pending — Waiting to run' for a run that had already completed, so the log is the source of truth here. §2b again. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8b5e950baa |
test(pm-agent): a committed negative-path probe for the worker's input guards
pm-worker's command is pinned by admission with no args, so everything a caller could influence arrives in the environment — and the guards at the top of that script are what stand between an injected variable and a clone from the wrong host or a push to the wrong ref. Nothing exercised them, which makes them a claim rather than a control. probe-validation.sh asserts 17 refusals, each on BOTH a non-zero exit and the specific message, so a guard that stopped firing cannot pass by failing somewhere else. It covers a redirected git host, a real and reachable WRONG one (the egress policy permits tcp/443, so that case is not hypothetical), the right host with a smuggled path, 'main' and five other bad branch shapes, a malformed owner/repo, a bad mode, a bad issue number, and each missing credential. build-image.sh runs it inside the built image too. Writing it found three things: - the PM_BRANCH != main guard is unreachable while the pattern check above it holds, so the comment claiming it as the defence was wrong about which line does the work. Kept as an assertion independent of the regex; the comment now says so. - the EXIT trap ended in a failing test, which under set -e can replace the script's real exit code with that failure. - a provenance LOG LINE read a file through a redirect that set -o pipefail would have turned into a dead worker if the file were absent. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
2cb891c05d |
docs(agents): one workspace per agent run — the shared checkout is off-limits
Agents have been working in the operator's checkout and clobbering each other. Nothing in the tree told them not to: worktrees appeared in CLAUDE.md only as §3's gate workarounds, and decisions/README.md described worktree isolation as something agents already do rather than as a rule. New CLAUDE.md §0 states the rule, with the commands and an explicit do-not list for the shared checkout. Placed before §1 so every existing §-reference in the repo keeps its meaning. Measured in the shared checkout on 2026-10-05, each row a distinct failure mode: 107 `checkout: moving` reflog entries; sitting on feat/pm-agent-issue-to-pr, not main; two other runs' untracked trees left in it; local main two commits behind origin/main; 42 registered worktrees of which 31 are on already-merged branches; 9 gate scripts carrying a `.claude/` exclusion because an in-tree worktree made find-based discovery count the same manifest up to 18 times. Corrects a claim carried in earlier briefs: `git worktree remove` is NOT blocked by the agent sandbox. Provoked on a throwaway worktree — rc 0. §3 reframed as the normal path and grown from two breakages to three: spec-immutability's skip is avoidable, not merely tolerated, with the spec repo mounted read-only in the same ENGINE shim plus -e SPEC_REPO=/home-network-k3s. CLAUDE.md is a sensitive path, so this PR needs a non-author review or an administrator /approve-override posted as a PR review. Not posted by me: §1 forbids an agent clearing that on its own PR. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
54418363aa |
fix(pm-agent): take the newest comment by list order, not by comparing timestamp strings
last_activity() is the whole trigger: the PM acts iff the newest thing in a thread was not written by it. It found that newest comment by comparing Gitea's RFC 3339 created_at STRINGS, which is only correct while every offset is identical — true today, silently wrong after a server timezone change, and the failure would be the PM answering its own last comment forever. The comments endpoint returns them in creation order, so the last element IS the newest. Using that removes the dependency entirely. Found by self-review, before anything ran. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f92707f38c |
feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers)
Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/** , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c5b5b8672a |
feat(pm-agent): an in-cluster AI Program Manager that turns Gitea issues into pull requests
A service that reads this repository's Gitea issues, decides what to do about each thread as the Program Manager persona, and runs one worker agent per assignment that authors a change, runs the merge gate and opens a pull request. It reacts to issue creation and to comments; the assignment and all correspondence travel through the issue thread and the PR. It authenticates with an Anthropic subscription token. It never pushes to main and never merges. Three choices carry the design, with alternatives recorded in docs/plans/: - it POLLS rather than receiving webhooks (Gitea's webhook SSRF guard, lost deliveries, and no inbound reach needed at all); - it keeps NO state: the trigger is "the newest thing in this thread was not written by me", so the issue thread is the state — no PVC, no backup obligation; - the model holds NO tools: its reply is one JSON object this code validates, so nothing in the path executes text. The boundary is platform/kyverno/admission/pm-agent-job-shape.yaml (Enforce, failurePolicy: Fail), not the RBAC: Kubernetes places no `use` check on serviceAccountName, so "may create Jobs" is "may run arbitrary code as the namespace's most privileged identity". Six rules admit only the committed shape, and the worker identity is bound to nothing with no token mounted, so there is nothing to escalate to. 28/28 offline Kyverno assertions. Nothing has run: the image has never been built or pushed, and both the image digest and the two credentials are deliberate placeholders. See the handoff entry's "What I did NOT verify". Refs CLAUDE.md §1/§3/§6, R-SEC-02, R-GITOPS-02. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
22c94effff |
Merge pull request 'feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)' (#53) from feat/cilium-roll-out-on-config-change into main
Reviewed-on: #53 |
||
|
jakman85andClaude Opus 5
|
d0db63911a |
feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)
A merged Cilium config change did nothing, and everything reported success. The Hubble flow exporter (#50, merged 2026-10-04) never took effect: HelmRelease cilium Ready=True, "Helm upgrade succeeded ... cilium.v6" cilium-config hubble-export-file-path: stdout + the allowlist, PRESENT cilium agent pods all three still PRE-MERGE, startTime 2026-10-04T07:11Z exported flow lines 0, 0, 0 Flux every Kustomization Ready=True Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent reads that ConfigMap only at STARTUP. Merged, believed delivered, green everywhere, doing nothing. The same class of failure this repo already guards against elsewhere: a green signal over something that never ran. Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply -- both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has rolled them since. The inertness is specific to ConfigMap-only changes. Set in all three value files to hold the [parity] contract. Cost, and it is the point: every future Cilium config change now rolls the agents instead of waiting for a manual restart. Strictly more honest -- it applies when merged or fails visibly -- but config edits become disruptive by default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge at a quiet moment. Merging this IS one rollout (the ConfigMap checksum annotation changes the pod template) -- which is what finally activates the Hubble exporter. UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable after the agents restart (check for flow lines immediately); and the rollout's actual impact on LAN DNS is unmeasured. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
81f627167d |
docs(obs): record that the gate-sidecar scrape works — four places still said it did not
PR #49 (merged
|
||
|
jakman85
|
4386d1453b |
Merge pull request 'fix(plex): advertise a resolver-independent plex.direct name again (LAN-wide outage)' (#52) from fix/plex-resolver-independent-advertise into main
Reviewed-on: #52 |
||
|
jakman85andClaude Opus 5
|
874c6703f0 |
fix(plex): advertise a resolver-independent plex.direct name again (LAN-wide outage)
I took Plex down for the whole LAN. ADVERTISE_IP was http://plex-vip.${BASE_DOMAIN}:32400 -- a name that exists ONLY in PiHole's split-horizon answers and in no public zone. A Plex client resolves the advertised host ITSELF, with whatever resolver it uses, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default, phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve the name, the router answered the failed lookup with ITSELF, and they all landed on its web UI -- hence the %D certificate. Measured 2026-10-05: 192.168.2.1:443 -> C=ZZ, O=%D342e7e6b, CN=%D (router, LAN side) 136.51.115.252:443 -> the same %D cert (router, WAN side) pihole-0 AND pihole-1 -> plex -> .20, plex-vip -> .24 (both CORRECT) gateway .20:443 -> CN=*.home.keithjackson.dev, valid http://192.168.2.24:32400/identity -> 200 Plex's published connections: all port 32400, nothing on 443 The DNS was right; the DEPENDENCY on it was the bug. And since nothing Plex advertises listens on 443, the clients had left the Plex path entirely. Fix: <dashed-ip>.<serverhash>.plex.direct, which is RESOLVER-INDEPENDENT -- a public zone returning the private address embedded in the label, so every resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers it with 192.168.2.24. No split-horizon entry, no Cloudflare record and no certificate work needed: the wildcard cert Plex already serves on 32400 covers that name. Re-verified today with full validation (no curl -k): 200, TLS verified. The nicer hostname cannot be made correct for clients that bypass PiHole -- a public record resolves to the WAN address and there is no NAT hairpin, so it is right only for PiHole users and wrong for the rest. There is no configuration of it that is right for both. PiHole's plex-vip answer is KEPT; it is just not what Plex advertises. Cost restated, not buried: the value embeds the VIP and the cert hash as a literal and nothing checks they still agree. Accepted because an unresolvable name is a worse failure -- silent, LAN-wide, and it points users at the router. UNVERIFIED: unreconciled, and the client-side resolver path is INFERRED -- my public-resolver probe was broken (NXDOMAIN even for a name that resolves) so I discarded it. The chain is the only explanation consistent with the measurements, but the NXDOMAIN-hijack step was not observed directly. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
bc7095677b |
Merge pull request 'fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http' (#51) from fix/plex-unblock-app-tier into main
Reviewed-on: #51 |
||
|
jakman85andClaude Opus 5
|
a0e6ef145f |
fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http
I broke the app tier with #48. Two faults, both mine, both measured after merge: 1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key Secrets (plex-vip-<random>) were denied by secret-provenance/secrets-must-be-flux-managed, which matches every Secret in a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes only those labelled secret-managed-by=controller or carrying a Flux label. `plex` HAS that label; `gateway` -- where the working wildcard cert lives -- does NOT. That asymmetry is why the existing Certificate was never affected and mine was, and I did not check it. secretTemplate cannot fix it: those Secrets come from the keymanager controller, not the Certificate template. So no CertificateRequest, Order or Challenge was ever created. 2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so Flux waits on the Certificate OBJECT: timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress'] apps-wave-0 never went Ready; wave-1/-2 and everything downstream of platform-config sat in DependencyNotReady behind it. I guarded the pod and missed the object. That distinction is the whole bug. Third, independent: the advertised URL was https with no servable certificate -- Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip -- the exact failure mode of the outage earlier the same day. So: delete the Certificate (pruning it clears the health gate) and advertise http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there. http is explicitly temporary and only tolerable because remote access does not work yet; the restore condition is written at the call site. DELETED, not merely unregistered: my first attempt kept the file as "tracked but not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02) fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization + the https flip, as ONE change) is recorded in catalog/plex.yaml. Kept and verified working from #48: the hostname design, PiHole's split-horizon answer, and the postStart hook -- LanNetworksBandwidth is now set live. Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
c163deaef9 |
Merge pull request 'feat(plex): one advertised hostname for local+remote, its own LE cert, and a LAN definition so LAN clients stop reading as remote' (#48) from feat/plex-vip-hostname-and-lan-networks into main
Reviewed-on: #48 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
642a7752d8 |
Merge pull request 'feat(cilium): export Plex's pre-SNAT client address via Hubble, scoped to tcp/32400' (#50) from feat/hubble-flow-export-plex into main
Reviewed-on: #50 |
||
|
jakman85andClaude Opus 5
|
989649e64a |
feat(cilium): export Plex's pre-SNAT client address via Hubble, scoped to tcp/32400
The requirement is log visibility only -- which address connected to Plex.
Plex's own log cannot show it. Every LoadBalancer Service here is
externalTrafficPolicy: Cluster, so the node holding the VIP's L2 lease SNATs
the client before forwarding to the backend pod. Measured: Plex had logged
3,214 requests from 10.0.1.x (lease holder k3s-prod-3's own address, Cilium
identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No
X-Forwarded-For either -- it serves raw tcp/32400, not HTTP via the Gateway.
Nothing on the router does this; there is no hairpin NAT. It is Cilium.
The address is not lost, only unrecorded. Cilium observes the flow at ingress,
BEFORE its own translation, so Hubble already has it:
192.168.2.133 -> 10.0.0.135 real LAN client -> the Plex pod
10.0.1.78 -> 10.0.0.135 same traffic post-SNAT, what Plex sees
What was missing is RETENTION: 4095/4095 flows at ~303/s is about thirteen
seconds of history, metrics disabled, no exporter configured.
So enable the static exporter, port-scoped, to the agent's stdout -- Alloy
already tails every pod into Loki, so this needs no new mount, no rotation
handling and no Alloy change.
{namespace="kube-system", app="cilium"} |= "32400"
Scoped BY PORT, not namespace, on evidence: --to-namespace plex returned only
the post-SNAT flows; the record carrying the real client address did not match
it. A namespace filter would drop exactly what this exists to capture.
Rejected, with reasons: eTP:Local is incompatible with cilium 1.20's L2
announcer per upstream docs for this version ("may cause service IPs to be
announced on nodes without pods causing traffic drops") -- and the VIP is on
prod-3 while the pod is on prod-1, so it would black-hole 32400 immediately;
making it safe costs Plex its ability to reschedule. DSR: routing-mode is
vxlan, not native/geneve, and DSR returns bypass the ingress node, breaking
the router's stateful NAT on the remote path.
Also corrects the repo's own record: 2026-10-03 says the SNAT lands in
192.168.0.0/16 (the node's LAN address). It is 10.0.1.x, the cilium_host
inside the pod CIDR -- which is why LanNetworksBandwidth can never separate
local from remote while eTP is Cluster.
UNVERIFIED and stated in the handoff: nothing is reconciled; `filePath: stdout`
is taken from a verbatim upstream sentence but not confirmed against 1.20.1
source (a wrong notation is a silent no-op, with the file+Alloy fallback
documented); the port FlowFilter has no upstream example; and this change
ROLLS CILIUM -- prod-3 announces the Gateway, PiHole DNS and Plex VIPs, so a
lease move during the rollout could briefly interrupt LAN DNS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
21c88b0dc7 |
Merge pull request 'fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable' (#49) from fix/oauth2-sidecar-metrics-scrape-netpol into main
Reviewed-on: #49 |
||
|
jakman85andClaude Opus 5
|
b156ee6cfa |
fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable
platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that have never once scraped. Measured on prod 2026-10-04: all six read health=down with lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the configured scrapeTimeout, i.e. packets DROPPED rather than refused. Cause: every app namespace ships a default-deny NetworkPolicy with policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180. Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr, qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object. render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this catalog declares one. It was wired in at the call site and had never emitted a single object on any branch: correct, and entirely latent. It now also fires for the forward-auth gate sidecar's own metrics port, keyed on _needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the sidecar exists. Keying on observability.metrics instead would let metrics: false produce a discovered-but-blocked target: the same "two halves authored apart" failure this function exists to prevent. Rendered here rather than in platform/oauth2-proxy because that tier would have to name app namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering per-app dissolves it. Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring. apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved. Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an object necessarily drifts the goldens that pin the renderer's output. The first gate run caught container-workload-render and egress-class-render as FAIL with exactly this CNP as the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the evidence that the regen is faithful and not a rubber stamp. That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author approving review. The local gate cannot see that — offline it reports approval-policy green (N/A). Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements. Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0aa1a0a78a |
feat(plex): one advertised hostname for local+remote, its own LE cert, and a LAN definition
Two symptoms after the publishing fix landed:
1. LAN clients were reported as Remote. Measured root cause:
LanNetworksBandwidth was ABSENT, so Plex had no notion of a local
network and classified everything WAN -- 468 requests logged
"Allowed Network (WAN)", ZERO "(LAN)", including Seerr from inside the
cluster. allowedNetworks only waives AUTH; it does not make a client
local. Two settings, and only one has an env hook.
2. The advertised URL hard-coded both the VIP and the server's cert hash
into a cluster-var -- two values nothing kept in sync.
Operator design: ONE hostname, plex-vip.${BASE_DOMAIN}, split-horizon --
PiHole answers it with ${PLEX_L4_VIP} on the LAN, Cloudflare answers it
publicly with the WAN address NATed to the same VIP.
- catalog/plex.yaml: ADVERTISE_IP -> https://plex-vip.${BASE_DOMAIN}:32400,
plus a level-triggered postStart hook (pihole/qbittorrent idiom) that
re-asserts LanNetworksBandwidth and the three customCertificate* prefs
every start, converting PEM -> PKCS#12 so cert renewals need no extra part.
- apps/wave-0/plex-cert.yaml: cert-manager Certificate for exactly that
name from letsencrypt-prod, so the advertised https URL is servable.
- schema 1.11.0: `optional` on secretKeyRef/configMapKeyRef. Load-bearing --
the cert is issued INTO the plex namespace, which the rendered plex.yaml
itself creates, so it lands in the same reconcile; hard refs would have
held Plex in CreateContainerConfigError and, since apps-wave-0 is
wait:true, stalled the whole app tier on a slow DNS01 challenge.
- catalog/pihole.yaml: the plex-vip split-horizon answer. Required --
measured, the name resolved to the Gateway VIP where nothing listens
on 32400 (http=000).
- PLEX_ADVERTISE_URL deleted; the hostname removes the cert-hash fragility.
Corrects my own earlier claim that a SAN match is REQUIRED: it is not
(httpsRequired=0, secureConnections defaults to Preferred, so clients can
fall back to cleartext). A matching cert keeps the remote leg ENCRYPTED.
Hence: obtain a real cert for the advertised name rather than pick the name
to suit Plex's built-in one.
UNVERIFIED, and stated plainly in the handoff: nothing here has run. Cert
issuance for this name is inferred from the wildcard's success, and whether
Plex accepts an OpenSSL-3 PKCS#12 bundle is untested -- the largest risk.
Touches clusters/** -> sensitive paths -> needs a non-author approval or the
operator's /approve-override; deliberately not self-posted (CLAUDE.md 1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
422834d4e2 |
Merge pull request 'fix(plex): advertise the plex.direct name, not the bare VIP (server vanished from the plex.tv account)' (#47) from fix/plex-advertise-plexdirect into main
Reviewed-on: #47 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
1f4552d409 |
fix(plex): advertise the plex.direct name, not the bare VIP (server vanished from the plex.tv account)
customConnections held two hand-typed URLs and neither could ever complete, so Plex had nothing publishable and the server disappeared from the account: - https://192.168.2.24:32400 -- the server's cert is *.130072f20c334d6c8f63d0dfef87309b.plex.direct with NO IP SAN, so bare-IP https fails TLS for every client. Measured: a continuous storm of "CERT: incomplete TLS handshake ... certificate unknown" from real LAN clients arriving through the plex-l4-tcp VIP. - https://plex.home.keithjackson.dev:32400 -- nothing listens; the Gateway serves that host on 443 (measured http=000 against 192.168.2.20:32400). With no completable connection plex.tv published nothing ("Mapped - Not Published"), Plex fell back to Relay, and clients reported "not authorized". Fix: advertise <dashed-ip>.<serverhash>.plex.direct -- public DNS that resolves to the embedded address, so it points at the LAN VIP and matches the wildcard cert. Verified with full TLS validation (no curl -k): 200 + verify OK, where the bare-IP form fails to connect. Declared as ADVERTISE_IP so git owns the value instead of a web form: the image's 40-plex-first-run writes it into customConnections on every pod start, outside the claim guard. Does NOT fix off-LAN access -- the advertised address is RFC1918 and the WAN tcp/32400 forward remains unverified. ManualPortMappingMode=1 with an absent port is left as-is (no GitOps path exists for it). Touches clusters/** -> sensitive paths -> needs the operator's /approve-override, deliberately not self-posted (CLAUDE.md 1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
c95b1c7121 |
Merge pull request 'fix(cilium): point cgroup.hostRoot at the host cgroup2 root (socket-LB never covered host processes)' (#42) from fix/cilium-cgroup-hostroot into main
Reviewed-on: #42 |
||
|
jakman85
|
ed6afc3d3f |
Merge pull request 'docs(decisions): record that PiHole HA is verified live, failover drill included' (#46) from docs/pihole-ha-verified into main
Reviewed-on: #46 |
||
|
jakman85andClaude Opus 5
|
254985baa6 |
docs(decisions): record that PiHole HA is verified live, failover drill included
No manifests change. The three entries merged earlier today each say, correctly
at the time, that nothing had been reconciled -- which is now stale in exactly
the way CLAUDE.md section 2b warns about: a reader would conclude the work is
unproven. This entry is the current state and does not edit the earlier files.
Measured on live prod after
|
||
|
jakman85
|
dce20b7be6 |
Merge pull request 'feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes' (#45) from feat/pihole-multi-replica into main
Reviewed-on: #45 |
||
|
jakman85andClaude Opus 5
|
028853c87c |
feat(pihole): two replicas on different nodes, and fix a PVC mounted where pihole never writes
Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO force-detach), while losing the node holding the pihole-dns L2 lease -- a DIFFERENT node -- meant tens of seconds. Two of three nodes were single points of failure with wildly different blast radii. Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent only; the mechanism is render.py's, documented in the renderer contract. 1 or absent renders a Deployment exactly as before -- verified byte-identical across the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a multi-replica Deployment never schedules the second pod), per-replica volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever). Pinned by ~20 new --selftest assertions rather than a conformance change. ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default /data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M -- gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn volume held lost+found and 24K of nothing. So every restart discarded all state and re-downloaded 72k gravity domains, and backup.policy has been backing up an empty volume while reporting success. Fixed here because otherwise the new per-replica volumes would hold nothing while state stayed ephemeral -- and because persisting gravity.db is what makes a restart cheap. The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as uid 0 and chowns the directory recursively (no fsGroup needed), and it calls migrate_gravity which creates the database when absent. The gate caught a real regression on the first run: the wave-0 alert unit test fed pihole a Deployment-shaped series. Updated; promtool SUCCESS. Touches no sensitive paths, so this one needs no override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
17f0c11403 |
Merge pull request 'feat(pihole): declare the admin password and adlists in git' (#44) from feat/pihole-declare-state into main
Reviewed-on: #44 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
3b1d6ae115 |
feat(pihole): declare the admin password and adlists in git
Prerequisite for running more than one PiHole replica. A second replica with a fresh PVC would have come up serving DNS and blocking nothing, with a different admin password -- answers varying by which pod replied. pihole.toml's own ### CHANGED markers gave the exact inventory: four settings already came from env (declared), one came from the image, and exactly ONE operator-set value lived only on the PVC -- webserver.api.pwhash, a one-way hash nobody could recover. Separately the adlists were undeclared: one list and 72,234 gravity domains that nothing in this repo named. Declares both: - A SOPS secret feeding FTLCONF_webserver_api_password. webserver.api.password takes cleartext and FTL hashes it itself (verified: pihole setpassword IS setFTLConfigValue on that key), so the PVC's pwhash becomes derived state. - A level-triggered postStart hook that reconciles the declared adlist set into gravity.db -- adds, re-enables, and disables (never deletes) undeclared lists, running pihole -g only when something moved. A file would not work: v6's adlists.list import is migration-only, guarded on the DB not existing. The rendered hook was run in the live pod against throwaway copies of gravity.db, exercising all four paths. A fresh DB built from the image's own schema template has ZERO adlists, confirming this is load-bearing rather than cosmetic. Password kept rather than set empty: the pihole Service still publishes port 80 past the oauth2-proxy, closed only by policy, and default-deny has drifted here before. Multi-replica is phase 2 and deliberately not in this branch -- it needs a schema field, a StatefulSet (the config PVC is RWO), anti-affinity and a PDB. Adds a *.sops.yaml -- a sensitive path -- so approval-policy will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
939461120a |
Merge pull request 'feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP' (#43) from feat/plex-remote-access-l4 into main
Reviewed-on: #43 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fbaeb3e038 |
feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP
Remote Plex clients had no path in, for two measured reasons: Plex advertised no remote connection at all (customConnections and ManualPortMappingMode both ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so 32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this server on a client's connection list. Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS with *.plex.direct and streams bypass the Envoy L7 hop. Adds a hand-written non-catalog unit (the SD schema cannot express a LoadBalancer Service or a pinned lbipam VIP), registered in render.py's EXTRA_UNITS so --emit-waves places it in plex's own wave. externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Known consequence, documented: SNAT puts remote clients inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change. Remote access does NOT work until the operator does two manual steps that cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually specify public port" toggle (no env var exists for it; verified NONE in the image's init surface). runbooks/plex-remote-access.md is the procedure. Touches clusters/** -- a sensitive path -- so the approval-policy stage will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
9a69f6906a |
fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root
hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing mounts cgroup2 there. The agent mounted one from inside its own private cgroup namespace, so socket-LB attached to the agent's own cgroup and host processes got no Service/LB translation. k3s-prod-1 (Gateway L2 lease holder + kube-vip VIP) therefore could not reach the Gateway VIP, and its apiserver's OIDC authenticator has failed since 2026-09-12. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85
|
7ef2af205d |
Merge pull request 'fix(node-baseline): pin the OIDC issuer to the Gateway VIP on prod nodes' (#41) from fix/node-keycloak-hosts-pin into main
Reviewed-on: #41 |
||
|
jakman85andClaude Opus 5.5
|
788eaeae62 |
fix(node-baseline): pin the OIDC issuer to the Gateway VIP on prod nodes
After public DNS moved to the WAN IP (#40), prod nodes resolve keycloak.home.keithjackson.dev to a WAN address the router does not hairpin, so the apiserver could not reach its OIDC issuer. Adds an opt-in node_baseline_hosts_pins /etc/hosts block; prod pins keycloak to GATEWAY_VIP. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85
|
576e1abb19 |
Merge pull request 'feat(dns): publish Gateway names to the WAN via in-cluster DDNS' (#40) from feat/public-dns-wan-target into main
Reviewed-on: #40 |
||
|
jakman85andClaude Opus 5.5
|
a1e808e697 |
feat(dns): publish Gateway names to the WAN via in-cluster DDNS
Gateway-published records now CNAME to wan.${BASE_DOMAIN}, which a
favonia/cloudflare-ddns Deployment keeps at the home WAN's current public
IP. Operator decision: every Gateway-published name goes public; internal
services (controller: none) remain unpublished.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
||
|
jakman85
|
bcb8931b4f |
Merge pull request 'fix(runbooks): Plex hw transcoding WORKS — the dashboard was never the proof' (#39) from fix/plex-quicksync-verification into main
Reviewed-on: #39 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
0d7f94098b | Merge branch 'main' into fix/plex-quicksync-verification | ||
|
jakman85
|
5ffd0c98bc |
Merge pull request 'fix(ci): let an administrator merge their own green PR — required_approvals 1 → 0, plus an attributed /approve-override' (#38) from fix/admin-approval-override into main
Reviewed-on: #38 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f0994c11a0 |
docs(decisions): record the PR's measured CI result — gitea-settings has drifted
Read PR #39's own gate log rather than the tick (CLAUDE.md §2b). Every stage
this branch could affect is green: decisions-log, lint, l0, render-golden, l1,
and approval-policy (which evaluated for real on the pull_request event and
passed — no sensitive path touched). The one failure is `gitea-settings`:
required_approvals: DECLARED 1 but LIVE is 0
Gitea has DRIFTED from infra/gitea/expected-branch-protection.json
Every other declared setting matched, and one runner is online. So live `main`
currently requires ZERO approvals while the declaration says 1, and because
.gitea/workflows/gate.yml sets REQUIRE_GITEA_SETTINGS=true, that drift blocks
EVERY PR in the repo, not only this one. The check that would notice approvals
are unenforced is the check that is failing.
Not touched here: infra/gitea/** is a sensitive path (non-author approval
required) and outside this branch's documentation-only scope. Re-asserting the
protection or changing the declaration is an operator decision in its own PR;
local branch fix/admin-approval-override (
|
||
|
jakman85andClaude Opus 5
|
b52c8d1af6 |
fix(ci): accept the override from a PR review, and stop misdiagnosing an unreadable comments API
The override did not work on its first real test, and the reason was invisible from the outside. Run 203 on PR #38: a well-formed `/approve-override` posted by jakman85 in the PR comment box, 29 minutes newer than the head commit, and the stage still blocked with: approval-override: COMMENTS_JSON is not a list SETTINGS_API_TOKEN is minted [read:repository, read:admin], and Gitea puts /issues/{n}/comments behind the *issue* scope — so Gitea answered with an error OBJECT, `jq -e .` accepted it as valid JSON, and the stage fell through to 'no approving review'. A permission failure wearing the costume of a missing override. /pulls/{n}/files and /pulls/{n}/reviews are under read:repository, which is why every other part of the same stage worked. 1. The diagnosis no longer lies. COMMENTS_JSON leaves the must-parse-or-block loop and is checked for being an ARRAY; when it is not, the stage prints Gitea's own .message, names the missing scope and says where to post instead. An unreadable source can only cause a false BLOCK, never a false pass, so a loud warning is the honest behaviour and blocking was not. 2. An override is accepted from a PR REVIEW body as well as a comment, so the hatch is not hostage to one API scope — a review of type 'Comment' works on the token CI has today. The gate names which source it used; PENDING (unsubmitted, author-only) reviews are dropped. 3. set-ci-token.sh requests read:issue, making the ordinary comment box work — on a RE-RUN only, since an existing token's scopes cannot be widened. ci/tests/approval-override.test.sh: 23 -> 38 assertions, covering the reviews source, PENDING reviews, an error object from the comments API, 'an unreadable comments source must not suppress a valid review override', latest-wins across both sources, and that the diagnostic quotes Gitea's message. Also measured and recorded: run 203 shows gitea-settings PASS after the operator ran the live PATCH, with block_admin_merge_override still true — the narrow change held, the approval count moved and the gate's teeth did not. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
1677dff568 |
fix(runbooks): Plex hw transcoding WORKS — the dashboard was never the proof
Hardware transcoding is confirmed working on prod (2026-10-02): VAAPI for both decode and encode on /dev/dri/renderD128, with zero-copy, measured in `Plex Media Server.log` in the running pod. The device-plugin mechanism from #37 works end to end. `runbooks/plex-quicksync.md` said the Plex dashboard showing `Transcode (hw)` was "the *only* proof available, and nothing else will ever tell you". That is false: Dashboard -> Activity showed nothing while the feature was fully working, so the page's authoritative check was silent on a healthy system and sent the operator hunting a fault that did not exist. Three inference errors corrected, two kept as named false leads: 1. The LOG is the primary check, not the dashboard. The grep ships in its REDACTING form -- that log carries API URLs with `X-Plex-Token` (CLAUDE.md §6). Also recorded: `Plex Transcoder Statistics.log` is the WRONG file (XML <Variant/> records, no ffmpeg output), and why `grep -a` is required. The dashboard is kept but demoted to best-effort, with its silence stated to prove nothing. 2. An fd snapshot is not evidence. A /proc/*/fd sweep found no holder of renderD128 while hardware transcoding was active -- sessions are short and the scan landed in a gap. The reliable process-level check is the transcoder's argv (-hwaccel, -hwaccel_device, vaapi, renderD128). 3. A missing preference key means nothing. `HardwareAcceleratedCodecs` was absent from all 27 attributes with the toggle on and working, while `HardwareDevicePath` was written and the log said "hardware transcoding: enabled". Only `="0"` is conclusive; the log settles it. Also stale and now fixed: GID 992 is a hostPath-era number. Under the plugin the injected device is root:root, so the image's hook resolves GID 0 and adds `plex` to the root group (`Groups: 0 100 1000`, `groups=1000(plex),0(root),100(users)`). Marked as an inherent property of the plugin's DeviceSpec, not a defect, and explicitly not a reason to reintroduce supplementalGroups (still a no-op -- s6-setuidgid rebuilds the group list from /etc/group). Added a per-title fallback entry (UHD 630 has no AV1 decode), labelled a HYPOTHESIS: the title that fell back was never checked. `runbooks/README.md`'s index entry was stale the same two ways and is corrected. Unchanged on purpose: the "STOP HERE IF PLEX IS Pending" section, the escape hatch, and the no-alert-is-possible reasoning (no metricsEndpoint, no ServiceMonitor, nothing a rule could evaluate). Documentation only -- no catalog/, schemas/, apps/ or platform/ change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8ccfdab180 |
fix(ci): let an administrator merge their own green PR
`required_approvals: 1` on `main` was a lock-out, not a protection. Gitea
refuses to let a PR's author approve their own PR, and `jakman85` is the only
collaborator — so no account in existence could approve a PR the operator
opened, including the PR that would fix that. Measured 2026-10-01 on PR #35:
gate green in 1m41s, reviews `[]`, `required_approvals: 1`, unmergeable.
It was never the documented design either: `ci/approval-policy.sh`'s header and
`ci/gate.sh:194` both already describe the model as "required_approvals drops
to 0 and the GATE decides", and `require-gate.sh`'s own closing note named this
failure and prescribed this remedy. 2026-09-30's entry flagged the 1-vs-0
disagreement as unresolved; this settles it at 0.
Layer 1 — branch protection. required_approvals 1 -> 0 in the declaration, plus
infra/gitea/allow-admin-self-merge.sh to apply it live (target read FROM the
declaration, so the script cannot drift from what the gate checks).
require-gate.sh moves to 0 too; left at 1 it would re-impose the lock-out on its
next run. The narrow choice is deliberate: the other way to unblock an owner is
block_admin_merge_override: false, which would re-open what
block-admin-override.sh was written to close (PRs #5-#9, zero approvals). This
drops the approval COUNT only — block_admin_merge_override and
enable_status_check stay true and are asserted after the PATCH, so a red gate
stays unmergeable by everyone, administrators included.
Layer 2 — ci/approval-policy.sh. Sensitive paths still need a non-author
approval the sole admin cannot get, so an administrator may waive it with an
`/approve-override <reason>` comment: directive must start a line, reason
mandatory, permission re-checked LIVE, invalidated by a new commit, and the
overrider + reason printed in the gate log and job summary. Candidate selection
is the pure, offline ci/approval-override.py (23 unit assertions, all passing);
authorisation stays in the fail-closed stage. Every new failure mode blocks.
The cost, stated not buried: for an administrator, "a change to the gate cannot
be gated by the gate" is now advisory rather than structural. Unavoidable with
one human; what is bought back is an attributed, commit-scoped record instead of
a silent bypass. And the uncomfortable part, accepted as an operator decision:
authorisation is by live permission alone, and the AGENT account `gitea-admin`
also holds `owner` — so an agent can clear its own PR this way. Documented in
approval-policy.json, the stage header, and CLAUDE.md (which tells agents not
to). Narrowing needs no code change: add `override_admins` and the stage already
requires membership.
Adjacent fix, in scope because `make -C ci test` is the only runner for the
suite added here: ci/tests/gate-decision.test.sh hand-maintained a stub list
that "must name EVERY stage gate.sh runs" and omitted `approval-policy` (added
to gate.sh by PR #17), so run_stage hit exit 127 and 15 of 50 assertions failed.
Verified pre-existing on pristine origin/main @
|