80592970360a411ab8aefc99fdf0e191de80938b
135
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
8059297036 |
fix(devcontainer): bake wg + qrencode into the image instead of splitting the VPN procedure across host and container
Operator correction: "sop, wg, and any other tool requirement is supposed to be
baked into the devcontainer."
My previous commit discovered that `wg` existed only on the host, `sops` only in
the container, and `qrencode` on neither -- and fixed it the wrong way, by
splitting every step of the key procedure across the boundary and telling the
operator to `dnf install qrencode` on their workstation. That breaks CLAUDE.md §5
and the devcontainer README's "Don't install the toolchain on the host", produces
a procedure that cannot be reproduced in CI, and is miserable to follow.
wireguard-tools + qrencode are now distro packages in the Containerfile. Both are
in Fedora 44's repos (wireguard-tools 1.0.20260223 -- the same upstream version as
the image this component runs -- and qrencode 4.1.1), unpinned like every other
dnf package there: the pinned thing is the Fedora base, and neither is a gate tool
whose version changes what green means.
Three things got SIMPLER rather than merely relocated, which is the tell that the
rule was right:
* Step 2 is one pipeline in one shell -- `sops set ... "$(wg genkey)"` writes the
key straight into the already-encrypted document. No staged copy in .cache, no
`*.key`-suffix warning, no sha256 comparison to prove the right bytes landed.
* new-peer.sh derives the server public key from wg-server-key.sops.yaml itself
(`sops -d --extract | wg pubkey`), so there is no local key copy to keep in
step with the encrypted one, and no magic host path it must live at.
* `.cache/` is gitignored at the repo root (mirroring infra/ansible/.gitignore),
giving the container a persistent non-committable scratch dir. new-peer.sh
writes profiles to .cache/wg-peers/ -- necessary because the repo bind-mount is
the container's only persistent read-write mount, so a profile in /tmp there is
gone before it can reach a desktop. Its blanket "refuse to write inside the
checkout" guard is now a `git check-ignore` test of the real property.
Verified by execution, not reasoning: the image rebuilds; wg/qrencode/sops all
resolve in it; `sops -d | wg pubkey` runs in one shell; new-peer.sh refuses the
REPLACE-ME placeholder with an actionable message; and a full run with a throwaway
server key renders a correct profile at 0600 under a 0700 gitignored dir, prints
the QR, and refuses a non-ignored destination inside the repo.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e35612dc7e |
fix(vpn-ingress): keep the server key in source through SOPS, not as an out-of-band Secret
Operator correction: secrets are kept in source through sops. #55 shipped the WireGuard server private key as a Secret to `kubectl create` by hand. Two real costs, not just a convention break: * R-SEC-01 / spec docs/07 -- secrets live in source, SOPS-encrypted, and reach the cluster only through Flux. * It broke DR for the one credential that cannot be re-derived. A rebuild restores everything in git; it does not restore a hand-made Secret, and losing this key means re-enrolling EVERY device, because each client profile pins its public half. #55 papered over that with "keep a copy in your password manager". (Note secret-provenance did NOT forbid the original shape -- it scopes to app-namespace=true and vpn-ingress is deliberately a platform namespace. The rule broken was the written requirement, not an enforced control.) An agent still must not mint a long-lived private key (CLAUDE.md §6), so the file is committed ALREADY ENCRYPTED with `privatekey: REPLACE-ME`. That makes the wiring complete and green from the first commit -- resource line plus both overlay decryption patches -- leaving the operator exactly one act: replace the value. A commented-out resource to uncomment later is the "in git, no generator entry, reviewed, merged, still broken" shape this repo keeps hitting. The init guard is now a validation rather than an emptiness check: the value is piped through `wg pubkey`, which accepts only a real curve25519 private key, and the pod names which case it hit -- placeholder, empty, mangled, or (the silent one) still `ENC[` ciphertext because an overlay is missing its decryption patch, which otherwise applies cleanly with every Flux condition green. Second, independent finding that also fixes #55's step 2: `wg` is on the HOST only and `sops` is in the CONTAINER only, so `sops -d | wg pubkey` cannot run in one shell -- the command #55 documented could not have worked. Every step is now split by where it runs, and the written key is verified by comparing sha256 digests rather than printing anything. `qrencode` is on neither, so the QR step says so and gives the install command. Touches clusters/** and adds a **/*.sops.yaml, so approval-policy will block pending the operator's /approve-override. Deliberately not posted by me. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
e4a9b02b00 |
docs(pm-agent): CLAUDE.md §0 is already satisfied in a worker — tell it not to make a worktree
PR #56 added §0 (work in your own worktree, never the shared checkout) while this branch was open. A worker agent reads CLAUDE.md first, and §0's instruction is to create a git worktree outside the shared checkout — which does not exist in a worker's container. It holds a fresh clone of its own, made for one assignment and thrown away with the pod, which is what §0 is actually asking for. A worker that obeyed §0 literally would 'git worktree add', edit somewhere the wrapper that started it does not look, and have its work committed by nobody while the issue got a 'changed nothing' report. worker-rules.md now says §0 is satisfied and not to do that. Also records in the handoff entry: the merge and its one conflict, and that this push dismisses the operator's approving review — measured, dismiss_stale_approvals=true — so the approval-policy stage needs them to re-approve. That is theirs to do, not an agent's. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0569d47d49 |
Merge branch 'main' into feat/pm-agent-issue-to-pr
One conflict, in clusters/base/kustomization.yaml: PR #55 (vpn-ingress) and this branch both appended a resource line to the same tail. Resolved by keeping BOTH, with main's vpn-ingress first — it is a platform-config-tier leaf, while pm-agent dependsOn apps-wave-0, which is later. Nothing else conflicted. runbooks/kustomization.yaml auto-merged and carries both the vpn-remote-access and pm-agent entries; the handoff entries are separate files and had nothing to merge, which is the whole point of decisions/ being a directory. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8b5e950baa |
test(pm-agent): a committed negative-path probe for the worker's input guards
pm-worker's command is pinned by admission with no args, so everything a caller could influence arrives in the environment — and the guards at the top of that script are what stand between an injected variable and a clone from the wrong host or a push to the wrong ref. Nothing exercised them, which makes them a claim rather than a control. probe-validation.sh asserts 17 refusals, each on BOTH a non-zero exit and the specific message, so a guard that stopped firing cannot pass by failing somewhere else. It covers a redirected git host, a real and reachable WRONG one (the egress policy permits tcp/443, so that case is not hypothetical), the right host with a smuggled path, 'main' and five other bad branch shapes, a malformed owner/repo, a bad mode, a bad issue number, and each missing credential. build-image.sh runs it inside the built image too. Writing it found three things: - the PM_BRANCH != main guard is unreachable while the pattern check above it holds, so the comment claiming it as the defence was wrong about which line does the work. Kept as an assertion independent of the regex; the comment now says so. - the EXIT trap ended in a failing test, which under set -e can replace the script's real exit code with that failure. - a provenance LOG LINE read a file through a redirect that set -o pipefail would have turned into a dead worker if the file were absent. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
54418363aa |
fix(pm-agent): take the newest comment by list order, not by comparing timestamp strings
last_activity() is the whole trigger: the PM acts iff the newest thing in a thread was not written by it. It found that newest comment by comparing Gitea's RFC 3339 created_at STRINGS, which is only correct while every offset is identical — true today, silently wrong after a server timezone change, and the failure would be the PM answering its own last comment forever. The comments endpoint returns them in creation order, so the last element IS the newest. Using that removes the dependency entirely. Found by self-review, before anything ran. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f92707f38c |
feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers)
Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/** , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c5b5b8672a |
feat(pm-agent): an in-cluster AI Program Manager that turns Gitea issues into pull requests
A service that reads this repository's Gitea issues, decides what to do about each thread as the Program Manager persona, and runs one worker agent per assignment that authors a change, runs the merge gate and opens a pull request. It reacts to issue creation and to comments; the assignment and all correspondence travel through the issue thread and the PR. It authenticates with an Anthropic subscription token. It never pushes to main and never merges. Three choices carry the design, with alternatives recorded in docs/plans/: - it POLLS rather than receiving webhooks (Gitea's webhook SSRF guard, lost deliveries, and no inbound reach needed at all); - it keeps NO state: the trigger is "the newest thing in this thread was not written by me", so the issue thread is the state — no PVC, no backup obligation; - the model holds NO tools: its reply is one JSON object this code validates, so nothing in the path executes text. The boundary is platform/kyverno/admission/pm-agent-job-shape.yaml (Enforce, failurePolicy: Fail), not the RBAC: Kubernetes places no `use` check on serviceAccountName, so "may create Jobs" is "may run arbitrary code as the namespace's most privileged identity". Six rules admit only the committed shape, and the worker identity is bound to nothing with no token mounted, so there is nothing to escalate to. 28/28 offline Kyverno assertions. Nothing has run: the image has never been built or pushed, and both the image digest and the two credentials are deliberate placeholders. See the handoff entry's "What I did NOT verify". Refs CLAUDE.md §1/§3/§6, R-SEC-02, R-GITOPS-02. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
d0db63911a |
feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)
A merged Cilium config change did nothing, and everything reported success. The Hubble flow exporter (#50, merged 2026-10-04) never took effect: HelmRelease cilium Ready=True, "Helm upgrade succeeded ... cilium.v6" cilium-config hubble-export-file-path: stdout + the allowlist, PRESENT cilium agent pods all three still PRE-MERGE, startTime 2026-10-04T07:11Z exported flow lines 0, 0, 0 Flux every Kustomization Ready=True Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent reads that ConfigMap only at STARTUP. Merged, believed delivered, green everywhere, doing nothing. The same class of failure this repo already guards against elsewhere: a green signal over something that never ran. Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply -- both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has rolled them since. The inertness is specific to ConfigMap-only changes. Set in all three value files to hold the [parity] contract. Cost, and it is the point: every future Cilium config change now rolls the agents instead of waiting for a manual restart. Strictly more honest -- it applies when merged or fails visibly -- but config edits become disruptive by default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge at a quiet moment. Merging this IS one rollout (the ConfigMap checksum annotation changes the pod template) -- which is what finally activates the Hubble exporter. UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable after the agents restart (check for flow lines immediately); and the rollout's actual impact on LAN DNS is unmeasured. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
989649e64a |
feat(cilium): export Plex's pre-SNAT client address via Hubble, scoped to tcp/32400
The requirement is log visibility only -- which address connected to Plex.
Plex's own log cannot show it. Every LoadBalancer Service here is
externalTrafficPolicy: Cluster, so the node holding the VIP's L2 lease SNATs
the client before forwarding to the backend pod. Measured: Plex had logged
3,214 requests from 10.0.1.x (lease holder k3s-prod-3's own address, Cilium
identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No
X-Forwarded-For either -- it serves raw tcp/32400, not HTTP via the Gateway.
Nothing on the router does this; there is no hairpin NAT. It is Cilium.
The address is not lost, only unrecorded. Cilium observes the flow at ingress,
BEFORE its own translation, so Hubble already has it:
192.168.2.133 -> 10.0.0.135 real LAN client -> the Plex pod
10.0.1.78 -> 10.0.0.135 same traffic post-SNAT, what Plex sees
What was missing is RETENTION: 4095/4095 flows at ~303/s is about thirteen
seconds of history, metrics disabled, no exporter configured.
So enable the static exporter, port-scoped, to the agent's stdout -- Alloy
already tails every pod into Loki, so this needs no new mount, no rotation
handling and no Alloy change.
{namespace="kube-system", app="cilium"} |= "32400"
Scoped BY PORT, not namespace, on evidence: --to-namespace plex returned only
the post-SNAT flows; the record carrying the real client address did not match
it. A namespace filter would drop exactly what this exists to capture.
Rejected, with reasons: eTP:Local is incompatible with cilium 1.20's L2
announcer per upstream docs for this version ("may cause service IPs to be
announced on nodes without pods causing traffic drops") -- and the VIP is on
prod-3 while the pod is on prod-1, so it would black-hole 32400 immediately;
making it safe costs Plex its ability to reschedule. DSR: routing-mode is
vxlan, not native/geneve, and DSR returns bypass the ingress node, breaking
the router's stateful NAT on the remote path.
Also corrects the repo's own record: 2026-10-03 says the SNAT lands in
192.168.0.0/16 (the node's LAN address). It is 10.0.1.x, the cilium_host
inside the pod CIDR -- which is why LanNetworksBandwidth can never separate
local from remote while eTP is Cluster.
UNVERIFIED and stated in the handoff: nothing is reconciled; `filePath: stdout`
is taken from a verbatim upstream sentence but not confirmed against 1.20.1
source (a wrong notation is a silent no-op, with the file+Alloy fallback
documented); the port FlowFilter has no upstream example; and this change
ROLLS CILIUM -- prod-3 announces the Gateway, PiHole DNS and Plex VIPs, so a
lease move during the rollout could briefly interrupt LAN DNS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5.5
|
9a69f6906a |
fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root
hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing mounts cgroup2 there. The agent mounted one from inside its own private cgroup namespace, so socket-LB attached to the agent's own cgroup and host processes got no Service/LB translation. k3s-prod-1 (Gateway L2 lease holder + kube-vip VIP) therefore could not reach the Gateway VIP, and its apiserver's OIDC authenticator has failed since 2026-09-12. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
788eaeae62 |
fix(node-baseline): pin the OIDC issuer to the Gateway VIP on prod nodes
After public DNS moved to the WAN IP (#40), prod nodes resolve keycloak.home.keithjackson.dev to a WAN address the router does not hairpin, so the apiserver could not reach its OIDC issuer. Adds an opt-in node_baseline_hosts_pins /etc/hosts block; prod pins keycloak to GATEWAY_VIP. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b52c8d1af6 |
fix(ci): accept the override from a PR review, and stop misdiagnosing an unreadable comments API
The override did not work on its first real test, and the reason was invisible from the outside. Run 203 on PR #38: a well-formed `/approve-override` posted by jakman85 in the PR comment box, 29 minutes newer than the head commit, and the stage still blocked with: approval-override: COMMENTS_JSON is not a list SETTINGS_API_TOKEN is minted [read:repository, read:admin], and Gitea puts /issues/{n}/comments behind the *issue* scope — so Gitea answered with an error OBJECT, `jq -e .` accepted it as valid JSON, and the stage fell through to 'no approving review'. A permission failure wearing the costume of a missing override. /pulls/{n}/files and /pulls/{n}/reviews are under read:repository, which is why every other part of the same stage worked. 1. The diagnosis no longer lies. COMMENTS_JSON leaves the must-parse-or-block loop and is checked for being an ARRAY; when it is not, the stage prints Gitea's own .message, names the missing scope and says where to post instead. An unreadable source can only cause a false BLOCK, never a false pass, so a loud warning is the honest behaviour and blocking was not. 2. An override is accepted from a PR REVIEW body as well as a comment, so the hatch is not hostage to one API scope — a review of type 'Comment' works on the token CI has today. The gate names which source it used; PENDING (unsubmitted, author-only) reviews are dropped. 3. set-ci-token.sh requests read:issue, making the ordinary comment box work — on a RE-RUN only, since an existing token's scopes cannot be widened. ci/tests/approval-override.test.sh: 23 -> 38 assertions, covering the reviews source, PENDING reviews, an error object from the comments API, 'an unreadable comments source must not suppress a valid review override', latest-wins across both sources, and that the diagnostic quotes Gitea's message. Also measured and recorded: run 203 shows gitea-settings PASS after the operator ran the live PATCH, with block_admin_merge_override still true — the narrow change held, the approval count moved and the gate's teeth did not. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8ccfdab180 |
fix(ci): let an administrator merge their own green PR
`required_approvals: 1` on `main` was a lock-out, not a protection. Gitea
refuses to let a PR's author approve their own PR, and `jakman85` is the only
collaborator — so no account in existence could approve a PR the operator
opened, including the PR that would fix that. Measured 2026-10-01 on PR #35:
gate green in 1m41s, reviews `[]`, `required_approvals: 1`, unmergeable.
It was never the documented design either: `ci/approval-policy.sh`'s header and
`ci/gate.sh:194` both already describe the model as "required_approvals drops
to 0 and the GATE decides", and `require-gate.sh`'s own closing note named this
failure and prescribed this remedy. 2026-09-30's entry flagged the 1-vs-0
disagreement as unresolved; this settles it at 0.
Layer 1 — branch protection. required_approvals 1 -> 0 in the declaration, plus
infra/gitea/allow-admin-self-merge.sh to apply it live (target read FROM the
declaration, so the script cannot drift from what the gate checks).
require-gate.sh moves to 0 too; left at 1 it would re-impose the lock-out on its
next run. The narrow choice is deliberate: the other way to unblock an owner is
block_admin_merge_override: false, which would re-open what
block-admin-override.sh was written to close (PRs #5-#9, zero approvals). This
drops the approval COUNT only — block_admin_merge_override and
enable_status_check stay true and are asserted after the PATCH, so a red gate
stays unmergeable by everyone, administrators included.
Layer 2 — ci/approval-policy.sh. Sensitive paths still need a non-author
approval the sole admin cannot get, so an administrator may waive it with an
`/approve-override <reason>` comment: directive must start a line, reason
mandatory, permission re-checked LIVE, invalidated by a new commit, and the
overrider + reason printed in the gate log and job summary. Candidate selection
is the pure, offline ci/approval-override.py (23 unit assertions, all passing);
authorisation stays in the fail-closed stage. Every new failure mode blocks.
The cost, stated not buried: for an administrator, "a change to the gate cannot
be gated by the gate" is now advisory rather than structural. Unavoidable with
one human; what is bought back is an attributed, commit-scoped record instead of
a silent bypass. And the uncomfortable part, accepted as an operator decision:
authorisation is by live permission alone, and the AGENT account `gitea-admin`
also holds `owner` — so an agent can clear its own PR this way. Documented in
approval-policy.json, the stage header, and CLAUDE.md (which tells agents not
to). Narrowing needs no code change: add `override_admins` and the stage already
requires membership.
Adjacent fix, in scope because `make -C ci test` is the only runner for the
suite added here: ci/tests/gate-decision.test.sh hand-maintained a stub list
that "must name EVERY stage gate.sh runs" and omitted `approval-policy` (added
to gate.sh by PR #17), so run_stage hit exit 127 and 15 of 50 assertions failed.
Verified pre-existing on pristine origin/main @
|
||
|
jakman85andClaude Opus 5
|
a78f010368 |
feat(node-health): probe the Intel render device, read-only
The prod nodes already have a working Intel render device — measured by hand on
all three on 2026-09-30. Nothing here installs, loads or changes anything; the
point is that the measurement now re-runs on every health run instead of decaying
silently in a handoff note.
The failure being hunted is silent by construction: Plex hardware-transcodes
through /dev/dri/renderD128, and if that node stops existing (node rebuild, a BIOS
change that disables the onboard IGD, a kernel that binds a different DRM driver)
Plex falls back to software transcoding on its own and the only symptom is a hot
CPU.
All of it is in the UNPRIVILEGED collector pass: lstat() on /dev/dri/*,
grp.getgrgid() for the owning group name, the /sys/class/drm/card*/device/driver
symlink for the bound driver, /sys/bus/pci/devices/*/class for display-class
devices whether or not a driver is bound, /sys/class/drm/card*/engine/* for the
vcs*/vecs* media engines, /proc/modules and /proc/cmdline. lspci and vainfo are
probed but never depended on — neither is in node_baseline's package floor.
Three checks, in the role's existing PASS/WARN/FAIL/SKIP/INFO shape:
graphics.render_device the outcome — a usable character-special renderD* node.
FAIL only when a display device IS on the PCI bus and
there is still no render node. SKIP when the machine has
no display device at all (a VM guest), which is why the
PCI scan is separate from the DRM scan: /sys/class/drm/
card* exists only once a driver has BOUND.
graphics.drm_driver what is bound, against the new
node_health_graphics_expected_driver default (i915), so
the comparison is data and not task logic. `xe` bound in
place of i915 is a WARN, not a FAIL: the device works but
the userspace media stack differs.
graphics.media_stack INFO only, deliberately. GuC/HuC off (i915.enable_guc=0)
is an accepted state that costs low-power H.264 encode
quality only, and no host vainfo is not a defect at all
because Plex ships its own libva inside its container.
The render node's owning group and GID are REPORTED, never asserted: `render` is a
systemd dynamic allocation rather than a Debian static gid, so a rebuild can
legitimately move it off the 992 measured today, and Plex's container entrypoint
stats the device and joins whichever group owns it. Drift stays visible without
being an error.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
296d6d3275 |
Merge pull request 'feat(ci): the pipeline is the approver' (#17) from feat/ci-is-the-approver into main
Reviewed-on: #17 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
ae5c7385a5 |
docs(gitea): the README still named the ILLEGAL secret GITEA_SETTINGS_TOKEN
PR #15 renamed the Actions secret to SETTINGS_API_TOKEN because Gitea RESERVES the GITEA_ prefix and rejects such a name with HTTP 400 "invalid secret name" -- the defect that left the branch-protection detector inert and unnoticed. The README in this very directory, which documents that story, went on naming the illegal name in its behaviour table and in its closing "remaining step" line. Both now say SETTINGS_API_TOKEN, plus a note recording what ci/gitea-settings.sh ACTUALLY does, quoted from the file rather than assumed: TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}" GITEA_SETTINGS_TOKEN is still honoured, but ONLY as an environment variable for a local run -- it can never be the CI wiring, because the secret that would feed it cannot be created at all. The closing line also predated PR #15 storing the secret; it now says so. Documentation only. No behaviour change. NOT fixed here, because it is outside this PR's diff: ci/gitea-settings.sh:45 still instructs the reader to "create a Gitea repo secret GITEA_SETTINGS_TOKEN", which is impossible. Same defect, different file -- worth its own one-line change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
19f9a8ae0c |
feat(sec): provision the narrow conformance CI trigger credential -- and refuse to arm a fake green
docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6, as an operator script:
infra/gitea/conformance-trigger-kubeconfig.sh. UNEXECUTED here -- minting a
cluster credential and changing Gitea variables are operator actions.
It mints a token for conformance/conformance-trigger (TokenRequest), wraps ONLY
that token with the cluster CA and server into a kubeconfig, and stores it as the
Actions secret CONFORMANCE_TRIGGER_KUBECONFIG. Never cluster-admin, never a copy
of the operator's own kubeconfig. The token is never printed and never reaches
argv: it goes into a 0600 file and into the PUT body FILE (--data @file), and the
script refuses to run under `set -x` because xtrace would print it.
Two traps this repo already paid for are designed around:
* GITEA_ is a RESERVED secret-name prefix (HTTP 400 "invalid secret name"), so
the name is CONFORMANCE_TRIGGER_KUBECONFIG -- which is also what gate.yml and
merge-gate.yaml already reference.
* `curl -sS` exits 0 on an HTTP error. Every write checks %{http_code} AND reads
the object back. The secret read-back proves presence by NAME only (Gitea
never returns secret values) and the script says so rather than implying its
content was checked; the variable read-back asserts the value is exactly
"true", which is what gate.yml compares against.
PRECONDITIONS ARE SPLIT IN TWO, and the split is the security argument:
mint-blocking (cluster) -- SA/Role/RoleBinding present, and ClusterPolicy
conformance-trigger-job-shape present + Enforce + failurePolicy Fail + Ready.
Any failure and NOTHING IS MINTED. Kubernetes places no `use` check on a Job's
serviceAccountName, so a `create jobs` token in that namespace IS the privileged
conformance-runner identity unless admission bounds its shape. RBAC cannot close
that (plan §4); minting before the boundary is live hands CI the escalation the
boundary exists to close.
arm-blocking (repo) -- digest-pinned CONFORMANCE_RUNNER_IMAGE, an initContainer
plus the admitted image in chainsaw-job.yaml, and CONFORMANCE_TRIGGER_KUBECONFIG
referenced in ci/conformance.sh CODE rather than a comment. These block only
vars.CLUSTER_AVAILABLE, because storing the secret early is harmless (the job is
gated off) and is the plan's own order.
All three arm-blocking checks FAIL in the tree today, so a live run would store
the secret and then REFUSE to arm CLUSTER_AVAILABLE, naming each missing item.
That is the intended behaviour: measured 2026-09-29, CLUSTER_AVAILABLE=true with
no trigger path scaffold-skipped all 54 tests at rc 0 and the stage printed "L2/L3
conformance passed". ci/conformance.sh now blocks that case, so arming early does
not enable L2/L3 -- it only breaks every PR. Arming needs the explicit flag, every
precondition green, and a typed confirmation on a TTY.
Before storing anything, the credential is PROVEN narrow: auth whoami must be
system:serviceaccount:conformance:conformance-trigger, a real `get jobs` round-trip
must succeed, and an auth can-i matrix asserts both halves -- granted (create/get
jobs, get pods, get pods/log) and DENIED (delete jobs, create pods/exec, get
secrets, create jobs in default, get nodes, '*' '*' --all-namespaces). A mismatch
aborts before any store, and says explicitly not to widen
conformance/incluster/rbac.yaml to make it pass.
Verified only by `bash -n` (clean) and reading. The Gitea variables API shape
(POST creates, PUT updates, body field `value` not `data`) is from documentation,
not measurement: the script tries POST then PUT, prints the response body on
failure, and treats the read-back as the authority. Kyverno readiness is accepted
from either .status.ready or a Ready condition, and "neither present" blocks
rather than assuming ready.
Gate (pinned toolbox): MERGE GATE: UNDETERMINED (rc 3) -- spec-immutability and
gitea-settings unrun, the expected local result.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
0c6949ed98 |
feat(ci): the pipeline is the approver; humans approve changes to the pipeline
Implements the operator intent: "CI/CD pipeline is the end-all-be-all for approval. If the CI pipeline is not good enough to be trusted for this, then it is not good enough to do its intended job and needs to be improved. Manual approvers should only be for architecturally or security relevant changes." Adds ci/approval-policy.sh as a gate stage. A green gate becomes sufficient for an ordinary change; the one exception is structural rather than a matter of taste: A CHANGE TO THE GATE CANNOT BE GATED BY THE GATE. A PR that weakens a check would be approved by the weakened check. The same holds for the admission policies, the Flux wiring that decides what reconciles, the SOPS recipients and the RBAC. Those paths -- and only those -- still require an APPROVED review from someone other than the author. The list is data with a reason per entry (infra/gitea/approval-policy.json) and is itself a sensitive path, so the policy cannot be widened or narrowed without a human. FAIL-CLOSED, and deliberately the opposite of gitea-settings. There, a missing credential must not block every PR -- that is a lock-out. Here, a missing credential would let an unreviewed change to the gate merge, so "cannot evaluate" is a BLOCK. Verified against real PRs rather than fixtures: PR #16 (sensitive paths, approved by jakman85, author gitea-admin) -> PASS PR #6 (.gitea/workflows/gate.yml, ZERO reviews) -> BLOCK, naming the file push event (not a PR) -> N/A, exit 0 bogus token -> BLOCK, not skip Also caught before shipping: the first draft passed the PR file list with a second stdin redirect after a heredoc, which silently wins and leaves the script body unread. Running it found that; reading it would not have. required_approvals is NOT changed here. Flipping it live must happen together with the declaration in expected-branch-protection.json, or gitea-settings correctly fails on the drift -- that is a separate, deliberate change. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
50a48e6f07 |
fix(ci): the settings token could NEVER have been stored -- Gitea reserves the GITEA_ prefix
The drift detector merged in #11 was inert and would have stayed inert silently. WHAT HAPPENED. infra/gitea/set-ci-token.sh minted a token and PUT it to the repo Actions secret GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for secret names -- exactly as GitHub reserves `GITHUB_` -- and rejects that PUT with: HTTP 400 {"message":"invalid secret name"} The script sent the PUT to /dev/null and printed "==> stored." unconditionally, and `curl -sS` exits 0 on an HTTP error, so it reported success for a store that never happened. Measured afterwards: the token existed in the user's token list, and the repo's secret list was EMPTY. ci/gitea-settings.sh would have skipped on every CI run forever, and with REQUIRE_GITEA_SETTINGS=true it would instead have BLOCKED EVERY PR -- including the one fixing it. Three changes: * the secret is SETTINGS_API_TOKEN; ci/gitea-settings.sh reads it, keeping GITEA_SETTINGS_TOKEN as a local-run fallback; * set-ci-token.sh checks the PUT's HTTP status AND reads the secret back, refusing to claim success otherwise. That missing verification is the actual defect -- the wrong name was just what it hid; * gate.yml maps the secret and ENABLES REQUIRE_GITEA_SETTINGS. WHY ENABLING IT HERE IS SAFE: this PR's own CI run is the proof. If the stage were still skipping, REQUIRE turns that into a failure and this PR goes RED rather than merging. Green means the token resolved and live Gitea matched infra/gitea/expected-branch-protection.json. Verified locally before pushing: REQUIRE=true, token present -> MERGE GATE: PASS REQUIRE=true, no token -> gitea-settings SKIPPED -> BLOCKED (not a silent skip) The secret is already stored and read back as present, so CI has what it needs. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
e6f6e4febb |
feat(ci): detect Gitea settings drift -- the blind spot that hid a month of outage
Every check in this repo reads the repository, the rendered manifests, or the cluster.
NONE could see Gitea's own database, and that is where branch protection and runner
registration live. Three failures came out of that blind spot on 2026-09-29, all the same
shape -- a setting that READ as enforced while not being, with nothing able to notice:
* main had NO branch protection for ~a month while DECISIONS-LOG recorded it as a
required status check;
* ZERO Actions runners were registered, so PR #1 -- the first PR this repo ever had --
sat pending for 30.6 minutes with nothing to execute it;
* after enable_status_check and required_approvals were turned on they STILL did not
bite, because block_admin_merge_override was false: PRs #5-#9 merged with zero
approvals recorded against required_approvals: 1.
infra/gitea/*.sh ASSERT the desired state. Nothing DETECTED drift from it. This is the
detector.
ci/gitea-settings.sh runs as a merge-gate stage: it diffs the live rule at
/branch_protections/main against infra/gitea/expected-branch-protection.json and asserts at
least one runner is online. It reads the RULE object, not /branches/main -- the branch view
does not expose block_admin_merge_override and showed all-green while the rule was
overridable by the only account that merges.
Verified by test, not assumed:
no token -> exit 3, skips loudly, does NOT block
live == declaration -> exit 0
injected drift -> exit 1, blocks, naming both the drifted key and the runner count
The skip is DELIBERATELY not in gate.sh's strict-skips loop. Making an absent credential
block every PR is precisely the lock-yourself-out failure this repo keeps walking into, so
a skip denies a PASS (exit 3) and says loudly what is unverified. REQUIRE_GITEA_SETTINGS=true
upgrades it to a block once the token is wired.
gate.yml's `|| [ "$?" -eq 3 ]` tolerance now covers a second stage. That widening is
declared in the file rather than left to be discovered, and it narrows back the moment
REQUIRE_GITEA_SETTINGS is set.
infra/gitea/set-ci-token.sh mints the token and stores it as the repo Actions secret without
ever printing the value.
UNTIL THAT TOKEN EXISTS THIS STAGE SKIPS, AND A SKIPPED DETECTOR DETECTS NOTHING.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
fcc66c7858 |
fix(gitea): admins could override the gate -- the enforcement was not enforcing
enable_status_check and required_approvals were set earlier today and recorded as done. They did not bite. block_admin_merge_override was false, and Gitea lets repo admins override merge requirements without it. jakman85 holds owner, so both requirements were bypassable by exactly the account that performs every merge. The evidence was already on the record: PRs #5-#9 all merged while required_approvals: 1 was in force, with ZERO approvals recorded on any of them. Two process failures, both the same shape as the defect this whole session started with: 1. Verified with the wrong view. /branches/main reports protected, enable_status_check and required_approvals but NOT block_admin_merge_override -- it showed everything green while the rule was overridable. /branch_protections/main is authoritative. 2. Probed for a field name assumed to exist. This Gitea (1.24.3) has no `apply_to_admins`; asking jq for it returns null, which reads like "set to false" but means "key absent", so a PATCH that silently did nothing produced a convincing before/after. jq returning null is not a value -- it may be an absence. Enumerate with to_entries. require-gate.sh now sets block_admin_merge_override: true; block-admin-override.sh fixes it in isolation. Both carry the field-name warning. This PR is the test. If it merges WITHOUT an approval, the fix did not work. Also records the governance question: with only jakman85 (human) and gitea-admin (agent), a human-authored PR now needs the automation account to approve it. That is ceremony, not review, and the choice between keeping it, dropping approvals in favour of the status check, or adding a second human is the operator's to make. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b2d9883958 |
docs(ci): close out the hardening -- land the scripts, correct a claim that shipped wrong
The branch-protection + PR-gate + sidecar sequence is complete and verified. This records the finished state and fixes what the sequence itself proved wrong. CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT. Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one benign startup line would slow every subsequent poll. The manifest comment now states this instead of the wrong prediction. Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online, and a real workflow goes green on it, before enabling enforcement and probing the push. Re-run it after any future act-runner change. Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were deleted, so no prefix produces a duplicate run any more. CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with all four rows marked verified and the push rejection noted as PROVOKED rather than assumed. It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3 never runs in CI and only the L0/L1 offline subset is covered. infra/gitea/README.md marks the sequence done and names the root cause that remains unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by the conformance gate, so NOTHING detects drift from what this directory declares. These scripts assert desired state; they are not a detector. The check that reads the live API and fails on divergence is the highest-value work left, and it is the difference between this being fixed and this being unable to recur. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8e017fad91 |
docs(runbooks): the PR flow, not git push origin main -- that instruction is now blocked
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
234cc6a71a |
docs(gitea): a skipped job reports SUCCESS, not pending -- correct the deadlock claim
The runner came up on 2026-09-29 and the first CI jobs this repo has ever run went green, which falsified a claim merged in #2. MEASURED: gate / gate (pull_request) success 1m58s (PR 1), 1m34s (PR 2) gate / gate (push) success 1m32s (PR 2 only) gate / conformance (pull_request) success "Has been skipped" #2 said listing the conformance context as a required check would deadlock every PR, because the job is `if: false` and never runs. That is wrong. Gitea reports a skipped job as SUCCESS. Requiring it would not deadlock anything -- it would pass VACUOUSLY, forever, while nothing ran. The conclusion is unchanged (never require it) but the reason inverts, and the corrected reason matters more: under HARD RULE 4 a permanently-green check that verifies nothing is worse than a deadlock. A deadlock is loud. This is the exact fail-open shape the 2026-09-18 audit and this one both keep finding. Also records that the (push) contexts only appear for branches matching gate.yml's push trigger -- feature/**, fix/**, docs/**, chore/** -- and NOT feat/**, which is why PR 1's feat/ branch produced only pull_request contexts. Require the pull_request context alone, or PRs from non-matching prefixes never satisfy the rule. CLAUDE.md's enforcement table is updated: the runner half is genuinely fixed and observed green, the branch-protection half is not. Adds the warning that a green tick is not by itself evidence that anything ran. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fe862ab527 |
infra(gitea): recover the Actions runner before anything requires the gate
No runner is registered at repo, org or site level, so gate.yml has never executed. PR #1 sat pending for 30.6 minutes across 122 polls and never reported. Requiring the gate as a status check in that state would lock the repo permanently. Adds runner-token-rotate.sh: mints a fresh instance-level registration token from the Gitea API and writes it into the SOPS secret via sops set, without ever printing the value. Records the likely root cause, which a new token alone does NOT fix. act_runner registers once and persists its identity in /data/.runner on the act-runner-data PVC. If Gitea no longer holds a matching runner row, the pod starts, sees .runner, SKIPS registration entirely, and polls with an identity Gitea does not recognise. Symptom: zero runners, pod running, nothing obviously broken. Clearing the PVC is what forces a genuine re-registration. The README now sequences runner-first, enforcement-second, and requires CONFIRMING the runner reports online rather than assuming it. That assumption is the whole defect: DECISIONS-LOG recorded the runner as online on 2026-08-31 and nobody checked since. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a2665c0511 |
infra(gitea): script the branch protection that was never applied
Gitea repo settings live in its own database, not in a manifest Flux reconciles, so they are drift-prone and invisible to the conformance gate -- which is how main came to accept direct pushes while the log recorded it as protected. Adds infra/gitea/branch-protection.sh (blocks direct pushes to main) and a README carrying the measured before-state and the ORDER the remaining steps must happen in. The script deliberately leaves enable_status_check:false and required_approvals:0. Both would deadlock today: there is no runner to satisfy a required context, and the only human account is not yet a repo collaborator, so no one could supply an approval. The README sequences both. It prints a push probe to run afterwards. The API response is not proof -- Gitea may let site admins bypass protection, and that has to be observed, not assumed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
78d0f1b0f6 |
feat(ci): promtool in the pinned toolchain, one alert-rule harness, and the gaps declared
The CNPG alert unit tests (324 lines, mutation-checked) were run by NOTHING. promtool was
not in the baked toolchain, no gate stage evaluated any *.promtool-test.yaml, and the only
runner `curl | tar -xz`'d a promtool release from GitHub at TEST-RUN time if one was
missing. Hours later a second, independent harness landed for the Velero rules
(platform/monitoring/tests/) with its own fallback — a `podman run` of
quay.io/prometheus/prometheus. So the repo had two harnesses, two acquisition paths, and
coverage that looked present and was enforced nowhere: the exact failure mode that had just
produced a permanently-firing CNPG alert and a structurally inert Velero alert.
PROMTOOL IS NOW PINNED LIKE EVERY OTHER TOOL. PROMTOOL_VERSION=3.14.0 in
infra/devcontainer/versions.env and ci/versions.env (the two stay in lockstep, per the
repo's rule), baked by the Containerfile from the prometheus release tarball (promtool ships
inside it), recorded in /etc/dev-toolchain-versions, passed by dev.sh, and installed for the
l0 stage by .gitea/workflows/scripts/install-tools.sh so the CI runner is not permanently
SKIPPED. The pin is not free-floating: promtool IS the rule engine, so it tracks the
Prometheus prod actually runs — verified against the live cluster, not assumed:
kubectl --context k3s-prod get prometheus -n monitoring -o jsonpath='{.items[*].spec.version}'
-> v3.14.0-distroless (image quay.io/prometheus/prometheus:v3.14.0-distroless)
ONE HARNESS. platform/cnpg/run-alert-tests.sh is folded into
platform/monitoring/tests/run.sh, which now covers every PrometheusRule in the repo that has
tests, via two conventions it discovers automatically: `<manifest>.promtool-test.yaml`
beside its manifest (cnpg), and `<component>/tests/*.test.yaml` over that component's
manifests (monitoring). It writes the flattened rules under whatever name each test file's
own `rule_files:` declares, so no test file needs to know the harness exists, and it
`promtool check rules` first — a rules file that does not parse would make every unit test
pass vacuously by finding no rules at all. The cnpg script's reasoning is preserved: why a
CR must be flattened (`.spec.groups`, never re-typed), and why generated files go to a temp
dir. Its `${BASE_DOMAIN}` substitution is dropped in favour of extract_rules.py's verbatim
placeholders, with the trade-off recorded there — the hazard it guarded against is addressed
by the temp dir, and verbatim leaves one transformation fewer between the shipped rule and
the rule under test.
NO ENGINE IS ACQUIRED AT RUN TIME any more — not the GitHub release, not the container
image. Not because a pinned download is unusual here (the Containerfile downloads every tool
it bakes) but because those two decided the version, the network and the machine SILENTLY,
at test time, on whatever host ran the tests; and each quietly turned "the toolchain has no
promtool" into a green run, which is how the toolchain stayed without one. Missing promtool
is exit 3 (UNWIRED, never green); a PROMTOOL= that does not resolve is exit 2.
ONE WIRING: conformance/tests/l0-static/alert-rules-unit-tested, NOT a gate stage. The
`prom-rules` stage (ci/prom-rules.sh + its Makefile target + ci/gate.sh entry) is deleted:
it is the bespoke-per-check pattern
|
||
|
jakman85andClaude Opus 5
|
9b4744d450 |
fix(devcontainer): dev.sh defaults to prod — dev is decommissioned
Every unprefixed './dev.sh run kubectl ...' pointed at the dev cluster, which was decommissioned on 2026-09-12. The banner cheerfully named dev and the command then hung or errored against a context that no longer answers, with nothing saying the cluster is gone. I hit this repeatedly tonight. CLUSTER=dev is still accepted rather than removed, because the dev VM disks were deliberately kept so the environment is reversible — but it now warns. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
9161cab9cd |
feat(dns): prod takes the apex — home.keithjackson.dev
Dev is decommissioned and has released the name. Prerequisites completed and
verified BEFORE this flip:
1. dev's 18 A records released by its own external-dns (policy=sync; sources
deleted after suspending dev's Flux so they could not be recreated).
2. dev's 18 LEGACY UNPREFIXED TXT registry entries deleted from Cloudflare by
hand. external-dns v0.21.0 deleted only the `a-` prefixed form and then logged
"All records are already up to date" — it would NEVER have cleared the older
unprefixed format it no longer writes but still READS for compatibility. They
carried `owner=default`; prod runs `owner=prod`, so prod could not have
claimed these names while they existed. This was the real blocker, and it was
invisible until measured.
3. dev's `*.home.keithjackson.dev` + apex Certificate deleted so it stops
renewing. Issued ~18d ago, so prod's request is #1 this week against LE's
duplicate limit (5 per identical name set per week).
Verified at the AUTHORITY (betty.ns.cloudflare.com), not via a resolver: 1.1.1.1
served stale TXT for minutes after deletion and would have shown a false failure.
UNCHANGED, deliberately: EXTERNALDNS_TXT_OWNER_ID stays `prod` — giving prod
`default` would let it adopt and sync-delete records it does not own, which is the
2026-09-04 incident. EXTERNALDNS_DOMAIN_FILTER stays the ZONE `keithjackson.dev`:
finding F-DNS1 proved a strict subdomain filter drops the hosted zone and external-dns
then publishes NOTHING.
The zone has wildcards `*.keithjackson.dev` and `*.home.keithjackson.dev` pointing at
the public IP (legacy non-k3s prod). Dev's specific records were shadowing the latter;
prod's new specific records shadow it again. The legacy stack lives on
`*.keithjackson.dev` and is untouched.
k3s_base_domain moves with it to hold the [parity] contract, but that value renders
the apiserver's install-time oidc-issuer-url: it does NOT take effect until the
k3s_server role re-renders config.yaml and k3s is restarted serial: 1. Until then the
apiserver trusts the OLD issuer while Keycloak serves the NEW one, so kubectl OIDC
fails in that window. Cert-based kubeconfigs are unaffected.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
37a6c94168 |
fix(vpn): two gateways on one pinned PIA server evict each other every 2 minutes
Standing up prod broke dev, and the mechanism is worth spelling out because
"same PIA account, well inside its connection limit" is true and still not enough.
The vaulted .ovpn pins ONE server. With both gateways using it, dev and prod
landed on the SAME server AND were handed the SAME tunnel address — both journals
read `[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198`
and `ifconfig 10.72.192.7 255.255.192.0`. Each reconnect steals that address on
that server; the loser then receives nothing, trips `--ping-restart` after 60s,
reconnects and steals it back. Measured cycle: dev restarting at 22:57:05,
22:59:03, 23:01:05 UTC with prod interleaved about a minute off, each ~2 minutes
apart, indefinitely. Neither environment had reliable vpn-class egress — prod's
probes looked green only because they happened to land inside an up-window.
Fix: `vpn_egress_gateway_pia_remote` ("<host> <port>", empty = honour the .ovpn)
rewrites just the `remote` line of the deployed .ovpn, and prod's group_vars sets
it to 158.173.241.135. That is another member of the same PIA Stockholm LB pool —
`sweden.privacy.network` resolved from k3s-prod-3 to
158.173.241.{34,55,112,135,239} — and every server in the pool takes the same
credentials and the same inline CA, so only the address differs. Dev keeps the
.ovpn's .112 untouched, which is why the override is per-group rather than in the
role or a second vaulted .ovpn.
Two details that matter for whoever touches this next:
* The regex is anchored to a line STARTING with `remote` plus whitespace, so
`remote-cert-tls` and `remote-random` are not collateral damage. The task keeps
`no_log: true` — the transformation never widens what is logged.
* A PIA server address is NOT a credential, so it lives in
group_vars/vpn_egress_gateway_prod/vars.yml in the clear, next to the vault
rather than inside it. Keeping it out of the vault is what makes it reviewable.
If prod ever gets its own PIA account, this override can be deleted again.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
04606d48b2 |
docs(vpn): stop crediting the nftables kill-switch for a drop it never makes
Four places in the tree told the reader that the host nftables table is the authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress README, egress-steering.yaml, the role README, and the vpn-killswitch test's own GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway datapath bpf_redirect()s those skbs to the egress device without invoking netfilter POSTROUTING. Corrected with the measurement in each place rather than softened, because a guard the repo trusts and that does not fire is worse than no guard. What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops the traffic when it cannot use its cached egress interface, verified by tcpdump on both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out either). The nft table keeps a narrower, now-proven job: node-originated traffic sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on k3s-prod-3 when such a packet was routed at the WAN). Also recorded: * egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the gateway node is what tells Cilium it changed (the egress-uplink node label). A future reader's instinct will be to have the node write to that object; the comment says not to, because it is Flux-managed, and points at the label and `cilium-dbg bpf egress list` as the debugging entry points instead. * egress-steering.yaml — why the new NFS media mounts are not affected by pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns, so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server too, but that is the second reason, not the first. * vpn-killswitch test — its recovery step depends on the re-steer path, and could not have passed before it existed. Its no-leak assertion was always right; only the explanation was wrong. Also states the prerequisite plainly: the test cannot run at all without a labelled gateway node. * platform README — prod is live on k3s-prod-3 with the before/after egress IPs, and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway restored egress rather than moving it off the WAN. * role README — the standing "add the shared fwmark to the interface contract" escalation is closed as not implementable on Cilium OSS, rather than left open for someone to attempt. DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 / docs-03 amendment for the architect (specs are immutable to agents) and the five things left for the operator — chief among them that a silent gateway outage still has no alert, which is what let the dev gateway sit dead for 80 minutes. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
60e3f09c67 |
feat(vpn): prod egress-gateway group — k3s-prod-3, singleton, opt-in only
`[vpn_egress_gateway_prod]` is a SEPARATE inventory group with its own vaulted credentials, not a second member of `[vpn_egress_gateway]`. That group is the default target of playbooks/vpn.egress.gateway.yml, so adding a prod host to it would mean a flagless `ansible-playbook playbooks/vpn.egress.gateway.yml` silently reconfigures a prod node's kill-switch datapath. Prod is reached only by asking for it, with `--list-hosts` first: -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 Node 3 because node 1's SSD was replaced yesterday (least operational history) and node 2 carries the §0b memattr history. Exactly one node, forever: Cilium OSS picks "the first node in lexical ordering" among nodeSelector matches, so a labelled k3s-prod-1 would silently steal the gateway role — and the public egress IP — from k3s-prod-3, with no event and no alert. vault.yml is a verbatim copy of the dev group's CIPHERTEXT: same PIA account per the operator, and copying the encrypted blob means the plaintext was never decrypted to disk, through a shell, or into a transcript. The two files are now independent — rotating means re-encrypting both, which is the right behaviour for two environments. Also corrects a stale inventory note that claimed k3s-prod-3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it as the gateway, because a second live uplink in a kill-switch datapath is not a footnote: there is no wl* netdev on the node at all, no phy80211 device, the /etc/network/interfaces wlp1s0 stanza is commented out, 192.168.2.195 is assigned nowhere, and the only IPv4 addresses are lo and eno1 192.168.2.12. Nothing needed bringing down. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6f698f59af |
fix(vpn): Cilium never re-resolves tun0 — the gateway dies on every reconnect
Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the dev gateway silently dead for ~80 minutes before this change: 1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's `interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the map to 1987 and egress recovered instantly. `egress-gateway-reconciliation- trigger-interval: 1s` does NOT cover netlink link events — dev has that same 1s setting and still did not self-heal, 20s+ after recovery. Fix: vpn-egress-resteer.sh publishes "<tun0 addr>-<ifindex>" as a node label. A node update makes the gateway re-process the policy (verified: cached ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the policy, because the policy is Flux-managed GitOps and the node's labels are already this role's territory. Fired by the OpenVPN up-script on every connect, plus a 2-minute convergence timer for the boot case where the kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the identical sequence stayed broken indefinitely. 2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only diagnostic table on the gateway node recorded, while a vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA range, against 45166 packets on the same postrouting hook for WAN traffic and 16 packets for a node-originated curl out tun0. Cilium's eBPF datapath bpf_redirect()s these skbs to the egress device without invoking netfilter. So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods. What actually fails closed is Cilium's own drop when the cached egress interface is unusable — tcpdump during a tunnel stop AND during a restart showed 0 packets leaving either the WAN or tun0. Every rule now carries a `counter` so the claim stays falsifiable, and every comment that called this table the authoritative vpn-class backstop is corrected rather than left to mislead the next reader. Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable: nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is a hazard to whatever picks it next. 3. The up-script flushed the nft source set on every connect, so a packet still sourced from the previous tunnel address matched neither the ip-rule nor the drop rule. The set is now additive (`add element` is idempotent; the down-script already retained it): an element can only ever tighten the guard. Finding 2 means that window could not have leaked POD traffic, but node-originated tunnel traffic does traverse netfilter, and additive is free. Kill-switch reloads no longer disarm the guard either: the .nft file uses `table` + `flush table`, which clears rules but leaves the named set's elements intact, so an Ansible run cannot empty the set mid-flight. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
a624c638ef | merge: docs/node-1-drive-replaced | ||
|
jakman85
|
90b39de527 | merge: fix/swap-off-persists-across-reboot | ||
|
jakman85andClaude Opus 5
|
df1d2ab44d |
docs(prod): node 1's drive was replaced — and §2's exit condition still is not met
Records the 2026-09-07 I/O-starvation incident and the drive swap: the C400 was replaced with a Samsung 850 EVO, and under the same load class (an active 18-replica Longhorn rebuild, with node 1 as the write target) I/O pressure fell from full avg300 11.36 to 3.79 — from ~4x worse than its peers into node 3's band. Updates the three §0 cells that had become false (disk model, drive health, kernel — all three nodes now run 7.1.8, not 6.12) and adds §0c rather than rewriting the 'NOT hardware-symmetric' analysis, which is still the correct reasoning about the state it described. Two things the rebuild taught, recorded so the next one is cheaper: Longhorn refuses a replaced disk with DiskFilesystemChanged and needs disable -> remove -> re-add in that order, and the inventory's [k3s_prod_primary] decides BOTH --cluster-init and k3s_api_host, so rebuilding the founder without moving that designation would have founded a second cluster. The replacement does NOT close §2: etcd and Longhorn still share one SATA spindle. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
69a25cff82 |
fix(node_health): say 'active via <unit>', not a bare parenthetical
The unit name in the swap-active detail read as an unlabelled aside. Naming it as the unit that activated the swap is what makes the line actionable. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c1e36d9353 |
docs(node_baseline): say why the Longhorn claim guard's swap list is now always empty
The claim guard builds node_baseline_lh_protected_devices from ansible_mounts plus `swapon --show`. With swap.target masked, a baselined node has no active swap, so that half of the list contributes nothing — which reads like a guard that quietly stopped guarding. It is not: the claim candidate comes from whatever is mounted at node_baseline_longhorn_claim_mountpoint, and a swap partition is neither mounted nor ext4, so it could never be the candidate. Say so where the next reader will ask. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
9d00a3fcd3 |
fix(node_health): memory.swap blamed the wrong thing, and could not see the defect
The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry; this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod nodes HAD been baselined, had their fstab swap line commented out by the role, and were still swapping 977 MiB each after a reboot. Anyone reading that WARN would have re-run the role that had already run, and watched swap come back next boot. The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at every boot, without reading /etc/fstab — says in as many words that this is not "the node has not been baselined", and points at `node-baseline.yml --tags system`. The check was also blind in the other direction: SwapTotal == 0 is not evidence that swap is disabled, only that nothing has been swapped on yet this boot, so a node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly the state the old role left behind. The collector now reports memory.swap_persistence (swap.target masked?, .swap units the generators emitted this boot, which of them are masked, leftover active fstab swap lines — all world-readable, no root, no systemctl parsing), and the check WARNs on "clean now, swapping after the next reboot". Rendered against every fact shape (swap active + unmasked, swap off + unmasked, after the fix, a dev VM with no swap partition at all, per-unit mask only, persistence unreadable, an older collector's output with the key absent, and /proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot: [PASS] memory.swap — no swap active and none can be activated: swap.target is masked Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0e92d258dd |
fix(node_baseline): swap-off now survives a reboot — mask swap.target
The role's `swapoff -a` + commented fstab line held only until the next boot,
on every prod node. Measured 2026-09-07: all three came back from a reboot with
977 MiB of swap ACTIVE (k3s-prod-1 /dev/sda4, k3s-prod-3 /dev/sda4, k3s-prod-2
/dev/nvme0n1p4) while their fstab swap entry was commented out by this role. The
role reported success and the setting silently expired.
CAUSE. systemd-gpt-auto-generator claims any GPT partition typed
0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE,
synthesises a .swap unit into /run/systemd/generator.late/ and links it into
swap.target.wants/. It never reads /etc/fstab, so neutralising fstab cannot stop
it. Re-running the generator by hand on k3s-prod-1 says so verbatim:
Probed fstype 'swap' on partition /dev/disk/by-diskseq/1-part4.
Adding swap: /dev/disk/by-diskseq/1-part4
MECHANISM. Mask swap.target. `systemctl show` on the generated unit says its only
inbound dependency is `WantedBy=swap.target` — Requires/RequiredBy/BoundBy empty,
and the matching .device unit has an empty Wants=, so a device appearing does not
pull its swap unit in by itself. swap.target is the single choke point for every
activation path: gpt-auto, systemd-fstab-generator, systemd-cryptsetup-generator.
It is device-shape independent (no device name appears in the change, so one task
covers sda4, nvme0n1p4 and dev's vda VMs), boot-safe (swap.target is Wants=/After=
from sysinit.target and After= from tmp.mount; RequiredBy= is EMPTY on all three
prod and all three dev nodes, and a masked unit reached only via Wants= is skipped,
not failed), and inert if the device is absent or renamed. The generated units are
masked too, as a second independent barrier.
Rejected: `systemd.gpt_auto=0` (disables GPT auto-discovery wholesale, needs a GRUB
edit + reboot to become true, and on DEV would switch off root auto-discovery — dev's
vda1 carries the discoverable GUID 4f68bce3-e8cd-4db1-96e7-fbcaf984b709); clearing
the partition's GPT type GUID (a partition-table write to the live boot disk of a node
serving etcd and Longhorn replicas, and an irreversible decision about that 977 MiB
that belongs to the operator); `systemd.swap=0` (systemd-fstab-generator only — this
swap does not come from fstab).
`swapoff -a` is now conditional on `swapon --show` returning something, so the task
reports honestly instead of hiding a command that ran anyway behind changed_when: false.
PROVEN on k3s-prod-1 (all 18 Longhorn volumes healthy before the reboot):
run 1 changed=2, run 2 changed=0
after `reboot`: /proc/swaps empty, `free` Swap 0B, swap.target masked, the
generated unit still emitted by the generator and still masked, `systemctl
--failed` empty, `systemctl is-system-running` = running.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
08fbe85638 |
fix(ansible): move the founder designation to node 2
Node 1's SSD failed and was replaced; Debian was reinstalled, so node 1 is now a
JOINING node with no cluster state. It was still listed in [k3s_prod_primary],
and that group decides two things at once:
1. which play gets k3s_server_mode: primary -> --cluster-init, which FOUNDS A
NEW CLUSTER, and
2. what k3s_api_host resolves to -- the --server join target.
So re-provisioning node 1 unchanged would have pointed a rebuilt, empty node at
ITSELF and founded a second cluster beside the live one. The role's
`creates: /etc/rancher/k3s/k3s.yaml` guard does not help: on a fresh install that
file is absent, so cluster-init WOULD have run.
Node 2 is the senior surviving server (up since 2026-09-01) and takes the founder
slot. Its own cluster-init task is inert precisely because k3s.yaml already exists
there. Verified after the change: k3s_api_host renders 192.168.2.11, and
groups[k3s_primary_group][0] is k3s-prod-2.
This is a latent gap the role documented but did not handle -- its comment assumes
the founder is always alive ('On a WARM single-node rejoin the founder is up and
answering too'). Rebuilding the founder itself was not covered.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
863af01062 |
docs: the §2 shared-device measurement came back, and it does not hold
infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as "accepted to be MEASURED, not assumed", with an exit condition. The measurement arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage. §2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400 with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that reasoning assumed, and §0 had already named the three-way drive asymmetry as the fault line. The exit condition is OPEN and the second bullet (dedicated device) applies; that remains the operator's. Also records what the three software mitigations in this branch do and do not buy, including the point that no cgroup or scheduler control arbitrates the ext4 journal both tenants share — and the gap that etcd's own fsync histograms, which the exit condition literally asks for, are not collected because kubeEtcd is disabled for k3s. PSI is the signal that exists. DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the finding that concurrentReplicaRebuildPerNodeLimit was already set and already held during the outage, the third instance of config that looked applied and was not (orphanAutoDeletion), and the three things still requiring the operator. infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented out" of the inventory. They have been live in [k3s_prod_secondary] since 2026-09-03. That sentence is exactly the kind used to reason about blast radius — `-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the --list-hosts command that settles it. The new etcd-io-priority play is added to the run order, the layer table and the prod command list. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4) |
||
|
jakman85
|
f7a9229d81 |
fix(ansible): the NVMe node would have died on an undefined variable
Self-review of the role I had just written. On k3s-prod-2 the scheduler path is
deliberately skipped, so the `stat` on /sys/fs/cgroup/system.slice/k3s.service is
skipped too — and a SKIPPED registration has no `.stat` key at all. Ansible
templates `loop:` BEFORE it evaluates `when:`, so
loop: "{{ ['system.slice'] + (['system.slice/k3s.service']
if _eio_k3s_cgroup.stat.exists else []) }}"
raises on that host even though the task would never have run. The role would have
failed on the one node it is designed to leave alone — and it would have failed at
the very end, after doing nothing, which reads like a broken mitigation rather than
an intentional skip.
`| default(false)` on both uses. Exercised offline against a simulated skipped
registration; `--syntax-check` re-run.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit 5ccf5111a54857abb934a64c41332b7aa5046323)
|
||
|
jakman85
|
a819c50ffe |
feat(ansible): arbitrate the shared etcd/Longhorn device — and prove it landed
roles/etcd_io_priority + playbooks/etcd-io-priority.yml, enabled for [k3s_prod]
only. Mitigation for the 2026-09-07 k3s-prod-1 outage: etcd's WAL fdatasync took
3.4-6.7 s (expects 1 s), the node went NotReady while the host stayed up, and
/proc/pressure/io full avg300 read 11.36 there against 0.75 (node 2, NVMe) and
1.29 (node 3).
WHAT IT DOES, on the SATA nodes only:
1. k3s-etcd-io-priority.service (oneshot, Before=k3s.service) switches the
shared device to BFQ and drops nr_requests 64 -> 32.
2. IOWeight=10000 drop-ins on system.slice AND k3s.service, which BFQ turns
into io.bfq.weight=1000 — a 10:1 share against the default-weighted
kubepods.slice where Longhorn's instance-managers live.
3. Reads all of it back out of the kernel and ASSERTS. If the scheduler did
not switch or the weight did not land, the play fails.
THE THREE OBVIOUS ANSWERS ARE INERT ON THESE NODES, ALL CHECKED LIVE, ALL
DELIBERATELY ABSENT:
* IOSchedulingClass=realtime on k3s.service. mq-deadline DOES honour the
priority class, so this looks right — but ioprio is inherited across fork
and kept across exec (block/blk-ioc.c __copy_io copies ioc->ioprio) and
CONTAINERD IS A CHILD OF THE K3S SERVER PROCESS. Verified on k3s-prod-1:
containerd pid 84553 has ppid 84333 = k3s-server, and
ionice -c2 -n0 -- sh -c 'ionice -p $$; sh -c "ionice -p \$\$"'
prints best-effort prio 0 at both levels. Realtime for k3s is realtime for
every container on the node, Longhorn included. No relative gain.
* IOSchedulingPriority (the level inside a class): mq-deadline buckets by
CLASS only, so BE-0 vs BE-4 is a no-op.
* io.latency, which would be the ideal scheduler-independent protection:
CONFIG_BLK_CGROUP_IOLATENCY is not built in Debian trixie's 7.1.8 kernel
(only IOCOST is) and no io.latency file exists under /sys/fs/cgroup.
AND ONE NON-OBVIOUS TRAP: bare IOWeight= is DECORATION here. systemd writes it
to io.weight (blk-iocost — compiled in but disabled, io.cost.qos is empty) and
io.bfq.weight (BFQ — not loaded). Neither reader exists on the stock nodes, so
the scheduler change and the weights have to be one role with one verification.
The value is 10000 rather than a "reasonable" 1000 for the same reason: systemd
rescales 1..10000 onto io.bfq.weight's 1..1000, so 1000 maps to ~101 against
BFQ's unweighted default of 100. The role asserts the result is >= 500.
k3s-prod-2 IS SKIPPED ON PURPOSE. The role discovers the device under
k3s_data_dir per node (verified: prod-1/3 -> /dev/sda5 -> sda, prod-2 ->
/dev/nvme0n1p5 -> nvme0n1) and writes NOTHING on NVMe rather than leaving inert
files behind — BFQ is a poor fit for NVMe and node 2 measured 0.75 IO PSI.
DEV IS UNTOUCHED. Master switch defaults false and is set true only in
group_vars/k3s_prod; dev has a dedicated etcd device (docs/plans/ssd-etcd-
isolation.md) and nothing to arbitrate. Verified with --list-hosts: no extra
vars -> the three dev hosts (where the role is a no-op); -e k3s_hosts=k3s_prod
-> the three prod hosts; --limit k3s-prod-3 -> one.
NOTHING IS RESTARTED. The oneshot writes two sysfs files; the weights go in
through daemon-reload. If daemon-reload turns out not to re-realize a running
k3s.service's cgroup, the verification FAILS and names the rolling restart as
the fix rather than quietly leaving the weight unapplied.
WHAT IT CANNOT DO, stated in the role README and worth repeating: it does not
separate the two tenants. Both still write one ext4 filesystem, and no cgroup
control arbitrates the shared jbd2 journal that etcd's fdatasync waits on. It is
also UNPROVEN on this hardware — it fails loudly if the settings do not land, it
cannot tell you they helped. Apply to k3s-prod-3 first and watch
NodeIOPressureHigh. Replacing the C400 / giving etcd its own device remains the
real fix and the operator's (PROD-PROVISIONING.md §2 exit condition).
Syntax-checked, and the fact-derivation and assert expressions were exercised
offline against all three node shapes plus negative controls (a node still on
mq-deadline fails the gate; IOWeight=1000 -> 101 fails the floor).
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit de8c1f832724c011b03466842e18296de07a6776)
|
||
|
jakman85
|
3ca4ad6011 | merge: R2 off-cluster backups + per-environment backup credentials | ||
|
jakman85andClaude Opus 5
|
708a05dfc1 |
fix(devcontainer): restore multi-cluster kubeconfig selection
infra/devcontainer/kubeconfig.sh was never staged into the delivery-A commit, so the branch built on main carried main's single-cluster tooling and `--context k3s-prod` failed with 'context does not exist' mid-incident. Not Flux-reconciled, so it never affected the cluster — but it breaks prod diagnosis from a checkout of this branch, which is when you need it most. Carries the concurrency fix: the merged kubeconfig holds both contexts and NO current-context, with each invocation getting a per-invocation selector file prepended to KUBECONFIG. Two concurrent dev.sh runs with different CLUSTER values previously overwrote each other's current-context mid-command — reproduced live, a CLUSTER=dev reader returned k3s-prod nodes on 3 of 7 iterations. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6a8953e288 |
feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision
|
||
|
jakman85andClaude Opus 5
|
9739e9be91 |
chore(backup): R2 credentials for the off-cluster DR target
Written by platform/backup/set-r2-credentials.sh (operator, 2026-09-07T05:41:34Z) into all four consumers, in the two shapes they need: velero wants an AWS INI file, CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY. All four verified to carry the SAME 32-character key id, so no consumer was missed. Proven from a pod INSIDE prod against https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com: head-bucket on k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Byte-integrity of GET is NOT yet proven (the aws-cli image has no cmp); s3check is what will establish it. Credentials only — no endpoint or bucket wiring. That is feat/r2-backup-prod. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
d1a6628a97 |
feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
|