Compare commits

..
11 Commits
Author SHA1 Message Date
jakman85andClaude Opus 5 72f0ba4e1f fix(qbittorrent): the comment warning about doubled dollars contained one
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
The new flux-postbuild-substitution guard failed the merge on
apps/wave-0/qbittorrent.yaml:155 — a run of two dollars in a substituted
path that nothing justifies. It was inside the postStart hook's own comment
EXPLAINING the hazard: prose about what envsubst does to a doubled dollar,
written with a literal doubled dollar.

Functionally harmless — it is a bash comment, and collapsing it changes no
behaviour — but the guard cannot distinguish a comment from content that
reaches the workload, and it is right not to try: that is precisely how the
nextcloud-db break slipped through. Reworded in words instead.

Worth recording that this is the second time today the same trap caught an
explanation of itself: my own fix for the nextcloud-db incident (16b7fcd)
also shipped dollars inside the comment describing the danger, and I only
caught it because I re-read the file. This time the guard caught it, on work
merged minutes earlier — which is the guard earning its place.

Re-rendered so the golden matches the catalog.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:38:14 -05:00
jakman85 088ff13d15 merge: coverage debt paid — 51/51, and the root really did hide a service
R-C2-01 and R-C2-02 were the last two uncovered requirements (49/51). Now
100% with 0 gaps, via an offline L0 check (49 assertions) that compares two
repository facts: every home-zone hostname the repo routes over HTTP must be
linked from the apex dashboard, and every link the dashboard offers must
resolve to something the repo actually serves at L7 or L4.

'A service the root hid' was still a live defect: crafty.${BASE_DOMAIN} was
served and reconciled while nothing on the root linked to it. Committed RED
first so the history shows the check catching it, then fixed.

Also fixes a regex that could never have worked: ci/touched-requirements.sh
matched R-[A-Z]+-[0-9]+, where [A-Z]+ eats the C of R-C2-01 and then demands
a dash where the 2 is — so a manifest declaring either ID attributed nothing.
Every other consumer already used [A-Z0-9]+.

16 mutations proven caught. R-TLS-02's drafted amendment is recovered into
DECISIONS-LOG: revocation is not implementable with cert-manager, and the
choice between amending the requirement and a reviewed DOCUMENTED_GAPS entry
is the owner's.

# Conflicts:
#	DECISIONS-LOG.md
2026-09-13 16:34:54 -05:00
jakman85 cc648ba02c merge: qBittorrent has a durable credential, and the arrs work again
qBittorrent 5.x minted a new temporary admin password on every process
start, so the Sonarr/Radarr download clients broke on the next restart — and
had already broken: both reported isValid: false, Authentication Failure.

A SOPS secret is now the single source of truth and a postStart hook derives
qBittorrent's stored PBKDF2 form on every container start, then logs in to
prove it. The hook rather than a CronJob because the failure IS a container
start: qbittorrent-nox reads the credential once at start from an RWO PVC the
app holds, so a 15-minute cron leaves a window after every restart where both
arrs are broken. An initContainer cannot do the ground-truth login.

Proven: testall flipped false -> true on both arrs, and survived a real pod
deletion and recreation with the conf still holding both keys.

Two bugs found in the inherited work, both the envsubst trap again: a
doubled-dollar port expansion that bash read as its own PID, which made the
hook report 'the WebUI rejected the credential' three times without ever
asking it anything; and one shared wait budget that let the first attempt
burn all 90s. Also caught that the SOPS file was never listed in
apps/wave-0/kustomization.yaml, so Flux would never have applied it — the
live Secret was not going to be adopted at all.
2026-09-13 16:34:35 -05:00
jakman85andClaude Opus 5 8b36242643 fix(monitoring): kube-state-metrics was OOMKilled 91 times — 13 rules went intermittent
Working set measured at 111 MiB with a 6h peak of 123.3 MiB against a 128Mi
limit, so it died on every spike — exit 137, roughly every 11 minutes, 91
restarts in 16h.

The consequence is worse than the restarts. Thirteen alert rules read kube_*
series, so each kill made them intermittently unable to fire, which is the
quiet version of the inert-alert class found elsewhere today; KubePodNotReady
and KubeDeploymentReplicasMismatch were both sitting pending on it.

Self-inflicted: enabling customResourceState for the Velero Schedule metrics
earlier today added a CRD informer plus the mandatory CRD list/watch, and
that marginal cost crossed a limit already sitting about 4 MiB above the
working set. Raised to 384Mi, roughly 3x the observed peak, which costs
nothing on nodes reporting 18.7-22.3 GiB available.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:32:28 -05:00
jakman85andClaude Opus 5 e4bd76070e fix(qbittorrent): a durable WebUI credential, and the two bugs the WIP hid
Finishes the inherited WIP. qBittorrent 5.x had no WebUI\Username or
WebUI\Password_PBKDF2 in its conf, so it minted a NEW random admin password on
every process start. Both consequences had ALREADY fired: measured before this
change, `POST /api/v3/downloadclient/testall` returned isValid:false with
"Authentication Failure" on BOTH Sonarr and Radarr, so Completed Download
Handling was silently broken, and no human had a stable login either.

Mechanism: the SOPS/age qbittorrent-bootstrap Secret is the single source of
truth; a lifecycle.postStart hook derives qBittorrent's own stored form
(PBKDF2-HMAC-SHA512, 100000 iterations, 16-byte salt, 64-byte digest) and
asserts it into the conf on EVERY container start, then logs in to the WebUI to
prove it. Level-triggered like the repo's CronJob reconcilers, but triggered by
the container start because the failure it closes IS a container start: a cron
would leave a window after every restart in which the app serves a random
password, and would have to edit an RWO PVC from outside, under a running
process. AuthSubnetWhitelist was rejected — authentication is not weakened.

Two real bugs found in the WIP, both the postBuild-substitution trap:

  * PORT came from a doubled-dollar WEBUI_PORT expansion. Correct on the cluster
    (envsubst collapses it) and the renderer accepts it, but run any other way
    bash reads the doubled dollar as its PID, so every curl got an unparseable
    URL and the hook logged "the WebUI rejected the durable credential" three
    times having never asked it anything — see the marker log at 05:35Z. Now a
    literal PORT=8080; the script contains no braced expansion at all and
    behaves identically however it is run.
  * One shared wait budget covered both the s6 wait and the API wait, so the
    first consumed all 90s and attempts 2-3 fired instantly. Separate counters
    now; "the API never answered" is reported as a distinct fault from "the
    password was rejected"; and a rejection restarts qbittorrent-nox so a retry
    can actually change its outcome.

apps/wave-0/kustomization.yaml did not list the SOPS secret, so Flux would never
have applied it and the adoption of the live-only, kubectl-client-side-apply
qbittorrent-bootstrap Secret would silently not have happened. Listed now; the
decrypted values are SHA-256-identical to the live Secret's, so adopting it
cannot change the password.

Proven on prod: testall flipped isValid false -> true on both apps; after
deleting and recreating the pod the conf still held both keys, the WebUI answered
Ok. to a login with the credential decrypted from git, and both arrs still
reported isValid:true. The post-envsubst manifest was audited as `flux build
kustomization` emits it (no doubled dollar, no braced expansion, PORT=8080,
bash -n clean) and the hook's write path was exercised in-container against a
scratch conf: exit 10 when the keys are absent, exit 0 and a byte-identical file
on a second run, digest verified by an independent PBKDF2.

NOT proven, and recorded as such in DECISIONS-LOG: the hook has not yet run as a
real postStart on prod — the Deployment is Flux-owned from main and both routes
to the live object were refused by the session's permission classifier. Verify
on merge that the marker log gains a VERIFIED line on the next pod start.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:31:58 -05:00
jakman85andClaude Opus 5 aff80d1d7a docs(pm): R-TLS-02 needs the owner — the drafted spec amendment, recovered and restated
The R-TLS-02 analysis and its drafted spec amendment were written on
feat/coverage-debt-paydown (b37defe, 2026-09-07) and lived ONLY in that branch's copy of
DECISIONS-LOG.md. That branch's tree is ~107 commits behind main and must never be merged,
so the amendment went out of view with it — the second time this decision has been raised
and lost. Recovered, re-checked against today's main, and written where it cannot vanish
with a branch.

THE SUBSTANCE IS UNCHANGED. R-TLS-02 says certificates MUST be automatically "issued,
renewed, and revoked without manual steps". Issuance and renewal hold. Revocation is not
implemented and is not implementable with the stack ADR-0006 selects: cert-manager has no
revoke-on-delete, no Certificate field requesting revocation, and no controller that calls
ACME revokeCert, and neither ADR-0006 nor docs/08 specifies a mechanism.
enableCertificateOwnerRef is NOT the remedy — it garbage-collects the Secret, which is
cleanup, not revocation — and it is a global controller flag that would drop the Gateway's
wildcard-home-tls Secret on any future Certificate rename/prune, so it needs its own risk
assessment rather than a side effect of a test task. Still not set.

WHAT CHANGED SINCE THE FIRST ESCALATION, and why it now matters more: R-TLS-02 is no
longer a COVERAGE gap (l0-static/tls-lifecycle-automated + l2-behavioral/
cert-lifecycle-automated both carry the label and main counts it covered), and the L0 check
is honest about the boundary in its own verdict line ("'revoked' is NOT proven at any
layer"). But the L2 test deletes a Certificate and requires its Secret to disappear, and
its failure message says "set enableCertificateOwnerRef: true ... (owner decision — WS-TEST
does not weaken the assertion instead)". That assertion is written to FAIL on a live run
until the owner decides. The decision is now arriving as a red test rather than as a
question, which is exactly how it gets weakened by whoever meets it next.

Two options are recorded verbatim for the architect, in the drafted preference order:
amend the requirement so revocation is an explicitly manual, documented break-glass action
(which also makes enableCertificateOwnerRef a CONSEQUENCE of the requirement), or keep the
text and record a reviewed DOCUMENTED_GAPS entry naming the clause, the reason and the
removal condition.

NOT APPLIED, and not appliable by an agent: the spec repo is read-only under R-TEST-06.
This is a request for a human decision, stated plainly as one.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:28:21 -05:00
jakman85andClaude Opus 5 a76ae3bdfa fix(ci): the content scan could not match R-C2-01/02 — a digit in the workstream segment
Checked as part of closing the C2 coverage gaps, and it was live on main. The content
scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match an ID whose workstream segment
carries a digit: `[A-Z]+` consumes the `C` of `R-C2-01` and then demands a `-` where the
`2` is. `R-C2-01` and `R-C2-02` are the only two such IDs in
conformance/requirements/registry.yaml, and the scan was blind to BOTH.

REPRODUCED before the fix, on a throwaway repo whose only change is a manifest declaring
both IDs on real (non-comment) lines, in a path the map keys to nothing:

    $ REPO_ROOT=$T BASE_REF=main ci/touched-requirements.sh
    R-NET-06                     # the all-letter control ID was the ONLY output

It failed by UNDER-attributing, which is the fail-OPEN direction: a change that retyped a
`conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02, contributed
those IDs to NOBODY, so the C2 tests were filtered out of the merge gate. Same defect
class as the map's `*) : ;;` arm — a miss that reads as "nothing touched".

Every other consumer in the repo already allowed the digit, which is why this was the lone
dissenter: test_requirements_for_path() twenty lines above in this same script,
conformance/lib/coverage.py, conformance/lib/status.py and conformance/lib/
run_l1_offline.py all use `R-[A-Z0-9]+-[0-9]+`. Aligned to that form, with the reason
recorded at the line.

SCOPE OF THE HOLE. path_requirements() has no arm for platform/homepage/* or
platform/headlamp/* either, so before this the ONLY route to R-C2-01/02 was detection rule
3 (editing the C2 conformance test itself). No path-map arm is added here: c2-root-surface
declares `scope: whole-repo`, so ci/l0.sh Stage 3b's --include-scope union selects it on
EVERY gate invocation regardless of the touched set — which is the right mechanism for an
invariant whose input surface is the whole tree, and strictly stronger than a two-component
path arm would be.

REGRESSION-TESTED, in ci/tests/touched-requirements.test.sh (the suite `make -C ci test`
runs): a fixture declaring R-C2-01 / R-C2-02 / R-OBS-04 from `notes/` — a path outside
path_requirements(), so the IDs can only come from the content scan — asserts both digit
IDs ARE attributed, that the all-letter form still is (control), and that a purely numeric
workstream is not invented (the pattern must not widen the gate on noise).

PROVEN TO DETECT BREAKAGE: with the pattern reverted to `R-[A-Z]+-[0-9]+` the suite reports
`FAIL (2 failed, 14 passed)` naming both C2 IDs as missing; restored, all three ci/ suites
pass (16 assertions in this one).

ALSO: conformance/tests/l0-static/whole-repo-invariants-wired/check.sh cites this pattern
in the comment explaining why its BOGUS_REQ fixture is lowercase. Citation updated so the
reasoning still reads true; the fixture is unaffected (grep -E is case-sensitive, and the
check still passes 26/26).

NOTE: another branch may be editing this same script concurrently. This is a one-line
change to one statement plus its comment — expect a trivial conflict and keep the
`[A-Z0-9]` form.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:28:21 -05:00
jakman85andClaude Opus 5 02c58aae15 fix(homepage): the C2 root hid a running service — Crafty had no tile
The R-C2-01 check added in the previous commit is RED on main for a real reason.
platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} on the shared Gateway,
clusters/base/crafty.yaml reconciles it (dependsOn platform-config, wait:false), the web
UI is Keycloak-gated by a per-pod oauth2-proxy sidecar — and nothing on the operator's
front door linked to it. The service was running and the root did not admit it existed.

R-C2-01: "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services
running in the cluster and links to them." An unsurfaced route is a requirement gap, not
a missing nicety, and it is the exact failure mode nothing else in the gate notices: every
cluster test stays green, the apex answers 200, and the estate is one service larger than
the dashboard says.

This is the SAME defect the unmerged 49c160d found on 2026-09-07 (feat/coverage-debt-
paydown, ~107 commits behind main). That branch was never merged, so the gap survived —
re-fixed here against today's tree, in token form (`crafty.${BASE_DOMAIN}`, not the
literal hostname that branch predated).

Grouped under Media because Crafty manages game servers; Platform would be equally
defensible and purely cosmetic. The tile also does duty for the L4 game ports
(crafty-mc.${BASE_DOMAIN}, TCP 25565-25567 / UDP 19132-19134), which have no browsable URL
of their own — the check reports those as an advisory rather than requiring a link.

ALSO: clusters/base/homepage.yaml's postBuild comment claimed "Homepage references no
${VARS} today; the substitution is retained for parity". That is no longer true and the
claim is dangerous, because it invites someone to delete the substitution as dead weight.
This component now references ${BASE_DOMAIN} in httproute.yaml's apex hostname AND in every
tile href; without substitution the token is applied literally, the apex route binds a host
matching no listener, every link on the root is dead, and Flux still reports Ready. Comment
corrected, and the property is asserted by c2-root-surface (5) so it cannot rot back.

Verified: conformance/tests/l0-static/c2-root-surface -> check OK, 49 assertions.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:28:21 -05:00
jakman85andClaude Opus 5 f57c411953 test(conformance): L0 covers R-C2-01 / R-C2-02 — 49 assertions, and it is RED
The Test DECLARATION landed in the preceding WIP commit with no check.sh, so the test
could not run at all. This is the check: 49 offline assertions over the repository as
git tracks it, no cluster, no network, no spec repo (R-TEST-06).

THE RULE. Every home-zone hostname the repository routes over HTTP must be linked from
the apex dashboard's configuration; every home-zone link the dashboard offers must
resolve to something the repository actually serves (an HTTPRoute at L7 or a
hostname-publishing LoadBalancer Service at L4); and the two R-C2-02 surfaces must be
linked, reconciled and backed by real alert rules, with operational rights arriving
per-user through OIDC groups rather than through a shared cluster-admin identity.

IT IS RED ON MAIN, on the forward direction:

    FAIL (8) every service the repo ROUTES over HTTP is surfaced on the C2 root.
             Unsurfaced: crafty.${BASE_DOMAIN} (platform/crafty)

platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} through the shared Gateway,
clusters/base/crafty.yaml reconciles it, and nothing on the C2 root links to it. R-C2-01
says the root MUST surface "the services running in the cluster and links to them", so
an unsurfaced route is a requirement gap, not a styling preference. Fixed in the next
commit, deliberately separate so the RED is in the history.

WHAT IS ASSERTED BEYOND THE PRIOR ATTEMPT (49c160d on feat/coverage-debt-paydown, whose
tree is ~107 commits behind and was NOT merged):

  (0) THE INPUT SURFACE ITSELF. The forward direction is only as good as the set of
      manifests it reads, so three assertions guard the guard: the scanned roots are
      present, every manifest in them PARSES (an unparseable file is a service the check
      cannot see), and EVERY Flux-reconciled path lies inside a scanned root — so a
      future tier reconciling, say, ./services fails loudly instead of letting the
      dashboard hide all of it. conformance/ is deliberately excluded from the scan: its
      fixtures and goldens declare synthetic HTTPRoutes (paas-demo, gw-declared-
      conformance) that are test data, not services the cluster serves. The prior
      attempt excluded them only by accident, through a filename prefilter.

  (5) postBuild.substituteFrom ON THE DASHBOARD'S TIER. Hostnames are ${BASE_DOMAIN}
      tokens now, not the literals the prior attempt compared. Without substitution the
      apex route binds the literal host `${BASE_DOMAIN}` (matching no listener) and
      every tile links to a host that does not exist — while Flux still reports the
      Kustomization Ready. It fails OPEN, which is this repo's most-repeated incident
      shape.

 (12) TOKENS, NEVER LITERALS. No href may name a literal `home.keithjackson.dev` host,
      so prod's front door cannot link dev's services — the 2026-09-04 class (two
      clusters authored to describe the same names, prod's external-dns then deleting
      dev's records), applied to the C2.

  (2) THE SECTIONNAME IS PROVEN LOAD-BEARING rather than asserted from prose: the
      wildcard listeners are shown to exist AND to be distinct from the apex listener,
      which is why `*.${BASE_DOMAIN}` cannot serve the bare apex.

 (10) THE REVERSE DIRECTION NOW ACCEPTS L4. The UniFi tile points at :11443 on a
      hostname published by apps/vms/unifi/50-service-l4.yaml (a KubeVirt VM behind a
      pinned Cilium LoadBalancer VIP). That is something this repo serves, so it
      resolves — no off-cluster declaration is needed for it any more. The declaration
      hatch is kept and constrained (specific home-zone hostnames, may not shadow a
      hostname the repo does serve) but is EMPTY today, which is the honest state.

WHOLE-REPO SCOPE, as the declaration says. The invariant quantifies over every tracked
manifest that can declare an HTTPRoute; the change that BREAKS it is a route added
somewhere else entirely (`apps/wave-1/newthing.yaml` attributes R-GITOPS-02 or nothing,
never R-C2-01), and ci/touched-requirements.sh has no path-map arm for platform/homepage
or platform/headlamp at all. Requirement-scoped selection would filter this check out of
exactly the change that introduces the bug — the R-TEST-03 blind spot. No ci/l0.sh stage
is added: Stage 3b's --include-scope union selects it, and whole-repo-invariants-wired
fails the gate if anyone hand-adds one.

HONEST GAPS, in the check's own header: it does not fetch the apex (L2, covered for the
Gateway/TLS half under R-DNS-01 / R-TLS-01); (19) reads a DECLARED auth posture because
no Gateway ext-authz attachment exists anywhere in this repo, and says so in the
assertion text; it does not judge whether a tile is useful; and an L4-ONLY endpoint
(crafty-mc, a Minecraft game port) is reported as an ADVISORY rather than required,
because a raw TCP/UDP endpoint has no URL a browser can follow.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:28:21 -05:00
jakman85andClaude Opus 5 0fe12b8332 WIP: L0 test declaration for R-C2-01 / R-C2-02 (no check.sh yet)
INCOMPLETE — preserved by the PM after the authoring agent died to a server
error mid-task (its third attempt; the first two died to a session rate
limit). Do not merge.

R-C2-01 and R-C2-02 are the only 2 remaining coverage gaps on main (49/51).
This is the chainsaw Test declaration only: it quotes both requirements
verbatim from the spec, cites docs/12-command-center-ui.md for which surface
serves which half (Homepage at the root for R-C2-01, Headlamp for R-C2-02's
'controlled operational actions where safe', Grafana for dashboards/alerts),
and argues why the deciding half is L0 rather than L2 — reachability is
already covered by l2-behavioral/gateway-ingress-reachable, whereas whether
the root surfaces THE SERVICES THE CLUSTER RUNS is a comparison of two sets
that are both repository facts.

WHAT REMAINS: check.sh does not exist, so this test cannot run at all. Also
still open: whether 'a service the root hid' (from the prior art commit
49c160d on feat/coverage-debt-paydown) is still a live defect, whether
ci/touched-requirements.sh can attribute R-C2-01/02 at all (a digit in the
workstream segment reportedly broke the content scan), and the R-TLS-02
decision summary.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 16:28:21 -05:00
jakman85andClaude Opus 5 204f0ec28e WIP: durable qBittorrent WebUI credential
INCOMPLETE — preserved by the PM after the authoring agent hit the session
rate limit mid-task. Do not merge without finishing and gating it.

Problem: qBittorrent 5.1.2 has no WebUI\Username or WebUI\Password_PBKDF2 in
its config, so it mints a NEW temporary admin password on every process start
and announces it only on stdout. Two consequences, both real: the Sonarr and
Radarr download clients configured earlier will silently stop working at the
next pod restart, and a human operator has no stable login either.

Adds a SOPS-encrypted qbittorrent-bootstrap Secret (verified encrypted: both
values are ENC[AES256_GCM]) plus the catalog/rendered wiring to consume it.

WHAT REMAINS:
  - The agent's last note: the postStart hook works, but its output never
    reaches kubectl logs, so it was switching to the marker-file logging
    pattern gitea's hook already uses. That change is NOT here.
  - The Secret exists on prod as kubectl-client-side-apply, i.e. live-only;
    merging adopts it into git.
  - Sonarr's and Radarr's stored download-client passwords were to be updated
    to the durable value — NOT done, so the chain may break on restart until
    it is.
  - The required proof is unfinished: restart the pod, then show the WebUI
    accepts the credential AND both arrs still report isValid: true.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 15:57:42 -05:00
13 changed files with 1426 additions and 6 deletions

No files matched your search

+202
View File
@@ -3042,3 +3042,205 @@ shared etcd/Longhorn device plus the I/O-pressure alert that missed the prod-1 o
Branch pruning was deliberately NOT done: four agent branches read as "merged, no unique commits"
only because their agents had not committed yet, and deleting those would have destroyed live work.
## 2026-09-13 — qBittorrent minted a NEW admin password on every start; both arrs were already broken (branch `fix/qbittorrent-durable-webui-credential`)
**THE DEFECT.** `/config/qBittorrent/qBittorrent.conf` on prod carried neither `WebUI\Username` nor
`WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and generates a
fresh random `admin` password on EVERY process start, announcing it only on stdout. Two consequences,
and **both had already fired by the time this was finished** — this was not a hypothesis:
- Sonarr's and Radarr's download-client credentials live in their own databases. Measured at 21:0xZ,
before any change: `POST /api/v3/downloadclient/testall` returned
`isValid: false, validationFailures: [{propertyName: "Username", errorMessage: "Authentication Failure"}]`
on **both** apps. Completed Download Handling was broken for both, silently — the arrs report a
failed client, not a failed import.
- No human had a stable login either; every restart meant reading the pod log.
**THE MECHANISM.** A SOPS/age Secret `qbittorrent-bootstrap` (`webui-username`, `webui-password`, 32
alphanumeric chars) is the single source of truth, consumed as env by the app container. A
`lifecycle.postStart` hook derives qBittorrent's own stored form — PBKDF2-HMAC-SHA512, 100000
iterations, 16-byte random salt, 64-byte digest, stored as
`WebUI\Password_PBKDF2="@ByteArray(<b64 salt>:<b64 digest>)"` — asserts both keys into the conf on
**every** container start, and then **logs in to the WebUI to prove it works**. Authentication is
never weakened: `WebUI\AuthSubnetWhitelist` was rejected outright (it leaves the human login broken
and makes auth depend on network policy staying correct for ever).
**HOW A REBUILD FROM GIT REPRODUCES IT.** All three paths converge on the same credential:
`pod restart` → the conf already holds it, the hook verifies and writes nothing (steady state does
not even rotate the salt); `config-PVC restored from an older snapshot` → a restore requires the pod
to be recreated, so the hook runs and re-asserts over whatever the snapshot held; `rebuild from
git onto an EMPTY PVC` → the image copies `/defaults/qBittorrent.conf`, the hook writes the
credential into it before the WebUI is serving. The only input is the SOPS file in git.
**WHY postStart AND NOT THE REPO'S CronJob RECONCILER IDIOM** (`platform/keycloak/client-secret-reconciler.yaml`,
the CNPG grant reconciler in `apps/wave-0/nextcloud-db.yaml`). The idiom's *principle* — desired state
in git, RE-ASSERTED unconditionally, never set once — is exactly what is applied; only the trigger
differs, because the failure being closed IS a container start. qbittorrent-nox reads the credential
once, at process start, from a file on the PVC, so a 15-minute cron leaves a 0–15 minute window after
every restart in which the app serves a random password and both arrs are broken. A CronJob would also
have to reach the conf from OUTSIDE the pod (mounting an RWO Longhorn PVC the app already holds, or
`kubectl exec` with RBAC it would need granting) and then edit a file under a running process whose
own shutdown save can overwrite the edit. The hook needs no ServiceAccount, RBAC, NetworkPolicy or
API token: strictly less privilege. An **initContainer** was the genuinely tempting third option — it
would run strictly before the app and its stdout would reach `kubectl logs` — but it cannot perform
the ground-truth login (the WebUI is not up yet), and the ServiceDefinition schema 1.5.0 has
`lifecycle` but no initContainers field, so it would cost a schema bump plus a renderer change across
every service's golden to buy something strictly weaker.
**TWO REAL BUGS FOUND IN THE INHERITED WIP, both from the postBuild-substitution trap.** The hook
computed its port as a doubled-dollar `WEBUI_PORT` default-operator expansion. That is *correct on the
cluster* (envsubst collapses the doubled dollar) and the renderer's `assert_substitution_safe` passes
it — but run any other way, bash reads the doubled dollar as its own PID, so `PORT` became
`<pid>{WEBUI_PORT:-8080}`, every curl got an unparseable URL, and the hook logged **"the WebUI
rejected the durable credential" three times having never asked it anything**. That misdiagnosis is
preserved in the service's own marker log at `2026-09-13T05:35Z`. Fixed by making it a literal
`PORT=8080` — the script now contains no braced dollar expansion at all and behaves identically
however it is run. Second bug: one shared wait budget covered both "wait for the s6 service" and
"wait for the API", so the first consumed all 90s and attempts 2 and 3 fired instantly. They now have
separate counters, "the API never answered" is reported as a distinct fault from "the password was
rejected", and a rejection now restarts qbittorrent-nox so a retry can actually change its outcome.
**A SELF-INFLICTED IP BAN IS PART OF THE BLAST RADIUS, AND IT WILL FOOL AN OPERATOR.** Repointing the
arrs at the correct password FAILED at first: the arr's own save-time validation returned
`Authentication Failure`, and from inside the sonarr pod qBittorrent answered
`Your IP address has been banned after too many failed authentication attempts.` A day of arr retries
against the stale password had banned both arr pod IPs. The credential was right the whole time. The
ban is in-memory, so the qbittorrent **process restart cleared it**; nothing was relaxed
(`MaxAuthenticationFailCount`/`BanDuration` untouched — the fix is that the arrs stop generating
failures). The hook spends at most 3 logins per start for exactly this reason.
**THE ARRS' STORED PASSWORD CANNOT BE READ BACK, so idempotency is driven by `testall`.** Sonarr and
Radarr return `"value": "********"` (a literal 8-asterisk mask, `privacy: password`) for every
password field. Comparing against that makes every run look like a change — and it also means the
earlier reading of "the arrs store an 8-char password" was reading the mask, not a value. The
convergence step therefore asks `POST /api/v3/downloadclient/testall` first and writes only when it
says `isValid: false`; a second run reports `ALREADY CONVERGED` and writes nothing.
**PROOF ON PROD.** `testall` flipped `isValid: false → true` on both apps. Then the pod was deleted
and recreated (`...-v8rj4` → `...-gpjcz`, startTime `21:22:18Z`): the conf still held both keys, the
WebUI answered `Ok.` to a login with the credential decrypted from git, and **both** arrs still
reported `isValid: true` — so qBittorrent's shutdown rewrite of its own conf preserves the credential
rather than clobbering it. The post-envsubst manifest was audited as `flux build kustomization`
produces it: no doubled dollar, no braced expansion, `PORT=8080`, `bash -n` clean. The hook's write
path was exercised in the real container against a scratch copy of the conf under `/tmp`, using the
helper extracted verbatim from that post-envsubst output: exit 10 on a conf with the keys removed
(wrote them), exit 0 on a second run (converged, byte-identical file — no salt rotation), and the
digest it produced verifies under an independent PBKDF2 implementation with the git password and
fails with a wrong one.
**WHAT IS *NOT* PROVEN, stated plainly.** The hook has **not** yet run as a real `postStart` on prod.
The Deployment is Flux-owned from `main`, and both ways of getting the branch's hook onto the live
object — suspending `apps-wave-0` and patching the Deployment — were refused by this session's
permission classifier. The durability shown above is therefore the *credential's* durability across
two real pod restarts, plus the hook's write/verify logic proven in-container; the kubelet actually
invoking it is verified only on merge. **First thing to check after merging:** that
`/config/qbittorrent-webui-cred.log` gains a `VERIFIED: the WebUI accepts the durable credential`
line on the next pod start.
**STILL OUTSIDE GIT (unchanged by this, and it is app state by nature).** Sonarr's and Radarr's
download-client rows — including this password — live in their own databases on their config PVCs.
Rotating `webui-password` in the SOPS file therefore means re-running the convergence step against
both arrs; git alone will not do it. Same caveat the Remote Path Mappings entry already records.
**The live-only Secret is now adopted.** `qbittorrent-bootstrap` existed on prod only as
`kubectl-client-side-apply`. Its SOPS file was in the WIP commit but **`apps/wave-0/kustomization.yaml`
did not list it**, so Flux would never have applied it and the adoption the last handoff expected
would silently not have happened. It is listed now (verified: the decrypted SOPS values are
byte-identical to the live Secret's, compared by SHA-256, so adoption cannot change the password).
Note also that the live Secret carries the plaintext in a
`kubectl.kubernetes.io/last-applied-configuration` annotation — a side effect of how it was created by
hand; Flux's server-side apply on merge does not write that annotation.
**GATE: PASS** (`make -C ci gate`, exit 0, from a real clone — gating from inside a linked worktree
cannot work here because the container mounts only the worktree while its `.git` points outside).
spec-immutability, lint, l0 (13 cases, 0 failures — including `no-plaintext-secrets`), render-golden
(all 10 onboarded goldens reproduce from the canonical render) and l1 all green. The touched
requirement is **R-GITOPS-02**. Its L2 stage was then run deliberately against prod rather than left
skipped, and passed — **but that green is worth nothing and should not be cited as coverage**: the
only test covering R-GITOPS-02, `conformance/tests/l2-behavioral/onboarding-completeness`, is a
scaffold containing 8 `TODO` echoes and asserts none of the behaviours it names (compute up, PVC
bound, default-deny, egress class, DNS/TLS, metrics, update policy). This is a concrete instance of
the fail-open the `feat/coverage-debt-paydown` and `fix/touched-requirements-fail-closed` branches are
about: a stage that reports green while verifying nothing.
## R-TLS-02 — the "revoked" clause still needs the OWNER's decision (recovered 2026-09-13)
**This is a decision only the repo owner/architect can make. It has now been escalated twice and
lost twice, which is why it is written into this log rather than into another branch's commit
message.** The analysis was first drafted on `feat/coverage-debt-paydown` (`b37defe`, 2026-09-07),
a branch whose tree is ~107 commits behind main and which must never be merged; the drafted
amendment lived only in that branch's copy of this file, so it disappeared from view the moment the
branch went stale. Recovered here, re-checked against today's main, and restated unchanged in
substance.
**R-TLS-02, verbatim (spec `docs/00-vision-and-scope.md`):** *"Certificates MUST be automatically
**issued, renewed, and revoked** without manual steps."*
Three verbs. Two hold; the third does not, and cannot with the stack ADR-0006 selects:
- **Issued — satisfied.** Both ClusterIssuers are declarative (ACME DNS-01, Cloudflare token
referenced by name from SOPS), the wildcard Certificate plus its apex SAN make onboarding
certificate-free, and the active issuer is chosen per environment by `${ACME_CLUSTER_ISSUER}`.
- **Renewed — satisfied, and unhobbled.** cert-manager renews at ~2/3 of lifetime with no operator
action; nothing in the repo sets a `duration`/`renewBefore` that would defeat it, and
`CertificateExpiringSoon` / `CertificateExpiryImminent` / `CertificateNotReady` fire if renewal
silently stops.
- **Revoked — NOT implemented, and not implementable here.** cert-manager has no automatic ACME
revocation: no revoke-on-delete, no `Certificate` field that requests revocation, no controller
that calls ACME `revokeCert`. Deleting a `Certificate` stops renewal; it does not revoke, and the
issued certificate stays valid at the CA until it expires. Neither ADR-0006 nor `docs/08`
specifies a mechanism — `docs/08`'s own "Renewal & revocation" section describes replacement
issuance, which is rotation.
**`enableCertificateOwnerRef` is NOT the remedy, and is still not set.** That cert-manager
controller flag makes a `Certificate` own its `Secret`, so deleting the Certificate
garbage-collects the key material. That is *cleanup*, not revocation. It does remove one real
manual step (today a deleted Certificate leaves an orphaned TLS Secret and private key to be
deleted by hand), so it is defensible on its own merits — but it is a GLOBAL controller flag, and
any future rename or prune that deletes and recreates a `Certificate` would take the Gateway's
`wildcard-home-tls` Secret with it, dropping TLS on every service until reissuance, against Let's
Encrypt rate limits, on two clusters. It deserves its own risk assessment, not a side effect of a
coverage or test task.
**WHAT HAS CHANGED SINCE THE FIRST ESCALATION.** R-TLS-02 is no longer a COVERAGE gap — main
carries `l0-static/tls-lifecycle-automated` and `l2-behavioral/cert-lifecycle-automated`, and the
coverage report counts it covered. The honesty was kept where it matters: the L0 check's own verdict
line ends *"R-TLS-02 'without manual steps' clause only — issued/renewed/revoked are RUNTIME verbs
(see l2-behavioral/cert-lifecycle-automated; 'revoked' is NOT proven at any layer)"*. But the
decision is now also LOAD-BEARING on the gate in a second way: the L2 test deletes a Certificate and
requires its Secret to disappear, and on failure prints *"set enableCertificateOwnerRef: true in
platform/cert-manager values (owner decision — WS-TEST does not weaken the assertion instead)"*. So
that test is written to FAIL on any live cluster run until the owner decides. Nobody should weaken
it to make a run green; that is the decision below, arriving as a red test instead of as a question.
**THE TWO OPTIONS, in the drafted preference order. The spec repo is READ-ONLY to every agent
(R-TEST-06), so neither has been applied — this is a request for a human decision.**
1. **PREFERRED — amend R-TLS-02 to the property that is achievable and actually protective**, and
move revocation to an explicitly manual, documented break-glass procedure:
> **R-TLS-02** — Certificates MUST be automatically **issued and renewed** without manual steps,
> and decommissioning a service MUST NOT leave its key material behind. Certificate
> **revocation** is an exceptional, operator-initiated action (suspected key compromise); it is
> NOT automated, and MUST have a documented procedure. Rationale: the ACME / Let's Encrypt model
> relies on SHORT certificate lifetimes plus automated rotation rather than on revocation — Let's
> Encrypt retired OCSP in 2025 and cert-manager implements no automatic revocation — so an
> "auto-revoked" requirement cannot be met by the stack ADR-0006 selects, and could only be met
> by bespoke revocation automation the threat model does not justify.
Under this wording R-TLS-02 becomes fully testable: issuance and renewal are already asserted
statically, and "no key material left behind" is exactly what `enableCertificateOwnerRef`
implements — which makes that flag a *consequence of the requirement*, decided on its own risk
assessment, rather than a fudge to turn a gate green.
2. **Keep R-TLS-02 as written and accept it as permanently partially satisfied**, with a reviewed
`DOCUMENTED_GAPS` entry in `conformance/run.sh` naming the revocation clause, the reason (no ACME
revocation exists in cert-manager) and the removal condition (cert-manager ships revocation, or
the requirement is amended). Cheaper, and records a requirement the system does not meet.
**Until the owner picks one, the state is: R-TLS-02 counted as covered, its third clause
unimplemented and labelled as such in the test's own output, and the L2 withdrawal assertion
standing as a deliberate red.** Writing a test tagged R-TLS-02 that silently verifies only issuance
and renewal would report the requirement as met while a third of its normative text is
unimplemented — which is the one thing this harness exists to prevent. The choice is the
architect's, not an agent's.
+1
View File
@@ -17,6 +17,7 @@ resources:
- pihole-oauth2-proxy-credentials.sops.yaml
- pihole.yaml
- plex.yaml
- qbittorrent-bootstrap-secret.sops.yaml
- qbittorrent-oauth2-proxy-credentials.sops.yaml
- qbittorrent.yaml
- semaphore-db-app.sops.yaml
@@ -0,0 +1,60 @@
# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux.
#
# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13):
# qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR
# `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW
# random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
# password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's
# saved download-client credentials stop working at the next qbittorrent pod restart, silently
# breaking Completed Download Handling; (b) a human operator has no stable login at all and must
# read the pod log after every restart.
#
# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env
# (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as
# gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores
# and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then
# verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore
# from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME
# credential, because the hook re-asserts it on every start rather than once.
#
# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is
# the name its own temporary-password path uses, so keeping it avoids inventing a second identity.
# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client
# entries store in their own databases (app state on their config PVCs, NOT in git — see
# DECISIONS-LOG), so rotating it here means re-pointing both arrs.
# Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON
# payloads, and a shell/URL-special character in any of those is a bug waiting to happen.
#
# NEVER put a value in a comment: SOPS does NOT encrypt comments.
# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml
apiVersion: v1
kind: Secret
metadata:
name: qbittorrent-bootstrap
namespace: qbittorrent
labels:
app.kubernetes.io/name: qbittorrent
type: Opaque
stringData:
webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str]
webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
hc_vault: []
age:
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2
TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3
UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv
VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv
FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw==
-----END AGE ENCRYPTED FILE-----
lastmodified: "2026-09-13T05:23:23Z"
mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str]
pgp: []
encrypted_regex: ^(data|stringData)$
version: 3.9.4
+19
View File
@@ -130,12 +130,31 @@ spec:
value: '1000'
- name: PGID
value: '3000'
- name: QBT_WEBUI_USERNAME
valueFrom:
secretKeyRef:
name: qbittorrent-bootstrap
key: webui-username
- name: QBT_WEBUI_PASSWORD
valueFrom:
secretKeyRef:
name: qbittorrent-bootstrap
key: webui-password
volumeMounts:
- name: config
mountPath: /config
- name: downloads
mountPath: /downloads
subPath: downloads
lifecycle:
postStart:
exec:
command:
- /bin/bash
- -c
- "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
, \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\
\ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
+289
View File
@@ -44,6 +44,295 @@ env:
value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay
# readable to the EXISTING non-k3s consumers of the same dataset, and land
# group-owned the same way the arrs' own writes do
# --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
# Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
# NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
# NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
# password is provided for this session: ..."). Two real consequences, not hypotheticals:
# * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
# tested valid against the then-current temporary password; both would have started failing at
# the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
# report a failed client, not a failed import).
# * The human operator had no stable login: every restart meant reading the pod log.
# Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
# same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
- name: QBT_WEBUI_USERNAME
valueFrom:
secretKeyRef:
name: qbittorrent-bootstrap
key: webui-username
- name: QBT_WEBUI_PASSWORD
valueFrom:
secretKeyRef:
name: qbittorrent-bootstrap
key: webui-password
lifecycle:
# LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
#
# WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
# directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
# apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
# RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
# the container start rather than a clock, because the failure this closes IS a container start:
# * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
# PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
# restore or rebuild in which the app serves a random password and both arrs are broken. The
# hook has no such window: it runs before/with the very start that would otherwise regenerate.
# * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
# PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
# then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
# In-container, before the service starts, none of that exists.
# * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
# privilege than the CronJob alternative (R-SEC/least-privilege).
# The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
# keeps in its own state), and the schema documents this hook for exactly that use.
#
# AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
# BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
# would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
# * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
# credential — because the WebUI is not running yet. That assertion is the most valuable line
# in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
# writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
# * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
# mean a schema version bump plus a renderer change — a blast radius across every service's
# golden — to buy something strictly weaker than what the hook already does.
# The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
# the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
#
# DURABILITY, the whole point — all three paths end with the SAME credential:
# pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
# config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
# hook runs and re-asserts over whatever the snapshot contained.
# rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
# credential into it before the service is serving. Nothing outside git.
#
# WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
# for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
# depend on network policy staying correct forever. The WebUI keeps demanding a password; that
# password is simply now known, fixed, and reproducible from git.
postStart:
exec:
command:
- /bin/bash
- -c
- |
# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
#
# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
# CLUSTER — but only there:
# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;
# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
# however it is run.
set -u
CONF=/config/qBittorrent/qBittorrent.conf
SVC=/run/service/svc-qbittorrent
# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
# ServiceDefinition — which is the single place the renderer takes the containerPort, the
# Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
# env var for a lookup to read anyway.
# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
# so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
# matched. Every curl below then got an unparseable URL, returned nothing, and the hook
# logged "the WebUI rejected the durable
# credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
# service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
PORT=8080
# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
# that only echoes is effectively silent (gitea's hits the same wall and writes a marker
# file for it). Every outcome therefore also lands, timestamped, on the persistent config
# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
# credential from git was asserted and accepted on this start.
log() {
echo "[qbt-webui-cred] $*"
printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
>> /config/qbittorrent-webui-cred.log 2>/dev/null || true
}
# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
# racing to create one it would then not overwrite.
n=0
while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
waited=0
attempt=0
while [ "$attempt" -lt 3 ]; do
attempt=$((attempt + 1))
# (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
# 10 = rewritten, anything else = could not write.
python3 - "$CONF" <<'PY'
import base64, hashlib, hmac, os, stat, sys
conf = sys.argv[1]
user = os.environ.get("QBT_WEBUI_USERNAME", "")
pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
if not user or not pw:
sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
raise SystemExit(1)
# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
# iterations, 16-byte random salt, 64-byte digest, stored as
# WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
# Get any one of them wrong and the app simply rejects every login.
ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
def matches(raw):
"""True iff the stored digest is one OUR password produces (salt is read from it)."""
v = raw.strip().strip('"')
if v.startswith("@ByteArray(") and v.endswith(")"):
v = v[len("@ByteArray("):-1]
try:
salt_b64, digest_b64 = v.split(":", 1)
salt = base64.b64decode(salt_b64, validate=True)
digest = base64.b64decode(digest_b64, validate=True)
except Exception:
return False
if not salt or not digest:
return False
return hmac.compare_digest(
hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
try:
lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
except FileNotFoundError:
lines = []
def value_of(key):
for i, line in enumerate(lines):
if line.startswith(key + "="):
return i, line.split("=", 1)[1]
return -1, None
ui, uval = value_of(UKEY)
pi, pval = value_of(PKEY)
if uval == user and pval is not None and matches(pval):
raise SystemExit(0) # converged - leave the file (and its salt) alone
salt = os.urandom(SALT_LEN)
digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
if SECTION in out:
at = out.index(SECTION) + 1
else:
if out and out[-1].strip():
out.append("")
out.append(SECTION)
at = len(out)
out[at:at] = desired
try:
os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
tmp = conf + ".webui-cred.new"
with open(tmp, "w", encoding="utf-8") as fh:
fh.write("\n".join(out) + "\n")
fh.flush()
os.fsync(fh.fileno())
# This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
# cost the app the ability to save its own settings, so carry the ownership over.
try:
st = os.stat(conf)
os.chown(tmp, st.st_uid, st.st_gid)
os.chmod(tmp, stat.S_IMODE(st.st_mode))
except FileNotFoundError:
os.chown(tmp, int(os.environ.get("PUID", "1000")),
int(os.environ.get("PGID", "1000")))
os.chmod(tmp, 0o664)
os.replace(tmp, conf) # atomic: the app never reads a half-written conf
except OSError as exc:
sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
raise SystemExit(1)
raise SystemExit(10)
PY
rc=$?
if [ "$rc" = 0 ]; then
log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
elif [ "$rc" = 10 ]; then
log "asserted WebUI Username + Password_PBKDF2 into $CONF"
# postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
# have read the old, credential-less conf and be serving a random temporary password.
# Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
# Nothing to do when the service has not started yet - it will read the new file.
# Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
# 2026-09-13 failure spent the entire verification budget before the first login and
# then fired attempts 2 and 3 with no wait left at all.
sv=0
while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
sv=$((sv + 1)); sleep 1
done
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
log "qbittorrent-nox was already running - restarting it to load the credential"
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
sleep 3
fi
else
log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
exit 0
fi
# (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
# (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
# at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
#
# "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
# conflating them is what made this hook lie once already (see the PORT comment). An
# unreachable WebUI is reported as unreachable, and the hook stops rather than spending
# its remaining login attempts to produce two more identical wrong diagnoses.
api_ok=0
while [ "$waited" -lt 120 ]; do
code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
waited=$((waited + 1)); sleep 1
done
if [ "$api_ok" != 1 ]; then
log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
log "never submitted. Do not rotate the credential on the strength of this line." >&2
exit 0
fi
body=$(curl -sS -m 10 \
--data-urlencode "username=$QBT_WEBUI_USERNAME" \
--data-urlencode "password=$QBT_WEBUI_PASSWORD" \
"http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
if [ "$body" = "Ok." ]; then
log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
exit 0
fi
log "WebUI rejected the durable credential (login said '$body')" >&2
# A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
# the running qbittorrent-nox is serving something it read from an EARLIER conf, so
# rewriting the same bytes on the next attempt would change nothing. The remedy is to
# make the process RE-READ the file, so restart the supervised service and let the next
# attempt re-verify. Without this the retry loop could never change its own outcome.
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
sleep 3
fi
done
log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
# app, and the arrs' own download-client health check surfaces a broken credential.
exit 0
podSecurityContext:
# For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
# on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
+33
View File
@@ -136,6 +136,39 @@ printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01"
# ---------------------------------------------------------------------------------------
# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
#
# The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match `R-C2-01` or
# `R-C2-02` — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. Those are
# the only two registry IDs whose workstream segment carries a digit, and the scan was blind
# to BOTH, in the fail-OPEN direction: a change self-declaring a C2 requirement contributed
# it to nobody, so the C2 conformance tests were filtered out of the merge gate. Found while
# wiring conformance/tests/l0-static/c2-root-surface, which closes those two coverage gaps.
#
# `notes/` is outside path_requirements() on purpose, so the IDs below can only have come
# from the CONTENT SCAN. A neighbouring all-letter ID is asserted too, so a regex that
# somehow matched only the digit form would still fail here.
git -C "$TMP" checkout -q -b feature5 main
cat > "$TMP/notes/c2.yaml" <<'YAML'
kind: Test
metadata:
labels:
conformance.home.k3s/R-C2-01: "true"
conformance.home.k3s/R-C2-02: "true"
conformance.home.k3s/R-OBS-04: "true"
YAML
git -C "$TMP" add -A && git -C "$TMP" commit -qm "declare R-C2-01 / R-C2-02"
OUT5="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
printf 'touched set (digit in the workstream segment):\n%s\n---\n' "$OUT5"
assert_present "R-C2-01" "$OUT5" "an R-ID with a DIGIT in the workstream segment IS attributed (R-C2-01)"
assert_present "R-C2-02" "$OUT5" "the second such ID is attributed too (R-C2-02)"
assert_present "R-OBS-04" "$OUT5" "an all-letter workstream segment still works (control)"
# The pattern must not become so loose that a non-ID token is read as a requirement: a bare
# numeric workstream is not a registry shape, and attributing one would WIDEN the gate on
# noise (over-broad selection, never a false green, but noisy and misleading).
assert_absent "R-1-01" "$OUT5" "a purely numeric workstream segment is not invented from thin air"
echo
if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
+16 -1
View File
@@ -345,6 +345,21 @@ main() {
done <<< "$files"
# Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
#
# THE ID PATTERN IS `R-[A-Z0-9]+-[0-9]+`, NOT `R-[A-Z]+-[0-9]+`. The WORKSTREAM segment may
# contain a DIGIT: `R-C2-01` and `R-C2-02` (command & control UI) are the only two such IDs
# in conformance/requirements/registry.yaml today, and the stricter pattern silently failed
# to match BOTH of them — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is.
# Every other consumer in the repo already allows the digit: test_requirements_for_path()
# twenty lines above uses `R-[A-Z0-9]+-[0-9]+`, as do conformance/lib/coverage.py,
# conformance/lib/status.py and conformance/lib/run_l1_offline.py. This grep was the lone
# dissenter, and it dissented by UNDER-attributing — the fail-OPEN direction. A change that
# retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02 from a
# path the map keys elsewhere, contributed those IDs to NOBODY, so the C2 tests were
# filtered out of the merge gate. Same defect class as the map's `*) : ;;` arm: a miss that
# reads as "nothing touched". path_requirements() has no arm for platform/homepage/* or
# platform/headlamp/* either, so before this fix the ONLY route to R-C2-01/02 was rule 3
# (editing the C2 test itself). Regression-tested in ci/tests/touched-requirements.test.sh.
# `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
# `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so
# what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
@@ -358,7 +373,7 @@ main() {
| grep -vE '^(\+\+\+|---)' \
| sed -E 's/^[+-]//' \
| grep -vE '^[[:space:]]*#' \
| grep -hoE 'R-[A-Z]+-[0-9]+' \
| grep -hoE 'R-[A-Z0-9]+-[0-9]+' \
| tr '\n' ' ' || true)"
# Normalize: split, unique, sort. Emit to stdout.
+6 -2
View File
@@ -34,8 +34,12 @@ spec:
kind: GitRepository
name: flux-system
path: ./platform/homepage
# Homepage references no ${VARS} today; the substitution is retained for parity with the other
# tiers / future overlay knobs.
# LOAD-BEARING, not parity decoration (the comment here used to say the opposite). This component
# references ${BASE_DOMAIN} in TWO places: httproute.yaml's apex hostname, and EVERY tile href in
# configmap.yaml. Without this substitution the token is applied LITERALLY — the apex route binds
# the host `${BASE_DOMAIN}`, which matches no Gateway listener, and every link on the operator's
# front door points at a host that does not exist — while Flux still reports this Kustomization
# Ready. It fails OPEN. Asserted by conformance/tests/l0-static/c2-root-surface (5).
postBuild:
substituteFrom:
- kind: ConfigMap
@@ -0,0 +1,116 @@
# =============================================================================
# L0 STATIC — the C2 root surfaces every service the cluster runs, plus platform
# health and controlled actions (R-C2-01, R-C2-02) [EXECUTABLE]
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# THE REQUIREMENTS, VERBATIM (spec docs/00-vision-and-scope.md):
# R-C2-01 "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services
# running in the cluster and links to them."
# R-C2-02 "The C2 UI MUST surface platform health (cluster, storage, GitOps, alerts) and
# provide controlled operational actions where safe."
# Detail: spec docs/12-command-center-ui.md — Homepage at the root is "the primary R-C2-01
# surface"; Headlamp is "the R-C2-02 'controlled operational actions where safe' surface";
# Grafana is the dashboards/alerts surface; "never a raw cluster-admin kubeconfig in a browser".
#
# WHY L0, AND NOT L2
# ------------------
# The REACHABILITY half of R-C2-01 is already exercised against a live cluster:
# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls
# verifies the certificate it serves. What no live probe can decide is the half the
# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING
# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts:
# what git declares the cluster serves (every in-repo HTTPRoute hostname, every L4 Service
# that publishes a hostname, every catalog ServiceDefinition) against what the dashboard's
# ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides half the
# estate and cannot tell the difference.
#
# It is also the assertion that ROTS, in the direction nothing else notices: onboard a
# service, forget the dashboard, and every cluster test stays green while the operator's
# front door hides a running service. That is not a styling preference — it is exactly the
# R-C2-01 failure mode, and this check found a live instance of it when it was written (see
# the commit message: platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root
# linked to it).
#
# GIVEN the repository as git tracks it,
# WHEN the apex ingress, the dashboard's declared link set, every in-repo HTTPRoute, every
# hostname-publishing L4 Service, the versioned catalog and the platform-health
# surfaces are read and compared,
# THEN — R-C2-01 —
# the bare apex is claimed by exactly one route, that route binds the ONE Gateway
# listener whose hostname matches a bare apex, the apex is a SAN on the certificate
# that listener terminates with, the dashboard is RECONCILED (not merely authored),
# FORWARD every HTTP service the repo routes is linked from the root, FORWARD every
# catalog `kind: container` service with a route is linked, and REVERSE every
# home-zone link the root offers resolves to something this repo actually serves —
# an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4);
# — R-C2-02 —
# both health surfaces docs/12 names are linked AND reconciled, the health DATA exists
# as code (PrometheusRule groups for cluster/nodes, storage, GitOps and cert expiry,
# reconciled by the platform-config tier), and actions are CONTROLLED: the console
# binds NO ClusterRole to its own ServiceAccount, authenticates per-user by native
# OIDC from a named SOPS Secret, and cluster rights arrive through OIDC GROUPS at more
# than one privilege level.
#
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files + PyYAML —
# the same dependency catalog/render.py already requires). conformance/run.sh --layer l0 and
# ci/l0.sh run it directly; this Chainsaw Test wraps the SAME script.
#
# WHAT IT DOES NOT PROVE — honest gaps, restated in check.sh's own header:
# - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2;
# the Gateway/TLS half is covered under R-DNS-01 / R-TLS-01.
# - That the C2 is access-CONTROLLED in fact. The auth assertions read a DECLARED posture,
# because the Gateway ext-authz attachment that would enforce forward-auth is not built
# anywhere in this repo (platform/homepage/httproute.yaml says so in its own header, and
# every forward-auth app shares the gap). Declared, not enforced — and labelled as such
# in the assertion text.
# - That an operational action in the console succeeds, or that it is audited. The
# assertions prove there is no shared cluster-admin identity to act with and that rights
# arrive per-user; the audit trail is a runtime property.
# - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up
# (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: c2-root-surface
labels:
conformance.home.k3s/R-C2-01: "true"
conformance.home.k3s/R-C2-02: "true"
annotations:
conformance.home.k3s/requirements: "R-C2-01, R-C2-02"
conformance.home.k3s/layer: "L0"
# WHOLE-REPO SCOPE, and it is the whole point of the forward assertion.
#
# The invariant is "every service the repo routes is linked from the C2 root". Its input
# surface is therefore EVERY tracked manifest that can declare an HTTPRoute — apps/wave-*,
# platform/*, runbooks/ — not just the two components that own the C2. ci/touched-
# requirements.sh keys `platform/homepage/*` and `platform/headlamp/*` to nothing at all
# today (there is no path-map arm for either), and even with one, the change that BREAKS
# this invariant is the change that adds a route somewhere else: `apps/wave-1/newthing.yaml`
# attributes R-GITOPS-02 / R-NET-* / nothing, never R-C2-01. Requirement-scoped selection
# would filter this check out of exactly the change that introduces the bug — the R-TEST-03
# blind spot recorded in conformance/lib/common.sh.
#
# Declaring the scope is the whole fix: conformance/run.sh --include-scope whole-repo
# (ci/l0.sh Stage 3b) unions this test into EVERY gate invocation. Do NOT hand-add a stage
# to ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
# A SKIP of a scope-declaring check is rc 3 (UNWIRED), never green.
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "false"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
The bare apex is bound to the landing dashboard over trusted TLS and is reconciled; the
dashboard links every HTTP service this repo routes and every link it offers resolves to
something the repo serves; and the C2 surfaces platform health backed by real alert rules
while granting operational rights per-user through OIDC groups rather than through a
shared cluster-admin identity.
spec:
steps:
- name: compare-the-c2-root-against-what-the-repo-actually-serves
description: >-
Read the apex ingress chain, the dashboard's declared links, every in-repo HTTPRoute and
hostname-publishing Service, the catalog, the platform-health alert rules and the
console's RBAC posture, and assert the two sets agree in both directions.
try:
- script:
content: bash ./check.sh
+656
View File
@@ -0,0 +1,656 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — c2-root-surface
#
# Serves (spec docs/00-vision-and-scope.md; detail in docs/12-command-center-ui.md):
#
# R-C2-01 "The root `home.keithjackson.dev` MUST serve a UI that surfaces the
# services running in the cluster and links to them."
# R-C2-02 "The C2 UI MUST surface platform health (cluster, storage, GitOps,
# alerts) and provide controlled operational actions where safe."
#
# WHY THIS IS AN L0 STATIC CHECK AND NOT AN L2 PROBE
# --------------------------------------------------
# The REACHABILITY half of R-C2-01 is already exercised against a live cluster:
# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls
# verifies the certificate it serves. What no live probe can decide is the half the
# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING
# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts:
# what git declares the cluster serves (every in-repo HTTPRoute hostname, every
# hostname-publishing L4 Service, every catalog ServiceDefinition) against what the
# dashboard's ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides
# half the estate and cannot tell the difference.
#
# It is also the assertion that ROTS, in the direction nothing else notices: onboard a
# service, forget the dashboard, and every cluster test stays green while the operator's
# front door hides a running service. That is not a styling preference — it is exactly the
# R-C2-01 failure mode, and it was a LIVE defect on main when this check was written
# (platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root linked to it).
#
# THE RULE THIS IMPLEMENTS, in one sentence: every home-zone hostname the repository routes
# over HTTP must be linked from the apex dashboard's configuration, every home-zone link the
# apex dashboard offers must resolve to something the repository actually serves, and the two
# R-C2-02 surfaces (health data + a per-user-authorized console) must be linked, reconciled
# and backed by real alert rules.
#
# GIVEN the repository as git tracks it,
# WHEN the apex ingress chain, the dashboard's declared link set, every in-repo HTTPRoute,
# every hostname-publishing L4 Service, the versioned catalog, the platform-health
# alert rules and the console's RBAC posture are read and compared,
# THEN — R-C2-01 —
# (0) the INPUT SURFACE is intact: every Flux-reconciled path lies inside the roots
# this check scans, and every manifest in them parses. A root this check does not
# scan, or a file it cannot parse, is a service it cannot see — which would
# silently weaken every assertion below;
# (1) EXACTLY ONE HTTPRoute claims the bare apex (an ambiguous root is not a root),
# and it belongs to the landing-dashboard component;
# (2) that route pins by sectionName the ONE Gateway listener whose hostname is the
# bare apex, and that listener is HTTPS — the wildcard `*.${BASE_DOMAIN}` listener
# does NOT intersect the bare apex, so without the sectionName the root binds
# nothing and serves nothing (asserted against the wildcard listener's existence,
# so the claim is proven rather than asserted from prose);
# (3) the apex is a SAN on the Certificate whose Secret that listener terminates with;
# (4) the dashboard is RECONCILED, not merely authored: one Flux Kustomization points
# at its path and is listed in the base entrypoint set;
# (5) that Kustomization carries postBuild.substituteFrom, because the dashboard's
# hrefs and its route's hostname are written as `${BASE_DOMAIN}` tokens — without
# substitution the apex route binds a literal `${BASE_DOMAIN}` and EVERY tile on
# the root links to a nonexistent host, while Flux still reports Ready;
# (6) the component's own kustomization ships config + workload + Service + route;
# (7) the dashboard is a service CATALOG: grouped entries carrying hrefs;
# (8) FORWARD — every home-zone hostname an in-repo HTTPRoute serves, other than the
# apex itself, is LINKED from the root. This is the rot-catcher;
# (9) FORWARD — every catalog `kind: container` ServiceDefinition whose declared
# ingress host is routed is likewise linked, tying the C2 to the VERSIONED
# onboarding contract and not only to whatever is rendered today;
# (10) REVERSE — every home-zone link the root offers resolves to something this repo
# actually serves: an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service
# (L4). A dead tile is a link the root offers to nothing;
# (11) the off-cluster escape hatch is not a blanket one: every name declared in the
# ConfigMap's `off-cluster-links` annotation is a specific home-zone hostname and
# none of them shadows a hostname the repo DOES serve;
# (12) no href names a LITERAL base-domain host. Every hostname is the `${BASE_DOMAIN}`
# token, so prod's root cannot link dev's services — the 2026-09-04 incident class
# (two clusters authored to describe the same names), applied to the front door;
# — R-C2-02 —
# (13) both health surfaces docs/12 names are linked: the Kubernetes console
# (cluster/storage/GitOps state) and the dashboards/alerts surface;
# (14) both are RECONCILED by Flux, so the links lead to something running;
# (15) the health DATA those surfaces render exists as code — PrometheusRule groups for
# each axis R-C2-02 enumerates (cluster/nodes, storage, GitOps reconciliation)
# plus certificate expiry — and those rule files are reconciled;
# (16) actions are CONTROLLED, not cluster-admin-in-a-browser (docs/12: "never a raw
# cluster-admin kubeconfig in a browser"): the console chart is explicitly told NOT
# to bind its ServiceAccount to a ClusterRole, so there is no shared privileged
# identity to act as;
# (17) actions are instead PER-USER: the console is configured for native OIDC with
# credentials from a named (SOPS) Secret, not inline;
# (18) cluster rights arrive through OIDC GROUPS at MORE THAN ONE privilege level (an
# admin group and a lesser one) — least-privilege, not one tier for everybody;
# (19) the C2 root DECLARES the authenticated-group posture docs/12 requires, on its own
# route (declared, NOT enforced — see the gap note below).
#
# WHAT THIS DOES *NOT* PROVE — honest gaps, the same ones the Test declaration names:
# - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2; the
# Gateway/TLS half is covered under R-DNS-01 / R-TLS-01.
# - That the C2 is access-CONTROLLED in fact. (19) reads a DECLARED posture, because the
# Gateway ext-authz attachment that would ENFORCE forward-auth is not built anywhere in
# this repo (platform/homepage/httproute.yaml says so in its own header, and every
# forward-auth app shares the gap). Declared, not enforced, and labelled as such in the
# assertion text itself.
# - That an operational action in the console succeeds, or that it is audited. (16)-(18)
# prove there is no shared cluster-admin identity to act with and that rights arrive
# per-user; the audit trail is a runtime property.
# - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up
# (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for.
# - That an L4-ONLY endpoint is surfaced. A raw TCP/UDP endpoint (a Minecraft game port) has
# no URL a browser can follow, so the FORWARD direction quantifies over HTTP routes;
# L4-published hostnames are listed as ADVISORIES and are honoured in the REVERSE
# direction (that is how the UniFi tile, an L4 LoadBalancer on :11443, resolves).
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and the
# requirement text above is quoted, not parsed.
#
# STATUS: EXECUTABLE. Pure offline: `git ls-files` + PyYAML (the dependency ten neighbouring
# l0-static checks already take). No cluster, no network, no spec repo.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
PYTHON="${PYTHON:-python3}"
# Honest SKIP (never green, and rc 3 is UNWIRED for a scope-declaring test) if the landing
# dashboard component is absent entirely — there is then no C2 root to audit, which is a
# report about the tree, not a pass.
if [ ! -f "${ROOT}/platform/homepage/configmap.yaml" ]; then
echo "check SKIPPED: platform/homepage/configmap.yaml absent — no C2 landing dashboard to audit" >&2
exit 3
fi
cd "${ROOT}"
"${PYTHON}" - "${ROOT}" <<'PY'
"""Static audit: the C2 root surfaces every service the repository serves, plus health.
Every parse contract is documented at its parser. PyYAML rather than regex because the
subjects are genuinely structured: multi-document apps/ bundles, and a Homepage config that
is nested YAML *inside* a ConfigMap string value.
"""
from __future__ import annotations
import re
import subprocess
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
FAILURES: list[str] = []
ADVISORIES: list[str] = []
CHECKS = 0
def check(cond, msg):
global CHECKS
CHECKS += 1
if cond:
print(f" ok {msg}")
else:
print(f" FAIL {msg}", file=sys.stderr)
FAILURES.append(msg)
def read(p: Path) -> str:
try:
return p.read_text()
except (OSError, UnicodeDecodeError):
return ""
# ============================================================ the hostname vocabulary
# BASE_DOMAIN is a cluster-vars knob (R-ENV-02): every public hostname the cluster serves is
# `<svc>.${BASE_DOMAIN}`, and dev/prod differ ONLY in that value. So a hostname in this repo
# is canonically the TOKEN form, and the literal value below exists only to be FORBIDDEN in
# (12): a literal is how prod came to be authored to serve dev's hostnames (2026-09-04,
# clusters/base/cluster-vars.yaml's own header).
TOKEN = "${BASE_DOMAIN}"
_vars = yaml.safe_load(read(ROOT / "clusters" / "base" / "cluster-vars.yaml")) or {}
LITERAL_BASE = ((_vars.get("data") or {}).get("BASE_DOMAIN") or "").strip()
def strip_port(host: str) -> str:
return host.split(":", 1)[0]
def zone_label(host: str):
"""The sub-label of a home-zone hostname, "" for the bare apex, or None if not in zone.
Accepts only the TOKEN form; a literal-base hostname returns None here and is caught
separately by (12), so a literal can never silently satisfy a set comparison.
"""
h = strip_port(host)
if h == TOKEN:
return ""
if h.endswith("." + TOKEN):
return h[: -len("." + TOKEN)]
return None
def is_literal_base(host: str) -> bool:
h = strip_port(host)
return bool(LITERAL_BASE) and (h == LITERAL_BASE or h.endswith("." + LITERAL_BASE))
# ============================================================ (0) the input surface
# DELIVERY ROOTS — the subtrees Flux reconciles, plus catalog/ (the versioned source the
# apps/ manifests are rendered FROM). conformance/ is deliberately EXCLUDED: its fixtures and
# goldens declare synthetic HTTPRoutes (`paas-demo.…`, `gw-declared-conformance.…`) that are
# test data, not services the cluster serves, and counting them would make (8) assert that
# the operator's dashboard links this harness's fixtures.
DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
"runbooks", "clusters", "bootstrap", "catalog")
def tracked_yaml() -> list[Path]:
out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
cwd=ROOT, capture_output=True, text=True, check=True)
return [ROOT / p for p in out.stdout.split("\0") if p]
ALL_TRACKED = tracked_yaml()
DELIVERY = [p for p in ALL_TRACKED
if p.relative_to(ROOT).parts[0] in DELIVERY_ROOTS]
parse_failed: list[str] = []
DOCS: dict[Path, list[dict]] = {}
for p in DELIVERY:
try:
DOCS[p] = [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
except yaml.YAMLError:
DOCS[p] = []
parse_failed.append(str(p.relative_to(ROOT)))
def docs_all():
for p, ds in DOCS.items():
for d in ds:
yield p, d
check(len(DELIVERY) >= 100,
f"(0) the delivery tree is present and scanned ({len(DELIVERY)} tracked manifest(s) "
f"under {'/, '.join(DELIVERY_ROOTS)}/)")
check(not parse_failed,
f"(0) every manifest in the scanned roots PARSES — an unparseable one is a service "
f"this check cannot see, so it would weaken every set comparison below "
f"{parse_failed if parse_failed else ''}")
# Every Flux-reconciled path must lie inside a root this check scans. This is the guard on
# the guard: if a future tier reconciles, say, ./services, the forward direction would stop
# seeing its routes and would go quietly green on a dashboard that hides all of them.
flux_ks = [(p, d) for p, d in docs_all()
if d.get("kind") == "Kustomization"
and str(d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io")]
# A RECONCILING Kustomization is one carrying a sourceRef. The pathless documents under
# bootstrap/flux/patches/ are strategic-merge PATCH FRAGMENTS (they add a decryption stanza to
# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
# a path of them would fail on a file that is correct.
reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
unscanned = sorted({
f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
for p, d in reconcilers
if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
not in DELIVERY_ROOTS
})
check(len(reconcilers) >= 10 and not unscanned,
f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
f"reconciled tree this check does not read is a service it cannot require a link for "
f"{unscanned if unscanned else ''}")
# ============================================================ what the repo SERVES
# L7: every hostname an in-repo HTTPRoute claims, and the component that owns it.
routes: dict[str, list[tuple[Path, dict]]] = {}
apex_inheriting: list[Path] = []
for p, d in docs_all():
if d.get("kind") != "HTTPRoute":
continue
hosts = (d.get("spec", {}) or {}).get("hostnames") or []
if not hosts:
# An HTTPRoute with NO hostnames inherits the hostname of every listener it attaches
# to (runbooks/serving/httproute.yaml binds `https-apex` only, so it serves a PATH on
# the apex). It introduces no new hostname, so it owes no tile.
apex_inheriting.append(p)
continue
for h in hosts:
routes.setdefault(str(h), []).append((p, d))
routed_labels = {} # sub-label -> owning component dir
for h, owners in routes.items():
lbl = zone_label(h)
if lbl: # "" is the apex; None is out of zone
routed_labels[lbl] = owners[0][0].parent
# L4: a Service that publishes a home-zone hostname to DNS is also "served by this repo" —
# it is how UniFi (a KubeVirt VM behind a pinned Cilium LoadBalancer VIP on :11443) reaches
# the LAN. external-dns's annotation IS the declaration; there is no HTTPRoute to find.
DNS_ANN = "external-dns.alpha.kubernetes.io/hostname"
l4_labels: dict[str, Path] = {}
for p, d in docs_all():
if d.get("kind") != "Service":
continue
ann = (d.get("metadata", {}) or {}).get("annotations", {}) or {}
for h in re.split(r"[,\s]+", str(ann.get(DNS_ANN, ""))):
lbl = zone_label(h) if h else None
if lbl:
l4_labels[lbl] = p
check(len(routed_labels) >= 10,
f"(0) the repo's north-south HTTP surface is discovered "
f"({len(routed_labels)} routed home-zone hostname(s), "
f"{len(apex_inheriting)} apex-path route(s), {len(l4_labels)} L4-published hostname(s))")
# ============================================================ (1) the apex is unambiguous
apex_claims = routes.get(TOKEN, [])
check(len(apex_claims) == 1,
f"(1) exactly ONE HTTPRoute claims the bare apex {TOKEN} — an ambiguous root is not a "
f"root (found {len(apex_claims)}: {[str(p.relative_to(ROOT)) for p, _ in apex_claims]})")
apex_path, apex_route = (apex_claims[0] if apex_claims else (None, {}))
DASH = apex_path.parent if apex_path is not None else None
check(DASH is not None and DASH.name == "homepage",
f"(1) the apex route belongs to the landing-dashboard component "
f"(found {DASH.name if DASH else None!r})")
# ============================================================ (2) it binds the apex listener
parents = (apex_route.get("spec", {}) or {}).get("parentRefs") or []
section = next((pr.get("sectionName") for pr in parents if pr.get("sectionName")), None)
gw_name = next((pr.get("name") for pr in parents), None)
gw_ns = next((pr.get("namespace") for pr in parents), None)
check(section is not None,
"(2) the apex route pins a Gateway listener by sectionName — the wildcard listener's "
"hostname cannot match a bare apex, so an unpinned apex route binds NOTHING")
gw = next((d for _, d in docs_all()
if d.get("kind") == "Gateway"
and (d.get("metadata", {}) or {}).get("name") == gw_name
and (gw_ns is None or (d.get("metadata", {}) or {}).get("namespace") == gw_ns)), None)
check(gw is not None, f"(2) the parent Gateway '{gw_name}' is declared in the repo")
listeners = ((gw or {}).get("spec", {}) or {}).get("listeners") or []
apex_listener = next((l for l in listeners if l.get("name") == section), None)
check(apex_listener is not None,
f"(2) the Gateway declares the '{section}' listener the apex route binds")
check(apex_listener is not None and apex_listener.get("hostname") == TOKEN,
f"(2) that listener's hostname is EXACTLY the bare apex "
f"(found {(apex_listener or {}).get('hostname')!r})")
check(apex_listener is not None and apex_listener.get("protocol") == "HTTPS",
"(2) the apex listener terminates HTTPS (docs/12 acceptance gate: trusted TLS on the root)")
# Prove the sectionName is LOAD-BEARING rather than decorative: a wildcard listener exists,
# it is a DIFFERENT listener, and its hostname does not equal the apex.
wildcard = [l for l in listeners if str(l.get("hostname", "")) == "*." + TOKEN]
check(bool(wildcard) and all(l.get("name") != section for l in wildcard),
f"(2) the wildcard listener(s) {[l.get('name') for l in wildcard]} are DISTINCT from "
f"'{section}' — which is why the apex needs its own listener, and why dropping the "
f"sectionName would leave the root unserved")
# ============================================================ (3) trusted TLS on the root
cert_secret = next((ref.get("name")
for ref in ((apex_listener or {}).get("tls", {}) or {}).get("certificateRefs") or []),
None)
cert = next((d for _, d in docs_all()
if d.get("kind") == "Certificate"
and (d.get("spec", {}) or {}).get("secretName") == cert_secret), None)
check(cert is not None,
f"(3) a cert-manager Certificate produces the apex listener's Secret ({cert_secret!r})")
check(cert is not None and TOKEN in ((cert.get("spec", {}) or {}).get("dnsNames") or []),
"(3) the bare apex is a SAN on that Certificate — the wildcard alone does NOT cover it, "
"so without this SAN the root serves an untrusted certificate")
# ============================================================ (4)(5) it is RECONCILED
dash_rel = f"./{DASH.relative_to(ROOT)}" if DASH else None
dash_ks = [(p, d) for p, d in flux_ks
if (d.get("spec", {}) or {}).get("path", "").rstrip("/") == dash_rel]
check(len(dash_ks) == 1,
f"(4) exactly one Flux Kustomization reconciles the dashboard ({dash_rel}) — authored "
f"but unreconciled is the 'looks done, is absent' class this repo keeps hitting "
f"(found {len(dash_ks)})")
if dash_ks:
base_txt = read(ROOT / "clusters" / "base" / "kustomization.yaml")
entry = dash_ks[0][0].name
check(re.search(rf"^\s*-\s*{re.escape(entry)}\s*(#.*)?$", base_txt, re.M) is not None,
f"(4) that Kustomization is listed in the base entrypoint set ({entry})")
# (5) The dashboard's hrefs and its route's hostname are ${BASE_DOMAIN} TOKENS. Without
# postBuild.substituteFrom the token reaches the cluster LITERALLY: the apex route binds
# the host `${BASE_DOMAIN}` (matching no listener) and every tile links to a host that
# does not exist — and Flux still reports the Kustomization Ready. It fails OPEN.
subs = (((dash_ks[0][1].get("spec", {}) or {}).get("postBuild") or {}).get("substituteFrom") or [])
check(any((s.get("name") == "cluster-vars") for s in subs),
f"(5) {dash_ks[0][0].name} carries postBuild.substituteFrom cluster-vars. It MUST: "
f"this component's route hostname and EVERY tile href are ${{BASE_DOMAIN}} tokens, "
f"and an unsubstituted token is applied literally while Flux still reports Ready — "
f"a root bound to no listener and a dashboard of dead links, with a green tier")
# ============================================================ (6) it ships the UI
dash_kust = yaml.safe_load(read(DASH / "kustomization.yaml")) if DASH else None
dash_res = set((dash_kust or {}).get("resources") or [])
for want in ("configmap.yaml", "deployment.yaml", "service.yaml", "httproute.yaml"):
check(want in dash_res, f"(6) the dashboard kustomization ships {want}")
# ================================================== the dashboard's DECLARED link set
# Homepage's config is nested YAML inside ConfigMap string values (services.yaml,
# bookmarks.yaml). Both are read: a bookmark is a link the root offers just as a tile is.
cm = next((d for d in (DOCS.get(DASH / "configmap.yaml") or []) if d.get("kind") == "ConfigMap"),
{}) if DASH else {}
cm_data = cm.get("data") or {}
def embedded(key):
try:
return yaml.safe_load(cm_data.get(key) or "") or []
except yaml.YAMLError:
return None
svc_cfg = embedded("services.yaml")
bm_cfg = embedded("bookmarks.yaml")
check(svc_cfg is not None and bm_cfg is not None,
"(7) the dashboard's embedded services.yaml / bookmarks.yaml parse as YAML — the app "
"mounts these verbatim, so a malformed one is a dashboard that renders nothing")
def hrefs(node):
"""Every href anywhere in Homepage's nested group/entry structure."""
if isinstance(node, dict):
for k, v in node.items():
if k == "href" and isinstance(v, str):
yield v
else:
yield from hrefs(v)
elif isinstance(node, list):
for v in node:
yield from hrefs(v)
def host_of(url):
m = re.match(r"^https?://([^/]+)", url or "")
return m.group(1) if m else None
tile_urls = list(hrefs(svc_cfg or []))
all_urls = tile_urls + list(hrefs(bm_cfg or []))
all_hosts = [h for h in (host_of(u) for u in all_urls) if h]
linked = {lbl for lbl in (zone_label(h) for h in all_hosts) if lbl}
tile_labels = {lbl for lbl in (zone_label(h) for h in (host_of(u) for u in tile_urls) if h) if lbl}
groups = [g for g in (svc_cfg or []) if isinstance(g, dict)]
check(len(groups) >= 1 and len(tile_labels) >= 5,
f"(7) the root is a grouped service catalog with links "
f"({len(groups)} group(s), {len(tile_labels)} linked home-zone service(s))")
# ============================================ (8) FORWARD: nothing routed is hidden
unsurfaced = sorted(set(routed_labels) - linked)
check(not unsurfaced,
"(8) every service the repo ROUTES over HTTP is surfaced on the C2 root. Unsurfaced: "
+ ", ".join(f"{n}.{TOKEN} ({routed_labels[n].relative_to(ROOT)})" for n in unsurfaced)
if unsurfaced else
f"(8) every service the repo ROUTES over HTTP is surfaced on the C2 root "
f"({len(routed_labels)} hostname(s) compared)")
# L4-only endpoints are reported, never required: a raw TCP/UDP port has no browsable URL.
for lbl in sorted(set(l4_labels) - linked):
ADVISORIES.append(
f"{lbl}.{TOKEN} is published to DNS by an L4 Service "
f"({l4_labels[lbl].relative_to(ROOT)}) and is not linked from the root. Not a "
f"failure: a raw TCP/UDP endpoint has no URL a browser can follow. Verify the "
f"service it belongs to is reachable through a tile that IS linked")
# ================================= (9) FORWARD: the VERSIONED onboarding contract
cat_dir = ROOT / "catalog"
cat_seen, cat_missing, cat_unrouted = 0, [], []
for p in sorted(cat_dir.glob("*.yaml")):
sd = next((d for d in (DOCS.get(p) or [])), None)
if not isinstance(sd, dict) or sd.get("kind") != "container":
continue # kind: vm has no north-south HTTP surface (docs/09)
for r in sd.get("ingressRoutes") or []:
lbl = zone_label(str(r.get("host", "")))
if not lbl:
continue
if lbl not in routed_labels:
# Declared in the catalog but not rendered into a route: an apps/ render drift
# problem owned by R-GITOPS-02, reported here rather than silently skipped.
cat_unrouted.append(f"{sd.get('name')}:{lbl}")
continue
cat_seen += 1
if lbl not in linked:
cat_missing.append(f"{sd.get('name')}:{lbl}")
check(cat_seen >= 5,
f"(9) the versioned catalog contributes routed HTTP services ({cat_seen} found) — the "
f"C2 is tied to the onboarding contract, not only to today's rendered output")
check(not cat_missing,
f"(9) every catalog `kind: container` service with a route is linked from the C2 root "
f"{cat_missing if cat_missing else ''}")
for u in cat_unrouted:
ADVISORIES.append(f"catalog declares an ingress host with no rendered HTTPRoute: {u} "
f"(render drift — R-GITOPS-02 owns it, not R-C2-01)")
# ============================ (10)(11) REVERSE: no link points at nothing
ann = (cm.get("metadata", {}) or {}).get("annotations", {}) or {}
OFF_ANN = "homelab.keithjackson.dev/off-cluster-links"
declared_off_raw = [t for t in re.split(r"[,\s]+", str(ann.get(OFF_ANN, ""))) if t]
declared_off = {zone_label(t) for t in declared_off_raw}
served = set(routed_labels) | set(l4_labels)
dead = sorted(linked - served - declared_off)
check(not dead,
f"(10) every home-zone link the root offers resolves to something this repo serves — "
f"an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4). Dead: "
f"{[f'{d}.{TOKEN}' for d in dead] if dead else ''}")
# The escape hatch is a DECLARATION, not a hole: same doctrine as an explicit `NONE` in
# ci/touched-requirements.sh's path map. Empty today, and these two hold vacuously when it is
# — they exist so it cannot be widened into a blanket without failing.
check(all(zone_label(t) for t in declared_off_raw),
f"(11) every declared off-cluster link is a specific home-zone hostname in TOKEN form "
f"({declared_off_raw})")
shadowed = sorted(declared_off & served)
check(not shadowed,
f"(11) no off-cluster declaration shadows a hostname the repo DOES serve "
f"{shadowed if shadowed else ''}")
# ============================ (12) tokens, never literals
literal_hosts = sorted({h for h in all_hosts if is_literal_base(h)})
check(not literal_hosts,
f"(12) no link on the root names a LITERAL '{LITERAL_BASE}' host — every hostname is the "
f"${{BASE_DOMAIN}} token, so prod's front door cannot link dev's services (the "
f"2026-09-04 class: two clusters authored to describe the same names). Literals: "
f"{literal_hosts if literal_hosts else ''}")
# =====================================================================
# R-C2-02 — platform health, and controlled operational actions
# =====================================================================
# docs/12 names the surfaces the C2 composes: Headlamp is "the R-C2-02 'controlled
# operational actions where safe' surface"; Grafana is the dashboards/alerts surface. Each is
# identified by the component that OWNS ITS ROUTE, not by a magic string in the config.
CONSOLE, DASHBOARDS = "headlamp", "grafana"
check(CONSOLE in linked,
f"(13) the C2 root links the Kubernetes console ({CONSOLE}.{TOKEN}) — cluster, storage "
f"and GitOps state, and the R-C2-02 operational-actions surface (docs/12)")
check(DASHBOARDS in linked,
f"(13) the C2 root links the dashboards/alerts surface ({DASHBOARDS}.{TOKEN})")
AGG = ("\n".join(read(ROOT / t / "kustomization.yaml")
for t in ("platform", "platform-config", "platform-config-cdi")))
for label, name in (("console", CONSOLE), ("dashboards", DASHBOARDS)):
comp = routed_labels.get(name)
check(comp is not None, f"(14) the {label} surface has an in-repo HTTPRoute")
if comp is None:
continue
rel = f"./{comp.relative_to(ROOT)}"
reconciled = any((d.get("spec", {}) or {}).get("path", "").rstrip("/") == rel
for _, d in flux_ks)
if not reconciled:
# A component may reconcile as a MEMBER of an aggregate tier's kustomize tree
# (platform / platform-config) instead of owning a Flux Kustomization. Both count;
# being in NEITHER is "authored, never applied".
reconciled = (f"\n - {comp.name}" in AGG
or f"../{comp.relative_to(ROOT)}/" in AGG)
check(reconciled,
f"(14) the {label} surface ({comp.name}) is reconciled by Flux, not merely authored "
f"— a link to an unreconciled component is a link to nothing")
# (15) the health DATA exists as code, for every axis R-C2-02 enumerates.
rule_files = sorted({p for p, d in docs_all() if d.get("kind") == "PrometheusRule"})
rule_groups = {g.get("name", "")
for p, d in docs_all() if d.get("kind") == "PrometheusRule"
for g in ((d.get("spec", {}) or {}).get("groups") or [])}
for axis, needle in (("cluster/nodes", "nodes"),
("storage", "longhorn"),
("GitOps reconciliation", "flux"),
("certificate expiry", "certificates")):
check(any(needle in g for g in rule_groups),
f"(15) platform-health alert rules exist for {axis} (a '*{needle}*' rule group) — "
f"the C2 cannot surface health that is not measured")
pc_txt = read(ROOT / "platform-config" / "kustomization.yaml")
reconciled_rules = [p for p in rule_files if f"../{p.relative_to(ROOT)}" in pc_txt]
check(len(reconciled_rules) >= 2,
f"(15) those alert rules are RECONCILED — PrometheusRule is a CR of a CRD the platform "
f"tier installs, so the rules are applied by platform-config "
f"({len(reconciled_rules)}/{len(rule_files)} rule file(s) referenced there)")
# (16)(17) controlled actions: no shared cluster-admin identity; per-user OIDC instead.
console_dir = routed_labels.get(CONSOLE)
hr = next((d for p in (sorted(console_dir.glob("*.yaml")) if console_dir else [])
for d in (DOCS.get(p) or []) if d.get("kind") == "HelmRelease"), None)
check(hr is not None, "(16) the Kubernetes console is deployed by a HelmRelease")
hv = ((hr or {}).get("spec", {}) or {}).get("values", {}) or {}
check((hv.get("clusterRoleBinding") or {}).get("create") is False,
"(16) the console chart is told NOT to bind its ServiceAccount to a ClusterRole. The "
"chart's DEFAULT binds it to cluster-admin, which is exactly the 'raw cluster-admin "
"kubeconfig in a browser' docs/12 forbids — every visitor would act as one shared "
"omnipotent identity")
oidc = ((hv.get("config") or {}).get("oidc") or {})
check(bool(oidc),
"(17) the console is configured for native OIDC login, so an action carries the "
"operator's OWN identity (docs/12: Headlamp/Grafana use native OIDC)")
check(bool((oidc.get("externalSecret") or {}).get("name")),
"(17) its OIDC client credentials come from a NAMED (SOPS) Secret, not inline in the "
"HelmRelease values — R-SEC-01")
# (18) rights arrive through GROUPS, at more than one privilege level.
group_bindings = []
for p, d in docs_all():
if d.get("kind") not in ("ClusterRoleBinding", "RoleBinding"):
continue
for s in d.get("subjects") or []:
if s.get("kind") == "Group" and str(s.get("name", "")).startswith("oidc:"):
group_bindings.append(((d.get("metadata", {}) or {}).get("name"),
s.get("name"),
(d.get("roleRef") or {}).get("name")))
check(len(group_bindings) >= 2,
f"(18) cluster rights are bound to OIDC GROUPS, so access follows Keycloak group "
f"membership rather than a per-user grant somebody must remember to revoke "
f"({[(g, r) for _, g, r in group_bindings]})")
check(any(r == "cluster-admin" for _, _, r in group_bindings)
and any(r != "cluster-admin" for _, _, r in group_bindings),
"(18) MORE THAN ONE privilege level exists (an admin group AND a lesser one) — "
"least-privilege, not one tier for everybody (docs/12 access control, R-IDP-06)")
# (19) the root DECLARES the authenticated-group posture. DECLARED, NOT ENFORCED.
route_ann = (apex_route.get("metadata", {}) or {}).get("annotations", {}) or {}
check(route_ann.get("homelab.keithjackson.dev/auth-mode") in ("forward-auth", "oidc", "oauth2-proxy"),
f"(19) the C2 root's route DECLARES an authentication mode "
f"(found {route_ann.get('homelab.keithjackson.dev/auth-mode')!r}) "
f"[DECLARED, NOT ENFORCED — no Gateway ext-authz attachment exists in this repo]")
check(bool(route_ann.get("homelab.keithjackson.dev/auth-groups")),
"(19) it DECLARES the authorized group(s) — docs/12 requires an authenticated user IN "
"AN AUTHORIZED GROUP, not merely logged in "
"[DECLARED, NOT ENFORCED — see the header's gap note]")
# ======================================================================== report
print()
print(f"C2 surface inventory (apex {TOKEN}, literal base default '{LITERAL_BASE}'):")
print(f" routed over HTTP : {len(routed_labels)} -> {', '.join(sorted(routed_labels))}")
print(f" L4-published : {len(l4_labels)} -> {', '.join(sorted(l4_labels)) or '(none)'}")
print(f" linked from root : {len(linked)} -> {', '.join(sorted(linked))}")
print(f" apex-path routes : {len(apex_inheriting)} -> "
f"{', '.join(str(p.relative_to(ROOT)) for p in apex_inheriting) or '(none)'}")
print(f" catalog services compared: {cat_seen} off-cluster declarations: "
f"{sorted(declared_off_raw) or '(none)'}")
if ADVISORIES:
print("\nADVISORIES (not failures):")
for a in ADVISORIES:
print(f" - {a}")
print()
if not FAILURES:
print(f"check OK: c2-root-surface — {CHECKS} assertions (R-C2-01, R-C2-02)")
raise SystemExit(0)
print(f"CHECK FAIL (c2-root-surface): {len(FAILURES)} failed, "
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
for f in FAILURES:
print(f" FAIL {f}", file=sys.stderr)
raise SystemExit(1)
PY
@@ -85,7 +85,7 @@ done
# second would fail.
# Deliberately LOWERCASE: conf_normalize_reqs() upper-cases it, so the selection behaves
# identically — but the literal here does not match ci/touched-requirements.sh's content-scan
# pattern `R-[A-Z]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement
# pattern `R-[A-Z0-9]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement
# to the change. (Same reason the synthetic fixture below carries no requirement label at all.)
BOGUS_REQ="r-nosuchreq-99"
sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
+12
View File
@@ -81,6 +81,18 @@ data:
href: https://qbittorrent.${BASE_DOMAIN}
description: Download client (VPN egress)
icon: mdi-download-network
# Crafty Controller — a running, north-south-exposed web UI (platform/crafty/httproute.yaml,
# reconciled by clusters/base/crafty.yaml) that this dashboard did NOT surface until
# 2026-09-13. R-C2-01 says the root MUST surface "the services running in the cluster and
# links to them", so an unsurfaced route is a requirement gap, not a styling preference —
# conformance/tests/l0-static/c2-root-surface fails on one now. Grouped under Media because
# it manages game servers; regrouping it under Platform would be purely cosmetic. The game
# ports themselves (crafty-mc.${BASE_DOMAIN}, TCP 25565-25567 / UDP 19132-19134) are an L4
# LoadBalancer with no browsable URL, so this tile is how they are surfaced.
- Crafty:
href: https://crafty.${BASE_DOMAIN}
description: Minecraft server manager (Keycloak-gated)
icon: mdi-minecraft
- Network & Infra:
- Pi-hole:
+15 -2
View File
@@ -328,12 +328,25 @@ spec:
labelsFromPath:
cron: [spec, schedule]
phase: [status, phase]
# MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB
# with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike,
# roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert
# rules read kube_* series, so each kill made them intermittently inert, and an alert that
# is only sometimes able to fire is the quiet version of the inert-alert class this repo
# keeps getting bitten by (KubePodNotReady and KubeDeploymentReplicasMismatch were both
# stuck pending on it). The trigger was enabling customResourceState below for the Velero
# Schedule metrics — it adds a CRD informer plus the mandatory CRD list/watch, and that
# marginal cost crossed a limit already sitting ~4 MiB above the working set.
# 384Mi is ~3x the observed peak and costs nothing here: the nodes report 18.7-22.3 GiB
# available. Re-measure with
# max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])
# before trimming it back.
resources:
requests:
cpu: 20m
memory: 64Mi
memory: 192Mi
limits:
memory: 128Mi
memory: 384Mi
prometheus-node-exporter:
resources:
requests: