Compare commits
11
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
72f0ba4e1f | ||
|
|
088ff13d15 | ||
|
|
cc648ba02c | ||
|
|
8b36242643 | ||
|
|
e4bd76070e | ||
|
|
aff80d1d7a | ||
|
|
a76ae3bdfa | ||
|
|
02c58aae15 | ||
|
|
f57c411953 | ||
|
|
0fe12b8332 | ||
|
|
204f0ec28e |
No files matched your search
@@ -3042,3 +3042,205 @@ shared etcd/Longhorn device plus the I/O-pressure alert that missed the prod-1 o
|
||||
|
||||
Branch pruning was deliberately NOT done: four agent branches read as "merged, no unique commits"
|
||||
only because their agents had not committed yet, and deleting those would have destroyed live work.
|
||||
|
||||
## 2026-09-13 — qBittorrent minted a NEW admin password on every start; both arrs were already broken (branch `fix/qbittorrent-durable-webui-credential`)
|
||||
|
||||
**THE DEFECT.** `/config/qBittorrent/qBittorrent.conf` on prod carried neither `WebUI\Username` nor
|
||||
`WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and generates a
|
||||
fresh random `admin` password on EVERY process start, announcing it only on stdout. Two consequences,
|
||||
and **both had already fired by the time this was finished** — this was not a hypothesis:
|
||||
|
||||
- Sonarr's and Radarr's download-client credentials live in their own databases. Measured at 21:0xZ,
|
||||
before any change: `POST /api/v3/downloadclient/testall` returned
|
||||
`isValid: false, validationFailures: [{propertyName: "Username", errorMessage: "Authentication Failure"}]`
|
||||
on **both** apps. Completed Download Handling was broken for both, silently — the arrs report a
|
||||
failed client, not a failed import.
|
||||
- No human had a stable login either; every restart meant reading the pod log.
|
||||
|
||||
**THE MECHANISM.** A SOPS/age Secret `qbittorrent-bootstrap` (`webui-username`, `webui-password`, 32
|
||||
alphanumeric chars) is the single source of truth, consumed as env by the app container. A
|
||||
`lifecycle.postStart` hook derives qBittorrent's own stored form — PBKDF2-HMAC-SHA512, 100000
|
||||
iterations, 16-byte random salt, 64-byte digest, stored as
|
||||
`WebUI\Password_PBKDF2="@ByteArray(<b64 salt>:<b64 digest>)"` — asserts both keys into the conf on
|
||||
**every** container start, and then **logs in to the WebUI to prove it works**. Authentication is
|
||||
never weakened: `WebUI\AuthSubnetWhitelist` was rejected outright (it leaves the human login broken
|
||||
and makes auth depend on network policy staying correct for ever).
|
||||
|
||||
**HOW A REBUILD FROM GIT REPRODUCES IT.** All three paths converge on the same credential:
|
||||
`pod restart` → the conf already holds it, the hook verifies and writes nothing (steady state does
|
||||
not even rotate the salt); `config-PVC restored from an older snapshot` → a restore requires the pod
|
||||
to be recreated, so the hook runs and re-asserts over whatever the snapshot held; `rebuild from
|
||||
git onto an EMPTY PVC` → the image copies `/defaults/qBittorrent.conf`, the hook writes the
|
||||
credential into it before the WebUI is serving. The only input is the SOPS file in git.
|
||||
|
||||
**WHY postStart AND NOT THE REPO'S CronJob RECONCILER IDIOM** (`platform/keycloak/client-secret-reconciler.yaml`,
|
||||
the CNPG grant reconciler in `apps/wave-0/nextcloud-db.yaml`). The idiom's *principle* — desired state
|
||||
in git, RE-ASSERTED unconditionally, never set once — is exactly what is applied; only the trigger
|
||||
differs, because the failure being closed IS a container start. qbittorrent-nox reads the credential
|
||||
once, at process start, from a file on the PVC, so a 15-minute cron leaves a 0–15 minute window after
|
||||
every restart in which the app serves a random password and both arrs are broken. A CronJob would also
|
||||
have to reach the conf from OUTSIDE the pod (mounting an RWO Longhorn PVC the app already holds, or
|
||||
`kubectl exec` with RBAC it would need granting) and then edit a file under a running process whose
|
||||
own shutdown save can overwrite the edit. The hook needs no ServiceAccount, RBAC, NetworkPolicy or
|
||||
API token: strictly less privilege. An **initContainer** was the genuinely tempting third option — it
|
||||
would run strictly before the app and its stdout would reach `kubectl logs` — but it cannot perform
|
||||
the ground-truth login (the WebUI is not up yet), and the ServiceDefinition schema 1.5.0 has
|
||||
`lifecycle` but no initContainers field, so it would cost a schema bump plus a renderer change across
|
||||
every service's golden to buy something strictly weaker.
|
||||
|
||||
**TWO REAL BUGS FOUND IN THE INHERITED WIP, both from the postBuild-substitution trap.** The hook
|
||||
computed its port as a doubled-dollar `WEBUI_PORT` default-operator expansion. That is *correct on the
|
||||
cluster* (envsubst collapses the doubled dollar) and the renderer's `assert_substitution_safe` passes
|
||||
it — but run any other way, bash reads the doubled dollar as its own PID, so `PORT` became
|
||||
`<pid>{WEBUI_PORT:-8080}`, every curl got an unparseable URL, and the hook logged **"the WebUI
|
||||
rejected the durable credential" three times having never asked it anything**. That misdiagnosis is
|
||||
preserved in the service's own marker log at `2026-09-13T05:35Z`. Fixed by making it a literal
|
||||
`PORT=8080` — the script now contains no braced dollar expansion at all and behaves identically
|
||||
however it is run. Second bug: one shared wait budget covered both "wait for the s6 service" and
|
||||
"wait for the API", so the first consumed all 90s and attempts 2 and 3 fired instantly. They now have
|
||||
separate counters, "the API never answered" is reported as a distinct fault from "the password was
|
||||
rejected", and a rejection now restarts qbittorrent-nox so a retry can actually change its outcome.
|
||||
|
||||
**A SELF-INFLICTED IP BAN IS PART OF THE BLAST RADIUS, AND IT WILL FOOL AN OPERATOR.** Repointing the
|
||||
arrs at the correct password FAILED at first: the arr's own save-time validation returned
|
||||
`Authentication Failure`, and from inside the sonarr pod qBittorrent answered
|
||||
`Your IP address has been banned after too many failed authentication attempts.` A day of arr retries
|
||||
against the stale password had banned both arr pod IPs. The credential was right the whole time. The
|
||||
ban is in-memory, so the qbittorrent **process restart cleared it**; nothing was relaxed
|
||||
(`MaxAuthenticationFailCount`/`BanDuration` untouched — the fix is that the arrs stop generating
|
||||
failures). The hook spends at most 3 logins per start for exactly this reason.
|
||||
|
||||
**THE ARRS' STORED PASSWORD CANNOT BE READ BACK, so idempotency is driven by `testall`.** Sonarr and
|
||||
Radarr return `"value": "********"` (a literal 8-asterisk mask, `privacy: password`) for every
|
||||
password field. Comparing against that makes every run look like a change — and it also means the
|
||||
earlier reading of "the arrs store an 8-char password" was reading the mask, not a value. The
|
||||
convergence step therefore asks `POST /api/v3/downloadclient/testall` first and writes only when it
|
||||
says `isValid: false`; a second run reports `ALREADY CONVERGED` and writes nothing.
|
||||
|
||||
**PROOF ON PROD.** `testall` flipped `isValid: false → true` on both apps. Then the pod was deleted
|
||||
and recreated (`...-v8rj4` → `...-gpjcz`, startTime `21:22:18Z`): the conf still held both keys, the
|
||||
WebUI answered `Ok.` to a login with the credential decrypted from git, and **both** arrs still
|
||||
reported `isValid: true` — so qBittorrent's shutdown rewrite of its own conf preserves the credential
|
||||
rather than clobbering it. The post-envsubst manifest was audited as `flux build kustomization`
|
||||
produces it: no doubled dollar, no braced expansion, `PORT=8080`, `bash -n` clean. The hook's write
|
||||
path was exercised in the real container against a scratch copy of the conf under `/tmp`, using the
|
||||
helper extracted verbatim from that post-envsubst output: exit 10 on a conf with the keys removed
|
||||
(wrote them), exit 0 on a second run (converged, byte-identical file — no salt rotation), and the
|
||||
digest it produced verifies under an independent PBKDF2 implementation with the git password and
|
||||
fails with a wrong one.
|
||||
|
||||
**WHAT IS *NOT* PROVEN, stated plainly.** The hook has **not** yet run as a real `postStart` on prod.
|
||||
The Deployment is Flux-owned from `main`, and both ways of getting the branch's hook onto the live
|
||||
object — suspending `apps-wave-0` and patching the Deployment — were refused by this session's
|
||||
permission classifier. The durability shown above is therefore the *credential's* durability across
|
||||
two real pod restarts, plus the hook's write/verify logic proven in-container; the kubelet actually
|
||||
invoking it is verified only on merge. **First thing to check after merging:** that
|
||||
`/config/qbittorrent-webui-cred.log` gains a `VERIFIED: the WebUI accepts the durable credential`
|
||||
line on the next pod start.
|
||||
|
||||
**STILL OUTSIDE GIT (unchanged by this, and it is app state by nature).** Sonarr's and Radarr's
|
||||
download-client rows — including this password — live in their own databases on their config PVCs.
|
||||
Rotating `webui-password` in the SOPS file therefore means re-running the convergence step against
|
||||
both arrs; git alone will not do it. Same caveat the Remote Path Mappings entry already records.
|
||||
|
||||
**The live-only Secret is now adopted.** `qbittorrent-bootstrap` existed on prod only as
|
||||
`kubectl-client-side-apply`. Its SOPS file was in the WIP commit but **`apps/wave-0/kustomization.yaml`
|
||||
did not list it**, so Flux would never have applied it and the adoption the last handoff expected
|
||||
would silently not have happened. It is listed now (verified: the decrypted SOPS values are
|
||||
byte-identical to the live Secret's, compared by SHA-256, so adoption cannot change the password).
|
||||
Note also that the live Secret carries the plaintext in a
|
||||
`kubectl.kubernetes.io/last-applied-configuration` annotation — a side effect of how it was created by
|
||||
hand; Flux's server-side apply on merge does not write that annotation.
|
||||
|
||||
**GATE: PASS** (`make -C ci gate`, exit 0, from a real clone — gating from inside a linked worktree
|
||||
cannot work here because the container mounts only the worktree while its `.git` points outside).
|
||||
spec-immutability, lint, l0 (13 cases, 0 failures — including `no-plaintext-secrets`), render-golden
|
||||
(all 10 onboarded goldens reproduce from the canonical render) and l1 all green. The touched
|
||||
requirement is **R-GITOPS-02**. Its L2 stage was then run deliberately against prod rather than left
|
||||
skipped, and passed — **but that green is worth nothing and should not be cited as coverage**: the
|
||||
only test covering R-GITOPS-02, `conformance/tests/l2-behavioral/onboarding-completeness`, is a
|
||||
scaffold containing 8 `TODO` echoes and asserts none of the behaviours it names (compute up, PVC
|
||||
bound, default-deny, egress class, DNS/TLS, metrics, update policy). This is a concrete instance of
|
||||
the fail-open the `feat/coverage-debt-paydown` and `fix/touched-requirements-fail-closed` branches are
|
||||
about: a stage that reports green while verifying nothing.
|
||||
## R-TLS-02 — the "revoked" clause still needs the OWNER's decision (recovered 2026-09-13)
|
||||
|
||||
**This is a decision only the repo owner/architect can make. It has now been escalated twice and
|
||||
lost twice, which is why it is written into this log rather than into another branch's commit
|
||||
message.** The analysis was first drafted on `feat/coverage-debt-paydown` (`b37defe`, 2026-09-07),
|
||||
a branch whose tree is ~107 commits behind main and which must never be merged; the drafted
|
||||
amendment lived only in that branch's copy of this file, so it disappeared from view the moment the
|
||||
branch went stale. Recovered here, re-checked against today's main, and restated unchanged in
|
||||
substance.
|
||||
|
||||
**R-TLS-02, verbatim (spec `docs/00-vision-and-scope.md`):** *"Certificates MUST be automatically
|
||||
**issued, renewed, and revoked** without manual steps."*
|
||||
|
||||
Three verbs. Two hold; the third does not, and cannot with the stack ADR-0006 selects:
|
||||
|
||||
- **Issued — satisfied.** Both ClusterIssuers are declarative (ACME DNS-01, Cloudflare token
|
||||
referenced by name from SOPS), the wildcard Certificate plus its apex SAN make onboarding
|
||||
certificate-free, and the active issuer is chosen per environment by `${ACME_CLUSTER_ISSUER}`.
|
||||
- **Renewed — satisfied, and unhobbled.** cert-manager renews at ~2/3 of lifetime with no operator
|
||||
action; nothing in the repo sets a `duration`/`renewBefore` that would defeat it, and
|
||||
`CertificateExpiringSoon` / `CertificateExpiryImminent` / `CertificateNotReady` fire if renewal
|
||||
silently stops.
|
||||
- **Revoked — NOT implemented, and not implementable here.** cert-manager has no automatic ACME
|
||||
revocation: no revoke-on-delete, no `Certificate` field that requests revocation, no controller
|
||||
that calls ACME `revokeCert`. Deleting a `Certificate` stops renewal; it does not revoke, and the
|
||||
issued certificate stays valid at the CA until it expires. Neither ADR-0006 nor `docs/08`
|
||||
specifies a mechanism — `docs/08`'s own "Renewal & revocation" section describes replacement
|
||||
issuance, which is rotation.
|
||||
|
||||
**`enableCertificateOwnerRef` is NOT the remedy, and is still not set.** That cert-manager
|
||||
controller flag makes a `Certificate` own its `Secret`, so deleting the Certificate
|
||||
garbage-collects the key material. That is *cleanup*, not revocation. It does remove one real
|
||||
manual step (today a deleted Certificate leaves an orphaned TLS Secret and private key to be
|
||||
deleted by hand), so it is defensible on its own merits — but it is a GLOBAL controller flag, and
|
||||
any future rename or prune that deletes and recreates a `Certificate` would take the Gateway's
|
||||
`wildcard-home-tls` Secret with it, dropping TLS on every service until reissuance, against Let's
|
||||
Encrypt rate limits, on two clusters. It deserves its own risk assessment, not a side effect of a
|
||||
coverage or test task.
|
||||
|
||||
**WHAT HAS CHANGED SINCE THE FIRST ESCALATION.** R-TLS-02 is no longer a COVERAGE gap — main
|
||||
carries `l0-static/tls-lifecycle-automated` and `l2-behavioral/cert-lifecycle-automated`, and the
|
||||
coverage report counts it covered. The honesty was kept where it matters: the L0 check's own verdict
|
||||
line ends *"R-TLS-02 'without manual steps' clause only — issued/renewed/revoked are RUNTIME verbs
|
||||
(see l2-behavioral/cert-lifecycle-automated; 'revoked' is NOT proven at any layer)"*. But the
|
||||
decision is now also LOAD-BEARING on the gate in a second way: the L2 test deletes a Certificate and
|
||||
requires its Secret to disappear, and on failure prints *"set enableCertificateOwnerRef: true in
|
||||
platform/cert-manager values (owner decision — WS-TEST does not weaken the assertion instead)"*. So
|
||||
that test is written to FAIL on any live cluster run until the owner decides. Nobody should weaken
|
||||
it to make a run green; that is the decision below, arriving as a red test instead of as a question.
|
||||
|
||||
**THE TWO OPTIONS, in the drafted preference order. The spec repo is READ-ONLY to every agent
|
||||
(R-TEST-06), so neither has been applied — this is a request for a human decision.**
|
||||
|
||||
1. **PREFERRED — amend R-TLS-02 to the property that is achievable and actually protective**, and
|
||||
move revocation to an explicitly manual, documented break-glass procedure:
|
||||
|
||||
> **R-TLS-02** — Certificates MUST be automatically **issued and renewed** without manual steps,
|
||||
> and decommissioning a service MUST NOT leave its key material behind. Certificate
|
||||
> **revocation** is an exceptional, operator-initiated action (suspected key compromise); it is
|
||||
> NOT automated, and MUST have a documented procedure. Rationale: the ACME / Let's Encrypt model
|
||||
> relies on SHORT certificate lifetimes plus automated rotation rather than on revocation — Let's
|
||||
> Encrypt retired OCSP in 2025 and cert-manager implements no automatic revocation — so an
|
||||
> "auto-revoked" requirement cannot be met by the stack ADR-0006 selects, and could only be met
|
||||
> by bespoke revocation automation the threat model does not justify.
|
||||
|
||||
Under this wording R-TLS-02 becomes fully testable: issuance and renewal are already asserted
|
||||
statically, and "no key material left behind" is exactly what `enableCertificateOwnerRef`
|
||||
implements — which makes that flag a *consequence of the requirement*, decided on its own risk
|
||||
assessment, rather than a fudge to turn a gate green.
|
||||
|
||||
2. **Keep R-TLS-02 as written and accept it as permanently partially satisfied**, with a reviewed
|
||||
`DOCUMENTED_GAPS` entry in `conformance/run.sh` naming the revocation clause, the reason (no ACME
|
||||
revocation exists in cert-manager) and the removal condition (cert-manager ships revocation, or
|
||||
the requirement is amended). Cheaper, and records a requirement the system does not meet.
|
||||
|
||||
**Until the owner picks one, the state is: R-TLS-02 counted as covered, its third clause
|
||||
unimplemented and labelled as such in the test's own output, and the L2 withdrawal assertion
|
||||
standing as a deliberate red.** Writing a test tagged R-TLS-02 that silently verifies only issuance
|
||||
and renewal would report the requirement as met while a third of its normative text is
|
||||
unimplemented — which is the one thing this harness exists to prevent. The choice is the
|
||||
architect's, not an agent's.
|
||||
@@ -17,6 +17,7 @@ resources:
|
||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||
- pihole.yaml
|
||||
- plex.yaml
|
||||
- qbittorrent-bootstrap-secret.sops.yaml
|
||||
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
||||
- qbittorrent.yaml
|
||||
- semaphore-db-app.sops.yaml
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# qBittorrent DURABLE WebUI credential (WS-APPS). SOPS-encrypted (age), decrypted in-cluster by Flux.
|
||||
#
|
||||
# WHY THIS FILE EXISTS (defect, measured on prod 2026-09-13):
|
||||
# qBittorrent.conf on the config PVC carried NEITHER `WebUI\Username` NOR
|
||||
# `WebUI\Password_PBKDF2`. qBittorrent 5.x treats that as "no credential set" and generates a NEW
|
||||
# random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
|
||||
# password is provided for this session: ..."). Consequences, both real: (a) Sonarr's and Radarr's
|
||||
# saved download-client credentials stop working at the next qbittorrent pod restart, silently
|
||||
# breaking Completed Download Handling; (b) a human operator has no stable login at all and must
|
||||
# read the pod log after every restart.
|
||||
#
|
||||
# HOW IT IS APPLIED: catalog/qbittorrent.yaml's app container reads BOTH keys below as env
|
||||
# (QBT_WEBUI_USERNAME / QBT_WEBUI_PASSWORD) and its `lifecycle.postStart` hook — the same idiom as
|
||||
# gitea's break-glass admin bootstrap — DERIVES the PBKDF2-HMAC-SHA512 digest qBittorrent stores
|
||||
# and asserts both keys into /config/qBittorrent/qBittorrent.conf on EVERY container start, then
|
||||
# verifies the WebUI actually accepts them. Level-triggered: a pod restart, a config-PVC restore
|
||||
# from an older backup, and a rebuild-from-git onto an EMPTY PVC all end with this SAME
|
||||
# credential, because the hook re-asserts it on every start rather than once.
|
||||
#
|
||||
# webui-username: the WebUI account name. qBittorrent has exactly ONE local account, and `admin` is
|
||||
# the name its own temporary-password path uses, so keeping it avoids inventing a second identity.
|
||||
# webui-password: the durable plaintext. It is ALSO what Sonarr's and Radarr's download-client
|
||||
# entries store in their own databases (app state on their config PVCs, NOT in git — see
|
||||
# DECISIONS-LOG), so rotating it here means re-pointing both arrs.
|
||||
# Deliberately alphanumeric: it travels through a urlencoded WebAPI login body and two arr JSON
|
||||
# payloads, and a shell/URL-special character in any of those is a bug waiting to happen.
|
||||
#
|
||||
# NEVER put a value in a comment: SOPS does NOT encrypt comments.
|
||||
# Recover with: infra/devcontainer/dev.sh run sops -d apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: qbittorrent-bootstrap
|
||||
namespace: qbittorrent
|
||||
labels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
type: Opaque
|
||||
stringData:
|
||||
webui-username: ENC[AES256_GCM,data:P022nG0=,iv:MXcCXsD5hs7MAJ+OAYHkZ3pXep7XGPqeAExChoqna08=,tag:WE34mxtfQRgiwPRaDA12pQ==,type:str]
|
||||
webui-password: ENC[AES256_GCM,data:+Lx0Ny7SBZeF/CzQYm8umntw27ewyXS7unWObpl0fCg=,iv:JkAtublBMS7SJUutAFip1XjdmsKnGe+S2E9QDT4rCJk=,tag:MYptPVgoEUEpXe5kmg1xIg==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1aWpUY04wZHoyWVdPbGU2
|
||||
TC9vaU5oYmhWSVh3YWVTdzkvU3Axdlh6QVFZCkV4MldaZXNyam5CejQyZStyczJ3
|
||||
UStFd0hRQmRsNGZLVEVTRnVjWXp6SVkKLS0tICtkWlRqVGN5OWYvVnZUaHYvbEZv
|
||||
VWtHMVdDU3hmdXo1TndLb05zQzJ0S28KfqlDBTocO0ZAKtH6ad/00EeVRqZ7WMwv
|
||||
FP2evd5vyn95QQICWhY/LH6EKbwVVlcIlL7i9l8K3kK4Ebdk54mAyw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-09-13T05:23:23Z"
|
||||
mac: ENC[AES256_GCM,data:hQ7jIgFqpNcFY5H4AfMxuslC2R/2wGmGWtghemOmTUUipSdoGdbNjWGQhxzk40lFjw6Zj2b+buOpXpg70s4xr+4feusqa2TJ3+bxAuRWmuZpTQAU2OaO5v0SdDEs3gwKm2mA3JFo/e6POm/P/kf2GIECcyU1AciZQt6ws7UJmQc=,iv:6SF0ZAuzHQThD/FGbWhFfa4dXU8hyW8jWFiDmwgfFU8=,tag:31i8HM2s/mGeS4zvFxO53Q==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -130,12 +130,31 @@ spec:
|
||||
value: '1000'
|
||||
- name: PGID
|
||||
value: '3000'
|
||||
- name: QBT_WEBUI_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-bootstrap
|
||||
key: webui-username
|
||||
- name: QBT_WEBUI_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-bootstrap
|
||||
key: webui-password
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: downloads
|
||||
mountPath: /downloads
|
||||
subPath: downloads
|
||||
lifecycle:
|
||||
postStart:
|
||||
exec:
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the\n# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.\n#\n# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.\n# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over\n# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;\n# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE\n# CLUSTER \u2014 but only there:\n# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl\n# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during\n# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;\n# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`\n# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY\n# however it is run.\nset -u\nCONF=/config/qBittorrent/qBittorrent.conf\nSVC=/run/service/svc-qbittorrent\n# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this\n# ServiceDefinition \u2014 which is the single place the renderer takes the containerPort, the\n# Service port and the HTTPRoute backend port from \u2014 and the image exposes no WEBUI_PORT\n# env var for a lookup to read anyway.\n# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and\n# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,\n# so PORT became \"<pid>{WEBUI_PORT:-8080}\" \u2014 the brace never expanded, the digits never\n# matched. Every curl below then got an unparseable URL, returned nothing, and the hook\n# logged \"the WebUI rejected the durable\n# credential\" \u2014 having never actually asked it. That MISDIAGNOSIS is preserved in this\n# service's own marker log at 2026-09-13T05:35Z (three \"rejected\" lines, then GAVE UP,\n# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.\nPORT=8080\n# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` \u2014 so a hook\n# that only echoes is effectively silent (gitea's hits the same wall and writes a marker\n# file for it). Every outcome therefore also lands, timestamped, on the persistent config\n# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the\n# credential from git was asserted and accepted on this start.\nlog() {\n echo \"[qbt-webui-cred] $*\"\n printf '%s [qbt-webui-cred] %s\\n' \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\" \"$*\" \\\n >> /config/qbittorrent-webui-cred.log 2>/dev/null || true\n}\n\n# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC\n# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of\n# racing to create one it would then not overwrite.\nn=0\nwhile [ ! -s \"$CONF\" ] && [ \"$n\" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done\n\nwaited=0\nattempt=0\nwhile [ \"$attempt\" -lt 3 ]; do\n attempt=$((attempt + 1))\n\n # (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),\n # 10 = rewritten, anything else = could not write.\n python3 - \"$CONF\" <<'PY'\nimport base64, hashlib, hmac, os, stat, sys\n\nconf = sys.argv[1]\nuser = os.environ.get(\"QBT_WEBUI_USERNAME\", \"\")\npw = os.environ.get(\"QBT_WEBUI_PASSWORD\", \"\")\nif not user or not pw:\n sys.stderr.write(\"[qbt-webui-cred] credential env is empty - refusing to touch the conf\\n\")\n raise SystemExit(1)\n\n# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000\n# iterations, 16-byte random salt, 64-byte digest, stored as\n# WebUI\\Password_PBKDF2=\"@ByteArray(<base64 salt>:<base64 digest>)\"\n# Get any one of them wrong and the app simply rejects every login.\nITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16\nUKEY, PKEY, SECTION = \"WebUI\\\\Username\", \"WebUI\\\\Password_PBKDF2\"\
|
||||
, \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\
|
||||
\ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
|
||||
@@ -44,6 +44,295 @@ env:
|
||||
value: "3000" # the export's dominant group, matching sonarr/radarr: files this writes stay
|
||||
# readable to the EXISTING non-k3s consumers of the same dataset, and land
|
||||
# group-owned the same way the arrs' own writes do
|
||||
# --- THE DURABLE WebUI CREDENTIAL (consumed by the postStart hook below) -------------------
|
||||
# Measured on prod 2026-09-13: /config/qBittorrent/qBittorrent.conf held NEITHER `WebUI\Username`
|
||||
# NOR `WebUI\Password_PBKDF2`. qBittorrent 5.x reads that as "no credential configured" and mints a
|
||||
# NEW random `admin` password on EVERY process start, announcing it ONLY on stdout ("A temporary
|
||||
# password is provided for this session: ..."). Two real consequences, not hypotheticals:
|
||||
# * Sonarr and Radarr store a download-client username/password in their OWN databases. Both
|
||||
# tested valid against the then-current temporary password; both would have started failing at
|
||||
# the NEXT qbittorrent pod restart, silently breaking Completed Download Handling (the arrs
|
||||
# report a failed client, not a failed import).
|
||||
# * The human operator had no stable login: every restart meant reading the pod log.
|
||||
# Provided as apps/wave-0/qbittorrent-bootstrap-secret.sops.yaml (SOPS/age, decrypted by Flux) —
|
||||
# same shape and naming as gitea-bootstrap, so `render.py --emit-waves` places it in this wave.
|
||||
- name: QBT_WEBUI_USERNAME
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-bootstrap
|
||||
key: webui-username
|
||||
- name: QBT_WEBUI_PASSWORD
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: qbittorrent-bootstrap
|
||||
key: webui-password
|
||||
lifecycle:
|
||||
# LEVEL-TRIGGERED CREDENTIAL ASSERTION, run on EVERY container start.
|
||||
#
|
||||
# WHY A postStart HOOK AND NOT A CronJob RECONCILER: the repo's idiom for "state Flux cannot reach
|
||||
# directly" is the level-triggered CronJob (platform/keycloak/client-secret-reconciler.yaml,
|
||||
# apps/wave-0/nextcloud-db.yaml). That idiom's PRINCIPLE — desired state lives in git and is
|
||||
# RE-ASSERTED unconditionally, never set once — is exactly what is applied here, but the trigger is
|
||||
# the container start rather than a clock, because the failure this closes IS a container start:
|
||||
# * The credential is read by qbittorrent-nox ONCE, at process start, from a file on the config
|
||||
# PVC. A 15-minute cron would therefore leave a 0-15 minute window after every restart, PVC
|
||||
# restore or rebuild in which the app serves a random password and both arrs are broken. The
|
||||
# hook has no such window: it runs before/with the very start that would otherwise regenerate.
|
||||
# * A CronJob would also have to reach the conf from OUTSIDE the pod — mounting an RWO Longhorn
|
||||
# PVC that the app already holds, or `kubectl exec`-ing with RBAC it would need granting — and
|
||||
# then edit a file UNDER the running process, whose own save-on-exit can overwrite the edit.
|
||||
# In-container, before the service starts, none of that exists.
|
||||
# * It needs no ServiceAccount, no RBAC, no NetworkPolicy and no API token: strictly less
|
||||
# privilege than the CronJob alternative (R-SEC/least-privilege).
|
||||
# The gitea entry's postStart is the same pattern (idempotent bootstrap of a credential the app
|
||||
# keeps in its own state), and the schema documents this hook for exactly that use.
|
||||
#
|
||||
# AND WHY NOT AN initContainer, which is the genuinely tempting third option: it would run strictly
|
||||
# BEFORE qbittorrent-nox, removing the race the `s6-svc -r` below exists to cover, and its stdout
|
||||
# would reach `kubectl logs -c <init>` instead of the kubelet. Two reasons it loses here:
|
||||
# * It CANNOT do step (3) — the ground-truth login that proves the WebUI actually accepts the
|
||||
# credential — because the WebUI is not running yet. That assertion is the most valuable line
|
||||
# in this whole mechanism: it is what turns "we wrote a file" into "the credential works", and
|
||||
# writing the right PBKDF2 bytes is precisely the part that is easy to get subtly wrong.
|
||||
# * The ServiceDefinition schema (1.5.0) has `lifecycle` but NO initContainers field, so it would
|
||||
# mean a schema version bump plus a renderer change — a blast radius across every service's
|
||||
# golden — to buy something strictly weaker than what the hook already does.
|
||||
# The honest summary: postStart trades a short, self-healing start-up race (detected and closed by
|
||||
# the s6 restart + re-verify loop) for the ability to VERIFY. That is the right trade.
|
||||
#
|
||||
# DURABILITY, the whole point — all three paths end with the SAME credential:
|
||||
# pod restart -> conf on the PVC already holds it; the hook verifies and writes NOTHING.
|
||||
# config-PVC restore from an older backup -> restore requires the pod to be recreated, so the
|
||||
# hook runs and re-asserts over whatever the snapshot contained.
|
||||
# rebuild from git -> empty PVC; the image copies /defaults/qBittorrent.conf, the hook writes the
|
||||
# credential into it before the service is serving. Nothing outside git.
|
||||
#
|
||||
# WHAT IT DOES NOT DO: it never weakens authentication. `WebUI\AuthSubnetWhitelist` (bypassing auth
|
||||
# for the pod CIDR) was REJECTED: it leaves the human login broken, and makes qBittorrent's auth
|
||||
# depend on network policy staying correct forever. The WebUI keeps demanding a password; that
|
||||
# password is simply now known, fixed, and reproducible from git.
|
||||
postStart:
|
||||
exec:
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# Assert the durable WebUI credential in the config PVC's qBittorrent.conf, then PROVE the
|
||||
# running WebUI accepts it. Idempotent: the steady-state path writes nothing at all.
|
||||
#
|
||||
# THIS SCRIPT CONTAINS NO BRACED DOLLAR EXPANSION AT ALL, AND THAT IS DELIBERATE.
|
||||
# apps-wave-0 declares postBuild.substituteFrom cluster-vars, so Flux runs envsubst over
|
||||
# this whole manifest at RECONCILE time. The repo's escape contract (double the dollar;
|
||||
# enforced by render.py's assert_substitution_safe) makes such a form correct ON THE
|
||||
# CLUSTER — but only there:
|
||||
# envsubst collapses the doubled dollar, so the SAME text run any other way (a `kubectl
|
||||
# apply` of the golden, which runs no envsubst; a hand `kubectl exec` of the script during
|
||||
# development) keeps the doubled dollar, which bash reads as its own PID. Not hypothetical;
|
||||
# see the PORT comment below for the bug it actually caused here. Bare `$VAR` and `$(cmd)`
|
||||
# are untouched by envsubst, so they are used throughout and the script behaves IDENTICALLY
|
||||
# however it is run.
|
||||
set -u
|
||||
CONF=/config/qBittorrent/qBittorrent.conf
|
||||
SVC=/run/service/svc-qbittorrent
|
||||
# The WebUI's listening port, as a LITERAL. It matches ingressRoutes[0].port in this
|
||||
# ServiceDefinition — which is the single place the renderer takes the containerPort, the
|
||||
# Service port and the HTTPRoute backend port from — and the image exposes no WEBUI_PORT
|
||||
# env var for a lookup to read anyway.
|
||||
# It USED to be a doubled-dollar default-operator expansion of a WEBUI_PORT env var, and
|
||||
# that cost real time: run without envsubst, bash read the doubled dollar as its own PID,
|
||||
# so PORT became "<pid>{WEBUI_PORT:-8080}" — the brace never expanded, the digits never
|
||||
# matched. Every curl below then got an unparseable URL, returned nothing, and the hook
|
||||
# logged "the WebUI rejected the durable
|
||||
# credential" — having never actually asked it. That MISDIAGNOSIS is preserved in this
|
||||
# service's own marker log at 2026-09-13T05:35Z (three "rejected" lines, then GAVE UP,
|
||||
# against a WebUI that accepted the credential perfectly well). A literal cannot be mangled.
|
||||
PORT=8080
|
||||
# A postStart hook's stdout/stderr goes to the KUBELET, not to `kubectl logs` — so a hook
|
||||
# that only echoes is effectively silent (gitea's hits the same wall and writes a marker
|
||||
# file for it). Every outcome therefore also lands, timestamped, on the persistent config
|
||||
# volume: /config/qbittorrent-webui-cred.log is where an operator looks to see whether the
|
||||
# credential from git was asserted and accepted on this start.
|
||||
log() {
|
||||
echo "[qbt-webui-cred] $*"
|
||||
printf '%s [qbt-webui-cred] %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" \
|
||||
>> /config/qbittorrent-webui-cred.log 2>/dev/null || true
|
||||
}
|
||||
|
||||
# (1) The image's init-qbittorrent-config copies /defaults/qBittorrent.conf onto the PVC
|
||||
# before it starts the service. Wait (bounded, ~30s) so we EDIT that file instead of
|
||||
# racing to create one it would then not overwrite.
|
||||
n=0
|
||||
while [ ! -s "$CONF" ] && [ "$n" -lt 120 ]; do n=$((n + 1)); sleep 0.25; done
|
||||
|
||||
waited=0
|
||||
attempt=0
|
||||
while [ "$attempt" -lt 3 ]; do
|
||||
attempt=$((attempt + 1))
|
||||
|
||||
# (2) Converge the conf. 0 = already correct (steady state: NO write, NO re-salt),
|
||||
# 10 = rewritten, anything else = could not write.
|
||||
python3 - "$CONF" <<'PY'
|
||||
import base64, hashlib, hmac, os, stat, sys
|
||||
|
||||
conf = sys.argv[1]
|
||||
user = os.environ.get("QBT_WEBUI_USERNAME", "")
|
||||
pw = os.environ.get("QBT_WEBUI_PASSWORD", "")
|
||||
if not user or not pw:
|
||||
sys.stderr.write("[qbt-webui-cred] credential env is empty - refusing to touch the conf\n")
|
||||
raise SystemExit(1)
|
||||
|
||||
# qBittorrent's OWN parameters (base/utils/password.cpp): PBKDF2-HMAC-SHA512, 100000
|
||||
# iterations, 16-byte random salt, 64-byte digest, stored as
|
||||
# WebUI\Password_PBKDF2="@ByteArray(<base64 salt>:<base64 digest>)"
|
||||
# Get any one of them wrong and the app simply rejects every login.
|
||||
ITERATIONS, DIGEST_LEN, SALT_LEN = 100000, 64, 16
|
||||
UKEY, PKEY, SECTION = "WebUI\\Username", "WebUI\\Password_PBKDF2", "[Preferences]"
|
||||
|
||||
def matches(raw):
|
||||
"""True iff the stored digest is one OUR password produces (salt is read from it)."""
|
||||
v = raw.strip().strip('"')
|
||||
if v.startswith("@ByteArray(") and v.endswith(")"):
|
||||
v = v[len("@ByteArray("):-1]
|
||||
try:
|
||||
salt_b64, digest_b64 = v.split(":", 1)
|
||||
salt = base64.b64decode(salt_b64, validate=True)
|
||||
digest = base64.b64decode(digest_b64, validate=True)
|
||||
except Exception:
|
||||
return False
|
||||
if not salt or not digest:
|
||||
return False
|
||||
return hmac.compare_digest(
|
||||
hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, len(digest)), digest)
|
||||
|
||||
try:
|
||||
lines = open(conf, encoding="utf-8", errors="replace").read().splitlines()
|
||||
except FileNotFoundError:
|
||||
lines = []
|
||||
|
||||
def value_of(key):
|
||||
for i, line in enumerate(lines):
|
||||
if line.startswith(key + "="):
|
||||
return i, line.split("=", 1)[1]
|
||||
return -1, None
|
||||
|
||||
ui, uval = value_of(UKEY)
|
||||
pi, pval = value_of(PKEY)
|
||||
if uval == user and pval is not None and matches(pval):
|
||||
raise SystemExit(0) # converged - leave the file (and its salt) alone
|
||||
|
||||
salt = os.urandom(SALT_LEN)
|
||||
digest = hashlib.pbkdf2_hmac("sha512", pw.encode(), salt, ITERATIONS, DIGEST_LEN)
|
||||
stored = base64.b64encode(salt).decode() + ":" + base64.b64encode(digest).decode()
|
||||
desired = [UKEY + "=" + user, PKEY + '="@ByteArray(' + stored + ')"']
|
||||
|
||||
out = [line for i, line in enumerate(lines) if i not in (ui, pi)]
|
||||
if SECTION in out:
|
||||
at = out.index(SECTION) + 1
|
||||
else:
|
||||
if out and out[-1].strip():
|
||||
out.append("")
|
||||
out.append(SECTION)
|
||||
at = len(out)
|
||||
out[at:at] = desired
|
||||
|
||||
try:
|
||||
os.makedirs(os.path.dirname(conf) or ".", exist_ok=True)
|
||||
tmp = conf + ".webui-cred.new"
|
||||
with open(tmp, "w", encoding="utf-8") as fh:
|
||||
fh.write("\n".join(out) + "\n")
|
||||
fh.flush()
|
||||
os.fsync(fh.fileno())
|
||||
# This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would
|
||||
# cost the app the ability to save its own settings, so carry the ownership over.
|
||||
try:
|
||||
st = os.stat(conf)
|
||||
os.chown(tmp, st.st_uid, st.st_gid)
|
||||
os.chmod(tmp, stat.S_IMODE(st.st_mode))
|
||||
except FileNotFoundError:
|
||||
os.chown(tmp, int(os.environ.get("PUID", "1000")),
|
||||
int(os.environ.get("PGID", "1000")))
|
||||
os.chmod(tmp, 0o664)
|
||||
os.replace(tmp, conf) # atomic: the app never reads a half-written conf
|
||||
except OSError as exc:
|
||||
sys.stderr.write("[qbt-webui-cred] cannot write %s: %s\n" % (conf, exc))
|
||||
raise SystemExit(1)
|
||||
raise SystemExit(10)
|
||||
PY
|
||||
rc=$?
|
||||
|
||||
if [ "$rc" = 0 ]; then
|
||||
log "conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)"
|
||||
elif [ "$rc" = 10 ]; then
|
||||
log "asserted WebUI Username + Password_PBKDF2 into $CONF"
|
||||
# postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY
|
||||
# have read the old, credential-less conf and be serving a random temporary password.
|
||||
# Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.
|
||||
# Nothing to do when the service has not started yet - it will read the new file.
|
||||
# Its own counter, NOT the shared `waited` API budget: sharing one counter is how the
|
||||
# 2026-09-13 failure spent the entire verification budget before the first login and
|
||||
# then fired attempts 2 and 3 with no wait left at all.
|
||||
sv=0
|
||||
while [ "$sv" -lt 60 ] && [ "$(s6-svstat -u "$SVC" 2>/dev/null)" != "true" ]; do
|
||||
sv=$((sv + 1)); sleep 1
|
||||
done
|
||||
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
|
||||
log "qbittorrent-nox was already running - restarting it to load the credential"
|
||||
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)"
|
||||
sleep 3
|
||||
fi
|
||||
else
|
||||
log "FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep" >&2
|
||||
log "minting a random password every start and BOTH arrs will break. Fix and restart." >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL
|
||||
# (shared ~120s budget across attempts), then spend exactly ONE login per attempt -
|
||||
# at most 3 rejected logins, which stays under qBittorrent's default ban threshold.
|
||||
#
|
||||
# "THE API NEVER ANSWERED" IS A DIFFERENT FAULT FROM "THE PASSWORD WAS REJECTED", and
|
||||
# conflating them is what made this hook lie once already (see the PORT comment). An
|
||||
# unreachable WebUI is reported as unreachable, and the hook stops rather than spending
|
||||
# its remaining login attempts to produce two more identical wrong diagnoses.
|
||||
api_ok=0
|
||||
while [ "$waited" -lt 120 ]; do
|
||||
code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' "http://127.0.0.1:$PORT/" 2>/dev/null)
|
||||
if [ -n "$code" ] && [ "$code" != "000" ]; then api_ok=1; break; fi
|
||||
waited=$((waited + 1)); sleep 1
|
||||
done
|
||||
if [ "$api_ok" != 1 ]; then
|
||||
log "the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the" >&2
|
||||
log "credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was" >&2
|
||||
log "never submitted. Do not rotate the credential on the strength of this line." >&2
|
||||
exit 0
|
||||
fi
|
||||
body=$(curl -sS -m 10 \
|
||||
--data-urlencode "username=$QBT_WEBUI_USERNAME" \
|
||||
--data-urlencode "password=$QBT_WEBUI_PASSWORD" \
|
||||
"http://127.0.0.1:$PORT/api/v2/auth/login" 2>/dev/null)
|
||||
if [ "$body" = "Ok." ]; then
|
||||
log "VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'"
|
||||
exit 0
|
||||
fi
|
||||
log "WebUI rejected the durable credential (login said '$body')" >&2
|
||||
# A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean
|
||||
# the running qbittorrent-nox is serving something it read from an EARLIER conf, so
|
||||
# rewriting the same bytes on the next attempt would change nothing. The remedy is to
|
||||
# make the process RE-READ the file, so restart the supervised service and let the next
|
||||
# attempt re-verify. Without this the retry loop could never change its own outcome.
|
||||
if [ "$(s6-svstat -u "$SVC" 2>/dev/null)" = "true" ]; then
|
||||
log "restarting qbittorrent-nox so it re-reads $CONF, then re-verifying" >&2
|
||||
s6-svc -r "$SVC" || log "s6-svc -r failed (continuing to verify anyway)" >&2
|
||||
sleep 3
|
||||
fi
|
||||
done
|
||||
|
||||
log "GAVE UP after $attempt attempts: the WebUI does not accept the credential from git." >&2
|
||||
# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container
|
||||
# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.
|
||||
# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the
|
||||
# app, and the arrs' own download-client health check surfaces a broken credential.
|
||||
exit 0
|
||||
podSecurityContext:
|
||||
# For the LONGHORN config PVC only, and for the same reason radarr/sonarr set it: fsGroup is a NO-OP
|
||||
# on the NFS volume (the in-tree plugin does no ownership management), but on a Longhorn filesystem
|
||||
|
||||
@@ -136,6 +136,39 @@ printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4"
|
||||
assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)"
|
||||
assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01"
|
||||
|
||||
# ---------------------------------------------------------------------------------------
|
||||
# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13)
|
||||
#
|
||||
# The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match `R-C2-01` or
|
||||
# `R-C2-02` — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. Those are
|
||||
# the only two registry IDs whose workstream segment carries a digit, and the scan was blind
|
||||
# to BOTH, in the fail-OPEN direction: a change self-declaring a C2 requirement contributed
|
||||
# it to nobody, so the C2 conformance tests were filtered out of the merge gate. Found while
|
||||
# wiring conformance/tests/l0-static/c2-root-surface, which closes those two coverage gaps.
|
||||
#
|
||||
# `notes/` is outside path_requirements() on purpose, so the IDs below can only have come
|
||||
# from the CONTENT SCAN. A neighbouring all-letter ID is asserted too, so a regex that
|
||||
# somehow matched only the digit form would still fail here.
|
||||
git -C "$TMP" checkout -q -b feature5 main
|
||||
cat > "$TMP/notes/c2.yaml" <<'YAML'
|
||||
kind: Test
|
||||
metadata:
|
||||
labels:
|
||||
conformance.home.k3s/R-C2-01: "true"
|
||||
conformance.home.k3s/R-C2-02: "true"
|
||||
conformance.home.k3s/R-OBS-04: "true"
|
||||
YAML
|
||||
git -C "$TMP" add -A && git -C "$TMP" commit -qm "declare R-C2-01 / R-C2-02"
|
||||
OUT5="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)"
|
||||
printf 'touched set (digit in the workstream segment):\n%s\n---\n' "$OUT5"
|
||||
assert_present "R-C2-01" "$OUT5" "an R-ID with a DIGIT in the workstream segment IS attributed (R-C2-01)"
|
||||
assert_present "R-C2-02" "$OUT5" "the second such ID is attributed too (R-C2-02)"
|
||||
assert_present "R-OBS-04" "$OUT5" "an all-letter workstream segment still works (control)"
|
||||
# The pattern must not become so loose that a non-ID token is read as a requirement: a bare
|
||||
# numeric workstream is not a registry shape, and attributing one would WIDEN the gate on
|
||||
# noise (over-broad selection, never a false green, but noisy and misleading).
|
||||
assert_absent "R-1-01" "$OUT5" "a purely numeric workstream segment is not invented from thin air"
|
||||
|
||||
echo
|
||||
if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0;
|
||||
else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi
|
||||
@@ -345,6 +345,21 @@ main() {
|
||||
done <<< "$files"
|
||||
|
||||
# Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff).
|
||||
#
|
||||
# THE ID PATTERN IS `R-[A-Z0-9]+-[0-9]+`, NOT `R-[A-Z]+-[0-9]+`. The WORKSTREAM segment may
|
||||
# contain a DIGIT: `R-C2-01` and `R-C2-02` (command & control UI) are the only two such IDs
|
||||
# in conformance/requirements/registry.yaml today, and the stricter pattern silently failed
|
||||
# to match BOTH of them — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is.
|
||||
# Every other consumer in the repo already allows the digit: test_requirements_for_path()
|
||||
# twenty lines above uses `R-[A-Z0-9]+-[0-9]+`, as do conformance/lib/coverage.py,
|
||||
# conformance/lib/status.py and conformance/lib/run_l1_offline.py. This grep was the lone
|
||||
# dissenter, and it dissented by UNDER-attributing — the fail-OPEN direction. A change that
|
||||
# retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02 from a
|
||||
# path the map keys elsewhere, contributed those IDs to NOBODY, so the C2 tests were
|
||||
# filtered out of the merge gate. Same defect class as the map's `*) : ;;` arm: a miss that
|
||||
# reads as "nothing touched". path_requirements() has no arm for platform/homepage/* or
|
||||
# platform/headlamp/* either, so before this fix the ONLY route to R-C2-01/02 was rule 3
|
||||
# (editing the C2 test itself). Regression-tested in ci/tests/touched-requirements.test.sh.
|
||||
# `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope.
|
||||
# `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so
|
||||
# what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line
|
||||
@@ -358,7 +373,7 @@ main() {
|
||||
| grep -vE '^(\+\+\+|---)' \
|
||||
| sed -E 's/^[+-]//' \
|
||||
| grep -vE '^[[:space:]]*#' \
|
||||
| grep -hoE 'R-[A-Z]+-[0-9]+' \
|
||||
| grep -hoE 'R-[A-Z0-9]+-[0-9]+' \
|
||||
| tr '\n' ' ' || true)"
|
||||
|
||||
# Normalize: split, unique, sort. Emit to stdout.
|
||||
|
||||
@@ -34,8 +34,12 @@ spec:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/homepage
|
||||
# Homepage references no ${VARS} today; the substitution is retained for parity with the other
|
||||
# tiers / future overlay knobs.
|
||||
# LOAD-BEARING, not parity decoration (the comment here used to say the opposite). This component
|
||||
# references ${BASE_DOMAIN} in TWO places: httproute.yaml's apex hostname, and EVERY tile href in
|
||||
# configmap.yaml. Without this substitution the token is applied LITERALLY — the apex route binds
|
||||
# the host `${BASE_DOMAIN}`, which matches no Gateway listener, and every link on the operator's
|
||||
# front door points at a host that does not exist — while Flux still reports this Kustomization
|
||||
# Ready. It fails OPEN. Asserted by conformance/tests/l0-static/c2-root-surface (5).
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
|
||||
@@ -0,0 +1,116 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the C2 root surfaces every service the cluster runs, plus platform
|
||||
# health and controlled actions (R-C2-01, R-C2-02) [EXECUTABLE]
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# THE REQUIREMENTS, VERBATIM (spec docs/00-vision-and-scope.md):
|
||||
# R-C2-01 "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services
|
||||
# running in the cluster and links to them."
|
||||
# R-C2-02 "The C2 UI MUST surface platform health (cluster, storage, GitOps, alerts) and
|
||||
# provide controlled operational actions where safe."
|
||||
# Detail: spec docs/12-command-center-ui.md — Homepage at the root is "the primary R-C2-01
|
||||
# surface"; Headlamp is "the R-C2-02 'controlled operational actions where safe' surface";
|
||||
# Grafana is the dashboards/alerts surface; "never a raw cluster-admin kubeconfig in a browser".
|
||||
#
|
||||
# WHY L0, AND NOT L2
|
||||
# ------------------
|
||||
# The REACHABILITY half of R-C2-01 is already exercised against a live cluster:
|
||||
# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls
|
||||
# verifies the certificate it serves. What no live probe can decide is the half the
|
||||
# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING
|
||||
# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts:
|
||||
# what git declares the cluster serves (every in-repo HTTPRoute hostname, every L4 Service
|
||||
# that publishes a hostname, every catalog ServiceDefinition) against what the dashboard's
|
||||
# ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides half the
|
||||
# estate and cannot tell the difference.
|
||||
#
|
||||
# It is also the assertion that ROTS, in the direction nothing else notices: onboard a
|
||||
# service, forget the dashboard, and every cluster test stays green while the operator's
|
||||
# front door hides a running service. That is not a styling preference — it is exactly the
|
||||
# R-C2-01 failure mode, and this check found a live instance of it when it was written (see
|
||||
# the commit message: platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root
|
||||
# linked to it).
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the apex ingress, the dashboard's declared link set, every in-repo HTTPRoute, every
|
||||
# hostname-publishing L4 Service, the versioned catalog and the platform-health
|
||||
# surfaces are read and compared,
|
||||
# THEN — R-C2-01 —
|
||||
# the bare apex is claimed by exactly one route, that route binds the ONE Gateway
|
||||
# listener whose hostname matches a bare apex, the apex is a SAN on the certificate
|
||||
# that listener terminates with, the dashboard is RECONCILED (not merely authored),
|
||||
# FORWARD every HTTP service the repo routes is linked from the root, FORWARD every
|
||||
# catalog `kind: container` service with a route is linked, and REVERSE every
|
||||
# home-zone link the root offers resolves to something this repo actually serves —
|
||||
# an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4);
|
||||
# — R-C2-02 —
|
||||
# both health surfaces docs/12 names are linked AND reconciled, the health DATA exists
|
||||
# as code (PrometheusRule groups for cluster/nodes, storage, GitOps and cert expiry,
|
||||
# reconciled by the platform-config tier), and actions are CONTROLLED: the console
|
||||
# binds NO ClusterRole to its own ServiceAccount, authenticates per-user by native
|
||||
# OIDC from a named SOPS Secret, and cluster rights arrive through OIDC GROUPS at more
|
||||
# than one privilege level.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files + PyYAML —
|
||||
# the same dependency catalog/render.py already requires). conformance/run.sh --layer l0 and
|
||||
# ci/l0.sh run it directly; this Chainsaw Test wraps the SAME script.
|
||||
#
|
||||
# WHAT IT DOES NOT PROVE — honest gaps, restated in check.sh's own header:
|
||||
# - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2;
|
||||
# the Gateway/TLS half is covered under R-DNS-01 / R-TLS-01.
|
||||
# - That the C2 is access-CONTROLLED in fact. The auth assertions read a DECLARED posture,
|
||||
# because the Gateway ext-authz attachment that would enforce forward-auth is not built
|
||||
# anywhere in this repo (platform/homepage/httproute.yaml says so in its own header, and
|
||||
# every forward-auth app shares the gap). Declared, not enforced — and labelled as such
|
||||
# in the assertion text.
|
||||
# - That an operational action in the console succeeds, or that it is audited. The
|
||||
# assertions prove there is no shared cluster-admin identity to act with and that rights
|
||||
# arrive per-user; the audit trail is a runtime property.
|
||||
# - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up
|
||||
# (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: c2-root-surface
|
||||
labels:
|
||||
conformance.home.k3s/R-C2-01: "true"
|
||||
conformance.home.k3s/R-C2-02: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-C2-01, R-C2-02"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE, and it is the whole point of the forward assertion.
|
||||
#
|
||||
# The invariant is "every service the repo routes is linked from the C2 root". Its input
|
||||
# surface is therefore EVERY tracked manifest that can declare an HTTPRoute — apps/wave-*,
|
||||
# platform/*, runbooks/ — not just the two components that own the C2. ci/touched-
|
||||
# requirements.sh keys `platform/homepage/*` and `platform/headlamp/*` to nothing at all
|
||||
# today (there is no path-map arm for either), and even with one, the change that BREAKS
|
||||
# this invariant is the change that adds a route somewhere else: `apps/wave-1/newthing.yaml`
|
||||
# attributes R-GITOPS-02 / R-NET-* / nothing, never R-C2-01. Requirement-scoped selection
|
||||
# would filter this check out of exactly the change that introduces the bug — the R-TEST-03
|
||||
# blind spot recorded in conformance/lib/common.sh.
|
||||
#
|
||||
# Declaring the scope is the whole fix: conformance/run.sh --include-scope whole-repo
|
||||
# (ci/l0.sh Stage 3b) unions this test into EVERY gate invocation. Do NOT hand-add a stage
|
||||
# to ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does.
|
||||
# A SKIP of a scope-declaring check is rc 3 (UNWIRED), never green.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
The bare apex is bound to the landing dashboard over trusted TLS and is reconciled; the
|
||||
dashboard links every HTTP service this repo routes and every link it offers resolves to
|
||||
something the repo serves; and the C2 surfaces platform health backed by real alert rules
|
||||
while granting operational rights per-user through OIDC groups rather than through a
|
||||
shared cluster-admin identity.
|
||||
spec:
|
||||
steps:
|
||||
- name: compare-the-c2-root-against-what-the-repo-actually-serves
|
||||
description: >-
|
||||
Read the apex ingress chain, the dashboard's declared links, every in-repo HTTPRoute and
|
||||
hostname-publishing Service, the catalog, the platform-health alert rules and the
|
||||
console's RBAC posture, and assert the two sets agree in both directions.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+656
@@ -0,0 +1,656 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — c2-root-surface
|
||||
#
|
||||
# Serves (spec docs/00-vision-and-scope.md; detail in docs/12-command-center-ui.md):
|
||||
#
|
||||
# R-C2-01 "The root `home.keithjackson.dev` MUST serve a UI that surfaces the
|
||||
# services running in the cluster and links to them."
|
||||
# R-C2-02 "The C2 UI MUST surface platform health (cluster, storage, GitOps,
|
||||
# alerts) and provide controlled operational actions where safe."
|
||||
#
|
||||
# WHY THIS IS AN L0 STATIC CHECK AND NOT AN L2 PROBE
|
||||
# --------------------------------------------------
|
||||
# The REACHABILITY half of R-C2-01 is already exercised against a live cluster:
|
||||
# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls
|
||||
# verifies the certificate it serves. What no live probe can decide is the half the
|
||||
# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING
|
||||
# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts:
|
||||
# what git declares the cluster serves (every in-repo HTTPRoute hostname, every
|
||||
# hostname-publishing L4 Service, every catalog ServiceDefinition) against what the
|
||||
# dashboard's ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides
|
||||
# half the estate and cannot tell the difference.
|
||||
#
|
||||
# It is also the assertion that ROTS, in the direction nothing else notices: onboard a
|
||||
# service, forget the dashboard, and every cluster test stays green while the operator's
|
||||
# front door hides a running service. That is not a styling preference — it is exactly the
|
||||
# R-C2-01 failure mode, and it was a LIVE defect on main when this check was written
|
||||
# (platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root linked to it).
|
||||
#
|
||||
# THE RULE THIS IMPLEMENTS, in one sentence: every home-zone hostname the repository routes
|
||||
# over HTTP must be linked from the apex dashboard's configuration, every home-zone link the
|
||||
# apex dashboard offers must resolve to something the repository actually serves, and the two
|
||||
# R-C2-02 surfaces (health data + a per-user-authorized console) must be linked, reconciled
|
||||
# and backed by real alert rules.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN the apex ingress chain, the dashboard's declared link set, every in-repo HTTPRoute,
|
||||
# every hostname-publishing L4 Service, the versioned catalog, the platform-health
|
||||
# alert rules and the console's RBAC posture are read and compared,
|
||||
# THEN — R-C2-01 —
|
||||
# (0) the INPUT SURFACE is intact: every Flux-reconciled path lies inside the roots
|
||||
# this check scans, and every manifest in them parses. A root this check does not
|
||||
# scan, or a file it cannot parse, is a service it cannot see — which would
|
||||
# silently weaken every assertion below;
|
||||
# (1) EXACTLY ONE HTTPRoute claims the bare apex (an ambiguous root is not a root),
|
||||
# and it belongs to the landing-dashboard component;
|
||||
# (2) that route pins by sectionName the ONE Gateway listener whose hostname is the
|
||||
# bare apex, and that listener is HTTPS — the wildcard `*.${BASE_DOMAIN}` listener
|
||||
# does NOT intersect the bare apex, so without the sectionName the root binds
|
||||
# nothing and serves nothing (asserted against the wildcard listener's existence,
|
||||
# so the claim is proven rather than asserted from prose);
|
||||
# (3) the apex is a SAN on the Certificate whose Secret that listener terminates with;
|
||||
# (4) the dashboard is RECONCILED, not merely authored: one Flux Kustomization points
|
||||
# at its path and is listed in the base entrypoint set;
|
||||
# (5) that Kustomization carries postBuild.substituteFrom, because the dashboard's
|
||||
# hrefs and its route's hostname are written as `${BASE_DOMAIN}` tokens — without
|
||||
# substitution the apex route binds a literal `${BASE_DOMAIN}` and EVERY tile on
|
||||
# the root links to a nonexistent host, while Flux still reports Ready;
|
||||
# (6) the component's own kustomization ships config + workload + Service + route;
|
||||
# (7) the dashboard is a service CATALOG: grouped entries carrying hrefs;
|
||||
# (8) FORWARD — every home-zone hostname an in-repo HTTPRoute serves, other than the
|
||||
# apex itself, is LINKED from the root. This is the rot-catcher;
|
||||
# (9) FORWARD — every catalog `kind: container` ServiceDefinition whose declared
|
||||
# ingress host is routed is likewise linked, tying the C2 to the VERSIONED
|
||||
# onboarding contract and not only to whatever is rendered today;
|
||||
# (10) REVERSE — every home-zone link the root offers resolves to something this repo
|
||||
# actually serves: an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service
|
||||
# (L4). A dead tile is a link the root offers to nothing;
|
||||
# (11) the off-cluster escape hatch is not a blanket one: every name declared in the
|
||||
# ConfigMap's `off-cluster-links` annotation is a specific home-zone hostname and
|
||||
# none of them shadows a hostname the repo DOES serve;
|
||||
# (12) no href names a LITERAL base-domain host. Every hostname is the `${BASE_DOMAIN}`
|
||||
# token, so prod's root cannot link dev's services — the 2026-09-04 incident class
|
||||
# (two clusters authored to describe the same names), applied to the front door;
|
||||
# — R-C2-02 —
|
||||
# (13) both health surfaces docs/12 names are linked: the Kubernetes console
|
||||
# (cluster/storage/GitOps state) and the dashboards/alerts surface;
|
||||
# (14) both are RECONCILED by Flux, so the links lead to something running;
|
||||
# (15) the health DATA those surfaces render exists as code — PrometheusRule groups for
|
||||
# each axis R-C2-02 enumerates (cluster/nodes, storage, GitOps reconciliation)
|
||||
# plus certificate expiry — and those rule files are reconciled;
|
||||
# (16) actions are CONTROLLED, not cluster-admin-in-a-browser (docs/12: "never a raw
|
||||
# cluster-admin kubeconfig in a browser"): the console chart is explicitly told NOT
|
||||
# to bind its ServiceAccount to a ClusterRole, so there is no shared privileged
|
||||
# identity to act as;
|
||||
# (17) actions are instead PER-USER: the console is configured for native OIDC with
|
||||
# credentials from a named (SOPS) Secret, not inline;
|
||||
# (18) cluster rights arrive through OIDC GROUPS at MORE THAN ONE privilege level (an
|
||||
# admin group and a lesser one) — least-privilege, not one tier for everybody;
|
||||
# (19) the C2 root DECLARES the authenticated-group posture docs/12 requires, on its own
|
||||
# route (declared, NOT enforced — see the gap note below).
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE — honest gaps, the same ones the Test declaration names:
|
||||
# - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2; the
|
||||
# Gateway/TLS half is covered under R-DNS-01 / R-TLS-01.
|
||||
# - That the C2 is access-CONTROLLED in fact. (19) reads a DECLARED posture, because the
|
||||
# Gateway ext-authz attachment that would ENFORCE forward-auth is not built anywhere in
|
||||
# this repo (platform/homepage/httproute.yaml says so in its own header, and every
|
||||
# forward-auth app shares the gap). Declared, not enforced, and labelled as such in the
|
||||
# assertion text itself.
|
||||
# - That an operational action in the console succeeds, or that it is audited. (16)-(18)
|
||||
# prove there is no shared cluster-admin identity to act with and that rights arrive
|
||||
# per-user; the audit trail is a runtime property.
|
||||
# - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up
|
||||
# (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for.
|
||||
# - That an L4-ONLY endpoint is surfaced. A raw TCP/UDP endpoint (a Minecraft game port) has
|
||||
# no URL a browser can follow, so the FORWARD direction quantifies over HTTP routes;
|
||||
# L4-published hostnames are listed as ADVISORIES and are honoured in the REVERSE
|
||||
# direction (that is how the UniFi tile, an L4 LoadBalancer on :11443, resolves).
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and the
|
||||
# requirement text above is quoted, not parsed.
|
||||
#
|
||||
# STATUS: EXECUTABLE. Pure offline: `git ls-files` + PyYAML (the dependency ten neighbouring
|
||||
# l0-static checks already take). No cluster, no network, no spec repo.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
# Honest SKIP (never green, and rc 3 is UNWIRED for a scope-declaring test) if the landing
|
||||
# dashboard component is absent entirely — there is then no C2 root to audit, which is a
|
||||
# report about the tree, not a pass.
|
||||
if [ ! -f "${ROOT}/platform/homepage/configmap.yaml" ]; then
|
||||
echo "check SKIPPED: platform/homepage/configmap.yaml absent — no C2 landing dashboard to audit" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
"""Static audit: the C2 root surfaces every service the repository serves, plus health.
|
||||
|
||||
Every parse contract is documented at its parser. PyYAML rather than regex because the
|
||||
subjects are genuinely structured: multi-document apps/ bundles, and a Homepage config that
|
||||
is nested YAML *inside* a ConfigMap string value.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree
|
||||
|
||||
FAILURES: list[str] = []
|
||||
ADVISORIES: list[str] = []
|
||||
CHECKS = 0
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if cond:
|
||||
print(f" ok {msg}")
|
||||
else:
|
||||
print(f" FAIL {msg}", file=sys.stderr)
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def read(p: Path) -> str:
|
||||
try:
|
||||
return p.read_text()
|
||||
except (OSError, UnicodeDecodeError):
|
||||
return ""
|
||||
|
||||
|
||||
# ============================================================ the hostname vocabulary
|
||||
# BASE_DOMAIN is a cluster-vars knob (R-ENV-02): every public hostname the cluster serves is
|
||||
# `<svc>.${BASE_DOMAIN}`, and dev/prod differ ONLY in that value. So a hostname in this repo
|
||||
# is canonically the TOKEN form, and the literal value below exists only to be FORBIDDEN in
|
||||
# (12): a literal is how prod came to be authored to serve dev's hostnames (2026-09-04,
|
||||
# clusters/base/cluster-vars.yaml's own header).
|
||||
TOKEN = "${BASE_DOMAIN}"
|
||||
_vars = yaml.safe_load(read(ROOT / "clusters" / "base" / "cluster-vars.yaml")) or {}
|
||||
LITERAL_BASE = ((_vars.get("data") or {}).get("BASE_DOMAIN") or "").strip()
|
||||
|
||||
|
||||
def strip_port(host: str) -> str:
|
||||
return host.split(":", 1)[0]
|
||||
|
||||
|
||||
def zone_label(host: str):
|
||||
"""The sub-label of a home-zone hostname, "" for the bare apex, or None if not in zone.
|
||||
|
||||
Accepts only the TOKEN form; a literal-base hostname returns None here and is caught
|
||||
separately by (12), so a literal can never silently satisfy a set comparison.
|
||||
"""
|
||||
h = strip_port(host)
|
||||
if h == TOKEN:
|
||||
return ""
|
||||
if h.endswith("." + TOKEN):
|
||||
return h[: -len("." + TOKEN)]
|
||||
return None
|
||||
|
||||
|
||||
def is_literal_base(host: str) -> bool:
|
||||
h = strip_port(host)
|
||||
return bool(LITERAL_BASE) and (h == LITERAL_BASE or h.endswith("." + LITERAL_BASE))
|
||||
|
||||
|
||||
# ============================================================ (0) the input surface
|
||||
# DELIVERY ROOTS — the subtrees Flux reconciles, plus catalog/ (the versioned source the
|
||||
# apps/ manifests are rendered FROM). conformance/ is deliberately EXCLUDED: its fixtures and
|
||||
# goldens declare synthetic HTTPRoutes (`paas-demo.…`, `gw-declared-conformance.…`) that are
|
||||
# test data, not services the cluster serves, and counting them would make (8) assert that
|
||||
# the operator's dashboard links this harness's fixtures.
|
||||
DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi",
|
||||
"runbooks", "clusters", "bootstrap", "catalog")
|
||||
|
||||
|
||||
def tracked_yaml() -> list[Path]:
|
||||
out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"],
|
||||
cwd=ROOT, capture_output=True, text=True, check=True)
|
||||
return [ROOT / p for p in out.stdout.split("\0") if p]
|
||||
|
||||
|
||||
ALL_TRACKED = tracked_yaml()
|
||||
DELIVERY = [p for p in ALL_TRACKED
|
||||
if p.relative_to(ROOT).parts[0] in DELIVERY_ROOTS]
|
||||
|
||||
parse_failed: list[str] = []
|
||||
DOCS: dict[Path, list[dict]] = {}
|
||||
for p in DELIVERY:
|
||||
try:
|
||||
DOCS[p] = [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
DOCS[p] = []
|
||||
parse_failed.append(str(p.relative_to(ROOT)))
|
||||
|
||||
|
||||
def docs_all():
|
||||
for p, ds in DOCS.items():
|
||||
for d in ds:
|
||||
yield p, d
|
||||
|
||||
|
||||
check(len(DELIVERY) >= 100,
|
||||
f"(0) the delivery tree is present and scanned ({len(DELIVERY)} tracked manifest(s) "
|
||||
f"under {'/, '.join(DELIVERY_ROOTS)}/)")
|
||||
check(not parse_failed,
|
||||
f"(0) every manifest in the scanned roots PARSES — an unparseable one is a service "
|
||||
f"this check cannot see, so it would weaken every set comparison below "
|
||||
f"{parse_failed if parse_failed else ''}")
|
||||
|
||||
# Every Flux-reconciled path must lie inside a root this check scans. This is the guard on
|
||||
# the guard: if a future tier reconciles, say, ./services, the forward direction would stop
|
||||
# seeing its routes and would go quietly green on a dashboard that hides all of them.
|
||||
flux_ks = [(p, d) for p, d in docs_all()
|
||||
if d.get("kind") == "Kustomization"
|
||||
and str(d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io")]
|
||||
# A RECONCILING Kustomization is one carrying a sourceRef. The pathless documents under
|
||||
# bootstrap/flux/patches/ are strategic-merge PATCH FRAGMENTS (they add a decryption stanza to
|
||||
# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring
|
||||
# a path of them would fail on a file that is correct.
|
||||
reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")]
|
||||
unscanned = sorted({
|
||||
f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}"
|
||||
for p, d in reconcilers
|
||||
if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0]
|
||||
not in DELIVERY_ROOTS
|
||||
})
|
||||
check(len(reconcilers) >= 10 and not unscanned,
|
||||
f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a "
|
||||
f"reconciled tree this check does not read is a service it cannot require a link for "
|
||||
f"{unscanned if unscanned else ''}")
|
||||
|
||||
# ============================================================ what the repo SERVES
|
||||
# L7: every hostname an in-repo HTTPRoute claims, and the component that owns it.
|
||||
routes: dict[str, list[tuple[Path, dict]]] = {}
|
||||
apex_inheriting: list[Path] = []
|
||||
for p, d in docs_all():
|
||||
if d.get("kind") != "HTTPRoute":
|
||||
continue
|
||||
hosts = (d.get("spec", {}) or {}).get("hostnames") or []
|
||||
if not hosts:
|
||||
# An HTTPRoute with NO hostnames inherits the hostname of every listener it attaches
|
||||
# to (runbooks/serving/httproute.yaml binds `https-apex` only, so it serves a PATH on
|
||||
# the apex). It introduces no new hostname, so it owes no tile.
|
||||
apex_inheriting.append(p)
|
||||
continue
|
||||
for h in hosts:
|
||||
routes.setdefault(str(h), []).append((p, d))
|
||||
|
||||
routed_labels = {} # sub-label -> owning component dir
|
||||
for h, owners in routes.items():
|
||||
lbl = zone_label(h)
|
||||
if lbl: # "" is the apex; None is out of zone
|
||||
routed_labels[lbl] = owners[0][0].parent
|
||||
|
||||
# L4: a Service that publishes a home-zone hostname to DNS is also "served by this repo" —
|
||||
# it is how UniFi (a KubeVirt VM behind a pinned Cilium LoadBalancer VIP on :11443) reaches
|
||||
# the LAN. external-dns's annotation IS the declaration; there is no HTTPRoute to find.
|
||||
DNS_ANN = "external-dns.alpha.kubernetes.io/hostname"
|
||||
l4_labels: dict[str, Path] = {}
|
||||
for p, d in docs_all():
|
||||
if d.get("kind") != "Service":
|
||||
continue
|
||||
ann = (d.get("metadata", {}) or {}).get("annotations", {}) or {}
|
||||
for h in re.split(r"[,\s]+", str(ann.get(DNS_ANN, ""))):
|
||||
lbl = zone_label(h) if h else None
|
||||
if lbl:
|
||||
l4_labels[lbl] = p
|
||||
|
||||
check(len(routed_labels) >= 10,
|
||||
f"(0) the repo's north-south HTTP surface is discovered "
|
||||
f"({len(routed_labels)} routed home-zone hostname(s), "
|
||||
f"{len(apex_inheriting)} apex-path route(s), {len(l4_labels)} L4-published hostname(s))")
|
||||
|
||||
# ============================================================ (1) the apex is unambiguous
|
||||
apex_claims = routes.get(TOKEN, [])
|
||||
check(len(apex_claims) == 1,
|
||||
f"(1) exactly ONE HTTPRoute claims the bare apex {TOKEN} — an ambiguous root is not a "
|
||||
f"root (found {len(apex_claims)}: {[str(p.relative_to(ROOT)) for p, _ in apex_claims]})")
|
||||
apex_path, apex_route = (apex_claims[0] if apex_claims else (None, {}))
|
||||
DASH = apex_path.parent if apex_path is not None else None
|
||||
check(DASH is not None and DASH.name == "homepage",
|
||||
f"(1) the apex route belongs to the landing-dashboard component "
|
||||
f"(found {DASH.name if DASH else None!r})")
|
||||
|
||||
# ============================================================ (2) it binds the apex listener
|
||||
parents = (apex_route.get("spec", {}) or {}).get("parentRefs") or []
|
||||
section = next((pr.get("sectionName") for pr in parents if pr.get("sectionName")), None)
|
||||
gw_name = next((pr.get("name") for pr in parents), None)
|
||||
gw_ns = next((pr.get("namespace") for pr in parents), None)
|
||||
check(section is not None,
|
||||
"(2) the apex route pins a Gateway listener by sectionName — the wildcard listener's "
|
||||
"hostname cannot match a bare apex, so an unpinned apex route binds NOTHING")
|
||||
|
||||
gw = next((d for _, d in docs_all()
|
||||
if d.get("kind") == "Gateway"
|
||||
and (d.get("metadata", {}) or {}).get("name") == gw_name
|
||||
and (gw_ns is None or (d.get("metadata", {}) or {}).get("namespace") == gw_ns)), None)
|
||||
check(gw is not None, f"(2) the parent Gateway '{gw_name}' is declared in the repo")
|
||||
listeners = ((gw or {}).get("spec", {}) or {}).get("listeners") or []
|
||||
apex_listener = next((l for l in listeners if l.get("name") == section), None)
|
||||
check(apex_listener is not None,
|
||||
f"(2) the Gateway declares the '{section}' listener the apex route binds")
|
||||
check(apex_listener is not None and apex_listener.get("hostname") == TOKEN,
|
||||
f"(2) that listener's hostname is EXACTLY the bare apex "
|
||||
f"(found {(apex_listener or {}).get('hostname')!r})")
|
||||
check(apex_listener is not None and apex_listener.get("protocol") == "HTTPS",
|
||||
"(2) the apex listener terminates HTTPS (docs/12 acceptance gate: trusted TLS on the root)")
|
||||
# Prove the sectionName is LOAD-BEARING rather than decorative: a wildcard listener exists,
|
||||
# it is a DIFFERENT listener, and its hostname does not equal the apex.
|
||||
wildcard = [l for l in listeners if str(l.get("hostname", "")) == "*." + TOKEN]
|
||||
check(bool(wildcard) and all(l.get("name") != section for l in wildcard),
|
||||
f"(2) the wildcard listener(s) {[l.get('name') for l in wildcard]} are DISTINCT from "
|
||||
f"'{section}' — which is why the apex needs its own listener, and why dropping the "
|
||||
f"sectionName would leave the root unserved")
|
||||
|
||||
# ============================================================ (3) trusted TLS on the root
|
||||
cert_secret = next((ref.get("name")
|
||||
for ref in ((apex_listener or {}).get("tls", {}) or {}).get("certificateRefs") or []),
|
||||
None)
|
||||
cert = next((d for _, d in docs_all()
|
||||
if d.get("kind") == "Certificate"
|
||||
and (d.get("spec", {}) or {}).get("secretName") == cert_secret), None)
|
||||
check(cert is not None,
|
||||
f"(3) a cert-manager Certificate produces the apex listener's Secret ({cert_secret!r})")
|
||||
check(cert is not None and TOKEN in ((cert.get("spec", {}) or {}).get("dnsNames") or []),
|
||||
"(3) the bare apex is a SAN on that Certificate — the wildcard alone does NOT cover it, "
|
||||
"so without this SAN the root serves an untrusted certificate")
|
||||
|
||||
# ============================================================ (4)(5) it is RECONCILED
|
||||
dash_rel = f"./{DASH.relative_to(ROOT)}" if DASH else None
|
||||
dash_ks = [(p, d) for p, d in flux_ks
|
||||
if (d.get("spec", {}) or {}).get("path", "").rstrip("/") == dash_rel]
|
||||
check(len(dash_ks) == 1,
|
||||
f"(4) exactly one Flux Kustomization reconciles the dashboard ({dash_rel}) — authored "
|
||||
f"but unreconciled is the 'looks done, is absent' class this repo keeps hitting "
|
||||
f"(found {len(dash_ks)})")
|
||||
if dash_ks:
|
||||
base_txt = read(ROOT / "clusters" / "base" / "kustomization.yaml")
|
||||
entry = dash_ks[0][0].name
|
||||
check(re.search(rf"^\s*-\s*{re.escape(entry)}\s*(#.*)?$", base_txt, re.M) is not None,
|
||||
f"(4) that Kustomization is listed in the base entrypoint set ({entry})")
|
||||
# (5) The dashboard's hrefs and its route's hostname are ${BASE_DOMAIN} TOKENS. Without
|
||||
# postBuild.substituteFrom the token reaches the cluster LITERALLY: the apex route binds
|
||||
# the host `${BASE_DOMAIN}` (matching no listener) and every tile links to a host that
|
||||
# does not exist — and Flux still reports the Kustomization Ready. It fails OPEN.
|
||||
subs = (((dash_ks[0][1].get("spec", {}) or {}).get("postBuild") or {}).get("substituteFrom") or [])
|
||||
check(any((s.get("name") == "cluster-vars") for s in subs),
|
||||
f"(5) {dash_ks[0][0].name} carries postBuild.substituteFrom cluster-vars. It MUST: "
|
||||
f"this component's route hostname and EVERY tile href are ${{BASE_DOMAIN}} tokens, "
|
||||
f"and an unsubstituted token is applied literally while Flux still reports Ready — "
|
||||
f"a root bound to no listener and a dashboard of dead links, with a green tier")
|
||||
|
||||
# ============================================================ (6) it ships the UI
|
||||
dash_kust = yaml.safe_load(read(DASH / "kustomization.yaml")) if DASH else None
|
||||
dash_res = set((dash_kust or {}).get("resources") or [])
|
||||
for want in ("configmap.yaml", "deployment.yaml", "service.yaml", "httproute.yaml"):
|
||||
check(want in dash_res, f"(6) the dashboard kustomization ships {want}")
|
||||
|
||||
# ================================================== the dashboard's DECLARED link set
|
||||
# Homepage's config is nested YAML inside ConfigMap string values (services.yaml,
|
||||
# bookmarks.yaml). Both are read: a bookmark is a link the root offers just as a tile is.
|
||||
cm = next((d for d in (DOCS.get(DASH / "configmap.yaml") or []) if d.get("kind") == "ConfigMap"),
|
||||
{}) if DASH else {}
|
||||
cm_data = cm.get("data") or {}
|
||||
|
||||
|
||||
def embedded(key):
|
||||
try:
|
||||
return yaml.safe_load(cm_data.get(key) or "") or []
|
||||
except yaml.YAMLError:
|
||||
return None
|
||||
|
||||
|
||||
svc_cfg = embedded("services.yaml")
|
||||
bm_cfg = embedded("bookmarks.yaml")
|
||||
check(svc_cfg is not None and bm_cfg is not None,
|
||||
"(7) the dashboard's embedded services.yaml / bookmarks.yaml parse as YAML — the app "
|
||||
"mounts these verbatim, so a malformed one is a dashboard that renders nothing")
|
||||
|
||||
|
||||
def hrefs(node):
|
||||
"""Every href anywhere in Homepage's nested group/entry structure."""
|
||||
if isinstance(node, dict):
|
||||
for k, v in node.items():
|
||||
if k == "href" and isinstance(v, str):
|
||||
yield v
|
||||
else:
|
||||
yield from hrefs(v)
|
||||
elif isinstance(node, list):
|
||||
for v in node:
|
||||
yield from hrefs(v)
|
||||
|
||||
|
||||
def host_of(url):
|
||||
m = re.match(r"^https?://([^/]+)", url or "")
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
tile_urls = list(hrefs(svc_cfg or []))
|
||||
all_urls = tile_urls + list(hrefs(bm_cfg or []))
|
||||
all_hosts = [h for h in (host_of(u) for u in all_urls) if h]
|
||||
linked = {lbl for lbl in (zone_label(h) for h in all_hosts) if lbl}
|
||||
tile_labels = {lbl for lbl in (zone_label(h) for h in (host_of(u) for u in tile_urls) if h) if lbl}
|
||||
|
||||
groups = [g for g in (svc_cfg or []) if isinstance(g, dict)]
|
||||
check(len(groups) >= 1 and len(tile_labels) >= 5,
|
||||
f"(7) the root is a grouped service catalog with links "
|
||||
f"({len(groups)} group(s), {len(tile_labels)} linked home-zone service(s))")
|
||||
|
||||
# ============================================ (8) FORWARD: nothing routed is hidden
|
||||
unsurfaced = sorted(set(routed_labels) - linked)
|
||||
check(not unsurfaced,
|
||||
"(8) every service the repo ROUTES over HTTP is surfaced on the C2 root. Unsurfaced: "
|
||||
+ ", ".join(f"{n}.{TOKEN} ({routed_labels[n].relative_to(ROOT)})" for n in unsurfaced)
|
||||
if unsurfaced else
|
||||
f"(8) every service the repo ROUTES over HTTP is surfaced on the C2 root "
|
||||
f"({len(routed_labels)} hostname(s) compared)")
|
||||
|
||||
# L4-only endpoints are reported, never required: a raw TCP/UDP port has no browsable URL.
|
||||
for lbl in sorted(set(l4_labels) - linked):
|
||||
ADVISORIES.append(
|
||||
f"{lbl}.{TOKEN} is published to DNS by an L4 Service "
|
||||
f"({l4_labels[lbl].relative_to(ROOT)}) and is not linked from the root. Not a "
|
||||
f"failure: a raw TCP/UDP endpoint has no URL a browser can follow. Verify the "
|
||||
f"service it belongs to is reachable through a tile that IS linked")
|
||||
|
||||
# ================================= (9) FORWARD: the VERSIONED onboarding contract
|
||||
cat_dir = ROOT / "catalog"
|
||||
cat_seen, cat_missing, cat_unrouted = 0, [], []
|
||||
for p in sorted(cat_dir.glob("*.yaml")):
|
||||
sd = next((d for d in (DOCS.get(p) or [])), None)
|
||||
if not isinstance(sd, dict) or sd.get("kind") != "container":
|
||||
continue # kind: vm has no north-south HTTP surface (docs/09)
|
||||
for r in sd.get("ingressRoutes") or []:
|
||||
lbl = zone_label(str(r.get("host", "")))
|
||||
if not lbl:
|
||||
continue
|
||||
if lbl not in routed_labels:
|
||||
# Declared in the catalog but not rendered into a route: an apps/ render drift
|
||||
# problem owned by R-GITOPS-02, reported here rather than silently skipped.
|
||||
cat_unrouted.append(f"{sd.get('name')}:{lbl}")
|
||||
continue
|
||||
cat_seen += 1
|
||||
if lbl not in linked:
|
||||
cat_missing.append(f"{sd.get('name')}:{lbl}")
|
||||
check(cat_seen >= 5,
|
||||
f"(9) the versioned catalog contributes routed HTTP services ({cat_seen} found) — the "
|
||||
f"C2 is tied to the onboarding contract, not only to today's rendered output")
|
||||
check(not cat_missing,
|
||||
f"(9) every catalog `kind: container` service with a route is linked from the C2 root "
|
||||
f"{cat_missing if cat_missing else ''}")
|
||||
for u in cat_unrouted:
|
||||
ADVISORIES.append(f"catalog declares an ingress host with no rendered HTTPRoute: {u} "
|
||||
f"(render drift — R-GITOPS-02 owns it, not R-C2-01)")
|
||||
|
||||
# ============================ (10)(11) REVERSE: no link points at nothing
|
||||
ann = (cm.get("metadata", {}) or {}).get("annotations", {}) or {}
|
||||
OFF_ANN = "homelab.keithjackson.dev/off-cluster-links"
|
||||
declared_off_raw = [t for t in re.split(r"[,\s]+", str(ann.get(OFF_ANN, ""))) if t]
|
||||
declared_off = {zone_label(t) for t in declared_off_raw}
|
||||
served = set(routed_labels) | set(l4_labels)
|
||||
dead = sorted(linked - served - declared_off)
|
||||
check(not dead,
|
||||
f"(10) every home-zone link the root offers resolves to something this repo serves — "
|
||||
f"an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4). Dead: "
|
||||
f"{[f'{d}.{TOKEN}' for d in dead] if dead else ''}")
|
||||
# The escape hatch is a DECLARATION, not a hole: same doctrine as an explicit `NONE` in
|
||||
# ci/touched-requirements.sh's path map. Empty today, and these two hold vacuously when it is
|
||||
# — they exist so it cannot be widened into a blanket without failing.
|
||||
check(all(zone_label(t) for t in declared_off_raw),
|
||||
f"(11) every declared off-cluster link is a specific home-zone hostname in TOKEN form "
|
||||
f"({declared_off_raw})")
|
||||
shadowed = sorted(declared_off & served)
|
||||
check(not shadowed,
|
||||
f"(11) no off-cluster declaration shadows a hostname the repo DOES serve "
|
||||
f"{shadowed if shadowed else ''}")
|
||||
|
||||
# ============================ (12) tokens, never literals
|
||||
literal_hosts = sorted({h for h in all_hosts if is_literal_base(h)})
|
||||
check(not literal_hosts,
|
||||
f"(12) no link on the root names a LITERAL '{LITERAL_BASE}' host — every hostname is the "
|
||||
f"${{BASE_DOMAIN}} token, so prod's front door cannot link dev's services (the "
|
||||
f"2026-09-04 class: two clusters authored to describe the same names). Literals: "
|
||||
f"{literal_hosts if literal_hosts else ''}")
|
||||
|
||||
# =====================================================================
|
||||
# R-C2-02 — platform health, and controlled operational actions
|
||||
# =====================================================================
|
||||
# docs/12 names the surfaces the C2 composes: Headlamp is "the R-C2-02 'controlled
|
||||
# operational actions where safe' surface"; Grafana is the dashboards/alerts surface. Each is
|
||||
# identified by the component that OWNS ITS ROUTE, not by a magic string in the config.
|
||||
CONSOLE, DASHBOARDS = "headlamp", "grafana"
|
||||
check(CONSOLE in linked,
|
||||
f"(13) the C2 root links the Kubernetes console ({CONSOLE}.{TOKEN}) — cluster, storage "
|
||||
f"and GitOps state, and the R-C2-02 operational-actions surface (docs/12)")
|
||||
check(DASHBOARDS in linked,
|
||||
f"(13) the C2 root links the dashboards/alerts surface ({DASHBOARDS}.{TOKEN})")
|
||||
|
||||
AGG = ("\n".join(read(ROOT / t / "kustomization.yaml")
|
||||
for t in ("platform", "platform-config", "platform-config-cdi")))
|
||||
for label, name in (("console", CONSOLE), ("dashboards", DASHBOARDS)):
|
||||
comp = routed_labels.get(name)
|
||||
check(comp is not None, f"(14) the {label} surface has an in-repo HTTPRoute")
|
||||
if comp is None:
|
||||
continue
|
||||
rel = f"./{comp.relative_to(ROOT)}"
|
||||
reconciled = any((d.get("spec", {}) or {}).get("path", "").rstrip("/") == rel
|
||||
for _, d in flux_ks)
|
||||
if not reconciled:
|
||||
# A component may reconcile as a MEMBER of an aggregate tier's kustomize tree
|
||||
# (platform / platform-config) instead of owning a Flux Kustomization. Both count;
|
||||
# being in NEITHER is "authored, never applied".
|
||||
reconciled = (f"\n - {comp.name}" in AGG
|
||||
or f"../{comp.relative_to(ROOT)}/" in AGG)
|
||||
check(reconciled,
|
||||
f"(14) the {label} surface ({comp.name}) is reconciled by Flux, not merely authored "
|
||||
f"— a link to an unreconciled component is a link to nothing")
|
||||
|
||||
# (15) the health DATA exists as code, for every axis R-C2-02 enumerates.
|
||||
rule_files = sorted({p for p, d in docs_all() if d.get("kind") == "PrometheusRule"})
|
||||
rule_groups = {g.get("name", "")
|
||||
for p, d in docs_all() if d.get("kind") == "PrometheusRule"
|
||||
for g in ((d.get("spec", {}) or {}).get("groups") or [])}
|
||||
for axis, needle in (("cluster/nodes", "nodes"),
|
||||
("storage", "longhorn"),
|
||||
("GitOps reconciliation", "flux"),
|
||||
("certificate expiry", "certificates")):
|
||||
check(any(needle in g for g in rule_groups),
|
||||
f"(15) platform-health alert rules exist for {axis} (a '*{needle}*' rule group) — "
|
||||
f"the C2 cannot surface health that is not measured")
|
||||
pc_txt = read(ROOT / "platform-config" / "kustomization.yaml")
|
||||
reconciled_rules = [p for p in rule_files if f"../{p.relative_to(ROOT)}" in pc_txt]
|
||||
check(len(reconciled_rules) >= 2,
|
||||
f"(15) those alert rules are RECONCILED — PrometheusRule is a CR of a CRD the platform "
|
||||
f"tier installs, so the rules are applied by platform-config "
|
||||
f"({len(reconciled_rules)}/{len(rule_files)} rule file(s) referenced there)")
|
||||
|
||||
# (16)(17) controlled actions: no shared cluster-admin identity; per-user OIDC instead.
|
||||
console_dir = routed_labels.get(CONSOLE)
|
||||
hr = next((d for p in (sorted(console_dir.glob("*.yaml")) if console_dir else [])
|
||||
for d in (DOCS.get(p) or []) if d.get("kind") == "HelmRelease"), None)
|
||||
check(hr is not None, "(16) the Kubernetes console is deployed by a HelmRelease")
|
||||
hv = ((hr or {}).get("spec", {}) or {}).get("values", {}) or {}
|
||||
check((hv.get("clusterRoleBinding") or {}).get("create") is False,
|
||||
"(16) the console chart is told NOT to bind its ServiceAccount to a ClusterRole. The "
|
||||
"chart's DEFAULT binds it to cluster-admin, which is exactly the 'raw cluster-admin "
|
||||
"kubeconfig in a browser' docs/12 forbids — every visitor would act as one shared "
|
||||
"omnipotent identity")
|
||||
oidc = ((hv.get("config") or {}).get("oidc") or {})
|
||||
check(bool(oidc),
|
||||
"(17) the console is configured for native OIDC login, so an action carries the "
|
||||
"operator's OWN identity (docs/12: Headlamp/Grafana use native OIDC)")
|
||||
check(bool((oidc.get("externalSecret") or {}).get("name")),
|
||||
"(17) its OIDC client credentials come from a NAMED (SOPS) Secret, not inline in the "
|
||||
"HelmRelease values — R-SEC-01")
|
||||
|
||||
# (18) rights arrive through GROUPS, at more than one privilege level.
|
||||
group_bindings = []
|
||||
for p, d in docs_all():
|
||||
if d.get("kind") not in ("ClusterRoleBinding", "RoleBinding"):
|
||||
continue
|
||||
for s in d.get("subjects") or []:
|
||||
if s.get("kind") == "Group" and str(s.get("name", "")).startswith("oidc:"):
|
||||
group_bindings.append(((d.get("metadata", {}) or {}).get("name"),
|
||||
s.get("name"),
|
||||
(d.get("roleRef") or {}).get("name")))
|
||||
check(len(group_bindings) >= 2,
|
||||
f"(18) cluster rights are bound to OIDC GROUPS, so access follows Keycloak group "
|
||||
f"membership rather than a per-user grant somebody must remember to revoke "
|
||||
f"({[(g, r) for _, g, r in group_bindings]})")
|
||||
check(any(r == "cluster-admin" for _, _, r in group_bindings)
|
||||
and any(r != "cluster-admin" for _, _, r in group_bindings),
|
||||
"(18) MORE THAN ONE privilege level exists (an admin group AND a lesser one) — "
|
||||
"least-privilege, not one tier for everybody (docs/12 access control, R-IDP-06)")
|
||||
|
||||
# (19) the root DECLARES the authenticated-group posture. DECLARED, NOT ENFORCED.
|
||||
route_ann = (apex_route.get("metadata", {}) or {}).get("annotations", {}) or {}
|
||||
check(route_ann.get("homelab.keithjackson.dev/auth-mode") in ("forward-auth", "oidc", "oauth2-proxy"),
|
||||
f"(19) the C2 root's route DECLARES an authentication mode "
|
||||
f"(found {route_ann.get('homelab.keithjackson.dev/auth-mode')!r}) "
|
||||
f"[DECLARED, NOT ENFORCED — no Gateway ext-authz attachment exists in this repo]")
|
||||
check(bool(route_ann.get("homelab.keithjackson.dev/auth-groups")),
|
||||
"(19) it DECLARES the authorized group(s) — docs/12 requires an authenticated user IN "
|
||||
"AN AUTHORIZED GROUP, not merely logged in "
|
||||
"[DECLARED, NOT ENFORCED — see the header's gap note]")
|
||||
|
||||
# ======================================================================== report
|
||||
print()
|
||||
print(f"C2 surface inventory (apex {TOKEN}, literal base default '{LITERAL_BASE}'):")
|
||||
print(f" routed over HTTP : {len(routed_labels)} -> {', '.join(sorted(routed_labels))}")
|
||||
print(f" L4-published : {len(l4_labels)} -> {', '.join(sorted(l4_labels)) or '(none)'}")
|
||||
print(f" linked from root : {len(linked)} -> {', '.join(sorted(linked))}")
|
||||
print(f" apex-path routes : {len(apex_inheriting)} -> "
|
||||
f"{', '.join(str(p.relative_to(ROOT)) for p in apex_inheriting) or '(none)'}")
|
||||
print(f" catalog services compared: {cat_seen} off-cluster declarations: "
|
||||
f"{sorted(declared_off_raw) or '(none)'}")
|
||||
|
||||
if ADVISORIES:
|
||||
print("\nADVISORIES (not failures):")
|
||||
for a in ADVISORIES:
|
||||
print(f" - {a}")
|
||||
|
||||
print()
|
||||
if not FAILURES:
|
||||
print(f"check OK: c2-root-surface — {CHECKS} assertions (R-C2-01, R-C2-02)")
|
||||
raise SystemExit(0)
|
||||
print(f"CHECK FAIL (c2-root-surface): {len(FAILURES)} failed, "
|
||||
f"{CHECKS - len(FAILURES)} passed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" FAIL {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
PY
|
||||
@@ -85,7 +85,7 @@ done
|
||||
# second would fail.
|
||||
# Deliberately LOWERCASE: conf_normalize_reqs() upper-cases it, so the selection behaves
|
||||
# identically — but the literal here does not match ci/touched-requirements.sh's content-scan
|
||||
# pattern `R-[A-Z]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement
|
||||
# pattern `R-[A-Z0-9]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement
|
||||
# to the change. (Same reason the synthetic fixture below carries no requirement label at all.)
|
||||
BOGUS_REQ="r-nosuchreq-99"
|
||||
sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)"
|
||||
|
||||
@@ -81,6 +81,18 @@ data:
|
||||
href: https://qbittorrent.${BASE_DOMAIN}
|
||||
description: Download client (VPN egress)
|
||||
icon: mdi-download-network
|
||||
# Crafty Controller — a running, north-south-exposed web UI (platform/crafty/httproute.yaml,
|
||||
# reconciled by clusters/base/crafty.yaml) that this dashboard did NOT surface until
|
||||
# 2026-09-13. R-C2-01 says the root MUST surface "the services running in the cluster and
|
||||
# links to them", so an unsurfaced route is a requirement gap, not a styling preference —
|
||||
# conformance/tests/l0-static/c2-root-surface fails on one now. Grouped under Media because
|
||||
# it manages game servers; regrouping it under Platform would be purely cosmetic. The game
|
||||
# ports themselves (crafty-mc.${BASE_DOMAIN}, TCP 25565-25567 / UDP 19132-19134) are an L4
|
||||
# LoadBalancer with no browsable URL, so this tile is how they are surfaced.
|
||||
- Crafty:
|
||||
href: https://crafty.${BASE_DOMAIN}
|
||||
description: Minecraft server manager (Keycloak-gated)
|
||||
icon: mdi-minecraft
|
||||
|
||||
- Network & Infra:
|
||||
- Pi-hole:
|
||||
|
||||
@@ -328,12 +328,25 @@ spec:
|
||||
labelsFromPath:
|
||||
cron: [spec, schedule]
|
||||
phase: [status, phase]
|
||||
# MEMORY RAISED 2026-09-13 after 91 OOMKills. Measured, not guessed: working set 111 MiB
|
||||
# with a 6h peak of 123.3 MiB against the old 128Mi limit — KSM was dying on every spike,
|
||||
# roughly every 11 minutes (exit 137). That matters beyond the restarts: THIRTEEN alert
|
||||
# rules read kube_* series, so each kill made them intermittently inert, and an alert that
|
||||
# is only sometimes able to fire is the quiet version of the inert-alert class this repo
|
||||
# keeps getting bitten by (KubePodNotReady and KubeDeploymentReplicasMismatch were both
|
||||
# stuck pending on it). The trigger was enabling customResourceState below for the Velero
|
||||
# Schedule metrics — it adds a CRD informer plus the mandatory CRD list/watch, and that
|
||||
# marginal cost crossed a limit already sitting ~4 MiB above the working set.
|
||||
# 384Mi is ~3x the observed peak and costs nothing here: the nodes report 18.7-22.3 GiB
|
||||
# available. Re-measure with
|
||||
# max_over_time(container_memory_working_set_bytes{container="kube-state-metrics"}[6h])
|
||||
# before trimming it back.
|
||||
resources:
|
||||
requests:
|
||||
cpu: 20m
|
||||
memory: 64Mi
|
||||
memory: 192Mi
|
||||
limits:
|
||||
memory: 128Mi
|
||||
memory: 384Mi
|
||||
prometheus-node-exporter:
|
||||
resources:
|
||||
requests:
|
||||
|
||||
Reference in new issue
Block a user