From 0fe12b83325e29a290676944ba5f6a9438122544 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 16:08:23 -0500 Subject: [PATCH 1/5] WIP: L0 test declaration for R-C2-01 / R-C2-02 (no check.sh yet) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit INCOMPLETE — preserved by the PM after the authoring agent died to a server error mid-task (its third attempt; the first two died to a session rate limit). Do not merge. R-C2-01 and R-C2-02 are the only 2 remaining coverage gaps on main (49/51). This is the chainsaw Test declaration only: it quotes both requirements verbatim from the spec, cites docs/12-command-center-ui.md for which surface serves which half (Homepage at the root for R-C2-01, Headlamp for R-C2-02's 'controlled operational actions where safe', Grafana for dashboards/alerts), and argues why the deciding half is L0 rather than L2 — reachability is already covered by l2-behavioral/gateway-ingress-reachable, whereas whether the root surfaces THE SERVICES THE CLUSTER RUNS is a comparison of two sets that are both repository facts. WHAT REMAINS: check.sh does not exist, so this test cannot run at all. Also still open: whether 'a service the root hid' (from the prior art commit 49c160d on feat/coverage-debt-paydown) is still a live defect, whether ci/touched-requirements.sh can attribute R-C2-01/02 at all (a digit in the workstream segment reportedly broke the content scan), and the R-TLS-02 decision summary. Co-Authored-By: Claude Opus 5 --- .../c2-root-surface/chainsaw-test.yaml | 116 ++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 conformance/tests/l0-static/c2-root-surface/chainsaw-test.yaml diff --git a/conformance/tests/l0-static/c2-root-surface/chainsaw-test.yaml b/conformance/tests/l0-static/c2-root-surface/chainsaw-test.yaml new file mode 100644 index 0000000..07773f4 --- /dev/null +++ b/conformance/tests/l0-static/c2-root-surface/chainsaw-test.yaml @@ -0,0 +1,116 @@ +# ============================================================================= +# L0 STATIC — the C2 root surfaces every service the cluster runs, plus platform +# health and controlled actions (R-C2-01, R-C2-02) [EXECUTABLE] +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# THE REQUIREMENTS, VERBATIM (spec docs/00-vision-and-scope.md): +# R-C2-01 "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services +# running in the cluster and links to them." +# R-C2-02 "The C2 UI MUST surface platform health (cluster, storage, GitOps, alerts) and +# provide controlled operational actions where safe." +# Detail: spec docs/12-command-center-ui.md — Homepage at the root is "the primary R-C2-01 +# surface"; Headlamp is "the R-C2-02 'controlled operational actions where safe' surface"; +# Grafana is the dashboards/alerts surface; "never a raw cluster-admin kubeconfig in a browser". +# +# WHY L0, AND NOT L2 +# ------------------ +# The REACHABILITY half of R-C2-01 is already exercised against a live cluster: +# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls +# verifies the certificate it serves. What no live probe can decide is the half the +# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING +# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts: +# what git declares the cluster serves (every in-repo HTTPRoute hostname, every L4 Service +# that publishes a hostname, every catalog ServiceDefinition) against what the dashboard's +# ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides half the +# estate and cannot tell the difference. +# +# It is also the assertion that ROTS, in the direction nothing else notices: onboard a +# service, forget the dashboard, and every cluster test stays green while the operator's +# front door hides a running service. That is not a styling preference — it is exactly the +# R-C2-01 failure mode, and this check found a live instance of it when it was written (see +# the commit message: platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root +# linked to it). +# +# GIVEN the repository as git tracks it, +# WHEN the apex ingress, the dashboard's declared link set, every in-repo HTTPRoute, every +# hostname-publishing L4 Service, the versioned catalog and the platform-health +# surfaces are read and compared, +# THEN — R-C2-01 — +# the bare apex is claimed by exactly one route, that route binds the ONE Gateway +# listener whose hostname matches a bare apex, the apex is a SAN on the certificate +# that listener terminates with, the dashboard is RECONCILED (not merely authored), +# FORWARD every HTTP service the repo routes is linked from the root, FORWARD every +# catalog `kind: container` service with a route is linked, and REVERSE every +# home-zone link the root offers resolves to something this repo actually serves — +# an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4); +# — R-C2-02 — +# both health surfaces docs/12 names are linked AND reconciled, the health DATA exists +# as code (PrometheusRule groups for cluster/nodes, storage, GitOps and cert expiry, +# reconciled by the platform-config tier), and actions are CONTROLLED: the console +# binds NO ClusterRole to its own ServiceAccount, authenticates per-user by native +# OIDC from a named SOPS Secret, and cluster rights arrive through OIDC GROUPS at more +# than one privilege level. +# +# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git ls-files + PyYAML — +# the same dependency catalog/render.py already requires). conformance/run.sh --layer l0 and +# ci/l0.sh run it directly; this Chainsaw Test wraps the SAME script. +# +# WHAT IT DOES NOT PROVE — honest gaps, restated in check.sh's own header: +# - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2; +# the Gateway/TLS half is covered under R-DNS-01 / R-TLS-01. +# - That the C2 is access-CONTROLLED in fact. The auth assertions read a DECLARED posture, +# because the Gateway ext-authz attachment that would enforce forward-auth is not built +# anywhere in this repo (platform/homepage/httproute.yaml says so in its own header, and +# every forward-auth app shares the gap). Declared, not enforced — and labelled as such +# in the assertion text. +# - That an operational action in the console succeeds, or that it is audited. The +# assertions prove there is no shared cluster-admin identity to act with and that rights +# arrive per-user; the audit trail is a runtime property. +# - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up +# (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: c2-root-surface + labels: + conformance.home.k3s/R-C2-01: "true" + conformance.home.k3s/R-C2-02: "true" + annotations: + conformance.home.k3s/requirements: "R-C2-01, R-C2-02" + conformance.home.k3s/layer: "L0" + # WHOLE-REPO SCOPE, and it is the whole point of the forward assertion. + # + # The invariant is "every service the repo routes is linked from the C2 root". Its input + # surface is therefore EVERY tracked manifest that can declare an HTTPRoute — apps/wave-*, + # platform/*, runbooks/ — not just the two components that own the C2. ci/touched- + # requirements.sh keys `platform/homepage/*` and `platform/headlamp/*` to nothing at all + # today (there is no path-map arm for either), and even with one, the change that BREAKS + # this invariant is the change that adds a route somewhere else: `apps/wave-1/newthing.yaml` + # attributes R-GITOPS-02 / R-NET-* / nothing, never R-C2-01. Requirement-scoped selection + # would filter this check out of exactly the change that introduces the bug — the R-TEST-03 + # blind spot recorded in conformance/lib/common.sh. + # + # Declaring the scope is the whole fix: conformance/run.sh --include-scope whole-repo + # (ci/l0.sh Stage 3b) unions this test into EVERY gate invocation. Do NOT hand-add a stage + # to ci/l0.sh — tests/l0-static/whole-repo-invariants-wired fails the gate if anyone does. + # A SKIP of a scope-declaring check is rc 3 (UNWIRED), never green. + conformance.home.k3s/scope: "whole-repo" + conformance.home.k3s/safety-critical: "false" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + The bare apex is bound to the landing dashboard over trusted TLS and is reconciled; the + dashboard links every HTTP service this repo routes and every link it offers resolves to + something the repo serves; and the C2 surfaces platform health backed by real alert rules + while granting operational rights per-user through OIDC groups rather than through a + shared cluster-admin identity. +spec: + steps: + - name: compare-the-c2-root-against-what-the-repo-actually-serves + description: >- + Read the apex ingress chain, the dashboard's declared links, every in-repo HTTPRoute and + hostname-publishing Service, the catalog, the platform-health alert rules and the + console's RBAC posture, and assert the two sets agree in both directions. + try: + - script: + content: bash ./check.sh From f57c41195341657a071ebf95c6cd89ecfdf13aeb Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 16:20:53 -0500 Subject: [PATCH 2/5] =?UTF-8?q?test(conformance):=20L0=20covers=20R-C2-01?= =?UTF-8?q?=20/=20R-C2-02=20=E2=80=94=2049=20assertions,=20and=20it=20is?= =?UTF-8?q?=20RED?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Test DECLARATION landed in the preceding WIP commit with no check.sh, so the test could not run at all. This is the check: 49 offline assertions over the repository as git tracks it, no cluster, no network, no spec repo (R-TEST-06). THE RULE. Every home-zone hostname the repository routes over HTTP must be linked from the apex dashboard's configuration; every home-zone link the dashboard offers must resolve to something the repository actually serves (an HTTPRoute at L7 or a hostname-publishing LoadBalancer Service at L4); and the two R-C2-02 surfaces must be linked, reconciled and backed by real alert rules, with operational rights arriving per-user through OIDC groups rather than through a shared cluster-admin identity. IT IS RED ON MAIN, on the forward direction: FAIL (8) every service the repo ROUTES over HTTP is surfaced on the C2 root. Unsurfaced: crafty.${BASE_DOMAIN} (platform/crafty) platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} through the shared Gateway, clusters/base/crafty.yaml reconciles it, and nothing on the C2 root links to it. R-C2-01 says the root MUST surface "the services running in the cluster and links to them", so an unsurfaced route is a requirement gap, not a styling preference. Fixed in the next commit, deliberately separate so the RED is in the history. WHAT IS ASSERTED BEYOND THE PRIOR ATTEMPT (49c160d on feat/coverage-debt-paydown, whose tree is ~107 commits behind and was NOT merged): (0) THE INPUT SURFACE ITSELF. The forward direction is only as good as the set of manifests it reads, so three assertions guard the guard: the scanned roots are present, every manifest in them PARSES (an unparseable file is a service the check cannot see), and EVERY Flux-reconciled path lies inside a scanned root — so a future tier reconciling, say, ./services fails loudly instead of letting the dashboard hide all of it. conformance/ is deliberately excluded from the scan: its fixtures and goldens declare synthetic HTTPRoutes (paas-demo, gw-declared- conformance) that are test data, not services the cluster serves. The prior attempt excluded them only by accident, through a filename prefilter. (5) postBuild.substituteFrom ON THE DASHBOARD'S TIER. Hostnames are ${BASE_DOMAIN} tokens now, not the literals the prior attempt compared. Without substitution the apex route binds the literal host `${BASE_DOMAIN}` (matching no listener) and every tile links to a host that does not exist — while Flux still reports the Kustomization Ready. It fails OPEN, which is this repo's most-repeated incident shape. (12) TOKENS, NEVER LITERALS. No href may name a literal `home.keithjackson.dev` host, so prod's front door cannot link dev's services — the 2026-09-04 class (two clusters authored to describe the same names, prod's external-dns then deleting dev's records), applied to the C2. (2) THE SECTIONNAME IS PROVEN LOAD-BEARING rather than asserted from prose: the wildcard listeners are shown to exist AND to be distinct from the apex listener, which is why `*.${BASE_DOMAIN}` cannot serve the bare apex. (10) THE REVERSE DIRECTION NOW ACCEPTS L4. The UniFi tile points at :11443 on a hostname published by apps/vms/unifi/50-service-l4.yaml (a KubeVirt VM behind a pinned Cilium LoadBalancer VIP). That is something this repo serves, so it resolves — no off-cluster declaration is needed for it any more. The declaration hatch is kept and constrained (specific home-zone hostnames, may not shadow a hostname the repo does serve) but is EMPTY today, which is the honest state. WHOLE-REPO SCOPE, as the declaration says. The invariant quantifies over every tracked manifest that can declare an HTTPRoute; the change that BREAKS it is a route added somewhere else entirely (`apps/wave-1/newthing.yaml` attributes R-GITOPS-02 or nothing, never R-C2-01), and ci/touched-requirements.sh has no path-map arm for platform/homepage or platform/headlamp at all. Requirement-scoped selection would filter this check out of exactly the change that introduces the bug — the R-TEST-03 blind spot. No ci/l0.sh stage is added: Stage 3b's --include-scope union selects it, and whole-repo-invariants-wired fails the gate if anyone hand-adds one. HONEST GAPS, in the check's own header: it does not fetch the apex (L2, covered for the Gateway/TLS half under R-DNS-01 / R-TLS-01); (19) reads a DECLARED auth posture because no Gateway ext-authz attachment exists anywhere in this repo, and says so in the assertion text; it does not judge whether a tile is useful; and an L4-ONLY endpoint (crafty-mc, a Minecraft game port) is reported as an ADVISORY rather than required, because a raw TCP/UDP endpoint has no URL a browser can follow. Co-Authored-By: Claude Opus 5 --- .../tests/l0-static/c2-root-surface/check.sh | 656 ++++++++++++++++++ 1 file changed, 656 insertions(+) create mode 100755 conformance/tests/l0-static/c2-root-surface/check.sh diff --git a/conformance/tests/l0-static/c2-root-surface/check.sh b/conformance/tests/l0-static/c2-root-surface/check.sh new file mode 100755 index 0000000..63ef2fe --- /dev/null +++ b/conformance/tests/l0-static/c2-root-surface/check.sh @@ -0,0 +1,656 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — c2-root-surface +# +# Serves (spec docs/00-vision-and-scope.md; detail in docs/12-command-center-ui.md): +# +# R-C2-01 "The root `home.keithjackson.dev` MUST serve a UI that surfaces the +# services running in the cluster and links to them." +# R-C2-02 "The C2 UI MUST surface platform health (cluster, storage, GitOps, +# alerts) and provide controlled operational actions where safe." +# +# WHY THIS IS AN L0 STATIC CHECK AND NOT AN L2 PROBE +# -------------------------------------------------- +# The REACHABILITY half of R-C2-01 is already exercised against a live cluster: +# l2-behavioral/gateway-ingress-reachable dials the Gateway and l2-behavioral/trusted-tls +# verifies the certificate it serves. What no live probe can decide is the half the +# requirement is actually about — whether the UI at the root surfaces THE SERVICES RUNNING +# IN THE CLUSTER. Deciding that is a comparison of two SETS, and both are repository facts: +# what git declares the cluster serves (every in-repo HTTPRoute hostname, every +# hostname-publishing L4 Service, every catalog ServiceDefinition) against what the +# dashboard's ConfigMap links. A probe of the root gets HTTP 200 from a dashboard that hides +# half the estate and cannot tell the difference. +# +# It is also the assertion that ROTS, in the direction nothing else notices: onboard a +# service, forget the dashboard, and every cluster test stays green while the operator's +# front door hides a running service. That is not a styling preference — it is exactly the +# R-C2-01 failure mode, and it was a LIVE defect on main when this check was written +# (platform/crafty served crafty.${BASE_DOMAIN} and nothing on the root linked to it). +# +# THE RULE THIS IMPLEMENTS, in one sentence: every home-zone hostname the repository routes +# over HTTP must be linked from the apex dashboard's configuration, every home-zone link the +# apex dashboard offers must resolve to something the repository actually serves, and the two +# R-C2-02 surfaces (health data + a per-user-authorized console) must be linked, reconciled +# and backed by real alert rules. +# +# GIVEN the repository as git tracks it, +# WHEN the apex ingress chain, the dashboard's declared link set, every in-repo HTTPRoute, +# every hostname-publishing L4 Service, the versioned catalog, the platform-health +# alert rules and the console's RBAC posture are read and compared, +# THEN — R-C2-01 — +# (0) the INPUT SURFACE is intact: every Flux-reconciled path lies inside the roots +# this check scans, and every manifest in them parses. A root this check does not +# scan, or a file it cannot parse, is a service it cannot see — which would +# silently weaken every assertion below; +# (1) EXACTLY ONE HTTPRoute claims the bare apex (an ambiguous root is not a root), +# and it belongs to the landing-dashboard component; +# (2) that route pins by sectionName the ONE Gateway listener whose hostname is the +# bare apex, and that listener is HTTPS — the wildcard `*.${BASE_DOMAIN}` listener +# does NOT intersect the bare apex, so without the sectionName the root binds +# nothing and serves nothing (asserted against the wildcard listener's existence, +# so the claim is proven rather than asserted from prose); +# (3) the apex is a SAN on the Certificate whose Secret that listener terminates with; +# (4) the dashboard is RECONCILED, not merely authored: one Flux Kustomization points +# at its path and is listed in the base entrypoint set; +# (5) that Kustomization carries postBuild.substituteFrom, because the dashboard's +# hrefs and its route's hostname are written as `${BASE_DOMAIN}` tokens — without +# substitution the apex route binds a literal `${BASE_DOMAIN}` and EVERY tile on +# the root links to a nonexistent host, while Flux still reports Ready; +# (6) the component's own kustomization ships config + workload + Service + route; +# (7) the dashboard is a service CATALOG: grouped entries carrying hrefs; +# (8) FORWARD — every home-zone hostname an in-repo HTTPRoute serves, other than the +# apex itself, is LINKED from the root. This is the rot-catcher; +# (9) FORWARD — every catalog `kind: container` ServiceDefinition whose declared +# ingress host is routed is likewise linked, tying the C2 to the VERSIONED +# onboarding contract and not only to whatever is rendered today; +# (10) REVERSE — every home-zone link the root offers resolves to something this repo +# actually serves: an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service +# (L4). A dead tile is a link the root offers to nothing; +# (11) the off-cluster escape hatch is not a blanket one: every name declared in the +# ConfigMap's `off-cluster-links` annotation is a specific home-zone hostname and +# none of them shadows a hostname the repo DOES serve; +# (12) no href names a LITERAL base-domain host. Every hostname is the `${BASE_DOMAIN}` +# token, so prod's root cannot link dev's services — the 2026-09-04 incident class +# (two clusters authored to describe the same names), applied to the front door; +# — R-C2-02 — +# (13) both health surfaces docs/12 names are linked: the Kubernetes console +# (cluster/storage/GitOps state) and the dashboards/alerts surface; +# (14) both are RECONCILED by Flux, so the links lead to something running; +# (15) the health DATA those surfaces render exists as code — PrometheusRule groups for +# each axis R-C2-02 enumerates (cluster/nodes, storage, GitOps reconciliation) +# plus certificate expiry — and those rule files are reconciled; +# (16) actions are CONTROLLED, not cluster-admin-in-a-browser (docs/12: "never a raw +# cluster-admin kubeconfig in a browser"): the console chart is explicitly told NOT +# to bind its ServiceAccount to a ClusterRole, so there is no shared privileged +# identity to act as; +# (17) actions are instead PER-USER: the console is configured for native OIDC with +# credentials from a named (SOPS) Secret, not inline; +# (18) cluster rights arrive through OIDC GROUPS at MORE THAN ONE privilege level (an +# admin group and a lesser one) — least-privilege, not one tier for everybody; +# (19) the C2 root DECLARES the authenticated-group posture docs/12 requires, on its own +# route (declared, NOT enforced — see the gap note below). +# +# WHAT THIS DOES *NOT* PROVE — honest gaps, the same ones the Test declaration names: +# - That the apex answers HTTP 200, or that any tile's status ping is green. That is L2; the +# Gateway/TLS half is covered under R-DNS-01 / R-TLS-01. +# - That the C2 is access-CONTROLLED in fact. (19) reads a DECLARED posture, because the +# Gateway ext-authz attachment that would ENFORCE forward-auth is not built anywhere in +# this repo (platform/homepage/httproute.yaml says so in its own header, and every +# forward-auth app shares the gap). Declared, not enforced, and labelled as such in the +# assertion text itself. +# - That an operational action in the console succeeds, or that it is audited. (16)-(18) +# prove there is no shared cluster-admin identity to act with and that rights arrive +# per-user; the audit trail is a runtime property. +# - Whether a tile is USEFUL. Homepage service-widget API keys are a deliberate follow-up +# (platform/homepage/configmap.yaml); link presence is what R-C2-01 asks for. +# - That an L4-ONLY endpoint is surfaced. A raw TCP/UDP endpoint (a Minecraft game port) has +# no URL a browser can follow, so the FORWARD direction quantifies over HTTP routes; +# L4-published hostnames are listed as ADVISORIES and are honoured in the REVERSE +# direction (that is how the UniFi tile, an L4 LoadBalancer on :11443, resolves). +# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06) and the +# requirement text above is quoted, not parsed. +# +# STATUS: EXECUTABLE. Pure offline: `git ls-files` + PyYAML (the dependency ten neighbouring +# l0-static checks already take). No cluster, no network, no spec repo. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +PYTHON="${PYTHON:-python3}" + +# Honest SKIP (never green, and rc 3 is UNWIRED for a scope-declaring test) if the landing +# dashboard component is absent entirely — there is then no C2 root to audit, which is a +# report about the tree, not a pass. +if [ ! -f "${ROOT}/platform/homepage/configmap.yaml" ]; then + echo "check SKIPPED: platform/homepage/configmap.yaml absent — no C2 landing dashboard to audit" >&2 + exit 3 +fi + +cd "${ROOT}" +"${PYTHON}" - "${ROOT}" <<'PY' +"""Static audit: the C2 root surfaces every service the repository serves, plus health. + +Every parse contract is documented at its parser. PyYAML rather than regex because the +subjects are genuinely structured: multi-document apps/ bundles, and a Homepage config that +is nested YAML *inside* a ConfigMap string value. +""" +from __future__ import annotations + +import re +import subprocess +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("check FAIL: PyYAML is required (same dependency as catalog/render.py)", + file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +sys.dont_write_bytecode = True # keep __pycache__ out of the tracked tree + +FAILURES: list[str] = [] +ADVISORIES: list[str] = [] +CHECKS = 0 + + +def check(cond, msg): + global CHECKS + CHECKS += 1 + if cond: + print(f" ok {msg}") + else: + print(f" FAIL {msg}", file=sys.stderr) + FAILURES.append(msg) + + +def read(p: Path) -> str: + try: + return p.read_text() + except (OSError, UnicodeDecodeError): + return "" + + +# ============================================================ the hostname vocabulary +# BASE_DOMAIN is a cluster-vars knob (R-ENV-02): every public hostname the cluster serves is +# `.${BASE_DOMAIN}`, and dev/prod differ ONLY in that value. So a hostname in this repo +# is canonically the TOKEN form, and the literal value below exists only to be FORBIDDEN in +# (12): a literal is how prod came to be authored to serve dev's hostnames (2026-09-04, +# clusters/base/cluster-vars.yaml's own header). +TOKEN = "${BASE_DOMAIN}" +_vars = yaml.safe_load(read(ROOT / "clusters" / "base" / "cluster-vars.yaml")) or {} +LITERAL_BASE = ((_vars.get("data") or {}).get("BASE_DOMAIN") or "").strip() + + +def strip_port(host: str) -> str: + return host.split(":", 1)[0] + + +def zone_label(host: str): + """The sub-label of a home-zone hostname, "" for the bare apex, or None if not in zone. + + Accepts only the TOKEN form; a literal-base hostname returns None here and is caught + separately by (12), so a literal can never silently satisfy a set comparison. + """ + h = strip_port(host) + if h == TOKEN: + return "" + if h.endswith("." + TOKEN): + return h[: -len("." + TOKEN)] + return None + + +def is_literal_base(host: str) -> bool: + h = strip_port(host) + return bool(LITERAL_BASE) and (h == LITERAL_BASE or h.endswith("." + LITERAL_BASE)) + + +# ============================================================ (0) the input surface +# DELIVERY ROOTS — the subtrees Flux reconciles, plus catalog/ (the versioned source the +# apps/ manifests are rendered FROM). conformance/ is deliberately EXCLUDED: its fixtures and +# goldens declare synthetic HTTPRoutes (`paas-demo.…`, `gw-declared-conformance.…`) that are +# test data, not services the cluster serves, and counting them would make (8) assert that +# the operator's dashboard links this harness's fixtures. +DELIVERY_ROOTS = ("apps", "platform", "platform-config", "platform-config-cdi", + "runbooks", "clusters", "bootstrap", "catalog") + + +def tracked_yaml() -> list[Path]: + out = subprocess.run(["git", "ls-files", "-z", "--", "*.yaml", "*.yml"], + cwd=ROOT, capture_output=True, text=True, check=True) + return [ROOT / p for p in out.stdout.split("\0") if p] + + +ALL_TRACKED = tracked_yaml() +DELIVERY = [p for p in ALL_TRACKED + if p.relative_to(ROOT).parts[0] in DELIVERY_ROOTS] + +parse_failed: list[str] = [] +DOCS: dict[Path, list[dict]] = {} +for p in DELIVERY: + try: + DOCS[p] = [d for d in yaml.safe_load_all(read(p)) if isinstance(d, dict)] + except yaml.YAMLError: + DOCS[p] = [] + parse_failed.append(str(p.relative_to(ROOT))) + + +def docs_all(): + for p, ds in DOCS.items(): + for d in ds: + yield p, d + + +check(len(DELIVERY) >= 100, + f"(0) the delivery tree is present and scanned ({len(DELIVERY)} tracked manifest(s) " + f"under {'/, '.join(DELIVERY_ROOTS)}/)") +check(not parse_failed, + f"(0) every manifest in the scanned roots PARSES — an unparseable one is a service " + f"this check cannot see, so it would weaken every set comparison below " + f"{parse_failed if parse_failed else ''}") + +# Every Flux-reconciled path must lie inside a root this check scans. This is the guard on +# the guard: if a future tier reconciles, say, ./services, the forward direction would stop +# seeing its routes and would go quietly green on a dashboard that hides all of them. +flux_ks = [(p, d) for p, d in docs_all() + if d.get("kind") == "Kustomization" + and str(d.get("apiVersion", "")).startswith("kustomize.toolkit.fluxcd.io")] +# A RECONCILING Kustomization is one carrying a sourceRef. The pathless documents under +# bootstrap/flux/patches/ are strategic-merge PATCH FRAGMENTS (they add a decryption stanza to +# a Kustomization declared elsewhere); they name no source and reconcile nothing, so requiring +# a path of them would fail on a file that is correct. +reconcilers = [(p, d) for p, d in flux_ks if (d.get("spec", {}) or {}).get("sourceRef")] +unscanned = sorted({ + f"{p.relative_to(ROOT)}:{(d.get('spec', {}) or {}).get('path', '') or '(no path)'}" + for p, d in reconcilers + if (d.get("spec", {}) or {}).get("path", "").lstrip("./").split("/")[0] + not in DELIVERY_ROOTS +}) +check(len(reconcilers) >= 10 and not unscanned, + f"(0) all {len(reconcilers)} Flux-reconciled path(s) lie inside a scanned root — a " + f"reconciled tree this check does not read is a service it cannot require a link for " + f"{unscanned if unscanned else ''}") + +# ============================================================ what the repo SERVES +# L7: every hostname an in-repo HTTPRoute claims, and the component that owns it. +routes: dict[str, list[tuple[Path, dict]]] = {} +apex_inheriting: list[Path] = [] +for p, d in docs_all(): + if d.get("kind") != "HTTPRoute": + continue + hosts = (d.get("spec", {}) or {}).get("hostnames") or [] + if not hosts: + # An HTTPRoute with NO hostnames inherits the hostname of every listener it attaches + # to (runbooks/serving/httproute.yaml binds `https-apex` only, so it serves a PATH on + # the apex). It introduces no new hostname, so it owes no tile. + apex_inheriting.append(p) + continue + for h in hosts: + routes.setdefault(str(h), []).append((p, d)) + +routed_labels = {} # sub-label -> owning component dir +for h, owners in routes.items(): + lbl = zone_label(h) + if lbl: # "" is the apex; None is out of zone + routed_labels[lbl] = owners[0][0].parent + +# L4: a Service that publishes a home-zone hostname to DNS is also "served by this repo" — +# it is how UniFi (a KubeVirt VM behind a pinned Cilium LoadBalancer VIP on :11443) reaches +# the LAN. external-dns's annotation IS the declaration; there is no HTTPRoute to find. +DNS_ANN = "external-dns.alpha.kubernetes.io/hostname" +l4_labels: dict[str, Path] = {} +for p, d in docs_all(): + if d.get("kind") != "Service": + continue + ann = (d.get("metadata", {}) or {}).get("annotations", {}) or {} + for h in re.split(r"[,\s]+", str(ann.get(DNS_ANN, ""))): + lbl = zone_label(h) if h else None + if lbl: + l4_labels[lbl] = p + +check(len(routed_labels) >= 10, + f"(0) the repo's north-south HTTP surface is discovered " + f"({len(routed_labels)} routed home-zone hostname(s), " + f"{len(apex_inheriting)} apex-path route(s), {len(l4_labels)} L4-published hostname(s))") + +# ============================================================ (1) the apex is unambiguous +apex_claims = routes.get(TOKEN, []) +check(len(apex_claims) == 1, + f"(1) exactly ONE HTTPRoute claims the bare apex {TOKEN} — an ambiguous root is not a " + f"root (found {len(apex_claims)}: {[str(p.relative_to(ROOT)) for p, _ in apex_claims]})") +apex_path, apex_route = (apex_claims[0] if apex_claims else (None, {})) +DASH = apex_path.parent if apex_path is not None else None +check(DASH is not None and DASH.name == "homepage", + f"(1) the apex route belongs to the landing-dashboard component " + f"(found {DASH.name if DASH else None!r})") + +# ============================================================ (2) it binds the apex listener +parents = (apex_route.get("spec", {}) or {}).get("parentRefs") or [] +section = next((pr.get("sectionName") for pr in parents if pr.get("sectionName")), None) +gw_name = next((pr.get("name") for pr in parents), None) +gw_ns = next((pr.get("namespace") for pr in parents), None) +check(section is not None, + "(2) the apex route pins a Gateway listener by sectionName — the wildcard listener's " + "hostname cannot match a bare apex, so an unpinned apex route binds NOTHING") + +gw = next((d for _, d in docs_all() + if d.get("kind") == "Gateway" + and (d.get("metadata", {}) or {}).get("name") == gw_name + and (gw_ns is None or (d.get("metadata", {}) or {}).get("namespace") == gw_ns)), None) +check(gw is not None, f"(2) the parent Gateway '{gw_name}' is declared in the repo") +listeners = ((gw or {}).get("spec", {}) or {}).get("listeners") or [] +apex_listener = next((l for l in listeners if l.get("name") == section), None) +check(apex_listener is not None, + f"(2) the Gateway declares the '{section}' listener the apex route binds") +check(apex_listener is not None and apex_listener.get("hostname") == TOKEN, + f"(2) that listener's hostname is EXACTLY the bare apex " + f"(found {(apex_listener or {}).get('hostname')!r})") +check(apex_listener is not None and apex_listener.get("protocol") == "HTTPS", + "(2) the apex listener terminates HTTPS (docs/12 acceptance gate: trusted TLS on the root)") +# Prove the sectionName is LOAD-BEARING rather than decorative: a wildcard listener exists, +# it is a DIFFERENT listener, and its hostname does not equal the apex. +wildcard = [l for l in listeners if str(l.get("hostname", "")) == "*." + TOKEN] +check(bool(wildcard) and all(l.get("name") != section for l in wildcard), + f"(2) the wildcard listener(s) {[l.get('name') for l in wildcard]} are DISTINCT from " + f"'{section}' — which is why the apex needs its own listener, and why dropping the " + f"sectionName would leave the root unserved") + +# ============================================================ (3) trusted TLS on the root +cert_secret = next((ref.get("name") + for ref in ((apex_listener or {}).get("tls", {}) or {}).get("certificateRefs") or []), + None) +cert = next((d for _, d in docs_all() + if d.get("kind") == "Certificate" + and (d.get("spec", {}) or {}).get("secretName") == cert_secret), None) +check(cert is not None, + f"(3) a cert-manager Certificate produces the apex listener's Secret ({cert_secret!r})") +check(cert is not None and TOKEN in ((cert.get("spec", {}) or {}).get("dnsNames") or []), + "(3) the bare apex is a SAN on that Certificate — the wildcard alone does NOT cover it, " + "so without this SAN the root serves an untrusted certificate") + +# ============================================================ (4)(5) it is RECONCILED +dash_rel = f"./{DASH.relative_to(ROOT)}" if DASH else None +dash_ks = [(p, d) for p, d in flux_ks + if (d.get("spec", {}) or {}).get("path", "").rstrip("/") == dash_rel] +check(len(dash_ks) == 1, + f"(4) exactly one Flux Kustomization reconciles the dashboard ({dash_rel}) — authored " + f"but unreconciled is the 'looks done, is absent' class this repo keeps hitting " + f"(found {len(dash_ks)})") +if dash_ks: + base_txt = read(ROOT / "clusters" / "base" / "kustomization.yaml") + entry = dash_ks[0][0].name + check(re.search(rf"^\s*-\s*{re.escape(entry)}\s*(#.*)?$", base_txt, re.M) is not None, + f"(4) that Kustomization is listed in the base entrypoint set ({entry})") + # (5) The dashboard's hrefs and its route's hostname are ${BASE_DOMAIN} TOKENS. Without + # postBuild.substituteFrom the token reaches the cluster LITERALLY: the apex route binds + # the host `${BASE_DOMAIN}` (matching no listener) and every tile links to a host that + # does not exist — and Flux still reports the Kustomization Ready. It fails OPEN. + subs = (((dash_ks[0][1].get("spec", {}) or {}).get("postBuild") or {}).get("substituteFrom") or []) + check(any((s.get("name") == "cluster-vars") for s in subs), + f"(5) {dash_ks[0][0].name} carries postBuild.substituteFrom cluster-vars. It MUST: " + f"this component's route hostname and EVERY tile href are ${{BASE_DOMAIN}} tokens, " + f"and an unsubstituted token is applied literally while Flux still reports Ready — " + f"a root bound to no listener and a dashboard of dead links, with a green tier") + +# ============================================================ (6) it ships the UI +dash_kust = yaml.safe_load(read(DASH / "kustomization.yaml")) if DASH else None +dash_res = set((dash_kust or {}).get("resources") or []) +for want in ("configmap.yaml", "deployment.yaml", "service.yaml", "httproute.yaml"): + check(want in dash_res, f"(6) the dashboard kustomization ships {want}") + +# ================================================== the dashboard's DECLARED link set +# Homepage's config is nested YAML inside ConfigMap string values (services.yaml, +# bookmarks.yaml). Both are read: a bookmark is a link the root offers just as a tile is. +cm = next((d for d in (DOCS.get(DASH / "configmap.yaml") or []) if d.get("kind") == "ConfigMap"), + {}) if DASH else {} +cm_data = cm.get("data") or {} + + +def embedded(key): + try: + return yaml.safe_load(cm_data.get(key) or "") or [] + except yaml.YAMLError: + return None + + +svc_cfg = embedded("services.yaml") +bm_cfg = embedded("bookmarks.yaml") +check(svc_cfg is not None and bm_cfg is not None, + "(7) the dashboard's embedded services.yaml / bookmarks.yaml parse as YAML — the app " + "mounts these verbatim, so a malformed one is a dashboard that renders nothing") + + +def hrefs(node): + """Every href anywhere in Homepage's nested group/entry structure.""" + if isinstance(node, dict): + for k, v in node.items(): + if k == "href" and isinstance(v, str): + yield v + else: + yield from hrefs(v) + elif isinstance(node, list): + for v in node: + yield from hrefs(v) + + +def host_of(url): + m = re.match(r"^https?://([^/]+)", url or "") + return m.group(1) if m else None + + +tile_urls = list(hrefs(svc_cfg or [])) +all_urls = tile_urls + list(hrefs(bm_cfg or [])) +all_hosts = [h for h in (host_of(u) for u in all_urls) if h] +linked = {lbl for lbl in (zone_label(h) for h in all_hosts) if lbl} +tile_labels = {lbl for lbl in (zone_label(h) for h in (host_of(u) for u in tile_urls) if h) if lbl} + +groups = [g for g in (svc_cfg or []) if isinstance(g, dict)] +check(len(groups) >= 1 and len(tile_labels) >= 5, + f"(7) the root is a grouped service catalog with links " + f"({len(groups)} group(s), {len(tile_labels)} linked home-zone service(s))") + +# ============================================ (8) FORWARD: nothing routed is hidden +unsurfaced = sorted(set(routed_labels) - linked) +check(not unsurfaced, + "(8) every service the repo ROUTES over HTTP is surfaced on the C2 root. Unsurfaced: " + + ", ".join(f"{n}.{TOKEN} ({routed_labels[n].relative_to(ROOT)})" for n in unsurfaced) + if unsurfaced else + f"(8) every service the repo ROUTES over HTTP is surfaced on the C2 root " + f"({len(routed_labels)} hostname(s) compared)") + +# L4-only endpoints are reported, never required: a raw TCP/UDP port has no browsable URL. +for lbl in sorted(set(l4_labels) - linked): + ADVISORIES.append( + f"{lbl}.{TOKEN} is published to DNS by an L4 Service " + f"({l4_labels[lbl].relative_to(ROOT)}) and is not linked from the root. Not a " + f"failure: a raw TCP/UDP endpoint has no URL a browser can follow. Verify the " + f"service it belongs to is reachable through a tile that IS linked") + +# ================================= (9) FORWARD: the VERSIONED onboarding contract +cat_dir = ROOT / "catalog" +cat_seen, cat_missing, cat_unrouted = 0, [], [] +for p in sorted(cat_dir.glob("*.yaml")): + sd = next((d for d in (DOCS.get(p) or [])), None) + if not isinstance(sd, dict) or sd.get("kind") != "container": + continue # kind: vm has no north-south HTTP surface (docs/09) + for r in sd.get("ingressRoutes") or []: + lbl = zone_label(str(r.get("host", ""))) + if not lbl: + continue + if lbl not in routed_labels: + # Declared in the catalog but not rendered into a route: an apps/ render drift + # problem owned by R-GITOPS-02, reported here rather than silently skipped. + cat_unrouted.append(f"{sd.get('name')}:{lbl}") + continue + cat_seen += 1 + if lbl not in linked: + cat_missing.append(f"{sd.get('name')}:{lbl}") +check(cat_seen >= 5, + f"(9) the versioned catalog contributes routed HTTP services ({cat_seen} found) — the " + f"C2 is tied to the onboarding contract, not only to today's rendered output") +check(not cat_missing, + f"(9) every catalog `kind: container` service with a route is linked from the C2 root " + f"{cat_missing if cat_missing else ''}") +for u in cat_unrouted: + ADVISORIES.append(f"catalog declares an ingress host with no rendered HTTPRoute: {u} " + f"(render drift — R-GITOPS-02 owns it, not R-C2-01)") + +# ============================ (10)(11) REVERSE: no link points at nothing +ann = (cm.get("metadata", {}) or {}).get("annotations", {}) or {} +OFF_ANN = "homelab.keithjackson.dev/off-cluster-links" +declared_off_raw = [t for t in re.split(r"[,\s]+", str(ann.get(OFF_ANN, ""))) if t] +declared_off = {zone_label(t) for t in declared_off_raw} +served = set(routed_labels) | set(l4_labels) +dead = sorted(linked - served - declared_off) +check(not dead, + f"(10) every home-zone link the root offers resolves to something this repo serves — " + f"an HTTPRoute (L7) or a hostname-publishing LoadBalancer Service (L4). Dead: " + f"{[f'{d}.{TOKEN}' for d in dead] if dead else ''}") +# The escape hatch is a DECLARATION, not a hole: same doctrine as an explicit `NONE` in +# ci/touched-requirements.sh's path map. Empty today, and these two hold vacuously when it is +# — they exist so it cannot be widened into a blanket without failing. +check(all(zone_label(t) for t in declared_off_raw), + f"(11) every declared off-cluster link is a specific home-zone hostname in TOKEN form " + f"({declared_off_raw})") +shadowed = sorted(declared_off & served) +check(not shadowed, + f"(11) no off-cluster declaration shadows a hostname the repo DOES serve " + f"{shadowed if shadowed else ''}") + +# ============================ (12) tokens, never literals +literal_hosts = sorted({h for h in all_hosts if is_literal_base(h)}) +check(not literal_hosts, + f"(12) no link on the root names a LITERAL '{LITERAL_BASE}' host — every hostname is the " + f"${{BASE_DOMAIN}} token, so prod's front door cannot link dev's services (the " + f"2026-09-04 class: two clusters authored to describe the same names). Literals: " + f"{literal_hosts if literal_hosts else ''}") + +# ===================================================================== +# R-C2-02 — platform health, and controlled operational actions +# ===================================================================== +# docs/12 names the surfaces the C2 composes: Headlamp is "the R-C2-02 'controlled +# operational actions where safe' surface"; Grafana is the dashboards/alerts surface. Each is +# identified by the component that OWNS ITS ROUTE, not by a magic string in the config. +CONSOLE, DASHBOARDS = "headlamp", "grafana" +check(CONSOLE in linked, + f"(13) the C2 root links the Kubernetes console ({CONSOLE}.{TOKEN}) — cluster, storage " + f"and GitOps state, and the R-C2-02 operational-actions surface (docs/12)") +check(DASHBOARDS in linked, + f"(13) the C2 root links the dashboards/alerts surface ({DASHBOARDS}.{TOKEN})") + +AGG = ("\n".join(read(ROOT / t / "kustomization.yaml") + for t in ("platform", "platform-config", "platform-config-cdi"))) +for label, name in (("console", CONSOLE), ("dashboards", DASHBOARDS)): + comp = routed_labels.get(name) + check(comp is not None, f"(14) the {label} surface has an in-repo HTTPRoute") + if comp is None: + continue + rel = f"./{comp.relative_to(ROOT)}" + reconciled = any((d.get("spec", {}) or {}).get("path", "").rstrip("/") == rel + for _, d in flux_ks) + if not reconciled: + # A component may reconcile as a MEMBER of an aggregate tier's kustomize tree + # (platform / platform-config) instead of owning a Flux Kustomization. Both count; + # being in NEITHER is "authored, never applied". + reconciled = (f"\n - {comp.name}" in AGG + or f"../{comp.relative_to(ROOT)}/" in AGG) + check(reconciled, + f"(14) the {label} surface ({comp.name}) is reconciled by Flux, not merely authored " + f"— a link to an unreconciled component is a link to nothing") + +# (15) the health DATA exists as code, for every axis R-C2-02 enumerates. +rule_files = sorted({p for p, d in docs_all() if d.get("kind") == "PrometheusRule"}) +rule_groups = {g.get("name", "") + for p, d in docs_all() if d.get("kind") == "PrometheusRule" + for g in ((d.get("spec", {}) or {}).get("groups") or [])} +for axis, needle in (("cluster/nodes", "nodes"), + ("storage", "longhorn"), + ("GitOps reconciliation", "flux"), + ("certificate expiry", "certificates")): + check(any(needle in g for g in rule_groups), + f"(15) platform-health alert rules exist for {axis} (a '*{needle}*' rule group) — " + f"the C2 cannot surface health that is not measured") +pc_txt = read(ROOT / "platform-config" / "kustomization.yaml") +reconciled_rules = [p for p in rule_files if f"../{p.relative_to(ROOT)}" in pc_txt] +check(len(reconciled_rules) >= 2, + f"(15) those alert rules are RECONCILED — PrometheusRule is a CR of a CRD the platform " + f"tier installs, so the rules are applied by platform-config " + f"({len(reconciled_rules)}/{len(rule_files)} rule file(s) referenced there)") + +# (16)(17) controlled actions: no shared cluster-admin identity; per-user OIDC instead. +console_dir = routed_labels.get(CONSOLE) +hr = next((d for p in (sorted(console_dir.glob("*.yaml")) if console_dir else []) + for d in (DOCS.get(p) or []) if d.get("kind") == "HelmRelease"), None) +check(hr is not None, "(16) the Kubernetes console is deployed by a HelmRelease") +hv = ((hr or {}).get("spec", {}) or {}).get("values", {}) or {} +check((hv.get("clusterRoleBinding") or {}).get("create") is False, + "(16) the console chart is told NOT to bind its ServiceAccount to a ClusterRole. The " + "chart's DEFAULT binds it to cluster-admin, which is exactly the 'raw cluster-admin " + "kubeconfig in a browser' docs/12 forbids — every visitor would act as one shared " + "omnipotent identity") +oidc = ((hv.get("config") or {}).get("oidc") or {}) +check(bool(oidc), + "(17) the console is configured for native OIDC login, so an action carries the " + "operator's OWN identity (docs/12: Headlamp/Grafana use native OIDC)") +check(bool((oidc.get("externalSecret") or {}).get("name")), + "(17) its OIDC client credentials come from a NAMED (SOPS) Secret, not inline in the " + "HelmRelease values — R-SEC-01") + +# (18) rights arrive through GROUPS, at more than one privilege level. +group_bindings = [] +for p, d in docs_all(): + if d.get("kind") not in ("ClusterRoleBinding", "RoleBinding"): + continue + for s in d.get("subjects") or []: + if s.get("kind") == "Group" and str(s.get("name", "")).startswith("oidc:"): + group_bindings.append(((d.get("metadata", {}) or {}).get("name"), + s.get("name"), + (d.get("roleRef") or {}).get("name"))) +check(len(group_bindings) >= 2, + f"(18) cluster rights are bound to OIDC GROUPS, so access follows Keycloak group " + f"membership rather than a per-user grant somebody must remember to revoke " + f"({[(g, r) for _, g, r in group_bindings]})") +check(any(r == "cluster-admin" for _, _, r in group_bindings) + and any(r != "cluster-admin" for _, _, r in group_bindings), + "(18) MORE THAN ONE privilege level exists (an admin group AND a lesser one) — " + "least-privilege, not one tier for everybody (docs/12 access control, R-IDP-06)") + +# (19) the root DECLARES the authenticated-group posture. DECLARED, NOT ENFORCED. +route_ann = (apex_route.get("metadata", {}) or {}).get("annotations", {}) or {} +check(route_ann.get("homelab.keithjackson.dev/auth-mode") in ("forward-auth", "oidc", "oauth2-proxy"), + f"(19) the C2 root's route DECLARES an authentication mode " + f"(found {route_ann.get('homelab.keithjackson.dev/auth-mode')!r}) " + f"[DECLARED, NOT ENFORCED — no Gateway ext-authz attachment exists in this repo]") +check(bool(route_ann.get("homelab.keithjackson.dev/auth-groups")), + "(19) it DECLARES the authorized group(s) — docs/12 requires an authenticated user IN " + "AN AUTHORIZED GROUP, not merely logged in " + "[DECLARED, NOT ENFORCED — see the header's gap note]") + +# ======================================================================== report +print() +print(f"C2 surface inventory (apex {TOKEN}, literal base default '{LITERAL_BASE}'):") +print(f" routed over HTTP : {len(routed_labels)} -> {', '.join(sorted(routed_labels))}") +print(f" L4-published : {len(l4_labels)} -> {', '.join(sorted(l4_labels)) or '(none)'}") +print(f" linked from root : {len(linked)} -> {', '.join(sorted(linked))}") +print(f" apex-path routes : {len(apex_inheriting)} -> " + f"{', '.join(str(p.relative_to(ROOT)) for p in apex_inheriting) or '(none)'}") +print(f" catalog services compared: {cat_seen} off-cluster declarations: " + f"{sorted(declared_off_raw) or '(none)'}") + +if ADVISORIES: + print("\nADVISORIES (not failures):") + for a in ADVISORIES: + print(f" - {a}") + +print() +if not FAILURES: + print(f"check OK: c2-root-surface — {CHECKS} assertions (R-C2-01, R-C2-02)") + raise SystemExit(0) +print(f"CHECK FAIL (c2-root-surface): {len(FAILURES)} failed, " + f"{CHECKS - len(FAILURES)} passed", file=sys.stderr) +for f in FAILURES: + print(f" FAIL {f}", file=sys.stderr) +raise SystemExit(1) +PY From 02c58aae150d5fac4723d91f3ec2f2ffea62d9e3 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 16:21:44 -0500 Subject: [PATCH 3/5] =?UTF-8?q?fix(homepage):=20the=20C2=20root=20hid=20a?= =?UTF-8?q?=20running=20service=20=E2=80=94=20Crafty=20had=20no=20tile?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The R-C2-01 check added in the previous commit is RED on main for a real reason. platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} on the shared Gateway, clusters/base/crafty.yaml reconciles it (dependsOn platform-config, wait:false), the web UI is Keycloak-gated by a per-pod oauth2-proxy sidecar — and nothing on the operator's front door linked to it. The service was running and the root did not admit it existed. R-C2-01: "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services running in the cluster and links to them." An unsurfaced route is a requirement gap, not a missing nicety, and it is the exact failure mode nothing else in the gate notices: every cluster test stays green, the apex answers 200, and the estate is one service larger than the dashboard says. This is the SAME defect the unmerged 49c160d found on 2026-09-07 (feat/coverage-debt- paydown, ~107 commits behind main). That branch was never merged, so the gap survived — re-fixed here against today's tree, in token form (`crafty.${BASE_DOMAIN}`, not the literal hostname that branch predated). Grouped under Media because Crafty manages game servers; Platform would be equally defensible and purely cosmetic. The tile also does duty for the L4 game ports (crafty-mc.${BASE_DOMAIN}, TCP 25565-25567 / UDP 19132-19134), which have no browsable URL of their own — the check reports those as an advisory rather than requiring a link. ALSO: clusters/base/homepage.yaml's postBuild comment claimed "Homepage references no ${VARS} today; the substitution is retained for parity". That is no longer true and the claim is dangerous, because it invites someone to delete the substitution as dead weight. This component now references ${BASE_DOMAIN} in httproute.yaml's apex hostname AND in every tile href; without substitution the token is applied literally, the apex route binds a host matching no listener, every link on the root is dead, and Flux still reports Ready. Comment corrected, and the property is asserted by c2-root-surface (5) so it cannot rot back. Verified: conformance/tests/l0-static/c2-root-surface -> check OK, 49 assertions. Co-Authored-By: Claude Opus 5 --- clusters/base/homepage.yaml | 8 ++++++-- platform/homepage/configmap.yaml | 12 ++++++++++++ 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/clusters/base/homepage.yaml b/clusters/base/homepage.yaml index 716b83d..e55e8ab 100644 --- a/clusters/base/homepage.yaml +++ b/clusters/base/homepage.yaml @@ -34,8 +34,12 @@ spec: kind: GitRepository name: flux-system path: ./platform/homepage - # Homepage references no ${VARS} today; the substitution is retained for parity with the other - # tiers / future overlay knobs. + # LOAD-BEARING, not parity decoration (the comment here used to say the opposite). This component + # references ${BASE_DOMAIN} in TWO places: httproute.yaml's apex hostname, and EVERY tile href in + # configmap.yaml. Without this substitution the token is applied LITERALLY — the apex route binds + # the host `${BASE_DOMAIN}`, which matches no Gateway listener, and every link on the operator's + # front door points at a host that does not exist — while Flux still reports this Kustomization + # Ready. It fails OPEN. Asserted by conformance/tests/l0-static/c2-root-surface (5). postBuild: substituteFrom: - kind: ConfigMap diff --git a/platform/homepage/configmap.yaml b/platform/homepage/configmap.yaml index 8819a10..809e517 100644 --- a/platform/homepage/configmap.yaml +++ b/platform/homepage/configmap.yaml @@ -81,6 +81,18 @@ data: href: https://qbittorrent.${BASE_DOMAIN} description: Download client (VPN egress) icon: mdi-download-network + # Crafty Controller — a running, north-south-exposed web UI (platform/crafty/httproute.yaml, + # reconciled by clusters/base/crafty.yaml) that this dashboard did NOT surface until + # 2026-09-13. R-C2-01 says the root MUST surface "the services running in the cluster and + # links to them", so an unsurfaced route is a requirement gap, not a styling preference — + # conformance/tests/l0-static/c2-root-surface fails on one now. Grouped under Media because + # it manages game servers; regrouping it under Platform would be purely cosmetic. The game + # ports themselves (crafty-mc.${BASE_DOMAIN}, TCP 25565-25567 / UDP 19132-19134) are an L4 + # LoadBalancer with no browsable URL, so this tile is how they are surfaced. + - Crafty: + href: https://crafty.${BASE_DOMAIN} + description: Minecraft server manager (Keycloak-gated) + icon: mdi-minecraft - Network & Infra: - Pi-hole: From a76ae3bdfa2d55c3edb3b3b7465e315e6dcaed39 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 16:25:42 -0500 Subject: [PATCH 4/5] =?UTF-8?q?fix(ci):=20the=20content=20scan=20could=20n?= =?UTF-8?q?ot=20match=20R-C2-01/02=20=E2=80=94=20a=20digit=20in=20the=20wo?= =?UTF-8?q?rkstream=20segment?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Checked as part of closing the C2 coverage gaps, and it was live on main. The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match an ID whose workstream segment carries a digit: `[A-Z]+` consumes the `C` of `R-C2-01` and then demands a `-` where the `2` is. `R-C2-01` and `R-C2-02` are the only two such IDs in conformance/requirements/registry.yaml, and the scan was blind to BOTH. REPRODUCED before the fix, on a throwaway repo whose only change is a manifest declaring both IDs on real (non-comment) lines, in a path the map keys to nothing: $ REPO_ROOT=$T BASE_REF=main ci/touched-requirements.sh R-NET-06 # the all-letter control ID was the ONLY output It failed by UNDER-attributing, which is the fail-OPEN direction: a change that retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02, contributed those IDs to NOBODY, so the C2 tests were filtered out of the merge gate. Same defect class as the map's `*) : ;;` arm — a miss that reads as "nothing touched". Every other consumer in the repo already allowed the digit, which is why this was the lone dissenter: test_requirements_for_path() twenty lines above in this same script, conformance/lib/coverage.py, conformance/lib/status.py and conformance/lib/ run_l1_offline.py all use `R-[A-Z0-9]+-[0-9]+`. Aligned to that form, with the reason recorded at the line. SCOPE OF THE HOLE. path_requirements() has no arm for platform/homepage/* or platform/headlamp/* either, so before this the ONLY route to R-C2-01/02 was detection rule 3 (editing the C2 conformance test itself). No path-map arm is added here: c2-root-surface declares `scope: whole-repo`, so ci/l0.sh Stage 3b's --include-scope union selects it on EVERY gate invocation regardless of the touched set — which is the right mechanism for an invariant whose input surface is the whole tree, and strictly stronger than a two-component path arm would be. REGRESSION-TESTED, in ci/tests/touched-requirements.test.sh (the suite `make -C ci test` runs): a fixture declaring R-C2-01 / R-C2-02 / R-OBS-04 from `notes/` — a path outside path_requirements(), so the IDs can only come from the content scan — asserts both digit IDs ARE attributed, that the all-letter form still is (control), and that a purely numeric workstream is not invented (the pattern must not widen the gate on noise). PROVEN TO DETECT BREAKAGE: with the pattern reverted to `R-[A-Z]+-[0-9]+` the suite reports `FAIL (2 failed, 14 passed)` naming both C2 IDs as missing; restored, all three ci/ suites pass (16 assertions in this one). ALSO: conformance/tests/l0-static/whole-repo-invariants-wired/check.sh cites this pattern in the comment explaining why its BOGUS_REQ fixture is lowercase. Citation updated so the reasoning still reads true; the fixture is unaffected (grep -E is case-sensitive, and the check still passes 26/26). NOTE: another branch may be editing this same script concurrently. This is a one-line change to one statement plus its comment — expect a trivial conflict and keep the `[A-Z0-9]` form. Co-Authored-By: Claude Opus 5 --- ci/tests/touched-requirements.test.sh | 33 +++++++++++++++++++ ci/touched-requirements.sh | 17 +++++++++- .../whole-repo-invariants-wired/check.sh | 2 +- 3 files changed, 50 insertions(+), 2 deletions(-) diff --git a/ci/tests/touched-requirements.test.sh b/ci/tests/touched-requirements.test.sh index 9e906e1..5c0b028 100755 --- a/ci/tests/touched-requirements.test.sh +++ b/ci/tests/touched-requirements.test.sh @@ -136,6 +136,39 @@ printf 'touched set (clusters/ path map):\n%s\n---\n' "$OUT4" assert_present "R-ENV-02" "$OUT4" "clusters/* attributes R-ENV-02 (selects placeholder-guard-coupling HALF A)" assert_present "R-GITOPS-01" "$OUT4" "clusters/* attributes R-GITOPS-01" +# --------------------------------------------------------------------------------------- +# THE ID PATTERN MUST ALLOW A DIGIT IN THE WORKSTREAM SEGMENT (added 2026-09-13) +# +# The content scan's pattern was `R-[A-Z]+-[0-9]+`, which CANNOT match `R-C2-01` or +# `R-C2-02` — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. Those are +# the only two registry IDs whose workstream segment carries a digit, and the scan was blind +# to BOTH, in the fail-OPEN direction: a change self-declaring a C2 requirement contributed +# it to nobody, so the C2 conformance tests were filtered out of the merge gate. Found while +# wiring conformance/tests/l0-static/c2-root-surface, which closes those two coverage gaps. +# +# `notes/` is outside path_requirements() on purpose, so the IDs below can only have come +# from the CONTENT SCAN. A neighbouring all-letter ID is asserted too, so a regex that +# somehow matched only the digit form would still fail here. +git -C "$TMP" checkout -q -b feature5 main +cat > "$TMP/notes/c2.yaml" <<'YAML' +kind: Test +metadata: + labels: + conformance.home.k3s/R-C2-01: "true" + conformance.home.k3s/R-C2-02: "true" + conformance.home.k3s/R-OBS-04: "true" +YAML +git -C "$TMP" add -A && git -C "$TMP" commit -qm "declare R-C2-01 / R-C2-02" +OUT5="$(REPO_ROOT="$TMP" BASE_REF=main "$SCRIPT" 2>/dev/null)" +printf 'touched set (digit in the workstream segment):\n%s\n---\n' "$OUT5" +assert_present "R-C2-01" "$OUT5" "an R-ID with a DIGIT in the workstream segment IS attributed (R-C2-01)" +assert_present "R-C2-02" "$OUT5" "the second such ID is attributed too (R-C2-02)" +assert_present "R-OBS-04" "$OUT5" "an all-letter workstream segment still works (control)" +# The pattern must not become so loose that a non-ID token is read as a requirement: a bare +# numeric workstream is not a registry shape, and attributing one would WIDEN the gate on +# noise (over-broad selection, never a false green, but noisy and misleading). +assert_absent "R-1-01" "$OUT5" "a purely numeric workstream segment is not invented from thin air" + echo if [ "$FAIL" -eq 0 ]; then echo "touched-requirements tests: PASS ($PASS assertions)"; exit 0; else echo "touched-requirements tests: FAIL ($FAIL failed, $PASS passed)"; exit 1; fi diff --git a/ci/touched-requirements.sh b/ci/touched-requirements.sh index 3f07374..8566f40 100755 --- a/ci/touched-requirements.sh +++ b/ci/touched-requirements.sh @@ -345,6 +345,21 @@ main() { done <<< "$files" # Self-declared R-IDs on the ADDED/REMOVED lines of the change (see changed_diff). + # + # THE ID PATTERN IS `R-[A-Z0-9]+-[0-9]+`, NOT `R-[A-Z]+-[0-9]+`. The WORKSTREAM segment may + # contain a DIGIT: `R-C2-01` and `R-C2-02` (command & control UI) are the only two such IDs + # in conformance/requirements/registry.yaml today, and the stricter pattern silently failed + # to match BOTH of them — `[A-Z]+` consumes the `C` and then demands a `-` where the `2` is. + # Every other consumer in the repo already allows the digit: test_requirements_for_path() + # twenty lines above uses `R-[A-Z0-9]+-[0-9]+`, as do conformance/lib/coverage.py, + # conformance/lib/status.py and conformance/lib/run_l1_offline.py. This grep was the lone + # dissenter, and it dissented by UNDER-attributing — the fail-OPEN direction. A change that + # retyped a `conformance.home.k3s/R-C2-01` tag, or a manifest self-declaring R-C2-02 from a + # path the map keys elsewhere, contributed those IDs to NOBODY, so the C2 tests were + # filtered out of the merge gate. Same defect class as the map's `*) : ;;` arm: a miss that + # reads as "nothing touched". path_requirements() has no arm for platform/homepage/* or + # platform/headlamp/* either, so before this fix the ONLY route to R-C2-01/02 was rule 3 + # (editing the C2 test itself). Regression-tested in ci/tests/touched-requirements.test.sh. # `^[+-]` keeps hunk headers (`@@`) and the `+++`/`---` file headers out of scope. # `sed 's/^[+-]//'` then strips the single-char diff marker (safe: -U0 emits one), so # what remains is the SOURCE line, and `grep -vE '^[[:space:]]*#'` drops whole-line @@ -358,7 +373,7 @@ main() { | grep -vE '^(\+\+\+|---)' \ | sed -E 's/^[+-]//' \ | grep -vE '^[[:space:]]*#' \ - | grep -hoE 'R-[A-Z]+-[0-9]+' \ + | grep -hoE 'R-[A-Z0-9]+-[0-9]+' \ | tr '\n' ' ' || true)" # Normalize: split, unique, sort. Emit to stdout. diff --git a/conformance/tests/l0-static/whole-repo-invariants-wired/check.sh b/conformance/tests/l0-static/whole-repo-invariants-wired/check.sh index 4d13f82..4f22a3a 100755 --- a/conformance/tests/l0-static/whole-repo-invariants-wired/check.sh +++ b/conformance/tests/l0-static/whole-repo-invariants-wired/check.sh @@ -85,7 +85,7 @@ done # second would fail. # Deliberately LOWERCASE: conf_normalize_reqs() upper-cases it, so the selection behaves # identically — but the literal here does not match ci/touched-requirements.sh's content-scan -# pattern `R-[A-Z]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement +# pattern `R-[A-Z0-9]+-[0-9]+`, so editing this file does not attribute a nonexistent requirement # to the change. (Same reason the synthetic fixture below carries no requirement label at all.) BOGUS_REQ="r-nosuchreq-99" sel_without="$(bash "$RUN_SH" --layer l0 --requirements "$BOGUS_REQ" --print-selection 2>/dev/null || true)" From aff80d1d7a1d4823707b73921c4c9d26a300c3a8 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 13 Sep 2026 16:26:54 -0500 Subject: [PATCH 5/5] =?UTF-8?q?docs(pm):=20R-TLS-02=20needs=20the=20owner?= =?UTF-8?q?=20=E2=80=94=20the=20drafted=20spec=20amendment,=20recovered=20?= =?UTF-8?q?and=20restated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The R-TLS-02 analysis and its drafted spec amendment were written on feat/coverage-debt-paydown (b37defe, 2026-09-07) and lived ONLY in that branch's copy of DECISIONS-LOG.md. That branch's tree is ~107 commits behind main and must never be merged, so the amendment went out of view with it — the second time this decision has been raised and lost. Recovered, re-checked against today's main, and written where it cannot vanish with a branch. THE SUBSTANCE IS UNCHANGED. R-TLS-02 says certificates MUST be automatically "issued, renewed, and revoked without manual steps". Issuance and renewal hold. Revocation is not implemented and is not implementable with the stack ADR-0006 selects: cert-manager has no revoke-on-delete, no Certificate field requesting revocation, and no controller that calls ACME revokeCert, and neither ADR-0006 nor docs/08 specifies a mechanism. enableCertificateOwnerRef is NOT the remedy — it garbage-collects the Secret, which is cleanup, not revocation — and it is a global controller flag that would drop the Gateway's wildcard-home-tls Secret on any future Certificate rename/prune, so it needs its own risk assessment rather than a side effect of a test task. Still not set. WHAT CHANGED SINCE THE FIRST ESCALATION, and why it now matters more: R-TLS-02 is no longer a COVERAGE gap (l0-static/tls-lifecycle-automated + l2-behavioral/ cert-lifecycle-automated both carry the label and main counts it covered), and the L0 check is honest about the boundary in its own verdict line ("'revoked' is NOT proven at any layer"). But the L2 test deletes a Certificate and requires its Secret to disappear, and its failure message says "set enableCertificateOwnerRef: true ... (owner decision — WS-TEST does not weaken the assertion instead)". That assertion is written to FAIL on a live run until the owner decides. The decision is now arriving as a red test rather than as a question, which is exactly how it gets weakened by whoever meets it next. Two options are recorded verbatim for the architect, in the drafted preference order: amend the requirement so revocation is an explicitly manual, documented break-glass action (which also makes enableCertificateOwnerRef a CONSEQUENCE of the requirement), or keep the text and record a reviewed DOCUMENTED_GAPS entry naming the clause, the reason and the removal condition. NOT APPLIED, and not appliable by an agent: the spec repo is read-only under R-TEST-06. This is a request for a human decision, stated plainly as one. Co-Authored-By: Claude Opus 5 --- DECISIONS-LOG.md | 82 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+) diff --git a/DECISIONS-LOG.md b/DECISIONS-LOG.md index cdf348b..42956e5 100644 --- a/DECISIONS-LOG.md +++ b/DECISIONS-LOG.md @@ -3042,3 +3042,85 @@ shared etcd/Longhorn device plus the I/O-pressure alert that missed the prod-1 o Branch pruning was deliberately NOT done: four agent branches read as "merged, no unique commits" only because their agents had not committed yet, and deleting those would have destroyed live work. + +## R-TLS-02 — the "revoked" clause still needs the OWNER's decision (recovered 2026-09-13) + +**This is a decision only the repo owner/architect can make. It has now been escalated twice and +lost twice, which is why it is written into this log rather than into another branch's commit +message.** The analysis was first drafted on `feat/coverage-debt-paydown` (`b37defe`, 2026-09-07), +a branch whose tree is ~107 commits behind main and which must never be merged; the drafted +amendment lived only in that branch's copy of this file, so it disappeared from view the moment the +branch went stale. Recovered here, re-checked against today's main, and restated unchanged in +substance. + +**R-TLS-02, verbatim (spec `docs/00-vision-and-scope.md`):** *"Certificates MUST be automatically +**issued, renewed, and revoked** without manual steps."* + +Three verbs. Two hold; the third does not, and cannot with the stack ADR-0006 selects: + +- **Issued — satisfied.** Both ClusterIssuers are declarative (ACME DNS-01, Cloudflare token + referenced by name from SOPS), the wildcard Certificate plus its apex SAN make onboarding + certificate-free, and the active issuer is chosen per environment by `${ACME_CLUSTER_ISSUER}`. +- **Renewed — satisfied, and unhobbled.** cert-manager renews at ~2/3 of lifetime with no operator + action; nothing in the repo sets a `duration`/`renewBefore` that would defeat it, and + `CertificateExpiringSoon` / `CertificateExpiryImminent` / `CertificateNotReady` fire if renewal + silently stops. +- **Revoked — NOT implemented, and not implementable here.** cert-manager has no automatic ACME + revocation: no revoke-on-delete, no `Certificate` field that requests revocation, no controller + that calls ACME `revokeCert`. Deleting a `Certificate` stops renewal; it does not revoke, and the + issued certificate stays valid at the CA until it expires. Neither ADR-0006 nor `docs/08` + specifies a mechanism — `docs/08`'s own "Renewal & revocation" section describes replacement + issuance, which is rotation. + +**`enableCertificateOwnerRef` is NOT the remedy, and is still not set.** That cert-manager +controller flag makes a `Certificate` own its `Secret`, so deleting the Certificate +garbage-collects the key material. That is *cleanup*, not revocation. It does remove one real +manual step (today a deleted Certificate leaves an orphaned TLS Secret and private key to be +deleted by hand), so it is defensible on its own merits — but it is a GLOBAL controller flag, and +any future rename or prune that deletes and recreates a `Certificate` would take the Gateway's +`wildcard-home-tls` Secret with it, dropping TLS on every service until reissuance, against Let's +Encrypt rate limits, on two clusters. It deserves its own risk assessment, not a side effect of a +coverage or test task. + +**WHAT HAS CHANGED SINCE THE FIRST ESCALATION.** R-TLS-02 is no longer a COVERAGE gap — main +carries `l0-static/tls-lifecycle-automated` and `l2-behavioral/cert-lifecycle-automated`, and the +coverage report counts it covered. The honesty was kept where it matters: the L0 check's own verdict +line ends *"R-TLS-02 'without manual steps' clause only — issued/renewed/revoked are RUNTIME verbs +(see l2-behavioral/cert-lifecycle-automated; 'revoked' is NOT proven at any layer)"*. But the +decision is now also LOAD-BEARING on the gate in a second way: the L2 test deletes a Certificate and +requires its Secret to disappear, and on failure prints *"set enableCertificateOwnerRef: true in +platform/cert-manager values (owner decision — WS-TEST does not weaken the assertion instead)"*. So +that test is written to FAIL on any live cluster run until the owner decides. Nobody should weaken +it to make a run green; that is the decision below, arriving as a red test instead of as a question. + +**THE TWO OPTIONS, in the drafted preference order. The spec repo is READ-ONLY to every agent +(R-TEST-06), so neither has been applied — this is a request for a human decision.** + +1. **PREFERRED — amend R-TLS-02 to the property that is achievable and actually protective**, and + move revocation to an explicitly manual, documented break-glass procedure: + + > **R-TLS-02** — Certificates MUST be automatically **issued and renewed** without manual steps, + > and decommissioning a service MUST NOT leave its key material behind. Certificate + > **revocation** is an exceptional, operator-initiated action (suspected key compromise); it is + > NOT automated, and MUST have a documented procedure. Rationale: the ACME / Let's Encrypt model + > relies on SHORT certificate lifetimes plus automated rotation rather than on revocation — Let's + > Encrypt retired OCSP in 2025 and cert-manager implements no automatic revocation — so an + > "auto-revoked" requirement cannot be met by the stack ADR-0006 selects, and could only be met + > by bespoke revocation automation the threat model does not justify. + + Under this wording R-TLS-02 becomes fully testable: issuance and renewal are already asserted + statically, and "no key material left behind" is exactly what `enableCertificateOwnerRef` + implements — which makes that flag a *consequence of the requirement*, decided on its own risk + assessment, rather than a fudge to turn a gate green. + +2. **Keep R-TLS-02 as written and accept it as permanently partially satisfied**, with a reviewed + `DOCUMENTED_GAPS` entry in `conformance/run.sh` naming the revocation clause, the reason (no ACME + revocation exists in cert-manager) and the removal condition (cert-manager ships revocation, or + the requirement is amended). Cheaper, and records a requirement the system does not meet. + +**Until the owner picks one, the state is: R-TLS-02 counted as covered, its third clause +unimplemented and labelled as such in the test's own output, and the L2 withdrawal assertion +standing as a deliberate red.** Writing a test tagged R-TLS-02 that silently verifies only issuance +and renewal would report the requirement as met while a third of its normative text is +unimplemented — which is the one thing this harness exists to prevent. The choice is the +architect's, not an agent's.