roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.
TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
* which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
(k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
* which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
Follows the k3s_api_host convention already in this repo. No per-run flag.
DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.
BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).
FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.
KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.
Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.
Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Re-authors the fix stranded on origin/fix/act-runner-dind-readiness (242951a), which
applied cleanly but had never been merged.
The act_runner startup waited only for the dind TLS CERT FILE (/certs/client/ca.pem),
but dockerd writes the certs before it begins accepting TLS connections on :2376. A
runner proceeding on the file alone still hard-exits with "cannot ping the docker daemon
at tcp://localhost:2376" — observed crashlooping 11 times in ~8 minutes on a fresh pod
after a node bounce. Adds a second bounded wait (nc -z -w1 127.0.0.1 2376, 60s cap)
mirroring the existing cert-file loop, so the runner starts only once the daemon
actually answers. busybox nc in the Alpine-based gitea/act_runner:0.2.11 supports -z/-w.
This Deployment is the ENFORCED MERGE GATE for the whole repo, so it ships alone.
A REPORTED "latent envsubst bug" in this file was INVESTIGATED AND DISPROVEN. The claim
was that Flux envsubst expands bare `$i`, making the existing `[ "$i" -ge 60 ]` bound a
silent no-op. Running the real `flux envsubst` (v2.4.0) shows bare `$i` is NOT
substituted — even with i set in the environment, `$i` passes through literally while
`${i}` expands. This matches the repo's own contract in clusters/base/apps-vms.yaml,
which was itself written after empirical probing: "`$VAR` -> left alone by envsubst =>
bare shell vars need NO escaping." The existing 60s bound was always real. NO escaping
was applied: `$${i}` would be an ACTIVE REGRESSION, correct only on the envsubst path —
on a plain `kustomize build`/`kubectl apply -k`, sh reads `$$` as its own PID and the
bound compares garbage against 60. Bare is correct on both paths. A comment now records
the real contract so nobody "fixes" the bare form into escaping.
NEAR MISS worth recording: the first draft of that comment spelled the dangerous forms
out literally, and `flux envsubst` HARD-FAILED on it — "unable to parse variable name".
Flux's envsubst runs over the entire manifest INCLUDING YAML COMMENTS, so a braced name
containing punctuation is a fatal parse error that would have failed the whole
apps-wave-1 Kustomization — breaking every merge in the repo. From a comment. The
shipped comment names the forms in words; the file now contains no braced-dollar
sequence at all.
Verified without a cluster: flux envsubst over the file is byte-identical (exit 0, and
--strict 0); post-envsubst YAML parses with containers unchanged; `sh -n` clean; a
dry-run with a stubbed failing nc fires the stage-2 bound and exits 1; render==golden
green (gitea-actions.yaml is an EXTRA_UNITS entry — render.py places the filename into a
wave and never emits its bytes, so it is not among the 10 compared goldens).
Follow-up documented in-file, deliberately NOT done here: convert dind to an
initContainers sidecar (restartPolicy: Always) with a startupProbe that execs `docker
info` over TLS — NOT a bare tcpSocket probe, which would reopen the same "port open but
certs not ready" gap stage 1 closes — periodSeconds 1 / failureThreshold 60 to preserve
the cap, after which runner drops its command: override. k3s v1.36 has GA sidecars, so
this is the correct long-term shape; it ships separately because a failure in the merge
gate must be unambiguous.
Serves: R-GITOPS-01, R-TEST-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on
['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored;
the coverage rule was NOT narrowed.
=== THE FALSE GREEN (root cause) ===
`make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated
nothing, and EVERY way of failing to find a base degraded silently to an empty touched
set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing
three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on
`main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs
base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's
--require gate never runs at all. Green by vacuum.
Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to
resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base /
failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable
reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED
decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main
runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It
also now prints the diff base it used. A stale local main warns (widens, never falsely
greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites.
Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8
throwaway-repo scenarios including the exact false-green case, an already-merged branch,
no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and
origin/main-preferred-over-stale-main.
=== NEW L0 STATIC LAYER (conformance/tests/l0-static/) ===
Where docs/14 actually puts these invariants. Each check.sh really executes offline via
`run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds.
R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only
PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...]
values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries
a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is
tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces
and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists —
a .example must be referenced by no kustomization and hold only placeholders, and the
Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against
7 injected violations, including deleting the negative row and weakening Enforce->Audit.
R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no
escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to
cluster-admin or a wildcard role; the conformance runner's own test identity is scoped
and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class
policies Enforce and each carry a negative row. Proven against 6 injected violations.
R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL
ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code
in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1,
unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green),
unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is
load-bearing. The real spec repo is never written to.
Each check documents what it does NOT cover (runtime decryption, git history, runtime
least-privilege, branch protection) rather than overclaiming. status.py now reads
check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation.
Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 —
no sixth scaffold was added.
Serves: R-SEC-01, R-SEC-02, R-TEST-01..06.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator ask: implement the roles needed to check prod node hardware health, and run
them to find out our state. This is that role; the run's findings are below.
roles/node_health is a read-only sibling to cluster_verify (cluster health vs hardware
health), styled after node_baseline. One unprivileged collector script does the work in
a single SSH round trip — reading /sys/class/dmi, thermal + hwmon, EDAC, cpufreq +
thermal_throttle, /proc/pressure, /sys/block incl. SCSI ioerr/iotmo counters, lsblk,
lscpu, mounts + statvfs, per-NIC stats, resolv.conf.
THE DESIGN POINT: an unreadable field is emitted as null WITH A RECORDED REASON, never
as zero. The summary ends with a "NOT READ (and why) — these are UNKNOWNS, not passes"
section. A fake zero on a reallocated-sector count is exactly the kind of false green
this project refuses. 22 checks report PASS/WARN/FAIL/SKIP/INFO; every SKIP says why.
Report-only by default; thresholds live in defaults/main.yml.
Portable across bare metal AND VMs — validated against k3s-prod-1 (bare metal, no root)
and k3s-dev-1 (VM, root). VM-absent things (SMART on virtio, coretemp, EDAC) skip
cleanly, and iSCSI/Longhorn volumes are split out of the physical-disk set so a live
replica is never mistaken for a spare device.
playbooks/node-health.yml takes its target group as a variable (-e node_health_hosts=
k3s_prod), deliberately unlike the other playbooks that hardcode k3s_servers — see the
group-name gap already flagged in hosts.ini.
Two become facts found the hard way and documented in the role: a group_vars
`ansible_become: true` OUTRANKS `become: false` as a play/task keyword, so unprivileged
tasks must override the variable; and `failed_when: false` does NOT catch "Missing sudo
password" — only `ignore_errors` does.
FIRST RUN, k3s-prod-1, 2026-08-31: 0 FAIL · 3 WARN · 12 PASS · 3 SKIP · 4 INFO.
Box is healthy, cool and unstressed: i7-9700T 8C/8T, VT-x + /dev/kvm (KubeVirt OK), zero
throttle events, hottest sensor 33C, PSI 0.00 across the board, 31.1 GiB RAM, eno1 up at
1000 Mb/s, BIOS R21 02.26.00, no reboot pending.
DRIVE HEALTH IS STILL UNKNOWN — the headline. SMART did not run, blocked TWICE over: no
vaulted ansible_become_password for prod, AND smartmontools is not on the node at all
(node_baseline has never run there). Power-on hours, wear, reallocated/pending sectors
and drive temperature all remain unmeasured on a ~2011 consumer SSD.
WARNs: kernel ioerr_cnt=6 on sda (read cautiously — four freshly attached iSCSI disks on
dev each showed a flat 7, suggesting a fixed per-device discovery cost, not media
trouble; attribution needs dmesg/root); 0.95 GiB swap still active (node_baseline does
the swapoff); one IPv4 resolver only (the other two upstreams are IPv6 Google), which
refines rather than resolves the DNS-SPOF finding.
ESCALATION — ECC status is UNKNOWABLE on this hardware: /sys/devices/system/edac/mc
exists but no controller is bound. Non-ECC consumer RAM. Silent memory corruption on a
prod node would be undetectable. Recorded as a SKIP with reason, not a pass.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator directive: conformance and gates are enforced at the Gitea runner / CI-CD
level — use it. The authority on mergeability is the Actions run, not a local
`make -C ci gate`; local runs are a fast pre-check only.
Proven immediately on this branch. Same commit (a1fb31a):
local -> "no changed files vs base; empty touched-requirement set" -> MERGE GATE: PASS
CI -> 17 touched requirements -> REQUIRE: FAILED, uncovered
['R-SEC-01','R-SEC-02','R-TEST-06'] -> MERGE GATE: BLOCKED
The local run resolved an empty diff base, checked nothing, and reported green. A local
false green is this project's most dangerous failure mode — it hid real coverage debt.
Fix in progress so local agrees with CI or announces loudly that it cannot determine a
base; an honest noisy skip beats a quiet false pass (ci/README.md exit-code contract).
The gate was RIGHT: we authored a vaulted-secret flow (R-SEC-01) and status tooling
reasoning about spec immutability (R-TEST-06) while those requirements had NO covering
test. Tests are being authored; the rule is NOT being narrowed.
Also records how to read CI from a workstation — the API paths are non-obvious on this
Gitea version (actions/runs/<run>/jobs 404s; use actions/tasks + actions/jobs/<id>/logs,
noting the task-id -> job-id offset).
OPERATOR ATTESTATIONS: DR is proven (Velero -> Proxmox server, restore demonstrated) and
dev-cluster resilience is proven (slow only because Longhorn is on HDD; etcd is on its
dedicated SSD). Both recorded as fact. Kept distinct from automated proof: the status
report's R-STATE-03 finding is TEST debt, not capability debt — the system does the
thing, the suite does not yet assert it. The attestation does not close the requirement
and the gate must not be softened to make it appear closed.
Serves: R-TEST-03, R-TEST-05, R-TEST-06, R-STATE-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PROD-PROVISIONING.md §4 now carries the full operator-decided address table (nodes
.10/.11/.12, control-plane VIP .9, ingress Gateway VIP .20, Cilium LB pool .20-.30) with
the two invariants that make it safe: the control-plane VIP sits outside the LB pool so
kube-vip and LB-IPAM cannot contend, and GATEWAY_VIP is the FIRST address of the pool
because the shared Gateway pins nothing and LB-IPAM hands it the first free address.
§5 no longer tells the operator to hand-install smartmontools — package installation is
the node_baseline role's job (operator ruling). §0 gains the node 2/3 addresses.
Task: T-7.1. Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator ruling: "I care about the state of the repository and what the status of each
spec is based on that state" — not a hand-maintained task board. This is that report.
conformance/lib/status.py (stdlib-only, --selftest) sits beside lib/coverage.py and
IMPORTS it rather than forking the registry/test parsers. Driven by a new
`conformance/run.sh --status`; artifacts land in _artifacts/status/ alongside coverage
and are git-ignored by the existing convention. Exit codes follow ci/lib.sh (0/1/3);
--strict-coverage also applies the status hygiene gate.
Signals derived per requirement: is there an implementing artifact; is it RECONCILED
(walk resources: from clusters/dev, expand each Flux Kustomization's spec.path, recurse
— so a commented-out resource correctly does not count); is there a tagged test and is
its body executable or a TODO scaffold (TODO markers OUTRANK an executable: "true"
annotation — ambiguity never upgrades a row).
DELIBERATELY NOT DERIVED, and reported as such rather than defaulting to green:
proof-of-pass (nothing tracked in git records a run result — _artifacts/ and results/
are git-ignored) and cluster liveness. "Reconciled" means Flux WOULD apply it, not that
it is healthy. PROVEN is reported as "not derivable" for all 51.
WHAT IT SHOWS THAT coverage.md DOES NOT. Coverage reports 37/51 covered, 0
safety-critical gaps. Three of those claims do not survive repo state:
1. 5 of the 37 "covered" requirements have only a TODO-body scaffold (R-STATE-03,
R-OBS-02, R-OBS-04, R-GITOPS-03, R-GITOPS-02). Coverage counts a tagged file; it
never looks inside it.
2. R-STATE-03 is SAFETY-CRITICAL and its L3 fault test asserts nothing. Coverage reports
"0 safety-critical gaps" only because dr-restore/chainsaw-test.yaml exists and carries
an L3 label. --strict now fails on this.
3. R-UPD-01/02/03 are NOT reconciled: platform/image-automation/ is commented out of
platform/kustomization.yaml. Coverage shows them covered with executable L3 tests,
which reads as delivered; automated updates are actually 0/3 wired+testable.
Headline: 51 requirements | reconciled 20 (+12 partial) | executable test 32 |
wired+testable 22 (+8 partial) | scaffold-only 5 | PROVEN 0 (not derivable).
Also surfaces 18 requirements with no reconciled manifest citing their R-ID — listed as
a traceability gap (R-TEST-04) rather than guessed either way. Annotating those manifests
makes the report self-correct, so it acts as a forcing function instead of another
hand-maintained map.
Serves: R-TEST-01..06. Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Prod inventory scaffolding for the three bare-metal nodes, and the become posture the
operator ruled for production.
BECOME: prod does NOT get dev's NOPASSWD sudo. Prod keeps a real sudo boundary and
supplies the password to Ansible as an ansible-vault-encrypted `ansible_become_password`
in inventory/group_vars/k3s_prod/vault.yml, mirroring the existing hypervisor and
proxmox_hosts groups. The decrypting passphrase stays outside the repo
(~/.config/home-k3s/ansible-vault-pass), so plays still run unattended with no plaintext
secret in git. Committed here: README.md + vault.yml.example only — the operator creates
the encrypted vault (`ansible-vault create`), as with every other vaulted group.
PACKAGES BELONG IN THE ROLE (operator ruling): smartmontools is now declared in
node_baseline_packages, not hand-installed on a node. Drive health tooling is a
permanent part of a bare-metal node's baseline — dev nodes are VMs and never needed it.
Same click-ops-never principle as the rest of the repo.
ADDRESSING: nodes .10/.11/.12, reached DIRECTLY on the LAN (no pfSense NAT hop, unlike
dev's 192.168.2.102:225x mapping), so ansible_host is the real node IP. kube-vip
control-plane VIP is 192.168.2.9 — deliberately BELOW the nodes and OUTSIDE the Cilium
LB pool (.20-.30) so kube-vip and LB-IPAM can never contend for an address. The ingress
Gateway VIP (.20) and the LB pool are Flux/cluster-vars concerns and deliberately do
not appear in Ansible.
BLOCKED-BY-DESIGN: node_baseline_longhorn_device / node_baseline_etcd_device are present
but set to an obviously invalid /dev/BLOCKED-... path pending the operator's disk-layout
decision (node 1 is a single SATA SSD; the role's /dev/vdb,/dev/vdc defaults are virtio
names that do not exist on bare metal). Leaving them unset would silently fall back to
those equally-nonexistent defaults — a confusing failure that looks like a wiring bug.
This fails just as hard and says exactly why.
KNOWN FOLLOW-UP: roles/k3s_server and every node-host playbook hardcode the literal
group names k3s_primary/k3s_secondary/k3s_servers, so the k3s_prod_* groups are not
selected by them as-is. Harmless now (nothing can accidentally run against prod) but the
"same roles, unchanged" reuse R-ENV-02 promises needs the target group parameterized
before §6 of PROD-PROVISIONING.md can execute. Documented in hosts.ini and both READMEs.
Serves: R-ENV-01, R-ENV-03, R-SEC-01. Task: T-7.1. Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).
New cluster-vars knobs (base default = dev literal, per the file's existing convention):
K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml
CILIUM_LB_POOL_STOP "
UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml
CRAFTY_L4_VIP platform/crafty/service-l4.yaml
S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
semaphore}-db.yaml, and platform/velero/helmrelease.yaml
Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.
clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.
PROD ADDRESS PLAN (operator, 2026-08-31):
nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
Cilium LB pool .20-.30
CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.
Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.
DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.
Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two operator corrections:
1. PROD DOES NOT GET NOPASSWD SUDO. My initial checklist recommended mirroring dev's
passwordless sudo onto prod — rejected, correctly. Dev's NOPASSWD is a convenience
that must not propagate to production. Prod instead keeps a real sudo boundary and
drives Ansible `become` with an ansible-vault-encrypted `ansible_become_password` in
inventory/group_vars/k3s_prod/vault.yml, mirroring the EXISTING hypervisor and
proxmox_hosts groups. The decrypting passphrase stays outside the repo at
~/.config/home-k3s/ansible-vault-pass (ansible.cfg :: vault_password_file), so plays
still run unattended with no plaintext secret in git.
2. THE TASK BOARD IS NOT THE STATUS SURFACE — THE REPOSITORY IS. Operator ruling closes
the escalation raised in the previous commit. Requirement/spec status is to be DERIVED
from repo state (what is authored, reconciled, and covered by a green requirement-tagged
test), not from a hand-maintained board. The spec board is left as a historical seed;
no agent edits it and no SPEC_PIN churn is spent on it. ci/spec-immutability.sh keeps
its full strictness — no path exception was added, since weakening an R-TEST-06 gate
for a bookkeeping file would be a defect, not a fix. A repo-derived spec status report
becomes the artifact of record alongside the coverage report and this log.
Serves: R-ENV-03 (prod provisioning posture), R-SEC-01 (no plaintext secrets), R-TEST-06.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The operator added the first production node (k3s-prod-1, 192.168.2.10 — HP EliteDesk
800 G5 Mini, i7-9700T/31GiB, Debian 13, VT-x + /dev/kvm) and opened roadmap Phase 7 for
PREP WORK ONLY: no k3s install until the R-ENV-02 blockers close and the gate is green.
Records the Phase-7 entry-gate position honestly: the operator has declared hardware
ready, but "Phases 1-6 proven in dev/test" is NOT met (37/51 conformance; R-ENV-01/02/03
themselves uncovered). Proceeding is an explicit, accepted risk — not an oversight.
Adds infra/PROD-PROVISIONING.md (T-7.1, design-only): firmware/BIOS, disk layout, OS,
network, access, Ansible reuse, repo blockers, build sequence, Phase-7 exit criteria.
Key findings recorded in DECISIONS-LOG:
- R-ENV-02 is violated in the base — dev LAN addresses hardcoded into base manifests
(cilium k8sServiceHost, the Cilium LB pool, crafty/UniFi LB IPs, 3 S3 backup
endpoints). Prod is on 192.168.2.x, so prod would build WRONG today.
- clusters/prod never sets GATEWAY_VIP -> would inherit the base default 10.73.0.240,
an address on the RETIRED libvirt network.
- No NOPASSWD sudo on the prod node -> all Ansible fails at its first become.
- node_baseline device defaults are virtio (/dev/vdb,vdc) and absent on bare metal.
- ESCALATION: agents/task-board.md is structurally unmaintainable — AGENTS.md designates
it the one agent-editable surface, but ci/spec-immutability.sh fails the gate on ANY
spec-repo dirt with no path exception. Explains the all-todo board after six phases.
NOT worked around: exempting it would weaken an R-TEST-06 gate. Operator decision.
Operator decisions: Phase 7 open (prep only); wait for all 3 nodes before cluster-init;
check drive health before choosing the single-SSD disk layout.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03. Task: T-7.1.
Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Appends the definitive current-state/handoff entry to DECISIONS-LOG (SSO fixed via CVE-2025-30162
egress allow, 5 forward-auth UIs gated, 8 vCPU, Gitea Actions CI live, external-snapshotter vendored;
operational gotchas; prioritized Next Steps → Longhorn-SSD, L2/L3-CI, forward-auth long-term,
Phase-6 onboarding). Updates the reconnect guide (6→8 vCPU) and points new agents at the handoff.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The kustomization pulled two REMOTE github bases (?ref=v8.2.0) that Flux's kustomize-controller
git-fetched on every reconcile; those fetches intermittently exceeded the 27s timeout ->
'kustomize build failed' BuildFailed events -> the whole platform tier stalled whenever GitHub was
slow. Vendored the exact v8.2.0 output to local crds.yaml + snapshot-controller.yaml (built with
kubectl kustomize; images transformer still re-pins the controller to v8.2.0), so the build is now
offline + deterministic. No change to what's deployed. platform/kubevirt has the same latent pattern.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
HAND-OFF 2 (L2/L3 conformance wiring) — DEFERRED with a concrete plan, per the
hand-off's escape hatch (too risky to wire live cluster access from CI autonomously;
do NOT hand CI a broad kubeconfig).
The intended seam (in-cluster Chainsaw Job under its own scoped SA; CI holds only a
narrow trigger credential) is mostly scaffolded but not safe to finish autonomously:
- ci/conformance.sh still runs run.sh locally rather than triggering the in-cluster Job.
- conformance/incluster/chainsaw-job.yaml has no repo-clone init-container and the stock
chainsaw image lacks bash/python3/kubectl/git that run.sh needs.
- No trigger SA/RBAC/credential exists, and the core blocker is a privilege-escalation
design question: any `create jobs` credential in the conformance namespace can create a
Job running as the privileged conformance-runner SA. This needs an admission constraint
(recommended: narrow trigger SA + a Kyverno policy pinning the admissible Job shape), or
a GitOps-native zero-credential trigger — an operator decision.
- conformance/incluster/ isn't wired into Flux; Gitea vars/secrets aren't provisioned.
docs/plans/l2-l3-conformance-ci-wiring.md captures current state, the security analysis,
the functional gaps, a file-by-file step sequence, validation/rollback, and the open
decisions. gate.yml's conformance placeholder now points to it (still if:false, honest).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
HAND-OFF 1 (make -C ci gate parity).
touched-requirements.sh gained a third scoping rule: after stripping the diff
marker, whole-line YAML/shell `#` comments are dropped BEFORE the R-ID grep. A
requirement cited only in a code-file comment is prose (a REFERENCE), not a
declaration -- the same logic rule 2 already applies to `*.md`. This unblocks the
self-referential trap where editing gate.yml's header (which cites R-TEST-06, a
known-uncovered ID enforced by the spec-immutability STAGE, not a Chainsaw test)
re-attributed R-TEST-06 so the coverage `--require` check false-blocked the merge.
Conservative: only whole-line comments are dropped; an R-ID on a real code/spec
line is still detected, and authoritative declarations never live in a bare comment.
gate.yml now invokes the aggregate `make -C ci gate` (ci/gate.sh) instead of the
individual stage scripts, so CI runs byte-for-byte the same gate a human runs in the
devcontainer (R-TEST-02) and applies the R-TEST-03 decision. A BLOCKED gate exits
non-zero and fails the job. The cluster-gated L2/L3 stage SKIPS honestly (no
CLUSTER_AVAILABLE). spec-immutability is retained as an explicit early UI step.
Adds ci/tests/touched-requirements.test.sh (+ `make -C ci test`) asserting rule 3:
comment-only R-IDs (added or removed) are not attributed while real-line R-IDs are.
ci/lib.sh REPO_ROOT is now env-overridable so tests can run the real script against a
throwaway git repo (default unchanged).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Roll the verified Option-B per-app oauth2-proxy sidecar out to the remaining
four forward-auth apps after the pihole canary was confirmed fail-closed
(unauthenticated -> 302 to Keycloak, pod 2/2, clean sidecar OIDC discovery).
Each app's Gateway route now terminates at its oauth2-proxy sidecar (:4180,
--allowed-group=media-users, --upstream=http://127.0.0.1:<uiport>); the
gateway-ingress CiliumNetworkPolicy allows the Gateway to reach ONLY :4180, so
the bare UI is never served un-authenticated (fail-closed; R-IDP-02/06). The
apps' own app/API ports stay in-Service for EAST-WEST callers (arr ->
qbittorrent:8080 / jackett:9117) via the existing reciprocal-ingress allows.
- keycloak: confidential jackett/qbittorrent/radarr/sonarr clients (realm-import
+ realm-clients mirror), provisioned live via kcadm (groups membership mapper).
- secrets: four SOPS/age <app>-oauth2-proxy-credentials, wired into apps/wave-0
and apps/wave-1 by the renderer's sidecar-creds wave rule.
- ci: remove the temporary canary deferral (all five goldens now reproduce).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Close the un-gated-UI security finding for forward-auth apps (R-IDP-02) by
rendering the Option-B per-app oauth2-proxy reverse-proxy sidecar that crafty
proves working. catalog/render.py now emits, for any forward-auth app with an
exposed UI: the oauth2-proxy sidecar (upstream http://127.0.0.1:<uiport>,
:4180, --allowed-group per auth.group), an authproxy :4180 Service port, an
HTTPRoute backendRef -> :4180, and a gateway-ingress CiliumNetworkPolicy that
allows the Gateway to reach ONLY :4180 (fail-closed: the bare UI is never
served un-authenticated). oidc/native/none apps render byte-identical.
This is the CANARY: only pihole's golden carries the sidecar and is verified
fail-closed in-cluster first. jackett/qbittorrent/radarr/sonarr are deferred to
the rollout commit (temporarily listed in ci/render-golden.sh) so their UIs
keep serving exactly as today until their clients+secrets land.
- keycloak: confidential pihole client (realm-import + realm-clients mirror),
provisioned live via kcadm (groups membership mapper, bare names).
- secret: SOPS/age pihole-oauth2-proxy-credentials (client-id/secret/cookie),
wired into apps/wave-0 via the renderer's new sidecar-creds wave rule.
- L1 render goldens regenerated for the sidecar (forward-auth fixtures).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Investigation + phased implementation plan to move the churn-heavy,
latency-sensitive Longhorn volumes (Prometheus, the CNPG Postgres DBs,
Loki, gitea) off the slow/failing HDD 'Storage' pool onto a tagged
SSD-backed Longhorn disk on local-zfs, ending the reboot rebuild storm.
Key finding: the recorded 'SSD does not fit Longhorn' blocker was based
on the 384 GiB provisioned figure; actual working set is ~79 GB
cluster-wide (~26 GB/node) since all bulk media is on NFS, not Longhorn.
Hot tier is ~51 GB actual / 114 GiB provisioned and fits the ~323 GiB
SSD AVAIL with headroom. Read-only investigation; no live storage
touched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The three KubeVirt VM conformance tests (vm-lifecycle, vm-default-deny,
vm-data-survival) carried comments asserting the ALPHA `ImageVolume` feature
gate is DISABLED "on this k3s v1.31 cluster" and that the apiserver therefore
prunes the container-disk `image` volume, breaking the containerDisk boot path.
The cluster was upgraded v1.31 -> v1.36.4. Revalidated the gate live:
- kubernetes_feature_enabled{name="ImageVolume"} = 1 (enabled by default;
stage="" — no longer ALPHA)
- a server-side `kubectl apply --dry-run=server` of a Pod with an `image`
volume RETAINS the volume + mount instead of pruning them.
So the old premise is stale: containerDisk roots would boot on 1.36.4. The
tests never asserted the gate state and never used containerDisk — they boot
from CDI DataVolume roots, which remains the correct choice because it is the
spec-required, rebuild-surviving disk model (docs/06, R-STATE-01); an ephemeral
containerDisk could never satisfy data-survival. Test logic is therefore
unchanged; only the explanatory comments are corrected (v1.31 -> v1.36.4, and
the "containerDisk is broken here" rationale marked stale with the live
evidence).
Merge gate: PASS (lint, l0, render-golden, l1 green).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Research + implementation plan (no live change) on whether Cilium 1.20.1's
native Gateway API ExternalAuth filter (GEP-1494) can replace the un-wired
forward-auth CEC template and Crafty's oauth2-proxy sidecar.
Verdict: DEFER. The filter exists in 1.20.1 code (shipped 1.20.0, PR #45739) but
is EXPERIMENTAL-channel Gateway API; the cluster runs the standard-channel CRDs
(verified: HTTPRoute filters[].type enum lacks ExternalAuth), so it is not even
installable today. Also: fail-open bug (#47178, fixed#47265) in a brand-new
security filter, no 1.20 docs, and the ext_authz hop would hit the same
enforce_policy_on_l7lb hairpin re-evaluation just fixed for Keycloak (68690e5).
Plan is phased/reversible (pihole canary first) and keeps the working sidecar.
Resolves the DECISIONS-LOG open item on ext_authz workaround relevance under 1.20.1.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Steady-state CPU PSI 15-23% + 15-min load 5.6-8.0 on 6 cores (run queue at/over core count) left no
headroom for bursty workloads. Proxmox host has 24 cores at ~4% load, so 3x8 = 24 vCPU is 1:1 with
zero oversubscription. In-place tofu update (cores change is pending until a per-node VM stop/start);
rolled one node at a time to keep etcd quorum. Comment also corrected 16->24 GiB (stale).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The container-workload and egress-class L1 fixtures use auth.mode=forward-auth, so the renderer now
emits the keycloak backend egress allow for them too. Regenerate the committed goldens to match the
canonical render. MERGE GATE: PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
App SSO broke cluster-wide after the Cilium 1.16.5->1.20.1 upgrade: the CVE-2025-30162 fix (in
>=1.16.8/1.17.2) makes the shared Gateway's L7 proxy re-evaluate a pod's egress policy against
Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) when it hairpins to keycloak.home.
'toEntities: world' matches the Gateway VIP at L3/L4 but not the resolved keycloak identity at L7, so
OIDC discovery/token 403'd ('Access denied') for every app with a NetworkPolicy (gitea/nextcloud/
semaphore/headlamp/crafty...); Grafana survived only because monitoring has no NetworkPolicy.
Fix: explicit toEndpoints egress allow to the keycloak backend for SSO apps.
- catalog/render.py: emit it for auth.mode in {oidc,forward-auth}; re-rendered 8 app goldens.
- platform/headlamp + platform/crafty: hand-written policies get the same allow.
Split-horizon design unchanged. Verified live: headlamp-ns -> keycloak discovery 403 -> 200.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The browser-mfa conditional-user-role authenticator used config key 'condition.evaluated.user.role',
which is not a real Keycloak property — the role resolved to null, so Keycloak logged 'Invalid role
name submitted: null' and the REQUIRED conditional-OTP step threw, 403-ing every platform-admins login
(Headlamp, Grafana admin, etc.). Correct key for Keycloak 26 is condUserRole=platform-admin.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds NodeCPUPressureHigh (CPU PSI 'some' > 15% for 15m) and NodeCPULoadHigh (load5/core > 1 for 15m)
to platform-health.nodes, plus a 'Nodes — CPU / Memory Pressure' Grafana dashboard (PSI + load-per-core
+ utilization). These surface the bursty CPU contention that averaged CPU%/kubectl top hides — the
signal behind 'spikes cause blips of delay'. Makes the 6->8 vCPU resize measurable.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Both were pinned at ~98% of their limits (Prometheus 2002/2048Mi, Grafana 499/512Mi) with no
headroom for TSDB compaction / dashboard-render spikes — the classic OOM setups. Nodes are 24Gi at
38-44% used (~14Gi free each; the '~22 GiB ceiling' comment predates the 16->24Gi node bump), so the
extra headroom is comfortable. Grafana request 256->384Mi, Prometheus request unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A node rebuild (tofu -replace / fresh OS) wipes host-level config Flux/Helm don't manage.
site.yml omits kube-vip.yml and vpn.egress.gateway.yml, so a re-provision missed the node-host
OpenVPN on k3s-dev-3 and dropped all egress-class=vpn pod traffic. Document the node-host layers
that must be re-run after any re-provision, and correct the stale author-only banner (these plays
are live).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
chart 7 major bump. Verified chart 7.3.0 renders identically for this SingleBinary/filesystem/
Longhorn config (one loki StatefulSet, no extra read/write/backend/cache pods, no schema errors).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Manifest env-var config (manager/cp_enable/svc_enable/vip_arp/leaderelection) is unchanged across
0.8->1.2, so the bump is image + re-applied auto-deploy manifest.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Grafana 12.x bundled in kps 88 OOMKilled (exit 137) at 256Mi during grafana-apiserver/unified-storage
startup — no fatal log, just SIGKILL. The new Grafana pod never became Ready, so Helm --wait ran the full
20m and rolled back to 65.8.1 every time. 512Mi holds.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 65->88 multi-major upgrade exceeded the default 5m Helm timeout (full operator/KSM/
node-exporter + Prometheus/Alertmanager StatefulSet rollout), so Flux rolled back to 65.8.1
four times and Stalled, blocking the platform kustomization and everything downstream.
CRDs pre-applied server-side at operator v0.93.1.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 k3s nodes re-provisioned to Debian 13 (kernel 6.12) via immutable tofu -replace,
one at a time, primary/leader last (secondary-join to avoid cluster-init split-brain).
- variables.tf image defaults -> debian-13-generic (generic, not genericcloud)
- k3s_server: ensure /etc/rancher/k3s exists (fresh-node re-provision fix)
- gitignore vendored ansible collections (installed for re-provision)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 k3s nodes re-provisioned to Debian 13 (kernel 6.12) via immutable tofu -replace,
one at a time, leader/primary last. variables.tf image defaults -> trixie generic.
k3s_server role: ensure /etc/rancher/k3s exists (fresh-node re-provision fix).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Root cause of a W5 wedge: default 5m Helm timeout on slow-HDD Longhorn upgrades caused
a rollback attempt after the data had already migrated to the new minor, which the old
manager fatally refuses. Give the upgrade room and never auto-rollback (fail forward).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gateway API CRDs bumped v1.2.1->v1.3.0 (required by Cilium 1.18; applied server-side).
Land on current k8s 1.32 before the 1.33 hop.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
