Records what landed, the numbers (coverage 37->40/51, safety-critical scaffolds 1->0,
PROVEN still 0 by design), the corrections made mid-session so they are not re-litigated
(prod NOPASSWD rejected; the '$i envsubst bug' disproven by running the real tool; the
stranded DNS branch dropped rather than landed), three near-misses worth remembering, the
six items blocked on the operator, and what is knowingly left undone.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.
The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.
MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
* systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
silently bypass the local zone and ad-blocking.
* resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
would destroy PiHole-first ordering.
* contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.
SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.
PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.
tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.
ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.
Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Closes the LAST scaffold-only safety-critical requirement. The operator has attested DR
works (Velero -> Proxmox versitygw S3, restore demonstrated), but the L3 test carried the
right tag and layer label over a 5-line TODO body — so coverage counted it and reported
"0 safety-critical gaps" while the suite asserted nothing. That was test debt, not
capability debt. This makes the suite assert what the operator already demonstrated.
Three steps:
1. assert-backup-target-is-off-cluster (read-only): BSL must be phase Available (Velero
itself reached the store), s3Url host is not localhost/*.svc/*.cluster.local, and the
resolved IPv4 is none of — loopback, any Node Internal/ExternalIP, any Service
ClusterIP, any Pod IP, inside any node podCIDR, inside any ServiceCIDR. A BSL that
exists but never reaches Available FAILS (a live DR defect), it does not skip.
2. assert-backups-are-scheduled: R-STATE-03 says SCHEDULED off-cluster backups — Velero
+ BSL but zero Schedule in phase Enabled is a fail.
3. The drill: seed a nanosecond-unique canary onto a longhorn PVC in its own scratch
namespace; Backup to Completed; assert a PodVolumeBackup for that volume is Completed
WITH a non-empty status.snapshotID (the kopia snapshot of volume DATA reached the
remote repo, not just resource JSON); ANNIHILATE the source (delete ns, poll until
gone, force-delete a lingering PV, FAIL if the original PV survived — a survivor makes
the restore prove nothing); Restore with namespaceMapping into a clean namespace;
assert the restored PVC bound a DIFFERENT PV and that the canary bytes are identical.
Off-cluster proof = endpoint outside every cluster address + BSL Available + kopia
snapshotID present + every in-cluster copy destroyed + restore landing on a new PV. The
bytes could only have come from outside.
Skips GREEN on: no longhorn SC, Velero API absent, no BSL, Velero/node-agent not ready —
all three steps re-derive the same gates so it can never half-run. An RBAC denial is
deliberately NOT a skip: it fails loudly as a harness defect. Teardown is a step-level
`finally` that ALWAYS runs — deletes the Restore, issues a DeleteBackupRequest so objects
leave the bucket (2h ttl as backstop), deletes both scratch namespaces by name and by
label, releases leftover PVs whose claimRef is one of its own. Everything it destroys is
something it created.
conformance/incluster/rbac.yaml: authored the R-STATE-03 grants (velero.io backups/
restores/deletebackuprequests create+delete+read; BSL/schedules/podvolume* read-only;
persistentvolumes get/list/DELETE — the off-cluster proof requires destroying the PV).
Also two pre-existing gaps other tests already depended on: storageclasses get/list (the
skip gate of five tests) and apps deployments/daemonsets read. No write verbs on platform
controllers.
NOT ASSERTED, stated in the file header rather than implied: off-HOST (on dev the nodes
are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor
loss); no direct S3 client read (that would hand a privileged test identity the backup
credential for weaker evidence); real service data (Nextcloud/Gitea/CNPG drills stay
operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC.
status: scaffold-only 5 -> 4; safety-critical w/o executable L3 1 -> 0 (of 6); executable
35 -> 36. coverage unchanged at 40/51 (correct — coverage only asks whether a tagged test
exists). PROVEN stayed 0 / not-derivable: it did NOT flip to proven just because the body
is non-empty.
AUTHORED, NEVER EXECUTED — carries a conformance.home.k3s/never-executed annotation. No
cluster access from this workstation. To convert authored -> proven the operator must, IN
A MAINTENANCE WINDOW (it deletes namespaces and PVs and writes/deletes bucket objects):
apply the updated rbac.yaml, run `./conformance/run.sh --requirements R-STATE-03 --layer
l3`, confirm the log shows DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK and NOT
DR_RESTORE_SKIPPED_OK (green but proves nothing), confirm no leaked *-dr-src/*-dr-dst
namespaces or orphan PVs or *-dr-backup objects, and record the green run.
Serves: R-STATE-03, R-TEST-05.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.
TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
* which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
(k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
* which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
Follows the k3s_api_host convention already in this repo. No per-run flag.
DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.
BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).
FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.
KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.
Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.
Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Re-authors the fix stranded on origin/fix/act-runner-dind-readiness (242951a), which
applied cleanly but had never been merged.
The act_runner startup waited only for the dind TLS CERT FILE (/certs/client/ca.pem),
but dockerd writes the certs before it begins accepting TLS connections on :2376. A
runner proceeding on the file alone still hard-exits with "cannot ping the docker daemon
at tcp://localhost:2376" — observed crashlooping 11 times in ~8 minutes on a fresh pod
after a node bounce. Adds a second bounded wait (nc -z -w1 127.0.0.1 2376, 60s cap)
mirroring the existing cert-file loop, so the runner starts only once the daemon
actually answers. busybox nc in the Alpine-based gitea/act_runner:0.2.11 supports -z/-w.
This Deployment is the ENFORCED MERGE GATE for the whole repo, so it ships alone.
A REPORTED "latent envsubst bug" in this file was INVESTIGATED AND DISPROVEN. The claim
was that Flux envsubst expands bare `$i`, making the existing `[ "$i" -ge 60 ]` bound a
silent no-op. Running the real `flux envsubst` (v2.4.0) shows bare `$i` is NOT
substituted — even with i set in the environment, `$i` passes through literally while
`${i}` expands. This matches the repo's own contract in clusters/base/apps-vms.yaml,
which was itself written after empirical probing: "`$VAR` -> left alone by envsubst =>
bare shell vars need NO escaping." The existing 60s bound was always real. NO escaping
was applied: `$${i}` would be an ACTIVE REGRESSION, correct only on the envsubst path —
on a plain `kustomize build`/`kubectl apply -k`, sh reads `$$` as its own PID and the
bound compares garbage against 60. Bare is correct on both paths. A comment now records
the real contract so nobody "fixes" the bare form into escaping.
NEAR MISS worth recording: the first draft of that comment spelled the dangerous forms
out literally, and `flux envsubst` HARD-FAILED on it — "unable to parse variable name".
Flux's envsubst runs over the entire manifest INCLUDING YAML COMMENTS, so a braced name
containing punctuation is a fatal parse error that would have failed the whole
apps-wave-1 Kustomization — breaking every merge in the repo. From a comment. The
shipped comment names the forms in words; the file now contains no braced-dollar
sequence at all.
Verified without a cluster: flux envsubst over the file is byte-identical (exit 0, and
--strict 0); post-envsubst YAML parses with containers unchanged; `sh -n` clean; a
dry-run with a stubbed failing nc fires the stage-2 bound and exits 1; render==golden
green (gitea-actions.yaml is an EXTRA_UNITS entry — render.py places the filename into a
wave and never emits its bytes, so it is not among the 10 compared goldens).
Follow-up documented in-file, deliberately NOT done here: convert dind to an
initContainers sidecar (restartPolicy: Always) with a startupProbe that execs `docker
info` over TLS — NOT a bare tcpSocket probe, which would reopen the same "port open but
certs not ready" gap stage 1 closes — periodSeconds 1 / failureThreshold 60 to preserve
the cap, after which runner drops its command: override. k3s v1.36 has GA sidecars, so
this is the correct long-term shape; it ships separately because a failure in the merge
gate must be unambiguous.
Serves: R-GITOPS-01, R-TEST-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on
['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored;
the coverage rule was NOT narrowed.
=== THE FALSE GREEN (root cause) ===
`make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated
nothing, and EVERY way of failing to find a base degraded silently to an empty touched
set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing
three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on
`main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs
base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's
--require gate never runs at all. Green by vacuum.
Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to
resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base /
failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable
reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED
decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main
runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It
also now prints the diff base it used. A stale local main warns (widens, never falsely
greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites.
Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8
throwaway-repo scenarios including the exact false-green case, an already-merged branch,
no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and
origin/main-preferred-over-stale-main.
=== NEW L0 STATIC LAYER (conformance/tests/l0-static/) ===
Where docs/14 actually puts these invariants. Each check.sh really executes offline via
`run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds.
R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only
PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...]
values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries
a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is
tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces
and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists —
a .example must be referenced by no kustomization and hold only placeholders, and the
Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against
7 injected violations, including deleting the negative row and weakening Enforce->Audit.
R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no
escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to
cluster-admin or a wildcard role; the conformance runner's own test identity is scoped
and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class
policies Enforce and each carry a negative row. Proven against 6 injected violations.
R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL
ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code
in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1,
unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green),
unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is
load-bearing. The real spec repo is never written to.
Each check documents what it does NOT cover (runtime decryption, git history, runtime
least-privilege, branch protection) rather than overclaiming. status.py now reads
check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation.
Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 —
no sixth scaffold was added.
Serves: R-SEC-01, R-SEC-02, R-TEST-01..06.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator ask: implement the roles needed to check prod node hardware health, and run
them to find out our state. This is that role; the run's findings are below.
roles/node_health is a read-only sibling to cluster_verify (cluster health vs hardware
health), styled after node_baseline. One unprivileged collector script does the work in
a single SSH round trip — reading /sys/class/dmi, thermal + hwmon, EDAC, cpufreq +
thermal_throttle, /proc/pressure, /sys/block incl. SCSI ioerr/iotmo counters, lsblk,
lscpu, mounts + statvfs, per-NIC stats, resolv.conf.
THE DESIGN POINT: an unreadable field is emitted as null WITH A RECORDED REASON, never
as zero. The summary ends with a "NOT READ (and why) — these are UNKNOWNS, not passes"
section. A fake zero on a reallocated-sector count is exactly the kind of false green
this project refuses. 22 checks report PASS/WARN/FAIL/SKIP/INFO; every SKIP says why.
Report-only by default; thresholds live in defaults/main.yml.
Portable across bare metal AND VMs — validated against k3s-prod-1 (bare metal, no root)
and k3s-dev-1 (VM, root). VM-absent things (SMART on virtio, coretemp, EDAC) skip
cleanly, and iSCSI/Longhorn volumes are split out of the physical-disk set so a live
replica is never mistaken for a spare device.
playbooks/node-health.yml takes its target group as a variable (-e node_health_hosts=
k3s_prod), deliberately unlike the other playbooks that hardcode k3s_servers — see the
group-name gap already flagged in hosts.ini.
Two become facts found the hard way and documented in the role: a group_vars
`ansible_become: true` OUTRANKS `become: false` as a play/task keyword, so unprivileged
tasks must override the variable; and `failed_when: false` does NOT catch "Missing sudo
password" — only `ignore_errors` does.
FIRST RUN, k3s-prod-1, 2026-08-31: 0 FAIL · 3 WARN · 12 PASS · 3 SKIP · 4 INFO.
Box is healthy, cool and unstressed: i7-9700T 8C/8T, VT-x + /dev/kvm (KubeVirt OK), zero
throttle events, hottest sensor 33C, PSI 0.00 across the board, 31.1 GiB RAM, eno1 up at
1000 Mb/s, BIOS R21 02.26.00, no reboot pending.
DRIVE HEALTH IS STILL UNKNOWN — the headline. SMART did not run, blocked TWICE over: no
vaulted ansible_become_password for prod, AND smartmontools is not on the node at all
(node_baseline has never run there). Power-on hours, wear, reallocated/pending sectors
and drive temperature all remain unmeasured on a ~2011 consumer SSD.
WARNs: kernel ioerr_cnt=6 on sda (read cautiously — four freshly attached iSCSI disks on
dev each showed a flat 7, suggesting a fixed per-device discovery cost, not media
trouble; attribution needs dmesg/root); 0.95 GiB swap still active (node_baseline does
the swapoff); one IPv4 resolver only (the other two upstreams are IPv6 Google), which
refines rather than resolves the DNS-SPOF finding.
ESCALATION — ECC status is UNKNOWABLE on this hardware: /sys/devices/system/edac/mc
exists but no controller is bound. Non-ECC consumer RAM. Silent memory corruption on a
prod node would be undetectable. Recorded as a SKIP with reason, not a pass.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator directive: conformance and gates are enforced at the Gitea runner / CI-CD
level — use it. The authority on mergeability is the Actions run, not a local
`make -C ci gate`; local runs are a fast pre-check only.
Proven immediately on this branch. Same commit (a1fb31a):
local -> "no changed files vs base; empty touched-requirement set" -> MERGE GATE: PASS
CI -> 17 touched requirements -> REQUIRE: FAILED, uncovered
['R-SEC-01','R-SEC-02','R-TEST-06'] -> MERGE GATE: BLOCKED
The local run resolved an empty diff base, checked nothing, and reported green. A local
false green is this project's most dangerous failure mode — it hid real coverage debt.
Fix in progress so local agrees with CI or announces loudly that it cannot determine a
base; an honest noisy skip beats a quiet false pass (ci/README.md exit-code contract).
The gate was RIGHT: we authored a vaulted-secret flow (R-SEC-01) and status tooling
reasoning about spec immutability (R-TEST-06) while those requirements had NO covering
test. Tests are being authored; the rule is NOT being narrowed.
Also records how to read CI from a workstation — the API paths are non-obvious on this
Gitea version (actions/runs/<run>/jobs 404s; use actions/tasks + actions/jobs/<id>/logs,
noting the task-id -> job-id offset).
OPERATOR ATTESTATIONS: DR is proven (Velero -> Proxmox server, restore demonstrated) and
dev-cluster resilience is proven (slow only because Longhorn is on HDD; etcd is on its
dedicated SSD). Both recorded as fact. Kept distinct from automated proof: the status
report's R-STATE-03 finding is TEST debt, not capability debt — the system does the
thing, the suite does not yet assert it. The attestation does not close the requirement
and the gate must not be softened to make it appear closed.
Serves: R-TEST-03, R-TEST-05, R-TEST-06, R-STATE-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PROD-PROVISIONING.md §4 now carries the full operator-decided address table (nodes
.10/.11/.12, control-plane VIP .9, ingress Gateway VIP .20, Cilium LB pool .20-.30) with
the two invariants that make it safe: the control-plane VIP sits outside the LB pool so
kube-vip and LB-IPAM cannot contend, and GATEWAY_VIP is the FIRST address of the pool
because the shared Gateway pins nothing and LB-IPAM hands it the first free address.
§5 no longer tells the operator to hand-install smartmontools — package installation is
the node_baseline role's job (operator ruling). §0 gains the node 2/3 addresses.
Task: T-7.1. Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator ruling: "I care about the state of the repository and what the status of each
spec is based on that state" — not a hand-maintained task board. This is that report.
conformance/lib/status.py (stdlib-only, --selftest) sits beside lib/coverage.py and
IMPORTS it rather than forking the registry/test parsers. Driven by a new
`conformance/run.sh --status`; artifacts land in _artifacts/status/ alongside coverage
and are git-ignored by the existing convention. Exit codes follow ci/lib.sh (0/1/3);
--strict-coverage also applies the status hygiene gate.
Signals derived per requirement: is there an implementing artifact; is it RECONCILED
(walk resources: from clusters/dev, expand each Flux Kustomization's spec.path, recurse
— so a commented-out resource correctly does not count); is there a tagged test and is
its body executable or a TODO scaffold (TODO markers OUTRANK an executable: "true"
annotation — ambiguity never upgrades a row).
DELIBERATELY NOT DERIVED, and reported as such rather than defaulting to green:
proof-of-pass (nothing tracked in git records a run result — _artifacts/ and results/
are git-ignored) and cluster liveness. "Reconciled" means Flux WOULD apply it, not that
it is healthy. PROVEN is reported as "not derivable" for all 51.
WHAT IT SHOWS THAT coverage.md DOES NOT. Coverage reports 37/51 covered, 0
safety-critical gaps. Three of those claims do not survive repo state:
1. 5 of the 37 "covered" requirements have only a TODO-body scaffold (R-STATE-03,
R-OBS-02, R-OBS-04, R-GITOPS-03, R-GITOPS-02). Coverage counts a tagged file; it
never looks inside it.
2. R-STATE-03 is SAFETY-CRITICAL and its L3 fault test asserts nothing. Coverage reports
"0 safety-critical gaps" only because dr-restore/chainsaw-test.yaml exists and carries
an L3 label. --strict now fails on this.
3. R-UPD-01/02/03 are NOT reconciled: platform/image-automation/ is commented out of
platform/kustomization.yaml. Coverage shows them covered with executable L3 tests,
which reads as delivered; automated updates are actually 0/3 wired+testable.
Headline: 51 requirements | reconciled 20 (+12 partial) | executable test 32 |
wired+testable 22 (+8 partial) | scaffold-only 5 | PROVEN 0 (not derivable).
Also surfaces 18 requirements with no reconciled manifest citing their R-ID — listed as
a traceability gap (R-TEST-04) rather than guessed either way. Annotating those manifests
makes the report self-correct, so it acts as a forcing function instead of another
hand-maintained map.
Serves: R-TEST-01..06. Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Prod inventory scaffolding for the three bare-metal nodes, and the become posture the
operator ruled for production.
BECOME: prod does NOT get dev's NOPASSWD sudo. Prod keeps a real sudo boundary and
supplies the password to Ansible as an ansible-vault-encrypted `ansible_become_password`
in inventory/group_vars/k3s_prod/vault.yml, mirroring the existing hypervisor and
proxmox_hosts groups. The decrypting passphrase stays outside the repo
(~/.config/home-k3s/ansible-vault-pass), so plays still run unattended with no plaintext
secret in git. Committed here: README.md + vault.yml.example only — the operator creates
the encrypted vault (`ansible-vault create`), as with every other vaulted group.
PACKAGES BELONG IN THE ROLE (operator ruling): smartmontools is now declared in
node_baseline_packages, not hand-installed on a node. Drive health tooling is a
permanent part of a bare-metal node's baseline — dev nodes are VMs and never needed it.
Same click-ops-never principle as the rest of the repo.
ADDRESSING: nodes .10/.11/.12, reached DIRECTLY on the LAN (no pfSense NAT hop, unlike
dev's 192.168.2.102:225x mapping), so ansible_host is the real node IP. kube-vip
control-plane VIP is 192.168.2.9 — deliberately BELOW the nodes and OUTSIDE the Cilium
LB pool (.20-.30) so kube-vip and LB-IPAM can never contend for an address. The ingress
Gateway VIP (.20) and the LB pool are Flux/cluster-vars concerns and deliberately do
not appear in Ansible.
BLOCKED-BY-DESIGN: node_baseline_longhorn_device / node_baseline_etcd_device are present
but set to an obviously invalid /dev/BLOCKED-... path pending the operator's disk-layout
decision (node 1 is a single SATA SSD; the role's /dev/vdb,/dev/vdc defaults are virtio
names that do not exist on bare metal). Leaving them unset would silently fall back to
those equally-nonexistent defaults — a confusing failure that looks like a wiring bug.
This fails just as hard and says exactly why.
KNOWN FOLLOW-UP: roles/k3s_server and every node-host playbook hardcode the literal
group names k3s_primary/k3s_secondary/k3s_servers, so the k3s_prod_* groups are not
selected by them as-is. Harmless now (nothing can accidentally run against prod) but the
"same roles, unchanged" reuse R-ENV-02 promises needs the target group parameterized
before §6 of PROD-PROVISIONING.md can execute. Documented in hosts.ini and both READMEs.
Serves: R-ENV-01, R-ENV-03, R-SEC-01. Task: T-7.1. Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).
New cluster-vars knobs (base default = dev literal, per the file's existing convention):
K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml
CILIUM_LB_POOL_STOP "
UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml
CRAFTY_L4_VIP platform/crafty/service-l4.yaml
S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
semaphore}-db.yaml, and platform/velero/helmrelease.yaml
Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.
clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.
PROD ADDRESS PLAN (operator, 2026-08-31):
nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
Cilium LB pool .20-.30
CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.
Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.
DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.
Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two operator corrections:
1. PROD DOES NOT GET NOPASSWD SUDO. My initial checklist recommended mirroring dev's
passwordless sudo onto prod — rejected, correctly. Dev's NOPASSWD is a convenience
that must not propagate to production. Prod instead keeps a real sudo boundary and
drives Ansible `become` with an ansible-vault-encrypted `ansible_become_password` in
inventory/group_vars/k3s_prod/vault.yml, mirroring the EXISTING hypervisor and
proxmox_hosts groups. The decrypting passphrase stays outside the repo at
~/.config/home-k3s/ansible-vault-pass (ansible.cfg :: vault_password_file), so plays
still run unattended with no plaintext secret in git.
2. THE TASK BOARD IS NOT THE STATUS SURFACE — THE REPOSITORY IS. Operator ruling closes
the escalation raised in the previous commit. Requirement/spec status is to be DERIVED
from repo state (what is authored, reconciled, and covered by a green requirement-tagged
test), not from a hand-maintained board. The spec board is left as a historical seed;
no agent edits it and no SPEC_PIN churn is spent on it. ci/spec-immutability.sh keeps
its full strictness — no path exception was added, since weakening an R-TEST-06 gate
for a bookkeeping file would be a defect, not a fix. A repo-derived spec status report
becomes the artifact of record alongside the coverage report and this log.
Serves: R-ENV-03 (prod provisioning posture), R-SEC-01 (no plaintext secrets), R-TEST-06.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The operator added the first production node (k3s-prod-1, 192.168.2.10 — HP EliteDesk
800 G5 Mini, i7-9700T/31GiB, Debian 13, VT-x + /dev/kvm) and opened roadmap Phase 7 for
PREP WORK ONLY: no k3s install until the R-ENV-02 blockers close and the gate is green.
Records the Phase-7 entry-gate position honestly: the operator has declared hardware
ready, but "Phases 1-6 proven in dev/test" is NOT met (37/51 conformance; R-ENV-01/02/03
themselves uncovered). Proceeding is an explicit, accepted risk — not an oversight.
Adds infra/PROD-PROVISIONING.md (T-7.1, design-only): firmware/BIOS, disk layout, OS,
network, access, Ansible reuse, repo blockers, build sequence, Phase-7 exit criteria.
Key findings recorded in DECISIONS-LOG:
- R-ENV-02 is violated in the base — dev LAN addresses hardcoded into base manifests
(cilium k8sServiceHost, the Cilium LB pool, crafty/UniFi LB IPs, 3 S3 backup
endpoints). Prod is on 192.168.2.x, so prod would build WRONG today.
- clusters/prod never sets GATEWAY_VIP -> would inherit the base default 10.73.0.240,
an address on the RETIRED libvirt network.
- No NOPASSWD sudo on the prod node -> all Ansible fails at its first become.
- node_baseline device defaults are virtio (/dev/vdb,vdc) and absent on bare metal.
- ESCALATION: agents/task-board.md is structurally unmaintainable — AGENTS.md designates
it the one agent-editable surface, but ci/spec-immutability.sh fails the gate on ANY
spec-repo dirt with no path exception. Explains the all-todo board after six phases.
NOT worked around: exempting it would weaken an R-TEST-06 gate. Operator decision.
Operator decisions: Phase 7 open (prep only); wait for all 3 nodes before cluster-init;
check drive health before choosing the single-SSD disk layout.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-PLAT-01/03. Task: T-7.1.
Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Appends the definitive current-state/handoff entry to DECISIONS-LOG (SSO fixed via CVE-2025-30162
egress allow, 5 forward-auth UIs gated, 8 vCPU, Gitea Actions CI live, external-snapshotter vendored;
operational gotchas; prioritized Next Steps → Longhorn-SSD, L2/L3-CI, forward-auth long-term,
Phase-6 onboarding). Updates the reconnect guide (6→8 vCPU) and points new agents at the handoff.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The kustomization pulled two REMOTE github bases (?ref=v8.2.0) that Flux's kustomize-controller
git-fetched on every reconcile; those fetches intermittently exceeded the 27s timeout ->
'kustomize build failed' BuildFailed events -> the whole platform tier stalled whenever GitHub was
slow. Vendored the exact v8.2.0 output to local crds.yaml + snapshot-controller.yaml (built with
kubectl kustomize; images transformer still re-pins the controller to v8.2.0), so the build is now
offline + deterministic. No change to what's deployed. platform/kubevirt has the same latent pattern.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
HAND-OFF 2 (L2/L3 conformance wiring) — DEFERRED with a concrete plan, per the
hand-off's escape hatch (too risky to wire live cluster access from CI autonomously;
do NOT hand CI a broad kubeconfig).
The intended seam (in-cluster Chainsaw Job under its own scoped SA; CI holds only a
narrow trigger credential) is mostly scaffolded but not safe to finish autonomously:
- ci/conformance.sh still runs run.sh locally rather than triggering the in-cluster Job.
- conformance/incluster/chainsaw-job.yaml has no repo-clone init-container and the stock
chainsaw image lacks bash/python3/kubectl/git that run.sh needs.
- No trigger SA/RBAC/credential exists, and the core blocker is a privilege-escalation
design question: any `create jobs` credential in the conformance namespace can create a
Job running as the privileged conformance-runner SA. This needs an admission constraint
(recommended: narrow trigger SA + a Kyverno policy pinning the admissible Job shape), or
a GitOps-native zero-credential trigger — an operator decision.
- conformance/incluster/ isn't wired into Flux; Gitea vars/secrets aren't provisioned.
docs/plans/l2-l3-conformance-ci-wiring.md captures current state, the security analysis,
the functional gaps, a file-by-file step sequence, validation/rollback, and the open
decisions. gate.yml's conformance placeholder now points to it (still if:false, honest).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
HAND-OFF 1 (make -C ci gate parity).
touched-requirements.sh gained a third scoping rule: after stripping the diff
marker, whole-line YAML/shell `#` comments are dropped BEFORE the R-ID grep. A
requirement cited only in a code-file comment is prose (a REFERENCE), not a
declaration -- the same logic rule 2 already applies to `*.md`. This unblocks the
self-referential trap where editing gate.yml's header (which cites R-TEST-06, a
known-uncovered ID enforced by the spec-immutability STAGE, not a Chainsaw test)
re-attributed R-TEST-06 so the coverage `--require` check false-blocked the merge.
Conservative: only whole-line comments are dropped; an R-ID on a real code/spec
line is still detected, and authoritative declarations never live in a bare comment.
gate.yml now invokes the aggregate `make -C ci gate` (ci/gate.sh) instead of the
individual stage scripts, so CI runs byte-for-byte the same gate a human runs in the
devcontainer (R-TEST-02) and applies the R-TEST-03 decision. A BLOCKED gate exits
non-zero and fails the job. The cluster-gated L2/L3 stage SKIPS honestly (no
CLUSTER_AVAILABLE). spec-immutability is retained as an explicit early UI step.
Adds ci/tests/touched-requirements.test.sh (+ `make -C ci test`) asserting rule 3:
comment-only R-IDs (added or removed) are not attributed while real-line R-IDs are.
ci/lib.sh REPO_ROOT is now env-overridable so tests can run the real script against a
throwaway git repo (default unchanged).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Roll the verified Option-B per-app oauth2-proxy sidecar out to the remaining
four forward-auth apps after the pihole canary was confirmed fail-closed
(unauthenticated -> 302 to Keycloak, pod 2/2, clean sidecar OIDC discovery).
Each app's Gateway route now terminates at its oauth2-proxy sidecar (:4180,
--allowed-group=media-users, --upstream=http://127.0.0.1:<uiport>); the
gateway-ingress CiliumNetworkPolicy allows the Gateway to reach ONLY :4180, so
the bare UI is never served un-authenticated (fail-closed; R-IDP-02/06). The
apps' own app/API ports stay in-Service for EAST-WEST callers (arr ->
qbittorrent:8080 / jackett:9117) via the existing reciprocal-ingress allows.
- keycloak: confidential jackett/qbittorrent/radarr/sonarr clients (realm-import
+ realm-clients mirror), provisioned live via kcadm (groups membership mapper).
- secrets: four SOPS/age <app>-oauth2-proxy-credentials, wired into apps/wave-0
and apps/wave-1 by the renderer's sidecar-creds wave rule.
- ci: remove the temporary canary deferral (all five goldens now reproduce).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Close the un-gated-UI security finding for forward-auth apps (R-IDP-02) by
rendering the Option-B per-app oauth2-proxy reverse-proxy sidecar that crafty
proves working. catalog/render.py now emits, for any forward-auth app with an
exposed UI: the oauth2-proxy sidecar (upstream http://127.0.0.1:<uiport>,
:4180, --allowed-group per auth.group), an authproxy :4180 Service port, an
HTTPRoute backendRef -> :4180, and a gateway-ingress CiliumNetworkPolicy that
allows the Gateway to reach ONLY :4180 (fail-closed: the bare UI is never
served un-authenticated). oidc/native/none apps render byte-identical.
This is the CANARY: only pihole's golden carries the sidecar and is verified
fail-closed in-cluster first. jackett/qbittorrent/radarr/sonarr are deferred to
the rollout commit (temporarily listed in ci/render-golden.sh) so their UIs
keep serving exactly as today until their clients+secrets land.
- keycloak: confidential pihole client (realm-import + realm-clients mirror),
provisioned live via kcadm (groups membership mapper, bare names).
- secret: SOPS/age pihole-oauth2-proxy-credentials (client-id/secret/cookie),
wired into apps/wave-0 via the renderer's new sidecar-creds wave rule.
- L1 render goldens regenerated for the sidecar (forward-auth fixtures).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Investigation + phased implementation plan to move the churn-heavy,
latency-sensitive Longhorn volumes (Prometheus, the CNPG Postgres DBs,
Loki, gitea) off the slow/failing HDD 'Storage' pool onto a tagged
SSD-backed Longhorn disk on local-zfs, ending the reboot rebuild storm.
Key finding: the recorded 'SSD does not fit Longhorn' blocker was based
on the 384 GiB provisioned figure; actual working set is ~79 GB
cluster-wide (~26 GB/node) since all bulk media is on NFS, not Longhorn.
Hot tier is ~51 GB actual / 114 GiB provisioned and fits the ~323 GiB
SSD AVAIL with headroom. Read-only investigation; no live storage
touched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The three KubeVirt VM conformance tests (vm-lifecycle, vm-default-deny,
vm-data-survival) carried comments asserting the ALPHA `ImageVolume` feature
gate is DISABLED "on this k3s v1.31 cluster" and that the apiserver therefore
prunes the container-disk `image` volume, breaking the containerDisk boot path.
The cluster was upgraded v1.31 -> v1.36.4. Revalidated the gate live:
- kubernetes_feature_enabled{name="ImageVolume"} = 1 (enabled by default;
stage="" — no longer ALPHA)
- a server-side `kubectl apply --dry-run=server` of a Pod with an `image`
volume RETAINS the volume + mount instead of pruning them.
So the old premise is stale: containerDisk roots would boot on 1.36.4. The
tests never asserted the gate state and never used containerDisk — they boot
from CDI DataVolume roots, which remains the correct choice because it is the
spec-required, rebuild-surviving disk model (docs/06, R-STATE-01); an ephemeral
containerDisk could never satisfy data-survival. Test logic is therefore
unchanged; only the explanatory comments are corrected (v1.31 -> v1.36.4, and
the "containerDisk is broken here" rationale marked stale with the live
evidence).
Merge gate: PASS (lint, l0, render-golden, l1 green).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Research + implementation plan (no live change) on whether Cilium 1.20.1's
native Gateway API ExternalAuth filter (GEP-1494) can replace the un-wired
forward-auth CEC template and Crafty's oauth2-proxy sidecar.
Verdict: DEFER. The filter exists in 1.20.1 code (shipped 1.20.0, PR #45739) but
is EXPERIMENTAL-channel Gateway API; the cluster runs the standard-channel CRDs
(verified: HTTPRoute filters[].type enum lacks ExternalAuth), so it is not even
installable today. Also: fail-open bug (#47178, fixed#47265) in a brand-new
security filter, no 1.20 docs, and the ext_authz hop would hit the same
enforce_policy_on_l7lb hairpin re-evaluation just fixed for Keycloak (68690e5).
Plan is phased/reversible (pihole canary first) and keeps the working sidecar.
Resolves the DECISIONS-LOG open item on ext_authz workaround relevance under 1.20.1.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Steady-state CPU PSI 15-23% + 15-min load 5.6-8.0 on 6 cores (run queue at/over core count) left no
headroom for bursty workloads. Proxmox host has 24 cores at ~4% load, so 3x8 = 24 vCPU is 1:1 with
zero oversubscription. In-place tofu update (cores change is pending until a per-node VM stop/start);
rolled one node at a time to keep etcd quorum. Comment also corrected 16->24 GiB (stale).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The container-workload and egress-class L1 fixtures use auth.mode=forward-auth, so the renderer now
emits the keycloak backend egress allow for them too. Regenerate the committed goldens to match the
canonical render. MERGE GATE: PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
App SSO broke cluster-wide after the Cilium 1.16.5->1.20.1 upgrade: the CVE-2025-30162 fix (in
>=1.16.8/1.17.2) makes the shared Gateway's L7 proxy re-evaluate a pod's egress policy against
Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) when it hairpins to keycloak.home.
'toEntities: world' matches the Gateway VIP at L3/L4 but not the resolved keycloak identity at L7, so
OIDC discovery/token 403'd ('Access denied') for every app with a NetworkPolicy (gitea/nextcloud/
semaphore/headlamp/crafty...); Grafana survived only because monitoring has no NetworkPolicy.
Fix: explicit toEndpoints egress allow to the keycloak backend for SSO apps.
- catalog/render.py: emit it for auth.mode in {oidc,forward-auth}; re-rendered 8 app goldens.
- platform/headlamp + platform/crafty: hand-written policies get the same allow.
Split-horizon design unchanged. Verified live: headlamp-ns -> keycloak discovery 403 -> 200.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The browser-mfa conditional-user-role authenticator used config key 'condition.evaluated.user.role',
which is not a real Keycloak property — the role resolved to null, so Keycloak logged 'Invalid role
name submitted: null' and the REQUIRED conditional-OTP step threw, 403-ing every platform-admins login
(Headlamp, Grafana admin, etc.). Correct key for Keycloak 26 is condUserRole=platform-admin.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds NodeCPUPressureHigh (CPU PSI 'some' > 15% for 15m) and NodeCPULoadHigh (load5/core > 1 for 15m)
to platform-health.nodes, plus a 'Nodes — CPU / Memory Pressure' Grafana dashboard (PSI + load-per-core
+ utilization). These surface the bursty CPU contention that averaged CPU%/kubectl top hides — the
signal behind 'spikes cause blips of delay'. Makes the 6->8 vCPU resize measurable.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Both were pinned at ~98% of their limits (Prometheus 2002/2048Mi, Grafana 499/512Mi) with no
headroom for TSDB compaction / dashboard-render spikes — the classic OOM setups. Nodes are 24Gi at
38-44% used (~14Gi free each; the '~22 GiB ceiling' comment predates the 16->24Gi node bump), so the
extra headroom is comfortable. Grafana request 256->384Mi, Prometheus request unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A node rebuild (tofu -replace / fresh OS) wipes host-level config Flux/Helm don't manage.
site.yml omits kube-vip.yml and vpn.egress.gateway.yml, so a re-provision missed the node-host
OpenVPN on k3s-dev-3 and dropped all egress-class=vpn pod traffic. Document the node-host layers
that must be re-run after any re-provision, and correct the stale author-only banner (these plays
are live).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
chart 7 major bump. Verified chart 7.3.0 renders identically for this SingleBinary/filesystem/
Longhorn config (one loki StatefulSet, no extra read/write/backend/cache pods, no schema errors).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Manifest env-var config (manager/cp_enable/svc_enable/vip_arp/leaderelection) is unchanged across
0.8->1.2, so the bump is image + re-applied auto-deploy manifest.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Grafana 12.x bundled in kps 88 OOMKilled (exit 137) at 256Mi during grafana-apiserver/unified-storage
startup — no fatal log, just SIGKILL. The new Grafana pod never became Ready, so Helm --wait ran the full
20m and rolled back to 65.8.1 every time. 512Mi holds.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 65->88 multi-major upgrade exceeded the default 5m Helm timeout (full operator/KSM/
node-exporter + Prometheus/Alertmanager StatefulSet rollout), so Flux rolled back to 65.8.1
four times and Stalled, blocking the platform kustomization and everything downstream.
CRDs pre-applied server-side at operator v0.93.1.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 k3s nodes re-provisioned to Debian 13 (kernel 6.12) via immutable tofu -replace,
one at a time, primary/leader last (secondary-join to avoid cluster-init split-brain).
- variables.tf image defaults -> debian-13-generic (generic, not genericcloud)
- k3s_server: ensure /etc/rancher/k3s exists (fresh-node re-provision fix)
- gitignore vendored ansible collections (installed for re-provision)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
All 3 k3s nodes re-provisioned to Debian 13 (kernel 6.12) via immutable tofu -replace,
one at a time, leader/primary last. variables.tf image defaults -> trixie generic.
k3s_server role: ensure /etc/rancher/k3s exists (fresh-node re-provision fix).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Root cause of a W5 wedge: default 5m Helm timeout on slow-HDD Longhorn upgrades caused
a rollback attempt after the data had already migrated to the new minor, which the old
manager fatally refuses. Give the upgrade room and never auto-rollback (fail forward).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gateway API CRDs bumped v1.2.1->v1.3.0 (required by Cilium 1.18; applied server-side).
Land on current k8s 1.32 before the 1.33 hop.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Land the two pacing components on current k8s 1.31 before the 1.32 k3s hop.
Cilium 1.17 supports k8s 1.29-1.32; Longhorn 1.8 supports 1.25+; Gateway API CRDs
stay v1.2.1 (Cilium 1.17 targets v1.2.x). Also commits the full upgrade plan doc.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Prereq for the Flux v2.7+ upgrade (W4). Flux 2.4 already serves image.toolkit/v1,
so this is safe now. render.py source + platform/image-automation + L3 test + L1 goldens.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Migrate image.toolkit.fluxcd.io/v1beta2 -> /v1 for ImageRepository,
ImagePolicy, and ImageUpdateAutomation ahead of the Flux v2.7+ upgrade
(cluster-upgrade-plan §3.2). Flux v2.4 (current) already serves /v1, so
migrating the manifests now is a safe no-op at runtime.
- catalog/render.py: r_image_automation emits v1 for ImageRepository/ImagePolicy
- platform/image-automation/{image-update-automation,per-service/pihole.example}
- conformance L3 hands-off-update chainsaw applies v1 CRs
- conformance L1 goldens re-rendered to match (apiVersion-only diff)
No spec-field changes required: all spec fields used (image, interval,
imageRepositoryRef, filterTags.pattern/extract, policy.semver.range,
sourceRef, git.*, update.strategy/path) are unchanged in v1; the v1
messageTemplate already uses .Changed (not the removed .Updated).
apps/ golden unaffected (canonical render uses --no-image-automation).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Dedicated SSD (rpool/local-zfs) virtio2 disk per k3s VM, etcd migrated to
/var/lib/rancher/k3s/server/db on /dev/vdc, isolating etcd WAL fsync from the
HDD Longhorn churn. Applied live one node at a time (dev-2/dev-1/dev-3, leader
last) with rebuild-freeze; dev-3 also bumped 16->24GB. All 3 nodes verified:
etcd on SSD, 3/3 healthy, 18/18 volumes healthy, tofu plan clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Interim bridge to stop Longhorn replica-rebuild I/O storms on the shared HDD
Storage pool from starving etcd WAL fsync (crash-looping k3s / flapping the
kube-vip VIP). Adds a small dedicated SSD disk per node for etcd only; Longhorn
stays on HDD. See docs/plans/ssd-etcd-isolation.md.
tofu (infra/tofu):
- New vars etcd_ssd_datastore (default "local-zfs" = rpool/data, Micron 5100
SATA SSD mirror w/ PLP) and etcd_disk_gib (default 16).
- New virtio2 disk block on the SSD datastore per VM: discard=on, cache=none
(honor guest fsync straight to the ZFS ZIL — safe on PLP SSD), iothread=false.
Kept last (ascending interface) so bpg does not reorder virtio0/virtio1.
- Revert the debian_cloud_image resource to the long type name
proxmox_virtual_environment_download_file to match the R2 state: bpg >= 0.111
dropped the cross-type moved{} linkage to the short proxmox_download_file, so
the stale moved block failed EVERY plan with "Resource type mismatch" (broken
on main too). Aligning code to state yields a clean plan with no image
re-download. Short-name migration is deferred to a deliberate `tofu state mv`.
ansible (node_baseline):
- Format /dev/vdc (label k3s-etcd) and mount it at
/var/lib/rancher/k3s/server/db (defaults,noatime; no nofail so a missing etcd
disk fails loudly rather than silently landing etcd on the root HDD) before
k3s starts, so a rebuilt node comes up with etcd isolated on SSD. Data-safe:
filesystem module never reformats an existing fs.
Phase 1 only: non-disruptive. No tofu apply, no node/VM/etcd/Longhorn changes.
tofu plan verified: 0 to add, 3 to change, 0 to destroy — every VM an in-place
disk ADD (virtio2), dev-3 also an in-place memory 16384->24576; no VM replace.
Merge gate green (render==golden included).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Headlamp OIDC login failed with `unsupported protocol scheme ""`. The
kubernetes-sigs/headlamp chart `envFrom`s the headlamp-oidc Secret verbatim
and renders args like `-oidc-idp-issuer-url=$(OIDC_ISSUER_URL)`, so the Secret
keys must be the exact env var names. The keys were camelCase
(clientID/issuerURL/scopes/callbackURL/clientSecret), following the mistaken
`.example`, so `OIDC_ISSUER_URL` was never defined. Kubernetes left the literal
`$(OIDC_ISSUER_URL)` in the arg and Headlamp's OIDC discovery hit
`Get "$(OIDC_ISSUER_URL)/.well-known/openid-configuration": unsupported
protocol scheme ""`.
Rename the SOPS Secret keys to OIDC_CLIENT_ID / OIDC_CLIENT_SECRET /
OIDC_ISSUER_URL / OIDC_SCOPES / OIDC_CALLBACK_URL (values unchanged, still
SOPS/age-encrypted). Fix the `.example` and README so the correct, load-bearing
key names are documented and the camelCase footgun can't recur.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The k3s VM disks are HDD-backed, so a full Longhorn replica rebuild's
sequential I/O can saturate the spindle and starve etcd's WAL fsync,
crash-looping etcd and flapping the kube-vip VIP (observed during a
rolling node resize). Cap rebuilds to one replica per node at a time so a
node bounce or replica replenishment can never storm the disks.
Set via HelmRelease defaultSettings.concurrentReplicaRebuildPerNodeLimit
(GitOps-managed, permanent). replica-replenishment-wait-interval stays at
the Longhorn default of 600s. Reference values.yaml kept in sync.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Raise per-node RAM from 16 GiB to 24 GiB for proper failover headroom.
Matches the live rolling resize of the three k3s-dev VMs on Proxmox
(qm set --memory 24576 per node, one at a time, KSM enabled first).
Single memory_mib var applies to all three nodes via for_each, so a
future `tofu apply` would resize all three at once and break etcd
quorum — the resize MUST be done live/rolling, not via apply.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Supersedes the previous kubectl-image repoint. The velero chart's pre-install
`velero-upgrade-crds` HOOK job runs docker.io/bitnami/kubectl, whose public
Docker Hub catalog Bitnami removed (Aug 2025); `bitnami/kubectl:1.31` 404s,
leaving the hook in Init:ImagePullBackOff so Velero never installs.
Repointing the hook to the exact-version rancher/kubectl:v1.31.5 (prior commit)
does not work: that image is distroless, and the hook's init runs
`/bin/sh -c "cp $(which sh)/... $(which kubectl) /tmp"` — with no /bin/sh it
crashes StartError (verified on-cluster).
The hook exists ONLY to UPDATE CRDs on chart upgrades — it is NOT how CRDs are
first installed. The chart ships all 13 Velero CRDs in its native `crds/`
directory, which Helm applies unconditionally on install (before hooks),
independent of this flag; the CRDs are already present on-cluster from Helm's
crds/ handling. Setting `upgradeCRDs: false` drops the fragile external
kubectl-image dependency entirely while CRDs still install. Trade-off: Helm
won't manage crds/ after install, so CRD schema upgrades become a
manual/Renovate step; acceptable given Velero's CRDs are stable.
All other Velero values (BSL -> versitygw s3://velero, kopia FSB node-agent, the
two Schedules, the velero-cloud-credentials secret) are unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The velero chart's pre-install `velero-upgrade-crds` hook defaults its kubectl
image to docker.io/bitnami/kubectl, whose public Docker Hub catalog Bitnami
removed (Aug 2025). `docker.io/bitnami/kubectl:1.31` now 404s, leaving the hook
job in Init:ImagePullBackOff so Velero never installs (no Deployment,
node-agent, or BackupStorageLocation).
Set kubectl.image to the Rancher-maintained mirror docker.io/rancher/kubectl,
pinned to the cluster's exact k8s minor v1.31.5 (multi-arch amd64/arm64,
verified pullable via the Docker Hub registry API). All other Velero values
(BSL → versitygw s3://velero, kopia FSB node-agent, the two Schedules, the
velero-cloud-credentials secret) are unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
MinIO's open-source edition is de-prioritised (console removed, commercial
pivot), so swap the off-cluster S3 backup target on the Proxmox host to
versitygw v1.7.0 (github.com/versity/versitygw) — a maintained Go S3 gateway
serving a POSIX filesystem as S3.
Rename role minio_backup_target -> s3_backup_target and playbook backup-minio
-> backup-s3. The role now:
- decommissions any pre-existing MinIO (service/binary/env/mc-client/user/
object-layout), KEEPING the ZFS dataset Storage/k3s-backups and its mount
- installs the pinned, checksummed versitygw binary (fetched+extracted on the
controller, host has no internet), a dedicated versitygw system user, and a
Type=simple hardened systemd unit running the posix backend rooted at
/mnt/Storage/k3s-backups/data, bound to 192.168.1.9:9000 (same endpoint),
plain HTTP / path-style
- ensures the dataset has xattr=sa (versitygw stores object metadata in xattrs)
- pre-creates the velero + cnpg buckets (bucket == top-level subdir)
Credentials are REUSED: versitygw's ROOT key is set to the same scoped
access_key/secret the cluster already uses, so endpoint + key + secret + bucket
names are byte-identical to the old MinIO target. The GitOps SOPS secret
(platform/backup/minio-credentials.sops.yaml) and all cluster-side Velero/CNPG
config need ZERO change. The vaulted group_vars now expose
s3_backup_target_access_key/_secret_key (same values, MinIO root creds retired).
Verified live on the Proxmox host: versitygw active+enabled on 192.168.1.9:9000,
MinIO fully removed, dataset intact; SigV4 ListBuckets with the reused key
returns velero+cnpg; a throwaway in-cluster pod reaches http://192.168.1.9:9000
(versitygw S3 response). Re-run is idempotent (no host changes).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Stand up a single-node/single-drive MinIO S3 endpoint on the Proxmox host as the
off-cluster backup target for Velero (R-STATE-03) and CloudNativePG (R-IDP-05).
Ansible (author + executed):
- roles/minio_backup_target + playbooks/backup-minio.yml — idempotent, additive only:
ZFS dataset Storage/k3s-backups (zstd, quota=100G, /mnt/Storage/k3s-backups, no
sharenfs); minio system user; pinned+checksummed MinIO/mc binaries (fetched on the
controller since the host has no WAN, copied to host); hardened systemd unit bound to
the host's cluster-reachable LAN IP (auto-discovered) on :9000; velero + cnpg buckets;
a non-root access key scoped by IAM policy to only those two buckets.
- inventory: new [proxmox_hosts] group ([email protected].2.102); overrides the cluster-wide
ansible_user. Does not disturb existing k3s node entries.
- group_vars/proxmox_hosts/vault.yml (ansible-vault): root + scoped MinIO creds.
GitOps hand-off:
- platform/backup/minio-credentials.sops.yaml (+ .example) — SOPS/age-encrypted endpoint +
scoped key/secret for the Velero/CNPG wiring (WS-STORAGE/WS-SEC). Not yet wired into a
kustomization (left for the consuming agents).
Verified live: dataset props correct; minio active+enabled bound to 192.168.1.9:9000;
S3 endpoint http://192.168.1.9:9000/minio/health/ready returns 200 from an in-cluster pod;
buckets velero+cnpg present; scoped key created+policy attached; re-run is idempotent.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The plural --allowed-groups does not exist in oauth2-proxy v7.8.1; the shared
oauth2-proxy new pod and the crafty auth sidecar both crashlooped on 'unknown flag'.
The flag is the singular, repeatable --allowed-group.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add Crafty Controller 4 (registry.gitlab.com/crafty-controller/crafty-4:4.10.8)
as a platform component: a web panel that runs Java + Bedrock Minecraft servers as
subprocesses in one pod + a 20Gi Longhorn PVC (mounted 5x by subPath).
Auth = Option B (per-app oauth2-proxy reverse-proxy sidecar), since Cilium 1.16.5
cannot do ext_authz on the shared Gateway:
- oauth2-proxy v7.8.1 sidecar (:4180) reverse-proxies to Crafty's self-signed HTTPS
UI (127.0.0.1:8443), gated to Keycloak homelab realm group platform-admins.
- Confidential `crafty` client provisioned live via kcadm + captured in the realm
import; client secret + cookie secret via SOPS/age (mirrored in realm-clients).
- Web UI: HTTPRoute crafty.home.keithjackson.dev -> Service :80 -> sidecar :4180.
- Game ports: Cilium L2 LoadBalancer on VIP 192.168.1.55 (next free after .53/.54),
starter set TCP 25565-25567 (Java) + UDP 19132-19134 (Bedrock), NOT auth-gated.
- fsGroup:0 / runAsGroup:0 / runAsUser:1000 for the image's crafty:root identity on
the root:root Longhorn PVC (the seerr fsGroup lesson).
- Own non-gating Flux tier clusters/base/crafty.yaml (dependsOn platform-config,
wait:false) + sops-decryption patch in clusters/dev — a leaf UI must never gate
the platform.
- Explicit default-deny NetworkPolicy + CiliumNetworkPolicies (platform namespace,
not app-namespace, so Kyverno does not auto-generate the baseline).
make -C ci gate green; kustomize build of platform/crafty, clusters/base,
clusters/dev clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Build the deferred forward-auth ENFORCEMENT for non-OIDC UIs (pihole/*arr/
qbittorrent/homepage; Crafty next), gated by Keycloak via oauth2-proxy.
- helmrelease: add `--allowed-groups=platform-admins` (R-IDP-06 floor:
authenticated AND in group) + `--skip-provider-button` (302 straight to
Keycloak; verified the interstitial 403 sign-in page otherwise).
- forward-auth-cec.yaml: interim self-authored CiliumClusterwideEnvoyConfig
ext_authz listener -> oauth2-proxy /oauth2/auth, failure_mode_allow:false,
per-route opt-in via virtualHost. TEMPLATE — not Flux-wired (PM deploys with
an L2 VIP + DNS repoint). The ext_authz stanza is verified in-cluster.
- README: document the mechanism, why per-route ext_authz on the shared Gateway
is NOT clean on Cilium 1.16.5 (machine-owned cilium-gateway-shared CEC; native
ExternalAuth/GEP-1494 in no released Cilium), the three options + recommendation,
the opt-in steps, the in-cluster proof, and the human browser test.
Keycloak `oauth2-proxy` client already carries the callback redirect URI + a
bare `groups` claim (full.path=false) — no realm change needed.
Proof (cilium-envoy v1.30.8): a standalone envoy running this exact ext_authz
stanza against the live oauth2-proxy denied unauthenticated requests (403,
oauth2-proxy sign-in body; upstream never served), while a no-filter control
route on the same backend returned 200. R-IDP-02 property holds.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).
Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
failed ... write to WAL: no space left on device". Scrapes still return 200
(targets show healthy, lastScrape=now) but samples are never stored, so the
TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
present) — the rules are correct; the monitor was simply blind.
`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.
- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale
Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Plex Deployment had no Plex env vars. Add via the ServiceDefinition env list
(rendered verbatim by catalog/render.py):
- PLEX_UID/PLEX_GID=1000 (image chowns /config to this and drops; matches the
Longhorn ownership convention seerr/semaphore use)
- TZ=Etc/UTC (no other service sets TZ repo-wide)
- ALLOWED_NETWORKS=10.0.0.0/8,192.168.0.0/16 so in-cluster clients (Seerr ->
plex:32400) are trusted as LAN (pod CIDR 10.0.0.0/16 + svc 10.43.0.0/16 + LAN)
PLEX_CLAIM intentionally not set (4-min TTL is unusable via git; one-time live
claim persists on /config). ADVERTISE_IP deferred: the only candidate host
resolves via split-horizon DNS to the internal Gateway VIP and its remote
reachability is unverified — a wrong value breaks remote access, so omitted
per prefer-safety, documented as a follow-up in catalog/plex.yaml.
Homepage still labeled seerr as "Overseerr"; renamed the tile to "Seerr"
(icon mdi-play-box-multiple keeps — not overseerr-specific).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add the offline_access optional client scope to the `kubernetes`
kubelogin client so the operator's kubectl can request long-lived
OFFLINE refresh tokens (--oidc-extra-scope=offline_access) instead of
being forced to re-auth on the realm's ONLINE session timeouts
(ssoSessionIdleTimeout 30m / ssoSessionMaxLifespan 10h). Offline tokens
honour offlineSessionIdleTimeout 30d with no hard max lifespan.
- keycloak-realm-import.yaml: add optionalClientScopes: [offline_access]
to the kubernetes client so a fresh realm import reproduces it.
- k8s-oidc-rbac.yaml: document the kubeconfig exec-arg
(--oidc-extra-scope=offline_access) + clear-cache/re-login steps.
Realm session timeouts unchanged; scope is optional (opt-in) and
client-only. Applied live via kcadm (reversible).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Homepage v2.1.2 treats proxmox.yaml as a required config file (present in
the image skeleton dir /app/src/skeleton alongside services/settings/
bookmarks/widgets/docker/kubernetes/custom.*). On startup it copies a
default skeleton into /app/config for any required file that is missing.
The ConfigMap provided every skeleton file except proxmox.yaml, so the pod
tried to copyfile the proxmox skeleton into the read-only subPath-mounted
/app/config and crashed with EACCES -> CrashLoopBackOff -> Gateway "no
healthy upstream".
Provide an empty proxmox.yaml in the ConfigMap and mount it via subPath
like the other config files, so Homepage finds all required files and
never writes at runtime. Verified the full required set against the running
image's skeleton dir (9 files); proxmox.yaml was the only missing one.
Bumped checksum/config annotation to re-roll the pod.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Deploy Homepage (gethomepage) as the apex landing dashboard aggregating
links to cluster services, in its OWN non-gating post-platform Flux
Kustomization (clusters/base/homepage.yaml, dependsOn platform-config,
wait:false) — mirrors headlamp/oauth2-proxy so a leaf UI can never gate
the platform chain.
- platform/homepage/: plain manifests (ns, config ConfigMap, Deployment,
Service, default-deny+Cilium ingress NP, apex HTTPRoute). Non-root pod,
HOMEPAGE_ALLOWED_HOSTS, bundled mdi-* icons (DNS-only egress).
- gateway.yaml: add https-apex + http-apex-redirect listeners for the bare
apex (the *.home wildcard listener does NOT match it). Apex is already a
SAN on the wildcard cert — no new Certificate.
- Forward-auth annotated (auth-mode=forward-auth) but NOT enforced: the
Gateway ext-authz mechanism is an unbuilt Phase-6 WS-NET TODO (same status
as all other forward-auth apps). Flagged as PM follow-up.
Gate: lint/l0/render-golden(10/10)/l1 green; conformance skipped (no cluster).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Headlamp's HelmRelease sat in the wait:true `platform` Flux tier. Its pod
crashlooped (`flag provided but not defined: -session-ttl` — chart 0.40.1
renders a container arg the appVersion v0.40.1 backend lacks; upstream
kubernetes-sigs/headlamp#4883), and because it gated `platform`, the whole
chain stalled: platform Ready=Unknown -> platform-config, oauth2-proxy,
apps-wave-0/1/2 and apps-vms all Ready=False "dependency not ready".
Part A (unblock + durable lesson): a leaf UI must never gate the chain. Move
all Headlamp resources into a new non-gating Flux Kustomization
clusters/base/headlamp.yaml (dependsOn platform-config, wait:false, SOPS
decryption patched per-env in clusters/dev) — mirroring oauth2-proxy exactly.
The new Kustomization owns the whole platform/headlamp component; its HTTPRoute
moves from platform-config into platform/headlamp/kustomization.yaml. Removed
headlamp from platform/kustomization.yaml and its httproute from
platform-config/kustomization.yaml. Now a crashlooping Headlamp can never stall
platform/platform-config/apps.
Part B (make it start): pin the chart to 0.41.0 (`>=0.41.0 <0.42.0`). The
backend added `f.Int("session-ttl", …)` in v0.41.0 and chart 0.41.0 pins that
same image (appVersion == chart version), so chart+image realign; every arg the
chart renders for our values is supported. OIDC config + per-user RBAC
(clusterRoleBinding.create:false) unchanged.
Gate: `make -C ci gate` PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add Headlamp (kubernetes-sigs/headlamp Helm chart) at
headlamp.home.keithjackson.dev with native Keycloak OIDC login and PER-USER
RBAC: Headlamp forwards each user's id_token to the k3s apiserver, so the
existing oidc:* group bindings (platform-admins->cluster-admin, viewers->view)
apply per user. The ServiceAccount is deliberately NOT bound to cluster-admin
(clusterRoleBinding.create=false), since user requests ride the forwarded token.
Key integration nuance: the apiserver is configured oidc-client-id=kubernetes,
so it only accepts tokens whose aud contains `kubernetes`. The new `headlamp`
Keycloak client carries an oidc-audience-mapper (included.client.audience=
kubernetes) so tokens carry aud=[headlamp, kubernetes] -- accepted by both
Headlamp's own verifier and the apiserver, leaving the apiserver config
untouched. A groups mapper emits the `groups` claim the apiserver maps to RBAC.
- platform/headlamp: namespace, HelmRepository/HelmRelease, SOPS OIDC secret
(externalSecret envFrom), default-deny NetworkPolicy + Cilium ingress
(gateway)/egress (apiserver+Keycloak) policies, README. Wired into
platform/kustomization.yaml (platform tier).
- platform/headlamp/httproute.yaml wired into platform-config (Gateway API CR,
dependsOn platform), modeled on grafana-httproute.
- keycloak-realm-import: add `headlamp` confidential client + groups/audience
mappers + HEADLAMP_CLIENT_SECRET placeholder. Client also created LIVE via
kcadm (realm import no-ops on the existing realm).
- keycloak-realm-clients.sops: add headlamp-client-secret (mirrored into
headlamp-oidc-credentials clientSecret). Both encrypted to the repo age key.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
bpg/proxmox deprecates proxmox_virtual_environment_download_file ("Use
proxmox_download_file instead. This resource will be removed in v1.0."),
the short-name rename from ADR-007. Both are the same dual-registered
Framework resource, so rename the resource type + import_from reference
and add a `moved` block. OpenTofu 1.9 drives the provider's MoveState
(migration.PrefixMoveState) to transfer state in place: same id, no
re-download of the Debian image, no change to the VMs.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The committed apps/ goldens were a MIX of two render flag sets: 8 services
were rendered with `--no-image-automation` (no image CRs) while gitea and
nextcloud carried stale ImageRepository/ImagePolicy CRs from a render WITHOUT
the flag. Neither single command reproduced all goldens, and the gate did not
catch it.
Part A — canonical render:
python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
`--no-image-automation` is canonical: image automation is authored-but-DISABLED
(DECISIONS-LOG; image-reflector/-automation controllers not installed), and
every app was onboarded via this flag. Re-render drops the ImageRepository +
ImagePolicy objects from gitea + nextcloud only (52 line deletions, purely the
trailing image-automation blocks). No functional content changed — the
nextcloud postStart hardening and gitea OIDC hook are untouched.
Part B — enforcement:
ci/render-golden.sh renders every onboarded catalog service with the canonical
command into a temp dir and FAILS on ANY diff vs committed apps/ (missing
golden or content drift), across all 10 services (not just the 3 L1 spot-checks).
Not-onboarded catalog entries (keycloak, vm-demo) are excluded per DESIGN-waves.
Wired as a required no-cluster stage in ci/gate.sh, ci/Makefile, and both the
live (.gitea/workflows/gate.yml) and dormant (merge-gate.yaml) pipelines.
Proof: tampering a golden makes the stage exit 1 (blocks merge); reverting
restores exit 0. `make -C ci gate` green in the devcontainer.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
apps/wave-1/nextcloud.yaml is GENERATED from catalog/nextcloud.yaml by
catalog/render.py, so the previous commit hardened only the rendered golden —
the next render would have reverted step (d) to the old single-pass version.
Apply the same verify-and-retry hardening to the postStart hook in
catalog/nextcloud.yaml (the source of truth). Re-rendered with
`render.py --all catalog -o apps --wave-layout`; apps/wave-1/nextcloud.yaml is
byte-identical to the golden committed in the previous commit (render round-trip
clean), so no apps/ change is needed here.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two SSO follow-ups the operator hit after Nextcloud OIDC login worked.
1. Sign-out "redirect failed". On logout user_oidc redirects the browser to
Keycloak's end_session_endpoint with post_logout_redirect_uri=<Nextcloud
base '/'> and client_id=nextcloud but NO id_token_hint (provider
sendIdTokenHint=0). Keycloak then validates that redirect against the
client's post.logout.redirect.uris attribute, which was empty -> "Invalid
redirect uri" -> the operator's "redirect failed". Register
https://nextcloud.home.keithjackson.dev/* on the nextcloud client
(attributes.post.logout.redirect.uris). Applied live via kcadm and captured
here so git stays authoritative (a RealmImport no-ops on the existing realm).
2. postStart step (d) mirrors platform-admins -> built-in admin group, but the
previous single best-effort pass ran occ group:adduser once with output
swallowed and no retry. At early boot (container Started ~9s after create)
that call can transiently fail (DB / maintenance race), silently dropping the
promotion until the next restart. Harden step (d) with a verify-and-retry
loop (like step (c)): re-read group:list and re-add any platform-admins
member still missing from admin until verified (bounded ~30s), no-op for
existing members.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replace the gitignored plaintext infra/tofu/.r2.env and backend.hcl with a
single committed, SOPS/age-encrypted dotenv (secrets.sops.env) plus a thin
wrapper (tf.sh), so tofu runs with no manual secret entry and secrets live
encrypted in git like the repo's other secrets.
- .sops.yaml: creation rule encrypting infra/tofu/secrets.sops.env (all values)
to the existing age recipient.
- secrets.sops.env: encrypted, PLACEHOLDER values only. Holds R2 S3 keys, the
state-encryption passphrase (TF_STATE_PASSPHRASE scalar), and the Proxmox API
token/endpoint. Operator fills real values via `sops`.
- tf.sh: sops exec-env decrypts the dotenv, exports PROXMOX_VE_*/AWS_*, builds
the multi-line TF_ENCRYPTION HCL from TF_STATE_PASSPHRASE, and runs tofu
(injecting -backend-config=backend.r2.hcl on init). Proxmox token stays env-only,
never a tofu variable/state.
- backend.r2.hcl: committed non-secret R2 bucket/key/endpoint (account id is not
a credential), passed to `tofu init`.
- Retire .r2.env/backend.hcl(.example): drop the committed templates; keep the
gitignore lines as a safety net for leftover local plaintext.
- Update backend.tf/README/REMOTE-STATE for the new day-2 flow and DR story
(repo + age key is sufficient).
Verified: placeholder file round-trips sops -e/-d with the repo age key; tf.sh
exports the expected VAR NAMES and builds valid TF_ENCRYPTION HCL (stub tofu, no
real plan/secrets).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 5