jakman85andClaude Opus 5
a2440a3125
Two halves of one repeatable upgrade process. Neither works alone. (a) OS upgrades and reboots platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml: that tier gates platform-config and every apps wave, so a wedged reboot daemon there could stall every application. A component added to make maintenance safer must not be able to cause an outage. infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other open branches, and this is a different lifecycle), enabled for prod in group_vars. The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's rejection of automating away the gate between nodes: u-u is security-only with the kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then does only the mechanical half, including the ordering rule node_baseline has to shout. Quorum: concurrent reboots are structurally impossible, not merely unlikely. concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency), and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node holds it forever and nothing cascades. Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait, serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m (unconditional) plus kured's Prometheus alert gate (semantic). alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and under the chart default kured would have been blocked on day one by the five alerts measured firing with no end in sight, and never rebooted anything. Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it. Asserted against apt-config dump, not against the file just written. 6.12 kernels are still installed and bootable on all three nodes, and nothing was held. Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING could create the sentinel. kured alone would have idled forever at 1/1 Running. platform/kured/alerts.yaml — three rules over kured's only metric, with a companion promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md. Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this cluster's 7d Prometheus retention. (b) Image/dependency automation Renovate was aspirational: renovate.json present and complete, with no workflow, no in-cluster workload, and its docker+kubernetes managers disabled — so every plain image: tag was maintained by nothing while seven files claimed otherwise. .gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a skipped Gitea job reports success and that is how this stayed inert. Renovate now owns both halves; Flux image-automation is formally closed: it pushes commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here anyway — main is protected user_can_push:false and the flux-system GitRepository has no secretRef. Its controllers are installed and have idled 43 days with zero CRs. conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing; tightens the check for this component (requires service.create+metrics.create, both chart-default false, plus a companion NetworkPolicy requirement). Required because assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is shrink-only. Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author approving review; /approve-override deliberately NOT posted. NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role never executed, and whether CNPG switches its primary on a drain-induced eviction (the most likely way the first real reboot fails). Full list in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No work in a shared checkout. Every agent run gets its own git worktree, outside the repo
root, branched from
origin/main. This checkout is shared mutable state — seeCLAUDE.md§0. - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, vpn-ingress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
(vpn-egress = pods OUT via PIA; vpn-ingress = remote clients IN via WireGuard)
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered, and Phase 7 (Production Readiness) is OPEN and in
progress. clusters/prod is no longer a design-only overlay: prod is the live cluster, serving the
apex home.keithjackson.dev. The dev cluster was decommissioned on 2026-09-12 (full migration:
dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
hardware rather than on Proxmox VMs.
Phase 7 exit criteria — prod runs the same config via the prod overlay; DR proven; monitoring + alerts live:
| criterion | state |
|---|---|
| prod runs the prod overlay | met — 3× mini-PC bare metal, one Flux tree, prod overlay only |
| monitoring + alerts live | met 2026-09-13 — see the note below; this was not true before that date |
| DR proven | R-STATE-03 PROVEN on prod 2026-09-14 — off-cluster restore from Cloudflare R2. R-STATE-02 (node-loss) is still dev-only |
DR restore is proven on prod. R-STATE-03 reads proven in --status as of 2026-09-14 —
the first requirement in this repo to reach that state. Both of its verifiers are recorded:
dr-restore (L3, run 20260914T021505Z-24b8111, sentinel DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK)
and placeholder-guard-coupling (L0). The L3 drill seeds a canary on the replicated longhorn
class, backs the namespace and volume data up off-cluster, destroys every in-cluster copy
(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
different PersistentVolume — so an in-cluster survival cannot pass as a restore.
What remains for Phase 7: R-STATE-02 (node-loss) is still proven only on the decommissioned
dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
exercise rather than something to run in passing.
Read --coverage-only and --status as two different questions; the earlier PROVEN: 0 was not a
bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; now 3× mini-PC bare metal (k3s-prod-1/2/3, 192.168.2.10/.11/.12, k3s v1.36.4+k3s1, Debian 13) |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, both halves: CoreDNS for pods, and PiHole as the authoritative LAN resolver for ${BASE_DOMAIN} on udp/53 + tcp/53 at a pinned VIP (192.168.2.23 on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client dnsmasq files |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero unblocked 2026-09-13 — backs up to Cloudflare R2 (OQ-BKP closed; the velero-plugin-for-aws tagging incompatibility needed a pinned plugin). Longhorn has no backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
| 7 Production readiness | IN PROGRESS (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). Open: node-loss (R-STATE-02) re-proven on prod. |
Conformance: 51 / 51 requirements covered (100%), 0 gaps, 0 safety-critical gaps (closed
2026-09-13; R-C2-01/R-C2-02 were the last two). Regenerate with
./conformance/run.sh --coverage-only; the full report is
conformance/_artifacts/coverage/coverage.md.
Coverage is not proof. A requirement is covered when an executable test exists for it, and
PROVEN only when a recorded run of that test is in conformance/runs/ledger.jsonl at the current
SHA on the cluster it claims. Read both numbers: --coverage-only for the first, --status for the
second.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
First, get out of this checkout. It is the operator's working copy and several agents share it; concurrent runs here have overwritten each other's branches and left each other's uncommitted files behind. One workspace per change:
git fetch origin main
git worktree add -b <type>/<slug> /path/outside/the/repo/wt-<slug> origin/main
cd /path/outside/the/repo/wt-<slug>
Outside the repo root specifically — a worktree under .claude/worktrees/ is a second full
copy that tree-walking gate checks then count twice. The rule, the measurements behind it, and
the three gate workarounds a worktree needs are in CLAUDE.md §0 and §3.
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.