jakman85andClaude Opus 5
8427b28b0e
FluxRolloutInProgress fired correctly on the first real deploy and its mail
was suppressed before Alertmanager dispatched it. One label was wrong.
kube-prometheus-stack ships general.rules -> InfoInhibitor:
group by (namespace) (ALERTS{severity="info"} == 1)
unless on (namespace) group by (namespace)
(ALERTS{alertname!="InfoInhibitor", alertstate="firing",
severity=~"warning|critical"} == 1)
which fires for any namespace holding a firing info alert AND NOTHING ELSE
wrong in it, and the chart's third inhibit_rule (source alertname=InfoInhibitor
-> target severity="info", equal [namespace]) then drops exactly that alert.
For a deploy notification that is the worst possible interaction, because the
suppressed case is the ONLY case that matters: on a healthy rollout of a healthy
cluster flux-system holds no warning or critical alert, so the notification is
inhibited precisely when it is wanted, and would arrive only when something else
in flux-system was already broken. Measured on prod: InfoInhibitor went active
2026-09-30T04:58:48Z, the same moment the alert began firing for 12
Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager
either active or --inhibited, with zero send failures.
Upstream's intent is explicit and fair: severity "info" means "only worth
showing next to a real problem". A deploy notice is not that, so it must not
claim that severity. FluxRolloutInProgress now carries `notification: deploy`
and no severity at all, and the route matches on that label — no inhibit_rule
in the chart targets anything but severity values, and omitting severity also
stops InfoInhibitor firing for flux-system rather than merely working around it.
FluxRolloutStuck keeps severity: warning and is unaffected.
Also verified on prod in the process, all previously UNVERIFIED: the 33 metric
series exist exactly as designed, the `revision` label IS populated (so the
lastAppliedRevision/artifact.revision paths are right), the podAnnotations roll
forced a new KSM pod, and the rule both fired for 12 lagging Kustomizations and
then CLEARED on convergence — the completion signal's whole mechanism, on real
data. Routing proven by `amtool config routes test notification=deploy` ->
email-deploy.
Still unverified: no mail observed in the inbox. The 226->227 email counter
increment during the window cannot be attributed (no receiver label, other
alerts active), and I am not claiming it either way. See the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered, and Phase 7 (Production Readiness) is OPEN and in
progress. clusters/prod is no longer a design-only overlay: prod is the live cluster, serving the
apex home.keithjackson.dev. The dev cluster was decommissioned on 2026-09-12 (full migration:
dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod
hardware rather than on Proxmox VMs.
Phase 7 exit criteria — prod runs the same config via the prod overlay; DR proven; monitoring + alerts live:
| criterion | state |
|---|---|
| prod runs the prod overlay | met — 3× mini-PC bare metal, one Flux tree, prod overlay only |
| monitoring + alerts live | met 2026-09-13 — see the note below; this was not true before that date |
| DR proven | R-STATE-03 PROVEN on prod 2026-09-14 — off-cluster restore from Cloudflare R2. R-STATE-02 (node-loss) is still dev-only |
DR restore is proven on prod. R-STATE-03 reads proven in --status as of 2026-09-14 —
the first requirement in this repo to reach that state. Both of its verifiers are recorded:
dr-restore (L3, run 20260914T021505Z-24b8111, sentinel DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK)
and placeholder-guard-coupling (L0). The L3 drill seeds a canary on the replicated longhorn
class, backs the namespace and volume data up off-cluster, destroys every in-cluster copy
(namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a
different PersistentVolume — so an in-cluster survival cannot pass as a restore.
What remains for Phase 7: R-STATE-02 (node-loss) is still proven only on the decommissioned
dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled
exercise rather than something to run in passing.
Read --coverage-only and --status as two different questions; the earlier PROVEN: 0 was not a
bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; now 3× mini-PC bare metal (k3s-prod-1/2/3, 192.168.2.10/.11/.12, k3s v1.36.4+k3s1, Debian 13) |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, both halves: CoreDNS for pods, and PiHole as the authoritative LAN resolver for ${BASE_DOMAIN} on udp/53 + tcp/53 at a pinned VIP (192.168.2.23 on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client dnsmasq files |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero unblocked 2026-09-13 — backs up to Cloudflare R2 (OQ-BKP closed; the velero-plugin-for-aws tagging incompatibility needed a pinned plugin). Longhorn has no backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
| 7 Production readiness | IN PROGRESS (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). Open: node-loss (R-STATE-02) re-proven on prod. |
Conformance: 51 / 51 requirements covered (100%), 0 gaps, 0 safety-critical gaps (closed
2026-09-13; R-C2-01/R-C2-02 were the last two). Regenerate with
./conformance/run.sh --coverage-only; the full report is
conformance/_artifacts/coverage/coverage.md.
Coverage is not proof. A requirement is covered when an executable test exists for it, and
PROVEN only when a recorded run of that test is in conformance/runs/ledger.jsonl at the current
SHA on the cluster it claims. Read both numbers: --coverage-only for the first, --status for the
second.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.