jakman85andClaude Opus 5
21491f901b
CNPGNoRecoverabilityPoint has been firing continuously on prod against six
pods — keycloak-db-1/-3, nextcloud-db-1/-3, semaphore-db-1/-3 — i.e. exactly
the REPLICAS of all three Clusters, while the databases themselves are fine:
every Cluster reports firstRecoverabilityPoint=2026-09-07T07:0x, a
lastSuccessfulBackup within the last few hours, ContinuousArchiving=True and
LastBackupSucceeded=True.
Root cause is aggregation scope, not archiving. `cnpg_collector_first_recoverability_point`
reads the backup CATALOGUE, which only the primary owns; a replica reports a
hard 0 for as long as it is a replica. The metric carries no cnpg.io/cluster
label (its label set on prod is container/endpoint/instance/job/namespace/pod),
so `max by (namespace, pod)` evaluated each replica as its own subject and an
`== 0` comparison then held forever. The rule went from correct to permanently
wrong the moment the Clusters went to `instances: 3`; nothing about the
expression changed.
WHY THIS IS WORSE THAN HAVING NO ALERT. An alert that is always firing has
zero information content: its value is identical in the healthy state and in
the failure state, so it cannot distinguish them. It is worse than silence in
two specific ways. First, it trains the operator to delete the notification on
sight, so the day a base backup genuinely stops the seventh identical email is
discarded with the other six — the alert actively MASKS the failure it was
written to catch, which for a DR alarm means discovering the loss at restore
time. Second, it is indistinguishable from a broken alert, so nobody can tell
whether the absence of new alerts means health or means the rule rotted. This
file exists because a real 38-hour archiving failure told nobody; a rule that
cries wolf about six healthy replicas rebuilds exactly the blindness it was
supposed to remove. This repo has been here before — see the permanently-firing
critical VPNTunnelDown post-mortem in platform/monitoring/alerts.yaml.
Fix: group by (namespace, job) and aggregate `pod` away. The CNPG operator
creates one PodMonitor per Cluster, named after the Cluster, in the Cluster's
namespace, so prometheus-operator derives job="<namespace>/<cluster>" —
keycloak/keycloak-db, nextcloud-db/nextcloud-db, semaphore-db/semaphore-db —
which identifies the CNPG Cluster uniquely. `job` rather than bare `namespace`
deliberately: namespace alone works today (one Cluster per namespace, verified
live) but would let two Clusters in one namespace mask each other, since the
healthy one's non-zero value would satisfy `max` for both. The alert now
states what its summary always claimed — this CLUSTER has no recoverability
point — and fires only when no instance of that Cluster reports one.
Proven live against prod Prometheus, read-only (promtool query instant):
OLD max by (namespace, pod) (...) == 0 -> 6 series, and 6 ALERTS firing
NEW max by (namespace, job) (...) == 0 -> empty
The new expression is not vacuous: forcing the metric to 0 for one cluster's
primary only yields exactly one series, {job="keycloak/keycloak-db"}, so a
genuine single-cluster loss of DR still alerts, per cluster. `promtool check
rules` passes on all 5 rules.
CNPGNoRecentBaseBackup carried the same defect and is fixed the same way.
`cnpg_collector_last_available_backup_timestamp` is also primary-only: on prod
the primaries read 3.1h/2.1h/0.1h old while all six replicas read 0, i.e.
time()-0 = ~56.7 YEARS. It was not firing only because the `> 0` guard was
load-bearing — a single clause was all that stood between this rule and six
more permanent false positives, too thin a margin for a DR alarm. Aggregating
`pod` away removes the dependence: `max` takes the primary's timestamp as the
cluster's answer, and the guard now means "this cluster has had at least one
backup". Verdict unchanged live (old empty, new empty).
The other three rules were audited and deliberately left on `by (namespace, pod)`:
* CNPGWALArchiveBacklog — every instance reports archive_status and the shape
is `> 0`, so a replica at 0 never fires and no pod can mask another; `pod`
names the instance whose archiver is stuck.
* CNPGWALVolumeNotRecycling — per-pod is REQUIRED, not merely safe:
aggregating it away would compare one instance's WAL count against
another's max_wal_size.
* CNPGMetricsCollectorErroring — unchanged in meaning AND grouping. Its
grouping is already correct: cnpg_last_error is genuinely per-instance, and
prod proves it discriminates rather than tracking role — all three
nextcloud-db pods report 1 while all six keycloak/semaphore pods report 0.
It is a true positive (the nextcloud 42501 CONNECT grant) owned by separate
work; nothing here silences or rescopes it.
The general rule, recorded in the file: `== 0`/staleness shapes must aggregate
`pod` away or every structurally-silent pod fires forever; `> 0` shapes are
safe per-pod and keep the instance name.
Co-Authored-By: Claude Opus 5 <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.