jakman85andClaude Opus 5
7880d84bce
The CNPG metrics exporter could not CONNECT to the `nextcloud` application
database, so every user-defined collector query against that target failed on
all three instances:
FATAL: permission denied for database "nextcloud" (SQLSTATE 42501)
DETAIL: User does not have CONNECT privilege.
The whole cnpg_pg_stat_* / pg_settings / pg_postmaster family was therefore
MISSING for nextcloud-db while the scrape target still reported up=1 — the blind
spot platform/cnpg/alerts.yaml's CNPGMetricsCollectorErroring was written to
catch. It had been firing on all three pods; measured before this change,
cnpg_pg_settings_setting had 765 series for keycloak, 765 for semaphore-db and
ZERO for nextcloud-db. Nextcloud's database had never been monitored.
Root cause is the live ACL, not the spec — all three Cluster specs bootstrap
identically:
nextcloud : datacl = {nextcloud=CTc/nextcloud} <- explicit, no PUBLIC entry
keycloak : datacl = NULL
semaphore : datacl = NULL
A NULL datacl means PostgreSQL's defaults, which include CONNECT for PUBLIC, so
keycloak and semaphore admit the exporter only by inheriting PUBLIC's implicit
CONNECT. On `nextcloud` something ran the equivalent of REVOKE ALL ON DATABASE
nextcloud FROM PUBLIC, which materialises datacl and drops that implicit entry.
That REVOKE is in neither this repo nor DECISIONS-LOG.md — it was applied out of
band, which is why the remedy has to be declarative rather than another hand-run
psql.
CNPG 1.30.0 cannot express a database-level GRANT (verified against the live
CRDs): `Database` has no ACL field (`allowConnections` is the datallowconn flag,
not a per-role grant), and managed.roles/DatabaseRole cover role attributes plus
`inRoles` membership only — and membership would hand the exporter the owner's
CREATE/TEMP and all table access to buy one CONNECT. postInitApplicationSQL runs
once at bootstrap, so it cannot repair the already-bootstrapped instances that
are actually broken. So this follows the pattern the repo already uses for state
Flux cannot reach: the level-triggered CronJob reconciler alongside
platform/keycloak/client-secret-reconciler.yaml. A one-shot Job is deliberately
avoided — a completed Job never re-runs, so a rebuild or PITR restore would
silently reopen the blind spot.
Exactly one privilege, and nothing else:
GRANT CONNECT ON DATABASE nextcloud TO cnpg_metrics_exporter;
giving `cnpg_metrics_exporter=c/nextcloud` — CONNECT only, no CREATE, no
TEMPORARY, no ownership, no superuser, no table or schema grants. Nothing more is
needed because CNPG already creates the role as a member of the built-in
pg_monitor role (verified on all three clusters), which carries the read access
the collector's pg_catalog queries want. It runs as the database OWNER, not a
superuser: enableSuperuserAccess stays off, and a database owner holds the grant
option on its own database.
Verified live on prod:
* the grant issued under SET ROLE nextcloud, proving no superuser is needed;
* permission-denied and "Error collecting user query" both went to 0 on
nextcloud-db-1/-2/-3, i.e. every collector query now succeeds;
* cnpg_pg_settings_setting now 765 series for nextcloud-db, matching the other
two, and cnpg_pg_postmaster_start_time present on all three instances;
* cnpg_last_error = 0 on all NINE instances, so keycloak-db and semaphore-db
are unaffected and their datacl is still NULL;
* convergence proven by revoking the grant again and letting the committed
CronJob restore it, then re-running it against a converged database to show
idempotence;
* CNPGMetricsCollectorErroring, firing on all three pods beforehand, has
cleared (`for: 6h` delays firing, not resolution).
Co-Authored-By: Claude Opus 5 <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.