0 Fork 0

Files

T

jakman85andClaude Opus 5 fd43f332d6 feat(pihole): authoritative LAN DNS for the zone on udp/53 + tcp/53 at a pinned VIP

platform/coredns/coredns-custom.yaml has always said its split-horizon is for
PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS:
its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03
that docs/09 and the README's Phase 2 row both claim did not exist, so LAN
clients ran on hand-maintained per-client dnsmasq files - one went stale after
the dev decommission and cost the operator sonarr.home.keithjackson.dev. The
SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips
RFC1918 answers for public domains as rebind protection and returns NOERROR
with no answer.

EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by
render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a
pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason
apps/vms/unifi and platform/crafty hand-write theirs):
  - two Services, udp/53 and tcp/53, sharing one pinned VIP
    ${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the
    .20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22
    crafty; arping from a LAN host got no reply).
  - no external-dns hostname annotation: nothing public is created, and
    platform/externaldns/ is untouched.

ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes
every FTL setting as an FTLCONF_ environment variable and treats it as
env-forced, applied ahead of pihole.toml on every cold start and read-only in
the UI. That is a real declarative entry point, which the Keycloak and CNPG
CronJob reconcilers exist precisely because their state lacks, so a reconciler
here would be ceremony. misc.dnsmasq_lines carries a wildcard
address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service
(unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching
domain, so precedence is structural and a newly onboarded Gateway-fronted
service needs no DNS change. dns.upstreams is declared too - the live pod
carried hand-set upstreams that no manifest knew about and that every restart
discarded.

externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the
LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not
filter lease candidates by which nodes hold a backend - the two Services landed
on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps
renewing after the pod moves away, so with Local a reschedule leaves LAN DNS
dead indefinitely. Cluster serves the VIP from any node; the LAN restriction
moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP
before load-balancing (verified by narrowing the range to exclude the test
client and back). The cost is SNAT: forwarded queries reach PiHole from the
ingress node, so PiHole's per-client rate limit is disabled in the catalog or
the whole LAN would count as one client against its 1000-per-60s default.

Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and
both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex
-> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion
(cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA
NODATA, NXDOMAIN outside the zone, and still answering after a forced
prod-1 -> prod-2 reschedule.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-14 00:57:58 -05:00

8.0 KiB

Raw Blame History

home-network-k3s-impl

Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts (Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies, Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only specification at ../home-network-k3s.

Rules (from the spec — non-negotiable)

The spec is read-only. Never modify anything under ../home-network-k3s. Implement against it; escalate spec problems, don't edit them (spec AGENTS.md §11, R-TEST-06).
Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a manual demo (spec docs/14, agents/definition-of-done.md).
No work on main. Every change is a feature branch merged into main (spec CONTRIBUTING.md).
No plaintext secrets — SOPS-encrypted only (spec docs/07).
Phase discipline — only build what the active roadmap phase authorizes (spec roadmap/phases.md).

Layout

infra/          node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
                pinned dev container that carries the whole toolchain
bootstrap/      one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/       platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
                longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
                vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
                (ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
                Split out because CRs and the CRDs they use cannot apply in one pass.
apps/           application tier, reconciled as dependency-ordered WAVES:
                  wave-0/ wave-1/ wave-2/   rendered container apps (see DESIGN-waves.md)
                  vms/                      KubeVirt VirtualMachines (non-gating tier)
catalog/        ServiceDefinition entries (the onboarding surface) + render.py
schemas/        ServiceDefinition JSON Schema + renderer contract
clusters/       Flux entrypoints + dev/ and prod/ overlays
conformance/    Chainsaw behavior/conformance suite + harness + coverage report
ci/             the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/         Gitea Actions CI running that gate on an in-cluster runner

Status

Roadmap phases 0–6 are delivered, and Phase 7 (Production Readiness) is OPEN and in progress. clusters/prod is no longer a design-only overlay: prod is the live cluster, serving the apex home.keithjackson.dev. The dev cluster was decommissioned on 2026-09-12 (full migration: dev out, prod onto the apex — see the DECISIONS-LOG), so phases 0–6 are now exercised on prod hardware rather than on Proxmox VMs.

Phase 7 exit criteria — prod runs the same config via the prod overlay; DR proven; monitoring + alerts live:

criterion	state
prod runs the prod overlay	met — 3× mini-PC bare metal, one Flux tree, prod overlay only
monitoring + alerts live	met 2026-09-13 — see the note below; this was not true before that date
DR proven	R-STATE-03 PROVEN on prod 2026-09-14 — off-cluster restore from Cloudflare R2. `R-STATE-02` (node-loss) is still dev-only

DR restore is proven on prod. R-STATE-03 reads proven in --status as of 2026-09-14 — the first requirement in this repo to reach that state. Both of its verifiers are recorded: dr-restore (L3, run 20260914T021505Z-24b8111, sentinel DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK) and placeholder-guard-coupling (L0). The L3 drill seeds a canary on the replicated longhorn class, backs the namespace and volume data up off-cluster, destroys every in-cluster copy (namespace, PVC and PV), restores into a clean namespace, and asserts the restored PVC binds a different PersistentVolume — so an in-cluster survival cannot pass as a restore.

What remains for Phase 7: R-STATE-02 (node-loss) is still proven only on the decommissioned dev cluster. That drill takes a real node down on a live cluster, so it is a deliberate, scheduled exercise rather than something to run in passing.

Read --coverage-only and --status as two different questions; the earlier PROVEN: 0 was not a bug but the honest answer, because proofs are claims about a specific tree on a specific cluster.

Phase	Delivered
0 Foundations	ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI
1 Cluster bring-up	3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; now 3× mini-PC bare metal (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13)
2 Platform core	Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, both halves: CoreDNS for pods, and PiHole as the authoritative LAN resolver for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files
3 Egress / VPN	ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch
4 Observability	kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero unblocked 2026-09-13 — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has no backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled
5 IaaS	KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest
6 Identity + CI	Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate
7 Production readiness	IN PROGRESS (DR restore proven 2026-09-14; node-loss outstanding). Prod live on bare metal serving the apex; dev decommissioned; CNPG HA 3 instances with primaries spread one per node; NFS bulk media + legacy Nextcloud mounted; VPN egress on prod. Alerting made real 2026-09-13 (Longhorn, cert-manager, Flux and twelve app SLIs had all been unable to fire). Open: `node-loss` (R-STATE-02) re-proven on prod.

Conformance: 51 / 51 requirements covered (100%), 0 gaps, 0 safety-critical gaps (closed 2026-09-13; R-C2-01/R-C2-02 were the last two). Regenerate with ./conformance/run.sh --coverage-only; the full report is conformance/_artifacts/coverage/coverage.md.

Coverage is not proof. A requirement is covered when an executable test exists for it, and PROVEN only when a recorded run of that test is in conformance/runs/ledger.jsonl at the current SHA on the cluster it claims. Read both numbers: --coverage-only for the first, --status for the second.

The running narrative — decisions, root causes, incidents, follow-ups — is DECISIONS-LOG.md. It, not this table, is the authoritative record.

Working on this repo

Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:

infra/devcontainer/dev.sh build            # once
infra/devcontainer/dev.sh run make -C ci gate   # the same gate CI runs
infra/devcontainer/dev.sh                  # interactive shell (sops, flux, kubectl, tofu, ansible)

The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only so the spec-immutability check can actually run. See infra/devcontainer/README.md.

A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.