ad3f56eee8b6daae80f81c57f03fd76e2915031f
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
f92707f38c |
feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers)
Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/** , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fd43f332d6 |
feat(pihole): authoritative LAN DNS for the zone on udp/53 + tcp/53 at a pinned VIP
platform/coredns/coredns-custom.yaml has always said its split-horizon is for
PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS:
its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03
that docs/09 and the README's Phase 2 row both claim did not exist, so LAN
clients ran on hand-maintained per-client dnsmasq files - one went stale after
the dev decommission and cost the operator sonarr.home.keithjackson.dev. The
SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips
RFC1918 answers for public domains as rebind protection and returns NOERROR
with no answer.
EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by
render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a
pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason
apps/vms/unifi and platform/crafty hand-write theirs):
- two Services, udp/53 and tcp/53, sharing one pinned VIP
${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the
.20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22
crafty; arping from a LAN host got no reply).
- no external-dns hostname annotation: nothing public is created, and
platform/externaldns/ is untouched.
ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes
every FTL setting as an FTLCONF_ environment variable and treats it as
env-forced, applied ahead of pihole.toml on every cold start and read-only in
the UI. That is a real declarative entry point, which the Keycloak and CNPG
CronJob reconcilers exist precisely because their state lacks, so a reconciler
here would be ceremony. misc.dnsmasq_lines carries a wildcard
address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service
(unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching
domain, so precedence is structural and a newly onboarded Gateway-fronted
service needs no DNS change. dns.upstreams is declared too - the live pod
carried hand-set upstreams that no manifest knew about and that every restart
discarded.
externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the
LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not
filter lease candidates by which nodes hold a backend - the two Services landed
on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps
renewing after the pod moves away, so with Local a reschedule leaves LAN DNS
dead indefinitely. Cluster serves the VIP from any node; the LAN restriction
moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP
before load-balancing (verified by narrowing the range to exclude the test
client and back). The cost is SNAT: forwarded queries reach PiHole from the
ingress node, so PiHole's per-client rate limit is disabled in the catalog or
the whole LAN would count as one client against its 1000-per-60s default.
Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and
both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex
-> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion
(cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA
NODATA, NXDOMAIN outside the zone, and still answering after a forced
prod-1 -> prod-2 reschedule.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
268b0a26b4 |
docs(readme): DR restore is proven on prod — Phase 7's remaining item is node-loss
R-STATE-03 reads proven in --status as of 2026-09-14, the first requirement in this repo to reach that state, with both verifiers recorded on prod. Also states why the earlier PROVEN: 0 was the honest answer rather than a bug: a proof is a claim about a specific tree on a specific cluster, so coverage (51/51) and proof are two different questions. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
229d7f2d9c |
docs: README status was stale by a whole phase; record the R-STATE-03 decision
The Status section still said phases 0-6 were deployed on the DEV cluster and that Phase 7 was not open with clusters/prod 'a design-only overlay'. Dev was decommissioned on 2026-09-12 and prod has been the live cluster serving the apex since. It also claimed 36/51 requirements covered with a safety-critical gap, and that Velero was blocked on OQ-BKP. Now states Phase 7 as open with its three exit criteria and which one is actually outstanding; corrects the node inventory from the Proxmox dev IPs to the bare-metal prod nodes; records Velero as unblocked to R2; and adds the distinction the old text elided — coverage (51/51) is not proof (PROVEN: 0), because the only dr-restore and node-loss proofs were recorded on a cluster that no longer exists, against MinIO rather than R2. Also records the owner's decision that Velero+kopia to R2 is the accepted volume backup, with no Longhorn backup target, and that Longhorn config belongs in source. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f7d3884ad2 |
fix(ci): scope the requirement matcher to the diff and to non-doc files
`ci/touched-requirements.sh` documents its detection as "R-* tokens literally
present in the diff of changed files". It did not do that — it grepped the ENTIRE
CONTENT of every changed file. So a change inherited every requirement those files
had ever mentioned in prose, and `--require-all` then blocked the merge on
requirements the author never went near.
On this branch that meant editing the decision log and the tofu docs dragged in
five unrelated requirements, all on untouched lines, all legitimately uncovered:
[coverage] REQUIRE: FAILED — required requirement(s) uncovered: [5 IDs]
The practical effect was that the PM could not append to the decision log without
failing CI — which is a fair part of why the log went stale in the first place.
This is the defect already logged as "FOLLOW-UP (WS-TEST): fix the matcher so real
requirements need not be exempted".
Two scoping rules, both learned from the gate blocking merges it should not have:
1. Scan the ADDED/REMOVED lines of the diff, as the header always claimed.
Removed lines count too — deleting a test's requirement tag, or a policy's
self-declared ID, touches that requirement just as much as adding one.
2. Skip `*.md`. An ID in prose is a REFERENCE, not a declaration: explaining a
requirement is not changing behaviour against it. Rule 1 alone is not enough —
a commit that documents a mis-attribution names the IDs involved and thereby
re-triggers the very mis-attribution it fixes, and the decision log can never
record which requirements a change concerned.
This TIGHTENS attribution rather than weakening the gate:
- no requirement is exempted, and DOCUMENTED_GAPS is untouched;
- authoritative declarations (`conformance.home.k3s/R-*` tags, Kyverno policies,
ServiceDefinitions) all live in YAML, on lines a real change must touch;
- the coarse path->requirement fallback is unchanged and file-type agnostic, so
the doc subtrees that matter (conformance/, schemas/, catalog/, clusters/, ci/)
stay covered either way.
Alternatives rejected: adding the five IDs to DOCUMENTED_GAPS genuinely weakens the
gate; scrubbing requirement IDs out of prose (precedent
|
||
|
jakman85andClaude Opus 5
|
12c40d8c9c |
docs: reconcile the documentation with the deployed reality
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.
Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.
README.md
- Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
- Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
- Real conformance number (36/51, 70.6%) and how to regenerate it.
- Devcontainer workflow; note that the gate was red and why.
infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
the toolchain, off-repo state that must not be lost, and the fact that Flux's
git source is an app inside the cluster (`git push origin main` IS the deploy).
infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
rationale are still worth keeping; the libvirt execution steps are not.
infra/README.md — Proxmox, plus a map of the subtree and where the cluster
actually lives (it is not on this workstation).
infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
talks to an HTTP API. Everything runs in the container now. Documents the full
CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
and the read-only spec repo especially — without the latter, spec-immutability
silently skips instead of verifying).
infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
against the running nodes), pfSense NAT marked done, and a warning that
terraform.tfstate is absent on a fresh workstation — an apply with empty state
would try to recreate the three live nodes.
infra/tofu/variables.tf — section comment disagreed with its own defaults.
infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
domains that do not exist; behaviour unchanged, but it no longer lies.
apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
since been dropped. Now: live status, 40Gi, resolved open questions marked
resolved, and the two constraints the next person will otherwise rediscover —
KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
postBuild.substituteFrom (envsubst corrupts the cloud-init).
DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
the Flux source cutover, CI going live in-cluster, ten app onboardings and the
renderer bugs they exposed, the wave split (verified: DBs adopted, not
recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
VM, and kubectl SSO. Plus verified live state and the carried-forward open list.
Two corrections to the record while verifying:
- kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
[oidc:platform-admins]), so the rolling restart
|
||
|
jakman85andClaude Opus 4.8
|
253b6600f4 |
Initial skeleton: home-network-k3s implementation repo
Phase 0 foundation shell (read-only spec at ../home-network-k3s). Top-level tree + rules; workstream agents fill their subtrees on feature branches. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |