Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8fe8d5df8a | ||
|
|
020e736541 | ||
|
|
92944486d1 | ||
|
|
2e777e13ca | ||
|
|
16222606ae | ||
|
|
d9ab57f2d1 | ||
|
|
9724445a08 |
No files matched your search
@@ -76,7 +76,7 @@ bug but the honest answer, because proofs are claims about a specific tree on a
|
||||
|---|---|
|
||||
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
|
||||
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the LAN resolver**, answering an explicit list of `${BASE_DOMAIN}` names (LAN-only apps, gitea/nextcloud/keycloak/auth, the L4 VIPs — everything else resolves publicly to Cloudflare, since 2026-11-09) on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
|
||||
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
|
||||
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
|
||||
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
|
||||
|
||||
@@ -100,6 +100,11 @@ metadata:
|
||||
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
|
||||
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
|
||||
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
|
||||
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
|
||||
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
|
||||
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
|
||||
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
|
||||
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
|
||||
@@ -176,8 +176,8 @@ or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
|
||||
|
||||
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
|
||||
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
|
||||
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
|
||||
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
|
||||
`apps/wave-0/pihole.yaml` (LAN split-horizon, an explicit name list since 2026-11-09) point
|
||||
`unifi.${BASE_DOMAIN}` at `${UNIFI_L4_VIP}` rather than `${GATEWAY_VIP}`. `${BASE_DOMAIN}` on prod is the
|
||||
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
|
||||
flip, so this hostname is current, not a dev leftover.
|
||||
|
||||
|
||||
@@ -73,7 +73,7 @@ spec:
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: '0'
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
value: local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
- name: FTLCONF_webserver_api_password
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
||||
@@ -302,6 +302,8 @@ metadata:
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
annotations:
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
+57
-30
@@ -108,41 +108,61 @@ env:
|
||||
# `loadBalancer.mode: dsr`, which preserves the client address.
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: "0"
|
||||
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
|
||||
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
|
||||
# so the first line is a wildcard for the whole zone and each later line is a specific override.
|
||||
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
|
||||
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
|
||||
# changes nothing.
|
||||
# THE LAN ANSWERS — AN EXPLICIT LIST, NOT A ZONE WILDCARD (2026-11-09). A dnsmasq
|
||||
# `address=/<domain>/<ip>` entry answers that domain AND every subdomain of it (on LABEL
|
||||
# boundaries: `pihole.` does not match `pihole-dns.`), and dnsmasq resolves overlapping entries by
|
||||
# LONGEST MATCHING DOMAIN, so the order below is irrelevant. Every name NOT listed here is
|
||||
# FORWARDED to the upstreams above and gets the PUBLIC answer.
|
||||
#
|
||||
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
|
||||
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
|
||||
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
|
||||
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
|
||||
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
|
||||
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
|
||||
# is the annotation that means "this Service publishes its own hostname at its own address":
|
||||
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
|
||||
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
|
||||
# resolver address, are the complete set of exceptions today.
|
||||
# WHY NO WILDCARD ANY MORE. Until 2026-11-09 the first entry was `address=/${BASE_DOMAIN}/
|
||||
# ${GATEWAY_VIP}`, answering the whole zone. That made sense while every public record was a
|
||||
# DNS-only CNAME to `wan.${BASE_DOMAIN}` (no NAT hairpin on this router, so the LAN NEEDED a
|
||||
# private answer). Since PR #156 every public name is a PROXIED CNAME to the Cloudflare Tunnel
|
||||
# (platform/externaldns/README.md), which works identically from the LAN — so a name that is
|
||||
# public now SHOULD resolve publicly from the LAN too, and the LAN no longer depends on PiHole for
|
||||
# it: a client that bypasses PiHole (DoH browser, phone private DNS, the router's fallback) gets
|
||||
# the same working answer. The wildcard had also become actively misleading: it answered names
|
||||
# that exist nowhere (`wan.` itself resolved to the Gateway VIP on the LAN).
|
||||
#
|
||||
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
|
||||
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
|
||||
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
|
||||
# WHAT IS LISTED, and why each — enforced by conformance/tests/l0-static/pihole-lan-names:
|
||||
# -> ${GATEWAY_VIP}, LAN-ONLY: every HTTPRoute disowned from public DNS
|
||||
# (`external-dns.alpha.kubernetes.io/controller: none`, i.e. catalog `exposure: internal`,
|
||||
# plus plex and headlamp) — jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr,
|
||||
# headlamp. These have NO public record and the tunnel 404s them, so THIS LINE IS THEIR ONLY
|
||||
# PATH: drop one and the app is unreachable everywhere. A NEWLY ONBOARDED `internal` SERVICE
|
||||
# MUST BE ADDED HERE (the L0 check fails until it is). A new `public` service needs nothing.
|
||||
# -> ${GATEWAY_VIP}, PUBLIC BUT KEPT ON THE LAN PATH: gitea (git push, LFS, container registry)
|
||||
# and nextcloud (uploads) — Cloudflare's proxy caps a request body at 100 MB and a response
|
||||
# at 100 s, which a large push or upload exceeds; keycloak and auth (oauth2-proxy) — so the
|
||||
# OIDC login of those two LAN-path apps stays on the same path as the apps themselves.
|
||||
# -> their own L4 VIPs: unifi (${UNIFI_L4_VIP}) and crafty-mc (${CRAFTY_L4_VIP}) are not behind
|
||||
# the Gateway (grep `external-dns.alpha.kubernetes.io/hostname` on a Service:
|
||||
# apps/vms/unifi/50-service-l4.yaml, platform/crafty/service-l4.yaml); their public records
|
||||
# are DNS-only A records to the same RFC1918 VIPs, which this router's rebind protection
|
||||
# strips, so the LAN answer is still required. pihole-dns (${PIHOLE_DNS_VIP}) is PiHole's own
|
||||
# resolver address; plex-vip (${PLEX_L4_VIP}) is below.
|
||||
# NOT listed, deliberately, so they resolve via Cloudflare from the LAN as from anywhere: the apex,
|
||||
# grafana, crafty, seerr. In-cluster (platform/coredns/coredns-custom.yaml) and node
|
||||
# (/etc/hosts pins, node_baseline_hosts_pins) resolution are separate and unchanged.
|
||||
#
|
||||
# AAAA: dnsmasq answers a AAAA query for an `address=`d name with NODATA (NOERROR, no answer),
|
||||
# so IPv6-capable clients fall back to the A record instead of getting NXDOMAIN — the same
|
||||
# behaviour coredns-custom hand-writes a `template IN AAAA` block for. Unlisted (proxied) names
|
||||
# now get Cloudflare's real A and AAAA answers.
|
||||
#
|
||||
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
|
||||
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
|
||||
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
|
||||
# while resolution for the zone does not work. Confirmed both ways against the live image.
|
||||
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
|
||||
# hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
|
||||
# (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
|
||||
# LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
|
||||
# the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
|
||||
# wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
|
||||
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
|
||||
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
|
||||
# every client that takes the ADVERTISED connection cannot reach the server at all.
|
||||
#
|
||||
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the name of the tcp/32400 LoadBalancer
|
||||
# (apps/wave-0/plex-remote.yaml) and the domain of the certificate Plex serves there
|
||||
# (catalog/plex.yaml CERTDOMAIN / customCertificateDomain, Secret `plex-vip-tls`). It is NOT what
|
||||
# Plex advertises any more (ADVERTISE_IP is a resolver-independent plex.direct name since
|
||||
# 2026-10-05) and it has NO public record, so this entry is its only resolution — kept for
|
||||
# reaching the VIP by name with a matching certificate from a LAN shell. Measured 2026-10-04,
|
||||
# before it existed the zone wildcard answered it with ${GATEWAY_VIP}, where nothing listens on
|
||||
# 32400; without it now, the name is NXDOMAIN.
|
||||
#
|
||||
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
|
||||
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
|
||||
@@ -172,10 +192,17 @@ env:
|
||||
# cache expires, the router falls back, and the public answer latches again. That is a real
|
||||
# outage, and the fix for it is PiHole availability, not DNS caching.
|
||||
#
|
||||
# SINCE 2026-11-09 THE LATCH IS MOSTLY HARMLESS. The measurement above predates the tunnel: the
|
||||
# public answer was then `wan.`, unreachable from the LAN. Now the public answer for every
|
||||
# proxied name is Cloudflare, which works from the LAN, and those names are not listed below at
|
||||
# all. A latch still bites only the listed names: gitea/nextcloud/keycloak/auth fall back to
|
||||
# Cloudflare (working, but under its 100 MB / 100 s limits); the LAN-only names and the L4 VIPs
|
||||
# get NXDOMAIN or a stripped answer until the router re-queries PiHole.
|
||||
#
|
||||
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
|
||||
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
|
||||
# configuration, /etc/hosts and DHCP leases), so one directive covers every listed name.
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
value: "local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
|
||||
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
|
||||
# carries the full rationale and the measurement that found it.
|
||||
|
||||
+8
-1
@@ -347,7 +347,14 @@ lifecycle:
|
||||
# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
|
||||
hardware:
|
||||
gpu: intel-quicksync
|
||||
exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
|
||||
# EXPOSURE: internal (operator decision 2026-11-08, Cloudflare Tunnel cut-over). The plex.${BASE_DOMAIN}
|
||||
# web UI is LAN/VPN-only: `internal` renders `external-dns ... controller: none` on the HTTPRoute, so
|
||||
# external-dns DELETES the public plex record and its TXT, and platform/cloudflared/config.yaml maps the
|
||||
# name to http_status:404 so it can never be served through the tunnel either. REMOTE STREAMING IS NOT
|
||||
# AFFECTED: it never used this name. It rides plex.tv + the advertised plex.direct URL (ADVERTISE_IP
|
||||
# above) to the raw tcp/32400 L4 VIP (${PLEX_L4_VIP}, apps/wave-0/plex-remote.yaml) via the WAN
|
||||
# tcp/32400 forward — none of which this field touches. See runbooks/plex-remote-access.md.
|
||||
exposure: internal # web UI LAN/VPN-only; remote streaming = plex.tv + WAN tcp/32400 (docs/09; R-DNS/R-NET)
|
||||
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
||||
storage:
|
||||
- name: config
|
||||
|
||||
@@ -39,7 +39,9 @@ spec:
|
||||
Parse platform/cloudflared/config.yaml with BASE_DOMAIN substituted, assert every rule's
|
||||
target against the Gateway Service derived from platform/gateway/gateway.yaml, assert the
|
||||
excluded hostnames 404 first, check the egress CiliumNetworkPolicy and the Flux wiring, and
|
||||
run the evaluator on mutated configs to prove it is not vacuous.
|
||||
run the evaluator on mutated configs to prove it is not vacuous; and assert the
|
||||
credentials Secret volume is readable by the container's run-as uid/groups (fsGroup +
|
||||
mode), self-tested by mutation.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -26,9 +26,19 @@
|
||||
# (5) the Flux tier is ISOLATED: its own Kustomization, wait:false, not listed in
|
||||
# platform/kustomization.yaml, nothing anywhere dependsOn it, and both overlays carry its
|
||||
# SOPS decryption patch — so a missing or invalid credential can stall nothing else;
|
||||
# AND (6) the rule evaluator is not vacuous: it is run against mutated copies of the real config
|
||||
# (6) the rule evaluator is not vacuous: it is run against mutated copies of the real config
|
||||
# (an API rule added, plex moved below the wildcard, the catch-all removed) and must
|
||||
# flag each one.
|
||||
# flag each one;
|
||||
# AND (7) every Secret volume the container mounts — the tunnel credentials above all — is
|
||||
# READABLE by the container's run-as identity, modelled the way the kubelet lays it out:
|
||||
# files owned by uid 0, group-owned by fsGroup if set (else gid 0), with the item mode or
|
||||
# defaultMode (API default 0644). The identity is the container securityContext over
|
||||
# the pod's; the process's groups are runAsGroup + fsGroup + supplementalGroups. An
|
||||
# unset runAsUser is a FAIL, not a pass: the image default cannot be read offline. The
|
||||
# model is deliberately conservative — it does not credit the kubelet's habit of
|
||||
# OR-ing group-read into fsGroup-owned volumes, so 0400+fsGroup fails here though it
|
||||
# would work. Self-tested by mutation (fsGroup removed; mode 0400; runAsUser unset).
|
||||
# This is the 2026-11-08 CrashLoop: runAsUser 65532, no fsGroup, 0440 root:root.
|
||||
#
|
||||
# NOT proven (static): that Cloudflare routes only the hostnames whose DNS points at the tunnel,
|
||||
# that the Gateway's certificate validates from cloudflared, or that Envoy enforces (4) — those are
|
||||
@@ -236,6 +246,69 @@ check(dep["spec"]["template"]["metadata"]["labels"].get("homelab.keithjackson.de
|
||||
check(dep["spec"]["template"]["spec"].get("automountServiceAccountToken") is False,
|
||||
"(4) the pods mount no ServiceAccount token")
|
||||
|
||||
# (7) the credentials Secret is readable by the run-as identity
|
||||
def unreadable_secret_volumes(pod_spec):
|
||||
"""Return a description per Secret volume the containers mount but cannot read."""
|
||||
bad = []
|
||||
psc = pod_spec.get("securityContext") or {}
|
||||
vols = {v["name"]: v for v in pod_spec.get("volumes") or [] if "secret" in v}
|
||||
for c in (pod_spec.get("initContainers") or []) + (pod_spec.get("containers") or []):
|
||||
csc = c.get("securityContext") or {}
|
||||
uid = csc.get("runAsUser", psc.get("runAsUser"))
|
||||
gid = csc.get("runAsGroup", psc.get("runAsGroup"))
|
||||
fs_group = psc.get("fsGroup")
|
||||
groups = {g for g in [gid, fs_group, *(psc.get("supplementalGroups") or [])]
|
||||
if g is not None}
|
||||
for vm in c.get("volumeMounts") or []:
|
||||
v = vols.get(vm["name"])
|
||||
if v is None:
|
||||
continue
|
||||
sec = v["secret"]
|
||||
if uid is None:
|
||||
bad.append(f"container {c['name']} mounts Secret volume {vm['name']} but sets no "
|
||||
f"runAsUser (pod or container) — readability cannot be proven offline")
|
||||
continue
|
||||
default = sec.get("defaultMode", 0o644)
|
||||
modes = [it.get("mode", default) for it in sec.get("items") or []] or [default]
|
||||
owner_gid = fs_group if fs_group is not None else 0
|
||||
for mode in modes:
|
||||
readable = ((uid == 0 and mode & 0o400) or (owner_gid in groups and mode & 0o040)
|
||||
or mode & 0o004)
|
||||
if not readable:
|
||||
bad.append(f"container {c['name']} (uid {uid}, groups {sorted(groups)}) cannot "
|
||||
f"read Secret volume {vm['name']} ({sec.get('secretName')}): files "
|
||||
f"are 0:{owner_gid} mode {mode:04o} — set pod fsGroup to a group "
|
||||
f"the container runs with, keeping 0440")
|
||||
return bad
|
||||
|
||||
|
||||
pod_spec = dep["spec"]["template"]["spec"]
|
||||
creds_vol = [v for v in pod_spec.get("volumes") or []
|
||||
if (v.get("secret") or {}).get("secretName") == "cloudflared-credentials"]
|
||||
check(len(creds_vol) == 1, "(7) the Deployment mounts Secret cloudflared-credentials as a volume")
|
||||
for v in unreadable_secret_volumes(pod_spec):
|
||||
check(False, f"(7) {v}")
|
||||
sec_mutants = {}
|
||||
m = copy.deepcopy(pod_spec)
|
||||
m["securityContext"].pop("fsGroup", None)
|
||||
m["securityContext"].pop("supplementalGroups", None)
|
||||
sec_mutants["fsGroup removed"] = m
|
||||
m = copy.deepcopy(pod_spec)
|
||||
for v in m["volumes"]:
|
||||
if "secret" in v:
|
||||
v["secret"]["defaultMode"] = 0o400
|
||||
for it in v["secret"].get("items") or []:
|
||||
it["mode"] = 0o400
|
||||
sec_mutants["mode 0400"] = m
|
||||
m = copy.deepcopy(pod_spec)
|
||||
m["securityContext"].pop("runAsUser", None)
|
||||
for c in m["containers"]:
|
||||
(c.get("securityContext") or {}).pop("runAsUser", None)
|
||||
sec_mutants["runAsUser unset"] = m
|
||||
for name, mut in sec_mutants.items():
|
||||
check(unreadable_secret_volumes(mut),
|
||||
f"(7) self-test: the readability model did NOT flag the mutant '{name}' — vacuous")
|
||||
|
||||
if CHECKS < 25:
|
||||
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
|
||||
if FAILURES:
|
||||
|
||||
+4
-3
@@ -92,9 +92,10 @@
|
||||
# the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
|
||||
# - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
|
||||
# each publishes so the claim is reviewable, which is all a static check can offer.
|
||||
# - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
|
||||
# rather than a routable address; that is an open operator decision recorded in
|
||||
# platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
|
||||
# - That the public names RESOLVE to Cloudflare at runtime. check.sh (9) pins the TREE: the
|
||||
# Gateway target is the tunnel UUID in platform/cloudflared/config.yaml, proxying is on, the L4
|
||||
# `service` publishers opt out, and every name the tunnel 404s is unpublished. Whether Cloudflare
|
||||
# then answers anycast for them is a live read (`dig @1.1.1.1`), not a static one.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
|
||||
@@ -29,7 +29,17 @@
|
||||
# ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears);
|
||||
# (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the
|
||||
# prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after
|
||||
# the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime.
|
||||
# the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime;
|
||||
# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08) — every public name is a PROXIED CNAME to the tunnel:
|
||||
# (a) the shared Gateway's `target` is EXACTLY `<tunnel:>.cfargotunnel.com`, the UUID read from
|
||||
# platform/cloudflared/config.yaml, and the HelmRelease sets `--cloudflare-proxied`;
|
||||
# (b) every `service`-source publisher carries `cloudflare-proxied: "false"` (L4 A records stay
|
||||
# DNS-only), and no HTTPRoute opts out of proxying (a DNS-only tunnel CNAME is dead);
|
||||
# (c) no hostname the tunnel maps to `http_status:404` has a public record — no undisowned route
|
||||
# names it, and the only `service`-source publishers of one are the declared L4 DNS-only
|
||||
# names (unifi, crafty-mc); conversely every disowned route's hostname IS 404'd by the
|
||||
# tunnel, so the "unpublished" and "refused at the tunnel" lists cannot drift apart;
|
||||
# (d) the operator's LAN/VPN-only names (plex, headlamp) are disowned everywhere they appear.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
|
||||
# - That the record is gone from Cloudflare. Live read; belongs beside
|
||||
@@ -39,8 +49,11 @@
|
||||
# both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
|
||||
# - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
|
||||
# This check owns the NAME axis only.
|
||||
# - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
|
||||
# routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
|
||||
# - That the tunnel actually SERVES the public names, or that Cloudflare honoured `proxied`. (9)
|
||||
# proves the tree asks for a proxied CNAME to the right tunnel; the live answer (`dig @1.1.1.1`
|
||||
# returning Cloudflare anycast) is a runtime read. Nor does (8)/(9) prove the RUNNING controller
|
||||
# honours the prefix — a future chart bump that drops it would publish PROXIED A records at the
|
||||
# Gateway VIP; (8) only keeps the tree and the HelmRelease value consistent.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
#
|
||||
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
|
||||
@@ -141,9 +154,12 @@ def docs_of(p: Path):
|
||||
# unadjudicated set cannot grow without a review event. Same ratchet reasoning as
|
||||
# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
|
||||
#
|
||||
# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
|
||||
# the honest state after this change, and it is deliberate: converting them is an operator
|
||||
# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
|
||||
# WHAT THE ENTRIES PUBLISH NOW (2026-11-08). Every HTTPRoute entry that is not disowned publishes a
|
||||
# PROXIED CNAME to the Cloudflare Tunnel (the shared Gateway's `target`, assertion 9a), so no origin
|
||||
# address of ours is in the public zone for it. Headlamp is disowned (operator: LAN/VPN-only, 9d).
|
||||
# The two `service`-source entries still publish their RFC1918 LB VIP as a DNS-only A record —
|
||||
# deliberate (L4 ports cannot ride Cloudflare's HTTP proxy), and pinned DNS-only by 9b. Whether each
|
||||
# remaining name should be public at all stays an operator decision per name, not a renderer change,
|
||||
# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
|
||||
# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
|
||||
#
|
||||
@@ -151,27 +167,28 @@ def docs_of(p: Path):
|
||||
# =============================================================================================
|
||||
KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
|
||||
"platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
|
||||
"publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
|
||||
"publishes auth.${BASE_DOMAIN} -> the tunnel (proxied CNAME). The oauth2-proxy login endpoint is a "
|
||||
"platform component, not a catalog service; it must be reachable by every browser that "
|
||||
"any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
|
||||
"whether the five public services are — an operator decision, not a renderer one.",
|
||||
"platform/keycloak/httproute.yaml:keycloak/keycloak":
|
||||
"publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
|
||||
"publishes keycloak.${BASE_DOMAIN} -> the tunnel (proxied CNAME), and it is the route that actually "
|
||||
"OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
|
||||
"the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
|
||||
"hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
|
||||
"is meant to be public either way; it is listed because the PUBLISHING object is the "
|
||||
"hand-authored platform route, which carries no exposure field.",
|
||||
"platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
|
||||
"publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
|
||||
"certainly should not be publicly resolvable, but it is not a catalog service and "
|
||||
"withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
|
||||
"internal services (Pi-hole must answer first).",
|
||||
"publishes grafana.${BASE_DOMAIN} -> the tunnel (proxied CNAME). An admin dashboard "
|
||||
"behind Keycloak; it is not a catalog service, so withdrawing it is an operator call per "
|
||||
"name (it stayed public in the 2026-11-08 tunnel decision).",
|
||||
"platform/headlamp/httproute.yaml:headlamp/headlamp":
|
||||
"publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
|
||||
"Keycloak OIDC; same adjudication as grafana.",
|
||||
"publishes NOTHING since 2026-11-08: the route carries `controller: none` (operator "
|
||||
"decision: Headlamp is LAN/VPN-only) and the tunnel 404s the name. A Kubernetes web UI "
|
||||
"with native Keycloak OIDC; listed because it is a hand-authored route, and pinned "
|
||||
"disowned by (9d).",
|
||||
"platform/homepage/httproute.yaml:homepage/homepage":
|
||||
"publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
|
||||
"publishes the APEX ${BASE_DOMAIN} itself -> the tunnel (proxied CNAME), via the https-apex "
|
||||
"listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
|
||||
"declaration.",
|
||||
"runbooks/serving/httproute.yaml:runbooks/runbooks":
|
||||
@@ -180,16 +197,16 @@ KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
|
||||
"entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
|
||||
"header for why gating runbooks on Keycloak is a loop with no exit.",
|
||||
"platform/crafty/httproute.yaml:crafty/crafty":
|
||||
"publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
|
||||
"publishes crafty.${BASE_DOMAIN} -> the tunnel (proxied CNAME) (the Minecraft admin UI, gated by an "
|
||||
"oauth2-proxy sidecar). Platform component, no exposure field.",
|
||||
"platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
|
||||
"the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
|
||||
"${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
|
||||
"Gateway cannot express. A game server is the one case where a public name may be "
|
||||
"genuinely wanted; the address it answers with is still RFC1918.",
|
||||
"genuinely wanted; the address it answers with is still RFC1918, DNS-only (9b).",
|
||||
"apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
|
||||
"the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
|
||||
"(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
|
||||
"(192.168.2.21), DNS-only (9b), for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
|
||||
"under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
|
||||
"vm-demo, which declares no ingressRoutes and publishes nothing.",
|
||||
}
|
||||
@@ -515,6 +532,113 @@ check(not bad_render,
|
||||
f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller "
|
||||
f"reads {ANNOTATION_PREFIX!r}")
|
||||
|
||||
# =============================================================================================
|
||||
# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08). The public names are PROXIED CNAMEs to the tunnel.
|
||||
# =============================================================================================
|
||||
CF_CONFIG = ROOT / "platform/cloudflared/config.yaml"
|
||||
GATEWAY_FILE = ROOT / "platform/gateway/gateway.yaml"
|
||||
TARGET_KEY = ANNOTATION_PREFIX + "target"
|
||||
PROXIED_KEY = ANNOTATION_PREFIX + "cloudflare-proxied"
|
||||
UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
|
||||
|
||||
# The L4 names the tunnel 404s that are NONETHELESS published — as DNS-only A records from the
|
||||
# `service` source, at their own LB VIP, because they are not HTTP and never touch the tunnel
|
||||
# (operator decision 2026-11-08). Shrink-only, and must agree with the tunnel's 404 list (9c).
|
||||
L4_DNS_ONLY_NAMES = {"unifi.${BASE_DOMAIN}", "crafty-mc.${BASE_DOMAIN}"}
|
||||
# The operator's LAN/VPN-only HTTP names (2026-11-08). Pinned here as well as in the tunnel config, so
|
||||
# deleting one from config.yaml cannot quietly also release it from (9c).
|
||||
OPERATOR_LAN_ONLY_NAMES = {"plex.${BASE_DOMAIN}", "headlamp.${BASE_DOMAIN}"}
|
||||
|
||||
cf = next(iter(docs_of(CF_CONFIG)), None)
|
||||
check(cf is not None and isinstance(cf.get("ingress"), list),
|
||||
f"(9) {CF_CONFIG.relative_to(ROOT)} unreadable or has no `ingress` list — cannot prove what "
|
||||
f"the tunnel refuses or which tunnel the Gateway must target")
|
||||
cf = cf or {}
|
||||
tunnel = str(cf.get("tunnel") or "")
|
||||
check(bool(UUID_RE.match(tunnel)),
|
||||
f"(9a) platform/cloudflared/config.yaml `tunnel:` is {tunnel!r}, not a literal tunnel UUID — "
|
||||
f"the Gateway target is derived from it")
|
||||
gw = next((d for d in docs_of(GATEWAY_FILE) if d.get("kind") == "Gateway"
|
||||
and ident(d) == ("gateway", "shared")), None)
|
||||
check(gw is not None, f"(9a) {GATEWAY_FILE.relative_to(ROOT)} holds no Gateway gateway/shared")
|
||||
want_target = f"{tunnel}.cfargotunnel.com"
|
||||
got_target = annotations_of(gw or {}).get(TARGET_KEY)
|
||||
check(got_target == want_target,
|
||||
f"(9a) Gateway gateway/shared `{TARGET_KEY}` is {got_target!r}; it must be EXACTLY "
|
||||
f"{want_target!r} (the tunnel in platform/cloudflared/config.yaml). Any other value publishes "
|
||||
f"every public name at the wrong place — a dead cfargotunnel.com name (Cloudflare 1016), or "
|
||||
f"an origin address of ours.")
|
||||
check("--cloudflare-proxied" in [str(a) for a in (extra if isinstance(extra, list) else [])]
|
||||
or (isinstance(extra, dict) and str(extra.get("cloudflare-proxied", "")).lower() == "true"),
|
||||
"(9a) external-dns HelmRelease extraArgs must include `--cloudflare-proxied`: a DNS-only "
|
||||
"CNAME to <uuid>.cfargotunnel.com resolves to nothing routable, so every public name is dead")
|
||||
|
||||
for rel, d in dns_services:
|
||||
if is_disowned(d):
|
||||
continue
|
||||
if HOSTNAME_KEY not in annotations_of(d):
|
||||
continue
|
||||
v = annotations_of(d).get(PROXIED_KEY)
|
||||
check(str(v).lower() == "false",
|
||||
f"(9b) {rel} Service {'/'.join(ident(d))} publishes via the `service` source but carries "
|
||||
f"`{PROXIED_KEY}: {v!r}`, not \"false\". With --cloudflare-proxied global it would become "
|
||||
f"an orange-clouded A record, which cannot carry L4 traffic and hides nothing (the LB VIP "
|
||||
f"is RFC1918). Every `service`-source publisher must opt out explicitly.")
|
||||
for rel, d in routes:
|
||||
v = annotations_of(d).get(PROXIED_KEY)
|
||||
check(v is None or str(v).lower() == "true",
|
||||
f"(9b) {rel} HTTPRoute {'/'.join(ident(d))} sets `{PROXIED_KEY}: {v!r}`. A gateway-sourced "
|
||||
f"record is a CNAME to the tunnel; DNS-only, it is dead. Disown the route instead "
|
||||
f"(`{CONTROLLER_KEY}: none`) if the name should not be public.")
|
||||
|
||||
refused = {str(r.get("hostname")) for r in cf.get("ingress") or []
|
||||
if isinstance(r, dict) and r.get("hostname") and r.get("service") == "http_status:404"}
|
||||
check(len(refused) > 0, "(9c) the tunnel config 404s no hostname — parse contract broken (the "
|
||||
"operator's LAN/VPN-only list is known to be non-empty)")
|
||||
published_l4: set[str] = set()
|
||||
for rel, d in routes:
|
||||
hosts = {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}
|
||||
leaked = sorted(hosts & refused)
|
||||
if is_disowned(d):
|
||||
unrefused = sorted(hosts - refused)
|
||||
check(not unrefused,
|
||||
f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} is disowned (no public record) but the "
|
||||
f"tunnel does not 404 {unrefused}: add them to platform/cloudflared/config.yaml above "
|
||||
f"the wildcard, or the two halves of \"LAN/VPN-only\" have drifted apart.")
|
||||
continue
|
||||
check(not leaked,
|
||||
f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} PUBLISHES {leaked}, which the tunnel maps to "
|
||||
f"http_status:404 (LAN/VPN-only). The name would be a public proxied record that serves "
|
||||
f"only 404s — disown the route (`{CONTROLLER_KEY}: none`) or adjudicate the name.")
|
||||
for rel, d in dns_services:
|
||||
if is_disowned(d):
|
||||
continue
|
||||
hosts = {h.strip() for h in str(annotations_of(d).get(HOSTNAME_KEY) or "").split(",")
|
||||
if h.strip()}
|
||||
for h in sorted(hosts & refused):
|
||||
published_l4.add(h)
|
||||
check(h in L4_DNS_ONLY_NAMES,
|
||||
f"(9c) {rel} Service {'/'.join(ident(d))} publishes {h}, which the tunnel 404s, and it "
|
||||
f"is not a declared L4 DNS-only name {sorted(L4_DNS_ONLY_NAMES)}")
|
||||
check(L4_DNS_ONLY_NAMES <= refused,
|
||||
f"(9c) L4 DNS-only names {sorted(L4_DNS_ONLY_NAMES - refused)} are not 404'd by the tunnel; a "
|
||||
f"stray CNAME to the tunnel for them would hit the Gateway's catch-all instead of a 404")
|
||||
check(published_l4 == L4_DNS_ONLY_NAMES,
|
||||
f"(9c) L4_DNS_ONLY_NAMES is shrink-only: {sorted(L4_DNS_ONLY_NAMES - published_l4)} no longer "
|
||||
f"published by any `service`-source publisher — delete them from the list")
|
||||
|
||||
check(OPERATOR_LAN_ONLY_NAMES <= refused,
|
||||
f"(9d) operator LAN/VPN-only names {sorted(OPERATOR_LAN_ONLY_NAMES - refused)} are missing "
|
||||
f"from the tunnel's http_status:404 rules (platform/cloudflared/config.yaml)")
|
||||
for name in sorted(OPERATOR_LAN_ONLY_NAMES):
|
||||
naming = [(rel, d) for rel, d in routes
|
||||
if name in {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}]
|
||||
check(len(naming) > 0, f"(9d) no HTTPRoute names {name} — stale OPERATOR_LAN_ONLY_NAMES entry")
|
||||
for rel, d in naming:
|
||||
check(is_disowned(d),
|
||||
f"(9d) {rel} HTTPRoute {'/'.join(ident(d))} names LAN/VPN-only {name} but is not "
|
||||
f"disowned, so external-dns publishes it publicly (operator decision 2026-11-08)")
|
||||
|
||||
# =============================================================================================
|
||||
if FAILURES:
|
||||
for f in FAILURES:
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — PiHole answers an explicit LAN name list, and no LAN-only name is left out of it
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# Since PR #156 every public name is a proxied CNAME to the Cloudflare Tunnel and works from the
|
||||
# LAN, so PiHole's zone wildcard was replaced by an explicit list (catalog/pihole.yaml). That makes
|
||||
# the list load-bearing: a route disowned from public DNS and absent from it resolves nowhere. This
|
||||
# test derives the required names from the tree (every `controller: none` HTTPRoute, every hostname
|
||||
# the tunnel 404s, the L4 Services' own VIPs) and asserts the list covers them, carries no
|
||||
# zone-wide entry, and pins gitea and nextcloud to the Gateway VIP. It proves itself non-vacuous by
|
||||
# mutation. Rationale and the honest NOT-proven list: check.sh.
|
||||
#
|
||||
# WHY SCOPE, NOT REQUIREMENTS: the property spans catalog/, apps/, platform/ and runbooks/; a new
|
||||
# internal service in any of them can break it, and ci/touched-requirements.sh maps apps/* to no
|
||||
# requirement at all, so a requirement-scoped selection would skip it on precisely those changes.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: pihole-lan-names
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
PiHole answers an explicit list of LAN names rather than the whole zone; every HTTPRoute
|
||||
disowned from public DNS has an entry to the Gateway VIP; every hostname the Cloudflare
|
||||
Tunnel 404s has an entry, with the L4 names on their own VIPs; and gitea and nextcloud stay
|
||||
on the LAN path, off Cloudflare's request-size and timeout limits.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-pihole-explicit-lan-names
|
||||
description: >-
|
||||
Parse FTLCONF_misc_dnsmasq_lines from catalog/pihole.yaml and the rendered
|
||||
apps/wave-0/pihole.yaml, derive the required names from every `controller: none` HTTPRoute,
|
||||
the http_status:404 hostnames in platform/cloudflared/config.yaml and the L4 Services'
|
||||
lbipam annotations, assert the list covers them and contains no zone-wide entry, and run
|
||||
the evaluator on mutated lists to prove it is not vacuous.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+269
@@ -0,0 +1,269 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — pihole-lan-names
|
||||
# Serves: R-DNS-03 (LAN split-horizon answers), and decisions/2026-11-09-pihole-explicit-lan-names.md.
|
||||
#
|
||||
# WHY THIS EXISTS. Since PR #156 every PUBLIC name is a proxied CNAME to the Cloudflare Tunnel and
|
||||
# works from the LAN, so PiHole no longer answers the whole zone with a wildcard: it answers an
|
||||
# EXPLICIT list (catalog/pihole.yaml, FTLCONF_misc_dnsmasq_lines) and forwards everything else to
|
||||
# the public resolvers. That inverts the old failure mode. With the wildcard, a forgotten name still
|
||||
# resolved; now a forgotten LAN-ONLY name — a route disowned from public DNS, which the tunnel also
|
||||
# 404s — resolves NOWHERE, and the app is silently unreachable from every client. Nothing in the
|
||||
# cluster turns red when that happens. So the list is derived from the tree and asserted here.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# THEN (0) the dnsmasq value parses (only `local-ttl=` and `address=/<domain>/<${VAR}>` entries,
|
||||
# every VAR is a cluster-var), the catalog value and the rendered apps/wave-0/pihole.yaml
|
||||
# value are identical, and the inventories below are non-empty (anti-vacuous-pass);
|
||||
# (a) NO entry covers the whole zone: no `address=/${BASE_DOMAIN}/`, no `address=/#/`, and no
|
||||
# entry for any parent of the zone (dnsmasq matches every subdomain of an entry);
|
||||
# (b) every HTTPRoute hostname disowned from public DNS
|
||||
# (`external-dns.alpha.kubernetes.io/controller: none`, anywhere in apps/ platform/
|
||||
# runbooks/) has an explicit entry to ${GATEWAY_VIP} — a disowned route with no
|
||||
# `hostnames:` is itself a FAIL, because it cannot be checked;
|
||||
# (c) every hostname platform/cloudflared/config.yaml maps to http_status:404 has an entry;
|
||||
# the L4 DNS-only names (derived: Services carrying both
|
||||
# `external-dns.alpha.kubernetes.io/hostname` and `lbipam.cilium.io/ips` — unifi,
|
||||
# crafty-mc) must map to THAT Service's VIP, not the Gateway;
|
||||
# (d) gitea and nextcloud are pinned to ${GATEWAY_VIP} (Cloudflare's 100 MB body / 100 s
|
||||
# limits break git push/LFS and large uploads);
|
||||
# AND (e) the evaluator is not vacuous: it is run against mutated copies of the real entry list
|
||||
# (zone wildcard added; a disowned host dropped; a 404'd host dropped; gitea repointed;
|
||||
# unifi repointed to the Gateway) and must flag each one.
|
||||
#
|
||||
# NOT proven (static): that the running PiHole pods carry this value (FTL applies FTLCONF_* at
|
||||
# start; the env change rolls the StatefulSet), that public resolution of the unlisted names
|
||||
# reaches Cloudflare, or the substituted IPs. Runtime check: `dig @${PIHOLE_DNS_VIP} <name>`.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
exec python3 - "$ROOT" <<'PY'
|
||||
import copy
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("CHECK FAIL (pihole-lan-names): PyYAML is required", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES, CHECKS = [], 0
|
||||
NAME = "pihole-lan-names"
|
||||
ZONE = "${BASE_DOMAIN}"
|
||||
GW = "${GATEWAY_VIP}"
|
||||
DISOWN = "external-dns.alpha.kubernetes.io/controller"
|
||||
EDNS_HOST = "external-dns.alpha.kubernetes.io/hostname"
|
||||
LB_IPS = "lbipam.cilium.io/ips"
|
||||
# Public, but must bypass Cloudflare's 100 MB request body / 100 s response limits.
|
||||
LAN_PINNED = ("gitea", "nextcloud")
|
||||
# Trees that hold applied manifests. conformance/ fixtures and .claude/ worktrees are not authored
|
||||
# surface (CLAUDE.md §0); decisions/ is prose.
|
||||
SCAN = ("apps", "platform", "runbooks")
|
||||
|
||||
|
||||
def check(cond, desc):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(desc)
|
||||
|
||||
|
||||
def docs(path):
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
|
||||
def dnsmasq_value(objs):
|
||||
"""The FTLCONF_misc_dnsmasq_lines value from a list of docs (a ServiceDefinition or manifests)."""
|
||||
found = []
|
||||
for d in objs:
|
||||
envs = list(d.get("env") or [])
|
||||
tmpl = (((d.get("spec") or {}).get("template") or {}).get("spec") or {})
|
||||
for c in tmpl.get("containers") or []:
|
||||
envs += c.get("env") or []
|
||||
found += [e.get("value") for e in envs if e.get("name") == "FTLCONF_misc_dnsmasq_lines"]
|
||||
return found
|
||||
|
||||
|
||||
def die(msg):
|
||||
print(f"CHECK FAIL ({NAME}): {msg}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
CATALOG = ROOT / "catalog" / "pihole.yaml"
|
||||
RENDERED = ROOT / "apps" / "wave-0" / "pihole.yaml"
|
||||
CF = ROOT / "platform" / "cloudflared" / "config.yaml"
|
||||
for p in (CATALOG, RENDERED, CF):
|
||||
if not p.is_file():
|
||||
die(f"{p.relative_to(ROOT)} is absent — the surface this check guards moved; re-point or "
|
||||
"delete this check in the SAME change")
|
||||
|
||||
cat_vals = dnsmasq_value(docs(CATALOG))
|
||||
ren_vals = dnsmasq_value(docs(RENDERED))
|
||||
check(len(cat_vals) == 1, f"(0) catalog/pihole.yaml declares FTLCONF_misc_dnsmasq_lines exactly "
|
||||
f"once (found {len(cat_vals)})")
|
||||
check(len(ren_vals) == 1, f"(0) apps/wave-0/pihole.yaml carries FTLCONF_misc_dnsmasq_lines exactly "
|
||||
f"once (found {len(ren_vals)})")
|
||||
if not cat_vals:
|
||||
die("no dnsmasq value to check")
|
||||
check(cat_vals[:1] == ren_vals[:1],
|
||||
"(0) the rendered apps/wave-0/pihole.yaml dnsmasq value differs from catalog/pihole.yaml — "
|
||||
"re-render pihole (catalog/render.py --no-image-automation catalog/pihole.yaml)")
|
||||
|
||||
cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
|
||||
if d.get("kind") == "ConfigMap")
|
||||
CLUSTER_VARS = set(cv["data"])
|
||||
BASE_DOMAIN_VALUE = cv["data"]["BASE_DOMAIN"]
|
||||
|
||||
ENTRY = re.compile(r"^address=/([^/]+)/(\$\{[A-Z0-9_]+\})$")
|
||||
|
||||
|
||||
def parse(value):
|
||||
"""Return (entries {domain: target}, problems)."""
|
||||
entries, problems = {}, []
|
||||
for raw in value.split(";"):
|
||||
line = raw.strip()
|
||||
if raw != line:
|
||||
problems.append(f"entry {raw!r} has surrounding whitespace (FTL writes it verbatim)")
|
||||
if not line:
|
||||
problems.append("empty entry (doubled or trailing `;`)")
|
||||
continue
|
||||
if re.fullmatch(r"local-ttl=\d+", line):
|
||||
continue
|
||||
m = ENTRY.match(line)
|
||||
if not m:
|
||||
problems.append(f"entry {line!r} is not `local-ttl=N` or `address=/<domain>/${{VAR}}` — "
|
||||
f"this check cannot reason about it")
|
||||
continue
|
||||
dom, tgt = m.groups()
|
||||
if tgt[2:-1] not in CLUSTER_VARS:
|
||||
problems.append(f"{dom} -> {tgt}: not a key of clusters/base/cluster-vars.yaml")
|
||||
if dom in entries and entries[dom] != tgt:
|
||||
problems.append(f"{dom} is listed twice with different targets")
|
||||
entries[dom] = tgt
|
||||
return entries, problems
|
||||
|
||||
|
||||
entries, problems = parse(cat_vals[0])
|
||||
for pr in problems:
|
||||
check(False, f"(0) {pr}")
|
||||
|
||||
# ── inventories, derived from the tree ──────────────────────────────────────────────────────────
|
||||
disowned, unnamed = set(), []
|
||||
l4 = {}
|
||||
for top in SCAN:
|
||||
for f in sorted((ROOT / top).rglob("*.yaml")):
|
||||
for d in docs(f):
|
||||
meta = d.get("metadata") or {}
|
||||
ann = meta.get("annotations") or {}
|
||||
if d.get("kind") == "HTTPRoute" and str(ann.get(DISOWN, "")).strip() == "none":
|
||||
hosts = (d.get("spec") or {}).get("hostnames") or []
|
||||
if not hosts:
|
||||
unnamed.append(f"{f.relative_to(ROOT)}:{meta.get('name')}")
|
||||
disowned.update(hosts)
|
||||
if d.get("kind") == "Service" and EDNS_HOST in ann and LB_IPS in ann:
|
||||
for h in str(ann[EDNS_HOST]).split(","):
|
||||
l4[h.strip()] = str(ann[LB_IPS]).strip()
|
||||
for u in unnamed:
|
||||
check(False, f"(b) disowned HTTPRoute {u} declares no `hostnames:` — it inherits a listener "
|
||||
f"hostname this check cannot see; name it explicitly")
|
||||
check(len(disowned) >= 5, f"(0) only {len(disowned)} disowned HTTPRoute hostnames found — the scan "
|
||||
f"is vacuous (the disown annotation was renamed?)")
|
||||
check(len(l4) >= 2, f"(0) only {len(l4)} L4 Services publishing their own hostname found (expected "
|
||||
f"unifi and crafty-mc) — the scan is vacuous")
|
||||
|
||||
cf = yaml.safe_load(CF.read_text())
|
||||
cf_404 = {r["hostname"] for r in cf.get("ingress") or []
|
||||
if r.get("hostname") and r.get("service") == "http_status:404"}
|
||||
check(len(cf_404) >= 5, f"(0) only {len(cf_404)} http_status:404 hostnames in the cloudflared "
|
||||
f"config — the scan is vacuous")
|
||||
|
||||
|
||||
def covers_zone(dom):
|
||||
"""True if an entry for `dom` answers every name in the zone (dom is the zone or a parent)."""
|
||||
if dom in ("#", "", "."):
|
||||
return True
|
||||
if dom == ZONE:
|
||||
return True
|
||||
# A literal parent (e.g. `keithjackson.dev`, `dev`) of the zone's substituted value.
|
||||
real = dom.replace(ZONE, BASE_DOMAIN_VALUE)
|
||||
return real == BASE_DOMAIN_VALUE or BASE_DOMAIN_VALUE.endswith("." + real)
|
||||
|
||||
|
||||
def evaluate(entries):
|
||||
bad = []
|
||||
# (a)
|
||||
for dom in entries:
|
||||
if covers_zone(dom):
|
||||
bad.append(f"(a) `address=/{dom}/` answers EVERY name in the zone — the wildcard is gone "
|
||||
f"on purpose (public names must resolve to Cloudflare from the LAN)")
|
||||
# (b)
|
||||
for h in sorted(disowned):
|
||||
if entries.get(h) != GW:
|
||||
bad.append(f"(b) {h} is disowned from public DNS but PiHole maps it to "
|
||||
f"{entries.get(h, 'NOTHING')} — expected {GW}; without it the app is "
|
||||
f"unreachable everywhere")
|
||||
# (c)
|
||||
for h in sorted(cf_404):
|
||||
if h in l4:
|
||||
if entries.get(h) != l4[h]:
|
||||
bad.append(f"(c) {h} is an L4 DNS-only name; PiHole maps it to "
|
||||
f"{entries.get(h, 'NOTHING')}, expected its own VIP {l4[h]}")
|
||||
elif h not in entries:
|
||||
bad.append(f"(c) the tunnel 404s {h} but PiHole has no entry for it — it would resolve "
|
||||
f"only publicly, and the tunnel refuses it")
|
||||
# (d)
|
||||
for n in LAN_PINNED:
|
||||
h = f"{n}.{ZONE}"
|
||||
if entries.get(h) != GW:
|
||||
bad.append(f"(d) {h} maps to {entries.get(h, 'NOTHING (public/Cloudflare)')}; it must be "
|
||||
f"pinned to {GW} — Cloudflare caps request bodies at 100 MB and responses at "
|
||||
f"100 s")
|
||||
return bad
|
||||
|
||||
|
||||
for v in evaluate(entries):
|
||||
check(False, v)
|
||||
|
||||
# (e) mutation self-test
|
||||
mutants = {}
|
||||
m = dict(entries); m[ZONE] = GW
|
||||
mutants["zone wildcard added"] = m
|
||||
m = dict(entries); m[BASE_DOMAIN_VALUE.split(".", 1)[1]] = GW
|
||||
mutants["parent-of-zone wildcard added"] = m
|
||||
m = dict(entries); m.pop(sorted(disowned)[0], None)
|
||||
mutants[f"disowned host {sorted(disowned)[0]} dropped"] = m
|
||||
pure_404 = sorted(h for h in cf_404 if h not in l4 and h not in disowned)
|
||||
if pure_404:
|
||||
m = dict(entries); m.pop(pure_404[0], None)
|
||||
mutants[f"404'd host {pure_404[0]} dropped"] = m
|
||||
m = dict(entries); m.pop(f"gitea.{ZONE}", None)
|
||||
mutants["gitea unpinned"] = m
|
||||
m = dict(entries); m[f"nextcloud.{ZONE}"] = "${UNIFI_L4_VIP}"
|
||||
mutants["nextcloud repointed"] = m
|
||||
if l4:
|
||||
h = sorted(l4)[0]
|
||||
m = dict(entries); m[h] = GW
|
||||
mutants[f"{h} repointed to the Gateway"] = m
|
||||
for name, mut in mutants.items():
|
||||
check(evaluate(mut), f"(e) self-test: the evaluator did NOT flag the mutant '{name}' — vacuous")
|
||||
check(len(mutants) >= 5, f"(e) only {len(mutants)} mutants built — the self-test degenerated")
|
||||
_, pr = parse("local-ttl=60;address=/x.${BASE_DOMAIN}/${NOT_A_CLUSTER_VAR}; address=/y/${GATEWAY_VIP}")
|
||||
check(len(pr) >= 2, "(e) self-test: the parser accepted an unknown VAR or a whitespace-padded entry")
|
||||
|
||||
if CHECKS < 15:
|
||||
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"PASS ({NAME}): {CHECKS} assertions; {len(entries)} PiHole names, {len(disowned)} disowned "
|
||||
f"routes, {len(cf_404)} tunnel-404 names, {len(l4)} L4 names")
|
||||
PY
|
||||
@@ -64,6 +64,12 @@ spec:
|
||||
annotations:
|
||||
# conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
|
||||
external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
|
||||
# DNS-ONLY (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
|
||||
# gateway names point at the Cloudflare Tunnel). Without this opt-out the probe
|
||||
# record is orange-clouded, 1.1.1.1 answers Cloudflare anycast instead of the LB IP,
|
||||
# and the exact-IP poll below can never match. Same opt-out every real
|
||||
# `service`-source publisher carries (unifi, crafty-mc).
|
||||
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
# No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: fix/cloudflared-creds-permission
|
||||
title: cloudflared CrashLoop — credentials Secret unreadable by uid 65532; add fsGroup, pin it in L0
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
- **`platform/cloudflared/deployment.yaml`**: pod `securityContext.fsGroup: 65532`. Root cause of
|
||||
the CrashLoop that followed #152/#153: the pod runs as `65532:65532` with no `fsGroup`, and the
|
||||
Secret volume is `defaultMode: 0440`. The kubelet therefore laid `credentials.json` down as
|
||||
`root:root 0440`, which uid 65532 (not owner, not in gid 0) cannot read. cloudflared logged
|
||||
`couldn't read tunnel credentials ... permission denied`, then fell back to origin-cert lookup
|
||||
and ended on the misleading `error parsing tunnel ID: Error locating origin cert`. With
|
||||
`fsGroup` the files are group-owned by 65532, readable by the container and no other uid; the
|
||||
mode stays 0440 (not loosened to 0444). Same fix as `platform/keycloak/client-secret-reconciler.yaml`.
|
||||
Still non-root, `runAsNonRoot: true`, unchanged otherwise.
|
||||
- **`conformance/tests/l0-static/cloudflared-ingress-rules/check.sh`**: new assertion (7). For each
|
||||
Secret volume a container mounts, it models kubelet ownership (uid 0, gid = fsGroup or 0, item
|
||||
mode / defaultMode / API default 0644) against the container's effective runAsUser and its groups
|
||||
(runAsGroup + fsGroup + supplementalGroups) and fails if the file is unreadable. An unset
|
||||
runAsUser fails (image default is not knowable offline). Conservative: it does not credit the
|
||||
kubelet OR-ing group-read into fsGroup volumes. Mutation self-test: fsGroup removed, mode 0400,
|
||||
runAsUser unset must each be flagged. Chainsaw step description updated to match.
|
||||
- **`runbooks/cloudflare-tunnel.md`**: new first "likely cause" (permission denied on
|
||||
credentials.json → the origin-cert error is a red herring) plus a "read the first error line"
|
||||
note; causes renumbered. New section **Rotating the tunnel credential**: create a new tunnel,
|
||||
swap the UUID and the encrypted credentials, merge, verify (new tunnel has two connectors, old
|
||||
has none), then `cloudflared tunnel cleanup`/`delete <old>`.
|
||||
- NOT touched, on purpose: `tunnel:` in `config.yaml` and `cloudflared-credentials.sops.yaml`. The
|
||||
operator is rotating the credential (it was exposed in a transcript) in a separate PR.
|
||||
|
||||
## What I verified
|
||||
|
||||
- Live (k3s-prod, read-only, 2026-11-08 ~21:38Z): Deployment pod securityContext is
|
||||
`{"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532,"seccompProfile":...}`, no
|
||||
`fsGroup`; creds volume `defaultMode: 288` (= 0440). All three pods CrashLoopBackOff (38
|
||||
restarts). Secret `cloudflared-credentials` has the one key `credentials.json` (key names only).
|
||||
- Pinned image `cloudflare/cloudflared@sha256:4955524a...` config (`skopeo inspect --config`):
|
||||
`User: 65532:65532`, `Entrypoint: [cloudflared]`, `Cmd: [version]`.
|
||||
- The new assertion FAILS (rc 1) on the pre-fix deployment.yaml with
|
||||
`container cloudflared (uid 65532, groups [65532]) cannot read Secret volume creds ... files are
|
||||
0:0 mode 0440`, and PASSES (51 assertions) with the fix. All three mutants are flagged.
|
||||
- Kyverno admission policies under `platform/kyverno/admission/` that mention securityContext
|
||||
fields are scoped to the conformance-trigger and pm-agent Jobs only; none constrains fsGroup here.
|
||||
- Toolbox gate: see the PR description for the rc and skipped stages.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That the pods actually become Ready after merge.** Not deployed by me; Flux applies it. The
|
||||
file mode inside the running container was not observed (pods crash before an exec is possible,
|
||||
and exec would be an operator action anyway). The readability claim rests on the kubelet's
|
||||
documented fsGroup behaviour and the identical, working keycloak reconciler.
|
||||
- Whether the CURRENT credential is otherwise valid (UUID/TunnelID match, tunnel exists). Moot:
|
||||
the operator is rotating it.
|
||||
- The rotation runbook section has not been executed; `cloudflared tunnel cleanup` / `delete`
|
||||
syntax is from the CLI's documented commands, not run here.
|
||||
- The deployment comment says the image ENTRYPOINT is `cloudflared --no-autoupdate`; the pinned
|
||||
image's ENTRYPOINT is plain `cloudflared`. Left unchanged (out of scope); flagged for follow-up.
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: feat/dns-via-cloudflare-tunnel
|
||||
title: Public DNS via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only, L0 guard
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
One atomic change, so the conformance guard lands with the DNS flip.
|
||||
|
||||
- `platform/gateway/gateway.yaml`: the shared Gateway's `external-dns.alpha.kubernetes.io/target`
|
||||
is now the literal `fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com` (was
|
||||
`wan.${BASE_DOMAIN}`). Comment block rewritten.
|
||||
- `platform/externaldns/helmrelease.yaml`: `extraArgs: [--cloudflare-proxied]` (global, because
|
||||
v0.23.0 reads per-object `cloudflare-proxied` from the route/Service, never the Gateway). The
|
||||
public-target comment block rewritten, including the prefix-regression risk.
|
||||
- `apps/vms/unifi/50-service-l4.yaml`, `platform/crafty/service-l4.yaml`:
|
||||
`external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"` (L4 A records stay DNS-only). They
|
||||
are the only `service`-source publishers in the tree and live (measured).
|
||||
- Plex web UI LAN/VPN-only: `catalog/plex.yaml` `exposure: public` → `internal`; `apps/wave-0/plex.yaml`
|
||||
re-rendered with the canonical command into a scratch dir and ONLY plex.yaml copied back. The
|
||||
render diff is exactly +2 lines (`annotations: external-dns.alpha.kubernetes.io/controller: none`
|
||||
on HTTPRoute plex-32400). No other golden moved, and nothing was onboarded (vm-demo/keycloak files
|
||||
were not touched). ADVERTISE_IP, the L4 VIP .24 and `plex-remote.yaml` are untouched.
|
||||
- Headlamp LAN/VPN-only: `controller: none` on the hand-authored `platform/headlamp/httproute.yaml`.
|
||||
- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh`: header and
|
||||
register prose updated; new assertion (9):
|
||||
(9a) Gateway target == `<tunnel: from platform/cloudflared/config.yaml>.cfargotunnel.com`, and
|
||||
`--cloudflare-proxied` is set; (9b) every `service`-source publisher has `cloudflare-proxied:
|
||||
"false"` and no HTTPRoute opts out of proxying; (9c) no undisowned route names a hostname the
|
||||
tunnel 404s, the only `service` publishers of a 404'd name are the declared L4 set
|
||||
{unifi, crafty-mc}, and every disowned route's hostname IS 404'd by the tunnel (lists kept
|
||||
consistent both ways); (9d) plex and headlamp pinned disowned. Its `chainsaw-test.yaml` gap prose
|
||||
updated.
|
||||
- `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml`: fixture Service gets
|
||||
`cloudflare-proxied: "false"`, else the global flag proxies it and the exact-IP poll fails.
|
||||
- Docs: `platform/externaldns/ddns.yaml` header (wan. is WireGuard-only), `platform/externaldns/README.md`,
|
||||
`platform/gateway/README.md`, `runbooks/plex-remote-access.md`, `runbooks/cloudflare-tunnel.md`
|
||||
(Phase 2 done; operator step to close WAN tcp/80+443 and keep udp/51820 + tcp/32400; rotation
|
||||
now also swaps the Gateway UUID; rollback path).
|
||||
|
||||
## What I verified
|
||||
|
||||
- external-dns v0.23.0 source, read at the tag: `source/gateway.go` takes target overrides from
|
||||
`gw.Annotations` and provider-specific (incl. `cloudflare-proxied`) from the ROUTE's annotations;
|
||||
`annotations.SetAnnotationPrefix` re-derives `CloudflarePrefix`, so the alpha-prefixed key is the
|
||||
one read here; `provider/cloudflare` `shouldBeProxied` starts from `--cloudflare-proxied` and a
|
||||
parsed `"false"` annotation overrides it.
|
||||
- (8b) only forbids `annotation-prefix` / `enable-legacy-annotation-prefix` in extraArgs; a mutation
|
||||
confirmed `--cloudflare-proxied` passes and `--annotation-prefix=x/` fails.
|
||||
- The L0 check passes (116 assertions) and each new assertion FAILS on its mutation: Gateway target
|
||||
back to wan (9a), flag removed (9a), unifi proxied "true" (9b), headlamp undisowned (9c,9d), plex
|
||||
golden undisowned (9c,9d,(1)), jackett removed from the tunnel's 404 list (9c).
|
||||
`cloudflared-ingress-rules` still passes; `render.py --selftest` PASS.
|
||||
- Live, read-only (kubectl k3s-prod, ~23:55Z): 17 HTTPRoutes; the six internal apps carry
|
||||
`controller: none`; plex and headlamp do not (yet); the only Services with an external-dns
|
||||
hostname are crafty/crafty-l4-tcp and unifi/unifi-l4-tcp; Gateway target is currently
|
||||
`wan.home.keithjackson.dev`.
|
||||
- Toolbox gate result: see the PR description (rc and skipped stages quoted there).
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- Anything after merge: that external-dns actually UPDATEs the eight public names to proxied CNAMEs
|
||||
to the tunnel, DELETEs plex + headlamp and their TXT, and leaves unifi/crafty-mc/wan untouched.
|
||||
Expected effect is listed in the PR; the PM must read external-dns logs and `dig @1.1.1.1`.
|
||||
- A rendered HelmRelease: I read the chart 1.23.0 `templates/deployment.yaml` (a list `extraArgs`
|
||||
is emitted item-by-item via `tpl . $ | quote`), but did not `helm template` it.
|
||||
- kubectl OIDC from nodes: nodes resolve publicly, so `keycloak.` discovery/JWKS now goes through
|
||||
Cloudflare + the tunnel. No `oidc-ca-file` is set (system roots), so the edge cert should validate,
|
||||
but Cloudflare bot/WAF settings on that path are untested.
|
||||
- RISK: if the annotation prefix stops being honoured at runtime again (11-07 incident class),
|
||||
`target` is ignored and every name becomes a PROXIED A record at 192.168.2.20, while internal
|
||||
names publish. Assertion (8) only keeps the tree and `values.annotationPrefix` consistent; it
|
||||
cannot catch a controller that ignores the flag. No runtime alert exists for it.
|
||||
- The router WAN 80/443 close is an operator step, not done. The Pi-hole wildcard trim is a later PR.
|
||||
- Follow-up: spec requirement IDs for this (proposed R-DNS-04/05, R-SEC-04) once the spec ADR merges;
|
||||
`platform/pm-agent/README.md` and a comment in `catalog/pihole.yaml` still describe the wan. CNAME.
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
date: 2026-11-09
|
||||
branch: feat/pihole-explicit-lan-names
|
||||
title: PiHole answers an explicit LAN name list instead of a zone wildcard (public names resolve to Cloudflare from the LAN)
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**`catalog/pihole.yaml` → `apps/wave-0/pihole.yaml`.** `FTLCONF_misc_dnsmasq_lines` no longer carries
|
||||
`address=/${BASE_DOMAIN}/${GATEWAY_VIP}`. Every name not listed is now forwarded to 8.8.8.8/8.8.4.4 and
|
||||
gets the public answer. The final list (`local-ttl=60` kept):
|
||||
|
||||
| Name (`.${BASE_DOMAIN}`) | Answer | Why |
|
||||
|---|---|---|
|
||||
| jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr, headlamp | `${GATEWAY_VIP}` | LAN-only: every HTTPRoute with `external-dns.alpha.kubernetes.io/controller: none`, derived from the tree. No public record, tunnel 404s them; this entry is their only path |
|
||||
| gitea, nextcloud | `${GATEWAY_VIP}` | public, but Cloudflare's proxy caps bodies at 100 MB / responses at 100 s (git push, LFS, uploads) |
|
||||
| keycloak, auth | `${GATEWAY_VIP}` | keep the OIDC flow of the LAN-path apps on one path |
|
||||
| unifi | `${UNIFI_L4_VIP}` | L4 VIP, not behind the Gateway |
|
||||
| crafty-mc | `${CRAFTY_L4_VIP}` | L4 VIP, not behind the Gateway |
|
||||
| pihole-dns | `${PIHOLE_DNS_VIP}` | the resolver's own address |
|
||||
| plex-vip | `${PLEX_L4_VIP}` | kept, see below |
|
||||
|
||||
Not listed, so they resolve to Cloudflare from the LAN: the apex, grafana, crafty, seerr.
|
||||
|
||||
**The derived LAN-only set matches the brief's list exactly** (8 names; the tree was walked, not the
|
||||
list trusted). The `runbooks/serving` route has no `hostnames:` but is not disowned (it inherits the
|
||||
apex, public).
|
||||
|
||||
**plex-vip is kept.** Nothing advertises it any more (`ADVERTISE_IP` is a `plex.direct` name since
|
||||
2026-10-05), but it is not unused: it is Plex's `customCertificateDomain` (`CERTDOMAIN`, Secret
|
||||
`plex-vip-tls`), the name of the tcp/32400 VIP, and the 404'd name in the tunnel config. It has no
|
||||
public record, so this entry is its only resolution.
|
||||
|
||||
**New L0 check `conformance/tests/l0-static/pihole-lan-names`** (`scope: whole-repo`, so picked up by
|
||||
the runner's scope selection with no other wiring). Asserts: (a) no zone-wide entry (zone, `#`, or any
|
||||
parent of the zone); (b) every `controller: none` hostname → `${GATEWAY_VIP}`, and a disowned route
|
||||
with no `hostnames:` fails; (c) every tunnel-404 hostname has an entry, with the L4 names (derived
|
||||
from Services carrying both the external-dns hostname and an lbipam VIP) on their own VIP; (d) gitea
|
||||
and nextcloud → `${GATEWAY_VIP}`; plus catalog == rendered value and a built-in mutation self-test.
|
||||
|
||||
**Stale docs corrected:** `platform/pm-agent/README.md`, `runbooks/pm-agent.md`,
|
||||
`platform/externaldns/README.md`, `platform/cloudflared/README.md`, `platform/gateway/gateway.yaml`
|
||||
(comment), `platform/headlamp/httproute.yaml` (comment), `apps/vms/unifi/README.md`, `README.md`,
|
||||
`schemas/renderer-contract.md`; `runbooks/cloudflare-tunnel.md` gains the two consequences below.
|
||||
`platform/cloudflared/deployment.yaml`: the image entrypoint comment was wrong (comment-only).
|
||||
|
||||
## What I verified
|
||||
|
||||
- **Render:** the baseline `render.py --no-image-automation catalog/pihole.yaml` was byte-identical to
|
||||
the committed `apps/wave-0/pihole.yaml` before the change, and the file was replaced by that
|
||||
command's output after it; `--emit-waves` was not run.
|
||||
- **Mutation, against the real files (10 mutants, each rc 1 with the right assertion, each restored):**
|
||||
(a) zone wildcard restored; (b) headlamp and sonarr entries removed; (b) a disowned route with no
|
||||
`hostnames:`; (c) a new tunnel-404 host with no entry; (c) pihole-dns removed; (c) unifi pointed at
|
||||
the Gateway; (d) gitea removed; (d) nextcloud repointed; (0) rendered value drifted from the catalog.
|
||||
- **Rollout mechanics, measured live (read-only):** StatefulSet `updateStrategy` RollingUpdate
|
||||
partition 0, `podManagementPolicy: Parallel`, no `minReadySeconds`, 2 replicas, PDB minAvailable 1.
|
||||
The last rollout (2026-11-08 06:47–06:48Z) replaced pihole-1 then pihole-0: FTL logged "started" ~1 s
|
||||
after the container, the pod went Ready ~16 s after the container, and pihole-0 started ~9 s later.
|
||||
- **Expected LAN DNS blip:** no total outage; one replica answers throughout. Each of the two pod
|
||||
replacements can lose a few UDP queries for ~1–2 s while the terminating pod leaves the Service
|
||||
(clients retry). For roughly 30–60 s the two pods answer differently for the apex, grafana, crafty
|
||||
and seerr; both answers work. Cached answers age out in 60 s (`local-ttl`).
|
||||
- **Live DNS before the change:** @8.8.8.8 returns Cloudflare (104.21.51.91 / 172.67.178.27) for the
|
||||
apex, grafana, seerr, crafty and gitea, and nothing for jackett; @192.168.2.23 returns 192.168.2.20
|
||||
for all of them (the wildcard).
|
||||
- **Cloudflared image:** `skopeo inspect --config` on the pinned digest (amd64): Entrypoint
|
||||
`["cloudflared"]`, Cmd `["version"]`, User `65532:65532`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The post-merge behaviour.** Nothing here ran against a live PiHole with the new list; the
|
||||
expected answers are from dnsmasq semantics and the tree. After reconcile, check
|
||||
`dig @192.168.2.23` for one name per row, and grafana returning Cloudflare.
|
||||
- **Whether every LAN client path works via Cloudflare.** Measured 2xx/3xx through Cloudflare by the
|
||||
brief, not re-measured from a LAN client here.
|
||||
- **The FTL DNS-listening moment.** "FTL started" is a log line, not a successful query; the rollout
|
||||
timing above assumes DNS is up shortly after it.
|
||||
|
||||
**Accepted consequences, stated plainly:**
|
||||
- The apex, grafana, crafty and seerr now depend on the internet and the tunnel from the LAN too. An
|
||||
ISP or Cloudflare outage, or stopping the connectors, makes them unreachable from the LAN where they
|
||||
previously kept working. If grafana in particular should survive an internet outage, add it to the
|
||||
list (one entry; the check permits it).
|
||||
- The tunnel rollback (revert to `wan.`) now also requires restoring the wildcard; recorded in
|
||||
`runbooks/cloudflare-tunnel.md`.
|
||||
|
||||
**Flagged, not changed:**
|
||||
- The `pihole` container has no readiness probe; only the `oauth2-proxy` sidecar does, so "Ready" says
|
||||
nothing about DNS. The measured margin (FTL ~1 s, Ready ~16 s) makes this benign in a normal rollout
|
||||
but a slow FTL start would not hold the rollout back. A DNS readiness probe needs a schema/renderer
|
||||
decision.
|
||||
|
||||
**Stale statements left alone:** `platform/coredns/coredns-custom.yaml` header ("both resolve the whole
|
||||
zone ... by wildcard", brief said not to touch CoreDNS); `clusters/base/cluster-vars.yaml` comments
|
||||
(sensitive path); `schemas/renderer-contract.md` §`exposure` first bullets (manual edge port-forward
|
||||
wording); historical `infra/PROD-PROVISIONING.md`, `platform/externaldns/README.md` TODO list.
|
||||
@@ -17,6 +17,8 @@ outbound-only from the cluster (no router port forward, no hairpin, no split-DNS
|
||||
| `networkpolicy.yaml` | default-deny (single owner — not an app namespace) + Prometheus scrape allow + egress CNP: world 7844 tcp/udp and 443, the Gateway (`ingress` entity), and Gateway backends by `gateway-access=shared` minus the LAN-only namespaces. |
|
||||
| `servicemonitor.yaml`, `alerts.yaml` | Headless metrics Service + ServiceMonitor scraping :2000, and the two alerts, both written to fire (not go quiet) when their series vanish. |
|
||||
|
||||
**Phase 1 only.** This component changes no live traffic: Cloudflare sends a hostname through the
|
||||
tunnel only when its DNS record is a CNAME to `<UUID>.cfargotunnel.com`, and nothing here creates
|
||||
one. Moving DNS (external-dns, Gateway annotations, Pi-hole/CoreDNS split horizon) is a later change.
|
||||
Cloudflare sends a hostname through the tunnel only when its DNS record is a proxied CNAME to
|
||||
`<UUID>.cfargotunnel.com`; nothing in this directory creates one. Those records are published by
|
||||
external-dns from the shared Gateway's target annotation (Phase 2, PR #156; see
|
||||
`platform/externaldns/README.md`). LAN resolution: `catalog/pihole.yaml` (explicit names only since
|
||||
2026-11-09), so public names reach Cloudflare from the LAN too.
|
||||
@@ -47,6 +47,14 @@ spec:
|
||||
# The image is distroless `nonroot` (65532).
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
# fsGroup makes the kubelet GROUP-own the Secret volume's files to 65532. Without it they
|
||||
# are root:root, and defaultMode 0440 (below) is unreadable to uid 65532 — every replica
|
||||
# CrashLooped on "couldn't read tunnel credentials ... permission denied" (2026-11-08),
|
||||
# which cloudflared then misreports as "Error locating origin cert". Same lesson as
|
||||
# platform/keycloak/client-secret-reconciler.yaml. fsGroup + 0440 keeps the credential
|
||||
# unreadable to any other uid rather than relaxing it to world-readable 0444.
|
||||
# Pinned by the L0 check cloudflared-ingress-rules, assertion (7).
|
||||
fsGroup: 65532
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
topologySpreadConstraints:
|
||||
@@ -62,8 +70,9 @@ spec:
|
||||
- name: cloudflared
|
||||
# Tag + digest: Renovate's `kubernetes` manager (renovate.json5, platform/**) bumps both.
|
||||
image: docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba
|
||||
# The image ENTRYPOINT is `cloudflared --no-autoupdate`; these replace its CMD. The tunnel
|
||||
# UUID, credentials path, metrics address and ingress rules all come from the file.
|
||||
# The image ENTRYPOINT is `cloudflared` and its CMD is `version` (read from the pinned
|
||||
# digest's image config, 2026-11-09); these args replace that CMD. `no-autoupdate`, the
|
||||
# tunnel UUID, credentials path, metrics address and ingress rules all come from the file.
|
||||
args:
|
||||
- tunnel
|
||||
- --config
|
||||
@@ -121,6 +130,8 @@ spec:
|
||||
secret:
|
||||
# cloudflared-credentials.sops.yaml — the tunnel credentials JSON, by name only.
|
||||
secretName: cloudflared-credentials
|
||||
# 0440, owner root, group fsGroup 65532 (pod securityContext) — readable by the
|
||||
# container user through its group, by nobody else.
|
||||
defaultMode: 0440
|
||||
---
|
||||
# One replica may be disrupted at a time — enough for kured's one-node-at-a-time drain to
|
||||
|
||||
@@ -101,6 +101,11 @@ metadata:
|
||||
lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
|
||||
# external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target).
|
||||
external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
|
||||
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
|
||||
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
|
||||
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
|
||||
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
|
||||
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
|
||||
@@ -20,29 +20,49 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
|
||||
its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
|
||||
Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.
|
||||
|
||||
## Public target + in-cluster DDNS (2026-10-03)
|
||||
## Public target: the Cloudflare Tunnel, proxied (2026-11-08)
|
||||
|
||||
Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
|
||||
Gateway-published records are **proxied CNAMEs to the Cloudflare Tunnel**
|
||||
(`platform/cloudflared`), not the WAN IP and not the Gateway VIP:
|
||||
|
||||
```
|
||||
plex.home.keithjackson.dev CNAME wan.home.keithjackson.dev <- external-dns (Gateway `target` annotation)
|
||||
wan.home.keithjackson.dev A <current WAN IP> <- cloudflare-ddns (ddns.yaml), every 5m
|
||||
gitea.home.keithjackson.dev CNAME (proxied) fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com <- external-dns
|
||||
unifi.home.keithjackson.dev A (DNS-only) 192.168.2.21 <- external-dns, `service` source (L4)
|
||||
crafty-mc.home... A (DNS-only) 192.168.2.22 <- external-dns, `service` source (L4)
|
||||
wan.home.keithjackson.dev A (DNS-only) <current WAN IP> <- cloudflare-ddns (ddns.yaml): WireGuard udp/51820 ONLY
|
||||
```
|
||||
|
||||
- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
|
||||
applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
|
||||
nothing.
|
||||
- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
|
||||
It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
|
||||
`egress-class: vpn`.
|
||||
- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
|
||||
(split-horizon) and never use the public path.
|
||||
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
|
||||
OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
|
||||
the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
|
||||
- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`); external-dns
|
||||
reads it from the Gateway only, so it applies to every route that publishes at all. Its UUID must
|
||||
equal `tunnel:` in `platform/cloudflared/config.yaml` (L0 check
|
||||
`internal-services-publish-no-public-dns` (9a)).
|
||||
- `--cloudflare-proxied` (HelmRelease `extraArgs`) orange-clouds every record by default — the only
|
||||
practical lever, because v0.23.0 reads the per-object `cloudflare-proxied` annotation from the
|
||||
route/Service, never the Gateway. **Every `service`-source publisher must carry
|
||||
`external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"`** (L4 ports cannot ride the HTTP
|
||||
proxy); (9b) fails the gate otherwise.
|
||||
- `internal` routes (`controller: none`) still publish nothing. Plex's web UI and Headlamp joined that
|
||||
set on 2026-11-08 (LAN/VPN-only); the tunnel also 404s every such name (9c/9d).
|
||||
- Public answers are Cloudflare anycast **A and AAAA** — Cloudflare's edge, not an origin address.
|
||||
- **Pods** resolve the whole zone to the Gateway VIP via CoreDNS (split-horizon) and never use the
|
||||
public path. **LAN clients** use PiHole, which since 2026-11-09 answers an EXPLICIT list
|
||||
(`catalog/pihole.yaml`): the LAN-only names, plus gitea/nextcloud/keycloak/auth (kept off
|
||||
Cloudflare's 100 MB / 100 s limits and on one OIDC path), go to the Gateway VIP and the L4 names to
|
||||
their own VIPs; every other public name (apex, grafana, crafty, seerr) resolves to Cloudflare from
|
||||
the LAN exactly as from anywhere else, so no split-DNS is needed for them. Guard:
|
||||
`conformance/tests/l0-static/pihole-lan-names`.
|
||||
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node the apiserver's
|
||||
OIDC issuer `keycloak.home.keithjackson.dev` now resolves to Cloudflare and its discovery/JWKS
|
||||
fetches go through the tunnel (publicly trusted edge cert; no `oidc-ca-file` is configured). The
|
||||
router NAT hairpin is no longer on that path.
|
||||
|
||||
Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
|
||||
`kubectl -n external-dns logs deploy/cloudflare-ddns`.
|
||||
Check: `dig +short @1.1.1.1 gitea.home.keithjackson.dev` → Cloudflare anycast addresses (no CNAME is
|
||||
visible for a proxied record); `kubectl -n external-dns logs deploy/external-dns`.
|
||||
|
||||
### History
|
||||
|
||||
2026-10-03 → 2026-11-08: records were DNS-only CNAMEs to `wan.home.keithjackson.dev`, with the router
|
||||
forwarding WAN tcp/80+443 to the Gateway VIP. Before 2026-10-03 they answered the Gateway VIP itself.
|
||||
|
||||
## Secrets
|
||||
|
||||
|
||||
@@ -2,9 +2,15 @@
|
||||
# In-cluster dynamic DNS for the home WAN (2026-10-03).
|
||||
#
|
||||
# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
|
||||
# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
|
||||
# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
|
||||
# moves all of them through this one record, with no Git change and no external-dns involvement.
|
||||
# current public IP.
|
||||
#
|
||||
# WIREGUARD-ONLY SINCE 2026-11-08. From 2026-10-03 every public name external-dns published from the
|
||||
# shared Gateway was a CNAME to this record. They are now proxied CNAMEs to the Cloudflare Tunnel
|
||||
# (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so NOTHING in the zone
|
||||
# points here any more. The record stays for exactly one consumer: the WireGuard VPN endpoint
|
||||
# (udp/51820, platform/vpn-ingress, infra/vpn-ingress/new-peer.sh), which cannot ride Cloudflare's
|
||||
# HTTP proxy and needs the real WAN address. It must stay DNS-only (`PROXIED` unset/false below) —
|
||||
# an orange-clouded A record would hand peers a Cloudflare anycast address that drops UDP.
|
||||
#
|
||||
# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
|
||||
# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
|
||||
@@ -64,7 +70,7 @@ spec:
|
||||
- name: IP6_PROVIDER
|
||||
value: "none" # IPv4 WAN only; leaves any AAAA alone
|
||||
- name: PROXIED
|
||||
value: "false" # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
|
||||
value: "false" # DNS-only: WireGuard udp/51820 needs the real WAN IP
|
||||
- name: UPDATE_CRON
|
||||
value: "@every 5m"
|
||||
- name: DELETE_ON_STOP
|
||||
|
||||
@@ -107,29 +107,47 @@ spec:
|
||||
# what this instance can see or own; per-resource exclusion only ever REMOVES an object from
|
||||
# this instance's view, so it cannot make prod capable of deleting a record it does not own.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
|
||||
# THE PUBLIC TARGET — THE CLOUDFLARE TUNNEL, PROXIED (2026-11-08; supersedes 2026-10-03)
|
||||
#
|
||||
# Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
|
||||
# RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
|
||||
# fix, and both still hold:
|
||||
# * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
|
||||
# from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
||||
# gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
|
||||
# shared Gateway therefore moves EVERY gateway-sourced record at once.
|
||||
# * The home WAN IP is residential and can change, so a literal IP in Git would rot.
|
||||
# History: until 2026-10-03 every gateway-sourced record answered the Gateway VIP (RFC1918);
|
||||
# from 2026-10-03 they were CNAMEs to `wan.${BASE_DOMAIN}` (DNS-only, the home WAN IP, with the
|
||||
# router forwarding tcp/80+443 to the VIP). Both published an origin address of ours. Now:
|
||||
# * platform/gateway/gateway.yaml sets `target: <tunnel-uuid>.cfargotunnel.com` on the shared
|
||||
# Gateway. `target` is read in v0.23.0's gateway-httproute source ONLY from the GATEWAY
|
||||
# (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(gw.Annotations)`),
|
||||
# never from an HTTPRoute, so this one value moves every gateway-sourced record at once.
|
||||
# * `--cloudflare-proxied` below orange-clouds every record by default. It is GLOBAL because
|
||||
# that is the only practical lever: v0.23.0 reads the per-object `cloudflare-proxied`
|
||||
# annotation from the ROUTE (or Service), never the Gateway (source/gateway.go:
|
||||
# `annotations.ProviderSpecificAnnotations(annots)` with `annots := meta.Annotations` of the
|
||||
# route), so a Gateway-level opt-in does not exist. A CNAME to cfargotunnel.com MUST be
|
||||
# proxied — DNS-only it resolves to nothing routable — so "proxied by default" is the right
|
||||
# default for this zone's gateway names.
|
||||
# * The provider honours a per-object `"false"` over the global default (provider/cloudflare
|
||||
# shouldBeProxied: ParseBool of the annotation overrides proxiedByDefault). The key it reads
|
||||
# is `<annotationPrefix>cloudflare-proxied` (annotations.SetAnnotationPrefix re-derives
|
||||
# CloudflarePrefix), i.e. `external-dns.alpha.kubernetes.io/cloudflare-proxied` here.
|
||||
#
|
||||
# THE `service` SOURCE MUST OPT OUT. The L4 LoadBalancers (crafty-mc, unifi) publish an A record
|
||||
# at their own LB VIP; Cloudflare's proxy is HTTP(S)-only, so a proxied A for a game or UniFi
|
||||
# inform port is broken by construction. Each carries `cloudflare-proxied: "false"` and stays
|
||||
# DNS-only, exactly as before this change. conformance/tests/l0-static/
|
||||
# internal-services-publish-no-public-dns (assertion 9b) FAILS on any `service`-source publisher
|
||||
# without it, so a new L4 Service cannot silently become a proxied record.
|
||||
#
|
||||
# The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
|
||||
# UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
|
||||
# (b) the address is kept current IN-CLUSTER. Implementation:
|
||||
# * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
|
||||
# each published name becomes a CNAME to that one hostname;
|
||||
# * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
|
||||
# record at the current WAN IP. That record has no external-dns TXT registry entry, so this
|
||||
# instance never owns or garbage-collects it.
|
||||
# Internal services are unchanged: their routes carry `controller: none` (above), which is
|
||||
# checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
|
||||
# LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
|
||||
# LB VIP. Whether those should go public is a separate, still-open question.
|
||||
# checked BEFORE the target is read, so they still publish nothing. Plex's web UI and Headlamp
|
||||
# joined that set on 2026-11-08 (operator: LAN/VPN-only). `wan.${BASE_DOMAIN}` (ddns.yaml) has no
|
||||
# external-dns TXT registry entry and is untouched; it now serves only WireGuard udp/51820.
|
||||
#
|
||||
# THE RISK THIS ADDS, AND WHAT GUARDS IT. If the annotation prefix ever stops being honoured
|
||||
# again (the 2026-11-07 v0.23.0 incident), `target` is ignored and every name falls back to the
|
||||
# Gateway's own address — and with this flag it would be published as a PROXIED A record at
|
||||
# ${GATEWAY_VIP} (RFC1918; Cloudflare 1003/522s for every public name) while `controller: none`
|
||||
# is also ignored and internal names publish. Assertion (8) of the L0 check above ties the
|
||||
# prefix this tree writes to `values.annotationPrefix`; (8b) forbids overriding it here.
|
||||
extraArgs:
|
||||
- --cloudflare-proxied
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
|
||||
#
|
||||
|
||||
@@ -30,5 +30,9 @@ east-west default-deny (docs/09).
|
||||
|
||||
- **Phase 2 (WS-NET):** pin the L2-announced ingress VIP / IP pool (Cilium L2 announcement, docs/09);
|
||||
add the HTTP→HTTPS redirect route; confirm the cross-namespace ReferenceGrant pattern with WS-APPS.
|
||||
- **Public exposure** is a **manual edge port-forward** of 80/443 to the Gateway VIP (single SOHO
|
||||
router, docs/09) — a documented manual step, the analogue of the old pfSense exception.
|
||||
- **Public exposure** is via the **Cloudflare Tunnel** (2026-11-08, `platform/cloudflared`): the
|
||||
Gateway's `external-dns.alpha.kubernetes.io/target` is `<tunnel-uuid>.cfargotunnel.com`, so every
|
||||
published name is a proxied CNAME to the tunnel and cloudflared forwards to this Gateway's
|
||||
ClusterIP. No inbound WAN 80/443 forward is needed (the old manual edge port-forward is retired —
|
||||
runbooks/cloudflare-tunnel.md). LAN/VPN-only names (internal apps, plex, headlamp) carry
|
||||
`controller: none` on their route and are 404'd by the tunnel.
|
||||
@@ -8,22 +8,36 @@ metadata:
|
||||
name: shared
|
||||
namespace: gateway
|
||||
annotations:
|
||||
# PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
|
||||
# route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
|
||||
# unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
|
||||
# becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
|
||||
# pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
|
||||
# PUBLIC DNS TARGET = THE CLOUDFLARE TUNNEL (2026-11-08, operator decision; replaces the
|
||||
# 2026-10-03 `wan.${BASE_DOMAIN}` target). external-dns v0.23.0's gateway-httproute source reads
|
||||
# `target` from the GATEWAY only (source/gateway.go: `overrides: annotations.
|
||||
# TargetsFromTargetAnnotation(gw.Annotations)`) — never from an HTTPRoute — so this one value moves
|
||||
# EVERY published route on this Gateway at once. Each published `<svc>.${BASE_DOMAIN}` (and the
|
||||
# apex) becomes a CNAME to `<tunnel-uuid>.cfargotunnel.com`, and platform/externaldns/helmrelease.yaml
|
||||
# sets `--cloudflare-proxied` so the CNAME is orange-clouded: public clients resolve Cloudflare
|
||||
# anycast, Cloudflare terminates edge TLS (Total TLS) and hands the request to the in-cluster
|
||||
# cloudflared connectors (platform/cloudflared), which forward to this Gateway's ClusterIP. No
|
||||
# origin address of ours appears in public DNS, and no inbound WAN port is needed for HTTP(S).
|
||||
#
|
||||
# Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
|
||||
# admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
|
||||
# `exposure: public` apps. `internal` services stay unpublished via their
|
||||
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
|
||||
# BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
|
||||
# from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
|
||||
# THE UUID IS A LITERAL, AND IT MUST EQUAL `tunnel:` IN platform/cloudflared/config.yaml. A CNAME
|
||||
# to any other <uuid>.cfargotunnel.com is a dead name (Cloudflare answers 1016). That equality is
|
||||
# asserted on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns
|
||||
# (assertion 9a). A literal rather than a cluster-var keeps this change off clusters/**.
|
||||
#
|
||||
# LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
|
||||
# ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
|
||||
external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
|
||||
# Scope: every route on this Gateway that publishes at all. `internal` services (and the
|
||||
# operator-designated LAN/VPN-only names: plex, headlamp) stay unpublished via their
|
||||
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns
|
||||
# applies BEFORE it reads this target. Those same names map to `http_status:404` in the tunnel's
|
||||
# ingress rules, so even a hand-made record cannot reach them through the tunnel.
|
||||
#
|
||||
# In-cluster clients are unaffected: CoreDNS answers ${BASE_DOMAIN} with ${GATEWAY_VIP} directly
|
||||
# (split-horizon). LAN clients use PiHole's EXPLICIT list (catalog/pihole.yaml, since 2026-11-09):
|
||||
# LAN-only names and gitea/nextcloud/keycloak/auth get ${GATEWAY_VIP}; every other published
|
||||
# name takes this public path through Cloudflare, from the LAN as from anywhere.
|
||||
#
|
||||
# `wan.${BASE_DOMAIN}` (platform/externaldns/ddns.yaml) is NO LONGER a target of anything here;
|
||||
# it stays only as the WireGuard udp/51820 endpoint.
|
||||
external-dns.alpha.kubernetes.io/target: "fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com"
|
||||
spec:
|
||||
gatewayClassName: cilium
|
||||
listeners:
|
||||
|
||||
@@ -18,6 +18,13 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: oidc
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
# LAN/VPN-ONLY (operator decision 2026-11-08). Disowned from external-dns, so the public
|
||||
# headlamp record and its TXT are DELETED and nothing republishes them; LAN/VPN clients keep
|
||||
# resolving the name via PiHole's explicit entry for it (-> the Gateway VIP). The Cloudflare
|
||||
# Tunnel also maps headlamp.${BASE_DOMAIN} to http_status:404 (platform/cloudflared/config.yaml).
|
||||
# Same mechanism the renderer emits for `exposure: internal` services; asserted by
|
||||
# conformance/tests/l0-static/internal-services-publish-no-public-dns (9c).
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -72,11 +72,14 @@ Both are deliberate, and both fail **visibly** rather than silently:
|
||||
- **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
|
||||
the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
|
||||
*node* process, so it resolves through the node's public upstreams
|
||||
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
|
||||
answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
|
||||
IP — which this router does not hairpin; it answers :443 with its own placeholder
|
||||
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. At the time,
|
||||
the public answer for every name in this zone was a CNAME to `wan.home.keithjackson.dev`
|
||||
— the WAN IP — which this router does not hairpin; it answered :443 with its own placeholder
|
||||
certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
|
||||
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
|
||||
`x509: certificate is not valid for any names`. (Since 2026-11-08 the public answer is
|
||||
Cloudflare's edge via the Tunnel, not the WAN IP — see `platform/externaldns/README.md`.
|
||||
The pin is still the right path: it keeps registry pulls on the LAN and off Cloudflare's
|
||||
proxy.) The Gateway, the wildcard certificate
|
||||
and the digest were never implicated. The fix is the pin this repository already uses
|
||||
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
|
||||
in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
|
||||
|
||||
@@ -31,10 +31,29 @@ The operator decisions it encodes (2026-11-08):
|
||||
policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check
|
||||
`cloudflared-ingress-rules` fails the gate if that ever changes.
|
||||
|
||||
**Nothing goes through the tunnel until DNS points at it.** Cloudflare routes a hostname through a
|
||||
tunnel only when its record is a CNAME to `<UUID>.cfargotunnel.com`. Phase 1 (this component)
|
||||
creates no such record; moving public DNS is a separate, later change. So in Phase 1 the alerts
|
||||
firing means "the tunnel is not ready", not "users are down".
|
||||
**Cloudflare routes a hostname through the tunnel only when its record is a proxied CNAME to
|
||||
`<UUID>.cfargotunnel.com`.** Phase 1 created no such record.
|
||||
|
||||
**Phase 2 — DONE (2026-11-08, branch `feat/dns-via-cloudflare-tunnel`).** external-dns now publishes
|
||||
every gateway name as a proxied CNAME to the tunnel: the shared Gateway's
|
||||
`external-dns.alpha.kubernetes.io/target` is `<UUID>.cfargotunnel.com` and the HelmRelease sets
|
||||
`--cloudflare-proxied` (see `platform/externaldns/README.md`). Plex's web UI and Headlamp are
|
||||
disowned (`controller: none`), so their public records are deleted; unifi and crafty-mc stay DNS-only
|
||||
A records (`cloudflare-proxied: "false"`); `wan.` stays for WireGuard. The L0 check
|
||||
`internal-services-publish-no-public-dns` (9) pins all of this, including that the Gateway target's
|
||||
UUID equals `tunnel:` in `config.yaml`. **So from Phase 2 on, these alerts firing DOES mean public
|
||||
users are down** — every public HTTP(S) name depends on the connectors.
|
||||
|
||||
### After the Phase 2 merge — operator step (router)
|
||||
|
||||
Once `dig +short @1.1.1.1 <public name>` returns Cloudflare anycast addresses for the public names
|
||||
(and external-dns logs show the UPDATEs), nothing legitimate reaches the WAN on 80/443 any more:
|
||||
|
||||
1. **Close** the router's WAN **tcp/80** and **tcp/443** port-forwards to the Gateway VIP.
|
||||
2. **Keep** WAN **udp/51820** (WireGuard → `wan.home.keithjackson.dev`) and WAN **tcp/32400**
|
||||
(Plex remote streaming via plex.tv; `runbooks/plex-remote-access.md`).
|
||||
3. Verify from off-LAN (phone on cellular): a public name still loads (via the tunnel), and
|
||||
`curl -m5 https://<WAN IP>/` now times out instead of answering.
|
||||
|
||||
Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and
|
||||
`CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization
|
||||
@@ -63,23 +82,36 @@ patch is missing; see causes below.
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
1. **The tunnel has not been enrolled yet** (Phase 1, steps 1-5 below not done). Logs show an
|
||||
error about the tunnel `REPLACE-ME-TUNNEL-UUID` or a missing origin certificate. Do steps 1-5.
|
||||
2. **UUID and credentials disagree** — `config.yaml` `tunnel:` is not the `TunnelID` in the
|
||||
**Read the FIRST error line, not the last.** When cloudflared cannot use the credentials file it
|
||||
falls back to looking for an origin certificate, and the error it ends on —
|
||||
`Cannot determine default origin certificate path` / `error parsing tunnel ID: Error locating
|
||||
origin cert` — is a **red herring** whatever the real cause was. The credentials file is the
|
||||
whole of what this Deployment needs; no `cert.pem` ever belongs in the pod.
|
||||
|
||||
1. **`permission denied` on `credentials.json` → the "origin cert" error is a red herring.** Logs:
|
||||
`couldn't read tunnel credentials from /etc/cloudflared/creds/credentials.json: permission
|
||||
denied`, then the origin-cert lines. The container (uid 65532) cannot read the Secret file:
|
||||
the pod `securityContext.fsGroup: 65532` is missing, so the 0440 file is `root:root`. Fix it in
|
||||
git (`platform/cloudflared/deployment.yaml`); the L0 check `cloudflared-ingress-rules`,
|
||||
assertion (7), should already have refused that. Do NOT loosen `defaultMode` to 0444. (This is
|
||||
what CrashLooped every replica on 2026-11-08.)
|
||||
2. **The tunnel has not been enrolled yet** (Phase 1, steps 1-5 below not done). Logs show an
|
||||
error about the tunnel `REPLACE-ME-TUNNEL-UUID` (and the origin-cert red herring). Do steps 1-5.
|
||||
3. **UUID and credentials disagree** — `config.yaml` `tunnel:` is not the `TunnelID` in the
|
||||
credentials JSON. Logs: authentication / "tunnel not found". Check with step 4's verify command.
|
||||
3. **Credentials still encrypted in the cluster** — the `cloudflared` decryption patch is missing
|
||||
4. **Credentials still encrypted in the cluster** — the `cloudflared` decryption patch is missing
|
||||
from `clusters/prod/kustomization.yaml`. The L0 check guards this, so this means someone bypassed
|
||||
it.
|
||||
4. **Egress blocked** — a policy change, or the edge moved off 7844. Look for drops:
|
||||
5. **Egress blocked** — a policy change, or the edge moved off 7844. Look for drops:
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
|
||||
hubble observe --verdict DROPPED --from-namespace cloudflared --last 100 -o compact'
|
||||
```
|
||||
(`ds/cilium` picks one node's agent; repeat per node if empty.)
|
||||
5. **The tunnel was deleted or its secret rotated in Cloudflare** — `cloudflared tunnel list`
|
||||
6. **The tunnel was deleted or its secret rotated in Cloudflare** — `cloudflared tunnel list`
|
||||
(step 2's podman form) no longer shows it. Re-enrol with steps 2-5.
|
||||
6. **`CloudflaredReplicasDegraded` only**: one replica cannot schedule (the spread is hard, one per
|
||||
7. **`CloudflaredReplicasDegraded` only**: one replica cannot schedule (the spread is hard, one per
|
||||
node — a cordoned node during a kured reboot is normal and clears) or an image pull failed.
|
||||
|
||||
## When to escalate or stop
|
||||
@@ -90,8 +122,8 @@ patch is missing; see causes below.
|
||||
- **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one.
|
||||
**external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`);
|
||||
a record created behind its back either fights it or is deleted by it, and a real hostname
|
||||
routed that way is an unreviewed DNS flip. Moving records onto the tunnel is the separate Phase 2
|
||||
change. The only record this runbook creates is the hand-made canary in step 6, and you delete it.
|
||||
routed that way is an unreviewed DNS flip. Records reach the tunnel only through external-dns
|
||||
(Phase 2, done). The only record this runbook creates is the hand-made canary in step 6, and you delete it.
|
||||
- Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is
|
||||
Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2.
|
||||
|
||||
@@ -270,6 +302,46 @@ disappears with the pod on the next rollout.)
|
||||
Finally **delete the `tunnel-canary` CNAME** in the dashboard and confirm it is gone:
|
||||
`dig +short tunnel-canary.home.keithjackson.dev @1.1.1.1` returns nothing.
|
||||
|
||||
## Rotating the tunnel credential
|
||||
|
||||
Do this whenever the credentials JSON (`TunnelSecret`) or `cert.pem` may have been exposed — a
|
||||
transcript, a log, a screen share. A tunnel's secret cannot be changed in place from this
|
||||
workflow, so rotation is **a new tunnel**: the old credential stays valid until the old tunnel
|
||||
is deleted, which is why the last step is not optional.
|
||||
|
||||
1. **Create the new tunnel** (*Enrolling*, step 2's `cf` helper; `cf tunnel login` again if
|
||||
`cert.pem` is not to hand). Use a NEW name — the old one is still taken:
|
||||
```
|
||||
cf tunnel list # note the OLD tunnel's id and name
|
||||
cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s-2
|
||||
```
|
||||
2. **Swap the UUID** in your own worktree, on a new branch (CLAUDE.md §0):
|
||||
```
|
||||
OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml)
|
||||
sed -i "s/^tunnel: ${OLD}$/tunnel: <NEW UUID>/" platform/cloudflared/config.yaml
|
||||
# Phase 2: the Gateway's external-dns target carries the SAME UUID. Change both in one commit;
|
||||
# the L0 check internal-services-publish-no-public-dns (9a) fails the gate if they disagree.
|
||||
sed -i "s/${OLD}\.cfargotunnel\.com/<NEW UUID>.cfargotunnel.com/" platform/gateway/gateway.yaml
|
||||
```
|
||||
3. **Swap the encrypted credentials** with *Enrolling* step 4 — the same `sops set`, then the
|
||||
same `ok`-only verify, which now checks the JSON's `TunnelID` against the NEW UUID. Never
|
||||
`sops -d` to the terminal.
|
||||
4. **Merge** — gate, PR, `/approve-override` by the operator (it touches `**/*.sops.yaml`), merge.
|
||||
Flux rolls the Deployment because the content-hashed ConfigMap changed.
|
||||
5. **Verify** with *Enrolling* step 6: both replicas Ready, `Registered tunnel connection` in
|
||||
each replica's log, `cf tunnel info home-k3s-2` lists two connectors, and
|
||||
`cf tunnel info <old name>` lists **none**.
|
||||
6. **Only then delete the old tunnel**, which revokes the exposed credential:
|
||||
```
|
||||
cf tunnel cleanup <old name> # drops any stale connector records first
|
||||
cf tunnel delete <old name>
|
||||
cf tunnel list # the old tunnel is gone
|
||||
```
|
||||
Since Phase 2, records point at `<UUID>.cfargotunnel.com`: confirm external-dns has repointed
|
||||
every record to the NEW UUID (its logs show an UPDATE per name) before deleting, or those
|
||||
hostnames go dark.
|
||||
Then replace `tunnel.json`/`cert.pem` in the password manager and delete them from disk.
|
||||
|
||||
## Rollback
|
||||
|
||||
- **Phase 1 (no DNS on the tunnel):** revert the PR. Flux prunes the `cloudflared` namespace and
|
||||
@@ -281,8 +353,15 @@ Finally **delete the `tunnel-canary` CNAME** in the dashboard and confirm it is
|
||||
'flux --context k3s-prod suspend kustomization cloudflared &&
|
||||
kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0'
|
||||
```
|
||||
Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then
|
||||
Cloudflare then answers tunnel hostnames with a 530/1033 error — **from the LAN too** for the
|
||||
names PiHole does not list (apex, grafana, crafty, seerr; since 2026-11-09). Follow up in git, then
|
||||
`flux resume kustomization cloudflared`.
|
||||
- **After the DNS flip:** the fast path back is DNS, not the tunnel — restore the previous records
|
||||
(that change's own runbook). Revoking the tunnel credential (`cf tunnel delete`, or rotating it
|
||||
- **After the DNS flip (Phase 2):** the path back is a revert PR of `feat/dns-via-cloudflare-tunnel`
|
||||
(Gateway target back to `wan.${BASE_DOMAIN}`, drop `--cloudflare-proxied`) — external-dns then
|
||||
rewrites every record DNS-only to `wan.`. That only works if the router's WAN tcp/80+443 forwards
|
||||
are **re-opened** first (they are closed after Phase 2). **Also restore PiHole's zone wildcard**
|
||||
(`address=/${BASE_DOMAIN}/${GATEWAY_VIP}` in `catalog/pihole.yaml`, and drop the
|
||||
`pihole-lan-names` L0 assertion (a)): since 2026-11-09 the apex, grafana, crafty and seerr are not
|
||||
listed there and resolve publicly, and a public `wan.` answer does not hairpin from the LAN. Never edit the records by hand:
|
||||
external-dns owns them and `policy: sync` reverts a hand edit. Revoking the tunnel credential (`cf tunnel delete`, or rotating it
|
||||
in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.
|
||||
@@ -11,6 +11,19 @@ env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway UR
|
||||
**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
|
||||
procedures with a manual last mile.
|
||||
|
||||
## The Plex web UI is LAN/VPN-only (2026-11-08)
|
||||
|
||||
When public DNS moved to the Cloudflare Tunnel (`runbooks/cloudflare-tunnel.md`, Phase 2), the
|
||||
operator kept **Plex off the tunnel**. `catalog/plex.yaml` is now `exposure: internal`, so the
|
||||
`plex.home.keithjackson.dev` route carries `external-dns ... controller: none` and its public record
|
||||
is deleted; the tunnel also 404s `plex` and `plex-vip`. On the LAN (and over WireGuard) the web UI
|
||||
at `https://plex.home.keithjackson.dev` works exactly as before through PiHole's split-horizon answer.
|
||||
|
||||
**Remote streaming is unaffected** — it never used that name. It is the path below: plex.tv hands
|
||||
clients the `plex.direct` URL (`ADVERTISE_IP`) for the L4 VIP `192.168.2.24`, reached through the
|
||||
WAN **tcp/32400** forward. That forward must stay open when the WAN 80/443 forwards are closed after
|
||||
the tunnel cut-over. Remote users use app.plex.tv, not our hostname.
|
||||
|
||||
## What was actually broken (measured 2026-10-03)
|
||||
|
||||
Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
|
||||
@@ -44,9 +57,10 @@ either way — this repo has no such vantage point and must not pretend otherwis
|
||||
router by hand, so it must never move on a reschedule.
|
||||
|
||||
While in the router, **check what else is bound to the WAN interface.** If remote management is
|
||||
listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
|
||||
Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
|
||||
though remote *streaming* works. These are two separate paths and they fail independently.
|
||||
listening on 443 (see the certificate measurement above), move it or disable it. (Historical: this
|
||||
used to shadow the Gateway's 80/443 forward for the Plex *web UI* over the internet. Since
|
||||
2026-11-08 the web UI is LAN/VPN-only and public HTTP(S) goes through the Cloudflare Tunnel, so the
|
||||
80/443 forwards are retired; only tcp/32400 matters here.)
|
||||
|
||||
Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
|
||||
|
||||
|
||||
@@ -74,7 +74,7 @@ Work through these in order; the first three are by far the most likely.
|
||||
|---|---|---|
|
||||
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
|
||||
| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
|
||||
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
||||
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and (before the 2026-11-08 Tunnel cutover; the public answer is now Cloudflare) got the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
||||
| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
|
||||
| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets. Add `read:notification` if you want the cheap pull-request mention query (`platform/pm-agent/README.md`) |
|
||||
| a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
|
||||
|
||||
@@ -82,13 +82,10 @@ platform must render, and cites the requirement each mapping satisfies. Changing
|
||||
all seven the moment it is set. Per-resource exclusion has exactly the blast radius of the
|
||||
defect. The filter stays available as hardening once those seven are adjudicated — see
|
||||
`platform/externaldns/helmrelease.yaml` and the register in the L0 check.
|
||||
- `public` renders the record but the renderer does **not** yet set a target: the record still
|
||||
resolves to the shared Gateway VIP, i.e. an RFC1918 address, so a `public` service is not in fact
|
||||
publicly reachable. Fixing that is **not** a renderer change on v0.21.0 —
|
||||
`source/gateway.go` reads `external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and
|
||||
a ListenerSet), never from an HTTPRoute, so a per-service target override is not expressible while
|
||||
the public services share the `shared` Gateway. The open decision is recorded in
|
||||
`platform/externaldns/helmrelease.yaml`.
|
||||
- `public` renders the record but the renderer does **not** set a target: external-dns reads
|
||||
`external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and a ListenerSet), never from
|
||||
an HTTPRoute, so the target is set once, on the `shared` Gateway — since 2026-11-08 the Cloudflare
|
||||
Tunnel, proxied (`platform/gateway/gateway.yaml`, `platform/externaldns/README.md`).
|
||||
- Exposure does **not** relax east-west default-deny (docs/09 §"Interaction with default-deny").
|
||||
- **Requirements:** R-DNS-02/03 (docs/09), R-NET-04 (docs/03).
|
||||
|
||||
@@ -323,8 +320,9 @@ R-OBS-01 (the SLI must follow the rendered kind).
|
||||
`public` service**. An `internal` service's route is rendered with ExternalDNS's controller
|
||||
annotation set to `none`, which removes it from ExternalDNS's view entirely (see §`exposure`), so
|
||||
its `<host>` exists **only** in the internal split-horizon answer PiHole serves, never in the
|
||||
public zone. A `public` record's TARGET is still the Gateway VIP, not a routable address — open
|
||||
operator decision, see §`exposure`.
|
||||
public zone. A `public` record's TARGET is the Cloudflare Tunnel (a proxied CNAME, from the
|
||||
shared Gateway's target annotation; `platform/externaldns/README.md`), and a `public` name is
|
||||
not listed in PiHole unless it must stay on the LAN path.
|
||||
- The `host` pattern is enforced to `<label>.home.keithjackson.dev` by the schema.
|
||||
- **Requirements:** **R-TLS-01/02/03** (docs/08), **R-DNS-01/02/03** (docs/09).
|
||||
|
||||
|
||||
Reference in new issue
Block a user