Compare commits

...
1 Commits
Author SHA1 Message Date
jakman85andClaude Opus 5.5 8fe8d5df8a feat(dns): PiHole answers an explicit LAN name list, not a zone wildcard
gate / gate (pull_request) Failing after 43s
gate / conformance (pull_request) Skipped
Public names are proxied CNAMEs to the Cloudflare Tunnel since PR #156 and
work from the LAN, so they should resolve publicly there too. Replace the
address=/${BASE_DOMAIN}/ wildcard with explicit entries: the LAN-only names
(derived from every controller:none HTTPRoute), gitea/nextcloud (off
Cloudflare's 100 MB / 100 s limits), keycloak/auth, and the L4 VIPs.

Add L0 check pihole-lan-names, mutation-tested. Correct docs that still
described public names as CNAME to wan. and the cloudflared entrypoint comment.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-11-08 19:51:49 -06:00
16 changed files with 513 additions and 60 deletions

No files matched your search

+1 -1
View File
@@ -76,7 +76,7 @@ bug but the honest answer, because proofs are claims about a specific tree on a
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the LAN resolver**, answering an explicit list of `${BASE_DOMAIN}` names (LAN-only apps, gitea/nextcloud/keycloak/auth, the L4 VIPs — everything else resolves publicly to Cloudflare, since 2026-11-09) on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
+2 -2
View File
@@ -176,8 +176,8 @@ or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
`apps/wave-0/pihole.yaml` (LAN split-horizon, an explicit name list since 2026-11-09) point
`unifi.${BASE_DOMAIN}` at `${UNIFI_L4_VIP}` rather than `${GATEWAY_VIP}`. `${BASE_DOMAIN}` on prod is the
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
flip, so this hostname is current, not a dev leftover.
+1 -1
View File
@@ -73,7 +73,7 @@ spec:
- name: FTLCONF_dns_rateLimit_count
value: '0'
- name: FTLCONF_misc_dnsmasq_lines
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
value: local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
- name: FTLCONF_webserver_api_password
valueFrom:
secretKeyRef:
+57 -30
View File
@@ -108,41 +108,61 @@ env:
# `loadBalancer.mode: dsr`, which preserves the client address.
- name: FTLCONF_dns_rateLimit_count
value: "0"
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
# so the first line is a wildcard for the whole zone and each later line is a specific override.
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
# changes nothing.
# THE LAN ANSWERS — AN EXPLICIT LIST, NOT A ZONE WILDCARD (2026-11-09). A dnsmasq
# `address=/<domain>/<ip>` entry answers that domain AND every subdomain of it (on LABEL
# boundaries: `pihole.` does not match `pihole-dns.`), and dnsmasq resolves overlapping entries by
# LONGEST MATCHING DOMAIN, so the order below is irrelevant. Every name NOT listed here is
# FORWARDED to the upstreams above and gets the PUBLIC answer.
#
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
# is the annotation that means "this Service publishes its own hostname at its own address":
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
# resolver address, are the complete set of exceptions today.
# WHY NO WILDCARD ANY MORE. Until 2026-11-09 the first entry was `address=/${BASE_DOMAIN}/
# ${GATEWAY_VIP}`, answering the whole zone. That made sense while every public record was a
# DNS-only CNAME to `wan.${BASE_DOMAIN}` (no NAT hairpin on this router, so the LAN NEEDED a
# private answer). Since PR #156 every public name is a PROXIED CNAME to the Cloudflare Tunnel
# (platform/externaldns/README.md), which works identically from the LAN — so a name that is
# public now SHOULD resolve publicly from the LAN too, and the LAN no longer depends on PiHole for
# it: a client that bypasses PiHole (DoH browser, phone private DNS, the router's fallback) gets
# the same working answer. The wildcard had also become actively misleading: it answered names
# that exist nowhere (`wan.` itself resolved to the Gateway VIP on the LAN).
#
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
# WHAT IS LISTED, and why each — enforced by conformance/tests/l0-static/pihole-lan-names:
# -> ${GATEWAY_VIP}, LAN-ONLY: every HTTPRoute disowned from public DNS
# (`external-dns.alpha.kubernetes.io/controller: none`, i.e. catalog `exposure: internal`,
# plus plex and headlamp) — jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr,
# headlamp. These have NO public record and the tunnel 404s them, so THIS LINE IS THEIR ONLY
# PATH: drop one and the app is unreachable everywhere. A NEWLY ONBOARDED `internal` SERVICE
# MUST BE ADDED HERE (the L0 check fails until it is). A new `public` service needs nothing.
# -> ${GATEWAY_VIP}, PUBLIC BUT KEPT ON THE LAN PATH: gitea (git push, LFS, container registry)
# and nextcloud (uploads) — Cloudflare's proxy caps a request body at 100 MB and a response
# at 100 s, which a large push or upload exceeds; keycloak and auth (oauth2-proxy) — so the
# OIDC login of those two LAN-path apps stays on the same path as the apps themselves.
# -> their own L4 VIPs: unifi (${UNIFI_L4_VIP}) and crafty-mc (${CRAFTY_L4_VIP}) are not behind
# the Gateway (grep `external-dns.alpha.kubernetes.io/hostname` on a Service:
# apps/vms/unifi/50-service-l4.yaml, platform/crafty/service-l4.yaml); their public records
# are DNS-only A records to the same RFC1918 VIPs, which this router's rebind protection
# strips, so the LAN answer is still required. pihole-dns (${PIHOLE_DNS_VIP}) is PiHole's own
# resolver address; plex-vip (${PLEX_L4_VIP}) is below.
# NOT listed, deliberately, so they resolve via Cloudflare from the LAN as from anywhere: the apex,
# grafana, crafty, seerr. In-cluster (platform/coredns/coredns-custom.yaml) and node
# (/etc/hosts pins, node_baseline_hosts_pins) resolution are separate and unchanged.
#
# AAAA: dnsmasq answers a AAAA query for an `address=`d name with NODATA (NOERROR, no answer),
# so IPv6-capable clients fall back to the A record instead of getting NXDOMAIN — the same
# behaviour coredns-custom hand-writes a `template IN AAAA` block for. Unlisted (proxied) names
# now get Cloudflare's real A and AAAA answers.
#
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
# while resolution for the zone does not work. Confirmed both ways against the live image.
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
# hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
# (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
# LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
# the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
# wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
# every client that takes the ADVERTISED connection cannot reach the server at all.
#
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the name of the tcp/32400 LoadBalancer
# (apps/wave-0/plex-remote.yaml) and the domain of the certificate Plex serves there
# (catalog/plex.yaml CERTDOMAIN / customCertificateDomain, Secret `plex-vip-tls`). It is NOT what
# Plex advertises any more (ADVERTISE_IP is a resolver-independent plex.direct name since
# 2026-10-05) and it has NO public record, so this entry is its only resolution — kept for
# reaching the VIP by name with a matching certificate from a LAN shell. Measured 2026-10-04,
# before it existed the zone wildcard answered it with ${GATEWAY_VIP}, where nothing listens on
# 32400; without it now, the name is NXDOMAIN.
#
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
@@ -172,10 +192,17 @@ env:
# cache expires, the router falls back, and the public answer latches again. That is a real
# outage, and the fix for it is PiHole availability, not DNS caching.
#
# SINCE 2026-11-09 THE LATCH IS MOSTLY HARMLESS. The measurement above predates the tunnel: the
# public answer was then `wan.`, unreachable from the LAN. Now the public answer for every
# proxied name is Cloudflare, which works from the LAN, and those names are not listed below at
# all. A latch still bites only the listed names: gitea/nextcloud/keycloak/auth fall back to
# Cloudflare (working, but under its 100 MB / 100 s limits); the LAN-only names and the L4 VIPs
# get NXDOMAIN or a stripped answer until the router re-queries PiHole.
#
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
# configuration, /etc/hosts and DHCP leases), so one directive covers every listed name.
- name: FTLCONF_misc_dnsmasq_lines
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
value: "local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
# carries the full rationale and the measurement that found it.
@@ -0,0 +1,45 @@
# =============================================================================
# L0 STATIC — PiHole answers an explicit LAN name list, and no LAN-only name is left out of it
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
#
# Since PR #156 every public name is a proxied CNAME to the Cloudflare Tunnel and works from the
# LAN, so PiHole's zone wildcard was replaced by an explicit list (catalog/pihole.yaml). That makes
# the list load-bearing: a route disowned from public DNS and absent from it resolves nowhere. This
# test derives the required names from the tree (every `controller: none` HTTPRoute, every hostname
# the tunnel 404s, the L4 Services' own VIPs) and asserts the list covers them, carries no
# zone-wide entry, and pins gitea and nextcloud to the Gateway VIP. It proves itself non-vacuous by
# mutation. Rationale and the honest NOT-proven list: check.sh.
#
# WHY SCOPE, NOT REQUIREMENTS: the property spans catalog/, apps/, platform/ and runbooks/; a new
# internal service in any of them can break it, and ci/touched-requirements.sh maps apps/* to no
# requirement at all, so a requirement-scoped selection would skip it on precisely those changes.
# =============================================================================
apiVersion: chainsaw.kyverno.io/v1alpha1
kind: Test
metadata:
name: pihole-lan-names
labels:
conformance.home.k3s/R-DNS-03: "true"
annotations:
conformance.home.k3s/requirements: "R-DNS-03"
conformance.home.k3s/layer: "L0"
conformance.home.k3s/scope: "whole-repo"
conformance.home.k3s/safety-critical: "true"
conformance.home.k3s/executable: "true"
conformance.home.k3s/intent: >-
PiHole answers an explicit list of LAN names rather than the whole zone; every HTTPRoute
disowned from public DNS has an entry to the Gateway VIP; every hostname the Cloudflare
Tunnel 404s has an entry, with the L4 names on their own VIPs; and gitea and nextcloud stay
on the LAN path, off Cloudflare's request-size and timeout limits.
spec:
steps:
- name: assert-pihole-explicit-lan-names
description: >-
Parse FTLCONF_misc_dnsmasq_lines from catalog/pihole.yaml and the rendered
apps/wave-0/pihole.yaml, derive the required names from every `controller: none` HTTPRoute,
the http_status:404 hostnames in platform/cloudflared/config.yaml and the L4 Services'
lbipam annotations, assert the list covers them and contains no zone-wide entry, and run
the evaluator on mutated lists to prove it is not vacuous.
try:
- script:
content: bash ./check.sh
+269
View File
@@ -0,0 +1,269 @@
#!/usr/bin/env bash
# =============================================================================
# L0 static check (offline, no cluster) — pihole-lan-names
# Serves: R-DNS-03 (LAN split-horizon answers), and decisions/2026-11-09-pihole-explicit-lan-names.md.
#
# WHY THIS EXISTS. Since PR #156 every PUBLIC name is a proxied CNAME to the Cloudflare Tunnel and
# works from the LAN, so PiHole no longer answers the whole zone with a wildcard: it answers an
# EXPLICIT list (catalog/pihole.yaml, FTLCONF_misc_dnsmasq_lines) and forwards everything else to
# the public resolvers. That inverts the old failure mode. With the wildcard, a forgotten name still
# resolved; now a forgotten LAN-ONLY name — a route disowned from public DNS, which the tunnel also
# 404s — resolves NOWHERE, and the app is silently unreachable from every client. Nothing in the
# cluster turns red when that happens. So the list is derived from the tree and asserted here.
#
# GIVEN the repository as authored,
# THEN (0) the dnsmasq value parses (only `local-ttl=` and `address=/<domain>/<${VAR}>` entries,
# every VAR is a cluster-var), the catalog value and the rendered apps/wave-0/pihole.yaml
# value are identical, and the inventories below are non-empty (anti-vacuous-pass);
# (a) NO entry covers the whole zone: no `address=/${BASE_DOMAIN}/`, no `address=/#/`, and no
# entry for any parent of the zone (dnsmasq matches every subdomain of an entry);
# (b) every HTTPRoute hostname disowned from public DNS
# (`external-dns.alpha.kubernetes.io/controller: none`, anywhere in apps/ platform/
# runbooks/) has an explicit entry to ${GATEWAY_VIP} — a disowned route with no
# `hostnames:` is itself a FAIL, because it cannot be checked;
# (c) every hostname platform/cloudflared/config.yaml maps to http_status:404 has an entry;
# the L4 DNS-only names (derived: Services carrying both
# `external-dns.alpha.kubernetes.io/hostname` and `lbipam.cilium.io/ips` — unifi,
# crafty-mc) must map to THAT Service's VIP, not the Gateway;
# (d) gitea and nextcloud are pinned to ${GATEWAY_VIP} (Cloudflare's 100 MB body / 100 s
# limits break git push/LFS and large uploads);
# AND (e) the evaluator is not vacuous: it is run against mutated copies of the real entry list
# (zone wildcard added; a disowned host dropped; a 404'd host dropped; gitea repointed;
# unifi repointed to the Gateway) and must flag each one.
#
# NOT proven (static): that the running PiHole pods carry this value (FTL applies FTLCONF_* at
# start; the env change rolls the StatefulSet), that public resolution of the unlisted names
# reaches Cloudflare, or the substituted IPs. Runtime check: `dig @${PIHOLE_DNS_VIP} <name>`.
# =============================================================================
set -euo pipefail
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
exec python3 - "$ROOT" <<'PY'
import copy
import re
import sys
from pathlib import Path
try:
import yaml
except ImportError: # pragma: no cover
print("CHECK FAIL (pihole-lan-names): PyYAML is required", file=sys.stderr)
raise SystemExit(1)
ROOT = Path(sys.argv[1])
FAILURES, CHECKS = [], 0
NAME = "pihole-lan-names"
ZONE = "${BASE_DOMAIN}"
GW = "${GATEWAY_VIP}"
DISOWN = "external-dns.alpha.kubernetes.io/controller"
EDNS_HOST = "external-dns.alpha.kubernetes.io/hostname"
LB_IPS = "lbipam.cilium.io/ips"
# Public, but must bypass Cloudflare's 100 MB request body / 100 s response limits.
LAN_PINNED = ("gitea", "nextcloud")
# Trees that hold applied manifests. conformance/ fixtures and .claude/ worktrees are not authored
# surface (CLAUDE.md §0); decisions/ is prose.
SCAN = ("apps", "platform", "runbooks")
def check(cond, desc):
global CHECKS
CHECKS += 1
if not cond:
FAILURES.append(desc)
def docs(path):
try:
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
except yaml.YAMLError:
return []
def dnsmasq_value(objs):
"""The FTLCONF_misc_dnsmasq_lines value from a list of docs (a ServiceDefinition or manifests)."""
found = []
for d in objs:
envs = list(d.get("env") or [])
tmpl = (((d.get("spec") or {}).get("template") or {}).get("spec") or {})
for c in tmpl.get("containers") or []:
envs += c.get("env") or []
found += [e.get("value") for e in envs if e.get("name") == "FTLCONF_misc_dnsmasq_lines"]
return found
def die(msg):
print(f"CHECK FAIL ({NAME}): {msg}", file=sys.stderr)
raise SystemExit(1)
CATALOG = ROOT / "catalog" / "pihole.yaml"
RENDERED = ROOT / "apps" / "wave-0" / "pihole.yaml"
CF = ROOT / "platform" / "cloudflared" / "config.yaml"
for p in (CATALOG, RENDERED, CF):
if not p.is_file():
die(f"{p.relative_to(ROOT)} is absent — the surface this check guards moved; re-point or "
"delete this check in the SAME change")
cat_vals = dnsmasq_value(docs(CATALOG))
ren_vals = dnsmasq_value(docs(RENDERED))
check(len(cat_vals) == 1, f"(0) catalog/pihole.yaml declares FTLCONF_misc_dnsmasq_lines exactly "
f"once (found {len(cat_vals)})")
check(len(ren_vals) == 1, f"(0) apps/wave-0/pihole.yaml carries FTLCONF_misc_dnsmasq_lines exactly "
f"once (found {len(ren_vals)})")
if not cat_vals:
die("no dnsmasq value to check")
check(cat_vals[:1] == ren_vals[:1],
"(0) the rendered apps/wave-0/pihole.yaml dnsmasq value differs from catalog/pihole.yaml — "
"re-render pihole (catalog/render.py --no-image-automation catalog/pihole.yaml)")
cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
if d.get("kind") == "ConfigMap")
CLUSTER_VARS = set(cv["data"])
BASE_DOMAIN_VALUE = cv["data"]["BASE_DOMAIN"]
ENTRY = re.compile(r"^address=/([^/]+)/(\$\{[A-Z0-9_]+\})$")
def parse(value):
"""Return (entries {domain: target}, problems)."""
entries, problems = {}, []
for raw in value.split(";"):
line = raw.strip()
if raw != line:
problems.append(f"entry {raw!r} has surrounding whitespace (FTL writes it verbatim)")
if not line:
problems.append("empty entry (doubled or trailing `;`)")
continue
if re.fullmatch(r"local-ttl=\d+", line):
continue
m = ENTRY.match(line)
if not m:
problems.append(f"entry {line!r} is not `local-ttl=N` or `address=/<domain>/${{VAR}}` — "
f"this check cannot reason about it")
continue
dom, tgt = m.groups()
if tgt[2:-1] not in CLUSTER_VARS:
problems.append(f"{dom} -> {tgt}: not a key of clusters/base/cluster-vars.yaml")
if dom in entries and entries[dom] != tgt:
problems.append(f"{dom} is listed twice with different targets")
entries[dom] = tgt
return entries, problems
entries, problems = parse(cat_vals[0])
for pr in problems:
check(False, f"(0) {pr}")
# ── inventories, derived from the tree ──────────────────────────────────────────────────────────
disowned, unnamed = set(), []
l4 = {}
for top in SCAN:
for f in sorted((ROOT / top).rglob("*.yaml")):
for d in docs(f):
meta = d.get("metadata") or {}
ann = meta.get("annotations") or {}
if d.get("kind") == "HTTPRoute" and str(ann.get(DISOWN, "")).strip() == "none":
hosts = (d.get("spec") or {}).get("hostnames") or []
if not hosts:
unnamed.append(f"{f.relative_to(ROOT)}:{meta.get('name')}")
disowned.update(hosts)
if d.get("kind") == "Service" and EDNS_HOST in ann and LB_IPS in ann:
for h in str(ann[EDNS_HOST]).split(","):
l4[h.strip()] = str(ann[LB_IPS]).strip()
for u in unnamed:
check(False, f"(b) disowned HTTPRoute {u} declares no `hostnames:` — it inherits a listener "
f"hostname this check cannot see; name it explicitly")
check(len(disowned) >= 5, f"(0) only {len(disowned)} disowned HTTPRoute hostnames found — the scan "
f"is vacuous (the disown annotation was renamed?)")
check(len(l4) >= 2, f"(0) only {len(l4)} L4 Services publishing their own hostname found (expected "
f"unifi and crafty-mc) — the scan is vacuous")
cf = yaml.safe_load(CF.read_text())
cf_404 = {r["hostname"] for r in cf.get("ingress") or []
if r.get("hostname") and r.get("service") == "http_status:404"}
check(len(cf_404) >= 5, f"(0) only {len(cf_404)} http_status:404 hostnames in the cloudflared "
f"config — the scan is vacuous")
def covers_zone(dom):
"""True if an entry for `dom` answers every name in the zone (dom is the zone or a parent)."""
if dom in ("#", "", "."):
return True
if dom == ZONE:
return True
# A literal parent (e.g. `keithjackson.dev`, `dev`) of the zone's substituted value.
real = dom.replace(ZONE, BASE_DOMAIN_VALUE)
return real == BASE_DOMAIN_VALUE or BASE_DOMAIN_VALUE.endswith("." + real)
def evaluate(entries):
bad = []
# (a)
for dom in entries:
if covers_zone(dom):
bad.append(f"(a) `address=/{dom}/` answers EVERY name in the zone — the wildcard is gone "
f"on purpose (public names must resolve to Cloudflare from the LAN)")
# (b)
for h in sorted(disowned):
if entries.get(h) != GW:
bad.append(f"(b) {h} is disowned from public DNS but PiHole maps it to "
f"{entries.get(h, 'NOTHING')} — expected {GW}; without it the app is "
f"unreachable everywhere")
# (c)
for h in sorted(cf_404):
if h in l4:
if entries.get(h) != l4[h]:
bad.append(f"(c) {h} is an L4 DNS-only name; PiHole maps it to "
f"{entries.get(h, 'NOTHING')}, expected its own VIP {l4[h]}")
elif h not in entries:
bad.append(f"(c) the tunnel 404s {h} but PiHole has no entry for it — it would resolve "
f"only publicly, and the tunnel refuses it")
# (d)
for n in LAN_PINNED:
h = f"{n}.{ZONE}"
if entries.get(h) != GW:
bad.append(f"(d) {h} maps to {entries.get(h, 'NOTHING (public/Cloudflare)')}; it must be "
f"pinned to {GW} — Cloudflare caps request bodies at 100 MB and responses at "
f"100 s")
return bad
for v in evaluate(entries):
check(False, v)
# (e) mutation self-test
mutants = {}
m = dict(entries); m[ZONE] = GW
mutants["zone wildcard added"] = m
m = dict(entries); m[BASE_DOMAIN_VALUE.split(".", 1)[1]] = GW
mutants["parent-of-zone wildcard added"] = m
m = dict(entries); m.pop(sorted(disowned)[0], None)
mutants[f"disowned host {sorted(disowned)[0]} dropped"] = m
pure_404 = sorted(h for h in cf_404 if h not in l4 and h not in disowned)
if pure_404:
m = dict(entries); m.pop(pure_404[0], None)
mutants[f"404'd host {pure_404[0]} dropped"] = m
m = dict(entries); m.pop(f"gitea.{ZONE}", None)
mutants["gitea unpinned"] = m
m = dict(entries); m[f"nextcloud.{ZONE}"] = "${UNIFI_L4_VIP}"
mutants["nextcloud repointed"] = m
if l4:
h = sorted(l4)[0]
m = dict(entries); m[h] = GW
mutants[f"{h} repointed to the Gateway"] = m
for name, mut in mutants.items():
check(evaluate(mut), f"(e) self-test: the evaluator did NOT flag the mutant '{name}' — vacuous")
check(len(mutants) >= 5, f"(e) only {len(mutants)} mutants built — the self-test degenerated")
_, pr = parse("local-ttl=60;address=/x.${BASE_DOMAIN}/${NOT_A_CLUSTER_VAR}; address=/y/${GATEWAY_VIP}")
check(len(pr) >= 2, "(e) self-test: the parser accepted an unknown VAR or a whitespace-padded entry")
if CHECKS < 15:
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
if FAILURES:
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
for f in FAILURES:
print(f" - {f}", file=sys.stderr)
raise SystemExit(1)
print(f"PASS ({NAME}): {CHECKS} assertions; {len(entries)} PiHole names, {len(disowned)} disowned "
f"routes, {len(cf_404)} tunnel-404 names, {len(l4)} L4 names")
PY
@@ -0,0 +1,97 @@
---
date: 2026-11-09
branch: feat/pihole-explicit-lan-names
title: PiHole answers an explicit LAN name list instead of a zone wildcard (public names resolve to Cloudflare from the LAN)
---
## What changed
**`catalog/pihole.yaml` → `apps/wave-0/pihole.yaml`.** `FTLCONF_misc_dnsmasq_lines` no longer carries
`address=/${BASE_DOMAIN}/${GATEWAY_VIP}`. Every name not listed is now forwarded to 8.8.8.8/8.8.4.4 and
gets the public answer. The final list (`local-ttl=60` kept):
| Name (`.${BASE_DOMAIN}`) | Answer | Why |
|---|---|---|
| jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr, headlamp | `${GATEWAY_VIP}` | LAN-only: every HTTPRoute with `external-dns.alpha.kubernetes.io/controller: none`, derived from the tree. No public record, tunnel 404s them; this entry is their only path |
| gitea, nextcloud | `${GATEWAY_VIP}` | public, but Cloudflare's proxy caps bodies at 100 MB / responses at 100 s (git push, LFS, uploads) |
| keycloak, auth | `${GATEWAY_VIP}` | keep the OIDC flow of the LAN-path apps on one path |
| unifi | `${UNIFI_L4_VIP}` | L4 VIP, not behind the Gateway |
| crafty-mc | `${CRAFTY_L4_VIP}` | L4 VIP, not behind the Gateway |
| pihole-dns | `${PIHOLE_DNS_VIP}` | the resolver's own address |
| plex-vip | `${PLEX_L4_VIP}` | kept, see below |
Not listed, so they resolve to Cloudflare from the LAN: the apex, grafana, crafty, seerr.
**The derived LAN-only set matches the brief's list exactly** (8 names; the tree was walked, not the
list trusted). The `runbooks/serving` route has no `hostnames:` but is not disowned (it inherits the
apex, public).
**plex-vip is kept.** Nothing advertises it any more (`ADVERTISE_IP` is a `plex.direct` name since
2026-10-05), but it is not unused: it is Plex's `customCertificateDomain` (`CERTDOMAIN`, Secret
`plex-vip-tls`), the name of the tcp/32400 VIP, and the 404'd name in the tunnel config. It has no
public record, so this entry is its only resolution.
**New L0 check `conformance/tests/l0-static/pihole-lan-names`** (`scope: whole-repo`, so picked up by
the runner's scope selection with no other wiring). Asserts: (a) no zone-wide entry (zone, `#`, or any
parent of the zone); (b) every `controller: none` hostname → `${GATEWAY_VIP}`, and a disowned route
with no `hostnames:` fails; (c) every tunnel-404 hostname has an entry, with the L4 names (derived
from Services carrying both the external-dns hostname and an lbipam VIP) on their own VIP; (d) gitea
and nextcloud → `${GATEWAY_VIP}`; plus catalog == rendered value and a built-in mutation self-test.
**Stale docs corrected:** `platform/pm-agent/README.md`, `runbooks/pm-agent.md`,
`platform/externaldns/README.md`, `platform/cloudflared/README.md`, `platform/gateway/gateway.yaml`
(comment), `platform/headlamp/httproute.yaml` (comment), `apps/vms/unifi/README.md`, `README.md`,
`schemas/renderer-contract.md`; `runbooks/cloudflare-tunnel.md` gains the two consequences below.
`platform/cloudflared/deployment.yaml`: the image entrypoint comment was wrong (comment-only).
## What I verified
- **Render:** the baseline `render.py --no-image-automation catalog/pihole.yaml` was byte-identical to
the committed `apps/wave-0/pihole.yaml` before the change, and the file was replaced by that
command's output after it; `--emit-waves` was not run.
- **Mutation, against the real files (10 mutants, each rc 1 with the right assertion, each restored):**
(a) zone wildcard restored; (b) headlamp and sonarr entries removed; (b) a disowned route with no
`hostnames:`; (c) a new tunnel-404 host with no entry; (c) pihole-dns removed; (c) unifi pointed at
the Gateway; (d) gitea removed; (d) nextcloud repointed; (0) rendered value drifted from the catalog.
- **Rollout mechanics, measured live (read-only):** StatefulSet `updateStrategy` RollingUpdate
partition 0, `podManagementPolicy: Parallel`, no `minReadySeconds`, 2 replicas, PDB minAvailable 1.
The last rollout (2026-11-08 06:47–06:48Z) replaced pihole-1 then pihole-0: FTL logged "started" ~1 s
after the container, the pod went Ready ~16 s after the container, and pihole-0 started ~9 s later.
- **Expected LAN DNS blip:** no total outage; one replica answers throughout. Each of the two pod
replacements can lose a few UDP queries for ~1–2 s while the terminating pod leaves the Service
(clients retry). For roughly 30–60 s the two pods answer differently for the apex, grafana, crafty
and seerr; both answers work. Cached answers age out in 60 s (`local-ttl`).
- **Live DNS before the change:** @8.8.8.8 returns Cloudflare (104.21.51.91 / 172.67.178.27) for the
apex, grafana, seerr, crafty and gitea, and nothing for jackett; @192.168.2.23 returns 192.168.2.20
for all of them (the wildcard).
- **Cloudflared image:** `skopeo inspect --config` on the pinned digest (amd64): Entrypoint
`["cloudflared"]`, Cmd `["version"]`, User `65532:65532`.
## What I did NOT verify
- **The post-merge behaviour.** Nothing here ran against a live PiHole with the new list; the
expected answers are from dnsmasq semantics and the tree. After reconcile, check
`dig @192.168.2.23` for one name per row, and grafana returning Cloudflare.
- **Whether every LAN client path works via Cloudflare.** Measured 2xx/3xx through Cloudflare by the
brief, not re-measured from a LAN client here.
- **The FTL DNS-listening moment.** "FTL started" is a log line, not a successful query; the rollout
timing above assumes DNS is up shortly after it.
**Accepted consequences, stated plainly:**
- The apex, grafana, crafty and seerr now depend on the internet and the tunnel from the LAN too. An
ISP or Cloudflare outage, or stopping the connectors, makes them unreachable from the LAN where they
previously kept working. If grafana in particular should survive an internet outage, add it to the
list (one entry; the check permits it).
- The tunnel rollback (revert to `wan.`) now also requires restoring the wildcard; recorded in
`runbooks/cloudflare-tunnel.md`.
**Flagged, not changed:**
- The `pihole` container has no readiness probe; only the `oauth2-proxy` sidecar does, so "Ready" says
nothing about DNS. The measured margin (FTL ~1 s, Ready ~16 s) makes this benign in a normal rollout
but a slow FTL start would not hold the rollout back. A DNS readiness probe needs a schema/renderer
decision.
**Stale statements left alone:** `platform/coredns/coredns-custom.yaml` header ("both resolve the whole
zone ... by wildcard", brief said not to touch CoreDNS); `clusters/base/cluster-vars.yaml` comments
(sensitive path); `schemas/renderer-contract.md` §`exposure` first bullets (manual edge port-forward
wording); historical `infra/PROD-PROVISIONING.md`, `platform/externaldns/README.md` TODO list.
+5 -3
View File
@@ -17,6 +17,8 @@ outbound-only from the cluster (no router port forward, no hairpin, no split-DNS
| `networkpolicy.yaml` | default-deny (single owner — not an app namespace) + Prometheus scrape allow + egress CNP: world 7844 tcp/udp and 443, the Gateway (`ingress` entity), and Gateway backends by `gateway-access=shared` minus the LAN-only namespaces. |
| `servicemonitor.yaml`, `alerts.yaml` | Headless metrics Service + ServiceMonitor scraping :2000, and the two alerts, both written to fire (not go quiet) when their series vanish. |
**Phase 1 only.** This component changes no live traffic: Cloudflare sends a hostname through the
tunnel only when its DNS record is a CNAME to `<UUID>.cfargotunnel.com`, and nothing here creates
one. Moving DNS (external-dns, Gateway annotations, Pi-hole/CoreDNS split horizon) is a later change.
Cloudflare sends a hostname through the tunnel only when its DNS record is a proxied CNAME to
`<UUID>.cfargotunnel.com`; nothing in this directory creates one. Those records are published by
external-dns from the shared Gateway's target annotation (Phase 2, PR #156; see
`platform/externaldns/README.md`). LAN resolution: `catalog/pihole.yaml` (explicit names only since
2026-11-09), so public names reach Cloudflare from the LAN too.
+3 -2
View File
@@ -70,8 +70,9 @@ spec:
- name: cloudflared
# Tag + digest: Renovate's `kubernetes` manager (renovate.json5, platform/**) bumps both.
image: docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba
# The image ENTRYPOINT is `cloudflared --no-autoupdate`; these replace its CMD. The tunnel
# UUID, credentials path, metrics address and ingress rules all come from the file.
# The image ENTRYPOINT is `cloudflared` and its CMD is `version` (read from the pinned
# digest's image config, 2026-11-09); these args replace that CMD. `no-autoupdate`, the
# tunnel UUID, credentials path, metrics address and ingress rules all come from the file.
args:
- tunnel
- --config
+7 -2
View File
@@ -44,8 +44,13 @@ wan.home.keithjackson.dev A (DNS-only) <current WAN IP> <- cloudflare-ddn
- `internal` routes (`controller: none`) still publish nothing. Plex's web UI and Headlamp joined that
set on 2026-11-08 (LAN/VPN-only); the tunnel also 404s every such name (9c/9d).
- Public answers are Cloudflare anycast **A and AAAA** — Cloudflare's edge, not an origin address.
- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
(split-horizon) and never use the public path.
- **Pods** resolve the whole zone to the Gateway VIP via CoreDNS (split-horizon) and never use the
public path. **LAN clients** use PiHole, which since 2026-11-09 answers an EXPLICIT list
(`catalog/pihole.yaml`): the LAN-only names, plus gitea/nextcloud/keycloak/auth (kept off
Cloudflare's 100 MB / 100 s limits and on one OIDC path), go to the Gateway VIP and the L4 names to
their own VIPs; every other public name (apex, grafana, crafty, seerr) resolves to Cloudflare from
the LAN exactly as from anywhere else, so no split-DNS is needed for them. Guard:
`conformance/tests/l0-static/pihole-lan-names`.
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node the apiserver's
OIDC issuer `keycloak.home.keithjackson.dev` now resolves to Cloudflare and its discovery/JWKS
fetches go through the tunnel (publicly trusted edge cert; no `oidc-ca-file` is configured). The
+4 -2
View File
@@ -30,8 +30,10 @@ metadata:
# applies BEFORE it reads this target. Those same names map to `http_status:404` in the tunnel's
# ingress rules, so even a hand-made record cannot reach them through the tunnel.
#
# LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
# ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
# In-cluster clients are unaffected: CoreDNS answers ${BASE_DOMAIN} with ${GATEWAY_VIP} directly
# (split-horizon). LAN clients use PiHole's EXPLICIT list (catalog/pihole.yaml, since 2026-11-09):
# LAN-only names and gitea/nextcloud/keycloak/auth get ${GATEWAY_VIP}; every other published
# name takes this public path through Cloudflare, from the LAN as from anywhere.
#
# `wan.${BASE_DOMAIN}` (platform/externaldns/ddns.yaml) is NO LONGER a target of anything here;
# it stays only as the WireGuard udp/51820 endpoint.
+1 -1
View File
@@ -20,7 +20,7 @@ metadata:
homelab.keithjackson.dev/auth-groups: platform-admins
# LAN/VPN-ONLY (operator decision 2026-11-08). Disowned from external-dns, so the public
# headlamp record and its TXT are DELETED and nothing republishes them; LAN/VPN clients keep
# resolving the name via PiHole's split-horizon wildcard (-> the Gateway VIP). The Cloudflare
# resolving the name via PiHole's explicit entry for it (-> the Gateway VIP). The Cloudflare
# Tunnel also maps headlamp.${BASE_DOMAIN} to http_status:404 (platform/cloudflared/config.yaml).
# Same mechanism the renderer emits for `exposure: internal` services; asserted by
# conformance/tests/l0-static/internal-services-publish-no-public-dns (9c).
+7 -4
View File
@@ -72,11 +72,14 @@ Both are deliberate, and both fail **visibly** rather than silently:
- **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
*node* process, so it resolves through the node's public upstreams
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
IP — which this router does not hairpin; it answers :443 with its own placeholder
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. At the time,
the public answer for every name in this zone was a CNAME to `wan.home.keithjackson.dev`
— the WAN IP — which this router does not hairpin; it answered :443 with its own placeholder
certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
`x509: certificate is not valid for any names`. (Since 2026-11-08 the public answer is
Cloudflare's edge via the Tunnel, not the WAN IP — see `platform/externaldns/README.md`.
The pin is still the right path: it keeps registry pulls on the LAN and off Cloudflare's
proxy.) The Gateway, the wildcard certificate
and the digest were never implicated. The fix is the pin this repository already uses
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
+6 -2
View File
@@ -353,11 +353,15 @@ is deleted, which is why the last step is not optional.
'flux --context k3s-prod suspend kustomization cloudflared &&
kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0'
```
Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then
Cloudflare then answers tunnel hostnames with a 530/1033 error — **from the LAN too** for the
names PiHole does not list (apex, grafana, crafty, seerr; since 2026-11-09). Follow up in git, then
`flux resume kustomization cloudflared`.
- **After the DNS flip (Phase 2):** the path back is a revert PR of `feat/dns-via-cloudflare-tunnel`
(Gateway target back to `wan.${BASE_DOMAIN}`, drop `--cloudflare-proxied`) — external-dns then
rewrites every record DNS-only to `wan.`. That only works if the router's WAN tcp/80+443 forwards
are **re-opened** first (they are closed after Phase 2). Never edit the records by hand:
are **re-opened** first (they are closed after Phase 2). **Also restore PiHole's zone wildcard**
(`address=/${BASE_DOMAIN}/${GATEWAY_VIP}` in `catalog/pihole.yaml`, and drop the
`pihole-lan-names` L0 assertion (a)): since 2026-11-09 the apex, grafana, crafty and seerr are not
listed there and resolve publicly, and a public `wan.` answer does not hairpin from the LAN. Never edit the records by hand:
external-dns owns them and `policy: sync` reverts a hand edit. Revoking the tunnel credential (`cf tunnel delete`, or rotating it
in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.
+1 -1
View File
@@ -74,7 +74,7 @@ Work through these in order; the first three are by far the most likely.
|---|---|---|
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and (before the 2026-11-08 Tunnel cutover; the public answer is now Cloudflare) got the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets. Add `read:notification` if you want the cheap pull-request mention query (`platform/pm-agent/README.md`) |
| a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
+7 -9
View File
@@ -82,13 +82,10 @@ platform must render, and cites the requirement each mapping satisfies. Changing
all seven the moment it is set. Per-resource exclusion has exactly the blast radius of the
defect. The filter stays available as hardening once those seven are adjudicated — see
`platform/externaldns/helmrelease.yaml` and the register in the L0 check.
- `public` renders the record but the renderer does **not** yet set a target: the record still
resolves to the shared Gateway VIP, i.e. an RFC1918 address, so a `public` service is not in fact
publicly reachable. Fixing that is **not** a renderer change on v0.21.0 —
`source/gateway.go` reads `external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and
a ListenerSet), never from an HTTPRoute, so a per-service target override is not expressible while
the public services share the `shared` Gateway. The open decision is recorded in
`platform/externaldns/helmrelease.yaml`.
- `public` renders the record but the renderer does **not** set a target: external-dns reads
`external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and a ListenerSet), never from
an HTTPRoute, so the target is set once, on the `shared` Gateway — since 2026-11-08 the Cloudflare
Tunnel, proxied (`platform/gateway/gateway.yaml`, `platform/externaldns/README.md`).
- Exposure does **not** relax east-west default-deny (docs/09 §"Interaction with default-deny").
- **Requirements:** R-DNS-02/03 (docs/09), R-NET-04 (docs/03).
@@ -323,8 +320,9 @@ R-OBS-01 (the SLI must follow the rendered kind).
`public` service**. An `internal` service's route is rendered with ExternalDNS's controller
annotation set to `none`, which removes it from ExternalDNS's view entirely (see §`exposure`), so
its `<host>` exists **only** in the internal split-horizon answer PiHole serves, never in the
public zone. A `public` record's TARGET is still the Gateway VIP, not a routable address — open
operator decision, see §`exposure`.
public zone. A `public` record's TARGET is the Cloudflare Tunnel (a proxied CNAME, from the
shared Gateway's target annotation; `platform/externaldns/README.md`), and a `public` name is
not listed in PiHole unless it must stay on the LAN path.
- The `host` pattern is enforced to `<label>.home.keithjackson.dev` by the schema.
- **Requirements:** **R-TLS-01/02/03** (docs/08), **R-DNS-01/02/03** (docs/09).