home-network-k3s-impl/decisions/2026-11-08-cloudflare-tunnel.md

date, branch, title

date	branch	title
2026-11-08	feat/cloudflare-tunnel	Cloudflare Tunnel connector, Phase 1 — the component only, isolated, with zero change to live traffic

What changed

Ingress is moving to a Cloudflare Tunnel because the GFiber router cannot hairpin and hands LAN clients Google's IPv6 DNS, so split DNS keeps breaking. This is Phase 1: the connector only. external-dns, the Gateway annotations, Pi-hole and CoreDNS are untouched. A later change, by a different agent, moves DNS.

Operator decisions (2026-11-08), recorded here as the source for the guard:

1. Public hostnames stay *.home.keithjackson.dev and the apex home.keithjackson.dev. The operator enables Cloudflare ACM (ordered edge cert or Total TLS) for edge certificates.
2. Plex stays OFF the tunnel. No plex hostname goes through cloudflared (plex, plex-vip → 404).
3. jackett, pihole and semaphore stay LAN/VPN-only, never via the tunnel. The same goes for the non-HTTP L4 VIPs unifi and crafty-mc. After the PM's read-only audit, the other three catalog exposure: internal apps, qbittorrent, radarr and sonarr, are on the same list. All six carry the external-dns controller: none route annotation live.
4. The Kubernetes API stays local-only. It must never be reachable through the tunnel.
5. Pi-hole keeps serving the LAN. Split DNS gets trimmed later, by another agent.
6. Headlamp is LAN/VPN-only (operator decision, relayed by the PM later the same day). It is permanently in the 404 list and excluded from the egress backends. Its catalog/route exposure is deliberately not changed here; the Phase 2 DNS-flip PR does that.

Files:

• platform/cloudflared/ (new):
  • config.yaml is the locally-managed tunnel config, projected through a content-hashed configMapGenerator, so edits roll the pods. It has explicit http_status:404 rules for plex, plex-vip, jackett, pihole, pihole-dns, semaphore, qbittorrent, radarr, sonarr, headlamp, unifi and crafty-mc. Then the apex and *.${BASE_DOMAIN} go to https://cilium-gateway-shared.gateway.svc.cluster.local:443 with matchSNItoHost: true. Last comes a catch-all 404. The tunnel UUID is a literal in this file (REPLACE-ME-TUNNEL-UUID until enrolment), not a cluster-var, so the Phase 2 PR can match it without touching clusters/**.
  • cloudflared-credentials.sops.yaml is SOPS-encrypted and holds REPLACE-ME placeholders, with a .example hand-off sibling. I encrypted the placeholder (not a credential) in the toolbox with the root .sops.yaml recipient.
  • deployment.yaml runs 2 replicas with a hard topologySpreadConstraints across nodes. The image is cloudflare/cloudflared:2026.10.7@sha256:4955524a… (tag + digest, Renovate kubernetes manager). It runs as non-root 65532, with a read-only root fs, all capabilities dropped and no SA token. Requests are 10m/64Mi and limits 500m/256Mi. Readiness uses /ready. Liveness uses /healthcheck, so an edge outage cannot restart-loop the pods. The pods are labelled egress-class: internet. A PDB sets minAvailable: 1.
  • networkpolicy.yaml:
    • A namespace default-deny with DNS only. It has a single owner: the namespace is not app-namespace=true, so Kyverno's generate rule never writes it.
    • A Prometheus scrape allow on :2000.
    • The egress CNP allows world tcp/udp 7844 and tcp 443, plus the ingress entity on tcp 443. It also allows Gateway backends by io.cilium.k8s.namespace.labels.gateway-access=shared, minus the LAN-only namespaces.
  • servicemonitor.yaml (headless metrics Service + ServiceMonitor) and alerts.yaml (CloudflaredTunnelDown, CloudflaredReplicasDegraded) ship with alerts.promtool-test.yaml, and README.md documents the component.
• clusters/base/cloudflared.yaml (new) is its own Flux Kustomization: dependsOn: platform-config, wait: false, prune, and postBuild cluster-vars. It is listed in clusters/base/kustomization.yaml.
• clusters/{dev,prod}/kustomization.yaml each gain a SOPS decryption patch for cloudflared.
• conformance/tests/l0-static/cloudflared-ingress-rules/ (new, whole-repo scope) asserts several things:
  • Every rule targets either 404 or exactly the Gateway Service. The Service name is derived from platform/gateway/gateway.yaml, and API markers are named explicitly.
  • The excluded hosts 404 before the first Gateway rule, the Gateway rules use matchSNItoHost, there is no noTLSVerify, and a catch-all closes the list.
  • The CNP has no kube-apiserver, host, remote-node, cluster or all entity, no CIDR, and the full NotIn list. The pods carry egress-class internet and have no SA token.
  • The Flux tier is isolated: wait:false, not in platform/kustomization.yaml, nothing dependsOn: cloudflared, and both overlays decrypt it.
  • Mutation self-test: an added API rule, plex moved below the wildcard, and a dropped catch-all must each be flagged.
• conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh gains an additive cloudflared_ scrape owner, with a networkpolicy companion, in the same shape as the kured entry.
• runbooks/cloudflare-tunnel.md (new) is registered in runbooks/kustomization.yaml and the README index.

Design choices:

• Isolation (the merge-before-credentials question). The PR is safe to merge with placeholders. The Deployment CrashLoops, but in its own wait:false Kustomization that nothing depends on. Its Ready condition tracks the apply, not pod health, so neither platform, platform-config nor any apps wave can stall. There is no Certificate or HelmRelease in the path for health checks to hang on. Health is reported by the two alerts instead. CloudflaredTunnelDown will fire from merge until enrolment. That is honest, and it is warning because nothing routes through the tunnel yet. Raise it to critical in the DNS-flip change.
• Egress to backends, not {}. Under Cilium 1.20 (CVE-2025-30162), the Gateway's Envoy re-checks the source pod's egress against the resolved backend. An egress rule that names only the Gateway would therefore make every tunnelled request 403 Access denied (decisions/2026-10-30-gitea-admit-buildkit.md; catalog/render.py, r_internet_egress). The gitea-actions - {} pattern would also work, but it admits kube-apiserver and host, which decision 4 rules out. The namespace-label selector names every Gateway backend and nothing else. Backends need no ingress change. Envoy reaches them as reserved:ingress, and the 2026-10-30 measurement showed the identity check sits on the source's egress: gitea-actions is in none of Gitea's ingress allows and still reaches it. So no app's inClusterClients was touched.
• The alerts cannot self-silence. Both expressions use or on() vector(0), so absent series (a broken scrape, zero pods, KSM gone) fire. The promtool tests pin the absent case and the working-then-stale re-fire case.

What I verified

• The Gateway Service, measured live (read-only, k3s-prod, 2026-11-08). It is gateway/cilium-gateway-shared: type LoadBalancer, ClusterIP 10.43.225.79, EXTERNAL-IP 192.168.2.20, ports 80/443, no selector, labelled gateway.networking.k8s.io/gateway-name=shared. It belongs to Gateway gateway/shared (class cilium).
• The namespace selector, measured live. Cilium identities carry the label k8s:io.cilium.k8s.namespace.labels.gateway-access (read from a gitea identity). Every namespace with an HTTPRoute (crafty, gitea, headlamp, homepage, jackett, keycloak, monitoring, nextcloud, oauth2-proxy, pihole, plex, qbittorrent, radarr, runbooks, seerr, semaphore, sonarr) is gateway-access=shared.
• Nothing exists yet. Namespace cloudflared does not exist on prod. cloudflared is not in the pinned toolbox, so the runbook runs the pinned image under podman instead.
• The image tag and digest came from Docker Hub's API: 2026.10.7 = latest = sha256:4955524a…c9b9c5ba, published 2026-10-30.
• The credentials Secret is encrypted: stringData.credentials.json is ENC[…], with the age recipient from the root .sops.yaml.
• The new L0 check passes on the host (47 assertions, mutation self-test included). The toolbox gate result is reported in the PR description. I ran it after committing.

What I did NOT verify

• UNVERIFIED: nothing ran in a cluster. No pod has started, and no tunnel exists. I have not observed placeholder CrashLoop behaviour itself. I expect cloudflared to treat REPLACE-ME-TUNNEL-UUID as a tunnel name, find no origin cert, and exit, but whatever it does, the pods are not Ready and the tier is non-gating.
• UNVERIFIED: the CVE-2025-30162 path for cloudflared specifically. The selector form was checked against live identity labels, but no tunnelled request has been made. The runbook's step 6 kubectl debug probe exists to prove "real backend, no 403" from the pod's own identity.
• UNVERIFIED: whether the ingress entity rule's toPorts: 443 restriction holds at Envoy. buildkit's working rule has no port restriction. If the canary shows drops, removing the port restriction there is the first thing to try.
• The metric name cloudflared_tunnel_ha_connections and the /ready / /healthcheck endpoints come from my knowledge of cloudflared, not measured against 2026.10.7.
• The Cloudflare edge egress is world + ports, not Cloudflare CIDRs. I did not narrow it blind.
• Egress world:443 "for the API" follows the brief. I have not confirmed that a credentials-file tunnel needs it at all.
• The Cloudflare-side settings (ACM, Total TLS / ordered cert, Bot Fight Mode, BIC/WAF, WebSockets) are documented for the operator but not inspected. I have no Cloudflare access.
• The headlamp, qbittorrent, radarr and sonarr catalog exposures are deliberately unchanged (Phase 2).