home-network-k3s-impl/decisions/2026-11-08-cloudflare-tunnel.md

---
date: 2026-11-08
branch: feat/cloudflare-tunnel
title: Cloudflare Tunnel connector, Phase 1 — the component only, isolated, with zero change to live traffic
---

## What changed

Ingress is moving to a Cloudflare Tunnel because the GFiber router cannot hairpin and hands LAN
clients Google's IPv6 DNS, so split DNS keeps breaking. This is **Phase 1: the connector only.**
external-dns, the Gateway annotations, Pi-hole and CoreDNS are untouched. A later change, by a
different agent, moves DNS.

**Operator decisions (2026-11-08), recorded here as the source for the guard:**

1. Public hostnames stay `*.home.keithjackson.dev` and the apex `home.keithjackson.dev`. The
   operator enables Cloudflare ACM (ordered edge cert or Total TLS) for edge certificates.
2. **Plex stays OFF the tunnel.** No plex hostname goes through cloudflared (`plex`, `plex-vip` → 404).
3. **jackett, pihole and semaphore stay LAN/VPN-only**, never via the tunnel. The same goes for the
   non-HTTP L4 VIPs **unifi** and **crafty-mc**. After the PM's read-only audit, the other three
   catalog `exposure: internal` apps, **qbittorrent, radarr and sonarr**, are on the same list. All
   six carry the external-dns `controller: none` route annotation live.
4. **The Kubernetes API stays local-only.** It must never be reachable through the tunnel.
5. Pi-hole keeps serving the LAN. Split DNS gets trimmed later, by another agent.
6. **Headlamp is LAN/VPN-only** (operator decision, relayed by the PM later the same day). It is
   permanently in the 404 list and excluded from the egress backends. Its catalog/route exposure is
   deliberately **not** changed here; the Phase 2 DNS-flip PR does that.

**Files:**

- `platform/cloudflared/` (new):
  - `config.yaml` is the locally-managed tunnel config, projected through a content-hashed
    `configMapGenerator`, so edits roll the pods. It has explicit `http_status:404` rules for
    plex, plex-vip, jackett, pihole, pihole-dns, semaphore, qbittorrent, radarr, sonarr, headlamp,
    unifi and crafty-mc. Then the apex and `*.${BASE_DOMAIN}` go to
    `https://cilium-gateway-shared.gateway.svc.cluster.local:443` with `matchSNItoHost: true`. Last
    comes a catch-all 404. The tunnel UUID is a **literal** in this file (`REPLACE-ME-TUNNEL-UUID`
    until enrolment), not a cluster-var, so the Phase 2 PR can match it without touching `clusters/**`.
  - `cloudflared-credentials.sops.yaml` is SOPS-encrypted and holds `REPLACE-ME` placeholders, with
    a `.example` hand-off sibling. I encrypted the placeholder (not a credential) in the toolbox
    with the root `.sops.yaml` recipient.
  - `deployment.yaml` runs 2 replicas with a hard `topologySpreadConstraints` across nodes. The image
    is `cloudflare/cloudflared:2026.10.7@sha256:4955524a…` (tag + digest, Renovate `kubernetes`
    manager). It runs as non-root 65532, with a read-only root fs, all capabilities dropped and no
    SA token. Requests are 10m/64Mi and limits 500m/256Mi. Readiness uses `/ready`. Liveness uses
    `/healthcheck`, so an edge outage cannot restart-loop the pods. The pods are labelled
    `egress-class: internet`. A PDB sets `minAvailable: 1`.
  - `networkpolicy.yaml`:
    - A namespace `default-deny` with DNS only. It has a **single owner**: the namespace is not
      `app-namespace=true`, so Kyverno's generate rule never writes it.
    - A Prometheus scrape allow on :2000.
    - The egress CNP allows world tcp/udp 7844 and tcp 443, plus the `ingress` entity on tcp 443.
      It also allows **Gateway backends by `io.cilium.k8s.namespace.labels.gateway-access=shared`,
      minus the LAN-only namespaces.**
  - `servicemonitor.yaml` (headless metrics Service + ServiceMonitor) and `alerts.yaml` (`CloudflaredTunnelDown`, `CloudflaredReplicasDegraded`)
    ship with `alerts.promtool-test.yaml`, and `README.md` documents the component.
- `clusters/base/cloudflared.yaml` (new) is its own Flux Kustomization: `dependsOn: platform-config`,
  **`wait: false`**, prune, and postBuild `cluster-vars`. It is listed in `clusters/base/kustomization.yaml`.
- `clusters/{dev,prod}/kustomization.yaml` each gain a SOPS `decryption` patch for `cloudflared`.
- `conformance/tests/l0-static/cloudflared-ingress-rules/` (new, whole-repo scope) asserts several
  things:
  - Every rule targets either 404 or exactly the Gateway Service. The Service name is derived from
    `platform/gateway/gateway.yaml`, and API markers are named explicitly.
  - The excluded hosts 404 before the first Gateway rule, the Gateway rules use `matchSNItoHost`,
    there is no `noTLSVerify`, and a catch-all closes the list.
  - The CNP has no kube-apiserver, host, remote-node, cluster or `all` entity, no CIDR, and the full
    NotIn list. The pods carry egress-class internet and have no SA token.
  - The Flux tier is isolated: wait:false, not in `platform/kustomization.yaml`, nothing
    `dependsOn: cloudflared`, and both overlays decrypt it.
  - **Mutation self-test:** an added API rule, plex moved below the wildcard, and a dropped
    catch-all must each be flagged.
- `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` gains an additive
  `cloudflared_` scrape owner, with a networkpolicy companion, in the same shape as the kured entry.
- `runbooks/cloudflare-tunnel.md` (new) is registered in `runbooks/kustomization.yaml` and the
  README index.

**Design choices:**

- **Isolation (the merge-before-credentials question).** The PR is safe to merge with placeholders.
  The Deployment CrashLoops, but in its own `wait:false` Kustomization that nothing depends on. Its
  Ready condition tracks the apply, not pod health, so neither `platform`, `platform-config` nor any
  apps wave can stall. There is no Certificate or HelmRelease in the path for health checks to hang
  on. Health is reported by the two alerts instead. `CloudflaredTunnelDown` **will fire from merge
  until enrolment**. That is honest, and it is `warning` because nothing routes through the tunnel
  yet. Raise it to critical in the DNS-flip change.
- **Egress to backends, not `{}`.** Under Cilium 1.20 (CVE-2025-30162), the Gateway's Envoy
  re-checks the source pod's egress against the resolved backend. An egress rule that names only
  the Gateway would therefore make every tunnelled request `403 Access denied`
  (decisions/2026-10-30-gitea-admit-buildkit.md; catalog/render.py, `r_internet_egress`). The
  gitea-actions `- {}` pattern would also work, but it admits kube-apiserver and host, which
  decision 4 rules out. The namespace-label selector names every Gateway backend and nothing else.
  **Backends need no ingress change.** Envoy reaches them as `reserved:ingress`, and the 2026-10-30
  measurement showed the identity check sits on the source's egress: gitea-actions is in none of
  Gitea's ingress allows and still reaches it. So no app's `inClusterClients` was touched.
- **The alerts cannot self-silence.** Both expressions use `or on() vector(0)`, so absent series
  (a broken scrape, zero pods, KSM gone) fire. The promtool tests pin the absent case and the
  working-then-stale re-fire case.

## What I verified

- **The Gateway Service, measured live (read-only, `k3s-prod`, 2026-11-08).** It is
  `gateway/cilium-gateway-shared`: type LoadBalancer, ClusterIP `10.43.225.79`, EXTERNAL-IP
  `192.168.2.20`, ports 80/443, no selector, labelled `gateway.networking.k8s.io/gateway-name=shared`.
  It belongs to Gateway `gateway/shared` (class `cilium`).
- **The namespace selector, measured live.** Cilium identities carry the label
  `k8s:io.cilium.k8s.namespace.labels.gateway-access` (read from a gitea identity). Every namespace
  with an HTTPRoute (crafty, gitea, headlamp, homepage, jackett, keycloak, monitoring, nextcloud,
  oauth2-proxy, pihole, plex, qbittorrent, radarr, runbooks, seerr, semaphore, sonarr) is
  `gateway-access=shared`.
- **Nothing exists yet.** Namespace `cloudflared` does not exist on prod. **cloudflared is not in
  the pinned toolbox**, so the runbook runs the pinned image under podman instead.
- The image tag and digest came from Docker Hub's API: `2026.10.7` = `latest` =
  `sha256:4955524a…c9b9c5ba`, published 2026-10-30.
- The credentials Secret is encrypted: `stringData.credentials.json` is `ENC[…]`, with the age
  recipient from the root `.sops.yaml`.
- The new L0 check passes on the host (47 assertions, mutation self-test included). The toolbox
  gate result is reported in the PR description. I ran it after committing.

## What I did NOT verify

- **UNVERIFIED: nothing ran in a cluster.** No pod has started, and no tunnel exists. I have not
  observed placeholder CrashLoop behaviour itself. I expect cloudflared to treat
  `REPLACE-ME-TUNNEL-UUID` as a tunnel name, find no origin cert, and exit, but whatever it does,
  the pods are not Ready and the tier is non-gating.
- **UNVERIFIED: the CVE-2025-30162 path for cloudflared specifically.** The selector form was
  checked against live identity labels, but no tunnelled request has been made. The runbook's
  step 6 `kubectl debug` probe exists to prove "real backend, no 403" from the pod's own identity.
- **UNVERIFIED: whether the `ingress` entity rule's `toPorts: 443` restriction holds at Envoy.**
  buildkit's working rule has no port restriction. If the canary shows drops, removing the port
  restriction there is the first thing to try.
- The metric name `cloudflared_tunnel_ha_connections` and the `/ready` / `/healthcheck` endpoints
  come from my knowledge of cloudflared, **not measured** against 2026.10.7.
- The Cloudflare edge egress is `world` + ports, not Cloudflare CIDRs. I did not narrow it blind.
- Egress `world:443` "for the API" follows the brief. I have not confirmed that a credentials-file
  tunnel needs it at all.
- The Cloudflare-side settings (ACM, Total TLS / ordered cert, Bot Fight Mode, BIC/WAF, WebSockets)
  are documented for the operator but not inspected. I have no Cloudflare access.
- The headlamp, qbittorrent, radarr and sonarr catalog exposures are deliberately unchanged
  (Phase 2).