From aabc8c60b6c5c9db25d363dcd98cabc27c5636f4 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 8 Nov 2026 12:38:22 -0600 Subject: [PATCH] feat(cloudflared): Cloudflare Tunnel connector, Phase 1 (isolated, no traffic change) Adds platform/cloudflared: a locally-managed tunnel whose ingress rules live in git. LAN/VPN-only hosts (plex, plex-vip, jackett, pihole, pihole-dns, semaphore, qbittorrent, radarr, sonarr, headlamp, unifi, crafty-mc) map to 404. The apex and *.home go to cilium-gateway-shared.gateway over TLS with matchSNItoHost. Last comes a catch-all 404. Nothing routes to the Kubernetes API. - 2-replica Deployment, hard node spread, non-root, pinned digest, PDB - default-deny plus an egress CNP: Cloudflare edge 7844/443, the Gateway, and Gateway backends by gateway-access=shared minus the LAN-only namespaces (needed because of the CVE-2025-30162 L7LB egress re-check) - PodMonitor plus two alerts that fire on absent series, with promtool tests - its own Flux Kustomization (dependsOn platform-config, wait:false, nothing depends on it), plus SOPS decryption patches in both overlays - placeholder credentials (SOPS-encrypted) so it can merge before enrolment - L0 cloudflared-ingress-rules, and a cloudflared_ scrape owner - runbooks/cloudflare-tunnel.md and decisions/2026-11-08-cloudflare-tunnel.md Co-Authored-By: Claude Opus 5.5 --- clusters/base/cloudflared.yaml | 61 ++++ clusters/base/kustomization.yaml | 7 + clusters/dev/kustomization.yaml | 17 ++ clusters/prod/kustomization.yaml | 17 ++ .../check.sh | 25 ++ .../chainsaw-test.yaml | 45 +++ .../cloudflared-ingress-rules/check.sh | 248 +++++++++++++++ decisions/2026-11-08-cloudflare-tunnel.md | 139 +++++++++ platform/cloudflared/README.md | 22 ++ .../cloudflared/alerts.promtool-test.yaml | 128 ++++++++ platform/cloudflared/alerts.yaml | 64 ++++ .../cloudflared-credentials.sops.yaml | 32 ++ .../cloudflared-credentials.sops.yaml.example | 34 +++ platform/cloudflared/config.yaml | 84 +++++ platform/cloudflared/deployment.yaml | 140 +++++++++ platform/cloudflared/kustomization.yaml | 35 +++ platform/cloudflared/namespace.yaml | 18 ++ platform/cloudflared/networkpolicy.yaml | 149 +++++++++ platform/cloudflared/podmonitor.yaml | 22 ++ runbooks/README.md | 3 + runbooks/cloudflare-tunnel.md | 288 ++++++++++++++++++ runbooks/kustomization.yaml | 1 + 22 files changed, 1579 insertions(+) create mode 100644 clusters/base/cloudflared.yaml create mode 100644 conformance/tests/l0-static/cloudflared-ingress-rules/chainsaw-test.yaml create mode 100755 conformance/tests/l0-static/cloudflared-ingress-rules/check.sh create mode 100644 decisions/2026-11-08-cloudflare-tunnel.md create mode 100644 platform/cloudflared/README.md create mode 100644 platform/cloudflared/alerts.promtool-test.yaml create mode 100644 platform/cloudflared/alerts.yaml create mode 100644 platform/cloudflared/cloudflared-credentials.sops.yaml create mode 100644 platform/cloudflared/cloudflared-credentials.sops.yaml.example create mode 100644 platform/cloudflared/config.yaml create mode 100644 platform/cloudflared/deployment.yaml create mode 100644 platform/cloudflared/kustomization.yaml create mode 100644 platform/cloudflared/namespace.yaml create mode 100644 platform/cloudflared/networkpolicy.yaml create mode 100644 platform/cloudflared/podmonitor.yaml create mode 100644 runbooks/cloudflare-tunnel.md diff --git a/clusters/base/cloudflared.yaml b/clusters/base/cloudflared.yaml new file mode 100644 index 0000000..fd081ae --- /dev/null +++ b/clusters/base/cloudflared.yaml @@ -0,0 +1,61 @@ +--- +# Flux Kustomization: Cloudflare Tunnel connector — reconciles ./platform/cloudflared. +# +# ISOLATION IS THE POINT OF THIS FILE. The Deployment it applies CANNOT become Ready until the +# operator enrols the tunnel (runbooks/cloudflare-tunnel.md): it ships with REPLACE-ME credentials +# and a placeholder UUID, and a wrong or revoked credential later produces the same state. So: +# +# * ITS OWN Kustomization, not a line in platform/kustomization.yaml. `platform` is wait:true and +# gates platform-config, which gates every apps wave; an unenrolled tunnel there would be a +# platform-wide stall ON MERGE, with the cause (a missing manual step) invisible from any Flux +# condition. Same reasoning, and same failure shape, as vpn-ingress's unfilled server key. +# * wait: false. Its pods are legitimately not-Ready whenever the credentials are absent or the +# edge is unreachable. Health is reported by the CloudflaredTunnelDown / ReplicasDegraded +# alerts (platform/cloudflared/alerts.yaml), not by this Kustomization's Ready condition — +# which reaches Ready on APPLY success. +# * NOTHING dependsOn THIS, and nothing should. A missing or invalid credentials Secret can then +# only ever affect the tunnel itself. conformance/tests/l0-static/cloudflared-ingress-rules +# fails the gate if any Kustomization gains `dependsOn: cloudflared`, or if this one stops +# being wait:false. +# * No Certificate, no HelmRelease, no CRD: nothing in the path for Flux's health checks to wait +# on even if wait were flipped (the Certificate-stall lesson). +# +# dependsOn: [platform-config] — A REAL EDGE. +# 1. CRD ORDERING: PodMonitor and PrometheusRule are monitoring.coreos.com CRs whose CRDs are +# installed by kube-prometheus-stack in `platform`; platform-config dependsOn platform, so this +# edge carries that guarantee transitively (Flux aborts a whole pass on an unknown kind). +# 2. THE FAR SIDE OF THE TUNNEL: the Gateway (and so the cilium-gateway-shared Service the ingress +# rules point at) is applied by platform-config. A tunnel that comes up before its origin is a +# support call. +# +# ZERO EFFECT ON LIVE TRAFFIC UNTIL DNS MOVES. Cloudflare only sends a hostname through a tunnel +# when that hostname's DNS record is a CNAME to .cfargotunnel.com. This change creates no +# such record and leaves external-dns, the Gateway annotations, Pi-hole and CoreDNS untouched. +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: cloudflared + namespace: flux-system +spec: + interval: 10m + retryInterval: 2m + timeout: 5m + prune: true + wait: false + dependsOn: + - name: platform-config + sourceRef: + kind: GitRepository + name: flux-system + path: ./platform/cloudflared + # REQUIRED, not boilerplate: platform/cloudflared/config.yaml renders `${BASE_DOMAIN}` into every + # ingress hostname, and alerts.yaml into both runbook_url annotations. Without this block the + # token becomes the EMPTY STRING and the Kustomization still reports Ready (substitution fails + # OPEN), leaving ingress rules for `plex.` and `*.` — a tunnel that 404s everything at best. + postBuild: + substituteFrom: + - kind: ConfigMap + name: cluster-vars + # SOPS decryption is layered per-environment: clusters/dev and clusters/prod each patch THIS + # Kustomization by name with a `decryption:` stanza for cloudflared-credentials.sops.yaml. + # Without it the Secret applies cleanly full of ENC[...] and cloudflared cannot authenticate. diff --git a/clusters/base/kustomization.yaml b/clusters/base/kustomization.yaml index d56e149..810eebe 100644 --- a/clusters/base/kustomization.yaml +++ b/clusters/base/kustomization.yaml @@ -78,6 +78,13 @@ resources: # `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have # stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch. - vpn-ingress.yaml # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config) + # Cloudflare Tunnel connector (north-south INGRESS from the Cloudflare edge) — POST-platform tier + # (dependsOn platform-config, wait:false), and NOTHING dependsOn it. Non-gating is load-bearing, + # as for vpn-ingress: the Deployment cannot become Ready until the operator enrols the tunnel + # (runbooks/cloudflare-tunnel.md), so in a gating tier an unfinished manual step — or a revoked + # credential later — would stall every apps wave. Carries cloudflared-credentials.sops.yaml, so + # both overlays add a SOPS decryption patch for it (by name). See clusters/base/cloudflared.yaml. + - cloudflared.yaml # Flux Kustomization -> ./platform/cloudflared (dependsOn platform-config) # PM agent — reads Gitea issues and runs one worker agent per assignment that authors a # change, gates it and opens a pull request. POST-APP tier (dependsOn apps-wave-0, # wait:false): what it needs is GITEA, not Keycloak or the Gateway, and it must never gate diff --git a/clusters/dev/kustomization.yaml b/clusters/dev/kustomization.yaml index 3e4fc87..f2fa504 100644 --- a/clusters/dev/kustomization.yaml +++ b/clusters/dev/kustomization.yaml @@ -140,6 +140,23 @@ patches: provider: sops secretRef: name: sops-age + # cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries + # platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one + # secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY + # SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot + # authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band + # (name only). + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: cloudflared + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot # account and the Anthropic subscription token from `claude setup-token`. Without this diff --git a/clusters/prod/kustomization.yaml b/clusters/prod/kustomization.yaml index ad562df..5904bce 100644 --- a/clusters/prod/kustomization.yaml +++ b/clusters/prod/kustomization.yaml @@ -131,6 +131,23 @@ patches: provider: sops secretRef: name: sops-age + # cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries + # platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one + # secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY + # SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot + # authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band + # (name only). + - patch: |- + apiVersion: kustomize.toolkit.fluxcd.io/v1 + kind: Kustomization + metadata: + name: cloudflared + namespace: flux-system + spec: + decryption: + provider: sops + secretRef: + name: sops-age # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot # account and the Anthropic subscription token from `claude setup-token`. Without this diff --git a/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh b/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh index 78d065e..4e446eb 100755 --- a/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh +++ b/conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh @@ -411,6 +411,31 @@ SCRAPE_OWNERS: list[dict] = [ "Keeping the two in one file makes that pair inseparable", }, }, + # ── ADDED 2026-11-08 (feat/cloudflare-tunnel). A NEW OWNER, NOT A RELAXED ASSERTION. ─────── + # Purely additive, the same shape as the kured entry: it registers the cloudflared_* family + # and the PodMonitor that scrapes it, and makes the check STRICTER for that component. + { + "owner": "cloudflared: the PodMonitor on the `metrics` (:2000) container port", + "prefixes": ("cloudflared_",), + "file": "platform/cloudflared/podmonitor.yaml", + "require": [ + r"^kind:\s*PodMonitor\s*$", + r"^\s*-\s*port:\s*metrics\s*$", + ], + "forbid": [], + "why": "cloudflared_tunnel_ha_connections powers CloudflaredTunnelDown, the only signal " + "that the tunnel has lost the Cloudflare edge. That rule is written to FIRE on an " + "absent series, so a lost scrape is loud rather than inert — but it would then be " + "permanently firing for a wiring reason, which teaches the operator to ignore it", + "companion": { + "file": "platform/cloudflared/kustomization.yaml", + "require": [r"^\s*-\s*networkpolicy\.yaml\s*$"], + "why": "platform/cloudflared/networkpolicy.yaml ships the namespace's default-deny " + "(a platform namespace, outside Kyverno's generator) together with the ingress " + "allow for Prometheus on :2000. Keeping them in one file, applied together, is " + "what stops a baseline-without-allow from emptying the TSDB", + }, + }, ] # `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1) diff --git a/conformance/tests/l0-static/cloudflared-ingress-rules/chainsaw-test.yaml b/conformance/tests/l0-static/cloudflared-ingress-rules/chainsaw-test.yaml new file mode 100644 index 0000000..82ea151 --- /dev/null +++ b/conformance/tests/l0-static/cloudflared-ingress-rules/chainsaw-test.yaml @@ -0,0 +1,45 @@ +# ============================================================================= +# L0 STATIC — the Cloudflare Tunnel exposes only what the operator decided, and cannot stall Flux +# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster. +# +# platform/cloudflared/config.yaml is the entire internet-facing surface of the tunnel. This test +# asserts the operator decisions of 2026-11-08 against it on every gate run: no rule may target +# anything but a 404 or the shared Gateway (so never the Kubernetes API); the LAN/VPN-only +# hostnames (plex, jackett, pihole, semaphore, qbittorrent, radarr, sonarr, headlamp, +# unifi, crafty-mc) 404 before the wildcard; the +# egress policy admits no API/node entity; and the Flux tier is isolated (wait:false, nothing +# dependsOn it, both overlays decrypt it). It proves itself non-vacuous by mutation. Rationale and +# the honest NOT-proven list: check.sh. +# +# WHY SCOPE, NOT REQUIREMENTS: the property spans platform/cloudflared, platform/gateway (the +# derived Service name), clusters/base and both overlays. A requirement-scoped selection would skip +# it on, e.g., a Gateway rename — precisely a change that breaks it. +# ============================================================================= +apiVersion: chainsaw.kyverno.io/v1alpha1 +kind: Test +metadata: + name: cloudflared-ingress-rules + labels: + conformance.home.k3s/R-NET-01: "true" + conformance.home.k3s/R-NET-02: "true" + annotations: + conformance.home.k3s/requirements: "R-NET-01, R-NET-02" + conformance.home.k3s/layer: "L0" + conformance.home.k3s/scope: "whole-repo" + conformance.home.k3s/safety-critical: "true" + conformance.home.k3s/executable: "true" + conformance.home.k3s/intent: >- + The Cloudflare Tunnel routes only to the shared Gateway or a 404, never to the Kubernetes + API; the LAN/VPN-only hostnames 404 ahead of the wildcard; the connector's egress admits no + API or node entity; and its Flux tier cannot stall any other tier. +spec: + steps: + - name: assert-tunnel-ingress-rules-and-isolation + description: >- + Parse platform/cloudflared/config.yaml with BASE_DOMAIN substituted, assert every rule's + target against the Gateway Service derived from platform/gateway/gateway.yaml, assert the + excluded hostnames 404 first, check the egress CiliumNetworkPolicy and the Flux wiring, and + run the evaluator on mutated configs to prove it is not vacuous. + try: + - script: + content: bash ./check.sh diff --git a/conformance/tests/l0-static/cloudflared-ingress-rules/check.sh b/conformance/tests/l0-static/cloudflared-ingress-rules/check.sh new file mode 100755 index 0000000..b1a4db6 --- /dev/null +++ b/conformance/tests/l0-static/cloudflared-ingress-rules/check.sh @@ -0,0 +1,248 @@ +#!/usr/bin/env bash +# ============================================================================= +# L0 static check (offline, no cluster) — cloudflared-ingress-rules +# Serves: R-NET-01 / R-NET-02 (default-deny; allow rules declarative and scoped), and the four +# operator decisions of 2026-11-08 recorded in decisions/2026-11-08-cloudflare-tunnel.md. +# +# WHY THIS EXISTS. platform/cloudflared/config.yaml is the whole of what the internet can reach +# through the Cloudflare Tunnel. A one-line edit there — a rule pointing at the Kubernetes API, a +# LAN-only hostname moved below the wildcard, a dropped catch-all — publishes something the operator +# decided must never be public, and nothing in the cluster turns red: the tunnel stays connected and +# the Kustomization stays Ready. So the decisions are asserted here, on every gate run. +# +# GIVEN the repository as authored, +# THEN (1) every ingress rule's `service` is `http_status:404` or EXACTLY the shared Gateway's +# in-cluster Service, derived from platform/gateway/gateway.yaml (so a Gateway rename +# fails here rather than in production) — which by construction excludes the +# Kubernetes API, and a named deny-list says so explicitly when it is violated; +# (2) each LAN/VPN-only hostname (plex, plex-vip, jackett, pihole, pihole-dns, semaphore, +# qbittorrent, radarr, sonarr, headlamp, unifi, crafty-mc) has a rule mapping it to http_status:404 that comes BEFORE any +# Gateway-bound rule (cloudflared matches first-rule-wins); +# (3) the apex and the wildcard are the Gateway rules, with matchSNItoHost, and nothing +# disables origin TLS verification; the last rule is a hostname-less 404 catch-all; +# (4) the egress CiliumNetworkPolicy admits none of kube-apiserver/host/remote-node/ +# cluster/all and no CIDR rule, and its backend selector excludes the LAN-only +# namespaces (the second, independent layer behind (2)); +# (5) the Flux tier is ISOLATED: its own Kustomization, wait:false, not listed in +# platform/kustomization.yaml, nothing anywhere dependsOn it, and both overlays carry its +# SOPS decryption patch — so a missing or invalid credential can stall nothing else; +# AND (6) the rule evaluator is not vacuous: it is run against mutated copies of the real config +# (an API rule added, plex moved below the wildcard, the catch-all removed) and must +# flag each one. +# +# NOT proven (static): that Cloudflare routes only the hostnames whose DNS points at the tunnel, +# that the Gateway's certificate validates from cloudflared, or that Envoy enforces (4) — those are +# runtime properties, verified by the operator per runbooks/cloudflare-tunnel.md. +# ============================================================================= +set -euo pipefail +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root +exec python3 - "$ROOT" <<'PY' +import copy +import re +import sys +from pathlib import Path + +try: + import yaml +except ImportError: # pragma: no cover + print("CHECK FAIL (cloudflared-ingress-rules): PyYAML is required", file=sys.stderr) + raise SystemExit(1) + +ROOT = Path(sys.argv[1]) +FAILURES, CHECKS = [], 0 + + +def check(cond, desc): + global CHECKS + CHECKS += 1 + if not cond: + FAILURES.append(desc) + + +def docs(path): + return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)] + + +COMP = ROOT / "platform" / "cloudflared" +CONFIG = COMP / "config.yaml" +if not CONFIG.is_file(): + print(f"CHECK FAIL (cloudflared-ingress-rules): {CONFIG.relative_to(ROOT)} is absent — the " + "component this check guards was moved or deleted; re-point or delete this check in the " + "SAME change", file=sys.stderr) + raise SystemExit(1) + +# The hostnames the operator decided never go through the tunnel (decisions 2026-11-08). +# headlamp: operator decision 2026-11-08 (LAN/VPN-only). qbittorrent/radarr/sonarr: catalog +# `exposure: internal`, like jackett/pihole/semaphore. +EXCLUDED_HOSTS = ("plex", "plex-vip", "jackett", "pihole", "pihole-dns", "semaphore", + "qbittorrent", "radarr", "sonarr", "headlamp", "unifi", "crafty-mc") +# Their Gateway-backend namespaces (unifi / crafty-mc are L4 VIPs, not Gateway backends). +EXCLUDED_NAMESPACES = ("plex", "jackett", "pihole", "semaphore", "qbittorrent", "radarr", + "sonarr", "headlamp") +FORBIDDEN_ENTITIES = {"kube-apiserver", "host", "remote-node", "cluster", "all"} +# Any of these in a `service` means a route toward the Kubernetes API (or a raw-TCP/SSH bastion). +API_MARKERS = ("kubernetes", "kube-apiserver", "apiserver", ":6443", "10.43.0.1", "tcp://", + "ssh://", "unix:", "bastion") + +# ── substitution: ${BASE_DOMAIN} from cluster-vars, the ONLY token allowed ────────────────────── +cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml") + if d.get("kind") == "ConfigMap") +base_domain = cv["data"]["BASE_DOMAIN"] +raw = CONFIG.read_text() +tokens = set(re.findall(r"\$\{([A-Za-z0-9_]+)\}", raw)) +check(tokens == {"BASE_DOMAIN"}, + f"(0) config.yaml uses exactly the ${{BASE_DOMAIN}} token (found {sorted(tokens)}); any other " + f"token is substituted from cluster-vars unseen by this check") +check(re.search(r"\$(?!\{BASE_DOMAIN\})", raw) is None, + "(0) config.yaml contains a bare `$` — Flux envsubst would rewrite or empty it") +cfg = yaml.safe_load(raw.replace("${BASE_DOMAIN}", base_domain)) + +# ── the Gateway Service, derived rather than hard-coded ───────────────────────────────────────── +gw = next(d for d in docs(ROOT / "platform" / "gateway" / "gateway.yaml") if d.get("kind") == "Gateway") +GATEWAY_SVC = (f"https://cilium-gateway-{gw['metadata']['name']}." + f"{gw['metadata']['namespace']}.svc.cluster.local:443") + + +def evaluate(cfg): + """Return the list of rule violations for a parsed cloudflared config.""" + bad = [] + rules = cfg.get("ingress") or [] + if len(rules) < len(EXCLUDED_HOSTS) + 3: + bad.append(f"only {len(rules)} ingress rules — the scan is vacuous") + return bad + for i, r in enumerate(rules): + svc = str(r.get("service", "")) + if any(m in svc.lower() for m in API_MARKERS): + bad.append(f"rule {i} ({r.get('hostname', '')}) targets {svc!r}, which " + f"is a route toward the Kubernetes API / a raw bastion — the API stays " + f"local-only (operator decision 2026-11-08)") + if svc not in ("http_status:404", GATEWAY_SVC): + bad.append(f"rule {i} ({r.get('hostname', '')}) targets {svc!r}; only " + f"http_status:404 or {GATEWAY_SVC} are permitted") + if (r.get("originRequest") or {}).get("noTLSVerify"): + bad.append(f"rule {i} disables origin TLS verification (noTLSVerify)") + if (cfg.get("originRequest") or {}).get("noTLSVerify"): + bad.append("top-level originRequest disables origin TLS verification (noTLSVerify)") + + gw_idx = [i for i, r in enumerate(rules) if r.get("service") == GATEWAY_SVC] + first_gw = min(gw_idx) if gw_idx else len(rules) + for h in EXCLUDED_HOSTS: + fqdn = f"{h}.{base_domain}" + hits = [i for i, r in enumerate(rules) if r.get("hostname") == fqdn] + if not hits: + bad.append(f"no rule for LAN/VPN-only host {fqdn}; it would fall through to the " + f"wildcard and reach the Gateway") + continue + i = hits[0] + if rules[i].get("service") != "http_status:404": + bad.append(f"{fqdn} maps to {rules[i].get('service')!r}, not http_status:404") + if i > first_gw: + bad.append(f"{fqdn}'s 404 rule (#{i}) comes AFTER the first Gateway rule (#{first_gw}); " + f"cloudflared is first-match, so the Gateway rule wins") + + gw_hosts = {rules[i].get("hostname") for i in gw_idx} + if gw_hosts != {base_domain, f"*.{base_domain}"}: + bad.append(f"the Gateway-bound hostnames are {sorted(map(str, gw_hosts))}; expected " + f"exactly the apex and the wildcard") + for i in gw_idx: + if (rules[i].get("originRequest") or {}).get("matchSNItoHost") is not True: + bad.append(f"Gateway rule #{i} lacks originRequest.matchSNItoHost: true — the Gateway " + f"would not see the SNI that selects its wildcard certificate") + last = rules[-1] + if "hostname" in last or last.get("service") != "http_status:404": + bad.append("the last rule is not a hostname-less http_status:404 catch-all") + return bad + + +# (1)-(3) the real config +for v in evaluate(cfg): + check(False, f"(1-3) {v}") +check(str(cfg.get("credentials-file", "")).startswith("/etc/cloudflared/creds/"), + "(3) credentials-file points into the mounted Secret volume") + +# (6) mutation self-test — the evaluator must flag each broken variant +mutants = {} +m = copy.deepcopy(cfg) +m["ingress"].insert(0, {"hostname": f"k8s.{base_domain}", + "service": "https://kubernetes.default.svc:443"}) +mutants["API rule added"] = m +m = copy.deepcopy(cfg) +plex = next(r for r in m["ingress"] if r.get("hostname") == f"plex.{base_domain}") +m["ingress"].remove(plex) +m["ingress"].insert(len(m["ingress"]) - 1, plex) +mutants["plex moved below the wildcard"] = m +m = copy.deepcopy(cfg) +m["ingress"] = m["ingress"][:-1] +mutants["catch-all removed"] = m +for name, mut in mutants.items(): + check(evaluate(mut), f"(6) self-test: the evaluator did NOT flag the mutant '{name}' — the " + f"check is vacuous") + +# (4) the egress policy +np_docs = docs(COMP / "networkpolicy.yaml") +cnps = [d for d in np_docs if d.get("kind") == "CiliumNetworkPolicy"] +check(len(cnps) == 1, f"(4) exactly one CiliumNetworkPolicy in networkpolicy.yaml (found {len(cnps)})") +for cnp in cnps: + for i, rule in enumerate(cnp["spec"].get("egress") or []): + ents = set(rule.get("toEntities") or []) + check(not (ents & FORBIDDEN_ENTITIES), + f"(4) egress rule {i} admits {sorted(ents & FORBIDDEN_ENTITIES)} — that reaches the " + f"Kubernetes API or the nodes; the API stays local-only") + check(not (rule.get("toCIDR") or rule.get("toCIDRSet")), + f"(4) egress rule {i} uses a CIDR — a CIDR can name the API VIP; use entities") + for ep in rule.get("toEndpoints") or []: + notin = [e for e in ep.get("matchExpressions") or [] + if e.get("key") == "io.kubernetes.pod.namespace" and e.get("operator") == "NotIn"] + missing = set(EXCLUDED_NAMESPACES) - set(notin[0]["values"] if notin else []) + check(not missing, + f"(4) backend selector in egress rule {i} does not exclude {sorted(missing)}") +for d in np_docs: + if d.get("kind") == "NetworkPolicy" and d["metadata"]["name"] == "default-deny": + check(set(d["spec"].get("policyTypes", [])) == {"Ingress", "Egress"}, + "(4) default-deny covers both directions") + break +else: + check(False, "(4) networkpolicy.yaml ships no default-deny baseline for the namespace") + +# (5) Flux isolation +fk = next(d for d in docs(ROOT / "clusters" / "base" / "cloudflared.yaml") + if d.get("kind") == "Kustomization") +check(fk["spec"].get("wait") is False, + "(5) clusters/base/cloudflared.yaml must be wait: false — its pods are legitimately not-Ready " + "until the tunnel is enrolled") +check(fk["spec"].get("path") == "./platform/cloudflared", "(5) the Kustomization reconciles ./platform/cloudflared") +check("cloudflared.yaml" in (ROOT / "clusters" / "base" / "kustomization.yaml").read_text(), + "(5) clusters/base/kustomization.yaml lists cloudflared.yaml") +check("cloudflared" not in (ROOT / "platform" / "kustomization.yaml").read_text(), + "(5) platform/kustomization.yaml must not include cloudflared — that is the gating tier") +for f in sorted((ROOT / "clusters").rglob("*.yaml")): + for d in docs(f): + for dep in (d.get("spec") or {}).get("dependsOn") or []: + check(not (isinstance(dep, dict) and dep.get("name") == "cloudflared"), + f"(5) {f.relative_to(ROOT)} {d.get('metadata', {}).get('name')} dependsOn " + f"cloudflared — that hands an unenrolled tunnel the power to stall it") +for env in ("dev", "prod"): + text = (ROOT / "clusters" / env / "kustomization.yaml").read_text() + check(re.search(r"name:\s*cloudflared\s*\n\s*namespace:\s*flux-system\s*\n\s*spec:\s*\n" + r"\s*decryption:", text) is not None, + f"(5) clusters/{env}/kustomization.yaml carries the cloudflared SOPS decryption patch") + +dep = next(d for d in docs(COMP / "deployment.yaml") if d.get("kind") == "Deployment") +check(dep["spec"].get("replicas", 1) >= 2, "(5) the Deployment runs at least 2 replicas") +check(dep["spec"]["template"]["metadata"]["labels"].get("homelab.keithjackson.dev/egress-class") + == "internet", + "(4) the pods are egress-class=internet — vpn would put the edge connections behind PIA") +check(dep["spec"]["template"]["spec"].get("automountServiceAccountToken") is False, + "(4) the pods mount no ServiceAccount token") + +if CHECKS < 25: + FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated") +if FAILURES: + print(f"CHECK FAIL (cloudflared-ingress-rules): {len(FAILURES)} of {CHECKS} assertions failed", + file=sys.stderr) + for f in FAILURES: + print(f" - {f}", file=sys.stderr) + raise SystemExit(1) +print(f"PASS (cloudflared-ingress-rules): {CHECKS} assertions") +PY diff --git a/decisions/2026-11-08-cloudflare-tunnel.md b/decisions/2026-11-08-cloudflare-tunnel.md new file mode 100644 index 0000000..252abd4 --- /dev/null +++ b/decisions/2026-11-08-cloudflare-tunnel.md @@ -0,0 +1,139 @@ +--- +date: 2026-11-08 +branch: feat/cloudflare-tunnel +title: Cloudflare Tunnel connector, Phase 1 — the component only, isolated, with zero change to live traffic +--- + +## What changed + +Ingress is moving to a Cloudflare Tunnel because the GFiber router cannot hairpin and hands LAN +clients Google's IPv6 DNS, so split DNS keeps breaking. This is **Phase 1: the connector only.** +external-dns, the Gateway annotations, Pi-hole and CoreDNS are untouched. A later change, by a +different agent, moves DNS. + +**Operator decisions (2026-11-08), recorded here as the source for the guard:** + +1. Public hostnames stay `*.home.keithjackson.dev` and the apex `home.keithjackson.dev`. The + operator enables Cloudflare ACM (ordered edge cert or Total TLS) for edge certificates. +2. **Plex stays OFF the tunnel.** No plex hostname goes through cloudflared (`plex`, `plex-vip` → 404). +3. **jackett, pihole and semaphore stay LAN/VPN-only**, never via the tunnel. The same goes for the + non-HTTP L4 VIPs **unifi** and **crafty-mc**. After the PM's read-only audit, the other three + catalog `exposure: internal` apps, **qbittorrent, radarr and sonarr**, are on the same list. All + six carry the external-dns `controller: none` route annotation live. +4. **The Kubernetes API stays local-only.** It must never be reachable through the tunnel. +5. Pi-hole keeps serving the LAN. Split DNS gets trimmed later, by another agent. +6. **Headlamp is LAN/VPN-only** (operator decision, relayed by the PM later the same day). It is + permanently in the 404 list and excluded from the egress backends. Its catalog/route exposure is + deliberately **not** changed here; the Phase 2 DNS-flip PR does that. + +**Files:** + +- `platform/cloudflared/` (new): + - `config.yaml` is the locally-managed tunnel config, projected through a content-hashed + `configMapGenerator`, so edits roll the pods. It has explicit `http_status:404` rules for + plex, plex-vip, jackett, pihole, pihole-dns, semaphore, qbittorrent, radarr, sonarr, headlamp, + unifi and crafty-mc. Then the apex and `*.${BASE_DOMAIN}` go to + `https://cilium-gateway-shared.gateway.svc.cluster.local:443` with `matchSNItoHost: true`. Last + comes a catch-all 404. The tunnel UUID is a **literal** in this file (`REPLACE-ME-TUNNEL-UUID` + until enrolment), not a cluster-var, so the Phase 2 PR can match it without touching `clusters/**`. + - `cloudflared-credentials.sops.yaml` is SOPS-encrypted and holds `REPLACE-ME` placeholders, with + a `.example` hand-off sibling. I encrypted the placeholder (not a credential) in the toolbox + with the root `.sops.yaml` recipient. + - `deployment.yaml` runs 2 replicas with a hard `topologySpreadConstraints` across nodes. The image + is `cloudflare/cloudflared:2026.10.7@sha256:4955524a…` (tag + digest, Renovate `kubernetes` + manager). It runs as non-root 65532, with a read-only root fs, all capabilities dropped and no + SA token. Requests are 10m/64Mi and limits 500m/256Mi. Readiness uses `/ready`. Liveness uses + `/healthcheck`, so an edge outage cannot restart-loop the pods. The pods are labelled + `egress-class: internet`. A PDB sets `minAvailable: 1`. + - `networkpolicy.yaml`: + - A namespace `default-deny` with DNS only. It has a **single owner**: the namespace is not + `app-namespace=true`, so Kyverno's generate rule never writes it. + - A Prometheus scrape allow on :2000. + - The egress CNP allows world tcp/udp 7844 and tcp 443, plus the `ingress` entity on tcp 443. + It also allows **Gateway backends by `io.cilium.k8s.namespace.labels.gateway-access=shared`, + minus the LAN-only namespaces.** + - `podmonitor.yaml` and `alerts.yaml` (`CloudflaredTunnelDown`, `CloudflaredReplicasDegraded`) + ship with `alerts.promtool-test.yaml`, and `README.md` documents the component. +- `clusters/base/cloudflared.yaml` (new) is its own Flux Kustomization: `dependsOn: platform-config`, + **`wait: false`**, prune, and postBuild `cluster-vars`. It is listed in `clusters/base/kustomization.yaml`. +- `clusters/{dev,prod}/kustomization.yaml` each gain a SOPS `decryption` patch for `cloudflared`. +- `conformance/tests/l0-static/cloudflared-ingress-rules/` (new, whole-repo scope) asserts several + things: + - Every rule targets either 404 or exactly the Gateway Service. The Service name is derived from + `platform/gateway/gateway.yaml`, and API markers are named explicitly. + - The excluded hosts 404 before the first Gateway rule, the Gateway rules use `matchSNItoHost`, + there is no `noTLSVerify`, and a catch-all closes the list. + - The CNP has no kube-apiserver, host, remote-node, cluster or `all` entity, no CIDR, and the full + NotIn list. The pods carry egress-class internet and have no SA token. + - The Flux tier is isolated: wait:false, not in `platform/kustomization.yaml`, nothing + `dependsOn: cloudflared`, and both overlays decrypt it. + - **Mutation self-test:** an added API rule, plex moved below the wildcard, and a dropped + catch-all must each be flagged. +- `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` gains an additive + `cloudflared_` scrape owner, with a networkpolicy companion, in the same shape as the kured entry. +- `runbooks/cloudflare-tunnel.md` (new) is registered in `runbooks/kustomization.yaml` and the + README index. + +**Design choices:** + +- **Isolation (the merge-before-credentials question).** The PR is safe to merge with placeholders. + The Deployment CrashLoops, but in its own `wait:false` Kustomization that nothing depends on. Its + Ready condition tracks the apply, not pod health, so neither `platform`, `platform-config` nor any + apps wave can stall. There is no Certificate or HelmRelease in the path for health checks to hang + on. Health is reported by the two alerts instead. `CloudflaredTunnelDown` **will fire from merge + until enrolment**. That is honest, and it is `warning` because nothing routes through the tunnel + yet. Raise it to critical in the DNS-flip change. +- **Egress to backends, not `{}`.** Under Cilium 1.20 (CVE-2025-30162), the Gateway's Envoy + re-checks the source pod's egress against the resolved backend. An egress rule that names only + the Gateway would therefore make every tunnelled request `403 Access denied` + (decisions/2026-10-30-gitea-admit-buildkit.md; catalog/render.py, `r_internet_egress`). The + gitea-actions `- {}` pattern would also work, but it admits kube-apiserver and host, which + decision 4 rules out. The namespace-label selector names every Gateway backend and nothing else. + **Backends need no ingress change.** Envoy reaches them as `reserved:ingress`, and the 2026-10-30 + measurement showed the identity check sits on the source's egress: gitea-actions is in none of + Gitea's ingress allows and still reaches it. So no app's `inClusterClients` was touched. +- **The alerts cannot self-silence.** Both expressions use `or on() vector(0)`, so absent series + (a broken scrape, zero pods, KSM gone) fire. The promtool tests pin the absent case and the + working-then-stale re-fire case. + +## What I verified + +- **The Gateway Service, measured live (read-only, `k3s-prod`, 2026-11-08).** It is + `gateway/cilium-gateway-shared`: type LoadBalancer, ClusterIP `10.43.225.79`, EXTERNAL-IP + `192.168.2.20`, ports 80/443, no selector, labelled `gateway.networking.k8s.io/gateway-name=shared`. + It belongs to Gateway `gateway/shared` (class `cilium`). +- **The namespace selector, measured live.** Cilium identities carry the label + `k8s:io.cilium.k8s.namespace.labels.gateway-access` (read from a gitea identity). Every namespace + with an HTTPRoute (crafty, gitea, headlamp, homepage, jackett, keycloak, monitoring, nextcloud, + oauth2-proxy, pihole, plex, qbittorrent, radarr, runbooks, seerr, semaphore, sonarr) is + `gateway-access=shared`. +- **Nothing exists yet.** Namespace `cloudflared` does not exist on prod. **cloudflared is not in + the pinned toolbox**, so the runbook runs the pinned image under podman instead. +- The image tag and digest came from Docker Hub's API: `2026.10.7` = `latest` = + `sha256:4955524a…c9b9c5ba`, published 2026-10-30. +- The credentials Secret is encrypted: `stringData.credentials.json` is `ENC[…]`, with the age + recipient from the root `.sops.yaml`. +- The new L0 check passes on the host (47 assertions, mutation self-test included). The toolbox + gate result is reported in the PR description. I ran it after committing. + +## What I did NOT verify + +- **UNVERIFIED: nothing ran in a cluster.** No pod has started, and no tunnel exists. I have not + observed placeholder CrashLoop behaviour itself. I expect cloudflared to treat + `REPLACE-ME-TUNNEL-UUID` as a tunnel name, find no origin cert, and exit, but whatever it does, + the pods are not Ready and the tier is non-gating. +- **UNVERIFIED: the CVE-2025-30162 path for cloudflared specifically.** The selector form was + checked against live identity labels, but no tunnelled request has been made. The runbook's + step 6 `kubectl debug` probe exists to prove "real backend, no 403" from the pod's own identity. +- **UNVERIFIED: whether the `ingress` entity rule's `toPorts: 443` restriction holds at Envoy.** + buildkit's working rule has no port restriction. If the canary shows drops, removing the port + restriction there is the first thing to try. +- The metric name `cloudflared_tunnel_ha_connections` and the `/ready` / `/healthcheck` endpoints + come from my knowledge of cloudflared, **not measured** against 2026.10.7. +- The Cloudflare edge egress is `world` + ports, not Cloudflare CIDRs. I did not narrow it blind. +- Egress `world:443` "for the API" follows the brief. I have not confirmed that a credentials-file + tunnel needs it at all. +- The Cloudflare-side settings (ACM, Total TLS / ordered cert, Bot Fight Mode, BIC/WAF, WebSockets) + are documented for the operator but not inspected. I have no Cloudflare access. +- The headlamp, qbittorrent, radarr and sonarr catalog exposures are deliberately unchanged + (Phase 2). diff --git a/platform/cloudflared/README.md b/platform/cloudflared/README.md new file mode 100644 index 0000000..e587b1c --- /dev/null +++ b/platform/cloudflared/README.md @@ -0,0 +1,22 @@ +# platform/cloudflared + +**Cloudflare Tunnel connector**: inbound HTTPS from the Cloudflare edge to the shared Gateway, +outbound-only from the cluster (no router port forward, no hairpin, no split-DNS dependency). + +- **Flux tier:** `clusters/base/cloudflared.yaml` — its own Kustomization, `dependsOn: [platform-config]`, + `wait: false`, and nothing depends on it. A missing or invalid credential can affect only the tunnel. +- **Runbook (enrolment, canary, alerts, rollback):** `runbooks/cloudflare-tunnel.md` +- **Design record:** `decisions/2026-11-08-cloudflare-tunnel.md` +- **Guard:** `conformance/tests/l0-static/cloudflared-ingress-rules` + +| File | What it is | +|---|---| +| `config.yaml` | The locally-managed tunnel: literal tunnel UUID + ingress rules. LAN/VPN-only hosts → 404, apex + wildcard → `https://cilium-gateway-shared.gateway.svc.cluster.local:443` with `matchSNItoHost`, catch-all 404. Never the Kubernetes API. | +| `cloudflared-credentials.sops.yaml` | SOPS-encrypted tunnel credentials JSON. Ships with `REPLACE-ME`; the operator fills it. | +| `deployment.yaml` | 2 replicas, hard spread across nodes, non-root, read-only root fs, pinned digest; PDB `minAvailable: 1`. | +| `networkpolicy.yaml` | default-deny (single owner — not an app namespace) + Prometheus scrape allow + egress CNP: world 7844 tcp/udp and 443, the Gateway (`ingress` entity), and Gateway backends by `gateway-access=shared` minus the LAN-only namespaces. | +| `podmonitor.yaml`, `alerts.yaml` | Scrape of :2000 and the two alerts, both written to fire (not go quiet) when their series vanish. | + +**Phase 1 only.** This component changes no live traffic: Cloudflare sends a hostname through the +tunnel only when its DNS record is a CNAME to `.cfargotunnel.com`, and nothing here creates +one. Moving DNS (external-dns, Gateway annotations, Pi-hole/CoreDNS split horizon) is a later change. diff --git a/platform/cloudflared/alerts.promtool-test.yaml b/platform/cloudflared/alerts.promtool-test.yaml new file mode 100644 index 0000000..d915763 --- /dev/null +++ b/platform/cloudflared/alerts.promtool-test.yaml @@ -0,0 +1,128 @@ +--- +# promtool unit tests for platform/cloudflared/alerts.yaml. +# +# Found by platform/monitoring/tests/run.sh through the COMPANION convention +# (`.promtool-test.yaml` beside `.yaml`). Run just this file with +# platform/monitoring/tests/run.sh platform/cloudflared/alerts.promtool-test.yaml +# The harness flattens the PrometheusRule's `.spec.groups` into alerts.rules.yaml. Assertions query +# the synthetic ALERTS series (not exp_alerts), so annotation wording is not pinned here. +# +# Each rule is pinned BOTH ways, plus the two self-silencing traps: +# * series ABSENT (nothing scraped) must FIRE, not read as healthy; +# * working-then-broken must RE-FIRE, not latch cleared. +rule_files: + - alerts.rules.yaml +evaluation_interval: 1m + +tests: + - interval: 1m + name: healthy - two replicas, four edge connections each, both rules quiet + input_series: + - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}' + values: '4x60' + - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-b"}' + values: '4x60' + - series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}' + values: '2x60' + promql_expr_test: + - expr: 'count(ALERTS{alertname=~"Cloudflared.*"})' + eval_time: 50m + exp_samples: [] + + - interval: 1m + name: nothing scraped at all - TunnelDown still fires (no self-silencing on absence) + input_series: + # kube-state-metrics says both pods are Available, but no cloudflared series exists — the + # broken-PodMonitor / dropped-scrape case. The rule must not read this as healthy. + - series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}' + values: '2x60' + promql_expr_test: + - expr: 'ALERTS{alertname="CloudflaredTunnelDown"}' + eval_time: 5m + exp_samples: + - labels: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="pending", severity="warning"}' + value: 1 + - expr: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing"}' + eval_time: 12m + exp_samples: + - labels: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing", severity="warning"}' + value: 1 + - expr: 'ALERTS{alertname="CloudflaredReplicasDegraded"}' + eval_time: 30m + exp_samples: [] + + - interval: 1m + name: scraped but zero edge connections - TunnelDown fires after 10m + input_series: + - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}' + values: '0x60' + - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-b"}' + values: '0x60' + - series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}' + values: '0x60' + promql_expr_test: + - expr: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing"}' + eval_time: 9m + exp_samples: [] + - expr: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing"}' + eval_time: 12m + exp_samples: + - labels: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing", severity="warning"}' + value: 1 + - expr: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing"}' + eval_time: 17m + exp_samples: + - labels: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing", severity="warning"}' + value: 1 + + - interval: 1m + name: working then the series vanish - TunnelDown re-fires instead of latching cleared + input_series: + # 20 healthy samples, then the series stop (stale) — the scrape broke or the pods died. + - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}' + values: '4x20 stale' + - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-b"}' + values: '4x20 stale' + - series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}' + values: '2x60' + promql_expr_test: + - expr: 'ALERTS{alertname="CloudflaredTunnelDown"}' + eval_time: 15m + exp_samples: [] + - expr: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing"}' + eval_time: 35m + exp_samples: + - labels: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing", severity="warning"}' + value: 1 + + - interval: 1m + name: one replica lost - ReplicasDegraded fires after 15m, tunnel itself still up + input_series: + - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}' + values: '4x60' + - series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}' + values: '1x60' + promql_expr_test: + - expr: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing"}' + eval_time: 14m + exp_samples: [] + - expr: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing"}' + eval_time: 17m + exp_samples: + - labels: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing", severity="warning"}' + value: 1 + - expr: 'ALERTS{alertname="CloudflaredTunnelDown"}' + eval_time: 30m + exp_samples: [] + + - interval: 1m + name: kube-state-metrics series absent - ReplicasDegraded fires rather than going quiet + input_series: + - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}' + values: '4x60' + promql_expr_test: + - expr: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing"}' + eval_time: 17m + exp_samples: + - labels: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing", severity="warning"}' + value: 1 diff --git a/platform/cloudflared/alerts.yaml b/platform/cloudflared/alerts.yaml new file mode 100644 index 0000000..1a79691 --- /dev/null +++ b/platform/cloudflared/alerts.yaml @@ -0,0 +1,64 @@ +--- +# Alert rules for the Cloudflare Tunnel connector. +# +# THE DESIGN CONSTRAINT: NEITHER RULE MAY GO QUIET WHEN ITS OWN DATA DISAPPEARS. The repo has +# shipped self-silencing rules before (a bare `up{}` that clears exactly when scraping breaks), so: +# +# * CloudflaredTunnelDown reads `cloudflared_tunnel_ha_connections` — the per-replica gauge of +# live edge connections — and falls back to `vector(0)` when NO such series exists. A dead +# tunnel, zero pods, a broken PodMonitor, or a NetworkPolicy that drops the scrape all make the +# series absent, and all of them FIRE rather than read as healthy. +# * CloudflaredReplicasDegraded reads kube-state-metrics, which does not depend on cloudflared +# being scrapable at all, and has the same `vector(0)` fallback. +# +# Both directions are pinned by the companion alerts.promtool-test.yaml (healthy is quiet; absent +# series fires; working-then-broken re-fires). +# +# NOTE FOR PHASE 1: until the operator completes runbooks/cloudflare-tunnel.md steps 1-5 the +# Deployment cannot connect and CloudflaredTunnelDown fires. That is the honest state — the tunnel +# is not serving — and nothing routes through it yet, which is why both rules are `warning`. +# Raise TunnelDown to `critical` in the change that points public DNS at the tunnel. +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: cloudflared + namespace: cloudflared + labels: + app.kubernetes.io/name: cloudflared +spec: + groups: + - name: platform-health.cloudflare-tunnel + rules: + - alert: CloudflaredTunnelDown + # Sum of live edge connections across every replica. `or on() vector(0)` turns an EMPTY + # result (nothing scraped) into 0 so it fires instead of evaluating to no series. + expr: (sum(cloudflared_tunnel_ha_connections{namespace="cloudflared"}) or on() vector(0)) < 1 + # 10m rides out a rolling update and a Cloudflare edge reconnect, not a real outage. + for: 10m + labels: + severity: warning + annotations: + summary: Cloudflare Tunnel has no healthy edge connections + description: >- + No cloudflared replica reports a live connection to the Cloudflare edge (or none is + being scraped at all). Every hostname routed through the tunnel is unreachable from + the internet. Before the operator has enrolled the tunnel this is expected. + runbook_url: "https://${BASE_DOMAIN}/runbooks/cloudflare-tunnel" + + - alert: CloudflaredReplicasDegraded + # Fewer than the 2 replicas deployment.yaml asks for are Available — the tunnel has lost + # its HA, and the next node drain or reboot takes it down. Keep `2` in step with + # spec.replicas. Absent series (kube-state-metrics gone, Deployment deleted) -> 0 -> fires. + expr: >- + (max(kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}) + or on() vector(0)) < 2 + for: 15m + labels: + severity: warning + annotations: + summary: cloudflared has fewer than 2 available replicas + description: >- + The cloudflared Deployment has fewer than two Available replicas, so the tunnel has + no redundancy (or is down entirely). Check for unenrolled credentials, a node that + cannot schedule the hard-spread pod, or an image pull failure. + runbook_url: "https://${BASE_DOMAIN}/runbooks/cloudflare-tunnel" diff --git a/platform/cloudflared/cloudflared-credentials.sops.yaml b/platform/cloudflared/cloudflared-credentials.sops.yaml new file mode 100644 index 0000000..575ca97 --- /dev/null +++ b/platform/cloudflared/cloudflared-credentials.sops.yaml @@ -0,0 +1,32 @@ +# Cloudflare Tunnel credentials for platform/cloudflared — SOPS-ENCRYPTED (R-SEC-01, spec docs/07). +# Keys stay cleartext for reviewable diffs; only `stringData` values are encrypted (.sops.yaml +# `encrypted_regex`). Plaintext template + operator hand-off: cloudflared-credentials.sops.yaml.example. +# Shipped with REPLACE-ME placeholders; runbooks/cloudflare-tunnel.md step 4 fills it. +apiVersion: v1 +kind: Secret +metadata: + name: cloudflared-credentials + namespace: cloudflared +type: Opaque +stringData: + credentials.json: ENC[AES256_GCM,data:PM5pgIW/JEUMLuhApkPhgMOkJCL0nrlgLBb3tG+pHdm8JWdNK2yYCZUWM3USY7Uy2IYphZh35RwPOrWSUCyBlSq+TeEKBkzTTgJhp4gqaQ==,iv:Sr0Z6d9LmHvGN5oebIf+YBbFyLIdNOi80d6WwOF3PtU=,tag:Enb5/9v7H1nu2kvh4zEjXg==,type:str] +sops: + kms: [] + gcp_kms: [] + azure_kv: [] + hc_vault: [] + age: + - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg + enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUanIxelFEUDZtWU5CMHd5 + VHl5MkJnM1JDUWhkM2daY0dtRUlFUmNUMDJrClM5UFBsdjh2Q0NIWmxpM1djYXUy + OFlZYVdBbURLanRRNzV1RVNmbkxWR28KLS0tIFE3d1Z1MjFZMTlaTFFIWm5ndVpN + emhHaFpEZWpMN0JXL2NVaW5PU1ZwVW8KkCDX7ED039X12BCmiSqHr0fmZKSZmfjP + DIiR6DvWWw1TcB3CjD5q29nIthCDf9zlrpuMfQ+9ODAlRr5t6zP+cg== + -----END AGE ENCRYPTED FILE----- + lastmodified: "2026-11-08T18:28:24Z" + mac: ENC[AES256_GCM,data:RGiftdP4Bd5Qwyfx/pXEyZTBzTvr5Tvc7k0foEzTQGiSg6p0meurwtmCe/gMjKUZLhwf/0xqi1DtPQGS6UY8m8YFq1v1nXtohUQWLPXgTukMT915tHmohZeG91wYdyG9kA5E03udn+QNdOiI8OtjKoqAtifBoymCYd/EDOzWLQU=,iv:0vDIoDJ4QGECq7YjUPIlsU1j/AHfr1W32/+RjEOfpWo=,tag:BWT7nxiUxkVhEDh0JiE1Eg==,type:str] + pgp: [] + encrypted_regex: ^(data|stringData)$ + version: 3.9.4 diff --git a/platform/cloudflared/cloudflared-credentials.sops.yaml.example b/platform/cloudflared/cloudflared-credentials.sops.yaml.example new file mode 100644 index 0000000..bee5329 --- /dev/null +++ b/platform/cloudflared/cloudflared-credentials.sops.yaml.example @@ -0,0 +1,34 @@ +# ============================================================================= +# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never +# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF. +# +# WHAT THIS IS: the credentials JSON of the locally-managed Cloudflare Tunnel that +# platform/cloudflared runs. `cloudflared tunnel create` writes it; it holds the account tag, the +# tunnel UUID and the tunnel SECRET. Anyone holding it can run a connector for this tunnel and +# receive its traffic, so it is a real credential: SOPS-encrypted in git, never printed. +# +# *** THE REAL FILE IS COMMITTED WITH REPLACE-ME PLACEHOLDERS, ALREADY ENCRYPTED. *** +# So the wiring is complete before the tunnel exists: the resource is listed in kustomization.yaml +# and both overlays already carry the `cloudflared` decryption patch. Until you fill it, the +# Deployment CrashLoops inside its own NON-GATING Flux Kustomization (clusters/base/cloudflared.yaml, +# wait:false) and CloudflaredTunnelDown fires. Nothing else in the cluster is affected. +# +# HAND-OFF (operator — an agent must not mint a long-lived credential, CLAUDE.md §6). +# The full procedure, with verification and rollback, is runbooks/cloudflare-tunnel.md. In short: +# 1. `cloudflared tunnel login` + `cloudflared tunnel create --credentials-file ... home-k3s` +# (pinned cloudflared image under podman; the toolbox does not ship cloudflared). +# 2. Write the JSON STRAIGHT INTO the encrypted file with `sops set`, inside the toolbox — the +# plaintext never reaches a terminal or the repo tree. +# 3. Put the tunnel UUID (not secret) into platform/cloudflared/config.yaml `tunnel:`. +# 4. Verify without printing (`sops -d --extract ... | jq -e ...`), then PR it. +# ============================================================================= +apiVersion: v1 +kind: Secret +metadata: + name: cloudflared-credentials + namespace: cloudflared +type: Opaque +stringData: + # One line of JSON, exactly as `cloudflared tunnel create` wrote it (compacted with `jq -c`). + # TunnelID MUST equal the `tunnel:` UUID in config.yaml. + credentials.json: '{"AccountTag":"REPLACE-ME","TunnelSecret":"REPLACE-ME","TunnelID":"REPLACE-ME"}' diff --git a/platform/cloudflared/config.yaml b/platform/cloudflared/config.yaml new file mode 100644 index 0000000..62cbbfd --- /dev/null +++ b/platform/cloudflared/config.yaml @@ -0,0 +1,84 @@ +--- +# cloudflared configuration for the LOCALLY-MANAGED tunnel (the ingress rules live here, in git, +# not in the Cloudflare dashboard). Projected into the pods by the configMapGenerator in +# kustomization.yaml as the content-hashed ConfigMap `cloudflared-config`, so editing this file +# changes the pod template and ROLLS the Deployment (the vpn-ingress peers.conf lesson: a +# fixed-name ConfigMap updates in place and the running process never re-reads it). +# +# `${BASE_DOMAIN}` is substituted by Flux postBuild (clusters/base/cloudflared.yaml). It is the ONLY +# token in this file; do not add a bare dollar sign anywhere here. +# +# INVARIANTS, enforced by conformance/tests/l0-static/cloudflared-ingress-rules: +# * every rule's `service` is either `http_status:404` or EXACTLY the shared Gateway's in-cluster +# Service — nothing else, and in particular NEVER the Kubernetes API (operator decision +# 2026-11-08: the API stays local-only and must never be reachable through the tunnel); +# * the LAN/VPN-only hostnames below map to `http_status:404` and come BEFORE the wildcard; +# * `tunnel:` is a LITERAL UUID here, not a cluster-var: the Phase 2 DNS-flip change matches +# records against it, and keeping it out of clusters/** keeps that change off a sensitive path; +# * the last rule is a hostname-less `http_status:404` catch-all. +# +# OPERATOR STEP: replace REPLACE-ME-TUNNEL-UUID with the UUID printed by `cloudflared tunnel create` +# (runbooks/cloudflare-tunnel.md step 3). Until then cloudflared treats the value as a tunnel NAME, +# finds no origin certificate to resolve it with, and exits — a CrashLoop in a non-gating tier. +tunnel: REPLACE-ME-TUNNEL-UUID +credentials-file: /etc/cloudflared/creds/credentials.json +# Serves /metrics (Prometheus), /ready (200 only with >=1 live edge connection) and /healthcheck. +metrics: 0.0.0.0:2000 +no-autoupdate: true + +ingress: + # ── NEVER THROUGH THE TUNNEL (operator decisions 2026-11-08) ───────────────────────────────── + # Plex stays off the tunnel; it keeps its own remote-access path (runbooks/plex-remote-access.md). + - hostname: plex.${BASE_DOMAIN} + service: http_status:404 + - hostname: plex-vip.${BASE_DOMAIN} + service: http_status:404 + # LAN/VPN-only services. The six catalog `exposure: internal` apps (each carries the external-dns + # `controller: none` route annotation live, so none has a public record today) ... + - hostname: jackett.${BASE_DOMAIN} + service: http_status:404 + - hostname: pihole.${BASE_DOMAIN} + service: http_status:404 + - hostname: pihole-dns.${BASE_DOMAIN} + service: http_status:404 + - hostname: semaphore.${BASE_DOMAIN} + service: http_status:404 + - hostname: qbittorrent.${BASE_DOMAIN} + service: http_status:404 + - hostname: radarr.${BASE_DOMAIN} + service: http_status:404 + - hostname: sonarr.${BASE_DOMAIN} + service: http_status:404 + # ... and Headlamp, the cluster dashboard (operator decision 2026-11-08: LAN/VPN-only). It holds a + # route to the Kubernetes API on the user's behalf, which is one more reason it stays off the + # tunnel alongside decision 4. + - hostname: headlamp.${BASE_DOMAIN} + service: http_status:404 + # Non-HTTP L4 VIPs (their own LoadBalancer addresses, not the Gateway). Listed so a DNS flip + # that CNAMEs them at the tunnel by mistake gets a 404, not the Gateway's catch-all. + - hostname: unifi.${BASE_DOMAIN} + service: http_status:404 + - hostname: crafty-mc.${BASE_DOMAIN} + service: http_status:404 + + # ── EVERYTHING ELSE ON THE ZONE -> THE SHARED GATEWAY, OVER TLS ────────────────────────────── + # cilium-gateway-shared is the LoadBalancer Service Cilium creates for Gateway gateway/shared + # (platform/gateway/gateway.yaml); its ClusterIP frontend is redirected to the Gateway's Envoy + # exactly like the VIP. Addressed by cluster DNS, not by a ${BASE_DOMAIN} name, so the tunnel + # does not depend on split-horizon DNS at all. + # + # matchSNItoHost: cloudflared sends the request's Host as the TLS SNI, so the Gateway selects the + # right listener and presents the existing Let's Encrypt `wildcard-home-tls` certificate, which + # cloudflared validates against the public roots for that same name. No TLS verification is + # disabled anywhere. + - hostname: ${BASE_DOMAIN} + service: https://cilium-gateway-shared.gateway.svc.cluster.local:443 + originRequest: + matchSNItoHost: true + - hostname: "*.${BASE_DOMAIN}" + service: https://cilium-gateway-shared.gateway.svc.cluster.local:443 + originRequest: + matchSNItoHost: true + + # ── CATCH-ALL (cloudflared requires the last rule to have no hostname) ─────────────────────── + - service: http_status:404 diff --git a/platform/cloudflared/deployment.yaml b/platform/cloudflared/deployment.yaml new file mode 100644 index 0000000..c6e0fa4 --- /dev/null +++ b/platform/cloudflared/deployment.yaml @@ -0,0 +1,140 @@ +--- +# cloudflared — the Cloudflare Tunnel connector. Outbound-only: it dials the Cloudflare edge on +# 7844 (QUIC, falling back to HTTP/2 over TCP) and carries requests for the tunnel's hostnames +# back to the shared Gateway. Nothing listens on the LAN and no router port is opened. +# +# HA: 2 replicas, hard-spread across nodes. Each replica registers its own set of edge +# connections for the SAME tunnel, so Cloudflare load-balances between them and losing one node +# (a kured reboot, a drain) loses no traffic. The PDB in this directory keeps a drain from taking +# both at once. +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cloudflared + namespace: cloudflared + labels: + app.kubernetes.io/name: cloudflared +spec: + replicas: 2 + revisionHistoryLimit: 3 + strategy: + type: RollingUpdate + rollingUpdate: + # Never drop below the running count during a rollout; with 3 nodes and a hard spread the + # surge pod always has a node to land on. + maxUnavailable: 0 + maxSurge: 1 + selector: + matchLabels: + app.kubernetes.io/name: cloudflared + template: + metadata: + labels: + app.kubernetes.io/name: cloudflared + # internet, NEVER vpn. The vpn class steers egress through the PIA tunnel (ADR-0015), which + # would put the edge connections behind a third party's exit and its kill-switch — the + # tunnel would drop whenever PIA does. This namespace is outside the app-namespace scope + # of require-egress-class-label, so the label is declared for the steering, not to pass + # admission; the L0 check cloudflared-ingress-rules pins it. + homelab.keithjackson.dev/egress-class: internet + spec: + # cloudflared never talks to the Kubernetes API. No token mounted, and the egress policy + # does not admit the kube-apiserver entity either. + automountServiceAccountToken: false + enableServiceLinks: false + securityContext: + runAsNonRoot: true + # The image is distroless `nonroot` (65532). + runAsUser: 65532 + runAsGroup: 65532 + seccompProfile: + type: RuntimeDefault + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: kubernetes.io/hostname + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + app.kubernetes.io/name: cloudflared + # cloudflared drains in-flight requests for up to its own 30s grace-period on SIGTERM. + terminationGracePeriodSeconds: 45 + containers: + - name: cloudflared + # Tag + digest: Renovate's `kubernetes` manager (renovate.json5, platform/**) bumps both. + image: docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba + # The image ENTRYPOINT is `cloudflared --no-autoupdate`; these replace its CMD. The tunnel + # UUID, credentials path, metrics address and ingress rules all come from the file. + args: + - tunnel + - --config + - /etc/cloudflared/config/config.yaml + - run + ports: + - name: metrics + containerPort: 2000 + protocol: TCP + # /ready is 200 only while this replica holds at least one live edge connection, so an + # unenrolled or edge-isolated replica is NotReady and the Deployment says so. + readinessProbe: + httpGet: + path: /ready + port: metrics + initialDelaySeconds: 5 + periodSeconds: 10 + failureThreshold: 3 + # Liveness is deliberately NOT /ready: an edge outage would otherwise restart-loop every + # replica, which reconnects nothing faster and churns the pods. /healthcheck answers + # whenever the process is serving. + livenessProbe: + httpGet: + path: /healthcheck + port: metrics + initialDelaySeconds: 10 + periodSeconds: 20 + failureThreshold: 3 + resources: + requests: + cpu: 10m + memory: 64Mi + limits: + cpu: 500m + memory: 256Mi + securityContext: + allowPrivilegeEscalation: false + readOnlyRootFilesystem: true + capabilities: + drop: + - ALL + volumeMounts: + - name: config + mountPath: /etc/cloudflared/config + readOnly: true + - name: creds + mountPath: /etc/cloudflared/creds + readOnly: true + volumes: + - name: config + configMap: + # Rewritten to the content-hashed name by kustomize (configMapGenerator). + name: cloudflared-config + - name: creds + secret: + # cloudflared-credentials.sops.yaml — the tunnel credentials JSON, by name only. + secretName: cloudflared-credentials + defaultMode: 0440 +--- +# One replica may be disrupted at a time — enough for kured's one-node-at-a-time drain to +# proceed, never enough to take the tunnel down. minAvailable 1 (not 2) on purpose: a 0-disruption +# PDB wedges a drain (platform/kured/alerts.yaml, KuredRebootPending). +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: cloudflared + namespace: cloudflared + labels: + app.kubernetes.io/name: cloudflared +spec: + minAvailable: 1 + selector: + matchLabels: + app.kubernetes.io/name: cloudflared diff --git a/platform/cloudflared/kustomization.yaml b/platform/cloudflared/kustomization.yaml new file mode 100644 index 0000000..b1d910b --- /dev/null +++ b/platform/cloudflared/kustomization.yaml @@ -0,0 +1,35 @@ +--- +# Cloudflare Tunnel connector. Reconciled by its OWN non-gating Flux Kustomization +# (clusters/base/cloudflared.yaml: dependsOn platform-config, wait:false) — NOT a line in +# platform/kustomization.yaml. See that file for why; in one line: this Deployment cannot become +# Ready until the operator enrols the tunnel, and in a gating tier that would stall every app wave. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +# REQUIRED for the configMapGenerator: a generated ConfigMap inherits no namespace, and kustomize +# will not rewrite the Deployment's reference to its hashed name across a namespace boundary +# (measured on platform/vpn-ingress). +namespace: cloudflared +resources: + - namespace.yaml + - deployment.yaml # Deployment (2 replicas, hard node spread) + PodDisruptionBudget + # Namespace default-deny + the Prometheus scrape allow + the egress CiliumNetworkPolicy. Keep + # them in ONE file: a baseline without its scrape allow silently empties the TSDB. + - networkpolicy.yaml + - podmonitor.yaml + - alerts.yaml + # The tunnel credentials JSON — the ONE secret here. SOPS-encrypted, decrypted per-env by the + # `cloudflared` Flux Kustomization (both overlays patch it), referenced by name only. Its + # plaintext template sibling is deliberately NOT named in this file (no-plaintext-secrets). + # Ships with REPLACE-ME placeholders; runbooks/cloudflare-tunnel.md fills it. + - cloudflared-credentials.sops.yaml + +# config.yaml becomes the CONTENT-HASHED ConfigMap `cloudflared-config`, so an edit to the ingress +# rules or the tunnel UUID changes the pod template and rolls the Deployment. Do NOT set +# disableNameSuffixHash: the hash is the rollout mechanism. +configMapGenerator: + - name: cloudflared-config + files: + - config.yaml + options: + labels: + app.kubernetes.io/name: cloudflared diff --git a/platform/cloudflared/namespace.yaml b/platform/cloudflared/namespace.yaml new file mode 100644 index 0000000..542535e --- /dev/null +++ b/platform/cloudflared/namespace.yaml @@ -0,0 +1,18 @@ +--- +# Cloudflare Tunnel connector (north-south INGRESS from the Cloudflare edge). See README.md. +# +# NOT homelab.keithjackson.dev/app-namespace=true. This is platform network infrastructure (like +# vpn-ingress / vpn-egress), not an application, so Kyverno's app-scoped policies — including the +# `require-default-deny-baseline` GENERATE rule — do not select this namespace. That is why +# networkpolicy.yaml ships its own `default-deny` here, and why doing so creates NO second owner +# of that object: Kyverno never generates one in a namespace without the app-namespace label +# (the drift loop recorded in the 2026-09-30 handoff only exists where both write the same name). +# +# NO `gateway-access` label. Nothing here attaches an HTTPRoute to the shared Gateway; cloudflared +# is a CLIENT of the Gateway, not a backend behind it. +apiVersion: v1 +kind: Namespace +metadata: + name: cloudflared + labels: + app.kubernetes.io/name: cloudflared diff --git a/platform/cloudflared/networkpolicy.yaml b/platform/cloudflared/networkpolicy.yaml new file mode 100644 index 0000000..330182a --- /dev/null +++ b/platform/cloudflared/networkpolicy.yaml @@ -0,0 +1,149 @@ +--- +# Namespace baseline: deny everything both ways, except DNS to kube-system. +# +# SINGLE OWNER. This namespace is not labelled app-namespace=true, so Kyverno's +# require-default-deny-baseline generate rule does not select it and never writes this object; +# Flux is its only writer (the same posture as kured / crafty / headlamp / homepage). Do NOT add +# the app-namespace label to namespace.yaml without deleting this object in the same change, or +# the two controllers fight over it every reconcile (the 2026-09-30 drift-loop root cause). +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-deny + namespace: cloudflared + labels: + homelab.keithjackson.dev/baseline: default-deny +spec: + podSelector: {} + policyTypes: + - Ingress + - Egress + ingress: [] + egress: + - to: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: kube-system + ports: + - protocol: UDP + port: 53 + - protocol: TCP + port: 53 +--- +# The ONLY ingress: Prometheus scraping :2000. Shipped in the same file as the baseline on purpose +# (the Longhorn/kured lesson): a deny-all without this allow leaves the PodMonitor's targets +# DISCOVERED and every scrape timing out, which is the state the CloudflaredTunnelDown rule is +# written to fire on rather than go quiet in. +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: cloudflared-allow-prometheus-scrape + namespace: cloudflared + labels: + app.kubernetes.io/name: cloudflared +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: cloudflared + policyTypes: + - Ingress + ingress: + - from: + - namespaceSelector: + matchLabels: + kubernetes.io/metadata.name: monitoring + ports: + - protocol: TCP + port: 2000 +--- +# Egress: the Cloudflare edge, the shared Gateway, and the Gateway's backends — nothing else. +# +# 1. world tcp+udp/7844 — the tunnel itself (QUIC on udp, HTTP/2 fallback on tcp). +# world tcp/443 — Cloudflare's API / update endpoints that cloudflared calls at start. +# Expressed as `world` + port rather than Cloudflare CIDRs: the edge ranges are Cloudflare's to +# change, and a stale CIDR list would fail as a silent outage. Narrowing to toCIDR is a +# possible later hardening; it is not done blind. +# +# 2. `ingress` entity tcp/443 — the shared Gateway's Envoy. Traffic to the +# cilium-gateway-shared Service is handed to Envoy, whose identity is reserved:ingress. +# +# 3. The Gateway's BACKENDS, by namespace. NOT optional: Cilium >= 1.16.8 (CVE-2025-30162; this +# cluster runs 1.20.x) makes the Gateway's Envoy re-evaluate THIS pod's egress policy against +# the identity of the backend it routes the request to. With only rule 2, Envoy accepts the TCP +# connection and then answers every request `403 Access denied` — measured for buildkit -> +# gitea on 2026-10-30, see platform/buildkit/networkpolicy.yaml. So the backends are named: +# * any namespace opted in to the shared Gateway (gateway-access=shared) — which is the set +# of namespaces that can serve a route at all, so this tracks new apps automatically; +# * MINUS the LAN/VPN-only services (operator decisions 2026-11-08): plex, jackett, pihole, +# semaphore, qbittorrent, radarr, sonarr, headlamp. config.yaml already 404s their +# hostnames; this is the second, independent layer: even a request that got past the +# ingress rules would be refused by Envoy for these backends. +# unifi and crafty-mc are not here because they are L4 LoadBalancer VIPs, not Gateway +# backends; the crafty WEB UI (crafty., an HTTPRoute) is deliberately reachable. +# +# WHY NOT the `- {}` allow-all egress gitea-actions uses (apps/wave-1/gitea-actions.yaml)? +# It would satisfy the L7 re-check just as well, but `{}` admits EVERY identity, including +# kube-apiserver and host — which operator decision 4 (the Kubernetes API is never reachable +# through the tunnel) rules out for the one pod that carries internet traffic in. The +# namespace-label selector is the narrowest rule that still names every possible backend. +# Verified 2026-11-08: Cilium identities carry `k8s:io.cilium.k8s.namespace.labels. +# gateway-access`, and every namespace with an HTTPRoute on the shared Gateway is labelled +# gateway-access=shared. +# +# BACKENDS NEED NO INGRESS CHANGE. Envoy forwards to a backend as reserved:ingress, which every +# backend already admits; the source-identity check happens on THIS pod's egress, not on the +# destination's ingress. Measured 2026-10-30 (decisions/2026-10-30-gitea-admit-buildkit.md): +# gitea-actions is in none of Gitea's ingress allows and its requests through the Gateway reach +# Gitea, while Gitea's endpoint shows only a `reserved:ingress` allow row. So, unlike a +# ClusterIP client, cloudflared is not added to any app's inClusterClients. That also means +# this rule does not open a direct cloudflared -> app pod path into any app namespace whose +# ingress admits only reserved:ingress and named peers. +# +# DELIBERATELY ABSENT: kube-apiserver, host, remote-node, cluster, all. The Kubernetes API stays +# local-only (operator decision 2026-11-08); conformance/tests/l0-static/cloudflared-ingress-rules +# fails the gate if any of those entities appears here. +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: cloudflared-allow-egress + namespace: cloudflared + labels: + app.kubernetes.io/name: cloudflared +spec: + endpointSelector: + matchLabels: + app.kubernetes.io/name: cloudflared + egress: + - toEntities: + - world + toPorts: + - ports: + - port: "7844" + protocol: TCP + - port: "7844" + protocol: UDP + - port: "443" + protocol: TCP + - toEntities: + - ingress + toPorts: + - ports: + - port: "443" + protocol: TCP + - toEndpoints: + - matchExpressions: + - key: io.cilium.k8s.namespace.labels.gateway-access + operator: In + values: + - shared + - key: io.kubernetes.pod.namespace + operator: NotIn + values: + - plex + - jackett + - pihole + - semaphore + - qbittorrent + - radarr + - sonarr + - headlamp diff --git a/platform/cloudflared/podmonitor.yaml b/platform/cloudflared/podmonitor.yaml new file mode 100644 index 0000000..082d7e9 --- /dev/null +++ b/platform/cloudflared/podmonitor.yaml @@ -0,0 +1,22 @@ +--- +# Scrapes cloudflared's own metrics server (:2000, `metrics:` in config.yaml) on every replica. +# kube-prometheus-stack runs with podMonitorSelectorNilUsesHelmValues: false, so Prometheus selects +# every PodMonitor in every namespace; the `release` label is kept for consistency with the +# platform's other monitors. The ingress allow that lets this scrape through the namespace's +# default-deny is cloudflared-allow-prometheus-scrape in networkpolicy.yaml. +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: cloudflared + namespace: cloudflared + labels: + app.kubernetes.io/name: cloudflared + release: kube-prometheus-stack +spec: + selector: + matchLabels: + app.kubernetes.io/name: cloudflared + podMetricsEndpoints: + - port: metrics + path: /metrics + interval: 30s diff --git a/runbooks/README.md b/runbooks/README.md index 070f485..b29a80b 100644 --- a/runbooks/README.md +++ b/runbooks/README.md @@ -123,6 +123,9 @@ Each affected runbook says so on its own page and gives the by-hand equivalent. - **`vpn-remote-access`** — the INBOUND remote-access VPN (WireGuard, udp/51820): standing it up, enrolling and revoking devices, and the off-LAN checks that are the only honest proof it works. The opposite direction to the two pages above — do not confuse them. +- **`cloudflare-tunnel`** — the Cloudflare Tunnel connector (inbound HTTPS from the Cloudflare edge to + the shared Gateway): enrolling it, the canary, `CloudflaredTunnelDown` / `CloudflaredReplicasDegraded`, + and rollback. Never `cloudflared tunnel route dns` — external-dns owns the records. ### Backups and DR - **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here. diff --git a/runbooks/cloudflare-tunnel.md b/runbooks/cloudflare-tunnel.md new file mode 100644 index 0000000..eabbaf8 --- /dev/null +++ b/runbooks/cloudflare-tunnel.md @@ -0,0 +1,288 @@ +# cloudflare-tunnel — enrolling the Cloudflare Tunnel, proving it, and the two alerts + +Alerts that link here: **`CloudflaredTunnelDown`** (no replica has a live edge connection, or none +is being scraped) and **`CloudflaredReplicasDegraded`** (fewer than 2 replicas Available). + +Component: `platform/cloudflared/` (Flux Kustomization `cloudflared`, `clusters/base/cloudflared.yaml`). +Design record: `decisions/2026-11-08-cloudflare-tunnel.md`. + +Every step below is an **operator** step. An agent must not mint the tunnel credential, run +`flux reconcile`, or mutate the cluster (CLAUDE.md §5, §6). + +## What this means here + +cloudflared is an **outbound-only** connector: two replicas dial the Cloudflare edge on 7844 +(QUIC/udp, falling back to HTTP/2 over tcp) and carry requests back to the shared Gateway +(`https://cilium-gateway-shared.gateway.svc.cluster.local:443`, SNI = the request's Host, so the +Gateway's existing `wildcard-home-tls` Let's Encrypt certificate is what cloudflared validates). +The tunnel is **locally managed**: its ingress rules are `platform/cloudflared/config.yaml`, in git, +not in the Cloudflare dashboard. + +The operator decisions it encodes (2026-11-08): + +1. Public hostnames stay `*.home.keithjackson.dev` and the apex; edge certificates come from + Cloudflare **ACM + Total TLS**. +2. **Plex is not on the tunnel** (`plex`, `plex-vip` → 404). +3. **jackett, pihole, semaphore, qbittorrent, radarr, sonarr** and **headlamp** stay LAN/VPN-only, + as do the L4 VIPs **unifi** and **crafty-mc** (all → 404 in cloudflared, and the LAN-only + Gateway backends are also refused by the egress policy). Headlamp is an operator decision of + 2026-11-08; the other six are the catalog's `exposure: internal` services. +4. **The Kubernetes API is never reachable through the tunnel.** No rule targets it and the egress + policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check + `cloudflared-ingress-rules` fails the gate if that ever changes. + +**Nothing goes through the tunnel until DNS points at it.** Cloudflare routes a hostname through a +tunnel only when its record is a CNAME to `.cfargotunnel.com`. Phase 1 (this component) +creates no such record; moving public DNS is a separate, later change. So in Phase 1 the alerts +firing means "the tunnel is not ready", not "users are down". + +Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and +`CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization +is `wait: false` and nothing depends on it. + +## First commands + +``` +# Are the pods running, and on different nodes? +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n cloudflared get deploy,pods -o wide' + +# What does cloudflared say? Healthy: four "Registered tunnel connection" lines per replica. +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n cloudflared logs deploy/cloudflared --tail=50' + +# Did Flux apply the current revision, and is the Secret decrypted (key NAMES only)? +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n flux-system get kustomization cloudflared; + kubectl --context k3s-prod -n cloudflared get secret cloudflared-credentials \ + -o go-template="{{range \$k, \$v := .data}}{{\$k}}{{\"\n\"}}{{end}}"' +``` + +Never print the Secret's values. `ENC[` in the decrypted Secret means the overlay's decryption +patch is missing; see causes below. + +## Likely causes, most likely first + +1. **The tunnel has not been enrolled yet** (Phase 1, steps 1-5 below not done). Logs show an + error about the tunnel `REPLACE-ME-TUNNEL-UUID` or a missing origin certificate. Do steps 1-5. +2. **UUID and credentials disagree** — `config.yaml` `tunnel:` is not the `TunnelID` in the + credentials JSON. Logs: authentication / "tunnel not found". Check with step 4's verify command. +3. **Credentials still encrypted in the cluster** — the `cloudflared` decryption patch is missing + from `clusters/prod/kustomization.yaml`. The L0 check guards this, so this means someone bypassed + it. +4. **Egress blocked** — a policy change, or the edge moved off 7844. Look for drops: + ``` + infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \ + hubble observe --verdict DROPPED --from-namespace cloudflared --last 100 -o compact' + ``` + (`ds/cilium` picks one node's agent; repeat per node if empty.) +5. **The tunnel was deleted or its secret rotated in Cloudflare** — `cloudflared tunnel list` + (step 2's podman form) no longer shows it. Re-enrol with steps 2-5. +6. **`CloudflaredReplicasDegraded` only**: one replica cannot schedule (the spread is hard, one per + node — a cordoned node during a kured reboot is normal and clears) or an image pull failed. + +## When to escalate or stop + +- **Stop** if anything suggests a rule reaching the Kubernetes API or a LAN-only service. Do not + hot-fix through the dashboard: the tunnel is locally managed, and a dashboard edit is silently + overridden by `config.yaml`. Use the emergency stop under *Rollback*, then fix in git. +- **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one. + **external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`); + a record created behind its back either fights it or is deleted by it, and a real hostname + routed that way is an unreviewed DNS flip. Moving records onto the tunnel is the separate Phase 2 + change. The only record this runbook creates is the hand-made canary in step 6, and you delete it. +- Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is + Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2. + +--- + +## Enrolling the tunnel (once) + +cloudflared is **not** in the pinned toolbox (checked 2026-11-08). Run it from the same pinned image +the Deployment uses, under podman — no host install. The working directory is +`~/.config/sops/cloudflared`: it is private to you, and it is the one host directory `dev.sh` +already mounts into the toolbox (as `/root/.config/sops`), so the toolbox's `sops` can read the +credentials JSON without it ever entering the repository tree. + +``` +CF_IMG='docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba' +CF_DIR="$HOME/.config/sops/cloudflared" +mkdir -p "$CF_DIR" && chmod 700 "$CF_DIR" +cf() { podman run --rm -it --user 0 --security-opt label=disable \ + -e HOME=/root -v "$CF_DIR:/root/.cloudflared" "$CF_IMG" "$@"; } +``` + +(Keep `CF_IMG` in step with `platform/cloudflared/deployment.yaml` if Renovate has bumped it.) + +### Step 1 — edge certificate (ACM) and edge settings (Cloudflare dashboard) + +Universal SSL covers only `keithjackson.dev` and `*.keithjackson.dev`; it does **not** cover the +second-level `*.home.keithjackson.dev`. In the `keithjackson.dev` zone: + +1. **SSL/TLS → Edge Certificates → Advanced Certificate Manager**: subscribe. +2. Then EITHER **Order an advanced certificate** for exactly `home.keithjackson.dev, + *.home.keithjackson.dev` (covers every name at once, independent of proxy state), OR turn on + **Total TLS** (issues one certificate per *proxied* hostname as each one appears). The ordered + certificate is the more predictable of the two for the later DNS flip; either satisfies the + operator decision. +3. **Wait until the certificate shows `Active`** in the Edge Certificates list. Do not do step 6's + canary before then — a TLS error at the edge would be the certificate, not the tunnel. +4. Leave **SSL/TLS encryption mode** at *Full (strict)*. (It does not govern tunnel origins — + cloudflared verifies the Gateway itself — but it is the safe default for anything else.) + +Edge settings that would break non-browser clients once traffic moves (check them now, in the zone): + +- **Security → Bots → Bot Fight Mode: OFF.** It cannot be scoped by hostname or skipped by a WAF + rule, and it challenges `git`, `docker`/registry pushes, Nextcloud sync clients and API calls. +- **Browser Integrity Check** and the **WAF / managed rules**: make sure neither challenges or + blocks non-browser user agents on `gitea.home.keithjackson.dev` (git over HTTPS and the container + registry under `/v2/`) or on API paths. If either does, add a WAF **Skip** custom rule for those + hostnames/paths rather than turning protection off zone-wide. +- **Network → WebSockets: ON** (Grafana live, Headlamp-style consoles, Nextcloud, KubeVirt VNC). + +### Step 2 — log in and create the tunnel + +``` +cf tunnel login +# Open the printed URL, pick the keithjackson.dev zone. Writes $CF_DIR/cert.pem — an ACCOUNT-level +# origin certificate. Treat it as a credential. + +cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s +# Prints "Created tunnel home-k3s with id ". The UUID is not secret; note it. +cf tunnel list +``` + +### Step 3 — put the UUID into the config + +Do this in your own worktree on a new branch, as CLAUDE.md §0 requires: + +``` +UUID= +sed -i "s/^tunnel: REPLACE-ME-TUNNEL-UUID$/tunnel: ${UUID}/" platform/cloudflared/config.yaml +grep '^tunnel:' platform/cloudflared/config.yaml +``` + +### Step 4 — encrypt the credentials into the Secret, without printing them + +Run from that worktree. `sops set` edits the already-encrypted document in place; the JSON goes +from the mounted file to `sops` inside the container and never reaches the terminal: + +``` +./infra/devcontainer/dev.sh run bash -c ' + set -euo pipefail + cd /work + f=/root/.config/sops/cloudflared/tunnel.json + v="$(jq -c . "$f" | tr -d "\n" | jq -Rs .)" + sops set platform/cloudflared/cloudflared-credentials.sops.yaml \ + "[\"stringData\"][\"credentials.json\"]" "$v" + echo "credentials written" +' +``` + +(From a worktree, `dev.sh` needs `DEV_KUBECONFIG=/dev/null` and the `.git` mount shim — CLAUDE.md §3. +Or run this step from the main checkout and copy the encrypted file across.) + +Verify — prints only `ok` or fails, never a value: + +``` +./infra/devcontainer/dev.sh run bash -c ' + cd /work + UUID=$(sed -n "s/^tunnel: //p" platform/cloudflared/config.yaml) + sops -d --extract "[\"stringData\"][\"credentials.json\"]" \ + platform/cloudflared/cloudflared-credentials.sops.yaml \ + | jq -e --arg u "$UUID" ".TunnelID == \$u and (.TunnelSecret | length) > 20 and (.AccountTag | length) > 0" \ + >/dev/null && echo ok' +``` + +Then store `tunnel.json` **and** `cert.pem` in your password manager, and delete them from disk: + +``` +rm -f "$CF_DIR/tunnel.json" "$CF_DIR/cert.pem" +``` + +(`cert.pem` is only needed again to inspect (`tunnel info`), create or delete a tunnel. DNS records are +external-dns's, never `tunnel route dns`'s.) + +### Step 5 — PR it + +Commit `config.yaml` and the re-encrypted `cloudflared-credentials.sops.yaml`, gate, open a PR. No +other wiring is needed: the resource line and both overlay decryption patches already exist. On +merge Flux rolls the Deployment (the ConfigMap is content-hashed). + +That PR touches `**/*.sops.yaml`, a sensitive path in `infra/gitea/approval-policy.json`, so +`approval-policy` will block it until you post `/approve-override ` as a PR **review** of +type Comment (CLAUDE.md §1). + +### Step 6 — verify + +``` +# Both replicas Ready, on different nodes: +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n cloudflared get pods -o wide' + +# Logs: "Registered tunnel connection connIndex=0..3" on each replica. +infra/devcontainer/dev.sh run bash -c \ + 'kubectl --context k3s-prod -n cloudflared logs -l app.kubernetes.io/name=cloudflared --tail=40' +``` + +From the workstation: `cf tunnel info home-k3s` (needs `cert.pem` back in `$CF_DIR` temporarily) +lists **two connectors**. In Grafana/Prometheus, `sum(cloudflared_tunnel_ha_connections)` is ~8 and +both alerts are clear. + +**Canary — end to end, without touching any real record.** Hand-make ONE proxied CNAME for a +throwaway name (Cloudflare dashboard → DNS → Records → Add record): + + Type CNAME Name tunnel-canary.home Target .cfargotunnel.com Proxy status: Proxied + +Do NOT use `cloudflared tunnel route dns` for it (see *When to escalate or stop*). The name hits the +wildcard rule, so it travels edge → cloudflared → Gateway, where no HTTPRoute matches it: + +``` +curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://tunnel-canary.home.keithjackson.dev/ +# expected: "404 0" — a 404 from the Gateway's Envoy (no route for that host) over a VALID edge +# certificate (ssl_verify_result 0). A TLS error = step 1 is not Active yet. 502 = cloudflared +# could not reach or verify the Gateway; check the logs and `hubble observe` above. +``` + +**Then prove a REAL backend answers the tunnel's identity without a 403.** The canary cannot show +this: Envoy 404s an unknown host before it resolves any backend, and the `403 Access denied` +failure (Cilium re-checking the source pod's egress against the backend, CVE-2025-30162) only +happens once a backend is chosen. An ephemeral container in a cloudflared pod shares that pod's +network identity, so it takes exactly the tunnel's path to the Gateway Service: + +``` +infra/devcontainer/dev.sh run bash -c ' + POD=$(kubectl --context k3s-prod -n cloudflared get pod -l app.kubernetes.io/name=cloudflared \ + -o jsonpath="{.items[0].metadata.name}") + kubectl --context k3s-prod -n cloudflared debug "$POD" -q -i --image=docker.io/curlimages/curl:8.11.1 \ + --target=cloudflared -- sh -c "for h in grafana homepage-unused gitea; do + printf \"%s \" \$h; curl -sS -o /dev/null -w \"%{http_code}\n\" \ + --connect-to ::cilium-gateway-shared.gateway.svc.cluster.local:443 https://\$h.home.keithjackson.dev/; done"' +# expected: grafana and gitea answer 200/302 (anything but 403); a 403 there means the egress +# policy's backend selector does not cover that namespace. "homepage-unused" is a control: 404. +# A LAN-only host (e.g. jackett) is EXPECTED to 403 here — that is the second layer working. +``` + +(`kubectl debug` adds an ephemeral container: a cluster mutation, so an operator action. It +disappears with the pod on the next rollout.) + +Finally **delete the `tunnel-canary` CNAME** in the dashboard and confirm it is gone: +`dig +short tunnel-canary.home.keithjackson.dev @1.1.1.1` returns nothing. + +## Rollback + +- **Phase 1 (no DNS on the tunnel):** revert the PR. Flux prunes the `cloudflared` namespace and + everything in it. No user traffic is affected either way. Optionally `cf tunnel delete home-k3s` + (needs `cert.pem`) to retire the credential. +- **Emergency stop (operator, cluster mutation):** stop the connectors without waiting for a PR: + ``` + infra/devcontainer/dev.sh run bash -c \ + 'flux --context k3s-prod suspend kustomization cloudflared && + kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0' + ``` + Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then + `flux resume kustomization cloudflared`. +- **After the DNS flip:** the fast path back is DNS, not the tunnel — restore the previous records + (that change's own runbook). Revoking the tunnel credential (`cf tunnel delete`, or rotating it + in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked. diff --git a/runbooks/kustomization.yaml b/runbooks/kustomization.yaml index e260cc1..b955213 100644 --- a/runbooks/kustomization.yaml +++ b/runbooks/kustomization.yaml @@ -58,6 +58,7 @@ configMapGenerator: # improvised live, and the one hazard it exposed -- a StatefulSet with # persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 -- # destroyed monitoring/loki's volume before anyone noticed. + - cloudflare-tunnel=cloudflare-tunnel.md - cluster-maintenance-shutdown=cluster-maintenance-shutdown.md - cdi-import-stuck=cdi-import-stuck.md - cert-expiry=cert-expiry.md