Compare commits
37
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8cd5ab2e41 | ||
|
|
50c8719b12 | ||
|
|
8f60e6103a | ||
|
|
da0515c8c8 | ||
|
|
3a2ec5a12c | ||
|
|
f13d336630 | ||
|
|
8fe8d5df8a | ||
|
|
020e736541 | ||
|
|
92944486d1 | ||
|
|
2e777e13ca | ||
|
|
063c8acf77 | ||
|
|
16222606ae | ||
|
|
d9ab57f2d1 | ||
|
|
9724445a08 | ||
|
|
32b7d5bac1 | ||
|
|
83e0ac194f | ||
|
|
09f219fe7f | ||
|
|
aabc8c60b6 | ||
|
|
eb62c13c6b | ||
|
|
4226292dd6 | ||
|
|
213429a41f | ||
|
|
0120bc10d4 | ||
|
|
d1963bd000 | ||
|
|
0b0b5abc8e | ||
|
|
6e2e353bcc | ||
|
|
045d81c7a2 | ||
|
|
7be693bf37 | ||
|
|
9e9a3ea8cc | ||
|
|
64c5617ddd | ||
|
|
35ba875380 | ||
|
|
27d603f726 | ||
|
|
514ed5d5a4 | ||
|
|
9537ca2ddb | ||
|
|
b548224db8 | ||
|
|
3d7ecbcfe9 | ||
|
|
893c9dee4f | ||
|
|
619c2943da |
No files matched your search
@@ -165,7 +165,8 @@ Cost: the job shares the single act_runner slot with the gate, but the build its
|
||||
merge-gate.yaml the pipeline (lint → l0 → l1 → conformance → decision)
|
||||
images.yaml build/verify/publish this repo's container images (see above)
|
||||
scripts/
|
||||
install-tools.sh installs the pinned tools per stage (from ci/versions.env)
|
||||
install-tools.sh installs the pinned tools per stage (from ci/versions.env) — used by the
|
||||
dormant merge-gate.yaml blueprint only; gate.yml runs in the toolbox image
|
||||
decide.sh the R-TEST-03 merge decision over job results
|
||||
images.sh the image pipeline's driver (what to build, base, push, digest)
|
||||
README.md this file
|
||||
|
||||
+100
-11
@@ -3,8 +3,10 @@
|
||||
# This is the LIVE gate wired to the in-cluster Gitea Actions runner (ns gitea-actions,
|
||||
# Docker executor + DinD). It invokes the AGGREGATE merge gate — `./ci/gate.sh`, which is
|
||||
# exactly what `make -C ci gate` runs, so CI and the devcontainer gate the same way
|
||||
# (R-TEST-02): the runner just installs the pinned tools from ci/versions.env first (and see
|
||||
# below for why the script, not `make`, is invoked here). It runs every stage in order —
|
||||
# (R-TEST-02) — and since 2026-11-08 it runs INSIDE the same image: the job `container:` is the
|
||||
# CI-built devcontainer toolbox (see "THE JOB IMAGE" below), not a generic runner image that
|
||||
# downloads the pinned tools on every run. (See further below for why the script, not `make`,
|
||||
# is invoked here.) It runs every stage in order —
|
||||
# spec-immutability, lint, Kyverno L0, render==golden, L1 offline render — and then applies the
|
||||
# R-TEST-03 decision, printing
|
||||
# "MERGE GATE: PASS" (0) / "BLOCKED" (1) / "UNDETERMINED" (3) / "NOT APPLICABLE" (0). It is the
|
||||
@@ -17,7 +19,8 @@
|
||||
# kinds of non-run must be told apart HERE, in the caller, because only the caller knows which
|
||||
# environment it is:
|
||||
# * a pinned tool missing or a seam unwired ON THIS RUNNER is a DEFECT — the toolchain is
|
||||
# installed for us two steps up. GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
|
||||
# baked into the job image and checked against ci/versions.env two steps up.
|
||||
# GATE_STRICT_SKIPS=true DECLARES that (no env sniffing, no
|
||||
# "does this look like CI?" heuristic) and makes those skips exit 1: untolerated, job fails.
|
||||
# * spec-immutability CANNOT be verified here at all: the spec repo is a separate repo and is
|
||||
# not checked out into this runner (SPEC_REPO_SLUG unwired; blueprint in merge-gate.yaml). It
|
||||
@@ -41,6 +44,32 @@
|
||||
# same script the laptop runs; the extra pre-step is a read-only git-diff check and is
|
||||
# purely additive. The full pipeline blueprint (spec-checkout, per-layer jobs, decision job)
|
||||
# is retained dormant in merge-gate.yaml.
|
||||
#
|
||||
# THE JOB IMAGE (2026-11-08, feat/gate-toolbox-container). The gate job runs in
|
||||
# gitea.home.keithjackson.dev/home-network/toolbox, pinned BY DIGEST — the image
|
||||
# .gitea/workflows/images.yaml builds from infra/devcontainer/Containerfile, verifies, and
|
||||
# publishes. It is the SAME image `infra/devcontainer/dev.sh` runs, so a local gate and the CI
|
||||
# gate now execute the identical toolchain (R-TEST-02), not two installs that agree by intent.
|
||||
# Before this, the job ran in ghcr.io/catthehacker/ubuntu:act-22.04 and spent ~11s of every run
|
||||
# re-downloading yamllint, check-jsonschema, kyverno, promtool and chainsaw
|
||||
# (.gitea/workflows/scripts/install-tools.sh l0 + l1; measured on runs 494/495).
|
||||
# * PINNED BY DIGEST, MOVED BY RENOVATE. The `# renovate-digest:` marker above the image line
|
||||
# is the repo-wide convention (renovate.json5, "This repo's own CI-published images"):
|
||||
# Renovate follows the `main` tag and opens a digest-bump PR, whose OWN gate run then
|
||||
# executes on the new image — the bump tests itself. infra/gitea/registry-retention.sh reads
|
||||
# the same marker, so the pinned digest is never pruned. Never hand-edit it to a tag.
|
||||
# * PULL AUTH: none needed. The Home-Network org is public and the registry issues an
|
||||
# anonymous pull token (measured 2026-11-08: anonymous /v2/token -> manifest 200), so the
|
||||
# runner's DinD pulls it like any public image. If the org is ever made private, add
|
||||
# `credentials:` here with a read:package token secret — the automatic Actions token has no
|
||||
# package scope at /v2/token (see images.yaml's header).
|
||||
# * COLD PULL: ~467 MB compressed (1.54 GB unpacked). The DinD store is an emptyDir, so the
|
||||
# first job after a runner-pod restart or a digest bump pays one pull; every other run finds
|
||||
# it cached (forcePull=false), exactly as it already did for the catthehacker image.
|
||||
# * NO NODE in the toolbox, so `actions/checkout` (a Node action) cannot run here. The checkout
|
||||
# step is plain git instead — see its comment for how it keeps fetch-depth-0 semantics.
|
||||
# * install-tools.sh is NOT deleted: merge-gate.yaml (the dormant blueprint) still calls it,
|
||||
# and conformance/tests/l0-static/alert-rules-unit-tested asserts its promtool wiring.
|
||||
---
|
||||
name: gate
|
||||
|
||||
@@ -78,16 +107,76 @@ jobs:
|
||||
# Single required status check: the static merge gate.
|
||||
gate:
|
||||
runs-on: ubuntu-latest
|
||||
# The CI-built toolbox — the same image dev.sh runs. See "THE JOB IMAGE" in the header.
|
||||
container:
|
||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/toolbox tag=main
|
||||
image: gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501efe34fa0b039e3f0d360738686a0f59109866c49779768f5d5ed0ef81
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
|
||||
with:
|
||||
fetch-depth: 0 # full history so ci/touched-requirements.sh can diff vs main
|
||||
|
||||
- name: Install pinned CI toolchain (ci/versions.env)
|
||||
# Plain git, because actions/checkout is a Node action and the toolbox has no Node. It
|
||||
# reproduces what actions/checkout@v7 with `fetch-depth: 0` did here (read from its logged
|
||||
# commands on runs 490/495): full history of every branch + tags, so
|
||||
# ci/touched-requirements.sh can diff against origin/<base>; the exact event commit
|
||||
# checked out (detached for a PR, as `-B <branch>` for a push). One difference, on
|
||||
# purpose: the job token is passed to the single fetch through GIT_CONFIG_* environment
|
||||
# variables and is NOT persisted into .git/config — nothing after this step needs
|
||||
# authenticated git (ci/ and conformance/ make no network git calls), so there is no
|
||||
# reason to leave a credential in the workspace for the gate's scripts to read.
|
||||
- name: Checkout (git, full history)
|
||||
env:
|
||||
GIT_TOKEN: ${{ github.token }}
|
||||
SERVER_URL: ${{ github.server_url }}
|
||||
REPOSITORY: ${{ github.repository }}
|
||||
SHA: ${{ github.sha }}
|
||||
REF: ${{ github.ref }}
|
||||
run: |
|
||||
./.gitea/workflows/scripts/install-tools.sh l0
|
||||
./.gitea/workflows/scripts/install-tools.sh l1
|
||||
set -euo pipefail
|
||||
# act runs steps with umask 0000, so without this every checked-out directory is
|
||||
# world-writable — and Ansible then IGNORES infra/ansible/ansible.cfg as a source
|
||||
# ("Ansible is being run in a world writable directory"), which made the L0 check
|
||||
# prod-baremetal-not-dev-surface fail on this PR's first CI run (run 500). The
|
||||
# toolbox has Ansible, so that check now drives it for real. 0022 = what a
|
||||
# workstation clone gets (reproduced locally: 0000 fails the check, 0022 passes).
|
||||
umask 0022
|
||||
git config --global --add safe.directory "$PWD"
|
||||
git init -q .
|
||||
git remote add origin "${SERVER_URL}/${REPOSITORY}"
|
||||
auth="$(printf 'x-access-token:%s' "$GIT_TOKEN" | base64 -w0)"
|
||||
GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=http.extraHeader \
|
||||
GIT_CONFIG_VALUE_0="AUTHORIZATION: basic ${auth}" \
|
||||
git -c protocol.version=2 fetch --quiet --no-tags --prune --no-recurse-submodules origin \
|
||||
'+refs/heads/*:refs/remotes/origin/*' '+refs/tags/*:refs/tags/*' \
|
||||
"+${SHA}:refs/remotes/checkout/head"
|
||||
unset auth GIT_TOKEN
|
||||
case "$REF" in
|
||||
refs/heads/*) git checkout -q --force -B "${REF#refs/heads/}" "$SHA" ;;
|
||||
*) git checkout -q --force --detach "$SHA" ;;
|
||||
esac
|
||||
[ "$(git rev-parse HEAD)" = "$SHA" ] || { echo "::error::HEAD is not ${SHA}" >&2; exit 1; }
|
||||
echo "checked out $(git log -1 --format='%H %s') (ref ${REF}); $(git rev-list --count HEAD) commits of history"
|
||||
|
||||
# The image is the toolchain, so prove it is the PINNED one before trusting it: every
|
||||
# *_VERSION in ci/versions.env must equal what the image recorded at build time
|
||||
# (/etc/dev-toolchain-versions, written by infra/devcontainer/Containerfile; its `verify`
|
||||
# stage already asserted the version-sensitive binaries report those pins). This replaces
|
||||
# the per-run install: a drifted toolbox fails HERE, naming the tool, instead of gating with
|
||||
# a different engine. ci/versions.env and infra/devcontainer/versions.env must move together.
|
||||
- name: Toolchain is the one ci/versions.env pins
|
||||
run: |
|
||||
set -euo pipefail
|
||||
. ci/versions.env
|
||||
drift=0
|
||||
for var in $(grep -oE '^[A-Z_]+_VERSION' ci/versions.env); do
|
||||
tool="$(printf '%s' "${var%_VERSION}" | tr 'A-Z_' 'a-z-')"
|
||||
want="${!var}"
|
||||
got="$(sed -n "s/^${tool}=//p" /etc/dev-toolchain-versions)"
|
||||
if [ "$got" = "$want" ]; then
|
||||
printf 'ok %-18s %s\n' "$tool" "$want"
|
||||
else
|
||||
printf '::error::DRIFT %s: ci/versions.env pins %s, the job image has %s\n' "$tool" "$want" "${got:-nothing}"
|
||||
drift=1
|
||||
fi
|
||||
done
|
||||
[ "$drift" -eq 0 ]
|
||||
|
||||
# R-TEST-06: the spec is read-only to agents. An EXPLICIT early surfacing of the
|
||||
# spec-immutability check, so its result is a distinct, tolerant status line in the
|
||||
|
||||
@@ -76,7 +76,7 @@ bug but the honest answer, because proofs are claims about a specific tree on a
|
||||
|---|---|
|
||||
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
|
||||
| 1 Cluster bring-up | 3-node k3s HA, embedded etcd, kube-vip VIP, Cilium, Flux, SOPS. Originally Proxmox VMs; **now 3× mini-PC bare metal** (`k3s-prod-1/2/3`, `192.168.2.10/.11/.12`, k3s v1.36.4+k3s1, Debian 13) |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the authoritative LAN resolver** for `${BASE_DOMAIN}` on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
|
||||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon DNS, **both halves**: CoreDNS for pods, and **PiHole as the LAN resolver**, answering an explicit list of `${BASE_DOMAIN}` names (LAN-only apps, gitea/nextcloud/keycloak/auth, the L4 VIPs — everything else resolves publicly to Cloudflare, since 2026-11-09) on udp/53 + tcp/53 at a pinned VIP (`192.168.2.23` on prod). The LAN half was claimed from the start but did not exist until 2026-09-14 — PiHole's Service was ClusterIP :80/:4180, so LAN clients ran on hand-maintained per-client `dnsmasq` files |
|
||||
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
|
||||
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero unblocked 2026-09-13** — backs up to Cloudflare R2 (OQ-BKP closed; the `velero-plugin-for-aws` tagging incompatibility needed a pinned plugin). Longhorn has **no** backup target by decision: volume data is protected by Velero + kopia to R2. image-automation authored but disabled |
|
||||
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
|
||||
|
||||
@@ -100,6 +100,11 @@ metadata:
|
||||
# external-dns (source: service) publishes A unifi.home.keithjackson.dev -> the pinned VIP.
|
||||
# This is the browser + device target; UI is https://unifi.home.keithjackson.dev:11443.
|
||||
external-dns.alpha.kubernetes.io/hostname: "unifi.${BASE_DOMAIN}"
|
||||
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
|
||||
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
|
||||
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
|
||||
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
|
||||
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
|
||||
@@ -176,8 +176,8 @@ or any UDP (`3478`/`10001`) — exactly the ports adoption rides. So:
|
||||
|
||||
**UI access:** `https://unifi.home.keithjackson.dev:11443`. That name resolves to the VIP and not to
|
||||
the Gateway on purpose: both `platform/coredns/coredns-custom.yaml` (in-cluster) and
|
||||
`apps/wave-0/pihole.yaml` (LAN split-horizon) carve `unifi.${BASE_DOMAIN}` out of the
|
||||
zone-wide `${GATEWAY_VIP}` answer and point it at `${UNIFI_L4_VIP}`. `${BASE_DOMAIN}` on prod is the
|
||||
`apps/wave-0/pihole.yaml` (LAN split-horizon, an explicit name list since 2026-11-09) point
|
||||
`unifi.${BASE_DOMAIN}` at `${UNIFI_L4_VIP}` rather than `${GATEWAY_VIP}`. `${BASE_DOMAIN}` on prod is the
|
||||
apex `home.keithjackson.dev` (confirmed live in the `cluster-vars` ConfigMap) after the 2026-09-13
|
||||
flip, so this hostname is current, not a dev leftover.
|
||||
|
||||
|
||||
@@ -85,7 +85,7 @@ spec:
|
||||
- name: config
|
||||
mountPath: /config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
|
||||
@@ -83,15 +83,25 @@
|
||||
# and would also fix the same latent defect in unifi and crafty; it belongs in its own change.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# loadBalancerSourceRanges — THE LAN RESTRICTION, ENFORCED IN THE DATAPATH.
|
||||
# loadBalancerSourceRanges — THE LAN + VPN RESTRICTION, ENFORCED IN THE DATAPATH.
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# Cilium enforces this list in eBPF on the ingress node, on the VIP, BEFORE load-balancing and
|
||||
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} is dropped by the
|
||||
# datapath — it never reaches a backend and never reaches a network policy. This is what makes
|
||||
# before SNAT. A packet to ${PIHOLE_DNS_VIP}:53 from outside ${LAN_CIDR} and ${WG_CLIENT_CIDR} is
|
||||
# dropped by the datapath — it never reaches a backend and never reaches a network policy. This is what makes
|
||||
# "LAN clients only" a real, tested restriction rather than a comment: VERIFIED by narrowing the
|
||||
# range to a /24 that excludes the test client, after which the same query that had been answering
|
||||
# stopped being answered at all, and restoring it, after which it answered again.
|
||||
#
|
||||
# ${WG_CLIENT_CIDR} IS LISTED BECAUSE VPN CLIENTS REACH THIS CHECK UNTRANSLATED. wg0 is one of
|
||||
# Cilium's kube-proxy-replacement devices (measured 2026-11-08: `cilium-dbg status` lists eno1, wg0,
|
||||
# tun0), so a tunnel client's query is load-balanced as it enters from wg0, ahead of the iptables
|
||||
# MASQUERADE in platform/vpn-ingress/daemonset.yaml, and the source tested here is still 10.90.0.x
|
||||
# rather than the node's address. Without this entry a full-tunnel client's DNS is predicted to die
|
||||
# with DROP_NOT_IN_SRC_RANGE (derived from Cilium's source; NOT observed live, because no peer has
|
||||
# ever completed a handshake). The range is private and reachable only by a key in peers.conf, so
|
||||
# PiHole stays off the internet; conformance/tests/l0-static/pihole-dns-source-ranges fails the gate
|
||||
# if either entry goes missing or any range here stops being RFC1918.
|
||||
#
|
||||
# It is also the layer that survives the SNAT above. A CiliumNetworkPolicy cannot express this
|
||||
# restriction once the source has been rewritten: measured on this cluster, the forwarded query
|
||||
# arrives from the ingress node's internal router address and Cilium resolves that source to the
|
||||
@@ -113,6 +123,7 @@ spec:
|
||||
externalTrafficPolicy: Cluster
|
||||
loadBalancerSourceRanges:
|
||||
- ${LAN_CIDR}
|
||||
- ${WG_CLIENT_CIDR}
|
||||
selector:
|
||||
app.kubernetes.io/name: pihole
|
||||
ports:
|
||||
@@ -133,6 +144,7 @@ spec:
|
||||
externalTrafficPolicy: Cluster
|
||||
loadBalancerSourceRanges:
|
||||
- ${LAN_CIDR}
|
||||
- ${WG_CLIENT_CIDR}
|
||||
selector:
|
||||
app.kubernetes.io/name: pihole
|
||||
ports:
|
||||
@@ -158,7 +170,9 @@ spec:
|
||||
# "Policy denied ... identity world" drops in `cilium-dbg monitor`). So `world` is the only peer
|
||||
# selector that can describe LAN traffic that has been through the LoadBalancer, and the set of
|
||||
# addresses that can actually BE that traffic is fixed by loadBalancerSourceRanges before this
|
||||
# policy is consulted. Together: ${LAN_CIDR} in the datapath, udp/53 + tcp/53 here.
|
||||
# policy is consulted. Together: ${LAN_CIDR} + ${WG_CLIENT_CIDR} in the datapath, udp/53 + tcp/53
|
||||
# here. A VPN client delivered to a backend on its own ingress node keeps its 10.90.0.x source,
|
||||
# which Cilium also classifies as `world`, so this rule needs no new peer for it.
|
||||
#
|
||||
# This is NOT wider than the precedent it follows. unifi-allow-l4-ingress and crafty-allow-ingress
|
||||
# admit `fromEntities: [world, cluster, host]` on their L4 ports with NO source-range filter at all;
|
||||
@@ -187,7 +201,7 @@ spec:
|
||||
app.kubernetes.io/name: pihole
|
||||
ingress:
|
||||
- fromEntities:
|
||||
- world # LAN clients, post-LoadBalancer: see the block above for why not fromCIDR
|
||||
- world # LAN + VPN clients, post-LoadBalancer: see the block above for why not fromCIDR
|
||||
- host # this node's kubelet probes
|
||||
- remote-node # the node-to-node hop that forwards a query to wherever the pod runs
|
||||
toPorts:
|
||||
|
||||
@@ -73,7 +73,7 @@ spec:
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: '0'
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
value: local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}
|
||||
- name: FTLCONF_webserver_api_password
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
@@ -90,7 +90,7 @@ spec:
|
||||
- -c
|
||||
- "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n [ -n \"$url\" ] || continue\n printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n [ -n \"$url\" ] || continue\n # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n have=$(sql \"select count(*) from adlist where address='$url';\")\n if [ \"$have\" = \"0\" ]; then\n echo \"pihole-adlists: adding $url\"\n sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n CHANGED=1\n else\n en=$(sql \"select enabled from adlist where address='$url';\")\n if [ \"$en\" != \"1\" ]; then\n echo \"pihole-adlists: re-enabling $url\"\n sql \"update adlist set enabled=1 where address='$url';\"\n CHANGED=1\n fi\n fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n [ -n \"$addr\" ] || continue\n if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n echo \"pihole-adlists: disabling undeclared $addr\"\n sql \"update adlist set enabled=0 where address='$addr';\"\n echo changed >> /tmp/adlist-changed\n fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n echo \"pihole-adlists: declared set changed - running gravity update\"\n pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
|
||||
@@ -302,6 +302,8 @@ metadata:
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
annotations:
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -156,7 +156,7 @@ spec:
|
||||
, \"[Preferences]\"\n\ndef matches(raw):\n \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n v = raw.strip().strip('\"')\n if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n v = v[len(\"@ByteArray(\"):-1]\n try:\n salt_b64, digest_b64 = v.split(\":\", 1)\n salt = base64.b64decode(salt_b64, validate=True)\n digest = base64.b64decode(digest_b64, validate=True)\n except Exception:\n return False\n if not salt or not digest:\n return False\n return hmac.compare_digest(\n hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n lines = []\n\ndef value_of(key):\n for i, line in enumerate(lines):\n if line.startswith(key + \"=\"):\n return i, line.split(\"=\", 1)[1]\n return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n raise SystemExit(0) # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n at = out.index(SECTION) + 1\nelse:\n if out and out[-1].strip():\n out.append(\"\")\n out.append(SECTION)\n at = len(out)\nout[at:at] = desired\n\ntry:\n os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n tmp = conf + \".webui-cred.new\"\n with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n fh.write(\"\\n\".join(out) + \"\\n\")\n fh.flush()\n os.fsync(fh.fileno())\n # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n # cost the app the ability to save its own settings, so carry the ownership over.\n try:\n st = os.stat(conf)\n os.chown(tmp, st.st_uid, st.st_gid)\n os.chmod(tmp, stat.S_IMODE(st.st_mode))\n except FileNotFoundError:\n os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n int(os.environ.get(\"PGID\", \"1000\")))\n os.chmod(tmp, 0o664)\n os.replace(tmp, conf) # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n raise SystemExit(1)\nraise SystemExit(10)\nPY\n rc=$?\n\n if [ \"$rc\" = 0 ]; then\n log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n elif [ \"$rc\" = 10 ]; then\n log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n # have read the old, credential-less conf and be serving a random temporary password.\n # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n # Nothing to do when the service has not started yet - it will read the new file.\n # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n # 2026-09-13 failure spent the entire verification budget before the first login and\n # then fired attempts 2 and 3 with no wait left at all.\n sv=0\n while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n sv=$((sv + 1)); sleep 1\n done\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"qbittorrent-nox was already running - restarting it to load the credential\"\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n sleep 3\n fi\n else\n log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n log \"minting a random password\
|
||||
\ every start and BOTH arrs will break. Fix and restart.\" >&2\n exit 0\n fi\n\n # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n # (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n # at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n #\n # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n # conflating them is what made this hook lie once already (see the PORT comment). An\n # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n # its remaining login attempts to produce two more identical wrong diagnoses.\n api_ok=0\n while [ \"$waited\" -lt 120 ]; do\n code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n waited=$((waited + 1)); sleep 1\n done\n if [ \"$api_ok\" != 1 ]; then\n log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n exit 0\n fi\n body=$(curl -sS -m 10 \\\n --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n if [ \"$body\" = \"Ok.\" ]; then\n log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n exit 0\n fi\n log \"WebUI rejected the durable credential (login said '$body')\" >&2\n # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n # make the process RE-READ the file, so restart the supervised service and let the next\n # attempt re-verify. Without this the retry loop could never change its own outcome.\n if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n sleep 3\n fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
|
||||
@@ -77,6 +77,42 @@ spec:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
# act_runner config file — DELTAS ONLY. act_runner v0.6.1 LoadDefault()
|
||||
# (internal/pkg/config/config.go) unmarshals this file onto a ZERO Config and then fills every
|
||||
# unset field with the same default it uses when there is no file at all, so a file holding
|
||||
# only `runner.capacity` changes exactly that and nothing else. Do NOT paste in the output of
|
||||
# `act_runner generate-config`: that example is NOT the defaults. Its `runner.labels` would
|
||||
# OVERRIDE the registered labels in /data/.runner (daemon.go: config labels win when non-empty)
|
||||
# and repoint ubuntu-latest at a different job image; its `force_pull: true` differs from the
|
||||
# no-file default (false); and its sample `envs` would be injected into every job.
|
||||
#
|
||||
# Using a file also makes act_runner IGNORE the legacy GITEA_RUNNER_CAPACITY / _FILE /
|
||||
# _ENVIRON / _ENV_FILE / GITEA_DEBUG / GITEA_TRACE env vars (deprecated.go). None of them is
|
||||
# set on this Deployment; the GITEA_INSTANCE_URL / _REGISTRATION_TOKEN / _NAME / _LABELS env
|
||||
# below is consumed by the image's run.sh for registration, not by that legacy path.
|
||||
#
|
||||
# A ConfigMap edit alone does NOT restart the runner (it reads the file once at start). If a
|
||||
# future change touches ONLY this ConfigMap, also bump the Deployment's pod template (e.g. the
|
||||
# config-revision annotation below) or the new value stays inert until the next restart.
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: act-runner-config
|
||||
namespace: gitea-actions
|
||||
labels:
|
||||
app.kubernetes.io/name: act-runner
|
||||
data:
|
||||
config.yaml: |
|
||||
runner:
|
||||
# Two concurrent jobs (default 1). Measured 2026-11-08: queue wait ~1s normally, but
|
||||
# 29-759s whenever gate / images / renovate runs collide (run 472 waited 759s behind a
|
||||
# renovate burst). Both jobs share the ONE dind daemon below; act_runner names each job's
|
||||
# containers GITEA-ACTIONS-TASK-<task id> and creates a network per job, so they do not
|
||||
# collide. Workflow `concurrency:` groups are enforced by Gitea server-side and are
|
||||
# unaffected: gate.yml's gate-<ref> group still cancels a superseded run on the SAME ref,
|
||||
# and renovate's group still serialises renovate. The dind resources are sized for TWO jobs.
|
||||
capacity: 2
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
@@ -95,6 +131,10 @@ spec:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: act-runner
|
||||
annotations:
|
||||
# Bump when act-runner-config changes without any other pod-template change, so the
|
||||
# Deployment rolls and act_runner re-reads its config (see the ConfigMap header).
|
||||
homelab.keithjackson.dev/act-runner-config-revision: "1" # capacity 2
|
||||
spec:
|
||||
# CI holds no cluster credentials — do not mount a SA token into the runner or
|
||||
# (via the shared netns) the job containers (R-SEC-02, docs/14).
|
||||
@@ -146,12 +186,23 @@ spec:
|
||||
mountPath: /certs
|
||||
- name: dind-storage
|
||||
mountPath: /var/lib/docker
|
||||
# Sized for runner.capacity: 2. Every job's containers run UNDER this daemon, so this
|
||||
# container's cgroup carries the whole CI workload. Measured 2026-11-08 (Prometheus,
|
||||
# capacity 1): peak CPU 0.69 core (1m rate, 30d), peak working set 447Mi (30d), 0 OOMs.
|
||||
# cpu request 2: two jobs at today's peak is ~1.4 core, and the gate is being
|
||||
# parallelised, so per-job demand rises. The request is what the CFS share is
|
||||
# weighted by under contention (100m was a 1/20 share); NO cpu limit, deliberately,
|
||||
# so an idle node's spare cores are still usable. k3s-prod-2 (8 CPU) had 2945m
|
||||
# requested; +1.9 core = ~4.85 (61%), and prod-1/prod-3 fit too if it reschedules.
|
||||
# memory request 1Gi / limit 4Gi: two jobs at today's peak is ~0.9Gi. An OOM kill here
|
||||
# takes down dockerd and with it BOTH running jobs, so the limit carries ~4x headroom
|
||||
# over that for the parallelised gate. Nodes have ~31Gi allocatable.
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 256Mi
|
||||
cpu: "2"
|
||||
memory: 1Gi
|
||||
limits:
|
||||
memory: 2Gi
|
||||
memory: 4Gi
|
||||
startupProbe:
|
||||
exec:
|
||||
command:
|
||||
@@ -195,23 +246,36 @@ spec:
|
||||
value: /certs/client
|
||||
- name: DOCKER_TLS_VERIFY
|
||||
value: "1"
|
||||
# The image's run.sh passes `--config <this>` to BOTH `act_runner register` and
|
||||
# `act_runner daemon` when CONFIG_FILE is set (act_runner v0.6.1 scripts/run.sh).
|
||||
- name: CONFIG_FILE
|
||||
value: /etc/act_runner/config.yaml
|
||||
volumeMounts:
|
||||
- name: data
|
||||
mountPath: /data
|
||||
- name: certs
|
||||
mountPath: /certs
|
||||
readOnly: true
|
||||
- name: config
|
||||
mountPath: /etc/act_runner
|
||||
readOnly: true
|
||||
# Measured 2026-11-08 at capacity 1: peak CPU 0.24 core, peak working set 314Mi (30d)
|
||||
# against the old 512Mi limit. It holds per-task state, log buffers and the embedded
|
||||
# actions/cache server for every concurrent task, so the limit is doubled for capacity 2.
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
memory: 256Mi
|
||||
limits:
|
||||
memory: 512Mi
|
||||
memory: 1Gi
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim:
|
||||
claimName: act-runner-data
|
||||
- name: certs
|
||||
emptyDir: {}
|
||||
- name: config
|
||||
configMap:
|
||||
name: act-runner-config
|
||||
- name: dind-storage
|
||||
emptyDir: {}
|
||||
@@ -138,7 +138,7 @@ spec:
|
||||
- name: media
|
||||
mountPath: /media
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
|
||||
@@ -138,7 +138,7 @@ spec:
|
||||
- name: media
|
||||
mountPath: /media
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
|
||||
+58
-31
@@ -85,7 +85,7 @@ env:
|
||||
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
|
||||
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
|
||||
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
|
||||
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
|
||||
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}, ${WG_CLIENT_CIDR}]` on the DNS Services,
|
||||
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
|
||||
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
|
||||
- name: FTLCONF_dns_listeningMode
|
||||
@@ -108,41 +108,61 @@ env:
|
||||
# `loadBalancer.mode: dsr`, which preserves the client address.
|
||||
- name: FTLCONF_dns_rateLimit_count
|
||||
value: "0"
|
||||
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
|
||||
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
|
||||
# so the first line is a wildcard for the whole zone and each later line is a specific override.
|
||||
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
|
||||
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
|
||||
# changes nothing.
|
||||
# THE LAN ANSWERS — AN EXPLICIT LIST, NOT A ZONE WILDCARD (2026-11-09). A dnsmasq
|
||||
# `address=/<domain>/<ip>` entry answers that domain AND every subdomain of it (on LABEL
|
||||
# boundaries: `pihole.` does not match `pihole-dns.`), and dnsmasq resolves overlapping entries by
|
||||
# LONGEST MATCHING DOMAIN, so the order below is irrelevant. Every name NOT listed here is
|
||||
# FORWARDED to the upstreams above and gets the PUBLIC answer.
|
||||
#
|
||||
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
|
||||
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
|
||||
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
|
||||
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
|
||||
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
|
||||
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
|
||||
# is the annotation that means "this Service publishes its own hostname at its own address":
|
||||
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
|
||||
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
|
||||
# resolver address, are the complete set of exceptions today.
|
||||
# WHY NO WILDCARD ANY MORE. Until 2026-11-09 the first entry was `address=/${BASE_DOMAIN}/
|
||||
# ${GATEWAY_VIP}`, answering the whole zone. That made sense while every public record was a
|
||||
# DNS-only CNAME to `wan.${BASE_DOMAIN}` (no NAT hairpin on this router, so the LAN NEEDED a
|
||||
# private answer). Since PR #156 every public name is a PROXIED CNAME to the Cloudflare Tunnel
|
||||
# (platform/externaldns/README.md), which works identically from the LAN — so a name that is
|
||||
# public now SHOULD resolve publicly from the LAN too, and the LAN no longer depends on PiHole for
|
||||
# it: a client that bypasses PiHole (DoH browser, phone private DNS, the router's fallback) gets
|
||||
# the same working answer. The wildcard had also become actively misleading: it answered names
|
||||
# that exist nowhere (`wan.` itself resolved to the Gateway VIP on the LAN).
|
||||
#
|
||||
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
|
||||
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
|
||||
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
|
||||
# WHAT IS LISTED, and why each — enforced by conformance/tests/l0-static/pihole-lan-names:
|
||||
# -> ${GATEWAY_VIP}, LAN-ONLY: every HTTPRoute disowned from public DNS
|
||||
# (`external-dns.alpha.kubernetes.io/controller: none`, i.e. catalog `exposure: internal`,
|
||||
# plus plex and headlamp) — jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr,
|
||||
# headlamp. These have NO public record and the tunnel 404s them, so THIS LINE IS THEIR ONLY
|
||||
# PATH: drop one and the app is unreachable everywhere. A NEWLY ONBOARDED `internal` SERVICE
|
||||
# MUST BE ADDED HERE (the L0 check fails until it is). A new `public` service needs nothing.
|
||||
# -> ${GATEWAY_VIP}, PUBLIC BUT KEPT ON THE LAN PATH: gitea (git push, LFS, container registry)
|
||||
# and nextcloud (uploads) — Cloudflare's proxy caps a request body at 100 MB and a response
|
||||
# at 100 s, which a large push or upload exceeds; keycloak and auth (oauth2-proxy) — so the
|
||||
# OIDC login of those two LAN-path apps stays on the same path as the apps themselves.
|
||||
# -> their own L4 VIPs: unifi (${UNIFI_L4_VIP}) and crafty-mc (${CRAFTY_L4_VIP}) are not behind
|
||||
# the Gateway (grep `external-dns.alpha.kubernetes.io/hostname` on a Service:
|
||||
# apps/vms/unifi/50-service-l4.yaml, platform/crafty/service-l4.yaml); their public records
|
||||
# are DNS-only A records to the same RFC1918 VIPs, which this router's rebind protection
|
||||
# strips, so the LAN answer is still required. pihole-dns (${PIHOLE_DNS_VIP}) is PiHole's own
|
||||
# resolver address; plex-vip (${PLEX_L4_VIP}) is below.
|
||||
# NOT listed, deliberately, so they resolve via Cloudflare from the LAN as from anywhere: the apex,
|
||||
# grafana, crafty, seerr. In-cluster (platform/coredns/coredns-custom.yaml) and node
|
||||
# (/etc/hosts pins, node_baseline_hosts_pins) resolution are separate and unchanged.
|
||||
#
|
||||
# AAAA: dnsmasq answers a AAAA query for an `address=`d name with NODATA (NOERROR, no answer),
|
||||
# so IPv6-capable clients fall back to the A record instead of getting NXDOMAIN — the same
|
||||
# behaviour coredns-custom hand-writes a `template IN AAAA` block for. Unlisted (proxied) names
|
||||
# now get Cloudflare's real A and AAAA answers.
|
||||
#
|
||||
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
|
||||
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
|
||||
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
|
||||
# while resolution for the zone does not work. Confirmed both ways against the live image.
|
||||
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the LAN HALF of Plex's single advertised
|
||||
# hostname. Plex publishes ONE URL to plex.tv for both local and remote clients
|
||||
# (catalog/plex.yaml ADVERTISE_IP): on the LAN this answer points it at the tcp/32400
|
||||
# LoadBalancer, while Cloudflare answers the same name publicly with the WAN address, NATed to
|
||||
# the same VIP. WITHOUT THIS LINE the entry above answers it with ${GATEWAY_VIP} via the zone
|
||||
# wildcard, where NOTHING listens on 32400 — measured 2026-10-04, plex-vip.${BASE_DOMAIN}
|
||||
# resolved to the Gateway VIP and a connect to :32400 there returned http=000. That is a silent
|
||||
# failure: the Plex web UI on plex.${BASE_DOMAIN} keeps working through the Gateway on 443 while
|
||||
# every client that takes the ADVERTISED connection cannot reach the server at all.
|
||||
#
|
||||
# plex-vip -> ${PLEX_L4_VIP} (added 2026-10-04) is the name of the tcp/32400 LoadBalancer
|
||||
# (apps/wave-0/plex-remote.yaml) and the domain of the certificate Plex serves there
|
||||
# (catalog/plex.yaml CERTDOMAIN / customCertificateDomain, Secret `plex-vip-tls`). It is NOT what
|
||||
# Plex advertises any more (ADVERTISE_IP is a resolver-independent plex.direct name since
|
||||
# 2026-10-05) and it has NO public record, so this entry is its only resolution — kept for
|
||||
# reaching the VIP by name with a matching certificate from a LAN shell. Measured 2026-10-04,
|
||||
# before it existed the zone wildcard answered it with ${GATEWAY_VIP}, where nothing listens on
|
||||
# 32400; without it now, the name is NXDOMAIN.
|
||||
#
|
||||
# ── local-ttl=60 — THE SPLIT-HORIZON ANSWER MUST BE CACHEABLE ────────────────────────────────
|
||||
# MEASURED 2026-10-18, diagnosing "DNS flapping between the cluster ingress and the public IP":
|
||||
@@ -172,10 +192,17 @@ env:
|
||||
# cache expires, the router falls back, and the public answer latches again. That is a real
|
||||
# outage, and the fix for it is PiHole availability, not DNS caching.
|
||||
#
|
||||
# SINCE 2026-11-09 THE LATCH IS MOSTLY HARMLESS. The measurement above predates the tunnel: the
|
||||
# public answer was then `wan.`, unreachable from the LAN. Now the public answer for every
|
||||
# proxied name is Cloudflare, which works from the LAN, and those names are not listed below at
|
||||
# all. A latch still bites only the listed names: gitea/nextcloud/keycloak/auth fall back to
|
||||
# Cloudflare (working, but under its 100 MB / 100 s limits); the LAN-only names and the L4 VIPs
|
||||
# get NXDOMAIN or a stripped answer until the router re-queries PiHole.
|
||||
#
|
||||
# Applies to every `address=` reply below (dnsmasq: local-ttl governs replies made from
|
||||
# configuration, /etc/hosts and DHCP leases), so one directive covers all five names.
|
||||
# configuration, /etc/hosts and DHCP leases), so one directive covers every listed name.
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: "local-ttl=60;address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
value: "local-ttl=60;address=/gitea.${BASE_DOMAIN}/${GATEWAY_VIP};address=/nextcloud.${BASE_DOMAIN}/${GATEWAY_VIP};address=/keycloak.${BASE_DOMAIN}/${GATEWAY_VIP};address=/auth.${BASE_DOMAIN}/${GATEWAY_VIP};address=/headlamp.${BASE_DOMAIN}/${GATEWAY_VIP};address=/jackett.${BASE_DOMAIN}/${GATEWAY_VIP};address=/pihole.${BASE_DOMAIN}/${GATEWAY_VIP};address=/plex.${BASE_DOMAIN}/${GATEWAY_VIP};address=/qbittorrent.${BASE_DOMAIN}/${GATEWAY_VIP};address=/radarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/semaphore.${BASE_DOMAIN}/${GATEWAY_VIP};address=/sonarr.${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP};address=/plex-vip.${BASE_DOMAIN}/${PLEX_L4_VIP}"
|
||||
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
|
||||
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
|
||||
# carries the full rationale and the measurement that found it.
|
||||
|
||||
+8
-1
@@ -347,7 +347,14 @@ lifecycle:
|
||||
# `runbooks/plex-quicksync.md` is that procedure, plus how to verify it took.
|
||||
hardware:
|
||||
gpu: intel-quicksync
|
||||
exposure: public # edge port-forward + public DNS; clients + Seerr reach it (docs/09; R-DNS/R-NET)
|
||||
# EXPOSURE: internal (operator decision 2026-11-08, Cloudflare Tunnel cut-over). The plex.${BASE_DOMAIN}
|
||||
# web UI is LAN/VPN-only: `internal` renders `external-dns ... controller: none` on the HTTPRoute, so
|
||||
# external-dns DELETES the public plex record and its TXT, and platform/cloudflared/config.yaml maps the
|
||||
# name to http_status:404 so it can never be served through the tunnel either. REMOTE STREAMING IS NOT
|
||||
# AFFECTED: it never used this name. It rides plex.tv + the advertised plex.direct URL (ADVERTISE_IP
|
||||
# above) to the raw tcp/32400 L4 VIP (${PLEX_L4_VIP}, apps/wave-0/plex-remote.yaml) via the WAN
|
||||
# tcp/32400 forward — none of which this field touches. See runbooks/plex-remote-access.md.
|
||||
exposure: internal # web UI LAN/VPN-only; remote streaming = plex.tv + WAN tcp/32400 (docs/09; R-DNS/R-NET)
|
||||
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
||||
storage:
|
||||
- name: config
|
||||
|
||||
+1
-1
@@ -129,7 +129,7 @@ AUTH_MODES = ("oidc", "forward-auth", "native", "none")
|
||||
# the sidecar authenticates the browser against Keycloak (homelab realm, the app's auth.groups)
|
||||
# and only THEN proxies to the co-located app UI at http://127.0.0.1:<uiport>. The gate is IN
|
||||
# the data path (fail-closed: no valid session in an authorized group => no upstream; R-IDP-02).
|
||||
OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1"
|
||||
OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.15.6"
|
||||
OAUTH2_PROXY_PORT = 4180
|
||||
KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
|
||||
|
||||
|
||||
+1
-1
@@ -99,7 +99,7 @@ caller that must distinguish them invokes `./ci/gate.sh` directly.
|
||||
|
||||
| stage | a skip there means | verdict |
|
||||
|---|---|---|
|
||||
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is installed for us by `.gitea/workflows/scripts/install-tools.sh`, so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
|
||||
| `lint`, `l0`, `render-golden`, `l1` | a **pinned tool is absent** or a **wired seam went unwired** — it *could not run* | **BLOCKED** under strict skips (a CI runner: the toolchain is the job image — the CI-built toolbox, `.gitea/workflows/gate.yml` `container:` — so a skip is a defect); **UNDETERMINED** otherwise (a workstation missing a tool is a real local condition — still non-zero, still not a pass; the remedy is `infra/devcontainer/dev.sh`, which carries the pinned toolchain) |
|
||||
| `spec-immutability` (R-TEST-06) | the sibling spec repo is not checked out, so the spec **could not be verified** | **UNDETERMINED** — unverified is unverified. Deliberately *not* upgraded to BLOCKED by strict skips: the live runner does not check the spec repo out at all, so blocking would leave the only enforced status check permanently red for a reason no PR could clear. It is therefore the **one** skip whose non-answer a caller may knowingly tolerate, and `.gitea/workflows/gate.yml` does so in one visible line. `REQUIRE_SPEC_VERIFICATION=true` makes it a hard block once that checkout is wired. Locally: check the spec repo out beside this one, or set `SPEC_REPO` (`dev.sh` mounts a sibling automatically). |
|
||||
| `decisions-log` | **nothing** — this stage has no skip path. It reads only files in the tree under test and needs no tool and no cluster, so there is no environment in which it "could not run" | a reported skip is a **defect in the stage** and BLOCKS. The stage fails when the frozen `DECISIONS-LOG.md` is modified (which silently re-creates, for every *other* open branch, the tail-hunk conflict that made four of five PRs unmergeable on 2026-09-29) or when a `decisions/` entry is malformed. There is deliberately **no** `REQUIRE_DECISIONS_LOG` knob: a guard with an off switch is not a guard. See `decisions/README.md`. |
|
||||
| `conformance` (L2/L3) | `CLUSTER_AVAILABLE!=true` — the environment **declared** the capability absent (documented Phase-0 behaviour) | **does not deny a PASS**: a declared-inapplicable stage was never *supposed* to run. It is NAMED in the PASS headline and step summary, the touched requirements it would have covered are reported UNVERIFIED, and `REQUIRE_CLUSTER_TESTS=true` (Phase 1+) turns that into a block. A skip **despite** `CLUSTER_AVAILABLE=true` is a defect and BLOCKS. |
|
||||
|
||||
+3
-3
@@ -46,8 +46,8 @@
|
||||
# WHICH SKIPS BLOCK, AND WHY (per stage — a skip is not one thing):
|
||||
#
|
||||
# lint / l0 / render-golden / l1 A skip here means a PINNED tool is absent or a wired
|
||||
# seam went unwired. On a CI runner the pinned toolchain is installed for us by
|
||||
# .gitea/workflows/scripts/install-tools.sh, so a skip there is a DEFECT, not an
|
||||
# seam went unwired. On a CI runner the pinned toolchain is the job image itself (the
|
||||
# CI-built toolbox, .gitea/workflows/gate.yml), so a skip there is a DEFECT, not an
|
||||
# environment: it BLOCKS. On a workstation a missing tool is a real local condition, so
|
||||
# it degrades to UNDETERMINED (still non-zero, still never a pass). See strict_skips()
|
||||
# in ci/lib.sh for how CI is detected and why misdetecting it cannot fail open.
|
||||
@@ -257,7 +257,7 @@ for s in lint l0 render-golden l1; do
|
||||
skip)
|
||||
if strict_skips; then
|
||||
err "${s}: skipped (unwired seam / missing tool) under STRICT skips, where the pinned"
|
||||
err " toolchain is installed for us (.gitea/workflows/scripts/install-tools.sh) and every"
|
||||
err " toolchain is baked into the job image (.gitea/workflows/gate.yml container:) and every"
|
||||
err " seam is on main — so this is a DEFECT, not an environment -> blocks merge"
|
||||
summary "- ${s}: **SKIPPED under strict skips** (unwired seam / missing tool) — blocks merge"
|
||||
BLOCK=1
|
||||
|
||||
@@ -97,9 +97,9 @@ cluster_available() { truthy "${CLUSTER_AVAILABLE:-false}"; }
|
||||
# --- Skip strictness: is this a CI runner, or a workstation? -------------------
|
||||
# A SKIPPED stage is never green anywhere (HARD RULE 4), but its SEVERITY differs:
|
||||
#
|
||||
# CI the pinned toolchain is installed by .gitea/workflows/scripts/install-tools.sh
|
||||
# before the gate runs, and every seam is on `main`. A stage that skips here is a
|
||||
# DEFECT (a tool dropped from the installer, a seam that went unwired) -> BLOCK.
|
||||
# CI the pinned toolchain IS the job image (the CI-built toolbox; gate.yml `container:`),
|
||||
# and every seam is on `main`. A stage that skips here is a
|
||||
# DEFECT (a tool dropped from the image, a seam that went unwired) -> BLOCK.
|
||||
# local a missing pinned tool is a real, ordinary local condition -> the gate reports
|
||||
# UNDETERMINED (exit 3): not green, not a pass, but distinguishable from a red.
|
||||
#
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
# Flux Kustomization: Cloudflare Tunnel connector — reconciles ./platform/cloudflared.
|
||||
#
|
||||
# ISOLATION IS THE POINT OF THIS FILE. The Deployment it applies CANNOT become Ready until the
|
||||
# operator enrols the tunnel (runbooks/cloudflare-tunnel.md): it ships with REPLACE-ME credentials
|
||||
# and a placeholder UUID, and a wrong or revoked credential later produces the same state. So:
|
||||
#
|
||||
# * ITS OWN Kustomization, not a line in platform/kustomization.yaml. `platform` is wait:true and
|
||||
# gates platform-config, which gates every apps wave; an unenrolled tunnel there would be a
|
||||
# platform-wide stall ON MERGE, with the cause (a missing manual step) invisible from any Flux
|
||||
# condition. Same reasoning, and same failure shape, as vpn-ingress's unfilled server key.
|
||||
# * wait: false. Its pods are legitimately not-Ready whenever the credentials are absent or the
|
||||
# edge is unreachable. Health is reported by the CloudflaredTunnelDown / ReplicasDegraded
|
||||
# alerts (platform/cloudflared/alerts.yaml), not by this Kustomization's Ready condition —
|
||||
# which reaches Ready on APPLY success.
|
||||
# * NOTHING dependsOn THIS, and nothing should. A missing or invalid credentials Secret can then
|
||||
# only ever affect the tunnel itself. conformance/tests/l0-static/cloudflared-ingress-rules
|
||||
# fails the gate if any Kustomization gains `dependsOn: cloudflared`, or if this one stops
|
||||
# being wait:false.
|
||||
# * No Certificate, no HelmRelease, no CRD: nothing in the path for Flux's health checks to wait
|
||||
# on even if wait were flipped (the Certificate-stall lesson).
|
||||
#
|
||||
# dependsOn: [platform-config] — A REAL EDGE.
|
||||
# 1. CRD ORDERING: PodMonitor and PrometheusRule are monitoring.coreos.com CRs whose CRDs are
|
||||
# installed by kube-prometheus-stack in `platform`; platform-config dependsOn platform, so this
|
||||
# edge carries that guarantee transitively (Flux aborts a whole pass on an unknown kind).
|
||||
# 2. THE FAR SIDE OF THE TUNNEL: the Gateway (and so the cilium-gateway-shared Service the ingress
|
||||
# rules point at) is applied by platform-config. A tunnel that comes up before its origin is a
|
||||
# support call.
|
||||
#
|
||||
# ZERO EFFECT ON LIVE TRAFFIC UNTIL DNS MOVES. Cloudflare only sends a hostname through a tunnel
|
||||
# when that hostname's DNS record is a CNAME to <uuid>.cfargotunnel.com. This change creates no
|
||||
# such record and leaves external-dns, the Gateway annotations, Pi-hole and CoreDNS untouched.
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: cloudflared
|
||||
namespace: flux-system
|
||||
spec:
|
||||
interval: 10m
|
||||
retryInterval: 2m
|
||||
timeout: 5m
|
||||
prune: true
|
||||
wait: false
|
||||
dependsOn:
|
||||
- name: platform-config
|
||||
sourceRef:
|
||||
kind: GitRepository
|
||||
name: flux-system
|
||||
path: ./platform/cloudflared
|
||||
# REQUIRED, not boilerplate: platform/cloudflared/config.yaml renders `${BASE_DOMAIN}` into every
|
||||
# ingress hostname, and alerts.yaml into both runbook_url annotations. Without this block the
|
||||
# token becomes the EMPTY STRING and the Kustomization still reports Ready (substitution fails
|
||||
# OPEN), leaving ingress rules for `plex.` and `*.` — a tunnel that 404s everything at best.
|
||||
postBuild:
|
||||
substituteFrom:
|
||||
- kind: ConfigMap
|
||||
name: cluster-vars
|
||||
# SOPS decryption is layered per-environment: clusters/dev and clusters/prod each patch THIS
|
||||
# Kustomization by name with a `decryption:` stanza for cloudflared-credentials.sops.yaml.
|
||||
# Without it the Secret applies cleanly full of ENC[...] and cloudflared cannot authenticate.
|
||||
@@ -78,6 +78,13 @@ resources:
|
||||
# `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have
|
||||
# stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch.
|
||||
- vpn-ingress.yaml # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config)
|
||||
# Cloudflare Tunnel connector (north-south INGRESS from the Cloudflare edge) — POST-platform tier
|
||||
# (dependsOn platform-config, wait:false), and NOTHING dependsOn it. Non-gating is load-bearing,
|
||||
# as for vpn-ingress: the Deployment cannot become Ready until the operator enrols the tunnel
|
||||
# (runbooks/cloudflare-tunnel.md), so in a gating tier an unfinished manual step — or a revoked
|
||||
# credential later — would stall every apps wave. Carries cloudflared-credentials.sops.yaml, so
|
||||
# both overlays add a SOPS decryption patch for it (by name). See clusters/base/cloudflared.yaml.
|
||||
- cloudflared.yaml # Flux Kustomization -> ./platform/cloudflared (dependsOn platform-config)
|
||||
# PM agent — reads Gitea issues and runs one worker agent per assignment that authors a
|
||||
# change, gates it and opens a pull request. POST-APP tier (dependsOn apps-wave-0,
|
||||
# wait:false): what it needs is GITEA, not Keycloak or the Gateway, and it must never gate
|
||||
|
||||
@@ -140,6 +140,23 @@ patches:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries
|
||||
# platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one
|
||||
# secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY
|
||||
# SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot
|
||||
# authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band
|
||||
# (name only).
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: cloudflared
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
|
||||
# pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
|
||||
# account and the Anthropic subscription token from `claude setup-token`. Without this
|
||||
|
||||
@@ -131,6 +131,23 @@ patches:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries
|
||||
# platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one
|
||||
# secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY
|
||||
# SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot
|
||||
# authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band
|
||||
# (name only).
|
||||
- patch: |-
|
||||
apiVersion: kustomize.toolkit.fluxcd.io/v1
|
||||
kind: Kustomization
|
||||
metadata:
|
||||
name: cloudflared
|
||||
namespace: flux-system
|
||||
spec:
|
||||
decryption:
|
||||
provider: sops
|
||||
secretRef:
|
||||
name: sops-age
|
||||
# pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
|
||||
# pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
|
||||
# account and the Anthropic subscription token from `claude setup-token`. Without this
|
||||
|
||||
@@ -120,6 +120,14 @@ requirements:
|
||||
doc: docs/09-dns-and-ingress.md
|
||||
safety_critical: false
|
||||
summary: "Internal clients resolve to the internal Gateway (split-horizon)."
|
||||
- id: R-DNS-04
|
||||
doc: docs/09-dns-and-ingress.md
|
||||
safety_critical: false
|
||||
summary: "Public HTTP(S) reachable with no inbound 80/443 forward; LAN works whichever resolver answers."
|
||||
- id: R-DNS-05
|
||||
doc: docs/09-dns-and-ingress.md
|
||||
safety_critical: false
|
||||
summary: "Only declared-public names reachable through public ingress; LAN/VPN-only names are refused."
|
||||
|
||||
# --- Automated updates ---
|
||||
- id: R-UPD-01
|
||||
@@ -129,7 +137,7 @@ requirements:
|
||||
- id: R-UPD-02
|
||||
doc: docs/11-automated-updates.md
|
||||
safety_critical: true
|
||||
summary: "Updates need no human/AI/hand-authored change; bad update auto-rolls-back."
|
||||
summary: "Allowed-class updates need no human/AI/hand-authored change; bad update auto-rolls-back."
|
||||
- id: R-UPD-03
|
||||
doc: docs/11-automated-updates.md
|
||||
safety_critical: false
|
||||
@@ -175,15 +183,15 @@ requirements:
|
||||
- id: R-ENV-01
|
||||
doc: docs/02-environments-and-provisioning.md
|
||||
safety_critical: false
|
||||
summary: "dev/test provisioned as local KVMs via IaC (create/baseline/rebuild)."
|
||||
summary: "Nodes baselined, cluster-joined and managed as code; bare-metal imaging is checklist-driven."
|
||||
- id: R-ENV-02
|
||||
doc: docs/02-environments-and-provisioning.md
|
||||
safety_critical: false
|
||||
summary: "Same GitOps config promotes dev/test -> prod (authored once)."
|
||||
summary: "GitOps config authored once as base plus overlay; a future environment needs overrides only."
|
||||
- id: R-ENV-03
|
||||
doc: docs/02-environments-and-provisioning.md
|
||||
safety_critical: false
|
||||
summary: "prod is 3x mini-PC bare-metal k3s with local storage."
|
||||
summary: "Production is 3x mini-PC bare metal; changes reach it only through the PR gate and Flux."
|
||||
|
||||
# --- GitOps & delivery ---
|
||||
- id: R-GITOPS-01
|
||||
@@ -212,6 +220,10 @@ requirements:
|
||||
doc: docs/07-security-and-secrets.md
|
||||
safety_critical: false
|
||||
summary: "All security properties demonstrable/auditable."
|
||||
- id: R-SEC-04
|
||||
doc: docs/07-security-and-secrets.md
|
||||
safety_critical: false
|
||||
summary: "Kubernetes API not reachable from the public internet by any path; LAN/VPN-only."
|
||||
|
||||
# --- Identity & access ---
|
||||
- id: R-IDP-01
|
||||
|
||||
@@ -411,6 +411,32 @@ SCRAPE_OWNERS: list[dict] = [
|
||||
"Keeping the two in one file makes that pair inseparable",
|
||||
},
|
||||
},
|
||||
# ── ADDED 2026-11-08 (feat/cloudflare-tunnel). A NEW OWNER, NOT A RELAXED ASSERTION. ───────
|
||||
# Purely additive, the same shape as the kured entry: it registers the cloudflared_* family
|
||||
# and the Service+ServiceMonitor that scrapes it, and makes the check STRICTER for that component.
|
||||
{
|
||||
"owner": "cloudflared: headless metrics Service + ServiceMonitor on port `metrics` (:2000)",
|
||||
"prefixes": ("cloudflared_",),
|
||||
"file": "platform/cloudflared/servicemonitor.yaml",
|
||||
"require": [
|
||||
r"^kind:\s*ServiceMonitor\s*$",
|
||||
r"^kind:\s*Service\s*$",
|
||||
r"^\s*-\s*port:\s*metrics\s*$",
|
||||
],
|
||||
"forbid": [],
|
||||
"why": "cloudflared_tunnel_ha_connections powers CloudflaredTunnelDown, the only signal "
|
||||
"that the tunnel has lost the Cloudflare edge. That rule is written to FIRE on an "
|
||||
"absent series, so a lost scrape is loud rather than inert — but it would then be "
|
||||
"permanently firing for a wiring reason, which teaches the operator to ignore it",
|
||||
"companion": {
|
||||
"file": "platform/cloudflared/kustomization.yaml",
|
||||
"require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
|
||||
"why": "platform/cloudflared/networkpolicy.yaml ships the namespace's default-deny "
|
||||
"(a platform namespace, outside Kyverno's generator) together with the ingress "
|
||||
"allow for Prometheus on :2000. Keeping them in one file, applied together, is "
|
||||
"what stops a baseline-without-allow from emptying the TSDB",
|
||||
},
|
||||
},
|
||||
]
|
||||
|
||||
# `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — the Cloudflare Tunnel exposes only what the operator decided, and cannot stall Flux
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# platform/cloudflared/config.yaml is the entire internet-facing surface of the tunnel. This test
|
||||
# asserts the operator decisions of 2026-11-08 against it on every gate run: no rule may target
|
||||
# anything but a 404 or the shared Gateway (so never the Kubernetes API); the LAN/VPN-only
|
||||
# hostnames (plex, jackett, pihole, semaphore, qbittorrent, radarr, sonarr, headlamp,
|
||||
# unifi, crafty-mc) 404 before the wildcard; the
|
||||
# egress policy admits no API/node entity; and the Flux tier is isolated (wait:false, nothing
|
||||
# dependsOn it, both overlays decrypt it). It proves itself non-vacuous by mutation. Rationale and
|
||||
# the honest NOT-proven list: check.sh.
|
||||
#
|
||||
# WHY SCOPE, NOT REQUIREMENTS: the property spans platform/cloudflared, platform/gateway (the
|
||||
# derived Service name), clusters/base and both overlays. A requirement-scoped selection would skip
|
||||
# it on, e.g., a Gateway rename — precisely a change that breaks it.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: cloudflared-ingress-rules
|
||||
labels:
|
||||
conformance.home.k3s/R-NET-01: "true"
|
||||
conformance.home.k3s/R-NET-02: "true"
|
||||
conformance.home.k3s/R-DNS-04: "true"
|
||||
conformance.home.k3s/R-DNS-05: "true"
|
||||
conformance.home.k3s/R-SEC-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-NET-01, R-NET-02, R-DNS-04, R-DNS-05, R-SEC-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
The Cloudflare Tunnel routes only to the shared Gateway or a 404, never to the Kubernetes
|
||||
API; the LAN/VPN-only hostnames 404 ahead of the wildcard; the connector's egress admits no
|
||||
API or node entity; and its Flux tier cannot stall any other tier.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-tunnel-ingress-rules-and-isolation
|
||||
description: >-
|
||||
Parse platform/cloudflared/config.yaml with BASE_DOMAIN substituted, assert every rule's
|
||||
target against the Gateway Service derived from platform/gateway/gateway.yaml, assert the
|
||||
excluded hostnames 404 first, check the egress CiliumNetworkPolicy and the Flux wiring, and
|
||||
run the evaluator on mutated configs to prove it is not vacuous; and assert the
|
||||
credentials Secret volume is readable by the container's run-as uid/groups (fsGroup +
|
||||
mode), self-tested by mutation.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,321 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — cloudflared-ingress-rules
|
||||
# Serves: R-NET-01 / R-NET-02 (default-deny; allow rules declarative and scoped), and the four
|
||||
# operator decisions of 2026-11-08 recorded in decisions/2026-11-08-cloudflare-tunnel.md.
|
||||
#
|
||||
# WHY THIS EXISTS. platform/cloudflared/config.yaml is the whole of what the internet can reach
|
||||
# through the Cloudflare Tunnel. A one-line edit there — a rule pointing at the Kubernetes API, a
|
||||
# LAN-only hostname moved below the wildcard, a dropped catch-all — publishes something the operator
|
||||
# decided must never be public, and nothing in the cluster turns red: the tunnel stays connected and
|
||||
# the Kustomization stays Ready. So the decisions are asserted here, on every gate run.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# THEN (1) every ingress rule's `service` is `http_status:404` or EXACTLY the shared Gateway's
|
||||
# in-cluster Service, derived from platform/gateway/gateway.yaml (so a Gateway rename
|
||||
# fails here rather than in production) — which by construction excludes the
|
||||
# Kubernetes API, and a named deny-list says so explicitly when it is violated;
|
||||
# (2) each LAN/VPN-only hostname (plex, plex-vip, jackett, pihole, pihole-dns, semaphore,
|
||||
# qbittorrent, radarr, sonarr, headlamp, unifi, crafty-mc) has a rule mapping it to http_status:404 that comes BEFORE any
|
||||
# Gateway-bound rule (cloudflared matches first-rule-wins);
|
||||
# (3) the apex and the wildcard are the Gateway rules, with matchSNItoHost, and nothing
|
||||
# disables origin TLS verification; the last rule is a hostname-less 404 catch-all;
|
||||
# (4) the egress CiliumNetworkPolicy admits none of kube-apiserver/host/remote-node/
|
||||
# cluster/all and no CIDR rule, and its backend selector excludes the LAN-only
|
||||
# namespaces (the second, independent layer behind (2));
|
||||
# (5) the Flux tier is ISOLATED: its own Kustomization, wait:false, not listed in
|
||||
# platform/kustomization.yaml, nothing anywhere dependsOn it, and both overlays carry its
|
||||
# SOPS decryption patch — so a missing or invalid credential can stall nothing else;
|
||||
# (6) the rule evaluator is not vacuous: it is run against mutated copies of the real config
|
||||
# (an API rule added, plex moved below the wildcard, the catch-all removed) and must
|
||||
# flag each one;
|
||||
# AND (7) every Secret volume the container mounts — the tunnel credentials above all — is
|
||||
# READABLE by the container's run-as identity, modelled the way the kubelet lays it out:
|
||||
# files owned by uid 0, group-owned by fsGroup if set (else gid 0), with the item mode or
|
||||
# defaultMode (API default 0644). The identity is the container securityContext over
|
||||
# the pod's; the process's groups are runAsGroup + fsGroup + supplementalGroups. An
|
||||
# unset runAsUser is a FAIL, not a pass: the image default cannot be read offline. The
|
||||
# model is deliberately conservative — it does not credit the kubelet's habit of
|
||||
# OR-ing group-read into fsGroup-owned volumes, so 0400+fsGroup fails here though it
|
||||
# would work. Self-tested by mutation (fsGroup removed; mode 0400; runAsUser unset).
|
||||
# This is the 2026-11-08 CrashLoop: runAsUser 65532, no fsGroup, 0440 root:root.
|
||||
#
|
||||
# NOT proven (static): that Cloudflare routes only the hostnames whose DNS points at the tunnel,
|
||||
# that the Gateway's certificate validates from cloudflared, or that Envoy enforces (4) — those are
|
||||
# runtime properties, verified by the operator per runbooks/cloudflare-tunnel.md.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
exec python3 - "$ROOT" <<'PY'
|
||||
import copy
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("CHECK FAIL (cloudflared-ingress-rules): PyYAML is required", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES, CHECKS = [], 0
|
||||
|
||||
|
||||
def check(cond, desc):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(desc)
|
||||
|
||||
|
||||
def docs(path):
|
||||
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
|
||||
|
||||
|
||||
COMP = ROOT / "platform" / "cloudflared"
|
||||
CONFIG = COMP / "config.yaml"
|
||||
if not CONFIG.is_file():
|
||||
print(f"CHECK FAIL (cloudflared-ingress-rules): {CONFIG.relative_to(ROOT)} is absent — the "
|
||||
"component this check guards was moved or deleted; re-point or delete this check in the "
|
||||
"SAME change", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
# The hostnames the operator decided never go through the tunnel (decisions 2026-11-08).
|
||||
# headlamp: operator decision 2026-11-08 (LAN/VPN-only). qbittorrent/radarr/sonarr: catalog
|
||||
# `exposure: internal`, like jackett/pihole/semaphore.
|
||||
EXCLUDED_HOSTS = ("plex", "plex-vip", "jackett", "pihole", "pihole-dns", "semaphore",
|
||||
"qbittorrent", "radarr", "sonarr", "headlamp", "unifi", "crafty-mc")
|
||||
# Their Gateway-backend namespaces (unifi / crafty-mc are L4 VIPs, not Gateway backends).
|
||||
EXCLUDED_NAMESPACES = ("plex", "jackett", "pihole", "semaphore", "qbittorrent", "radarr",
|
||||
"sonarr", "headlamp")
|
||||
FORBIDDEN_ENTITIES = {"kube-apiserver", "host", "remote-node", "cluster", "all"}
|
||||
# Any of these in a `service` means a route toward the Kubernetes API (or a raw-TCP/SSH bastion).
|
||||
API_MARKERS = ("kubernetes", "kube-apiserver", "apiserver", ":6443", "10.43.0.1", "tcp://",
|
||||
"ssh://", "unix:", "bastion")
|
||||
|
||||
# ── substitution: ${BASE_DOMAIN} from cluster-vars, the ONLY token allowed ──────────────────────
|
||||
cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
|
||||
if d.get("kind") == "ConfigMap")
|
||||
base_domain = cv["data"]["BASE_DOMAIN"]
|
||||
raw = CONFIG.read_text()
|
||||
tokens = set(re.findall(r"\$\{([A-Za-z0-9_]+)\}", raw))
|
||||
check(tokens == {"BASE_DOMAIN"},
|
||||
f"(0) config.yaml uses exactly the ${{BASE_DOMAIN}} token (found {sorted(tokens)}); any other "
|
||||
f"token is substituted from cluster-vars unseen by this check")
|
||||
check(re.search(r"\$(?!\{BASE_DOMAIN\})", raw) is None,
|
||||
"(0) config.yaml contains a bare `$` — Flux envsubst would rewrite or empty it")
|
||||
cfg = yaml.safe_load(raw.replace("${BASE_DOMAIN}", base_domain))
|
||||
|
||||
# ── the Gateway Service, derived rather than hard-coded ─────────────────────────────────────────
|
||||
gw = next(d for d in docs(ROOT / "platform" / "gateway" / "gateway.yaml") if d.get("kind") == "Gateway")
|
||||
GATEWAY_SVC = (f"https://cilium-gateway-{gw['metadata']['name']}."
|
||||
f"{gw['metadata']['namespace']}.svc.cluster.local:443")
|
||||
|
||||
|
||||
def evaluate(cfg):
|
||||
"""Return the list of rule violations for a parsed cloudflared config."""
|
||||
bad = []
|
||||
rules = cfg.get("ingress") or []
|
||||
if len(rules) < len(EXCLUDED_HOSTS) + 3:
|
||||
bad.append(f"only {len(rules)} ingress rules — the scan is vacuous")
|
||||
return bad
|
||||
for i, r in enumerate(rules):
|
||||
svc = str(r.get("service", ""))
|
||||
if any(m in svc.lower() for m in API_MARKERS):
|
||||
bad.append(f"rule {i} ({r.get('hostname', '<catch-all>')}) targets {svc!r}, which "
|
||||
f"is a route toward the Kubernetes API / a raw bastion — the API stays "
|
||||
f"local-only (operator decision 2026-11-08)")
|
||||
if svc not in ("http_status:404", GATEWAY_SVC):
|
||||
bad.append(f"rule {i} ({r.get('hostname', '<catch-all>')}) targets {svc!r}; only "
|
||||
f"http_status:404 or {GATEWAY_SVC} are permitted")
|
||||
if (r.get("originRequest") or {}).get("noTLSVerify"):
|
||||
bad.append(f"rule {i} disables origin TLS verification (noTLSVerify)")
|
||||
if (cfg.get("originRequest") or {}).get("noTLSVerify"):
|
||||
bad.append("top-level originRequest disables origin TLS verification (noTLSVerify)")
|
||||
|
||||
gw_idx = [i for i, r in enumerate(rules) if r.get("service") == GATEWAY_SVC]
|
||||
first_gw = min(gw_idx) if gw_idx else len(rules)
|
||||
for h in EXCLUDED_HOSTS:
|
||||
fqdn = f"{h}.{base_domain}"
|
||||
hits = [i for i, r in enumerate(rules) if r.get("hostname") == fqdn]
|
||||
if not hits:
|
||||
bad.append(f"no rule for LAN/VPN-only host {fqdn}; it would fall through to the "
|
||||
f"wildcard and reach the Gateway")
|
||||
continue
|
||||
i = hits[0]
|
||||
if rules[i].get("service") != "http_status:404":
|
||||
bad.append(f"{fqdn} maps to {rules[i].get('service')!r}, not http_status:404")
|
||||
if i > first_gw:
|
||||
bad.append(f"{fqdn}'s 404 rule (#{i}) comes AFTER the first Gateway rule (#{first_gw}); "
|
||||
f"cloudflared is first-match, so the Gateway rule wins")
|
||||
|
||||
gw_hosts = {rules[i].get("hostname") for i in gw_idx}
|
||||
if gw_hosts != {base_domain, f"*.{base_domain}"}:
|
||||
bad.append(f"the Gateway-bound hostnames are {sorted(map(str, gw_hosts))}; expected "
|
||||
f"exactly the apex and the wildcard")
|
||||
for i in gw_idx:
|
||||
if (rules[i].get("originRequest") or {}).get("matchSNItoHost") is not True:
|
||||
bad.append(f"Gateway rule #{i} lacks originRequest.matchSNItoHost: true — the Gateway "
|
||||
f"would not see the SNI that selects its wildcard certificate")
|
||||
last = rules[-1]
|
||||
if "hostname" in last or last.get("service") != "http_status:404":
|
||||
bad.append("the last rule is not a hostname-less http_status:404 catch-all")
|
||||
return bad
|
||||
|
||||
|
||||
# (1)-(3) the real config
|
||||
for v in evaluate(cfg):
|
||||
check(False, f"(1-3) {v}")
|
||||
check(str(cfg.get("credentials-file", "")).startswith("/etc/cloudflared/creds/"),
|
||||
"(3) credentials-file points into the mounted Secret volume")
|
||||
|
||||
# (6) mutation self-test — the evaluator must flag each broken variant
|
||||
mutants = {}
|
||||
m = copy.deepcopy(cfg)
|
||||
m["ingress"].insert(0, {"hostname": f"k8s.{base_domain}",
|
||||
"service": "https://kubernetes.default.svc:443"})
|
||||
mutants["API rule added"] = m
|
||||
m = copy.deepcopy(cfg)
|
||||
plex = next(r for r in m["ingress"] if r.get("hostname") == f"plex.{base_domain}")
|
||||
m["ingress"].remove(plex)
|
||||
m["ingress"].insert(len(m["ingress"]) - 1, plex)
|
||||
mutants["plex moved below the wildcard"] = m
|
||||
m = copy.deepcopy(cfg)
|
||||
m["ingress"] = m["ingress"][:-1]
|
||||
mutants["catch-all removed"] = m
|
||||
for name, mut in mutants.items():
|
||||
check(evaluate(mut), f"(6) self-test: the evaluator did NOT flag the mutant '{name}' — the "
|
||||
f"check is vacuous")
|
||||
|
||||
# (4) the egress policy
|
||||
np_docs = docs(COMP / "networkpolicy.yaml")
|
||||
cnps = [d for d in np_docs if d.get("kind") == "CiliumNetworkPolicy"]
|
||||
check(len(cnps) == 1, f"(4) exactly one CiliumNetworkPolicy in networkpolicy.yaml (found {len(cnps)})")
|
||||
for cnp in cnps:
|
||||
for i, rule in enumerate(cnp["spec"].get("egress") or []):
|
||||
ents = set(rule.get("toEntities") or [])
|
||||
check(not (ents & FORBIDDEN_ENTITIES),
|
||||
f"(4) egress rule {i} admits {sorted(ents & FORBIDDEN_ENTITIES)} — that reaches the "
|
||||
f"Kubernetes API or the nodes; the API stays local-only")
|
||||
check(not (rule.get("toCIDR") or rule.get("toCIDRSet")),
|
||||
f"(4) egress rule {i} uses a CIDR — a CIDR can name the API VIP; use entities")
|
||||
for ep in rule.get("toEndpoints") or []:
|
||||
notin = [e for e in ep.get("matchExpressions") or []
|
||||
if e.get("key") == "io.kubernetes.pod.namespace" and e.get("operator") == "NotIn"]
|
||||
missing = set(EXCLUDED_NAMESPACES) - set(notin[0]["values"] if notin else [])
|
||||
check(not missing,
|
||||
f"(4) backend selector in egress rule {i} does not exclude {sorted(missing)}")
|
||||
for d in np_docs:
|
||||
if d.get("kind") == "NetworkPolicy" and d["metadata"]["name"] == "default-deny":
|
||||
check(set(d["spec"].get("policyTypes", [])) == {"Ingress", "Egress"},
|
||||
"(4) default-deny covers both directions")
|
||||
break
|
||||
else:
|
||||
check(False, "(4) networkpolicy.yaml ships no default-deny baseline for the namespace")
|
||||
|
||||
# (5) Flux isolation
|
||||
fk = next(d for d in docs(ROOT / "clusters" / "base" / "cloudflared.yaml")
|
||||
if d.get("kind") == "Kustomization")
|
||||
check(fk["spec"].get("wait") is False,
|
||||
"(5) clusters/base/cloudflared.yaml must be wait: false — its pods are legitimately not-Ready "
|
||||
"until the tunnel is enrolled")
|
||||
check(fk["spec"].get("path") == "./platform/cloudflared", "(5) the Kustomization reconciles ./platform/cloudflared")
|
||||
check("cloudflared.yaml" in (ROOT / "clusters" / "base" / "kustomization.yaml").read_text(),
|
||||
"(5) clusters/base/kustomization.yaml lists cloudflared.yaml")
|
||||
check("cloudflared" not in (ROOT / "platform" / "kustomization.yaml").read_text(),
|
||||
"(5) platform/kustomization.yaml must not include cloudflared — that is the gating tier")
|
||||
for f in sorted((ROOT / "clusters").rglob("*.yaml")):
|
||||
for d in docs(f):
|
||||
for dep in (d.get("spec") or {}).get("dependsOn") or []:
|
||||
check(not (isinstance(dep, dict) and dep.get("name") == "cloudflared"),
|
||||
f"(5) {f.relative_to(ROOT)} {d.get('metadata', {}).get('name')} dependsOn "
|
||||
f"cloudflared — that hands an unenrolled tunnel the power to stall it")
|
||||
for env in ("dev", "prod"):
|
||||
text = (ROOT / "clusters" / env / "kustomization.yaml").read_text()
|
||||
check(re.search(r"name:\s*cloudflared\s*\n\s*namespace:\s*flux-system\s*\n\s*spec:\s*\n"
|
||||
r"\s*decryption:", text) is not None,
|
||||
f"(5) clusters/{env}/kustomization.yaml carries the cloudflared SOPS decryption patch")
|
||||
|
||||
dep = next(d for d in docs(COMP / "deployment.yaml") if d.get("kind") == "Deployment")
|
||||
check(dep["spec"].get("replicas", 1) >= 2, "(5) the Deployment runs at least 2 replicas")
|
||||
check(dep["spec"]["template"]["metadata"]["labels"].get("homelab.keithjackson.dev/egress-class")
|
||||
== "internet",
|
||||
"(4) the pods are egress-class=internet — vpn would put the edge connections behind PIA")
|
||||
check(dep["spec"]["template"]["spec"].get("automountServiceAccountToken") is False,
|
||||
"(4) the pods mount no ServiceAccount token")
|
||||
|
||||
# (7) the credentials Secret is readable by the run-as identity
|
||||
def unreadable_secret_volumes(pod_spec):
|
||||
"""Return a description per Secret volume the containers mount but cannot read."""
|
||||
bad = []
|
||||
psc = pod_spec.get("securityContext") or {}
|
||||
vols = {v["name"]: v for v in pod_spec.get("volumes") or [] if "secret" in v}
|
||||
for c in (pod_spec.get("initContainers") or []) + (pod_spec.get("containers") or []):
|
||||
csc = c.get("securityContext") or {}
|
||||
uid = csc.get("runAsUser", psc.get("runAsUser"))
|
||||
gid = csc.get("runAsGroup", psc.get("runAsGroup"))
|
||||
fs_group = psc.get("fsGroup")
|
||||
groups = {g for g in [gid, fs_group, *(psc.get("supplementalGroups") or [])]
|
||||
if g is not None}
|
||||
for vm in c.get("volumeMounts") or []:
|
||||
v = vols.get(vm["name"])
|
||||
if v is None:
|
||||
continue
|
||||
sec = v["secret"]
|
||||
if uid is None:
|
||||
bad.append(f"container {c['name']} mounts Secret volume {vm['name']} but sets no "
|
||||
f"runAsUser (pod or container) — readability cannot be proven offline")
|
||||
continue
|
||||
default = sec.get("defaultMode", 0o644)
|
||||
modes = [it.get("mode", default) for it in sec.get("items") or []] or [default]
|
||||
owner_gid = fs_group if fs_group is not None else 0
|
||||
for mode in modes:
|
||||
readable = ((uid == 0 and mode & 0o400) or (owner_gid in groups and mode & 0o040)
|
||||
or mode & 0o004)
|
||||
if not readable:
|
||||
bad.append(f"container {c['name']} (uid {uid}, groups {sorted(groups)}) cannot "
|
||||
f"read Secret volume {vm['name']} ({sec.get('secretName')}): files "
|
||||
f"are 0:{owner_gid} mode {mode:04o} — set pod fsGroup to a group "
|
||||
f"the container runs with, keeping 0440")
|
||||
return bad
|
||||
|
||||
|
||||
pod_spec = dep["spec"]["template"]["spec"]
|
||||
creds_vol = [v for v in pod_spec.get("volumes") or []
|
||||
if (v.get("secret") or {}).get("secretName") == "cloudflared-credentials"]
|
||||
check(len(creds_vol) == 1, "(7) the Deployment mounts Secret cloudflared-credentials as a volume")
|
||||
for v in unreadable_secret_volumes(pod_spec):
|
||||
check(False, f"(7) {v}")
|
||||
sec_mutants = {}
|
||||
m = copy.deepcopy(pod_spec)
|
||||
m["securityContext"].pop("fsGroup", None)
|
||||
m["securityContext"].pop("supplementalGroups", None)
|
||||
sec_mutants["fsGroup removed"] = m
|
||||
m = copy.deepcopy(pod_spec)
|
||||
for v in m["volumes"]:
|
||||
if "secret" in v:
|
||||
v["secret"]["defaultMode"] = 0o400
|
||||
for it in v["secret"].get("items") or []:
|
||||
it["mode"] = 0o400
|
||||
sec_mutants["mode 0400"] = m
|
||||
m = copy.deepcopy(pod_spec)
|
||||
m["securityContext"].pop("runAsUser", None)
|
||||
for c in m["containers"]:
|
||||
(c.get("securityContext") or {}).pop("runAsUser", None)
|
||||
sec_mutants["runAsUser unset"] = m
|
||||
for name, mut in sec_mutants.items():
|
||||
check(unreadable_secret_volumes(mut),
|
||||
f"(7) self-test: the readability model did NOT flag the mutant '{name}' — vacuous")
|
||||
|
||||
if CHECKS < 25:
|
||||
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL (cloudflared-ingress-rules): {len(FAILURES)} of {CHECKS} assertions failed",
|
||||
file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"PASS (cloudflared-ingress-rules): {CHECKS} assertions")
|
||||
PY
|
||||
+6
-4
@@ -92,9 +92,10 @@
|
||||
# the NAME axis only. R-NET-04 / the edge port-forward posture owns the other.
|
||||
# - Anything about the seven non-catalog publishers' own correctness. The register records WHAT
|
||||
# each publishes so the claim is reviewable, which is all a static check can offer.
|
||||
# - Anything about the five `public` services' TARGET. They still resolve to the Gateway VIP
|
||||
# rather than a routable address; that is an open operator decision recorded in
|
||||
# platform/externaldns/helmrelease.yaml, and asserting a target here would hard-code a guess.
|
||||
# - That the public names RESOLVE to Cloudflare at runtime. check.sh (9) pins the TREE: the
|
||||
# Gateway target is the tunnel UUID in platform/cloudflared/config.yaml, proxying is on, the L4
|
||||
# `service` publishers opt out, and every name the tunnel 404s is unpublished. Whether Cloudflare
|
||||
# then answers anycast for them is a live read (`dig @1.1.1.1`), not a static one.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
@@ -104,8 +105,9 @@ metadata:
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-02: "true"
|
||||
conformance.home.k3s/R-DNS-03: "true"
|
||||
conformance.home.k3s/R-DNS-05: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03"
|
||||
conformance.home.k3s/requirements: "R-DNS-02, R-DNS-03, R-DNS-05"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE — this invariant quantifies over every ServiceDefinition in catalog/, every
|
||||
# HTTPRoute in apps/wave-* and platform/* and runbooks/, and every external-dns-annotated
|
||||
|
||||
@@ -29,7 +29,17 @@
|
||||
# ci/l0.sh stage (whole-repo-invariants-wired fails the gate if one appears);
|
||||
# (8) the external-dns HelmRelease declares `annotationPrefix` explicitly and it matches the
|
||||
# prefix of EVERY external-dns annotation in the tree and in catalog/render.py — added after
|
||||
# the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime.
|
||||
# the 2026-11-07 v0.23.0 default-prefix change silently voided (1)-(3) at runtime;
|
||||
# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08) — every public name is a PROXIED CNAME to the tunnel:
|
||||
# (a) the shared Gateway's `target` is EXACTLY `<tunnel:>.cfargotunnel.com`, the UUID read from
|
||||
# platform/cloudflared/config.yaml, and the HelmRelease sets `--cloudflare-proxied`;
|
||||
# (b) every `service`-source publisher carries `cloudflare-proxied: "false"` (L4 A records stay
|
||||
# DNS-only), and no HTTPRoute opts out of proxying (a DNS-only tunnel CNAME is dead);
|
||||
# (c) no hostname the tunnel maps to `http_status:404` has a public record — no undisowned route
|
||||
# names it, and the only `service`-source publishers of one are the declared L4 DNS-only
|
||||
# names (unifi, crafty-mc); conversely every disowned route's hostname IS 404'd by the
|
||||
# tunnel, so the "unpublished" and "refused at the tunnel" lists cannot drift apart;
|
||||
# (d) the operator's LAN/VPN-only names (plex, headlamp) are disowned everywhere they appear.
|
||||
#
|
||||
# WHAT THIS DOES *NOT* PROVE (honest gaps — it is STATIC, offline, cluster-free):
|
||||
# - That the record is gone from Cloudflare. Live read; belongs beside
|
||||
@@ -39,8 +49,11 @@
|
||||
# both the A and its `a-` TXT within ~60s) — but a manual proof is not a gate.
|
||||
# - That an internal name is UNREACHABLE. A port-forward plus a literal IP needs no DNS record.
|
||||
# This check owns the NAME axis only.
|
||||
# - Anything about the five public services' TARGET. They still answer with the Gateway VIP; the
|
||||
# routable-target decision is open and recorded in platform/externaldns/helmrelease.yaml.
|
||||
# - That the tunnel actually SERVES the public names, or that Cloudflare honoured `proxied`. (9)
|
||||
# proves the tree asks for a proxied CNAME to the right tunnel; the live answer (`dig @1.1.1.1`
|
||||
# returning Cloudflare anycast) is a runtime read. Nor does (8)/(9) prove the RUNNING controller
|
||||
# honours the prefix — a future chart bump that drops it would publish PROXIED A records at the
|
||||
# Gateway VIP; (8) only keeps the tree and the HelmRelease value consistent.
|
||||
# - It does not read ../home-network-k3s. The spec is read-only (R-TEST-06).
|
||||
#
|
||||
# Exit 0 = all assertions pass. Exit 1 = a failure. Exit 3 (honest SKIP, which the runner turns
|
||||
@@ -141,9 +154,12 @@ def docs_of(p: Path):
|
||||
# unadjudicated set cannot grow without a review event. Same ratchet reasoning as
|
||||
# KNOWN_EXTERNAL_MONITORS in servicemonitor-ports-resolve.
|
||||
#
|
||||
# EVERY ENTRY HERE IS A NAME THAT STILL ANSWERS WITH AN RFC1918 ADDRESS IN A PUBLIC ZONE. That is
|
||||
# the honest state after this change, and it is deliberate: converting them is an operator
|
||||
# decision per name (is grafana meant to be publicly resolvable at all?), not a renderer change,
|
||||
# WHAT THE ENTRIES PUBLISH NOW (2026-11-08). Every HTTPRoute entry that is not disowned publishes a
|
||||
# PROXIED CNAME to the Cloudflare Tunnel (the shared Gateway's `target`, assertion 9a), so no origin
|
||||
# address of ours is in the public zone for it. Headlamp is disowned (operator: LAN/VPN-only, 9d).
|
||||
# The two `service`-source entries still publish their RFC1918 LB VIP as a DNS-only A record —
|
||||
# deliberate (L4 ports cannot ride Cloudflare's HTTP proxy), and pinned DNS-only by 9b. Whether each
|
||||
# remaining name should be public at all stays an operator decision per name, not a renderer change,
|
||||
# and a cluster-wide ExternalDNS opt-in filter — the available hardening, see
|
||||
# platform/externaldns/helmrelease.yaml — would DELETE all of them at once the moment it is set.
|
||||
#
|
||||
@@ -151,27 +167,28 @@ def docs_of(p: Path):
|
||||
# =============================================================================================
|
||||
KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
|
||||
"platform/oauth2-proxy/httproute.yaml:oauth2-proxy/oauth2-proxy":
|
||||
"publishes auth.${BASE_DOMAIN} -> the Gateway VIP. The oauth2-proxy login endpoint is a "
|
||||
"publishes auth.${BASE_DOMAIN} -> the tunnel (proxied CNAME). The oauth2-proxy login endpoint is a "
|
||||
"platform component, not a catalog service; it must be reachable by every browser that "
|
||||
"any forward-auth app redirects, so whether it is PUBLICLY resolvable is coupled to "
|
||||
"whether the five public services are — an operator decision, not a renderer one.",
|
||||
"platform/keycloak/httproute.yaml:keycloak/keycloak":
|
||||
"publishes keycloak.${BASE_DOMAIN} -> the Gateway VIP, and it is the route that actually "
|
||||
"publishes keycloak.${BASE_DOMAIN} -> the tunnel (proxied CNAME), and it is the route that actually "
|
||||
"OWNS that record (the TXT registry entry reads resource=httproute/keycloak/keycloak, not "
|
||||
"the catalog-rendered keycloak-8080 route, because external-dns merges duplicate "
|
||||
"hostnames and this one won). Keycloak is `exposure: public` in the catalog, so the name "
|
||||
"is meant to be public either way; it is listed because the PUBLISHING object is the "
|
||||
"hand-authored platform route, which carries no exposure field.",
|
||||
"platform/monitoring/grafana-httproute.yaml:monitoring/grafana":
|
||||
"publishes grafana.${BASE_DOMAIN} -> the Gateway VIP. An admin dashboard; almost "
|
||||
"certainly should not be publicly resolvable, but it is not a catalog service and "
|
||||
"withdrawing it is an operator call with the same LAN-resolution prerequisite as the six "
|
||||
"internal services (Pi-hole must answer first).",
|
||||
"publishes grafana.${BASE_DOMAIN} -> the tunnel (proxied CNAME). An admin dashboard "
|
||||
"behind Keycloak; it is not a catalog service, so withdrawing it is an operator call per "
|
||||
"name (it stayed public in the 2026-11-08 tunnel decision).",
|
||||
"platform/headlamp/httproute.yaml:headlamp/headlamp":
|
||||
"publishes headlamp.${BASE_DOMAIN} -> the Gateway VIP. A Kubernetes web UI with native "
|
||||
"Keycloak OIDC; same adjudication as grafana.",
|
||||
"publishes NOTHING since 2026-11-08: the route carries `controller: none` (operator "
|
||||
"decision: Headlamp is LAN/VPN-only) and the tunnel 404s the name. A Kubernetes web UI "
|
||||
"with native Keycloak OIDC; listed because it is a hand-authored route, and pinned "
|
||||
"disowned by (9d).",
|
||||
"platform/homepage/httproute.yaml:homepage/homepage":
|
||||
"publishes the APEX ${BASE_DOMAIN} itself -> the Gateway VIP, via the https-apex "
|
||||
"publishes the APEX ${BASE_DOMAIN} itself -> the tunnel (proxied CNAME), via the https-apex "
|
||||
"listener. The apex is not a per-service name and cannot be withdrawn by an exposure "
|
||||
"declaration.",
|
||||
"runbooks/serving/httproute.yaml:runbooks/runbooks":
|
||||
@@ -180,16 +197,16 @@ KNOWN_NON_CATALOG_PUBLISHERS: dict[str, str] = {
|
||||
"entry names the homepage route). Deliberately un-authenticated — see the manifest's own "
|
||||
"header for why gating runbooks on Keycloak is a loop with no exit.",
|
||||
"platform/crafty/httproute.yaml:crafty/crafty":
|
||||
"publishes crafty.${BASE_DOMAIN} -> the Gateway VIP (the Minecraft admin UI, gated by an "
|
||||
"publishes crafty.${BASE_DOMAIN} -> the tunnel (proxied CNAME) (the Minecraft admin UI, gated by an "
|
||||
"oauth2-proxy sidecar). Platform component, no exposure field.",
|
||||
"platform/crafty/service-l4.yaml:crafty/crafty-l4-tcp":
|
||||
"the `service` SOURCE, not a route: publishes crafty-mc.${BASE_DOMAIN} -> "
|
||||
"${CRAFTY_L4_VIP} (192.168.2.22) for the raw Minecraft TCP/UDP game ports the HTTPS "
|
||||
"Gateway cannot express. A game server is the one case where a public name may be "
|
||||
"genuinely wanted; the address it answers with is still RFC1918.",
|
||||
"genuinely wanted; the address it answers with is still RFC1918, DNS-only (9b).",
|
||||
"apps/vms/unifi/50-service-l4.yaml:unifi/unifi-l4-tcp":
|
||||
"the `service` source: publishes unifi.${BASE_DOMAIN} -> ${UNIFI_L4_VIP} "
|
||||
"(192.168.2.21) for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
|
||||
"(192.168.2.21), DNS-only (9b), for the UniFi controller VM's inform/STUN/adoption ports. Hand-authored "
|
||||
"under apps/vms (a VM, not a rendered container service); the catalog's only VM entry is "
|
||||
"vm-demo, which declares no ingressRoutes and publishes nothing.",
|
||||
}
|
||||
@@ -515,6 +532,113 @@ check(not bad_render,
|
||||
f"(8d) catalog/render.py emits external-dns keys under {bad_render}, but the controller "
|
||||
f"reads {ANNOTATION_PREFIX!r}")
|
||||
|
||||
# =============================================================================================
|
||||
# (9) THE CLOUDFLARE TUNNEL WIRING (2026-11-08). The public names are PROXIED CNAMEs to the tunnel.
|
||||
# =============================================================================================
|
||||
CF_CONFIG = ROOT / "platform/cloudflared/config.yaml"
|
||||
GATEWAY_FILE = ROOT / "platform/gateway/gateway.yaml"
|
||||
TARGET_KEY = ANNOTATION_PREFIX + "target"
|
||||
PROXIED_KEY = ANNOTATION_PREFIX + "cloudflare-proxied"
|
||||
UUID_RE = re.compile(r"^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$")
|
||||
|
||||
# The L4 names the tunnel 404s that are NONETHELESS published — as DNS-only A records from the
|
||||
# `service` source, at their own LB VIP, because they are not HTTP and never touch the tunnel
|
||||
# (operator decision 2026-11-08). Shrink-only, and must agree with the tunnel's 404 list (9c).
|
||||
L4_DNS_ONLY_NAMES = {"unifi.${BASE_DOMAIN}", "crafty-mc.${BASE_DOMAIN}"}
|
||||
# The operator's LAN/VPN-only HTTP names (2026-11-08). Pinned here as well as in the tunnel config, so
|
||||
# deleting one from config.yaml cannot quietly also release it from (9c).
|
||||
OPERATOR_LAN_ONLY_NAMES = {"plex.${BASE_DOMAIN}", "headlamp.${BASE_DOMAIN}"}
|
||||
|
||||
cf = next(iter(docs_of(CF_CONFIG)), None)
|
||||
check(cf is not None and isinstance(cf.get("ingress"), list),
|
||||
f"(9) {CF_CONFIG.relative_to(ROOT)} unreadable or has no `ingress` list — cannot prove what "
|
||||
f"the tunnel refuses or which tunnel the Gateway must target")
|
||||
cf = cf or {}
|
||||
tunnel = str(cf.get("tunnel") or "")
|
||||
check(bool(UUID_RE.match(tunnel)),
|
||||
f"(9a) platform/cloudflared/config.yaml `tunnel:` is {tunnel!r}, not a literal tunnel UUID — "
|
||||
f"the Gateway target is derived from it")
|
||||
gw = next((d for d in docs_of(GATEWAY_FILE) if d.get("kind") == "Gateway"
|
||||
and ident(d) == ("gateway", "shared")), None)
|
||||
check(gw is not None, f"(9a) {GATEWAY_FILE.relative_to(ROOT)} holds no Gateway gateway/shared")
|
||||
want_target = f"{tunnel}.cfargotunnel.com"
|
||||
got_target = annotations_of(gw or {}).get(TARGET_KEY)
|
||||
check(got_target == want_target,
|
||||
f"(9a) Gateway gateway/shared `{TARGET_KEY}` is {got_target!r}; it must be EXACTLY "
|
||||
f"{want_target!r} (the tunnel in platform/cloudflared/config.yaml). Any other value publishes "
|
||||
f"every public name at the wrong place — a dead cfargotunnel.com name (Cloudflare 1016), or "
|
||||
f"an origin address of ours.")
|
||||
check("--cloudflare-proxied" in [str(a) for a in (extra if isinstance(extra, list) else [])]
|
||||
or (isinstance(extra, dict) and str(extra.get("cloudflare-proxied", "")).lower() == "true"),
|
||||
"(9a) external-dns HelmRelease extraArgs must include `--cloudflare-proxied`: a DNS-only "
|
||||
"CNAME to <uuid>.cfargotunnel.com resolves to nothing routable, so every public name is dead")
|
||||
|
||||
for rel, d in dns_services:
|
||||
if is_disowned(d):
|
||||
continue
|
||||
if HOSTNAME_KEY not in annotations_of(d):
|
||||
continue
|
||||
v = annotations_of(d).get(PROXIED_KEY)
|
||||
check(str(v).lower() == "false",
|
||||
f"(9b) {rel} Service {'/'.join(ident(d))} publishes via the `service` source but carries "
|
||||
f"`{PROXIED_KEY}: {v!r}`, not \"false\". With --cloudflare-proxied global it would become "
|
||||
f"an orange-clouded A record, which cannot carry L4 traffic and hides nothing (the LB VIP "
|
||||
f"is RFC1918). Every `service`-source publisher must opt out explicitly.")
|
||||
for rel, d in routes:
|
||||
v = annotations_of(d).get(PROXIED_KEY)
|
||||
check(v is None or str(v).lower() == "true",
|
||||
f"(9b) {rel} HTTPRoute {'/'.join(ident(d))} sets `{PROXIED_KEY}: {v!r}`. A gateway-sourced "
|
||||
f"record is a CNAME to the tunnel; DNS-only, it is dead. Disown the route instead "
|
||||
f"(`{CONTROLLER_KEY}: none`) if the name should not be public.")
|
||||
|
||||
refused = {str(r.get("hostname")) for r in cf.get("ingress") or []
|
||||
if isinstance(r, dict) and r.get("hostname") and r.get("service") == "http_status:404"}
|
||||
check(len(refused) > 0, "(9c) the tunnel config 404s no hostname — parse contract broken (the "
|
||||
"operator's LAN/VPN-only list is known to be non-empty)")
|
||||
published_l4: set[str] = set()
|
||||
for rel, d in routes:
|
||||
hosts = {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}
|
||||
leaked = sorted(hosts & refused)
|
||||
if is_disowned(d):
|
||||
unrefused = sorted(hosts - refused)
|
||||
check(not unrefused,
|
||||
f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} is disowned (no public record) but the "
|
||||
f"tunnel does not 404 {unrefused}: add them to platform/cloudflared/config.yaml above "
|
||||
f"the wildcard, or the two halves of \"LAN/VPN-only\" have drifted apart.")
|
||||
continue
|
||||
check(not leaked,
|
||||
f"(9c) {rel} HTTPRoute {'/'.join(ident(d))} PUBLISHES {leaked}, which the tunnel maps to "
|
||||
f"http_status:404 (LAN/VPN-only). The name would be a public proxied record that serves "
|
||||
f"only 404s — disown the route (`{CONTROLLER_KEY}: none`) or adjudicate the name.")
|
||||
for rel, d in dns_services:
|
||||
if is_disowned(d):
|
||||
continue
|
||||
hosts = {h.strip() for h in str(annotations_of(d).get(HOSTNAME_KEY) or "").split(",")
|
||||
if h.strip()}
|
||||
for h in sorted(hosts & refused):
|
||||
published_l4.add(h)
|
||||
check(h in L4_DNS_ONLY_NAMES,
|
||||
f"(9c) {rel} Service {'/'.join(ident(d))} publishes {h}, which the tunnel 404s, and it "
|
||||
f"is not a declared L4 DNS-only name {sorted(L4_DNS_ONLY_NAMES)}")
|
||||
check(L4_DNS_ONLY_NAMES <= refused,
|
||||
f"(9c) L4 DNS-only names {sorted(L4_DNS_ONLY_NAMES - refused)} are not 404'd by the tunnel; a "
|
||||
f"stray CNAME to the tunnel for them would hit the Gateway's catch-all instead of a 404")
|
||||
check(published_l4 == L4_DNS_ONLY_NAMES,
|
||||
f"(9c) L4_DNS_ONLY_NAMES is shrink-only: {sorted(L4_DNS_ONLY_NAMES - published_l4)} no longer "
|
||||
f"published by any `service`-source publisher — delete them from the list")
|
||||
|
||||
check(OPERATOR_LAN_ONLY_NAMES <= refused,
|
||||
f"(9d) operator LAN/VPN-only names {sorted(OPERATOR_LAN_ONLY_NAMES - refused)} are missing "
|
||||
f"from the tunnel's http_status:404 rules (platform/cloudflared/config.yaml)")
|
||||
for name in sorted(OPERATOR_LAN_ONLY_NAMES):
|
||||
naming = [(rel, d) for rel, d in routes
|
||||
if name in {str(h) for h in (d.get("spec") or {}).get("hostnames") or []}]
|
||||
check(len(naming) > 0, f"(9d) no HTTPRoute names {name} — stale OPERATOR_LAN_ONLY_NAMES entry")
|
||||
for rel, d in naming:
|
||||
check(is_disowned(d),
|
||||
f"(9d) {rel} HTTPRoute {'/'.join(ident(d))} names LAN/VPN-only {name} but is not "
|
||||
f"disowned, so external-dns publishes it publicly (operator decision 2026-11-08)")
|
||||
|
||||
# =============================================================================================
|
||||
if FAILURES:
|
||||
for f in FAILURES:
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — PiHole's DNS frontends admit the LAN and the VPN, and nothing public
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# loadBalancerSourceRanges is the only thing keeping PiHole from being an open resolver, and it
|
||||
# must also carry the WireGuard client subnet, because Cilium load-balances tunnel traffic on wg0
|
||||
# before the iptables MASQUERADE (so the source it tests is still 10.90.0.x). check.sh's header
|
||||
# states the assertions and what they do not prove.
|
||||
#
|
||||
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (standard library + PyYAML);
|
||||
# conformance/run.sh --layer l0 runs it directly and this Chainsaw Test wraps the SAME script.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: pihole-dns-source-ranges
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-03: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-03"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
# WHOLE-REPO SCOPE: ci/touched-requirements.sh maps apps/* to no R-DNS requirement, so a change
|
||||
# to these Services would otherwise select nothing.
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "false"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
Every PiHole LoadBalancer Service lists both the LAN and the WireGuard client subnet in
|
||||
loadBalancerSourceRanges, and every range resolves to RFC1918, so VPN clients can resolve
|
||||
and PiHole stays unreachable from the internet.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-pihole-source-ranges
|
||||
description: >-
|
||||
Collect the PiHole LoadBalancer Services, resolve each source range through cluster-vars,
|
||||
and assert both required ranges are present and every range is private.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
@@ -0,0 +1,146 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — pihole-dns-source-ranges
|
||||
# Serves: R-DNS-03 (LAN split-horizon DNS — spec docs/09).
|
||||
#
|
||||
# THE BUG CLASS THIS CLOSES. PiHole's LoadBalancer frontends are the only thing stopping it being
|
||||
# an open resolver, and their `loadBalancerSourceRanges` is pulled in two opposite directions:
|
||||
# * Too WIDE, and the resolver is open: a public range, or 0.0.0.0/0, in this list is the whole
|
||||
# restriction gone, with nothing else in the path to catch it (the CiliumNetworkPolicy admits
|
||||
# `world` on :53, because that is the only peer that can describe post-LoadBalancer traffic).
|
||||
# * Too NARROW, and VPN DNS dies: wg0 is a Cilium kube-proxy-replacement device, so a tunnel
|
||||
# client's query is load-balanced with its 10.90.0.x source intact, ahead of the iptables
|
||||
# MASQUERADE. A LAN-only list is predicted to drop it as DROP_NOT_IN_SRC_RANGE, and the
|
||||
# tunnel would still report a healthy handshake.
|
||||
#
|
||||
# GIVEN the repository as git tracks it,
|
||||
# WHEN every type=LoadBalancer Service in namespace `pihole` is collected and each source range
|
||||
# is resolved through cluster-vars (base value AND the prod overlay's replacement),
|
||||
# THEN (1) the frontends exist and cover both udp/53 and tcp/53;
|
||||
# (2) every frontend lists ${LAN_CIDR} and ${WG_CLIENT_CIDR};
|
||||
# (3) every range resolves to a valid IPv4 CIDR inside RFC1918 — so no 0.0.0.0/0, no public
|
||||
# range, and no unresolvable token that could hide one;
|
||||
# AND (4) the check is NON-VACUOUS: a real checkout and a floor on the assertion count.
|
||||
#
|
||||
# WHAT THIS DOES NOT PROVE: that a query over the tunnel is actually answered. That needs a
|
||||
# handshake from a real client; see platform/vpn-ingress/README.md.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
PYTHON="${PYTHON:-python3}"
|
||||
|
||||
cd "${ROOT}"
|
||||
"${PYTHON}" - "${ROOT}" <<'PY'
|
||||
import ipaddress, re, subprocess, sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("check FAIL: PyYAML is required (same dependency as catalog/render.py)",
|
||||
file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES: list[str] = []
|
||||
CHECKS = 0
|
||||
REQUIRED_TOKENS = ("LAN_CIDR", "WG_CLIENT_CIDR")
|
||||
RFC1918 = [ipaddress.ip_network(n) for n in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16")]
|
||||
TOKEN = re.compile(r"^\$\{([A-Z0-9_]+)\}$")
|
||||
|
||||
|
||||
def check(cond, msg):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(msg)
|
||||
|
||||
|
||||
def docs_of(rel: str):
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all((ROOT / rel).read_text(errors="replace"))
|
||||
if isinstance(d, dict)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
FILES = [p for p in subprocess.run(["git", "-C", str(ROOT), "ls-files", "-z"],
|
||||
capture_output=True, text=True, check=True).stdout.split("\0")
|
||||
if p]
|
||||
YAML_FILES = [f for f in FILES if re.search(r"\.ya?ml$", f)]
|
||||
check(len(YAML_FILES) > 50, f"only {len(YAML_FILES)} YAML files seen — not a real checkout")
|
||||
|
||||
# ------------------------------------------------- every value a ${TOKEN} can take on this tree
|
||||
values: dict[str, set[str]] = {}
|
||||
for d in docs_of("clusters/base/cluster-vars.yaml"):
|
||||
if d.get("kind") == "ConfigMap":
|
||||
for k, v in (d.get("data") or {}).items():
|
||||
values.setdefault(k, set()).add(str(v))
|
||||
for d in docs_of("clusters/prod/kustomization.yaml"):
|
||||
for p in d.get("patches") or []:
|
||||
ops = p.get("patch") if isinstance(p, dict) else None
|
||||
ops = yaml.safe_load(ops) if isinstance(ops, str) else ops
|
||||
for op in ops if isinstance(ops, list) else []:
|
||||
m = re.fullmatch(r"/data/([A-Z0-9_]+)", str(op.get("path", "")))
|
||||
if m and op.get("op") in ("add", "replace"):
|
||||
values.setdefault(m.group(1), set()).add(str(op.get("value")))
|
||||
for tok in REQUIRED_TOKENS:
|
||||
check(values.get(tok), f"clusters/base/cluster-vars.yaml defines no {tok} (R-DNS-03)")
|
||||
|
||||
|
||||
def assert_private(where: str, cidr: str):
|
||||
try:
|
||||
net = ipaddress.ip_network(cidr, strict=False)
|
||||
except ValueError:
|
||||
check(False, f"{where}: {cidr!r} is not a valid CIDR (R-DNS-03)")
|
||||
return
|
||||
check(net.version == 4 and any(net.subnet_of(r) for r in RFC1918),
|
||||
f"{where}: {cidr} is not inside RFC1918, so it opens PiHole beyond the LAN and the VPN "
|
||||
"(R-DNS-03)")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------- the frontends
|
||||
frontends = []
|
||||
for rel in YAML_FILES:
|
||||
for d in docs_of(rel):
|
||||
md, spec = d.get("metadata") or {}, d.get("spec") or {}
|
||||
if d.get("kind") == "Service" and md.get("namespace") == "pihole" \
|
||||
and spec.get("type") == "LoadBalancer":
|
||||
frontends.append((rel, md.get("name"), spec))
|
||||
|
||||
protocols = {(p.get("protocol", "TCP"), p.get("port"))
|
||||
for _, _, spec in frontends for p in spec.get("ports") or []}
|
||||
for want in (("UDP", 53), ("TCP", 53)):
|
||||
check(want in protocols, f"no PiHole LoadBalancer Service serves {want[0]}/{want[1]} — this "
|
||||
"check's corpus has moved and must be re-derived (R-DNS-03)")
|
||||
|
||||
for rel, name, spec in frontends:
|
||||
where = f"{rel}: Service/{name}"
|
||||
ranges = spec.get("loadBalancerSourceRanges") or []
|
||||
check(ranges, f"{where} has no loadBalancerSourceRanges, so it answers anyone who can reach "
|
||||
"the VIP (R-DNS-03)")
|
||||
tokens = {m.group(1) for m in (TOKEN.match(str(r)) for r in ranges) if m}
|
||||
for tok in REQUIRED_TOKENS:
|
||||
check(tok in tokens, f"{where} does not list ${{{tok}}} in loadBalancerSourceRanges "
|
||||
"(R-DNS-03)")
|
||||
for r in ranges:
|
||||
m = TOKEN.match(str(r))
|
||||
if not m:
|
||||
assert_private(where, str(r))
|
||||
continue
|
||||
resolved = values.get(m.group(1))
|
||||
check(resolved, f"{where}: {r} is not defined in cluster-vars, so its value cannot be "
|
||||
"shown to be private (R-DNS-03)")
|
||||
for v in sorted(resolved or []):
|
||||
assert_private(f"{where} ({r})", v)
|
||||
|
||||
check(CHECKS >= 15, f"only {CHECKS} assertions ran — this check has gone vacuous (R-DNS-03)")
|
||||
|
||||
if FAILURES:
|
||||
for f in FAILURES:
|
||||
print(f"check FAIL: {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"check OK: pihole-dns-source-ranges — {CHECKS} assertions over "
|
||||
f"{len(frontends)} PiHole LoadBalancer Service(s)")
|
||||
PY
|
||||
@@ -0,0 +1,46 @@
|
||||
# =============================================================================
|
||||
# L0 STATIC — PiHole answers an explicit LAN name list, and no LAN-only name is left out of it
|
||||
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
||||
#
|
||||
# Since PR #156 every public name is a proxied CNAME to the Cloudflare Tunnel and works from the
|
||||
# LAN, so PiHole's zone wildcard was replaced by an explicit list (catalog/pihole.yaml). That makes
|
||||
# the list load-bearing: a route disowned from public DNS and absent from it resolves nowhere. This
|
||||
# test derives the required names from the tree (every `controller: none` HTTPRoute, every hostname
|
||||
# the tunnel 404s, the L4 Services' own VIPs) and asserts the list covers them, carries no
|
||||
# zone-wide entry, and pins gitea and nextcloud to the Gateway VIP. It proves itself non-vacuous by
|
||||
# mutation. Rationale and the honest NOT-proven list: check.sh.
|
||||
#
|
||||
# WHY SCOPE, NOT REQUIREMENTS: the property spans catalog/, apps/, platform/ and runbooks/; a new
|
||||
# internal service in any of them can break it, and ci/touched-requirements.sh maps apps/* to no
|
||||
# requirement at all, so a requirement-scoped selection would skip it on precisely those changes.
|
||||
# =============================================================================
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: pihole-lan-names
|
||||
labels:
|
||||
conformance.home.k3s/R-DNS-03: "true"
|
||||
conformance.home.k3s/R-DNS-04: "true"
|
||||
annotations:
|
||||
conformance.home.k3s/requirements: "R-DNS-03, R-DNS-04"
|
||||
conformance.home.k3s/layer: "L0"
|
||||
conformance.home.k3s/scope: "whole-repo"
|
||||
conformance.home.k3s/safety-critical: "true"
|
||||
conformance.home.k3s/executable: "true"
|
||||
conformance.home.k3s/intent: >-
|
||||
PiHole answers an explicit list of LAN names rather than the whole zone; every HTTPRoute
|
||||
disowned from public DNS has an entry to the Gateway VIP; every hostname the Cloudflare
|
||||
Tunnel 404s has an entry, with the L4 names on their own VIPs; and gitea and nextcloud stay
|
||||
on the LAN path, off Cloudflare's request-size and timeout limits.
|
||||
spec:
|
||||
steps:
|
||||
- name: assert-pihole-explicit-lan-names
|
||||
description: >-
|
||||
Parse FTLCONF_misc_dnsmasq_lines from catalog/pihole.yaml and the rendered
|
||||
apps/wave-0/pihole.yaml, derive the required names from every `controller: none` HTTPRoute,
|
||||
the http_status:404 hostnames in platform/cloudflared/config.yaml and the L4 Services'
|
||||
lbipam annotations, assert the list covers them and contains no zone-wide entry, and run
|
||||
the evaluator on mutated lists to prove it is not vacuous.
|
||||
try:
|
||||
- script:
|
||||
content: bash ./check.sh
|
||||
+269
@@ -0,0 +1,269 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# L0 static check (offline, no cluster) — pihole-lan-names
|
||||
# Serves: R-DNS-03 (LAN split-horizon answers), and decisions/2026-11-09-pihole-explicit-lan-names.md.
|
||||
#
|
||||
# WHY THIS EXISTS. Since PR #156 every PUBLIC name is a proxied CNAME to the Cloudflare Tunnel and
|
||||
# works from the LAN, so PiHole no longer answers the whole zone with a wildcard: it answers an
|
||||
# EXPLICIT list (catalog/pihole.yaml, FTLCONF_misc_dnsmasq_lines) and forwards everything else to
|
||||
# the public resolvers. That inverts the old failure mode. With the wildcard, a forgotten name still
|
||||
# resolved; now a forgotten LAN-ONLY name — a route disowned from public DNS, which the tunnel also
|
||||
# 404s — resolves NOWHERE, and the app is silently unreachable from every client. Nothing in the
|
||||
# cluster turns red when that happens. So the list is derived from the tree and asserted here.
|
||||
#
|
||||
# GIVEN the repository as authored,
|
||||
# THEN (0) the dnsmasq value parses (only `local-ttl=` and `address=/<domain>/<${VAR}>` entries,
|
||||
# every VAR is a cluster-var), the catalog value and the rendered apps/wave-0/pihole.yaml
|
||||
# value are identical, and the inventories below are non-empty (anti-vacuous-pass);
|
||||
# (a) NO entry covers the whole zone: no `address=/${BASE_DOMAIN}/`, no `address=/#/`, and no
|
||||
# entry for any parent of the zone (dnsmasq matches every subdomain of an entry);
|
||||
# (b) every HTTPRoute hostname disowned from public DNS
|
||||
# (`external-dns.alpha.kubernetes.io/controller: none`, anywhere in apps/ platform/
|
||||
# runbooks/) has an explicit entry to ${GATEWAY_VIP} — a disowned route with no
|
||||
# `hostnames:` is itself a FAIL, because it cannot be checked;
|
||||
# (c) every hostname platform/cloudflared/config.yaml maps to http_status:404 has an entry;
|
||||
# the L4 DNS-only names (derived: Services carrying both
|
||||
# `external-dns.alpha.kubernetes.io/hostname` and `lbipam.cilium.io/ips` — unifi,
|
||||
# crafty-mc) must map to THAT Service's VIP, not the Gateway;
|
||||
# (d) gitea and nextcloud are pinned to ${GATEWAY_VIP} (Cloudflare's 100 MB body / 100 s
|
||||
# limits break git push/LFS and large uploads);
|
||||
# AND (e) the evaluator is not vacuous: it is run against mutated copies of the real entry list
|
||||
# (zone wildcard added; a disowned host dropped; a 404'd host dropped; gitea repointed;
|
||||
# unifi repointed to the Gateway) and must flag each one.
|
||||
#
|
||||
# NOT proven (static): that the running PiHole pods carry this value (FTL applies FTLCONF_* at
|
||||
# start; the env change rolls the StatefulSet), that public resolution of the unlisted names
|
||||
# reaches Cloudflare, or the substituted IPs. Runtime check: `dig @${PIHOLE_DNS_VIP} <name>`.
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "${DIR}/../../../.." && pwd)" # repo root
|
||||
exec python3 - "$ROOT" <<'PY'
|
||||
import copy
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import yaml
|
||||
except ImportError: # pragma: no cover
|
||||
print("CHECK FAIL (pihole-lan-names): PyYAML is required", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
ROOT = Path(sys.argv[1])
|
||||
FAILURES, CHECKS = [], 0
|
||||
NAME = "pihole-lan-names"
|
||||
ZONE = "${BASE_DOMAIN}"
|
||||
GW = "${GATEWAY_VIP}"
|
||||
DISOWN = "external-dns.alpha.kubernetes.io/controller"
|
||||
EDNS_HOST = "external-dns.alpha.kubernetes.io/hostname"
|
||||
LB_IPS = "lbipam.cilium.io/ips"
|
||||
# Public, but must bypass Cloudflare's 100 MB request body / 100 s response limits.
|
||||
LAN_PINNED = ("gitea", "nextcloud")
|
||||
# Trees that hold applied manifests. conformance/ fixtures and .claude/ worktrees are not authored
|
||||
# surface (CLAUDE.md §0); decisions/ is prose.
|
||||
SCAN = ("apps", "platform", "runbooks")
|
||||
|
||||
|
||||
def check(cond, desc):
|
||||
global CHECKS
|
||||
CHECKS += 1
|
||||
if not cond:
|
||||
FAILURES.append(desc)
|
||||
|
||||
|
||||
def docs(path):
|
||||
try:
|
||||
return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
|
||||
except yaml.YAMLError:
|
||||
return []
|
||||
|
||||
|
||||
def dnsmasq_value(objs):
|
||||
"""The FTLCONF_misc_dnsmasq_lines value from a list of docs (a ServiceDefinition or manifests)."""
|
||||
found = []
|
||||
for d in objs:
|
||||
envs = list(d.get("env") or [])
|
||||
tmpl = (((d.get("spec") or {}).get("template") or {}).get("spec") or {})
|
||||
for c in tmpl.get("containers") or []:
|
||||
envs += c.get("env") or []
|
||||
found += [e.get("value") for e in envs if e.get("name") == "FTLCONF_misc_dnsmasq_lines"]
|
||||
return found
|
||||
|
||||
|
||||
def die(msg):
|
||||
print(f"CHECK FAIL ({NAME}): {msg}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
CATALOG = ROOT / "catalog" / "pihole.yaml"
|
||||
RENDERED = ROOT / "apps" / "wave-0" / "pihole.yaml"
|
||||
CF = ROOT / "platform" / "cloudflared" / "config.yaml"
|
||||
for p in (CATALOG, RENDERED, CF):
|
||||
if not p.is_file():
|
||||
die(f"{p.relative_to(ROOT)} is absent — the surface this check guards moved; re-point or "
|
||||
"delete this check in the SAME change")
|
||||
|
||||
cat_vals = dnsmasq_value(docs(CATALOG))
|
||||
ren_vals = dnsmasq_value(docs(RENDERED))
|
||||
check(len(cat_vals) == 1, f"(0) catalog/pihole.yaml declares FTLCONF_misc_dnsmasq_lines exactly "
|
||||
f"once (found {len(cat_vals)})")
|
||||
check(len(ren_vals) == 1, f"(0) apps/wave-0/pihole.yaml carries FTLCONF_misc_dnsmasq_lines exactly "
|
||||
f"once (found {len(ren_vals)})")
|
||||
if not cat_vals:
|
||||
die("no dnsmasq value to check")
|
||||
check(cat_vals[:1] == ren_vals[:1],
|
||||
"(0) the rendered apps/wave-0/pihole.yaml dnsmasq value differs from catalog/pihole.yaml — "
|
||||
"re-render pihole (catalog/render.py --no-image-automation catalog/pihole.yaml)")
|
||||
|
||||
cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
|
||||
if d.get("kind") == "ConfigMap")
|
||||
CLUSTER_VARS = set(cv["data"])
|
||||
BASE_DOMAIN_VALUE = cv["data"]["BASE_DOMAIN"]
|
||||
|
||||
ENTRY = re.compile(r"^address=/([^/]+)/(\$\{[A-Z0-9_]+\})$")
|
||||
|
||||
|
||||
def parse(value):
|
||||
"""Return (entries {domain: target}, problems)."""
|
||||
entries, problems = {}, []
|
||||
for raw in value.split(";"):
|
||||
line = raw.strip()
|
||||
if raw != line:
|
||||
problems.append(f"entry {raw!r} has surrounding whitespace (FTL writes it verbatim)")
|
||||
if not line:
|
||||
problems.append("empty entry (doubled or trailing `;`)")
|
||||
continue
|
||||
if re.fullmatch(r"local-ttl=\d+", line):
|
||||
continue
|
||||
m = ENTRY.match(line)
|
||||
if not m:
|
||||
problems.append(f"entry {line!r} is not `local-ttl=N` or `address=/<domain>/${{VAR}}` — "
|
||||
f"this check cannot reason about it")
|
||||
continue
|
||||
dom, tgt = m.groups()
|
||||
if tgt[2:-1] not in CLUSTER_VARS:
|
||||
problems.append(f"{dom} -> {tgt}: not a key of clusters/base/cluster-vars.yaml")
|
||||
if dom in entries and entries[dom] != tgt:
|
||||
problems.append(f"{dom} is listed twice with different targets")
|
||||
entries[dom] = tgt
|
||||
return entries, problems
|
||||
|
||||
|
||||
entries, problems = parse(cat_vals[0])
|
||||
for pr in problems:
|
||||
check(False, f"(0) {pr}")
|
||||
|
||||
# ── inventories, derived from the tree ──────────────────────────────────────────────────────────
|
||||
disowned, unnamed = set(), []
|
||||
l4 = {}
|
||||
for top in SCAN:
|
||||
for f in sorted((ROOT / top).rglob("*.yaml")):
|
||||
for d in docs(f):
|
||||
meta = d.get("metadata") or {}
|
||||
ann = meta.get("annotations") or {}
|
||||
if d.get("kind") == "HTTPRoute" and str(ann.get(DISOWN, "")).strip() == "none":
|
||||
hosts = (d.get("spec") or {}).get("hostnames") or []
|
||||
if not hosts:
|
||||
unnamed.append(f"{f.relative_to(ROOT)}:{meta.get('name')}")
|
||||
disowned.update(hosts)
|
||||
if d.get("kind") == "Service" and EDNS_HOST in ann and LB_IPS in ann:
|
||||
for h in str(ann[EDNS_HOST]).split(","):
|
||||
l4[h.strip()] = str(ann[LB_IPS]).strip()
|
||||
for u in unnamed:
|
||||
check(False, f"(b) disowned HTTPRoute {u} declares no `hostnames:` — it inherits a listener "
|
||||
f"hostname this check cannot see; name it explicitly")
|
||||
check(len(disowned) >= 5, f"(0) only {len(disowned)} disowned HTTPRoute hostnames found — the scan "
|
||||
f"is vacuous (the disown annotation was renamed?)")
|
||||
check(len(l4) >= 2, f"(0) only {len(l4)} L4 Services publishing their own hostname found (expected "
|
||||
f"unifi and crafty-mc) — the scan is vacuous")
|
||||
|
||||
cf = yaml.safe_load(CF.read_text())
|
||||
cf_404 = {r["hostname"] for r in cf.get("ingress") or []
|
||||
if r.get("hostname") and r.get("service") == "http_status:404"}
|
||||
check(len(cf_404) >= 5, f"(0) only {len(cf_404)} http_status:404 hostnames in the cloudflared "
|
||||
f"config — the scan is vacuous")
|
||||
|
||||
|
||||
def covers_zone(dom):
|
||||
"""True if an entry for `dom` answers every name in the zone (dom is the zone or a parent)."""
|
||||
if dom in ("#", "", "."):
|
||||
return True
|
||||
if dom == ZONE:
|
||||
return True
|
||||
# A literal parent (e.g. `keithjackson.dev`, `dev`) of the zone's substituted value.
|
||||
real = dom.replace(ZONE, BASE_DOMAIN_VALUE)
|
||||
return real == BASE_DOMAIN_VALUE or BASE_DOMAIN_VALUE.endswith("." + real)
|
||||
|
||||
|
||||
def evaluate(entries):
|
||||
bad = []
|
||||
# (a)
|
||||
for dom in entries:
|
||||
if covers_zone(dom):
|
||||
bad.append(f"(a) `address=/{dom}/` answers EVERY name in the zone — the wildcard is gone "
|
||||
f"on purpose (public names must resolve to Cloudflare from the LAN)")
|
||||
# (b)
|
||||
for h in sorted(disowned):
|
||||
if entries.get(h) != GW:
|
||||
bad.append(f"(b) {h} is disowned from public DNS but PiHole maps it to "
|
||||
f"{entries.get(h, 'NOTHING')} — expected {GW}; without it the app is "
|
||||
f"unreachable everywhere")
|
||||
# (c)
|
||||
for h in sorted(cf_404):
|
||||
if h in l4:
|
||||
if entries.get(h) != l4[h]:
|
||||
bad.append(f"(c) {h} is an L4 DNS-only name; PiHole maps it to "
|
||||
f"{entries.get(h, 'NOTHING')}, expected its own VIP {l4[h]}")
|
||||
elif h not in entries:
|
||||
bad.append(f"(c) the tunnel 404s {h} but PiHole has no entry for it — it would resolve "
|
||||
f"only publicly, and the tunnel refuses it")
|
||||
# (d)
|
||||
for n in LAN_PINNED:
|
||||
h = f"{n}.{ZONE}"
|
||||
if entries.get(h) != GW:
|
||||
bad.append(f"(d) {h} maps to {entries.get(h, 'NOTHING (public/Cloudflare)')}; it must be "
|
||||
f"pinned to {GW} — Cloudflare caps request bodies at 100 MB and responses at "
|
||||
f"100 s")
|
||||
return bad
|
||||
|
||||
|
||||
for v in evaluate(entries):
|
||||
check(False, v)
|
||||
|
||||
# (e) mutation self-test
|
||||
mutants = {}
|
||||
m = dict(entries); m[ZONE] = GW
|
||||
mutants["zone wildcard added"] = m
|
||||
m = dict(entries); m[BASE_DOMAIN_VALUE.split(".", 1)[1]] = GW
|
||||
mutants["parent-of-zone wildcard added"] = m
|
||||
m = dict(entries); m.pop(sorted(disowned)[0], None)
|
||||
mutants[f"disowned host {sorted(disowned)[0]} dropped"] = m
|
||||
pure_404 = sorted(h for h in cf_404 if h not in l4 and h not in disowned)
|
||||
if pure_404:
|
||||
m = dict(entries); m.pop(pure_404[0], None)
|
||||
mutants[f"404'd host {pure_404[0]} dropped"] = m
|
||||
m = dict(entries); m.pop(f"gitea.{ZONE}", None)
|
||||
mutants["gitea unpinned"] = m
|
||||
m = dict(entries); m[f"nextcloud.{ZONE}"] = "${UNIFI_L4_VIP}"
|
||||
mutants["nextcloud repointed"] = m
|
||||
if l4:
|
||||
h = sorted(l4)[0]
|
||||
m = dict(entries); m[h] = GW
|
||||
mutants[f"{h} repointed to the Gateway"] = m
|
||||
for name, mut in mutants.items():
|
||||
check(evaluate(mut), f"(e) self-test: the evaluator did NOT flag the mutant '{name}' — vacuous")
|
||||
check(len(mutants) >= 5, f"(e) only {len(mutants)} mutants built — the self-test degenerated")
|
||||
_, pr = parse("local-ttl=60;address=/x.${BASE_DOMAIN}/${NOT_A_CLUSTER_VAR}; address=/y/${GATEWAY_VIP}")
|
||||
check(len(pr) >= 2, "(e) self-test: the parser accepted an unknown VAR or a whitespace-padded entry")
|
||||
|
||||
if CHECKS < 15:
|
||||
FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
|
||||
if FAILURES:
|
||||
print(f"CHECK FAIL ({NAME}): {len(FAILURES)} of {CHECKS} assertions failed", file=sys.stderr)
|
||||
for f in FAILURES:
|
||||
print(f" - {f}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"PASS ({NAME}): {CHECKS} assertions; {len(entries)} PiHole names, {len(disowned)} disowned "
|
||||
f"routes, {len(cf_404)} tunnel-404 names, {len(l4)} L4 names")
|
||||
PY
|
||||
@@ -62,7 +62,7 @@ spec:
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
|
||||
+1
-1
@@ -85,7 +85,7 @@ spec:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
|
||||
@@ -85,7 +85,7 @@ spec:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
|
||||
@@ -64,6 +64,12 @@ spec:
|
||||
annotations:
|
||||
# conformance-dns.<this cluster's BASE_DOMAIN> — never a literal apex.
|
||||
external-dns.alpha.kubernetes.io/hostname: (join('.', ['conformance-dns', $baseDomain]))
|
||||
# DNS-ONLY (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
|
||||
# gateway names point at the Cloudflare Tunnel). Without this opt-out the probe
|
||||
# record is orange-clouded, 1.1.1.1 answers Cloudflare anycast instead of the LB IP,
|
||||
# and the exact-IP poll below can never match. Same opt-out every real
|
||||
# `service`-source publisher carries (unifi, crafty-mc).
|
||||
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
# No real backend needed — Cilium LB-IPAM assigns a VIP regardless, and
|
||||
|
||||
@@ -61,7 +61,7 @@ CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
|
||||
# chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the
|
||||
# `sha256:` prefix this value carries — the trigger must strip it.
|
||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main
|
||||
CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:65cc7c1ab576fe8cdf45e7f68c7cf4eb5ce7e0db46e62191f6190c51610b20d9"
|
||||
CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:63e1aa74289a8b77ef5cd2ec4e4d01e7fef638f965f2057329bc1991d3074f0d"
|
||||
CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}"
|
||||
#
|
||||
# AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: chore/adopt-spec-sync
|
||||
title: Adopt the spec sync (registry parity, ADR cross-references, spec pin)
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator ruled this repo's `main` (`50c8719`) the accepted state and had the System Architect
|
||||
bring the spec repo in line with it (spec PR `jakman85/home-network-k3s#2`). This is the
|
||||
implementation half of that sync.
|
||||
|
||||
- `conformance/requirements/registry.yaml`: added R-DNS-04, R-DNS-05 and R-SEC-04, which the spec
|
||||
defined and this mirror lacked. Reworded the R-ENV-01/02/03 and R-UPD-02 summaries to match the
|
||||
amended requirements.
|
||||
- Tagged existing L0 tests against those IDs, because a registry entry with no tagged test blocks
|
||||
the merge: `cloudflared-ingress-rules` (R-DNS-04, R-DNS-05, R-SEC-04), `pihole-lan-names`
|
||||
(R-DNS-04), and `internal-services-publish-no-public-dns` (R-DNS-05).
|
||||
- `platform/vpn-egress/README.md` and a comment in `egress-steering.yaml` pointed at a drafted
|
||||
"ADR-0016 supersede". That number went to the Cloudflare Tunnel; the accepted ADR is ADR-0020.
|
||||
- `ci/SPEC_PIN` must move to the spec merge commit. That SHA does not exist until the spec PR
|
||||
merges, so this PR carries a `WIP:` title until the bump is added.
|
||||
|
||||
## What I verified
|
||||
|
||||
- Read each tagged test's assertions before tagging it. `cloudflared-ingress-rules` asserts every
|
||||
LAN-only hostname maps to a 404, the catch-all exists, no rule targets the API, and the egress
|
||||
policy admits no API or host entity. `pihole-lan-names` asserts LAN names resolve independently
|
||||
of the public path. `internal-services-publish-no-public-dns` asserts internal routes carry the
|
||||
disown annotation and pins the declared L4 set.
|
||||
- The toolbox gate result for this commit is quoted in the PR description, with the spec branch
|
||||
mounted so that the checks parsing spec text ran against the new spec.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **These tags are static coverage, not proof.** No L2 test touches the tunnel, and nothing here
|
||||
shows router 80/443 closed or 6443 unforwarded (spec `docs/16-known-gaps.md`).
|
||||
- R-SEC-04 is registered `safety_critical: false`. Making it `true` needs an L3 fault-injection
|
||||
test that does not exist. UNVERIFIED whether the operator wants that.
|
||||
- No live cluster, tunnel or DNS state was measured.
|
||||
- `spec-immutability` did not run against a pinned checkout: the pin is not bumped yet.
|
||||
- Stale implementation prose found by the audit was deliberately left out of this PR, because the
|
||||
operator ruled this repo the accepted state and none of it changes behaviour. Examples:
|
||||
`catalog/gitea.yaml` still says "+ mirrored"; `platform/longhorn` still promises a backup target;
|
||||
`README.md` says `node-loss` is outstanding; `hands-off-update` exercises Flux image policy rather
|
||||
than Renovate; `dev-nodes-provisioned-as-code` is named for an environment that no longer exists.
|
||||
@@ -0,0 +1,139 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: feat/cloudflare-tunnel
|
||||
title: Cloudflare Tunnel connector, Phase 1 — the component only, isolated, with zero change to live traffic
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Ingress is moving to a Cloudflare Tunnel because the GFiber router cannot hairpin and hands LAN
|
||||
clients Google's IPv6 DNS, so split DNS keeps breaking. This is **Phase 1: the connector only.**
|
||||
external-dns, the Gateway annotations, Pi-hole and CoreDNS are untouched. A later change, by a
|
||||
different agent, moves DNS.
|
||||
|
||||
**Operator decisions (2026-11-08), recorded here as the source for the guard:**
|
||||
|
||||
1. Public hostnames stay `*.home.keithjackson.dev` and the apex `home.keithjackson.dev`. The
|
||||
operator enables Cloudflare ACM (ordered edge cert or Total TLS) for edge certificates.
|
||||
2. **Plex stays OFF the tunnel.** No plex hostname goes through cloudflared (`plex`, `plex-vip` → 404).
|
||||
3. **jackett, pihole and semaphore stay LAN/VPN-only**, never via the tunnel. The same goes for the
|
||||
non-HTTP L4 VIPs **unifi** and **crafty-mc**. After the PM's read-only audit, the other three
|
||||
catalog `exposure: internal` apps, **qbittorrent, radarr and sonarr**, are on the same list. All
|
||||
six carry the external-dns `controller: none` route annotation live.
|
||||
4. **The Kubernetes API stays local-only.** It must never be reachable through the tunnel.
|
||||
5. Pi-hole keeps serving the LAN. Split DNS gets trimmed later, by another agent.
|
||||
6. **Headlamp is LAN/VPN-only** (operator decision, relayed by the PM later the same day). It is
|
||||
permanently in the 404 list and excluded from the egress backends. Its catalog/route exposure is
|
||||
deliberately **not** changed here; the Phase 2 DNS-flip PR does that.
|
||||
|
||||
**Files:**
|
||||
|
||||
- `platform/cloudflared/` (new):
|
||||
- `config.yaml` is the locally-managed tunnel config, projected through a content-hashed
|
||||
`configMapGenerator`, so edits roll the pods. It has explicit `http_status:404` rules for
|
||||
plex, plex-vip, jackett, pihole, pihole-dns, semaphore, qbittorrent, radarr, sonarr, headlamp,
|
||||
unifi and crafty-mc. Then the apex and `*.${BASE_DOMAIN}` go to
|
||||
`https://cilium-gateway-shared.gateway.svc.cluster.local:443` with `matchSNItoHost: true`. Last
|
||||
comes a catch-all 404. The tunnel UUID is a **literal** in this file (`REPLACE-ME-TUNNEL-UUID`
|
||||
until enrolment), not a cluster-var, so the Phase 2 PR can match it without touching `clusters/**`.
|
||||
- `cloudflared-credentials.sops.yaml` is SOPS-encrypted and holds `REPLACE-ME` placeholders, with
|
||||
a `.example` hand-off sibling. I encrypted the placeholder (not a credential) in the toolbox
|
||||
with the root `.sops.yaml` recipient.
|
||||
- `deployment.yaml` runs 2 replicas with a hard `topologySpreadConstraints` across nodes. The image
|
||||
is `cloudflare/cloudflared:2026.10.7@sha256:4955524a…` (tag + digest, Renovate `kubernetes`
|
||||
manager). It runs as non-root 65532, with a read-only root fs, all capabilities dropped and no
|
||||
SA token. Requests are 10m/64Mi and limits 500m/256Mi. Readiness uses `/ready`. Liveness uses
|
||||
`/healthcheck`, so an edge outage cannot restart-loop the pods. The pods are labelled
|
||||
`egress-class: internet`. A PDB sets `minAvailable: 1`.
|
||||
- `networkpolicy.yaml`:
|
||||
- A namespace `default-deny` with DNS only. It has a **single owner**: the namespace is not
|
||||
`app-namespace=true`, so Kyverno's generate rule never writes it.
|
||||
- A Prometheus scrape allow on :2000.
|
||||
- The egress CNP allows world tcp/udp 7844 and tcp 443, plus the `ingress` entity on tcp 443.
|
||||
It also allows **Gateway backends by `io.cilium.k8s.namespace.labels.gateway-access=shared`,
|
||||
minus the LAN-only namespaces.**
|
||||
- `servicemonitor.yaml` (headless metrics Service + ServiceMonitor) and `alerts.yaml` (`CloudflaredTunnelDown`, `CloudflaredReplicasDegraded`)
|
||||
ship with `alerts.promtool-test.yaml`, and `README.md` documents the component.
|
||||
- `clusters/base/cloudflared.yaml` (new) is its own Flux Kustomization: `dependsOn: platform-config`,
|
||||
**`wait: false`**, prune, and postBuild `cluster-vars`. It is listed in `clusters/base/kustomization.yaml`.
|
||||
- `clusters/{dev,prod}/kustomization.yaml` each gain a SOPS `decryption` patch for `cloudflared`.
|
||||
- `conformance/tests/l0-static/cloudflared-ingress-rules/` (new, whole-repo scope) asserts several
|
||||
things:
|
||||
- Every rule targets either 404 or exactly the Gateway Service. The Service name is derived from
|
||||
`platform/gateway/gateway.yaml`, and API markers are named explicitly.
|
||||
- The excluded hosts 404 before the first Gateway rule, the Gateway rules use `matchSNItoHost`,
|
||||
there is no `noTLSVerify`, and a catch-all closes the list.
|
||||
- The CNP has no kube-apiserver, host, remote-node, cluster or `all` entity, no CIDR, and the full
|
||||
NotIn list. The pods carry egress-class internet and have no SA token.
|
||||
- The Flux tier is isolated: wait:false, not in `platform/kustomization.yaml`, nothing
|
||||
`dependsOn: cloudflared`, and both overlays decrypt it.
|
||||
- **Mutation self-test:** an added API rule, plex moved below the wildcard, and a dropped
|
||||
catch-all must each be flagged.
|
||||
- `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` gains an additive
|
||||
`cloudflared_` scrape owner, with a networkpolicy companion, in the same shape as the kured entry.
|
||||
- `runbooks/cloudflare-tunnel.md` (new) is registered in `runbooks/kustomization.yaml` and the
|
||||
README index.
|
||||
|
||||
**Design choices:**
|
||||
|
||||
- **Isolation (the merge-before-credentials question).** The PR is safe to merge with placeholders.
|
||||
The Deployment CrashLoops, but in its own `wait:false` Kustomization that nothing depends on. Its
|
||||
Ready condition tracks the apply, not pod health, so neither `platform`, `platform-config` nor any
|
||||
apps wave can stall. There is no Certificate or HelmRelease in the path for health checks to hang
|
||||
on. Health is reported by the two alerts instead. `CloudflaredTunnelDown` **will fire from merge
|
||||
until enrolment**. That is honest, and it is `warning` because nothing routes through the tunnel
|
||||
yet. Raise it to critical in the DNS-flip change.
|
||||
- **Egress to backends, not `{}`.** Under Cilium 1.20 (CVE-2025-30162), the Gateway's Envoy
|
||||
re-checks the source pod's egress against the resolved backend. An egress rule that names only
|
||||
the Gateway would therefore make every tunnelled request `403 Access denied`
|
||||
(decisions/2026-10-30-gitea-admit-buildkit.md; catalog/render.py, `r_internet_egress`). The
|
||||
gitea-actions `- {}` pattern would also work, but it admits kube-apiserver and host, which
|
||||
decision 4 rules out. The namespace-label selector names every Gateway backend and nothing else.
|
||||
**Backends need no ingress change.** Envoy reaches them as `reserved:ingress`, and the 2026-10-30
|
||||
measurement showed the identity check sits on the source's egress: gitea-actions is in none of
|
||||
Gitea's ingress allows and still reaches it. So no app's `inClusterClients` was touched.
|
||||
- **The alerts cannot self-silence.** Both expressions use `or on() vector(0)`, so absent series
|
||||
(a broken scrape, zero pods, KSM gone) fire. The promtool tests pin the absent case and the
|
||||
working-then-stale re-fire case.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **The Gateway Service, measured live (read-only, `k3s-prod`, 2026-11-08).** It is
|
||||
`gateway/cilium-gateway-shared`: type LoadBalancer, ClusterIP `10.43.225.79`, EXTERNAL-IP
|
||||
`192.168.2.20`, ports 80/443, no selector, labelled `gateway.networking.k8s.io/gateway-name=shared`.
|
||||
It belongs to Gateway `gateway/shared` (class `cilium`).
|
||||
- **The namespace selector, measured live.** Cilium identities carry the label
|
||||
`k8s:io.cilium.k8s.namespace.labels.gateway-access` (read from a gitea identity). Every namespace
|
||||
with an HTTPRoute (crafty, gitea, headlamp, homepage, jackett, keycloak, monitoring, nextcloud,
|
||||
oauth2-proxy, pihole, plex, qbittorrent, radarr, runbooks, seerr, semaphore, sonarr) is
|
||||
`gateway-access=shared`.
|
||||
- **Nothing exists yet.** Namespace `cloudflared` does not exist on prod. **cloudflared is not in
|
||||
the pinned toolbox**, so the runbook runs the pinned image under podman instead.
|
||||
- The image tag and digest came from Docker Hub's API: `2026.10.7` = `latest` =
|
||||
`sha256:4955524a…c9b9c5ba`, published 2026-10-30.
|
||||
- The credentials Secret is encrypted: `stringData.credentials.json` is `ENC[…]`, with the age
|
||||
recipient from the root `.sops.yaml`.
|
||||
- The new L0 check passes on the host (47 assertions, mutation self-test included). The toolbox
|
||||
gate result is reported in the PR description. I ran it after committing.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **UNVERIFIED: nothing ran in a cluster.** No pod has started, and no tunnel exists. I have not
|
||||
observed placeholder CrashLoop behaviour itself. I expect cloudflared to treat
|
||||
`REPLACE-ME-TUNNEL-UUID` as a tunnel name, find no origin cert, and exit, but whatever it does,
|
||||
the pods are not Ready and the tier is non-gating.
|
||||
- **UNVERIFIED: the CVE-2025-30162 path for cloudflared specifically.** The selector form was
|
||||
checked against live identity labels, but no tunnelled request has been made. The runbook's
|
||||
step 6 `kubectl debug` probe exists to prove "real backend, no 403" from the pod's own identity.
|
||||
- **UNVERIFIED: whether the `ingress` entity rule's `toPorts: 443` restriction holds at Envoy.**
|
||||
buildkit's working rule has no port restriction. If the canary shows drops, removing the port
|
||||
restriction there is the first thing to try.
|
||||
- The metric name `cloudflared_tunnel_ha_connections` and the `/ready` / `/healthcheck` endpoints
|
||||
come from my knowledge of cloudflared, **not measured** against 2026.10.7.
|
||||
- The Cloudflare edge egress is `world` + ports, not Cloudflare CIDRs. I did not narrow it blind.
|
||||
- Egress `world:443` "for the API" follows the brief. I have not confirmed that a credentials-file
|
||||
tunnel needs it at all.
|
||||
- The Cloudflare-side settings (ACM, Total TLS / ordered cert, Bot Fight Mode, BIC/WAF, WebSockets)
|
||||
are documented for the operator but not inspected. I have no Cloudflare access.
|
||||
- The headlamp, qbittorrent, radarr and sonarr catalog exposures are deliberately unchanged
|
||||
(Phase 2).
|
||||
@@ -0,0 +1,59 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: fix/cloudflared-creds-permission
|
||||
title: cloudflared CrashLoop — credentials Secret unreadable by uid 65532; add fsGroup, pin it in L0
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
- **`platform/cloudflared/deployment.yaml`**: pod `securityContext.fsGroup: 65532`. Root cause of
|
||||
the CrashLoop that followed #152/#153: the pod runs as `65532:65532` with no `fsGroup`, and the
|
||||
Secret volume is `defaultMode: 0440`. The kubelet therefore laid `credentials.json` down as
|
||||
`root:root 0440`, which uid 65532 (not owner, not in gid 0) cannot read. cloudflared logged
|
||||
`couldn't read tunnel credentials ... permission denied`, then fell back to origin-cert lookup
|
||||
and ended on the misleading `error parsing tunnel ID: Error locating origin cert`. With
|
||||
`fsGroup` the files are group-owned by 65532, readable by the container and no other uid; the
|
||||
mode stays 0440 (not loosened to 0444). Same fix as `platform/keycloak/client-secret-reconciler.yaml`.
|
||||
Still non-root, `runAsNonRoot: true`, unchanged otherwise.
|
||||
- **`conformance/tests/l0-static/cloudflared-ingress-rules/check.sh`**: new assertion (7). For each
|
||||
Secret volume a container mounts, it models kubelet ownership (uid 0, gid = fsGroup or 0, item
|
||||
mode / defaultMode / API default 0644) against the container's effective runAsUser and its groups
|
||||
(runAsGroup + fsGroup + supplementalGroups) and fails if the file is unreadable. An unset
|
||||
runAsUser fails (image default is not knowable offline). Conservative: it does not credit the
|
||||
kubelet OR-ing group-read into fsGroup volumes. Mutation self-test: fsGroup removed, mode 0400,
|
||||
runAsUser unset must each be flagged. Chainsaw step description updated to match.
|
||||
- **`runbooks/cloudflare-tunnel.md`**: new first "likely cause" (permission denied on
|
||||
credentials.json → the origin-cert error is a red herring) plus a "read the first error line"
|
||||
note; causes renumbered. New section **Rotating the tunnel credential**: create a new tunnel,
|
||||
swap the UUID and the encrypted credentials, merge, verify (new tunnel has two connectors, old
|
||||
has none), then `cloudflared tunnel cleanup`/`delete <old>`.
|
||||
- NOT touched, on purpose: `tunnel:` in `config.yaml` and `cloudflared-credentials.sops.yaml`. The
|
||||
operator is rotating the credential (it was exposed in a transcript) in a separate PR.
|
||||
|
||||
## What I verified
|
||||
|
||||
- Live (k3s-prod, read-only, 2026-11-08 ~21:38Z): Deployment pod securityContext is
|
||||
`{"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532,"seccompProfile":...}`, no
|
||||
`fsGroup`; creds volume `defaultMode: 288` (= 0440). All three pods CrashLoopBackOff (38
|
||||
restarts). Secret `cloudflared-credentials` has the one key `credentials.json` (key names only).
|
||||
- Pinned image `cloudflare/cloudflared@sha256:4955524a...` config (`skopeo inspect --config`):
|
||||
`User: 65532:65532`, `Entrypoint: [cloudflared]`, `Cmd: [version]`.
|
||||
- The new assertion FAILS (rc 1) on the pre-fix deployment.yaml with
|
||||
`container cloudflared (uid 65532, groups [65532]) cannot read Secret volume creds ... files are
|
||||
0:0 mode 0440`, and PASSES (51 assertions) with the fix. All three mutants are flagged.
|
||||
- Kyverno admission policies under `platform/kyverno/admission/` that mention securityContext
|
||||
fields are scoped to the conformance-trigger and pm-agent Jobs only; none constrains fsGroup here.
|
||||
- Toolbox gate: see the PR description for the rc and skipped stages.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That the pods actually become Ready after merge.** Not deployed by me; Flux applies it. The
|
||||
file mode inside the running container was not observed (pods crash before an exec is possible,
|
||||
and exec would be an operator action anyway). The readability claim rests on the kubelet's
|
||||
documented fsGroup behaviour and the identical, working keycloak reconciler.
|
||||
- Whether the CURRENT credential is otherwise valid (UUID/TunnelID match, tunnel exists). Moot:
|
||||
the operator is rotating it.
|
||||
- The rotation runbook section has not been executed; `cloudflared tunnel cleanup` / `delete`
|
||||
syntax is from the CLI's documented commands, not run here.
|
||||
- The deployment comment says the image ENTRYPOINT is `cloudflared --no-autoupdate`; the pinned
|
||||
image's ENTRYPOINT is plain `cloudflared`. Left unchanged (out of scope); flagged for follow-up.
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: feat/dns-via-cloudflare-tunnel
|
||||
title: Public DNS via the Cloudflare Tunnel (Phase 2) — proxied CNAMEs, Plex/Headlamp LAN-only, L0 guard
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
One atomic change, so the conformance guard lands with the DNS flip.
|
||||
|
||||
- `platform/gateway/gateway.yaml`: the shared Gateway's `external-dns.alpha.kubernetes.io/target`
|
||||
is now the literal `fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com` (was
|
||||
`wan.${BASE_DOMAIN}`). Comment block rewritten.
|
||||
- `platform/externaldns/helmrelease.yaml`: `extraArgs: [--cloudflare-proxied]` (global, because
|
||||
v0.23.0 reads per-object `cloudflare-proxied` from the route/Service, never the Gateway). The
|
||||
public-target comment block rewritten, including the prefix-regression risk.
|
||||
- `apps/vms/unifi/50-service-l4.yaml`, `platform/crafty/service-l4.yaml`:
|
||||
`external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"` (L4 A records stay DNS-only). They
|
||||
are the only `service`-source publishers in the tree and live (measured).
|
||||
- Plex web UI LAN/VPN-only: `catalog/plex.yaml` `exposure: public` → `internal`; `apps/wave-0/plex.yaml`
|
||||
re-rendered with the canonical command into a scratch dir and ONLY plex.yaml copied back. The
|
||||
render diff is exactly +2 lines (`annotations: external-dns.alpha.kubernetes.io/controller: none`
|
||||
on HTTPRoute plex-32400). No other golden moved, and nothing was onboarded (vm-demo/keycloak files
|
||||
were not touched). ADVERTISE_IP, the L4 VIP .24 and `plex-remote.yaml` are untouched.
|
||||
- Headlamp LAN/VPN-only: `controller: none` on the hand-authored `platform/headlamp/httproute.yaml`.
|
||||
- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh`: header and
|
||||
register prose updated; new assertion (9):
|
||||
(9a) Gateway target == `<tunnel: from platform/cloudflared/config.yaml>.cfargotunnel.com`, and
|
||||
`--cloudflare-proxied` is set; (9b) every `service`-source publisher has `cloudflare-proxied:
|
||||
"false"` and no HTTPRoute opts out of proxying; (9c) no undisowned route names a hostname the
|
||||
tunnel 404s, the only `service` publishers of a 404'd name are the declared L4 set
|
||||
{unifi, crafty-mc}, and every disowned route's hostname IS 404'd by the tunnel (lists kept
|
||||
consistent both ways); (9d) plex and headlamp pinned disowned. Its `chainsaw-test.yaml` gap prose
|
||||
updated.
|
||||
- `conformance/tests/l2-behavioral/dns-exposure/chainsaw-test.yaml`: fixture Service gets
|
||||
`cloudflare-proxied: "false"`, else the global flag proxies it and the exact-IP poll fails.
|
||||
- Docs: `platform/externaldns/ddns.yaml` header (wan. is WireGuard-only), `platform/externaldns/README.md`,
|
||||
`platform/gateway/README.md`, `runbooks/plex-remote-access.md`, `runbooks/cloudflare-tunnel.md`
|
||||
(Phase 2 done; operator step to close WAN tcp/80+443 and keep udp/51820 + tcp/32400; rotation
|
||||
now also swaps the Gateway UUID; rollback path).
|
||||
|
||||
## What I verified
|
||||
|
||||
- external-dns v0.23.0 source, read at the tag: `source/gateway.go` takes target overrides from
|
||||
`gw.Annotations` and provider-specific (incl. `cloudflare-proxied`) from the ROUTE's annotations;
|
||||
`annotations.SetAnnotationPrefix` re-derives `CloudflarePrefix`, so the alpha-prefixed key is the
|
||||
one read here; `provider/cloudflare` `shouldBeProxied` starts from `--cloudflare-proxied` and a
|
||||
parsed `"false"` annotation overrides it.
|
||||
- (8b) only forbids `annotation-prefix` / `enable-legacy-annotation-prefix` in extraArgs; a mutation
|
||||
confirmed `--cloudflare-proxied` passes and `--annotation-prefix=x/` fails.
|
||||
- The L0 check passes (116 assertions) and each new assertion FAILS on its mutation: Gateway target
|
||||
back to wan (9a), flag removed (9a), unifi proxied "true" (9b), headlamp undisowned (9c,9d), plex
|
||||
golden undisowned (9c,9d,(1)), jackett removed from the tunnel's 404 list (9c).
|
||||
`cloudflared-ingress-rules` still passes; `render.py --selftest` PASS.
|
||||
- Live, read-only (kubectl k3s-prod, ~23:55Z): 17 HTTPRoutes; the six internal apps carry
|
||||
`controller: none`; plex and headlamp do not (yet); the only Services with an external-dns
|
||||
hostname are crafty/crafty-l4-tcp and unifi/unifi-l4-tcp; Gateway target is currently
|
||||
`wan.home.keithjackson.dev`.
|
||||
- Toolbox gate result: see the PR description (rc and skipped stages quoted there).
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- Anything after merge: that external-dns actually UPDATEs the eight public names to proxied CNAMEs
|
||||
to the tunnel, DELETEs plex + headlamp and their TXT, and leaves unifi/crafty-mc/wan untouched.
|
||||
Expected effect is listed in the PR; the PM must read external-dns logs and `dig @1.1.1.1`.
|
||||
- A rendered HelmRelease: I read the chart 1.23.0 `templates/deployment.yaml` (a list `extraArgs`
|
||||
is emitted item-by-item via `tpl . $ | quote`), but did not `helm template` it.
|
||||
- kubectl OIDC from nodes: nodes resolve publicly, so `keycloak.` discovery/JWKS now goes through
|
||||
Cloudflare + the tunnel. No `oidc-ca-file` is set (system roots), so the edge cert should validate,
|
||||
but Cloudflare bot/WAF settings on that path are untested.
|
||||
- RISK: if the annotation prefix stops being honoured at runtime again (11-07 incident class),
|
||||
`target` is ignored and every name becomes a PROXIED A record at 192.168.2.20, while internal
|
||||
names publish. Assertion (8) only keeps the tree and `values.annotationPrefix` consistent; it
|
||||
cannot catch a controller that ignores the flag. No runtime alert exists for it.
|
||||
- The router WAN 80/443 close is an operator step, not done. The Pi-hole wildcard trim is a later PR.
|
||||
- Follow-up: spec requirement IDs for this (proposed R-DNS-04/05, R-SEC-04) once the spec ADR merges;
|
||||
`platform/pm-agent/README.md` and a comment in `catalog/pihole.yaml` still describe the wan. CNAME.
|
||||
@@ -0,0 +1,104 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: feat/gate-toolbox-container
|
||||
title: The merge gate runs inside the CI-built toolbox image, pinned by digest — no per-run tool install
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`.gitea/workflows/gate.yml`, job `gate` (job name, the gate step, its env and its
|
||||
`|| [ "$?" -eq 3 ]` tolerance are unchanged, so the required context `gate / gate (pull_request)`
|
||||
is unchanged):
|
||||
|
||||
- **`container:`** `gitea.home.keithjackson.dev/home-network/toolbox@sha256:83a3501e…ef81` — the
|
||||
image `images.yaml` published from `dd94101` (run 789; it is today's `main` and `sha-dd941016478c`
|
||||
tag). The same image `infra/devcontainer/dev.sh` runs, so local and CI gate on the identical
|
||||
toolchain. A `# renovate-digest: image=… tag=main` marker sits on the line above, the repo's
|
||||
existing convention; `infra/gitea/registry-retention.sh` reads the same marker, so this digest
|
||||
is protected from pruning.
|
||||
- **`actions/checkout` replaced by a plain-git step.** The toolbox has no Node (measured:
|
||||
`command -v node` → missing), and `actions/checkout` is a Node action. Chosen over adding Node to
|
||||
the toolbox because that would need two sequential PRs (Containerfile, then a publish, then the
|
||||
pin) and would put a JS runtime into the operator toolbox only to run one action. The step
|
||||
reproduces what checkout@v7 `fetch-depth: 0` actually ran here (read from runs 490 and 495):
|
||||
every branch to `refs/remotes/origin/*`, tags, plus the event SHA; detached for a PR,
|
||||
`-B <branch>` for a push; asserts `HEAD == github.sha`. The job token goes to that one fetch via
|
||||
`GIT_CONFIG_*` env and is **not** persisted in `.git/config` (nothing later needs network git).
|
||||
It sets **`umask 0022`** first: the PR's first CI run (run 500) failed L0
|
||||
`prod-baremetal-not-dev-surface` because act runs steps with umask 0000, every checked-out
|
||||
directory was world-writable, and Ansible (present in the toolbox, absent from the catthehacker
|
||||
image) then ignored `infra/ansible/ansible.cfg`. Reproduced locally in the pinned image: clone
|
||||
under umask 0000 → that check fails with the same message; under 0022 → `check OK`. Side
|
||||
effect worth knowing: the Ansible-driven parts of that check now actually execute in CI.
|
||||
- **"Install pinned CI toolchain" step removed**, replaced by **"Toolchain is the one
|
||||
ci/versions.env pins"**: every `*_VERSION` in `ci/versions.env` must equal the image's
|
||||
`/etc/dev-toolchain-versions`, else `::error::DRIFT` and the job fails. Without this, dropping
|
||||
the installer would have made `ci/versions.env` an unenforced claim for everything except
|
||||
promtool (which `alert-rules-unit-tested` already asserts).
|
||||
- Header comments rewritten for the new job image (pull auth, cold-pull size, no-Node, why
|
||||
`install-tools.sh` stays).
|
||||
|
||||
`renovate.json5`: the `renovate-digest` custom manager now also scans `.gitea/workflows/gate.yml`;
|
||||
the `github-actions` manager is disabled for this repo's own images (same reason the `kubernetes`
|
||||
manager already was: it would see a digest-only image and look up `latest`).
|
||||
|
||||
Comment/message wording only: `ci/gate.sh` (header + the strict-skip error text), `ci/lib.sh`,
|
||||
`ci/README.md`, `.gitea/workflows/README.md` — they said the CI toolchain comes from
|
||||
`install-tools.sh`.
|
||||
|
||||
`.gitea/workflows/scripts/install-tools.sh` is **kept**: `merge-gate.yaml` (dormant blueprint)
|
||||
calls it, and `conformance/tests/l0-static/alert-rules-unit-tested/check.sh` asserts its promtool
|
||||
wiring (not touched — another agent owns `conformance/`).
|
||||
|
||||
**Consequence to know:** every toolbox digest bump is now a Renovate PR touching `.gitea/**`,
|
||||
i.e. a sensitive path needing an operator `/approve-override`. That is arguably correct (a new
|
||||
toolchain changes what green means — and `ci/versions.env` was already sensitive), but it is a
|
||||
new recurring operator touch.
|
||||
|
||||
## What I verified
|
||||
|
||||
- Registry pull auth: the Home-Network org is `public`; anonymous `/v2/token` → manifest `200`,
|
||||
`docker-content-digest: sha256:83a3501e…ef81` for both `main` and `sha-dd941016478c`. So no
|
||||
`container: credentials:` is needed. (The Actions token could not have been used anyway — no
|
||||
package scope, per images.yaml's header.)
|
||||
- Image: 466,912,274 bytes compressed across 6 layers (manifest sum), 1,541,728,125 bytes
|
||||
unpacked (`podman image inspect`). Local cold pull 10.4s on the workstation.
|
||||
- Image tool versions equal `ci/versions.env` for all 8 pins (kyverno 1.19.0, kyverno-json 0.0.3,
|
||||
chainsaw 0.2.12, yamllint 1.35.1, kubectl 1.36.4, yq 4.44.6, promtool 3.14.0, check-jsonschema
|
||||
0.29.4) — both via `/etc/dev-toolchain-versions` and by running each binary's `--version`.
|
||||
`sleep`, `tail`, `git` 2.56.0, `bash`, `python3`, `jq`, `curl` present (act's entrypoint is
|
||||
`/bin/sleep 10800`).
|
||||
- The new checkout script, run in the pinned image against the public repo (auth header disabled
|
||||
for the test only): PR ref `refs/pull/143/head` → detached at `60e3a24`, 1018 commits; push ref
|
||||
`refs/heads/main` → `On branch main` at `e8d2666`; `origin/main` present in both. The
|
||||
toolchain-drift step passes on the real `ci/versions.env` and exits 1 naming both tools when
|
||||
yamllint/promtool pins are altered.
|
||||
- Renovate marker regex (copied from renovate.json5) matches gate.yml exactly once, capturing
|
||||
depName/tag/digest.
|
||||
- LOCAL GATE, run **in the pinned CI image itself** (`IMAGE=<the digest above>` through
|
||||
`dev.sh`, `DEV_KUBECONFIG=/dev/null`, ENGINE shim adding the main checkout's `.git` read-only),
|
||||
on commit `9537ca2`: **rc 3**; spec-immutability, decisions-log (78 entries), lint, l0,
|
||||
render-golden, l1, approval-policy (N/A locally — not a PR event) all pass; **gitea-settings
|
||||
the only stage not run** (no `SETTINGS_API_TOKEN` locally); L2/L3 conformance skipped as
|
||||
usual. rc 3 is not green.
|
||||
|
||||
- **CI, this PR's own runs** (Gitea API `/actions/runs/{id}/jobs` + job logs):
|
||||
- run 500 (`514ed5d`, cold DinD cache: `Image exists? false`, image pulled): job 151s; set-up
|
||||
incl. pull **9s**, checkout <1s, toolchain check 1s (all 8 pins `ok`), gate 140s → BLOCKED by
|
||||
L0 `prod-baremetal-not-dev-surface` (umask, fixed above) + approval-policy.
|
||||
- run 503 (`27d603f`, warm: `Image exists? true`): job **140s**; set-up 1s, checkout 1s,
|
||||
toolchain check 0s, gate 137s. Every stage passes — l0 (incl. prod-baremetal-not-dev-surface),
|
||||
lint, decisions-log, render-golden, l1, **gitea-settings** — except **approval-policy**,
|
||||
the expected sensitive-path block. spec-immutability skipped (expected, no spec repo in CI).
|
||||
- baseline before (runs 494/495, push on main): job 180s/179s; set-up 1–2s, checkout 1s,
|
||||
**install 11s**, gate 164–165s. The install step is gone; the remaining ~25s gate-step delta
|
||||
is NOT claimed as caused by this change (different base commits, one sample each).
|
||||
- Re-gated locally on `27d603f` in the pinned image: rc 3, gitea-settings the only not-run stage.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- Renovate actually opening a digest-bump PR for gate.yml — only the regex was tested, not a
|
||||
Renovate run.
|
||||
- Push-to-main behaviour in CI (only the PR event runs before merge).
|
||||
- Whether the gate-step speedup (165s → 137s) is real or run-to-run variance.
|
||||
- That the cold-pull cost stays ~9s under a busier node or a bigger digest delta; one sample.
|
||||
@@ -0,0 +1,60 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: fix/keycloak-servicemonitor-owner
|
||||
title: One owner for the keycloak ServiceMonitor — disable the 26.8 operator's (409 loop stalled platform-config)
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
- `platform/keycloak/keycloak-instance.yaml`: added `spec.serviceMonitor.enabled: false`.
|
||||
The 26.x Keycloak operator creates its own ServiceMonitor, named after the CR (`keycloak/keycloak`),
|
||||
whenever `metrics-enabled=true` and the ServiceMonitor CRD exists. `serviceMonitor.enabled` defaults
|
||||
to `true` in the 26.8.2 CRD, both in v2alpha1 and in v2beta1 (v2beta1 is now the storage version).
|
||||
That is the same name and namespace as the repo's Flux-owned `platform/keycloak/servicemonitor.yaml`.
|
||||
After the 26.0.6 -> 26.8.2 bump (#121/#144) the operator's PATCH against it returned
|
||||
`409 Conflict` and it ran out of retries. The CR then sat at `Ready=False`/`HasErrors=True`, and
|
||||
`platform-config` (`wait: true`, which health-checks every object it applies, this CR included)
|
||||
blocked everything that `dependsOn` it: velero, apps-vms, apps-wave-0 (and so waves 1/2 and
|
||||
pm-agent), homepage, oauth2-proxy, headlamp, conformance, vpn-ingress, crafty, runbooks.
|
||||
- `platform/keycloak/servicemonitor.yaml`: a comment saying it is the sole owner.
|
||||
- **Why ours is kept:** these are written against it: the L0 `servicemonitor-ports-resolve`
|
||||
allow-list entry `platform/keycloak/servicemonitor.yaml:keycloak`, the L2 `keycloak-monitored`
|
||||
chainsaw test, the README's ownership table, and the dashboard's `up{namespace="keycloak"}`.
|
||||
The operator's version (upstream `KeycloakServiceMonitorDependentResource` @26.8.2) scrapes the same
|
||||
target: port name `management` (9000 on `keycloak-service`), path `/metrics`, interval 30s,
|
||||
selector = the operator's instance labels. It offers nothing ours lacks, so having two owners
|
||||
would add a fight and gain nothing.
|
||||
|
||||
## What I verified
|
||||
|
||||
- Live, read-only, `k3s-prod` ~06:25Z: the CRD schema field `spec.serviceMonitor.enabled`
|
||||
(`default: true`) exists in both served versions. The live `keycloak` ServiceMonitor has exactly
|
||||
**one** managedFields manager, `kustomize-controller` (Apply, 2026-09-05), **no ownerReferences**,
|
||||
and labels `kustomize.toolkit.fluxcd.io/name=platform-config`, `release=kube-prometheus-stack`. So
|
||||
the operator's apply never landed, and nothing on the object ties it to the Keycloak CR.
|
||||
- The ServiceMonitor will not be deleted when the operator is switched off: the operator's dependent
|
||||
informer is filtered on `Constants.DEFAULT_LABELS_AS_STRING`
|
||||
(`app.kubernetes.io/managed-by=keycloak-operator`). Our object does not carry that label and has
|
||||
no ownerReference, so the operator cannot see it as its secondary and cannot delete or GC it.
|
||||
This conclusion comes from reading the code. It was not observed.
|
||||
- The other operator dependents in ns `keycloak` do not collide with anything: `keycloak-service`,
|
||||
`keycloak-discovery`, the StatefulSet `keycloak`, `keycloak-initial-admin`, and
|
||||
`keycloak-network-policy` (owner Keycloak, no Flux label). The repo declares none of these names.
|
||||
It only *references* `keycloak-service` by name (HTTPRoute, reconcilers). There are no PDBs in the
|
||||
namespace, and the Ingress is already `enabled: false`. The only NetworkPolicy in the namespace is
|
||||
the operator's. The repo has no Keycloak NetworkPolicy, and no Kyverno-generated default-deny is
|
||||
present there. NetworkPolicies add up, so the operator's allow-policy cannot fight another owner.
|
||||
Left at its default.
|
||||
- Local gate: see the PR body. It records the rc and stages.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- That the CR reaches `Ready=True`/`HasErrors=False` after merge. The status also says
|
||||
"Performing Keycloak upgrade, scaling down the deployment". I am inferring that this half clears
|
||||
once the workflow stops erroring. It is not proven.
|
||||
- That the operator stops issuing the PATCH once `enabled: false` is applied. Watch the operator
|
||||
logs for 409.
|
||||
- That `platform-config` and its downstream Kustomizations go Ready on the new revision, and that
|
||||
Prometheus keeps exactly one `serviceMonitor/keycloak/keycloak/0` pool with targets up.
|
||||
- The operator's actual label set for `DEFAULT_LABELS_AS_STRING`. I took it from the observed labels
|
||||
on operator-created objects, not from `Constants.java`.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: rework/136
|
||||
title: "PR #136 rework: regenerate L1 render goldens for the oauth2-proxy v7.15.6 sidecar bump"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
- Renovate PR #136 bumps `OAUTH2_PROXY_IMAGE` in `catalog/render.py` (and the generated
|
||||
`apps/wave-*` / `platform/crafty` pins) from `v7.8.1` to `v7.15.6`. Its gate failed at L1
|
||||
because three byte-for-byte render goldens still carried the old tag:
|
||||
- `conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml`
|
||||
- `conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml`
|
||||
- `conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml`
|
||||
(CI only reported the vpn one because `render-check.sh` stops at the first failing diff;
|
||||
the internet golden had drifted too.)
|
||||
- Regenerated all three in the pinned toolbox with the exact invocation the check scripts use:
|
||||
`python3 catalog/render.py <fixture> > <golden>`. The diff in each is **one line**:
|
||||
`image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` -> `...:v7.15.6`. Nothing else moved.
|
||||
- `platform/crafty/README.md`: the "Sidecar" row described the current pin; updated to v7.15.6.
|
||||
- Deliberately NOT changed (they record history, not the current pin):
|
||||
- `catalog/render.py` comments / assertion messages saying a flag was "confirmed present in the
|
||||
PINNED v7.8.1 binary" or that `--api-routes` "does not exist in v7.8.1". Rewriting them to
|
||||
v7.15.6 would assert a `--help` verification nobody has done against v7.15.6.
|
||||
- `conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/check.sh`, the schema
|
||||
description, `runbooks/oauth2-proxy-session.md`, `docs/plans/...` — same reasoning.
|
||||
- `platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml` — v7.8.1 appears in
|
||||
synthetic kube-state-metrics series (test input data), not as a pin.
|
||||
- `.gitea/workflows/renovate.yaml` comment — a dated measurement of the pre-Renovate state.
|
||||
- Branch was 55 commits behind `origin/main`, but main touches neither `catalog/render.py` nor
|
||||
the L1 goldens, and the merge is conflict-free; no merge of main was needed for the gate.
|
||||
|
||||
## What I verified
|
||||
|
||||
- The golden diff is exactly the image tag in each of the three files (`git diff -U0`).
|
||||
- No remaining `oauth2-proxy:v7.8.1` in any rendered/generated/golden file (`git grep`).
|
||||
- Local toolbox gate result: see the PR #136 comment posted with this push (rc and per-stage
|
||||
results are quoted there and in the worker report).
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- That the v7.15.6 sidecar actually starts and authenticates in the cluster with the rendered
|
||||
args. Earlier research said no flag used here was removed/renamed between 7.8.1 and 7.15.6;
|
||||
I did not re-run `oauth2-proxy --help` against the v7.15.6 image.
|
||||
- L2/L3 conformance (does not run in CI: `CLUSTER_AVAILABLE` unset).
|
||||
- `approval-policy`: `conformance/**` is a sensitive path, so this PR is CI-red until the
|
||||
operator posts `/approve-override <reason>` as a PR review. The local gate cannot evaluate it.
|
||||
- Whether Renovate keeps rebasing this branch now that it carries a non-Renovate commit.
|
||||
@@ -0,0 +1,155 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: feat/runner-capacity-2
|
||||
title: act_runner runs two jobs at once, and dind is sized for two
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`apps/wave-1/gitea-actions.yaml` only. That file is a **hand-owned** non-catalog unit
|
||||
(`EXTRA_UNITS["gitea-actions"]` in `catalog/render.py`): the renderer only decides which wave
|
||||
it lands in, it does not generate its bytes, and `ci/render-golden.sh` compares only files the
|
||||
catalog renders. So it is edited in place, and no re-render was needed or run.
|
||||
|
||||
1. **New ConfigMap `act-runner-config`** holding a config file with ONE key,
|
||||
`runner.capacity: 2`, mounted read-only at `/etc/act_runner/` and selected with
|
||||
`CONFIG_FILE=/etc/act_runner/config.yaml`. The image's `run.sh` (act_runner v0.6.1) turns
|
||||
that into `--config` for both `register` and `daemon`.
|
||||
- Deltas only, on evidence: v0.6.1 `LoadDefault()` unmarshals the file onto a zero Config and
|
||||
then applies the same defaults as the no-file path, so the other settings are unchanged.
|
||||
- The `generate-config` example was deliberately NOT used. Its `runner.labels` would override
|
||||
the registered labels in `/data/.runner` and repoint `ubuntu-latest` at
|
||||
`docker.gitea.com/runner-images`. Its `force_pull: true` is not the default. Its sample `envs`
|
||||
would be injected into every job.
|
||||
- With a file present, act_runner ignores the legacy `GITEA_RUNNER_CAPACITY` /
|
||||
`_FILE` / `_ENVIRON` / `_ENV_FILE` / `GITEA_DEBUG` / `GITEA_TRACE` env vars. None of
|
||||
them is set on this Deployment.
|
||||
2. **Pod-template annotation `homelab.keithjackson.dev/act-runner-config-revision: "1"`.**
|
||||
act_runner reads its config once, at start, so a change to the ConfigMap alone would never
|
||||
take effect. Bump this annotation whenever you change only the ConfigMap.
|
||||
3. **dind resources:** requests `cpu 100m -> 2` and `memory 256Mi -> 1Gi`; limit
|
||||
`memory 2Gi -> 4Gi`. There is still no CPU limit, on purpose.
|
||||
4. **runner resources:** request `memory 128Mi -> 256Mi`, limit `memory 512Mi -> 1Gi`. CPU stays
|
||||
at `100m`.
|
||||
|
||||
Why these sizes. Measured 2026-11-08 through Prometheus, at capacity 1:
|
||||
|
||||
| container | peak CPU (1m rate, 30d) | peak working set (30d) | OOM events (7d) |
|
||||
|---|---|---|---|
|
||||
| dind | 0.69 core | 447 MiB | 0 |
|
||||
| runner | 0.24 core | 314 MiB | 0 |
|
||||
|
||||
At the moment of measurement the pod was idle (`top`: dind 1m/138Mi, runner 1m/83Mi), on
|
||||
k3s-prod-2.
|
||||
|
||||
- **CPU request 2.** Two jobs at today's peak is about 1.4 cores. The gate is being parallelised
|
||||
in other work, so each job will use more than that. The request sets dind's CFS weight under
|
||||
contention; at 100m it got a 1/20 share. With no limit, dind can still use spare cores on an
|
||||
idle node.
|
||||
- **Memory 1Gi / 4Gi.** Two jobs at today's peak is about 0.9 GiB. If dind is OOM-killed,
|
||||
dockerd goes down and takes both running jobs with it, so the limit is about 4x that.
|
||||
- **Runner limit 512Mi -> 1Gi.** Its 30-day peak was already 61% of the old limit, and with
|
||||
capacity 2 it holds state for two tasks.
|
||||
|
||||
Scheduling headroom: CPU requests per node were measured with `describe node`. The pod's total
|
||||
CPU request goes from 200m to 2100m (+1.9 cores).
|
||||
|
||||
| node | CPU requested now | after the change | allocatable |
|
||||
|---|---|---|---|
|
||||
| prod-2 (current) | 2945m | ~4845m (61%) | 8 |
|
||||
| prod-1 | 3175m | ~5075m | 8 |
|
||||
| prod-3 | 3840m | ~5740m (72%) | 8 |
|
||||
|
||||
The pod fits on any of the three if the RWO PVC moves it.
|
||||
|
||||
Concurrency semantics. Workflow `concurrency:` groups are enforced by Gitea on the server, not by
|
||||
the runner, so capacity does not change them:
|
||||
|
||||
- `gate.yml` `gate-${{ github.ref }}` with cancel-in-progress still cancels a superseded run on
|
||||
the same ref.
|
||||
- `renovate` (group `renovate`, no cancel) and `images-${{ github.ref }}` still serialise
|
||||
themselves.
|
||||
|
||||
What capacity 2 does change: runs on *different* refs (a PR gate next to a renovate or images
|
||||
run, or two PRs) now run side by side instead of queueing. The required check
|
||||
`gate / gate (pull_request)` is still one job per PR head, and gate.yml's other job
|
||||
(`conformance`) is `if: false`, so it never runs.
|
||||
|
||||
Both jobs share one dockerd. act_runner gives each task its own container prefix
|
||||
(`GITEA-ACTIONS-TASK-<task id>`) and its own network, and no workflow uses fixed container names
|
||||
or host ports.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **Source of truth.** `catalog/render.py` `EXTRA_UNITS` places `gitea-actions.yaml` by
|
||||
anchor/offset only. `ci/render-golden.sh` iterates over rendered files only, so this file is
|
||||
outside render==golden. No `--emit-waves` was run.
|
||||
- **act_runner v0.6.1 behaviour, read in its source** (tag v0.6.1):
|
||||
- `config.LoadDefault()` treats a partial file as described above.
|
||||
- `daemon.go`: config labels override `.runner` labels only when non-empty.
|
||||
- `poller.go`: a semaphore of `cfg.Runner.Capacity` bounds concurrent tasks.
|
||||
- `runner.go`: per-task container name prefix.
|
||||
- The image's `run.sh`, read from `gitea/act_runner:0.6.1`, honours `CONFIG_FILE`.
|
||||
- **The live pod has no config file today.** `/data` holds only `.runner` and `lost+found`.
|
||||
`act_runner --version` reports v0.6.1. No LimitRange or ResourceQuota exists in
|
||||
`gitea-actions`.
|
||||
- **Resource numbers.** All figures above come from read-only `kubectl top`, `describe node` and
|
||||
Prometheus queries against k3s-prod.
|
||||
- **dind disk.** The emptyDir sits on the node's 11.8G ephemeral filesystem: 4.0G used, 7.2G
|
||||
free. Image cache is 2.18GB, which is the one `catthehacker/ubuntu:act-22.04` image. Two
|
||||
concurrent jobs share that image.
|
||||
- **Embedded config.** It parses as YAML to `{'runner': {'capacity': 2}}`. The only `$` in the
|
||||
manifest is the existing `${BASE_DOMAIN}`, so nothing new is exposed to Flux postBuild
|
||||
envsubst.
|
||||
- **Approval policy.** `apps/wave-1/` matches no `sensitive_paths` glob in
|
||||
`infra/gitea/approval-policy.json`, so this PR merges on a green gate alone.
|
||||
- **Local gate.** Pinned toolbox, run from the worktree on the committed tree. The rc and skipped
|
||||
stages are given in the PR description.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **UNVERIFIED: that the runner actually starts with the config and runs two jobs at once.**
|
||||
Nothing is applied until Flux reconciles `main`.
|
||||
- **Capacity cannot be seen through Gitea's API.** act_runner v0.6.1 does not report capacity to
|
||||
Gitea: `Declare` sends only version and labels. `GET /api/v1/admin/actions/runners` therefore
|
||||
cannot show "capacity 2". Concurrency has to be observed instead.
|
||||
- **UNVERIFIED: whether the parallelised gate fits in these sizes.** That gate has not landed.
|
||||
Re-measure after it does, with
|
||||
`max_over_time(sum by (container)(rate(container_cpu_usage_seconds_total{namespace="gitea-actions"}[1m]))[7d:30s])`.
|
||||
- **Rollout cancels in-flight jobs.** The rollout restarts the runner pod (strategy Recreate,
|
||||
`shutdown_timeout: 0s`), and any job running at that moment is cancelled, possibly including
|
||||
the `push`-to-main gate run triggered by this very merge. Re-run it if that happens.
|
||||
- **dind disk is not bounded by a limit.** With two jobs it holds two jobs' workspaces and layers
|
||||
on the same 11.8G node filesystem. I did not add an `ephemeral-storage` limit, because an
|
||||
eviction would kill both jobs. Watch kubelet DiskPressure on the runner's node.
|
||||
- **Stale comments in sensitive paths.** `.gitea/workflows/gate.yml` (line ~52) and
|
||||
`.gitea/workflows/README.md` (line ~86) still say "`act_runner` runs at capacity 1". I left
|
||||
them alone because `.gitea/**` is a sensitive path and editing them would put this PR behind
|
||||
human approval. Fix them in a separate PR.
|
||||
|
||||
### Operator post-merge checklist
|
||||
|
||||
1. Flux applied it:
|
||||
`kubectl --context k3s-prod -n flux-system get kustomization apps-wave-1 -o jsonpath='{.status.lastAppliedRevision}'`
|
||||
equals the merge commit.
|
||||
2. The pod restarted with the config:
|
||||
- `kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner`
|
||||
shows a new pod, 2/2 Running.
|
||||
- `kubectl ... exec deploy/act-runner -c runner -- cat /etc/act_runner/config.yaml` shows
|
||||
`capacity: 2`.
|
||||
- `kubectl ... exec deploy/act-runner -c runner -- ps` shows `act_runner daemon --config /etc/act_runner/config.yaml`.
|
||||
- The logs show "declare successfully" with the two expected labels (`ubuntu-latest`,
|
||||
`homelab` -> `catthehacker/ubuntu:act-22.04`). That confirms the labels did not change.
|
||||
3. The runner is registered and online:
|
||||
- `GET /api/v1/admin/actions/runners` lists `incluster-dind`, online, with the same ID as
|
||||
before (no duplicate row).
|
||||
- Capacity is NOT shown there; see above.
|
||||
4. Concurrency is real: trigger two runs on different refs (e.g. two PR gates, or a PR gate plus a
|
||||
`workflow_dispatch` of renovate). Then either:
|
||||
- in `GET /api/v1/repos/Home-Network/home-network-k3s-impl/actions/tasks`, confirm their
|
||||
`run_started_at` .. `updated_at` windows overlap; or
|
||||
- check
|
||||
`docker --host=tcp://127.0.0.1:2376 --tlsverify ... ps` in the dind container lists two
|
||||
`GITEA-ACTIONS-TASK-*` container sets at once.
|
||||
5. Resources: `kubectl --context k3s-prod -n gitea-actions describe pod` shows dind requests
|
||||
cpu 2 / memory 1Gi, limit memory 4Gi, and runner limit memory 1Gi.
|
||||
@@ -0,0 +1,74 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: fix/wg-client-mtu-docs
|
||||
title: WireGuard client profiles default to MTU 1280; VPN runbook corrected and given a troubleshooting section
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
- **`infra/vpn-ingress/new-peer.sh`**: `WG_MTU` default 1420 -> 1280, still overridable, reason in a
|
||||
comment. Only new profiles change; the phone's existing profiles are not touched.
|
||||
- **`runbooks/vpn-remote-access.md`**: the verification table no longer says the Gateway sees a
|
||||
`host`-identity source; the limitations no longer say everything arrives SNATed or that IPv6 leaks
|
||||
on Android; the "if nothing handshakes" advice no longer claims `wg show` separates a bad forward
|
||||
from a bad key. New "Troubleshooting" section covering the four cases in the brief.
|
||||
- **`platform/vpn-ingress/daemonset.yaml`**: header comment only, SNAT claim corrected.
|
||||
- **`platform/vpn-ingress/README.md`**: one sentence, so `MTU = 1420` reads as the server's value.
|
||||
|
||||
Rollout on merge: the WireGuard DaemonSet does **not** roll. The `runbooks` ConfigMap is content
|
||||
hashed and `vpn-remote-access.md` is in it, so the runbooks site rolls.
|
||||
|
||||
Left unchanged, for the operator:
|
||||
|
||||
- **`platform/vpn-ingress/peers.conf`** still says "No peers enrolled yet". A comment-only edit on a
|
||||
throwaway copy moved the ConfigMap from `wg-peers-29mf6k867t` to `wg-peers-gggkk5d4tm` and changed
|
||||
the DaemonSet, so correcting it restarts every `wg0`. Fix it alongside the next peer change.
|
||||
- **`clusters/base/cluster-vars.yaml`** coverage comment still omits `WG_CLIENT_CIDR` (sensitive path).
|
||||
- **Server `wg0` MTU stays 1420**, because changing it restarts all WireGuard pods. For TCP a 1280
|
||||
client should be enough on its own: it advertises an MSS of 1240, which caps what the far end
|
||||
sends. That protection does not reach UDP. A server-to-phone datagram with an inner size of
|
||||
1281-1420 bytes is still sent as up to 1480 outer bytes and could be lost on the same path; QUIC
|
||||
should recover by probing, large DNS answers are the likeliest casualty. If that shows up, the
|
||||
fix is a lower server MTU, as a deliberate rollout.
|
||||
- **Other stale text, outside this brief:** `platform/vpn-ingress/README.md` still says no peer has
|
||||
ever completed a handshake, and the runbook's status line still calls the VPN inert.
|
||||
|
||||
## What I verified
|
||||
|
||||
- Upstream `wireguard-android` `GoBackend.java`, master (last commit to that file `ecc3c0a`,
|
||||
2026-03-15): `builder.setMtu(config.getInterface().getMtu().orElse(1280))`, and `allowFamily()` is
|
||||
skipped when `sawDefaultRoute && config.getPeers().size() == 1`.
|
||||
- Live, read-only, 2026-11-09 UTC, on the `k3s-prod-1` pod and agent:
|
||||
- `wg0` MTU is 1420 on all three pods. Both peers have handshakes and transfer on prod-1 only; the
|
||||
other two pods read `0`.
|
||||
- BPF conntrack holds `UDP SVC 10.90.0.x -> 192.168.2.23:53` and flows to `192.168.2.24:32400`.
|
||||
Their backend entries carry `SourceSecurityID=2` (`world`). So the source stays `10.90.0.x`
|
||||
through the load balancer on those two VIPs.
|
||||
- `/proc/net/nf_conntrack` is readable from the pod, which has no `conntrack` binary.
|
||||
- Offline, pinned toolbox: `kubectl kustomize platform/vpn-ingress` gives identical ConfigMap and
|
||||
DaemonSet hashes on `origin/main` and on this branch.
|
||||
- `new-peer.sh` writes `MTU = 1280` by default and `MTU = 1420` with `WG_MTU=1420`. Run with
|
||||
throwaway keys and stdout discarded; profiles shredded; nothing enrolled.
|
||||
- No test or conformance check pins the client MTU (`grep` for `WG_MTU` and `1420`).
|
||||
- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not green.**
|
||||
`gitea-settings` is the only stage counted as not run (no token locally). `conformance` (L2/L3)
|
||||
also skipped as the declared no-cluster gap, so R-ENV-01 and R-OBS-04 are not certified.
|
||||
`spec-immutability`, `approval-policy`, `decisions-log`, `lint`, `l0`, `render-golden` and `l1`
|
||||
passed. No sensitive path is touched.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That MTU was the cause.** It is inferred from one before-and-after, on one phone and one carrier,
|
||||
reported by the operator. The path MTU was never measured.
|
||||
- **That 1280 is the best value.** It is the value that worked, and the IPv6 minimum. A higher one
|
||||
may work too.
|
||||
- That the phone's app is on `GoBackend` rather than the kernel backend, so that a blank field meant
|
||||
1280 on this device. Only the upstream default was read.
|
||||
- That TCP MSS behaves as described, or that large UDP replies are actually lost. No packet capture
|
||||
was taken.
|
||||
- That the Gateway VIP `.20` sees `10.90.0.x` / `world`. Conntrack held no tunnel flows to it when
|
||||
checked.
|
||||
- Android's IPv6 blocking on a real device; read from source only.
|
||||
- `wg-quick` IPv6 behaviour on Linux, and any non-Android client at 1280.
|
||||
- `/proc/net/nf_conntrack` output during a key mismatch. The `[UNREPLIED]` reading comes from the
|
||||
earlier measurement in the brief; I saw only healthy entries.
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
date: 2026-11-08
|
||||
branch: fix/wg-home-peer-pihole-sources
|
||||
title: Enrol keith-phone-home and admit the WireGuard subnet to PiHole's DNS frontends
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
- **`platform/vpn-ingress/peers.conf`**: new peer `keith-phone-home`, `10.90.0.3/32`, key generated
|
||||
on the phone. It is the `home` half of a two-tunnel profile (Endpoint `192.168.2.10:51820`),
|
||||
because the router has no hairpin NAT and `wan.` is unusable from inside the house.
|
||||
- **`apps/wave-0/pihole-dns.yaml`**: `${WG_CLIENT_CIDR}` added after `${LAN_CIDR}` on both
|
||||
`pihole-dns-udp` and `pihole-dns-tcp`. This file is hand-written, not rendered (`catalog/render.py`
|
||||
only lists it in `EXTRA_UNITS`), so it is the source of truth and `--emit-waves` was not run.
|
||||
`catalog/pihole.yaml` changes in one comment only, so nothing renders differently.
|
||||
- **New L0 guard `conformance/tests/l0-static/pihole-dns-source-ranges`** (whole-repo scope,
|
||||
R-DNS-03): every PiHole LoadBalancer Service lists both tokens, and every range resolves through
|
||||
cluster-vars (base and prod overlay) to RFC1918.
|
||||
- **`platform/vpn-ingress/README.md`**: corrected "clients arrive SNATed" (false for LoadBalancer
|
||||
VIPs) and the IPv6 claim (Android's `GoBackend` blocks IPv6 for a single-peer default route);
|
||||
documented the home/away pattern, why `wan.` fails from inside, and the prod-1 dependency.
|
||||
|
||||
Rollout on merge: the `wg-peers` ConfigMap hash changes (`m9kcckcb5t` -> `29mf6k867t`), so the
|
||||
DaemonSet rolls, `maxUnavailable: 1`, `maxSurge: 0`, one node at a time in an order the controller
|
||||
chooses. Each roll deletes and recreates that node's `wg0`, so an established session on that node
|
||||
drops and must re-handshake. `keith-phone` has no session today (see below), so nothing is
|
||||
interrupted in practice.
|
||||
|
||||
**The one risk worth watching:** `wg0` is in Cilium's auto-detected device list and `devices` is
|
||||
still unpinned. On 2026-10-06 a crash-looping `wg0` churned that list and all three nodes lost their
|
||||
main-table default route. A rollout is one delete/create per node, not a loop, so it should not
|
||||
repeat that, but after merge check `ip route get 1.1.1.1` on each node.
|
||||
|
||||
The Service change is an in-place update: same VIP, same Service objects, no pod restart.
|
||||
|
||||
Checked on the rest of the path and left unchanged:
|
||||
|
||||
- Gateway VIP `.20` (`gateway/cilium-gateway-shared`): no source ranges, live. Nothing needed.
|
||||
- `pihole-allow-lan-dns-ingress` admits `world` on :53, which covers an untranslated 10.90.0.x
|
||||
source. `platform/vpn-ingress/networkpolicy.yaml` does not apply to a hostNetwork pod.
|
||||
- No L0 check pinned these ranges to the LAN; none needed updating.
|
||||
|
||||
Flagged, not changed (out of scope):
|
||||
|
||||
- `runbooks/vpn-remote-access.md` "Known limitations" repeats both corrected claims, and step 3 of
|
||||
its verification table says the Gateway sees a `host`-identity source.
|
||||
- `platform/vpn-ingress/daemonset.yaml`'s header repeats the SNAT claim.
|
||||
- `clusters/base/cluster-vars.yaml`'s substitution-coverage comment does not list
|
||||
`WG_CLIENT_CIDR` under apps-wave-0. The variable is defined and substituted there regardless.
|
||||
- `peers.conf` still says "No peers enrolled yet" above the enrolled peers.
|
||||
- PiHole's NodePorts are not covered by source ranges (already acknowledged in `pihole-dns.yaml`).
|
||||
Tunnel clients can reach node addresses, so they can reach those too; they are enrolled devices.
|
||||
- Pinning Cilium's `devices` remains the open hardening for the churn risk above.
|
||||
|
||||
## What I verified
|
||||
|
||||
Live, read-only, 2026-11-08:
|
||||
|
||||
- `cilium-dbg status` on one agent: `KubeProxyReplacement: True [eno1 ..., wg0 10.90.0.1, tun0 ...]`
|
||||
and `Masquerading: BPF [eno1, wg0, tun0]`. So `wg0` really is a load-balancing device.
|
||||
- `pihole-dns-tcp` and `-udp` carry `ranges=[192.168.2.0/24]`; no other LoadBalancer Service has
|
||||
source ranges. Live `cluster-vars`: `WG_CLIENT_CIDR=10.90.0.0/24`, `LAN_CIDR=192.168.2.0/24`.
|
||||
- `wg show wg0 latest-handshakes` on all three pods: `keith-phone` = `0`.
|
||||
- `wan.home.keithjackson.dev` resolves publicly to a single A record (via 1.1.1.1).
|
||||
|
||||
Offline, in the pinned toolbox:
|
||||
|
||||
- `flux envsubst --strict` renders both Services to `[192.168.2.0/24, 10.90.0.0/24]`.
|
||||
- `kubectl kustomize platform/vpn-ingress`: `wg-peers-m9kcckcb5t` on the old `peers.conf`,
|
||||
`wg-peers-29mf6k867t` on the new one.
|
||||
- Guard mutation test, each restored afterwards. Baseline passes (22 assertions, 2 Services). All
|
||||
seven fail with rc 1 and a named message: WG range dropped from one Service; `0.0.0.0/0` added;
|
||||
`8.8.8.0/24` added; `WG_CLIENT_CIDR` made public in cluster-vars; prod overlay sets `LAN_CIDR` to
|
||||
`0.0.0.0/0`; ranges block removed; undefined token added.
|
||||
- The Android claim was read from upstream `GoBackend.java` (`wireguard-android`, master): the
|
||||
`allowFamily` calls are skipped when `sawDefaultRoute && config.getPeers().size() == 1`.
|
||||
- Gate (pinned toolbox, from this worktree, on the committed tree): **rc 3, UNDETERMINED, not
|
||||
green.** `gitea-settings` is the only stage counted as not run (no token locally). `conformance`
|
||||
(L2/L3) also skipped as the declared no-cluster gap. `spec-immutability`, `decisions-log`, `lint`,
|
||||
`l0` (including the new guard), `render-golden` and `l1` passed. `approval-policy` reported N/A
|
||||
offline; it will be red in CI until the operator overrides, because `conformance/**` is sensitive.
|
||||
- `git merge-tree` against `feat/pihole-explicit-lan-names` (#157): no conflicts.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **`DROP_NOT_IN_SRC_RANGE` was NOT observed live.** It is a prediction from Cilium's source. Only
|
||||
the `wg0` device membership was measured; the drop itself has never been seen.
|
||||
- **Full-tunnel behaviour was NOT observed live**, on either profile. **No handshake has ever been
|
||||
recorded for any peer**: `keith-phone` reads `0` on all three pods (a counter that resets when a
|
||||
pod restarts, last 22 days ago, so it covers only the time since; earlier entries here also
|
||||
record none).
|
||||
- That tunnel DNS works after this change, including the reply path from a backend on another node
|
||||
and from one on the ingress node. That needs the phone: `dig` over the tunnel, while watching
|
||||
`cilium-dbg monitor --type drop`.
|
||||
- That the Gateway serves tunnel clients. Its source is predicted to be `world`, as for a LAN
|
||||
browser, which is what the apps admit; not tested.
|
||||
- That the new peer's key is the one on the phone. I copied the public key from the brief.
|
||||
- That the rollout leaves node default routes intact. Reasoned from the 2026-10-06 incident being
|
||||
a loop, not measured.
|
||||
- That a LAN client is unaffected while the Service's source ranges are updated. Expected, not
|
||||
measured.
|
||||
- IPv6 behaviour on any non-Android client.
|
||||
- That the WAN forward targets prod-1: it is off-repo.
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
date: 2026-11-09
|
||||
branch: feat/pihole-explicit-lan-names
|
||||
title: PiHole answers an explicit LAN name list instead of a zone wildcard (public names resolve to Cloudflare from the LAN)
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**`catalog/pihole.yaml` → `apps/wave-0/pihole.yaml`.** `FTLCONF_misc_dnsmasq_lines` no longer carries
|
||||
`address=/${BASE_DOMAIN}/${GATEWAY_VIP}`. Every name not listed is now forwarded to 8.8.8.8/8.8.4.4 and
|
||||
gets the public answer. The final list (`local-ttl=60` kept):
|
||||
|
||||
| Name (`.${BASE_DOMAIN}`) | Answer | Why |
|
||||
|---|---|---|
|
||||
| jackett, pihole, plex, qbittorrent, radarr, semaphore, sonarr, headlamp | `${GATEWAY_VIP}` | LAN-only: every HTTPRoute with `external-dns.alpha.kubernetes.io/controller: none`, derived from the tree. No public record, tunnel 404s them; this entry is their only path |
|
||||
| gitea, nextcloud | `${GATEWAY_VIP}` | public, but Cloudflare's proxy caps bodies at 100 MB / responses at 100 s (git push, LFS, uploads) |
|
||||
| keycloak, auth | `${GATEWAY_VIP}` | keep the OIDC flow of the LAN-path apps on one path |
|
||||
| unifi | `${UNIFI_L4_VIP}` | L4 VIP, not behind the Gateway |
|
||||
| crafty-mc | `${CRAFTY_L4_VIP}` | L4 VIP, not behind the Gateway |
|
||||
| pihole-dns | `${PIHOLE_DNS_VIP}` | the resolver's own address |
|
||||
| plex-vip | `${PLEX_L4_VIP}` | kept, see below |
|
||||
|
||||
Not listed, so they resolve to Cloudflare from the LAN: the apex, grafana, crafty, seerr.
|
||||
|
||||
**The derived LAN-only set matches the brief's list exactly** (8 names; the tree was walked, not the
|
||||
list trusted). The `runbooks/serving` route has no `hostnames:` but is not disowned (it inherits the
|
||||
apex, public).
|
||||
|
||||
**plex-vip is kept.** Nothing advertises it any more (`ADVERTISE_IP` is a `plex.direct` name since
|
||||
2026-10-05), but it is not unused: it is Plex's `customCertificateDomain` (`CERTDOMAIN`, Secret
|
||||
`plex-vip-tls`), the name of the tcp/32400 VIP, and the 404'd name in the tunnel config. It has no
|
||||
public record, so this entry is its only resolution.
|
||||
|
||||
**New L0 check `conformance/tests/l0-static/pihole-lan-names`** (`scope: whole-repo`, so picked up by
|
||||
the runner's scope selection with no other wiring). Asserts: (a) no zone-wide entry (zone, `#`, or any
|
||||
parent of the zone); (b) every `controller: none` hostname → `${GATEWAY_VIP}`, and a disowned route
|
||||
with no `hostnames:` fails; (c) every tunnel-404 hostname has an entry, with the L4 names (derived
|
||||
from Services carrying both the external-dns hostname and an lbipam VIP) on their own VIP; (d) gitea
|
||||
and nextcloud → `${GATEWAY_VIP}`; plus catalog == rendered value and a built-in mutation self-test.
|
||||
|
||||
**Stale docs corrected:** `platform/pm-agent/README.md`, `runbooks/pm-agent.md`,
|
||||
`platform/externaldns/README.md`, `platform/cloudflared/README.md`, `platform/gateway/gateway.yaml`
|
||||
(comment), `platform/headlamp/httproute.yaml` (comment), `apps/vms/unifi/README.md`, `README.md`,
|
||||
`schemas/renderer-contract.md`; `runbooks/cloudflare-tunnel.md` gains the two consequences below.
|
||||
`platform/cloudflared/deployment.yaml`: the image entrypoint comment was wrong (comment-only).
|
||||
|
||||
## What I verified
|
||||
|
||||
- **Render:** the baseline `render.py --no-image-automation catalog/pihole.yaml` was byte-identical to
|
||||
the committed `apps/wave-0/pihole.yaml` before the change, and the file was replaced by that
|
||||
command's output after it; `--emit-waves` was not run.
|
||||
- **Mutation, against the real files (10 mutants, each rc 1 with the right assertion, each restored):**
|
||||
(a) zone wildcard restored; (b) headlamp and sonarr entries removed; (b) a disowned route with no
|
||||
`hostnames:`; (c) a new tunnel-404 host with no entry; (c) pihole-dns removed; (c) unifi pointed at
|
||||
the Gateway; (d) gitea removed; (d) nextcloud repointed; (0) rendered value drifted from the catalog.
|
||||
- **Rollout mechanics, measured live (read-only):** StatefulSet `updateStrategy` RollingUpdate
|
||||
partition 0, `podManagementPolicy: Parallel`, no `minReadySeconds`, 2 replicas, PDB minAvailable 1.
|
||||
The last rollout (2026-11-08 06:47–06:48Z) replaced pihole-1 then pihole-0: FTL logged "started" ~1 s
|
||||
after the container, the pod went Ready ~16 s after the container, and pihole-0 started ~9 s later.
|
||||
- **Expected LAN DNS blip:** no total outage; one replica answers throughout. Each of the two pod
|
||||
replacements can lose a few UDP queries for ~1–2 s while the terminating pod leaves the Service
|
||||
(clients retry). For roughly 30–60 s the two pods answer differently for the apex, grafana, crafty
|
||||
and seerr; both answers work. Cached answers age out in 60 s (`local-ttl`).
|
||||
- **Live DNS before the change:** @8.8.8.8 returns Cloudflare (104.21.51.91 / 172.67.178.27) for the
|
||||
apex, grafana, seerr, crafty and gitea, and nothing for jackett; @192.168.2.23 returns 192.168.2.20
|
||||
for all of them (the wildcard).
|
||||
- **Cloudflared image:** `skopeo inspect --config` on the pinned digest (amd64): Entrypoint
|
||||
`["cloudflared"]`, Cmd `["version"]`, User `65532:65532`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The post-merge behaviour.** Nothing here ran against a live PiHole with the new list; the
|
||||
expected answers are from dnsmasq semantics and the tree. After reconcile, check
|
||||
`dig @192.168.2.23` for one name per row, and grafana returning Cloudflare.
|
||||
- **Whether every LAN client path works via Cloudflare.** Measured 2xx/3xx through Cloudflare by the
|
||||
brief, not re-measured from a LAN client here.
|
||||
- **The FTL DNS-listening moment.** "FTL started" is a log line, not a successful query; the rollout
|
||||
timing above assumes DNS is up shortly after it.
|
||||
|
||||
**Accepted consequences, stated plainly:**
|
||||
- The apex, grafana, crafty and seerr now depend on the internet and the tunnel from the LAN too. An
|
||||
ISP or Cloudflare outage, or stopping the connectors, makes them unreachable from the LAN where they
|
||||
previously kept working. If grafana in particular should survive an internet outage, add it to the
|
||||
list (one entry; the check permits it).
|
||||
- The tunnel rollback (revert to `wan.`) now also requires restoring the wildcard; recorded in
|
||||
`runbooks/cloudflare-tunnel.md`.
|
||||
|
||||
**Flagged, not changed:**
|
||||
- The `pihole` container has no readiness probe; only the `oauth2-proxy` sidecar does, so "Ready" says
|
||||
nothing about DNS. The measured margin (FTL ~1 s, Ready ~16 s) makes this benign in a normal rollout
|
||||
but a slow FTL start would not hold the rollout back. A DNS readiness probe needs a schema/renderer
|
||||
decision.
|
||||
|
||||
**Stale statements left alone:** `platform/coredns/coredns-custom.yaml` header ("both resolve the whole
|
||||
zone ... by wildcard", brief said not to touch CoreDNS); `clusters/base/cluster-vars.yaml` comments
|
||||
(sensitive path); `schemas/renderer-contract.md` §`exposure` first bullets (manual edge port-forward
|
||||
wording); historical `infra/PROD-PROVISIONING.md`, `platform/externaldns/README.md` TODO list.
|
||||
@@ -53,6 +53,7 @@
|
||||
# WG_ENDPOINT_HOST default wan.home.keithjackson.dev
|
||||
# WG_CLIENT_CIDR default 10.90.0.0/24 (must match clusters/base/cluster-vars.yaml)
|
||||
# WG_LAN_CIDR default 192.168.2.0/24 (prod LAN_CIDR)
|
||||
# WG_MTU default 1280 (client tunnel MTU; see the comment on the default below)
|
||||
# WG_DNS default 192.168.2.23 (prod PIHOLE_DNS_VIP)
|
||||
#
|
||||
# REQUIRES: `wg`, `qrencode`, `sops` — all three in the dev container.
|
||||
@@ -80,7 +81,14 @@ PEERS_FILE="${REPO_ROOT}/platform/vpn-ingress/peers.conf"
|
||||
# cannot resolve until it is already connected.
|
||||
: "${WG_ENDPOINT_HOST:=wan.home.keithjackson.dev}"
|
||||
: "${WG_LISTEN_PORT:=51820}"
|
||||
: "${WG_MTU:=1420}"
|
||||
# The CLIENT's tunnel MTU. Deliberately lower than the server's 1420 (daemonset.yaml), which only
|
||||
# describes the 1500-byte path at the node's end. A phone's path is the carrier's, often behind
|
||||
# carrier NAT, and is not known to carry 1500. Measured 2026-11-09, one Android phone on one
|
||||
# carrier: at 1420 the handshake was healthy but names and pages failed; clearing the app's MTU
|
||||
# field (its userspace backend then uses 1280) fixed it. 1280 is also the IPv6 minimum link MTU, so
|
||||
# it stays valid if the tunnel ever gains IPv6. For a device only ever on a wired or known-1500
|
||||
# path, WG_MTU=1420 recovers about 10% of per-packet payload.
|
||||
: "${WG_MTU:=1280}"
|
||||
|
||||
SPLIT=0
|
||||
OUT_DIR=""
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
# platform/cloudflared
|
||||
|
||||
**Cloudflare Tunnel connector**: inbound HTTPS from the Cloudflare edge to the shared Gateway,
|
||||
outbound-only from the cluster (no router port forward, no hairpin, no split-DNS dependency).
|
||||
|
||||
- **Flux tier:** `clusters/base/cloudflared.yaml` — its own Kustomization, `dependsOn: [platform-config]`,
|
||||
`wait: false`, and nothing depends on it. A missing or invalid credential can affect only the tunnel.
|
||||
- **Runbook (enrolment, canary, alerts, rollback):** `runbooks/cloudflare-tunnel.md`
|
||||
- **Design record:** `decisions/2026-11-08-cloudflare-tunnel.md`
|
||||
- **Guard:** `conformance/tests/l0-static/cloudflared-ingress-rules`
|
||||
|
||||
| File | What it is |
|
||||
|---|---|
|
||||
| `config.yaml` | The locally-managed tunnel: literal tunnel UUID + ingress rules. LAN/VPN-only hosts → 404, apex + wildcard → `https://cilium-gateway-shared.gateway.svc.cluster.local:443` with `matchSNItoHost`, catch-all 404. Never the Kubernetes API. |
|
||||
| `cloudflared-credentials.sops.yaml` | SOPS-encrypted tunnel credentials JSON. Ships with `REPLACE-ME`; the operator fills it. |
|
||||
| `deployment.yaml` | 2 replicas, hard spread across nodes, non-root, read-only root fs, pinned digest; PDB `minAvailable: 1`. |
|
||||
| `networkpolicy.yaml` | default-deny (single owner — not an app namespace) + Prometheus scrape allow + egress CNP: world 7844 tcp/udp and 443, the Gateway (`ingress` entity), and Gateway backends by `gateway-access=shared` minus the LAN-only namespaces. |
|
||||
| `servicemonitor.yaml`, `alerts.yaml` | Headless metrics Service + ServiceMonitor scraping :2000, and the two alerts, both written to fire (not go quiet) when their series vanish. |
|
||||
|
||||
Cloudflare sends a hostname through the tunnel only when its DNS record is a proxied CNAME to
|
||||
`<UUID>.cfargotunnel.com`; nothing in this directory creates one. Those records are published by
|
||||
external-dns from the shared Gateway's target annotation (Phase 2, PR #156; see
|
||||
`platform/externaldns/README.md`). LAN resolution: `catalog/pihole.yaml` (explicit names only since
|
||||
2026-11-09), so public names reach Cloudflare from the LAN too.
|
||||
@@ -0,0 +1,128 @@
|
||||
---
|
||||
# promtool unit tests for platform/cloudflared/alerts.yaml.
|
||||
#
|
||||
# Found by platform/monitoring/tests/run.sh through the COMPANION convention
|
||||
# (`<manifest>.promtool-test.yaml` beside `<manifest>.yaml`). Run just this file with
|
||||
# platform/monitoring/tests/run.sh platform/cloudflared/alerts.promtool-test.yaml
|
||||
# The harness flattens the PrometheusRule's `.spec.groups` into alerts.rules.yaml. Assertions query
|
||||
# the synthetic ALERTS series (not exp_alerts), so annotation wording is not pinned here.
|
||||
#
|
||||
# Each rule is pinned BOTH ways, plus the two self-silencing traps:
|
||||
# * series ABSENT (nothing scraped) must FIRE, not read as healthy;
|
||||
# * working-then-broken must RE-FIRE, not latch cleared.
|
||||
rule_files:
|
||||
- alerts.rules.yaml
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
- interval: 1m
|
||||
name: healthy - two replicas, four edge connections each, both rules quiet
|
||||
input_series:
|
||||
- series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}'
|
||||
values: '4x60'
|
||||
- series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-b"}'
|
||||
values: '4x60'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}'
|
||||
values: '2x60'
|
||||
promql_expr_test:
|
||||
- expr: 'count(ALERTS{alertname=~"Cloudflared.*"})'
|
||||
eval_time: 50m
|
||||
exp_samples: []
|
||||
|
||||
- interval: 1m
|
||||
name: nothing scraped at all - TunnelDown still fires (no self-silencing on absence)
|
||||
input_series:
|
||||
# kube-state-metrics says both pods are Available, but no cloudflared series exists — the
|
||||
# broken-ServiceMonitor / dropped-scrape case. The rule must not read this as healthy.
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}'
|
||||
values: '2x60'
|
||||
promql_expr_test:
|
||||
- expr: 'ALERTS{alertname="CloudflaredTunnelDown"}'
|
||||
eval_time: 5m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="pending", severity="warning"}'
|
||||
value: 1
|
||||
- expr: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing"}'
|
||||
eval_time: 12m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing", severity="warning"}'
|
||||
value: 1
|
||||
- expr: 'ALERTS{alertname="CloudflaredReplicasDegraded"}'
|
||||
eval_time: 30m
|
||||
exp_samples: []
|
||||
|
||||
- interval: 1m
|
||||
name: scraped but zero edge connections - TunnelDown fires after 10m
|
||||
input_series:
|
||||
- series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}'
|
||||
values: '0x60'
|
||||
- series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-b"}'
|
||||
values: '0x60'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}'
|
||||
values: '0x60'
|
||||
promql_expr_test:
|
||||
- expr: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing"}'
|
||||
eval_time: 9m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing"}'
|
||||
eval_time: 12m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing", severity="warning"}'
|
||||
value: 1
|
||||
- expr: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing"}'
|
||||
eval_time: 17m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing", severity="warning"}'
|
||||
value: 1
|
||||
|
||||
- interval: 1m
|
||||
name: working then the series vanish - TunnelDown re-fires instead of latching cleared
|
||||
input_series:
|
||||
# 20 healthy samples, then the series stop (stale) — the scrape broke or the pods died.
|
||||
- series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}'
|
||||
values: '4x20 stale'
|
||||
- series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-b"}'
|
||||
values: '4x20 stale'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}'
|
||||
values: '2x60'
|
||||
promql_expr_test:
|
||||
- expr: 'ALERTS{alertname="CloudflaredTunnelDown"}'
|
||||
eval_time: 15m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing"}'
|
||||
eval_time: 35m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing", severity="warning"}'
|
||||
value: 1
|
||||
|
||||
- interval: 1m
|
||||
name: one replica lost - ReplicasDegraded fires after 15m, tunnel itself still up
|
||||
input_series:
|
||||
- series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}'
|
||||
values: '4x60'
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}'
|
||||
values: '1x60'
|
||||
promql_expr_test:
|
||||
- expr: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing"}'
|
||||
eval_time: 14m
|
||||
exp_samples: []
|
||||
- expr: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing"}'
|
||||
eval_time: 17m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing", severity="warning"}'
|
||||
value: 1
|
||||
- expr: 'ALERTS{alertname="CloudflaredTunnelDown"}'
|
||||
eval_time: 30m
|
||||
exp_samples: []
|
||||
|
||||
- interval: 1m
|
||||
name: kube-state-metrics series absent - ReplicasDegraded fires rather than going quiet
|
||||
input_series:
|
||||
- series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}'
|
||||
values: '4x60'
|
||||
promql_expr_test:
|
||||
- expr: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing"}'
|
||||
eval_time: 17m
|
||||
exp_samples:
|
||||
- labels: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing", severity="warning"}'
|
||||
value: 1
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
# Alert rules for the Cloudflare Tunnel connector.
|
||||
#
|
||||
# THE DESIGN CONSTRAINT: NEITHER RULE MAY GO QUIET WHEN ITS OWN DATA DISAPPEARS. The repo has
|
||||
# shipped self-silencing rules before (a bare `up{}` that clears exactly when scraping breaks), so:
|
||||
#
|
||||
# * CloudflaredTunnelDown reads `cloudflared_tunnel_ha_connections` — the per-replica gauge of
|
||||
# live edge connections — and falls back to `vector(0)` when NO such series exists. A dead
|
||||
# tunnel, zero pods, a broken ServiceMonitor, or a NetworkPolicy that drops the scrape all make the
|
||||
# series absent, and all of them FIRE rather than read as healthy.
|
||||
# * CloudflaredReplicasDegraded reads kube-state-metrics, which does not depend on cloudflared
|
||||
# being scrapable at all, and has the same `vector(0)` fallback.
|
||||
#
|
||||
# Both directions are pinned by the companion alerts.promtool-test.yaml (healthy is quiet; absent
|
||||
# series fires; working-then-broken re-fires).
|
||||
#
|
||||
# NOTE FOR PHASE 1: until the operator completes runbooks/cloudflare-tunnel.md steps 1-5 the
|
||||
# Deployment cannot connect and CloudflaredTunnelDown fires. That is the honest state — the tunnel
|
||||
# is not serving — and nothing routes through it yet, which is why both rules are `warning`.
|
||||
# Raise TunnelDown to `critical` in the change that points public DNS at the tunnel.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: cloudflared
|
||||
namespace: cloudflared
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
spec:
|
||||
groups:
|
||||
- name: platform-health.cloudflare-tunnel
|
||||
rules:
|
||||
- alert: CloudflaredTunnelDown
|
||||
# Sum of live edge connections across every replica. `or on() vector(0)` turns an EMPTY
|
||||
# result (nothing scraped) into 0 so it fires instead of evaluating to no series.
|
||||
expr: (sum(cloudflared_tunnel_ha_connections{namespace="cloudflared"}) or on() vector(0)) < 1
|
||||
# 10m rides out a rolling update and a Cloudflare edge reconnect, not a real outage.
|
||||
for: 10m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: Cloudflare Tunnel has no healthy edge connections
|
||||
description: >-
|
||||
No cloudflared replica reports a live connection to the Cloudflare edge (or none is
|
||||
being scraped at all). Every hostname routed through the tunnel is unreachable from
|
||||
the internet. Before the operator has enrolled the tunnel this is expected.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cloudflare-tunnel"
|
||||
|
||||
- alert: CloudflaredReplicasDegraded
|
||||
# Fewer than the 2 replicas deployment.yaml asks for are Available — the tunnel has lost
|
||||
# its HA, and the next node drain or reboot takes it down. Keep `2` in step with
|
||||
# spec.replicas. Absent series (kube-state-metrics gone, Deployment deleted) -> 0 -> fires.
|
||||
expr: >-
|
||||
(max(kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"})
|
||||
or on() vector(0)) < 2
|
||||
for: 15m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: cloudflared has fewer than 2 available replicas
|
||||
description: >-
|
||||
The cloudflared Deployment has fewer than two Available replicas, so the tunnel has
|
||||
no redundancy (or is down entirely). Check for unenrolled credentials, a node that
|
||||
cannot schedule the hard-spread pod, or an image pull failure.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/cloudflare-tunnel"
|
||||
@@ -0,0 +1,32 @@
|
||||
# Cloudflare Tunnel credentials for platform/cloudflared — SOPS-ENCRYPTED (R-SEC-01, spec docs/07).
|
||||
# Keys stay cleartext for reviewable diffs; only `stringData` values are encrypted (.sops.yaml
|
||||
# `encrypted_regex`). Plaintext template + operator hand-off: cloudflared-credentials.sops.yaml.example.
|
||||
# Shipped with REPLACE-ME placeholders; runbooks/cloudflare-tunnel.md step 4 fills it.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: cloudflared-credentials
|
||||
namespace: cloudflared
|
||||
type: Opaque
|
||||
stringData:
|
||||
credentials.json: ENC[AES256_GCM,data:68G438/jsM4KqPt/3wjFp6jVmAVnsidx/FEdK6Lke2euQgkDqRcJjhoKZurapgP8zsP+FrQ98VgsQgUpERHXUCRuJp5wRIgTVmL23CweuZLacaLxBo6eoKmNC/H5DzGZsyhDxGjcXc+8BGpfDFzHR39Uo2h56XstiHtyZhnhlhia8nprLip9RinA9mbKRMJKTR/6IsZsdc91E22aEycDjZ3+nYmr4KP2+km3fd15Nw==,iv:CuAUscGmvMqT2ixFJZVzJWSvDYwbfQl79sQ+FHnXdVQ=,tag:H4fFjXPCFoxE1Mqwx0R5sQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUanIxelFEUDZtWU5CMHd5
|
||||
VHl5MkJnM1JDUWhkM2daY0dtRUlFUmNUMDJrClM5UFBsdjh2Q0NIWmxpM1djYXUy
|
||||
OFlZYVdBbURLanRRNzV1RVNmbkxWR28KLS0tIFE3d1Z1MjFZMTlaTFFIWm5ndVpN
|
||||
emhHaFpEZWpMN0JXL2NVaW5PU1ZwVW8KkCDX7ED039X12BCmiSqHr0fmZKSZmfjP
|
||||
DIiR6DvWWw1TcB3CjD5q29nIthCDf9zlrpuMfQ+9ODAlRr5t6zP+cg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-11-08T22:05:00Z"
|
||||
mac: ENC[AES256_GCM,data:FaTWQhcdYD4sDu50K9NisBO81cOwHqWKt9b/0XBvZzV5FAMNHBIi4Md3by5XT16paMuS+kgz20/+CLX1PkJ6FNbCwnp7M0+a+xtT+UT67apgJ04B6GW7HZJ+iUS6sOIIFoefSDHfKpO4bYLDIoXWupLYFlbHHRJQqQXpU7yvVgI=,iv:YDSYmgc7YUsYVeh7tPm8Sztvr3+L6xb1lww4lZdzDaY=,tag:7IektWvjJpKG98edUqNadQ==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -0,0 +1,35 @@
|
||||
# =============================================================================
|
||||
# TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
|
||||
# auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
|
||||
#
|
||||
# WHAT THIS IS: the credentials JSON of the locally-managed Cloudflare Tunnel that
|
||||
# platform/cloudflared runs. `cloudflared tunnel create` writes it; it holds the account tag, the
|
||||
# tunnel UUID and the tunnel SECRET. Anyone holding it can run a connector for this tunnel and
|
||||
# receive its traffic, so it is a real credential: SOPS-encrypted in git, never printed.
|
||||
#
|
||||
# *** THE REAL FILE IS COMMITTED WITH REPLACE-ME PLACEHOLDERS, ALREADY ENCRYPTED. ***
|
||||
# So the wiring is complete before the tunnel exists: the resource is listed in kustomization.yaml
|
||||
# and both overlays already carry the `cloudflared` decryption patch. Until you fill it, the
|
||||
# Deployment CrashLoops inside its own NON-GATING Flux Kustomization (clusters/base/cloudflared.yaml,
|
||||
# wait:false) and CloudflaredTunnelDown fires. Nothing else in the cluster is affected.
|
||||
#
|
||||
# HAND-OFF (operator — an agent must not mint a long-lived credential, CLAUDE.md §6).
|
||||
# The full procedure, with verification and rollback, is runbooks/cloudflare-tunnel.md. In short:
|
||||
# 1. `cloudflared tunnel login` + `cloudflared tunnel create --credentials-file ... home-k3s`
|
||||
# (pinned cloudflared image under podman; the toolbox does not ship cloudflared).
|
||||
# 2. Write the JSON STRAIGHT INTO the encrypted file with `sops set`, inside the toolbox — the
|
||||
# plaintext never reaches a terminal or the repo tree.
|
||||
# 3. Put the tunnel UUID (not secret) into platform/cloudflared/config.yaml `tunnel:`.
|
||||
# 4. Verify without printing (`sops -d --extract ... | jq -e ...`), then PR it.
|
||||
# =============================================================================
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: cloudflared-credentials
|
||||
namespace: cloudflared
|
||||
type: Opaque
|
||||
stringData:
|
||||
# One line of JSON, exactly as `cloudflared tunnel create` wrote it (compacted with `jq -c`):
|
||||
# {"AccountTag":"<account>","TunnelSecret":"<base64>","TunnelID":"<uuid>"}
|
||||
# TunnelID MUST equal the `tunnel:` UUID in config.yaml.
|
||||
credentials.json: REPLACE-ME
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
# cloudflared configuration for the LOCALLY-MANAGED tunnel (the ingress rules live here, in git,
|
||||
# not in the Cloudflare dashboard). Projected into the pods by the configMapGenerator in
|
||||
# kustomization.yaml as the content-hashed ConfigMap `cloudflared-config`, so editing this file
|
||||
# changes the pod template and ROLLS the Deployment (the vpn-ingress peers.conf lesson: a
|
||||
# fixed-name ConfigMap updates in place and the running process never re-reads it).
|
||||
#
|
||||
# `${BASE_DOMAIN}` is substituted by Flux postBuild (clusters/base/cloudflared.yaml). It is the ONLY
|
||||
# token in this file; do not add a bare dollar sign anywhere here.
|
||||
#
|
||||
# INVARIANTS, enforced by conformance/tests/l0-static/cloudflared-ingress-rules:
|
||||
# * every rule's `service` is either `http_status:404` or EXACTLY the shared Gateway's in-cluster
|
||||
# Service — nothing else, and in particular NEVER the Kubernetes API (operator decision
|
||||
# 2026-11-08: the API stays local-only and must never be reachable through the tunnel);
|
||||
# * the LAN/VPN-only hostnames below map to `http_status:404` and come BEFORE the wildcard;
|
||||
# * `tunnel:` is a LITERAL UUID here, not a cluster-var: the Phase 2 DNS-flip change matches
|
||||
# records against it, and keeping it out of clusters/** keeps that change off a sensitive path;
|
||||
# * the last rule is a hostname-less `http_status:404` catch-all.
|
||||
#
|
||||
# OPERATOR STEP: replace REPLACE-ME-TUNNEL-UUID with the UUID printed by `cloudflared tunnel create`
|
||||
# (runbooks/cloudflare-tunnel.md step 3). Until then cloudflared treats the value as a tunnel NAME,
|
||||
# finds no origin certificate to resolve it with, and exits — a CrashLoop in a non-gating tier.
|
||||
tunnel: fc81938b-5855-47f1-8a27-6286f56dda93
|
||||
credentials-file: /etc/cloudflared/creds/credentials.json
|
||||
# Serves /metrics (Prometheus), /ready (200 only with >=1 live edge connection) and /healthcheck.
|
||||
metrics: 0.0.0.0:2000
|
||||
no-autoupdate: true
|
||||
|
||||
ingress:
|
||||
# ── NEVER THROUGH THE TUNNEL (operator decisions 2026-11-08) ─────────────────────────────────
|
||||
# Plex stays off the tunnel; it keeps its own remote-access path (runbooks/plex-remote-access.md).
|
||||
- hostname: plex.${BASE_DOMAIN}
|
||||
service: http_status:404
|
||||
- hostname: plex-vip.${BASE_DOMAIN}
|
||||
service: http_status:404
|
||||
# LAN/VPN-only services. The six catalog `exposure: internal` apps (each carries the external-dns
|
||||
# `controller: none` route annotation live, so none has a public record today) ...
|
||||
- hostname: jackett.${BASE_DOMAIN}
|
||||
service: http_status:404
|
||||
- hostname: pihole.${BASE_DOMAIN}
|
||||
service: http_status:404
|
||||
- hostname: pihole-dns.${BASE_DOMAIN}
|
||||
service: http_status:404
|
||||
- hostname: semaphore.${BASE_DOMAIN}
|
||||
service: http_status:404
|
||||
- hostname: qbittorrent.${BASE_DOMAIN}
|
||||
service: http_status:404
|
||||
- hostname: radarr.${BASE_DOMAIN}
|
||||
service: http_status:404
|
||||
- hostname: sonarr.${BASE_DOMAIN}
|
||||
service: http_status:404
|
||||
# ... and Headlamp, the cluster dashboard (operator decision 2026-11-08: LAN/VPN-only). It holds a
|
||||
# route to the Kubernetes API on the user's behalf, which is one more reason it stays off the
|
||||
# tunnel alongside decision 4.
|
||||
- hostname: headlamp.${BASE_DOMAIN}
|
||||
service: http_status:404
|
||||
# Non-HTTP L4 VIPs (their own LoadBalancer addresses, not the Gateway). Listed so a DNS flip
|
||||
# that CNAMEs them at the tunnel by mistake gets a 404, not the Gateway's catch-all.
|
||||
- hostname: unifi.${BASE_DOMAIN}
|
||||
service: http_status:404
|
||||
- hostname: crafty-mc.${BASE_DOMAIN}
|
||||
service: http_status:404
|
||||
|
||||
# ── EVERYTHING ELSE ON THE ZONE -> THE SHARED GATEWAY, OVER TLS ──────────────────────────────
|
||||
# cilium-gateway-shared is the LoadBalancer Service Cilium creates for Gateway gateway/shared
|
||||
# (platform/gateway/gateway.yaml); its ClusterIP frontend is redirected to the Gateway's Envoy
|
||||
# exactly like the VIP. Addressed by cluster DNS, not by a ${BASE_DOMAIN} name, so the tunnel
|
||||
# does not depend on split-horizon DNS at all.
|
||||
#
|
||||
# matchSNItoHost: cloudflared sends the request's Host as the TLS SNI, so the Gateway selects the
|
||||
# right listener and presents the existing Let's Encrypt `wildcard-home-tls` certificate, which
|
||||
# cloudflared validates against the public roots for that same name. No TLS verification is
|
||||
# disabled anywhere.
|
||||
- hostname: ${BASE_DOMAIN}
|
||||
service: https://cilium-gateway-shared.gateway.svc.cluster.local:443
|
||||
originRequest:
|
||||
matchSNItoHost: true
|
||||
- hostname: "*.${BASE_DOMAIN}"
|
||||
service: https://cilium-gateway-shared.gateway.svc.cluster.local:443
|
||||
originRequest:
|
||||
matchSNItoHost: true
|
||||
|
||||
# ── CATCH-ALL (cloudflared requires the last rule to have no hostname) ───────────────────────
|
||||
- service: http_status:404
|
||||
@@ -0,0 +1,151 @@
|
||||
---
|
||||
# cloudflared — the Cloudflare Tunnel connector. Outbound-only: it dials the Cloudflare edge on
|
||||
# 7844 (QUIC, falling back to HTTP/2 over TCP) and carries requests for the tunnel's hostnames
|
||||
# back to the shared Gateway. Nothing listens on the LAN and no router port is opened.
|
||||
#
|
||||
# HA: 2 replicas, hard-spread across nodes. Each replica registers its own set of edge
|
||||
# connections for the SAME tunnel, so Cloudflare load-balances between them and losing one node
|
||||
# (a kured reboot, a drain) loses no traffic. The PDB in this directory keeps a drain from taking
|
||||
# both at once.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: cloudflared
|
||||
namespace: cloudflared
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
spec:
|
||||
replicas: 2
|
||||
revisionHistoryLimit: 3
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
# Never drop below the running count during a rollout; with 3 nodes and a hard spread the
|
||||
# surge pod always has a node to land on.
|
||||
maxUnavailable: 0
|
||||
maxSurge: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
# internet, NEVER vpn. The vpn class steers egress through the PIA tunnel (ADR-0015), which
|
||||
# would put the edge connections behind a third party's exit and its kill-switch — the
|
||||
# tunnel would drop whenever PIA does. This namespace is outside the app-namespace scope
|
||||
# of require-egress-class-label, so the label is declared for the steering, not to pass
|
||||
# admission; the L0 check cloudflared-ingress-rules pins it.
|
||||
homelab.keithjackson.dev/egress-class: internet
|
||||
spec:
|
||||
# cloudflared never talks to the Kubernetes API. No token mounted, and the egress policy
|
||||
# does not admit the kube-apiserver entity either.
|
||||
automountServiceAccountToken: false
|
||||
enableServiceLinks: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
# The image is distroless `nonroot` (65532).
|
||||
runAsUser: 65532
|
||||
runAsGroup: 65532
|
||||
# fsGroup makes the kubelet GROUP-own the Secret volume's files to 65532. Without it they
|
||||
# are root:root, and defaultMode 0440 (below) is unreadable to uid 65532 — every replica
|
||||
# CrashLooped on "couldn't read tunnel credentials ... permission denied" (2026-11-08),
|
||||
# which cloudflared then misreports as "Error locating origin cert". Same lesson as
|
||||
# platform/keycloak/client-secret-reconciler.yaml. fsGroup + 0440 keeps the credential
|
||||
# unreadable to any other uid rather than relaxing it to world-readable 0444.
|
||||
# Pinned by the L0 check cloudflared-ingress-rules, assertion (7).
|
||||
fsGroup: 65532
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
topologySpreadConstraints:
|
||||
- maxSkew: 1
|
||||
topologyKey: kubernetes.io/hostname
|
||||
whenUnsatisfiable: DoNotSchedule
|
||||
labelSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
# cloudflared drains in-flight requests for up to its own 30s grace-period on SIGTERM.
|
||||
terminationGracePeriodSeconds: 45
|
||||
containers:
|
||||
- name: cloudflared
|
||||
# Tag + digest: Renovate's `kubernetes` manager (renovate.json5, platform/**) bumps both.
|
||||
image: docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba
|
||||
# The image ENTRYPOINT is `cloudflared` and its CMD is `version` (read from the pinned
|
||||
# digest's image config, 2026-11-09); these args replace that CMD. `no-autoupdate`, the
|
||||
# tunnel UUID, credentials path, metrics address and ingress rules all come from the file.
|
||||
args:
|
||||
- tunnel
|
||||
- --config
|
||||
- /etc/cloudflared/config/config.yaml
|
||||
- run
|
||||
ports:
|
||||
- name: metrics
|
||||
containerPort: 2000
|
||||
protocol: TCP
|
||||
# /ready is 200 only while this replica holds at least one live edge connection, so an
|
||||
# unenrolled or edge-isolated replica is NotReady and the Deployment says so.
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /ready
|
||||
port: metrics
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
failureThreshold: 3
|
||||
# Liveness is deliberately NOT /ready: an edge outage would otherwise restart-loop every
|
||||
# replica, which reconnects nothing faster and churns the pods. /healthcheck answers
|
||||
# whenever the process is serving.
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthcheck
|
||||
port: metrics
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 20
|
||||
failureThreshold: 3
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 64Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 256Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /etc/cloudflared/config
|
||||
readOnly: true
|
||||
- name: creds
|
||||
mountPath: /etc/cloudflared/creds
|
||||
readOnly: true
|
||||
volumes:
|
||||
- name: config
|
||||
configMap:
|
||||
# Rewritten to the content-hashed name by kustomize (configMapGenerator).
|
||||
name: cloudflared-config
|
||||
- name: creds
|
||||
secret:
|
||||
# cloudflared-credentials.sops.yaml — the tunnel credentials JSON, by name only.
|
||||
secretName: cloudflared-credentials
|
||||
# 0440, owner root, group fsGroup 65532 (pod securityContext) — readable by the
|
||||
# container user through its group, by nobody else.
|
||||
defaultMode: 0440
|
||||
---
|
||||
# One replica may be disrupted at a time — enough for kured's one-node-at-a-time drain to
|
||||
# proceed, never enough to take the tunnel down. minAvailable 1 (not 2) on purpose: a 0-disruption
|
||||
# PDB wedges a drain (platform/kured/alerts.yaml, KuredRebootPending).
|
||||
apiVersion: policy/v1
|
||||
kind: PodDisruptionBudget
|
||||
metadata:
|
||||
name: cloudflared
|
||||
namespace: cloudflared
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
spec:
|
||||
minAvailable: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
# Cloudflare Tunnel connector. Reconciled by its OWN non-gating Flux Kustomization
|
||||
# (clusters/base/cloudflared.yaml: dependsOn platform-config, wait:false) — NOT a line in
|
||||
# platform/kustomization.yaml. See that file for why; in one line: this Deployment cannot become
|
||||
# Ready until the operator enrols the tunnel, and in a gating tier that would stall every app wave.
|
||||
apiVersion: kustomize.config.k8s.io/v1beta1
|
||||
kind: Kustomization
|
||||
# REQUIRED for the configMapGenerator: a generated ConfigMap inherits no namespace, and kustomize
|
||||
# will not rewrite the Deployment's reference to its hashed name across a namespace boundary
|
||||
# (measured on platform/vpn-ingress).
|
||||
namespace: cloudflared
|
||||
resources:
|
||||
- namespace.yaml
|
||||
- deployment.yaml # Deployment (2 replicas, hard node spread) + PodDisruptionBudget
|
||||
# Namespace default-deny + the Prometheus scrape allow + the egress CiliumNetworkPolicy. Keep
|
||||
# them in ONE file: a baseline without its scrape allow silently empties the TSDB.
|
||||
- networkpolicy.yaml
|
||||
- servicemonitor.yaml # headless metrics Service + ServiceMonitor (:2000)
|
||||
- alerts.yaml
|
||||
# The tunnel credentials JSON — the ONE secret here. SOPS-encrypted, decrypted per-env by the
|
||||
# `cloudflared` Flux Kustomization (both overlays patch it), referenced by name only. Its
|
||||
# plaintext template sibling is deliberately NOT named in this file (no-plaintext-secrets).
|
||||
# Ships with REPLACE-ME placeholders; runbooks/cloudflare-tunnel.md fills it.
|
||||
- cloudflared-credentials.sops.yaml
|
||||
|
||||
# config.yaml becomes the CONTENT-HASHED ConfigMap `cloudflared-config`, so an edit to the ingress
|
||||
# rules or the tunnel UUID changes the pod template and rolls the Deployment. Do NOT set
|
||||
# disableNameSuffixHash: the hash is the rollout mechanism.
|
||||
configMapGenerator:
|
||||
- name: cloudflared-config
|
||||
files:
|
||||
- config.yaml
|
||||
options:
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
# Cloudflare Tunnel connector (north-south INGRESS from the Cloudflare edge). See README.md.
|
||||
#
|
||||
# NOT homelab.keithjackson.dev/app-namespace=true. This is platform network infrastructure (like
|
||||
# vpn-ingress / vpn-egress), not an application, so Kyverno's app-scoped policies — including the
|
||||
# `require-default-deny-baseline` GENERATE rule — do not select this namespace. That is why
|
||||
# networkpolicy.yaml ships its own `default-deny` here, and why doing so creates NO second owner
|
||||
# of that object: Kyverno never generates one in a namespace without the app-namespace label
|
||||
# (the drift loop recorded in the 2026-09-30 handoff only exists where both write the same name).
|
||||
#
|
||||
# NO `gateway-access` label. Nothing here attaches an HTTPRoute to the shared Gateway; cloudflared
|
||||
# is a CLIENT of the Gateway, not a backend behind it.
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: cloudflared
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
@@ -0,0 +1,149 @@
|
||||
---
|
||||
# Namespace baseline: deny everything both ways, except DNS to kube-system.
|
||||
#
|
||||
# SINGLE OWNER. This namespace is not labelled app-namespace=true, so Kyverno's
|
||||
# require-default-deny-baseline generate rule does not select it and never writes this object;
|
||||
# Flux is its only writer (the same posture as kured / crafty / headlamp / homepage). Do NOT add
|
||||
# the app-namespace label to namespace.yaml without deleting this object in the same change, or
|
||||
# the two controllers fight over it every reconcile (the 2026-09-30 drift-loop root cause).
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: default-deny
|
||||
namespace: cloudflared
|
||||
labels:
|
||||
homelab.keithjackson.dev/baseline: default-deny
|
||||
spec:
|
||||
podSelector: {}
|
||||
policyTypes:
|
||||
- Ingress
|
||||
- Egress
|
||||
ingress: []
|
||||
egress:
|
||||
- to:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: kube-system
|
||||
ports:
|
||||
- protocol: UDP
|
||||
port: 53
|
||||
- protocol: TCP
|
||||
port: 53
|
||||
---
|
||||
# The ONLY ingress: Prometheus scraping :2000. Shipped in the same file as the baseline on purpose
|
||||
# (the Longhorn/kured lesson): a deny-all without this allow leaves the ServiceMonitor's targets
|
||||
# DISCOVERED and every scrape timing out, which is the state the CloudflaredTunnelDown rule is
|
||||
# written to fire on rather than go quiet in.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: NetworkPolicy
|
||||
metadata:
|
||||
name: cloudflared-allow-prometheus-scrape
|
||||
namespace: cloudflared
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
spec:
|
||||
podSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
policyTypes:
|
||||
- Ingress
|
||||
ingress:
|
||||
- from:
|
||||
- namespaceSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/metadata.name: monitoring
|
||||
ports:
|
||||
- protocol: TCP
|
||||
port: 2000
|
||||
---
|
||||
# Egress: the Cloudflare edge, the shared Gateway, and the Gateway's backends — nothing else.
|
||||
#
|
||||
# 1. world tcp+udp/7844 — the tunnel itself (QUIC on udp, HTTP/2 fallback on tcp).
|
||||
# world tcp/443 — Cloudflare's API / update endpoints that cloudflared calls at start.
|
||||
# Expressed as `world` + port rather than Cloudflare CIDRs: the edge ranges are Cloudflare's to
|
||||
# change, and a stale CIDR list would fail as a silent outage. Narrowing to toCIDR is a
|
||||
# possible later hardening; it is not done blind.
|
||||
#
|
||||
# 2. `ingress` entity tcp/443 — the shared Gateway's Envoy. Traffic to the
|
||||
# cilium-gateway-shared Service is handed to Envoy, whose identity is reserved:ingress.
|
||||
#
|
||||
# 3. The Gateway's BACKENDS, by namespace. NOT optional: Cilium >= 1.16.8 (CVE-2025-30162; this
|
||||
# cluster runs 1.20.x) makes the Gateway's Envoy re-evaluate THIS pod's egress policy against
|
||||
# the identity of the backend it routes the request to. With only rule 2, Envoy accepts the TCP
|
||||
# connection and then answers every request `403 Access denied` — measured for buildkit ->
|
||||
# gitea on 2026-10-30, see platform/buildkit/networkpolicy.yaml. So the backends are named:
|
||||
# * any namespace opted in to the shared Gateway (gateway-access=shared) — which is the set
|
||||
# of namespaces that can serve a route at all, so this tracks new apps automatically;
|
||||
# * MINUS the LAN/VPN-only services (operator decisions 2026-11-08): plex, jackett, pihole,
|
||||
# semaphore, qbittorrent, radarr, sonarr, headlamp. config.yaml already 404s their
|
||||
# hostnames; this is the second, independent layer: even a request that got past the
|
||||
# ingress rules would be refused by Envoy for these backends.
|
||||
# unifi and crafty-mc are not here because they are L4 LoadBalancer VIPs, not Gateway
|
||||
# backends; the crafty WEB UI (crafty.<domain>, an HTTPRoute) is deliberately reachable.
|
||||
#
|
||||
# WHY NOT the `- {}` allow-all egress gitea-actions uses (apps/wave-1/gitea-actions.yaml)?
|
||||
# It would satisfy the L7 re-check just as well, but `{}` admits EVERY identity, including
|
||||
# kube-apiserver and host — which operator decision 4 (the Kubernetes API is never reachable
|
||||
# through the tunnel) rules out for the one pod that carries internet traffic in. The
|
||||
# namespace-label selector is the narrowest rule that still names every possible backend.
|
||||
# Verified 2026-11-08: Cilium identities carry `k8s:io.cilium.k8s.namespace.labels.
|
||||
# gateway-access`, and every namespace with an HTTPRoute on the shared Gateway is labelled
|
||||
# gateway-access=shared.
|
||||
#
|
||||
# BACKENDS NEED NO INGRESS CHANGE. Envoy forwards to a backend as reserved:ingress, which every
|
||||
# backend already admits; the source-identity check happens on THIS pod's egress, not on the
|
||||
# destination's ingress. Measured 2026-10-30 (decisions/2026-10-30-gitea-admit-buildkit.md):
|
||||
# gitea-actions is in none of Gitea's ingress allows and its requests through the Gateway reach
|
||||
# Gitea, while Gitea's endpoint shows only a `reserved:ingress` allow row. So, unlike a
|
||||
# ClusterIP client, cloudflared is not added to any app's inClusterClients. That also means
|
||||
# this rule does not open a direct cloudflared -> app pod path into any app namespace whose
|
||||
# ingress admits only reserved:ingress and named peers.
|
||||
#
|
||||
# DELIBERATELY ABSENT: kube-apiserver, host, remote-node, cluster, all. The Kubernetes API stays
|
||||
# local-only (operator decision 2026-11-08); conformance/tests/l0-static/cloudflared-ingress-rules
|
||||
# fails the gate if any of those entities appears here.
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: cloudflared-allow-egress
|
||||
namespace: cloudflared
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
egress:
|
||||
- toEntities:
|
||||
- world
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: "7844"
|
||||
protocol: TCP
|
||||
- port: "7844"
|
||||
protocol: UDP
|
||||
- port: "443"
|
||||
protocol: TCP
|
||||
- toEntities:
|
||||
- ingress
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: "443"
|
||||
protocol: TCP
|
||||
- toEndpoints:
|
||||
- matchExpressions:
|
||||
- key: io.cilium.k8s.namespace.labels.gateway-access
|
||||
operator: In
|
||||
values:
|
||||
- shared
|
||||
- key: io.kubernetes.pod.namespace
|
||||
operator: NotIn
|
||||
values:
|
||||
- plex
|
||||
- jackett
|
||||
- pihole
|
||||
- semaphore
|
||||
- qbittorrent
|
||||
- radarr
|
||||
- sonarr
|
||||
- headlamp
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
# Scrape of cloudflared's own metrics server (:2000, `metrics:` in config.yaml) on every replica.
|
||||
#
|
||||
# A Service + ServiceMonitor in ONE file rather than a PodMonitor, on purpose: the repo's L0 checks
|
||||
# (servicemonitor-ports-resolve, alert-metrics-have-scrape-targets assertion 4) can then prove the
|
||||
# monitor's port is a port this Service actually defines. A PodMonitor would have to be declared as
|
||||
# an unauditable exception. The Service is headless and carries only the metrics port — it is not a
|
||||
# way into the tunnel, and nothing routes to it but Prometheus (networkpolicy.yaml admits only the
|
||||
# monitoring namespace on :2000).
|
||||
#
|
||||
# kube-prometheus-stack runs with serviceMonitorSelectorNilUsesHelmValues: false, so this is
|
||||
# selected from any namespace; the `release` label is kept for consistency.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: cloudflared-metrics
|
||||
namespace: cloudflared
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
spec:
|
||||
clusterIP: None
|
||||
selector:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
ports:
|
||||
- name: metrics
|
||||
port: 2000
|
||||
targetPort: metrics
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: cloudflared
|
||||
namespace: cloudflared
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
release: kube-prometheus-stack
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
endpoints:
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
interval: 30s
|
||||
@@ -16,7 +16,7 @@ Cilium L2 LoadBalancer (game protocols authenticate players themselves).
|
||||
|---|---|
|
||||
| Image | `registry.gitlab.com/crafty-controller/crafty-4:4.10.8` (current stable; bare-semver tags, no `v`) |
|
||||
| Web UI | HTTPS **self-signed** on `:8443` (reached only via the sidecar / same-pod) |
|
||||
| Sidecar | `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1`, reverse-proxy on `:4180` |
|
||||
| Sidecar | `quay.io/oauth2-proxy/oauth2-proxy:v7.15.6`, reverse-proxy on `:4180` |
|
||||
| Data | ONE Longhorn RWO PVC `crafty-data` (20Gi), mounted 5× by subPath |
|
||||
| PVC subPaths | `servers`→`/crafty/servers`, `backups`→`/crafty/backups`, `logs`→`/crafty/logs`, `import`→`/crafty/import`, `config`→`/crafty/app/config` |
|
||||
| Identity | `runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true` (see below) |
|
||||
|
||||
@@ -130,7 +130,7 @@ spec:
|
||||
- name: oauth2-proxy
|
||||
# quay.io/oauth2-proxy/oauth2-proxy — same authenticator family as platform/oauth2-proxy, but
|
||||
# here in per-app REVERSE-PROXY mode (--upstream is the co-located Crafty UI, not static://202).
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
|
||||
imagePullPolicy: IfNotPresent
|
||||
args:
|
||||
- --http-address=0.0.0.0:4180
|
||||
|
||||
@@ -101,6 +101,11 @@ metadata:
|
||||
lbipam.cilium.io/ips: "${CRAFTY_L4_VIP}"
|
||||
# external-dns publishes A crafty-mc.home.keithjackson.dev -> the pinned VIP (Java server target).
|
||||
external-dns.alpha.kubernetes.io/hostname: "crafty-mc.${BASE_DOMAIN}"
|
||||
# DNS-ONLY, NEVER PROXIED (2026-11-08). external-dns runs with --cloudflare-proxied globally (the
|
||||
# gateway names point at the Cloudflare Tunnel); Cloudflare's proxy carries HTTP(S) only, so an
|
||||
# orange-clouded A for these L4 ports would be broken. Required by the L0 check
|
||||
# internal-services-publish-no-public-dns (9b) on every `service`-source publisher.
|
||||
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
|
||||
@@ -20,29 +20,49 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
|
||||
its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
|
||||
Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.
|
||||
|
||||
## Public target + in-cluster DDNS (2026-10-03)
|
||||
## Public target: the Cloudflare Tunnel, proxied (2026-11-08)
|
||||
|
||||
Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
|
||||
Gateway-published records are **proxied CNAMEs to the Cloudflare Tunnel**
|
||||
(`platform/cloudflared`), not the WAN IP and not the Gateway VIP:
|
||||
|
||||
```
|
||||
plex.home.keithjackson.dev CNAME wan.home.keithjackson.dev <- external-dns (Gateway `target` annotation)
|
||||
wan.home.keithjackson.dev A <current WAN IP> <- cloudflare-ddns (ddns.yaml), every 5m
|
||||
gitea.home.keithjackson.dev CNAME (proxied) fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com <- external-dns
|
||||
unifi.home.keithjackson.dev A (DNS-only) 192.168.2.21 <- external-dns, `service` source (L4)
|
||||
crafty-mc.home... A (DNS-only) 192.168.2.22 <- external-dns, `service` source (L4)
|
||||
wan.home.keithjackson.dev A (DNS-only) <current WAN IP> <- cloudflare-ddns (ddns.yaml): WireGuard udp/51820 ONLY
|
||||
```
|
||||
|
||||
- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
|
||||
applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
|
||||
nothing.
|
||||
- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
|
||||
It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
|
||||
`egress-class: vpn`.
|
||||
- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
|
||||
(split-horizon) and never use the public path.
|
||||
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
|
||||
OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
|
||||
the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
|
||||
- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`); external-dns
|
||||
reads it from the Gateway only, so it applies to every route that publishes at all. Its UUID must
|
||||
equal `tunnel:` in `platform/cloudflared/config.yaml` (L0 check
|
||||
`internal-services-publish-no-public-dns` (9a)).
|
||||
- `--cloudflare-proxied` (HelmRelease `extraArgs`) orange-clouds every record by default — the only
|
||||
practical lever, because v0.23.0 reads the per-object `cloudflare-proxied` annotation from the
|
||||
route/Service, never the Gateway. **Every `service`-source publisher must carry
|
||||
`external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"`** (L4 ports cannot ride the HTTP
|
||||
proxy); (9b) fails the gate otherwise.
|
||||
- `internal` routes (`controller: none`) still publish nothing. Plex's web UI and Headlamp joined that
|
||||
set on 2026-11-08 (LAN/VPN-only); the tunnel also 404s every such name (9c/9d).
|
||||
- Public answers are Cloudflare anycast **A and AAAA** — Cloudflare's edge, not an origin address.
|
||||
- **Pods** resolve the whole zone to the Gateway VIP via CoreDNS (split-horizon) and never use the
|
||||
public path. **LAN clients** use PiHole, which since 2026-11-09 answers an EXPLICIT list
|
||||
(`catalog/pihole.yaml`): the LAN-only names, plus gitea/nextcloud/keycloak/auth (kept off
|
||||
Cloudflare's 100 MB / 100 s limits and on one OIDC path), go to the Gateway VIP and the L4 names to
|
||||
their own VIPs; every other public name (apex, grafana, crafty, seerr) resolves to Cloudflare from
|
||||
the LAN exactly as from anywhere else, so no split-DNS is needed for them. Guard:
|
||||
`conformance/tests/l0-static/pihole-lan-names`.
|
||||
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node the apiserver's
|
||||
OIDC issuer `keycloak.home.keithjackson.dev` now resolves to Cloudflare and its discovery/JWKS
|
||||
fetches go through the tunnel (publicly trusted edge cert; no `oidc-ca-file` is configured). The
|
||||
router NAT hairpin is no longer on that path.
|
||||
|
||||
Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
|
||||
`kubectl -n external-dns logs deploy/cloudflare-ddns`.
|
||||
Check: `dig +short @1.1.1.1 gitea.home.keithjackson.dev` → Cloudflare anycast addresses (no CNAME is
|
||||
visible for a proxied record); `kubectl -n external-dns logs deploy/external-dns`.
|
||||
|
||||
### History
|
||||
|
||||
2026-10-03 → 2026-11-08: records were DNS-only CNAMEs to `wan.home.keithjackson.dev`, with the router
|
||||
forwarding WAN tcp/80+443 to the Gateway VIP. Before 2026-10-03 they answered the Gateway VIP itself.
|
||||
|
||||
## Secrets
|
||||
|
||||
|
||||
@@ -2,9 +2,15 @@
|
||||
# In-cluster dynamic DNS for the home WAN (2026-10-03).
|
||||
#
|
||||
# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
|
||||
# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
|
||||
# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
|
||||
# moves all of them through this one record, with no Git change and no external-dns involvement.
|
||||
# current public IP.
|
||||
#
|
||||
# WIREGUARD-ONLY SINCE 2026-11-08. From 2026-10-03 every public name external-dns published from the
|
||||
# shared Gateway was a CNAME to this record. They are now proxied CNAMEs to the Cloudflare Tunnel
|
||||
# (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so NOTHING in the zone
|
||||
# points here any more. The record stays for exactly one consumer: the WireGuard VPN endpoint
|
||||
# (udp/51820, platform/vpn-ingress, infra/vpn-ingress/new-peer.sh), which cannot ride Cloudflare's
|
||||
# HTTP proxy and needs the real WAN address. It must stay DNS-only (`PROXIED` unset/false below) —
|
||||
# an orange-clouded A record would hand peers a Cloudflare anycast address that drops UDP.
|
||||
#
|
||||
# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
|
||||
# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
|
||||
@@ -64,7 +70,7 @@ spec:
|
||||
- name: IP6_PROVIDER
|
||||
value: "none" # IPv4 WAN only; leaves any AAAA alone
|
||||
- name: PROXIED
|
||||
value: "false" # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
|
||||
value: "false" # DNS-only: WireGuard udp/51820 needs the real WAN IP
|
||||
- name: UPDATE_CRON
|
||||
value: "@every 5m"
|
||||
- name: DELETE_ON_STOP
|
||||
|
||||
@@ -107,29 +107,47 @@ spec:
|
||||
# what this instance can see or own; per-resource exclusion only ever REMOVES an object from
|
||||
# this instance's view, so it cannot make prod capable of deleting a record it does not own.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
|
||||
# THE PUBLIC TARGET — THE CLOUDFLARE TUNNEL, PROXIED (2026-11-08; supersedes 2026-10-03)
|
||||
#
|
||||
# Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
|
||||
# RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
|
||||
# fix, and both still hold:
|
||||
# * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
|
||||
# from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
||||
# gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
|
||||
# shared Gateway therefore moves EVERY gateway-sourced record at once.
|
||||
# * The home WAN IP is residential and can change, so a literal IP in Git would rot.
|
||||
# History: until 2026-10-03 every gateway-sourced record answered the Gateway VIP (RFC1918);
|
||||
# from 2026-10-03 they were CNAMEs to `wan.${BASE_DOMAIN}` (DNS-only, the home WAN IP, with the
|
||||
# router forwarding tcp/80+443 to the VIP). Both published an origin address of ours. Now:
|
||||
# * platform/gateway/gateway.yaml sets `target: <tunnel-uuid>.cfargotunnel.com` on the shared
|
||||
# Gateway. `target` is read in v0.23.0's gateway-httproute source ONLY from the GATEWAY
|
||||
# (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(gw.Annotations)`),
|
||||
# never from an HTTPRoute, so this one value moves every gateway-sourced record at once.
|
||||
# * `--cloudflare-proxied` below orange-clouds every record by default. It is GLOBAL because
|
||||
# that is the only practical lever: v0.23.0 reads the per-object `cloudflare-proxied`
|
||||
# annotation from the ROUTE (or Service), never the Gateway (source/gateway.go:
|
||||
# `annotations.ProviderSpecificAnnotations(annots)` with `annots := meta.Annotations` of the
|
||||
# route), so a Gateway-level opt-in does not exist. A CNAME to cfargotunnel.com MUST be
|
||||
# proxied — DNS-only it resolves to nothing routable — so "proxied by default" is the right
|
||||
# default for this zone's gateway names.
|
||||
# * The provider honours a per-object `"false"` over the global default (provider/cloudflare
|
||||
# shouldBeProxied: ParseBool of the annotation overrides proxiedByDefault). The key it reads
|
||||
# is `<annotationPrefix>cloudflare-proxied` (annotations.SetAnnotationPrefix re-derives
|
||||
# CloudflarePrefix), i.e. `external-dns.alpha.kubernetes.io/cloudflare-proxied` here.
|
||||
#
|
||||
# THE `service` SOURCE MUST OPT OUT. The L4 LoadBalancers (crafty-mc, unifi) publish an A record
|
||||
# at their own LB VIP; Cloudflare's proxy is HTTP(S)-only, so a proxied A for a game or UniFi
|
||||
# inform port is broken by construction. Each carries `cloudflare-proxied: "false"` and stays
|
||||
# DNS-only, exactly as before this change. conformance/tests/l0-static/
|
||||
# internal-services-publish-no-public-dns (assertion 9b) FAILS on any `service`-source publisher
|
||||
# without it, so a new L4 Service cannot silently become a proxied record.
|
||||
#
|
||||
# The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
|
||||
# UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
|
||||
# (b) the address is kept current IN-CLUSTER. Implementation:
|
||||
# * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
|
||||
# each published name becomes a CNAME to that one hostname;
|
||||
# * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
|
||||
# record at the current WAN IP. That record has no external-dns TXT registry entry, so this
|
||||
# instance never owns or garbage-collects it.
|
||||
# Internal services are unchanged: their routes carry `controller: none` (above), which is
|
||||
# checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
|
||||
# LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
|
||||
# LB VIP. Whether those should go public is a separate, still-open question.
|
||||
# checked BEFORE the target is read, so they still publish nothing. Plex's web UI and Headlamp
|
||||
# joined that set on 2026-11-08 (operator: LAN/VPN-only). `wan.${BASE_DOMAIN}` (ddns.yaml) has no
|
||||
# external-dns TXT registry entry and is untouched; it now serves only WireGuard udp/51820.
|
||||
#
|
||||
# THE RISK THIS ADDS, AND WHAT GUARDS IT. If the annotation prefix ever stops being honoured
|
||||
# again (the 2026-11-07 v0.23.0 incident), `target` is ignored and every name falls back to the
|
||||
# Gateway's own address — and with this flag it would be published as a PROXIED A record at
|
||||
# ${GATEWAY_VIP} (RFC1918; Cloudflare 1003/522s for every public name) while `controller: none`
|
||||
# is also ignored and internal names publish. Assertion (8) of the L0 check above ties the
|
||||
# prefix this tree writes to `values.annotationPrefix`; (8b) forbids overriding it here.
|
||||
extraArgs:
|
||||
- --cloudflare-proxied
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
|
||||
#
|
||||
|
||||
@@ -30,5 +30,9 @@ east-west default-deny (docs/09).
|
||||
|
||||
- **Phase 2 (WS-NET):** pin the L2-announced ingress VIP / IP pool (Cilium L2 announcement, docs/09);
|
||||
add the HTTP→HTTPS redirect route; confirm the cross-namespace ReferenceGrant pattern with WS-APPS.
|
||||
- **Public exposure** is a **manual edge port-forward** of 80/443 to the Gateway VIP (single SOHO
|
||||
router, docs/09) — a documented manual step, the analogue of the old pfSense exception.
|
||||
- **Public exposure** is via the **Cloudflare Tunnel** (2026-11-08, `platform/cloudflared`): the
|
||||
Gateway's `external-dns.alpha.kubernetes.io/target` is `<tunnel-uuid>.cfargotunnel.com`, so every
|
||||
published name is a proxied CNAME to the tunnel and cloudflared forwards to this Gateway's
|
||||
ClusterIP. No inbound WAN 80/443 forward is needed (the old manual edge port-forward is retired —
|
||||
runbooks/cloudflare-tunnel.md). LAN/VPN-only names (internal apps, plex, headlamp) carry
|
||||
`controller: none` on their route and are 404'd by the tunnel.
|
||||
@@ -8,22 +8,36 @@ metadata:
|
||||
name: shared
|
||||
namespace: gateway
|
||||
annotations:
|
||||
# PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
|
||||
# route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
|
||||
# unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
|
||||
# becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
|
||||
# pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
|
||||
# PUBLIC DNS TARGET = THE CLOUDFLARE TUNNEL (2026-11-08, operator decision; replaces the
|
||||
# 2026-10-03 `wan.${BASE_DOMAIN}` target). external-dns v0.23.0's gateway-httproute source reads
|
||||
# `target` from the GATEWAY only (source/gateway.go: `overrides: annotations.
|
||||
# TargetsFromTargetAnnotation(gw.Annotations)`) — never from an HTTPRoute — so this one value moves
|
||||
# EVERY published route on this Gateway at once. Each published `<svc>.${BASE_DOMAIN}` (and the
|
||||
# apex) becomes a CNAME to `<tunnel-uuid>.cfargotunnel.com`, and platform/externaldns/helmrelease.yaml
|
||||
# sets `--cloudflare-proxied` so the CNAME is orange-clouded: public clients resolve Cloudflare
|
||||
# anycast, Cloudflare terminates edge TLS (Total TLS) and hands the request to the in-cluster
|
||||
# cloudflared connectors (platform/cloudflared), which forward to this Gateway's ClusterIP. No
|
||||
# origin address of ours appears in public DNS, and no inbound WAN port is needed for HTTP(S).
|
||||
#
|
||||
# Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
|
||||
# admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
|
||||
# `exposure: public` apps. `internal` services stay unpublished via their
|
||||
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
|
||||
# BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
|
||||
# from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
|
||||
# THE UUID IS A LITERAL, AND IT MUST EQUAL `tunnel:` IN platform/cloudflared/config.yaml. A CNAME
|
||||
# to any other <uuid>.cfargotunnel.com is a dead name (Cloudflare answers 1016). That equality is
|
||||
# asserted on every gate run by conformance/tests/l0-static/internal-services-publish-no-public-dns
|
||||
# (assertion 9a). A literal rather than a cluster-var keeps this change off clusters/**.
|
||||
#
|
||||
# LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
|
||||
# ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
|
||||
external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
|
||||
# Scope: every route on this Gateway that publishes at all. `internal` services (and the
|
||||
# operator-designated LAN/VPN-only names: plex, headlamp) stay unpublished via their
|
||||
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns
|
||||
# applies BEFORE it reads this target. Those same names map to `http_status:404` in the tunnel's
|
||||
# ingress rules, so even a hand-made record cannot reach them through the tunnel.
|
||||
#
|
||||
# In-cluster clients are unaffected: CoreDNS answers ${BASE_DOMAIN} with ${GATEWAY_VIP} directly
|
||||
# (split-horizon). LAN clients use PiHole's EXPLICIT list (catalog/pihole.yaml, since 2026-11-09):
|
||||
# LAN-only names and gitea/nextcloud/keycloak/auth get ${GATEWAY_VIP}; every other published
|
||||
# name takes this public path through Cloudflare, from the LAN as from anywhere.
|
||||
#
|
||||
# `wan.${BASE_DOMAIN}` (platform/externaldns/ddns.yaml) is NO LONGER a target of anything here;
|
||||
# it stays only as the WireGuard udp/51820 endpoint.
|
||||
external-dns.alpha.kubernetes.io/target: "fc81938b-5855-47f1-8a27-6286f56dda93.cfargotunnel.com"
|
||||
spec:
|
||||
gatewayClassName: cilium
|
||||
listeners:
|
||||
|
||||
@@ -18,6 +18,13 @@ metadata:
|
||||
annotations:
|
||||
homelab.keithjackson.dev/auth-mode: oidc
|
||||
homelab.keithjackson.dev/auth-groups: platform-admins
|
||||
# LAN/VPN-ONLY (operator decision 2026-11-08). Disowned from external-dns, so the public
|
||||
# headlamp record and its TXT are DELETED and nothing republishes them; LAN/VPN clients keep
|
||||
# resolving the name via PiHole's explicit entry for it (-> the Gateway VIP). The Cloudflare
|
||||
# Tunnel also maps headlamp.${BASE_DOMAIN} to http_status:404 (platform/cloudflared/config.yaml).
|
||||
# Same mechanism the renderer emits for `exposure: internal` services; asserted by
|
||||
# conformance/tests/l0-static/internal-services-publish-no-public-dns (9c).
|
||||
external-dns.alpha.kubernetes.io/controller: none
|
||||
spec:
|
||||
parentRefs:
|
||||
- name: shared
|
||||
|
||||
@@ -38,6 +38,20 @@ spec:
|
||||
ingress:
|
||||
enabled: false # exposed via Gateway API HTTPRoute (httproute.yaml), not the
|
||||
# operator's built-in Ingress
|
||||
# --- ONE OWNER for the `keycloak` ServiceMonitor: this repo (servicemonitor.yaml), not the operator.
|
||||
# Since 26.x the operator creates its OWN ServiceMonitor, named after this CR (`keycloak`, i.e.
|
||||
# the SAME name/namespace as ours), whenever metrics-enabled=true and the CRD exists —
|
||||
# spec.serviceMonitor.enabled defaults TRUE. On 2026-11-08, right after the 26.0.6 -> 26.8.2
|
||||
# bump, its server-side apply onto our Flux-owned object failed `409 Conflict`, the operator
|
||||
# exhausted retries, the CR sat Ready=False/HasErrors=True, and `platform-config` (wait: true,
|
||||
# so it health-checks this CR) stalled every Kustomization downstream of it. Ours is kept
|
||||
# because the L0 servicemonitor-ports-resolve and L2 keycloak-monitored conformance tests and
|
||||
# the dashboard's up{namespace="keycloak"} are written against it; the operator's would scrape
|
||||
# the same thing (management port 9000, /metrics, 30s). Do NOT flip this to true without
|
||||
# deleting servicemonitor.yaml in the same change — two owners of one object is a fight
|
||||
# (cf. the default-deny NetworkPolicy drift loop).
|
||||
serviceMonitor:
|
||||
enabled: false
|
||||
# --- Hardening + observability options (docs/15). Brute-force detection is a REALM setting
|
||||
# (bruteForceProtected in the realm import); these enable health + metrics endpoints. ---
|
||||
additionalOptions:
|
||||
|
||||
@@ -8,6 +8,9 @@
|
||||
# /metrics) once metrics-enabled/health-enabled are set (see keycloak-instance.yaml additionalOptions).
|
||||
# We select the operator-created keycloak-service (which selects the Keycloak pods) and scrape the pods'
|
||||
# management port by number via targetPort.
|
||||
# SOLE OWNER: the operator's own ServiceMonitor (same name `keycloak`) is switched OFF in
|
||||
# keycloak-instance.yaml (spec.serviceMonitor.enabled: false) — 26.x defaults it ON and its apply
|
||||
# 409-looped against this object on 2026-11-08. Keep exactly one of the two.
|
||||
# TODO(WS-SEC, bring-up): confirm the management port number/name the operator exposes (9000 default)
|
||||
# and adjust targetPort if the pinned operator changes it. The KeycloakDown / KeycloakDatabaseDown
|
||||
# alerts (platform/monitoring/alerts.yaml) key on kube-state-metrics and fire regardless of this SM.
|
||||
|
||||
@@ -18,7 +18,7 @@ spec:
|
||||
chart:
|
||||
spec:
|
||||
chart: longhorn
|
||||
version: ">=1.12.0 <1.13.0" # W5 upgrade 1.11->1.12 (resolves 1.12.1; supports k8s 1.36). Renovate maintains
|
||||
version: "<1.14.0" # W5 upgrade 1.11->1.12 (resolves 1.12.1; supports k8s 1.36). Renovate maintains
|
||||
sourceRef:
|
||||
kind: HelmRepository
|
||||
name: longhorn
|
||||
|
||||
@@ -72,11 +72,14 @@ Both are deliberate, and both fail **visibly** rather than silently:
|
||||
- **The nodes resolve the registry hostname to the WAN, so the TLS they fail to verify is
|
||||
the router's — DIAGNOSED 2026-10-06, and fixed in this change.** containerd pulls as a
|
||||
*node* process, so it resolves through the node's public upstreams
|
||||
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. The public
|
||||
answer for every name in this zone is a CNAME to `wan.home.keithjackson.dev` — the WAN
|
||||
IP — which this router does not hairpin; it answers :443 with its own placeholder
|
||||
(`node_baseline_dns_servers` = the LAN gateway) rather than through Pi-hole. At the time,
|
||||
the public answer for every name in this zone was a CNAME to `wan.home.keithjackson.dev`
|
||||
— the WAN IP — which this router does not hairpin; it answered :443 with its own placeholder
|
||||
certificate `CN=%D` carrying **no extensions, therefore no SANs**, which is precisely
|
||||
`x509: certificate is not valid for any names`. The Gateway, the wildcard certificate
|
||||
`x509: certificate is not valid for any names`. (Since 2026-11-08 the public answer is
|
||||
Cloudflare's edge via the Tunnel, not the WAN IP — see `platform/externaldns/README.md`.
|
||||
The pin is still the right path: it keeps registry pulls on the LAN and off Cloudflare's
|
||||
proxy.) The Gateway, the wildcard certificate
|
||||
and the digest were never implicated. The fix is the pin this repository already uses
|
||||
for the Keycloak issuer, for the identical reason: `gitea.{{ k3s_base_domain }}` is now
|
||||
in `node_baseline_hosts_pins` — node configuration is Ansible's, declared in the
|
||||
|
||||
@@ -67,7 +67,7 @@ data:
|
||||
# adds the CLI failure envelope to triage errors and the PM_DECISION_MAX_TURNS knob
|
||||
# (decisions/2026-10-31-pm-agent-digest-99.md).
|
||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
|
||||
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2"
|
||||
PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:89ce7eead20ccadf95c2bc6a1ffd497c8c5e2bb93245c9082979742bda4c46a1"
|
||||
|
||||
# --- the model -------------------------------------------------------------
|
||||
# The PM only reads a thread and answers with one JSON object, so it holds no tools and
|
||||
|
||||
@@ -78,7 +78,7 @@ spec:
|
||||
# the registry's manifest endpoint on 2026-10-30 and pinned by hand once so the
|
||||
# dispatcher does not wait on Renovate's first run; every later bump is Renovate's.
|
||||
# renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
|
||||
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2
|
||||
image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:89ce7eead20ccadf95c2bc6a1ffd497c8c5e2bb93245c9082979742bda4c46a1
|
||||
imagePullPolicy: IfNotPresent
|
||||
command:
|
||||
- /usr/local/bin/pm-dispatch
|
||||
|
||||
@@ -217,9 +217,9 @@ Consequences, stated plainly:
|
||||
|
||||
**Governance:** ADR-0015 and `docs/03` are specifications and are immutable to agents
|
||||
(spec `AGENTS.md` §11, R-TEST-06, enforced by `ci/spec-immutability.sh`) — they are **not** edited
|
||||
here. The correction is raised as an **escalation with a drafted ADR-0016 supersede** in
|
||||
`DECISIONS-LOG.md` (entry of 2026-09-06) for the operator/architect to accept. Until then this file
|
||||
and `egress-steering.yaml` are the accurate statement of what the implementation can actually do.
|
||||
here. The correction, first escalated in `DECISIONS-LOG.md` (entry of 2026-09-06), is recorded in
|
||||
spec **ADR-0020**, which supersedes ADR-0015's Kill-switch and Resilience bullets. This file and
|
||||
`egress-steering.yaml` state what the implementation does.
|
||||
|
||||
## Secrets — host-side now
|
||||
PIA credentials + the `.ovpn` live **on the gateway node via Ansible vault** (see spec `docs/07`), **not**
|
||||
|
||||
@@ -109,7 +109,7 @@ spec:
|
||||
# that shard and re-shards the rest (breaking live connections), and each shard exits from a
|
||||
# DIFFERENT public IP. Upstream: cilium/cilium#30157, cilium/cilium#18230.
|
||||
# ADR-0015's "Resilience: two egress-gateway nodes + Cilium Egress Gateway HA (gateway failover)"
|
||||
# is therefore NOT IMPLEMENTABLE on Cilium OSS — escalated for an ADR-0016 supersede; see
|
||||
# is therefore NOT IMPLEMENTABLE on Cilium OSS — superseded by spec ADR-0020; see
|
||||
# platform/vpn-egress/README.md and the DECISIONS-LOG entry of 2026-09-06.
|
||||
#
|
||||
# `interface: tun0` IS RESOLVED ONCE — the gateway node has to tell Cilium when tun0 changes.
|
||||
|
||||
@@ -42,8 +42,9 @@ no version string to match against a CVE, and no amplification to abuse.
|
||||
|
||||
That is the whole security argument, and it is why the exposed port carries no IP allowlist (which
|
||||
would also have made roaming clients unusable — a phone's address changes with every cell tower).
|
||||
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN: a
|
||||
resolver answers anybody, so it must be scoped by address. WireGuard answers nobody.
|
||||
Compare `apps/wave-0/pihole-dns.yaml`, which *does* pin `loadBalancerSourceRanges` to the LAN and
|
||||
this tunnel's client subnet: a resolver answers anybody, so it must be scoped by address. WireGuard
|
||||
answers nobody.
|
||||
|
||||
What it does **not** protect against is a stolen device — anyone holding a device's private key
|
||||
*is* that device. That is what revocation is for: delete the stanza, merge, done (~1 min).
|
||||
@@ -55,9 +56,9 @@ What it does **not** protect against is a stolen device — anyone holding a dev
|
||||
┌──────────────┐ udp/51820 ┌───────────────┐ forward ┌────────────────────────────┐
|
||||
│ WireGuard app│─────────────▶│ WAN 51820 ──▶ │───────────▶│ wg0 10.90.0.1/24 │
|
||||
│ 10.90.0.2/32 │ over the │ node IP:51820 │ │ decrypt, then FORWARD │
|
||||
└──────────────┘ internet └───────────────┘ │ SNAT to the node's addr │
|
||||
└──────────────┘ internet └───────────────┘ │ SNAT (LAN), eBPF LB (VIPs)│
|
||||
└─────────────┬──────────────┘
|
||||
│ as `host`/`remote-node`
|
||||
│ translated per destination (below)
|
||||
┌─────────────────────────────────┼──────────────────────┐
|
||||
▼ ▼ ▼
|
||||
LAN 192.168.2.0/24 Gateway VIP .20 (16 web UIs) WAN / internet
|
||||
@@ -86,9 +87,21 @@ resolved, because `apps/wave-0/pihole-dns.yaml`'s ingress policy deliberately om
|
||||
entity. A VPN that looks up and is useless is the worst available outcome, and nothing in the
|
||||
manifests would have hinted at it.
|
||||
|
||||
In the **host** netns there is no pod identity to carry. Forwarded client traffic is SNATed to the
|
||||
node's own address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
|
||||
byte-for-byte what a LAN client looks like. Three further problems vanish with it:
|
||||
In the **host** netns there is no pod identity to carry, which is what avoids the 403s. Clients do
|
||||
**not**, however, all arrive SNATed to the node. Measured 2026-11-08: Cilium counts `wg0` among its
|
||||
kube-proxy-replacement devices (`cilium-dbg status`: `eno1`, `wg0`, `tun0`), so what a packet looks
|
||||
like depends on where it is going:
|
||||
|
||||
| destination | handled by | source the destination sees |
|
||||
|---|---|---|
|
||||
| LAN hosts, the internet | kernel FORWARD, then the iptables MASQUERADE in [`daemonset.yaml`](./daemonset.yaml) | the node's address |
|
||||
| LoadBalancer VIPs (Gateway `.20`, PiHole `.23`, ...) | Cilium's eBPF load balancer, as the packet enters from `wg0`, **before** MASQUERADE | `10.90.0.x` at the VIP's source-range check, identity `world` |
|
||||
|
||||
The second row is why PiHole's `loadBalancerSourceRanges` has to list `${WG_CLIENT_CIDR}` as well as
|
||||
the LAN: left LAN-only, tunnel DNS is predicted to die with `DROP_NOT_IN_SRC_RANGE`. That prediction
|
||||
and the row's source column come from reading Cilium's datapath source; **no peer has ever completed
|
||||
a handshake, so neither has been observed live.** The Gateway VIP carries no source ranges and needs
|
||||
no change. Three further problems vanish with the host netns:
|
||||
|
||||
* `net.ipv4.ip_forward` is **already 1** on these nodes (`node_baseline` writes
|
||||
`/etc/sysctl.d/99-k3s-cilium.conf`). It is *not* in the kubelet's safe-sysctl allowlist, so a
|
||||
@@ -99,7 +112,9 @@ byte-for-byte what a LAN client looks like. Three further problems vanish with i
|
||||
`conformance/tests/l0-static/l2-announced-vips-serve-from-any-node`) and no
|
||||
`externalTrafficPolicy` trade-off to make at all.
|
||||
* **No double encapsulation.** Client UDP lands on the node's own NIC, so the underlay is a flat
|
||||
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` simply correct.
|
||||
1500-byte path, not Cilium's 1450-byte VXLAN. That is what makes `MTU = 1420` correct on the
|
||||
server's `wg0`. It says nothing about a client's own path: `new-peer.sh` defaults client profiles
|
||||
to `1280` (see "Troubleshooting" in `runbooks/vpn-remote-access.md`).
|
||||
|
||||
`hostNetwork` is *permitted* here, and not by oversight:
|
||||
`platform/kyverno/policies/restrict-host-namespaces.yaml` matches `namespaceSelector:
|
||||
@@ -163,6 +178,30 @@ internet path. To enforce less *server-side*, add filtering to the `PostUp` chai
|
||||
before the accept rules) — the FORWARD chain there is the only enforcement point, since no
|
||||
NetworkPolicy applies.
|
||||
|
||||
## Phone profiles: one tunnel for home, one for away
|
||||
|
||||
A phone that needs the LAN-only apps runs full tunnel (`AllowedIPs = 0.0.0.0/0`, `DNS = 192.168.2.23`)
|
||||
everywhere, so it needs **two tunnels**, each its own peer with its own key and address:
|
||||
|
||||
| tunnel | peer | address | Endpoint | use when |
|
||||
|---|---|---|---|---|
|
||||
| `away` | `keith-phone` | `10.90.0.2` | `wan.home.keithjackson.dev:51820` | off the home network |
|
||||
| `home` | `keith-phone-home` | `10.90.0.3` | `192.168.2.10:51820` | on the home Wi-Fi |
|
||||
|
||||
**`wan.` cannot work from inside the house.** It is an unproxied A record holding the house's WAN
|
||||
address (`platform/externaldns/README.md`), and the router has no hairpin NAT, so a LAN client that
|
||||
sends to it never comes back in through the port-forward. The Cloudflare Tunnel is no way round: it
|
||||
carries HTTP(S) to the Gateway and WireGuard is UDP to a node, so `wan.` has to stay DNS-only and
|
||||
`away` has to keep using it.
|
||||
|
||||
**`home` depends on `k3s-prod-1`.** Its Endpoint is that node's address, with no failover: if
|
||||
prod-1 is down, `home` cannot handshake even though the other two listeners are healthy. The fix
|
||||
is client-side: edit the tunnel's Endpoint to `192.168.2.11` or `.12`, since every node answers the
|
||||
same keys. No server change and no new key is needed. The same dependency applies to `away` if the
|
||||
WAN forward points at prod-1.
|
||||
|
||||
Android allows one active VPN at a time, so switch tunnels by hand when leaving or arriving.
|
||||
|
||||
## Operating it
|
||||
|
||||
* **Enrol a device:** `./infra/vpn-ingress/new-peer.sh <name> <octet>` → paste the printed `[Peer]`
|
||||
@@ -215,10 +254,13 @@ right way; one was a defect in my own manifest.
|
||||
handshake — and would be a real secret, forcing `peers.conf` into SOPS and costing the
|
||||
reviewability above. Revisit only if the threat model changes; if it does, the PSK belongs in the
|
||||
`wg-server-key` Secret.
|
||||
* **No IPv6.** The server offers no IPv6 tunnel address, so a full-tunnel client with native IPv6
|
||||
still reaches IPv6 sites *outside* the tunnel. Functionally harmless, a privacy leak if the point
|
||||
was to hide the client's network. Fix is an `fd..` ULA on `wg0` plus `::/0` in the client's
|
||||
`AllowedIPs`; not done because nothing here needs it.
|
||||
* **No IPv6.** The server offers no IPv6 tunnel address and `new-peer.sh` emits only `0.0.0.0/0`.
|
||||
What happens to a client's IPv6 then depends on the client. The **Android app does not leak it**:
|
||||
its `GoBackend` skips `allowFamily()` when the config is one peer with a default route, so Android
|
||||
blocks IPv6 for the VPN's duration and apps fall back to IPv4. Other clients are not covered by
|
||||
that code; expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside the tunnel. Fix is
|
||||
an `fd..` ULA on `wg0` plus `::/0` in the client's `AllowedIPs`; not done because nothing here
|
||||
needs it.
|
||||
* **No in-cluster-service reach beyond what the LAN has.** Clients reach cluster apps the way a LAN
|
||||
browser does — through the Gateway VIP. Pod IPs and `10.43.0.0/16` service IPs are not routed
|
||||
into the tunnel.
|
||||
@@ -30,9 +30,12 @@
|
||||
# `cluster` entity, so VPN clients would not have been able to resolve `*.${BASE_DOMAIN}` either.
|
||||
# The VPN would have looked up and been useless, which is the worst failure shape available.
|
||||
#
|
||||
# In the host netns there is no pod identity to carry: forwarded client traffic is SNATed to the
|
||||
# NODE's address, so it reaches the Gateway, PiHole and every LAN host as `host`/`remote-node` —
|
||||
# exactly what a LAN client looks like. Three further problems disappear with it:
|
||||
# In the host netns there is no pod identity to carry, so a client never looks like a pod. What a
|
||||
# destination sees depends on how the packet is handled. LAN hosts and the internet: forwarded,
|
||||
# then MASQUERADEd to the NODE's address by the PostUp rule below. LoadBalancer VIPs: Cilium's load
|
||||
# balancer takes the packet as it enters from wg0, before that rule, so the source is still
|
||||
# 10.90.0.x with identity `world`, like a LAN browser (measured 2026-11-09 for PiHole .23 and Plex
|
||||
# .24; see README). Three further problems disappear with it:
|
||||
# * net.ipv4.ip_forward is ALREADY 1 on these nodes (node_baseline writes it to
|
||||
# /etc/sysctl.d/99-k3s-cilium.conf), so no pod sysctl is needed — and `net.ipv4.ip_forward`
|
||||
# is NOT in the kubelet's safe-sysctl allowlist, so a pod-netns version of this would have
|
||||
|
||||
@@ -90,3 +90,7 @@
|
||||
[Peer]
|
||||
PublicKey = pBRSHrRyG2hnsuZ6FvHfEzF8hOVTF5stHUb/7fW2Km8=
|
||||
AllowedIPs = 10.90.0.2/32
|
||||
# keith-phone-home — <owner>, enrolled 2026-11-08 (at-home profile, LAN Endpoint; no hairpin NAT)
|
||||
[Peer]
|
||||
PublicKey = AT8lBJHqrVkv8RL9n0qWDsh5nyl0e3qgrfTAaYKgBz0=
|
||||
AllowedIPs = 10.90.0.3/32
|
||||
+5
-3
@@ -144,7 +144,9 @@
|
||||
"description": "This repo's own CI-published images, pinned by digest, following the tag named in a `# renovate-digest:` marker on the previous line.",
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)versions\\.env$/",
|
||||
"/(^|/)platform/pm-agent/.+\\.ya?ml$/"
|
||||
"/(^|/)platform/pm-agent/.+\\.ya?ml$/",
|
||||
// ADDED 2026-11-08: the merge gate's job `container:` is the toolbox, by digest.
|
||||
"/^\\.gitea/workflows/gate\\.ya?ml$/"
|
||||
],
|
||||
"matchStrings": [
|
||||
"#\\s*renovate-digest:\\s*image=(?<depName>\\S+)\\s+tag=(?<currentValue>\\S+)\\s*\\n[^\\n]*?(?<currentDigest>sha256:[a-f0-9]{64})"
|
||||
@@ -313,8 +315,8 @@
|
||||
"enabled": false
|
||||
},
|
||||
{
|
||||
"description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids.",
|
||||
"matchManagers": ["kubernetes"],
|
||||
"description": "ADDED 2026-10-29. This repo's own images are followed ONLY through the `# renovate-digest:` markers above. The `kubernetes` manager also sees platform/pm-agent/deployment.yaml and infra/agent/worker-job-template.yaml, where the image has a digest and no tag; left on, it would look those up as `latest` (a tag CI never pushes) and, for the template, re-open the self-referential loop the custom manager deliberately avoids. `github-actions` ADDED 2026-11-08 for the same reason: it extracts the gate job's digest-only `container: image:` in .gitea/workflows/gate.yml, which the custom manager already follows through its marker.",
|
||||
"matchManagers": ["kubernetes", "github-actions"],
|
||||
"matchPackageNames": [
|
||||
"gitea.home.keithjackson.dev/home-network/toolbox",
|
||||
"gitea.home.keithjackson.dev/home-network/agent-runner",
|
||||
|
||||
@@ -123,6 +123,9 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
|
||||
- **`vpn-remote-access`** — the INBOUND remote-access VPN (WireGuard, udp/51820): standing it up,
|
||||
enrolling and revoking devices, and the off-LAN checks that are the only honest proof it works.
|
||||
The opposite direction to the two pages above — do not confuse them.
|
||||
- **`cloudflare-tunnel`** — the Cloudflare Tunnel connector (inbound HTTPS from the Cloudflare edge to
|
||||
the shared Gateway): enrolling it, the canary, `CloudflaredTunnelDown` / `CloudflaredReplicasDegraded`,
|
||||
and rollback. Never `cloudflared tunnel route dns` — external-dns owns the records.
|
||||
|
||||
### Backups and DR
|
||||
- **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here.
|
||||
|
||||
@@ -0,0 +1,367 @@
|
||||
# cloudflare-tunnel — enrolling the Cloudflare Tunnel, proving it, and the two alerts
|
||||
|
||||
Alerts that link here: **`CloudflaredTunnelDown`** (no replica has a live edge connection, or none
|
||||
is being scraped) and **`CloudflaredReplicasDegraded`** (fewer than 2 replicas Available).
|
||||
|
||||
Component: `platform/cloudflared/` (Flux Kustomization `cloudflared`, `clusters/base/cloudflared.yaml`).
|
||||
Design record: `decisions/2026-11-08-cloudflare-tunnel.md`.
|
||||
|
||||
Every step below is an **operator** step. An agent must not mint the tunnel credential, run
|
||||
`flux reconcile`, or mutate the cluster (CLAUDE.md §5, §6).
|
||||
|
||||
## What this means here
|
||||
|
||||
cloudflared is an **outbound-only** connector: two replicas dial the Cloudflare edge on 7844
|
||||
(QUIC/udp, falling back to HTTP/2 over tcp) and carry requests back to the shared Gateway
|
||||
(`https://cilium-gateway-shared.gateway.svc.cluster.local:443`, SNI = the request's Host, so the
|
||||
Gateway's existing `wildcard-home-tls` Let's Encrypt certificate is what cloudflared validates).
|
||||
The tunnel is **locally managed**: its ingress rules are `platform/cloudflared/config.yaml`, in git,
|
||||
not in the Cloudflare dashboard.
|
||||
|
||||
The operator decisions it encodes (2026-11-08):
|
||||
|
||||
1. Public hostnames stay `*.home.keithjackson.dev` and the apex; edge certificates come from
|
||||
Cloudflare **ACM + Total TLS**.
|
||||
2. **Plex is not on the tunnel** (`plex`, `plex-vip` → 404).
|
||||
3. **jackett, pihole, semaphore, qbittorrent, radarr, sonarr** and **headlamp** stay LAN/VPN-only,
|
||||
as do the L4 VIPs **unifi** and **crafty-mc** (all → 404 in cloudflared, and the LAN-only
|
||||
Gateway backends are also refused by the egress policy). Headlamp is an operator decision of
|
||||
2026-11-08; the other six are the catalog's `exposure: internal` services.
|
||||
4. **The Kubernetes API is never reachable through the tunnel.** No rule targets it and the egress
|
||||
policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check
|
||||
`cloudflared-ingress-rules` fails the gate if that ever changes.
|
||||
|
||||
**Cloudflare routes a hostname through the tunnel only when its record is a proxied CNAME to
|
||||
`<UUID>.cfargotunnel.com`.** Phase 1 created no such record.
|
||||
|
||||
**Phase 2 — DONE (2026-11-08, branch `feat/dns-via-cloudflare-tunnel`).** external-dns now publishes
|
||||
every gateway name as a proxied CNAME to the tunnel: the shared Gateway's
|
||||
`external-dns.alpha.kubernetes.io/target` is `<UUID>.cfargotunnel.com` and the HelmRelease sets
|
||||
`--cloudflare-proxied` (see `platform/externaldns/README.md`). Plex's web UI and Headlamp are
|
||||
disowned (`controller: none`), so their public records are deleted; unifi and crafty-mc stay DNS-only
|
||||
A records (`cloudflare-proxied: "false"`); `wan.` stays for WireGuard. The L0 check
|
||||
`internal-services-publish-no-public-dns` (9) pins all of this, including that the Gateway target's
|
||||
UUID equals `tunnel:` in `config.yaml`. **So from Phase 2 on, these alerts firing DOES mean public
|
||||
users are down** — every public HTTP(S) name depends on the connectors.
|
||||
|
||||
### After the Phase 2 merge — operator step (router)
|
||||
|
||||
Once `dig +short @1.1.1.1 <public name>` returns Cloudflare anycast addresses for the public names
|
||||
(and external-dns logs show the UPDATEs), nothing legitimate reaches the WAN on 80/443 any more:
|
||||
|
||||
1. **Close** the router's WAN **tcp/80** and **tcp/443** port-forwards to the Gateway VIP.
|
||||
2. **Keep** WAN **udp/51820** (WireGuard → `wan.home.keithjackson.dev`) and WAN **tcp/32400**
|
||||
(Plex remote streaming via plex.tv; `runbooks/plex-remote-access.md`).
|
||||
3. Verify from off-LAN (phone on cellular): a public name still loads (via the tunnel), and
|
||||
`curl -m5 https://<WAN IP>/` now times out instead of answering.
|
||||
|
||||
Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and
|
||||
`CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization
|
||||
is `wait: false` and nothing depends on it.
|
||||
|
||||
## First commands
|
||||
|
||||
```
|
||||
# Are the pods running, and on different nodes?
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n cloudflared get deploy,pods -o wide'
|
||||
|
||||
# What does cloudflared say? Healthy: four "Registered tunnel connection" lines per replica.
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n cloudflared logs deploy/cloudflared --tail=50'
|
||||
|
||||
# Did Flux apply the current revision, and is the Secret decrypted (key NAMES only)?
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n flux-system get kustomization cloudflared;
|
||||
kubectl --context k3s-prod -n cloudflared get secret cloudflared-credentials \
|
||||
-o go-template="{{range \$k, \$v := .data}}{{\$k}}{{\"\n\"}}{{end}}"'
|
||||
```
|
||||
|
||||
Never print the Secret's values. `ENC[` in the decrypted Secret means the overlay's decryption
|
||||
patch is missing; see causes below.
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
**Read the FIRST error line, not the last.** When cloudflared cannot use the credentials file it
|
||||
falls back to looking for an origin certificate, and the error it ends on —
|
||||
`Cannot determine default origin certificate path` / `error parsing tunnel ID: Error locating
|
||||
origin cert` — is a **red herring** whatever the real cause was. The credentials file is the
|
||||
whole of what this Deployment needs; no `cert.pem` ever belongs in the pod.
|
||||
|
||||
1. **`permission denied` on `credentials.json` → the "origin cert" error is a red herring.** Logs:
|
||||
`couldn't read tunnel credentials from /etc/cloudflared/creds/credentials.json: permission
|
||||
denied`, then the origin-cert lines. The container (uid 65532) cannot read the Secret file:
|
||||
the pod `securityContext.fsGroup: 65532` is missing, so the 0440 file is `root:root`. Fix it in
|
||||
git (`platform/cloudflared/deployment.yaml`); the L0 check `cloudflared-ingress-rules`,
|
||||
assertion (7), should already have refused that. Do NOT loosen `defaultMode` to 0444. (This is
|
||||
what CrashLooped every replica on 2026-11-08.)
|
||||
2. **The tunnel has not been enrolled yet** (Phase 1, steps 1-5 below not done). Logs show an
|
||||
error about the tunnel `REPLACE-ME-TUNNEL-UUID` (and the origin-cert red herring). Do steps 1-5.
|
||||
3. **UUID and credentials disagree** — `config.yaml` `tunnel:` is not the `TunnelID` in the
|
||||
credentials JSON. Logs: authentication / "tunnel not found". Check with step 4's verify command.
|
||||
4. **Credentials still encrypted in the cluster** — the `cloudflared` decryption patch is missing
|
||||
from `clusters/prod/kustomization.yaml`. The L0 check guards this, so this means someone bypassed
|
||||
it.
|
||||
5. **Egress blocked** — a policy change, or the edge moved off 7844. Look for drops:
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
|
||||
hubble observe --verdict DROPPED --from-namespace cloudflared --last 100 -o compact'
|
||||
```
|
||||
(`ds/cilium` picks one node's agent; repeat per node if empty.)
|
||||
6. **The tunnel was deleted or its secret rotated in Cloudflare** — `cloudflared tunnel list`
|
||||
(step 2's podman form) no longer shows it. Re-enrol with steps 2-5.
|
||||
7. **`CloudflaredReplicasDegraded` only**: one replica cannot schedule (the spread is hard, one per
|
||||
node — a cordoned node during a kured reboot is normal and clears) or an image pull failed.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
- **Stop** if anything suggests a rule reaching the Kubernetes API or a LAN-only service. Do not
|
||||
hot-fix through the dashboard: the tunnel is locally managed, and a dashboard edit is silently
|
||||
overridden by `config.yaml`. Use the emergency stop under *Rollback*, then fix in git.
|
||||
- **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one.
|
||||
**external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`);
|
||||
a record created behind its back either fights it or is deleted by it, and a real hostname
|
||||
routed that way is an unreviewed DNS flip. Records reach the tunnel only through external-dns
|
||||
(Phase 2, done). The only record this runbook creates is the hand-made canary in step 6, and you delete it.
|
||||
- Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is
|
||||
Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2.
|
||||
|
||||
---
|
||||
|
||||
## Enrolling the tunnel (once)
|
||||
|
||||
cloudflared is **not** in the pinned toolbox (checked 2026-11-08). Run it from the same pinned image
|
||||
the Deployment uses, under podman — no host install. The working directory is
|
||||
`~/.config/sops/cloudflared`: it is private to you, and it is the one host directory `dev.sh`
|
||||
already mounts into the toolbox (as `/root/.config/sops`), so the toolbox's `sops` can read the
|
||||
credentials JSON without it ever entering the repository tree.
|
||||
|
||||
```
|
||||
CF_IMG='docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba'
|
||||
CF_DIR="$HOME/.config/sops/cloudflared"
|
||||
mkdir -p "$CF_DIR" && chmod 700 "$CF_DIR"
|
||||
cf() { podman run --rm -it --user 0 --security-opt label=disable \
|
||||
-e HOME=/root -v "$CF_DIR:/root/.cloudflared" "$CF_IMG" "$@"; }
|
||||
```
|
||||
|
||||
(Keep `CF_IMG` in step with `platform/cloudflared/deployment.yaml` if Renovate has bumped it.)
|
||||
|
||||
### Step 1 — edge certificate (ACM) and edge settings (Cloudflare dashboard)
|
||||
|
||||
Universal SSL covers only `keithjackson.dev` and `*.keithjackson.dev`; it does **not** cover the
|
||||
second-level `*.home.keithjackson.dev`. In the `keithjackson.dev` zone:
|
||||
|
||||
1. **SSL/TLS → Edge Certificates → Advanced Certificate Manager**: subscribe.
|
||||
2. Then EITHER **Order an advanced certificate** for exactly `home.keithjackson.dev,
|
||||
*.home.keithjackson.dev` (covers every name at once, independent of proxy state), OR turn on
|
||||
**Total TLS** (issues one certificate per *proxied* hostname as each one appears). The ordered
|
||||
certificate is the more predictable of the two for the later DNS flip; either satisfies the
|
||||
operator decision.
|
||||
3. **Wait until the certificate shows `Active`** in the Edge Certificates list. Do not do step 6's
|
||||
canary before then — a TLS error at the edge would be the certificate, not the tunnel.
|
||||
4. Leave **SSL/TLS encryption mode** at *Full (strict)*. (It does not govern tunnel origins —
|
||||
cloudflared verifies the Gateway itself — but it is the safe default for anything else.)
|
||||
|
||||
Edge settings that would break non-browser clients once traffic moves (check them now, in the zone):
|
||||
|
||||
- **Security → Bots → Bot Fight Mode: OFF.** It cannot be scoped by hostname or skipped by a WAF
|
||||
rule, and it challenges `git`, `docker`/registry pushes, Nextcloud sync clients and API calls.
|
||||
- **Browser Integrity Check** and the **WAF / managed rules**: make sure neither challenges or
|
||||
blocks non-browser user agents on `gitea.home.keithjackson.dev` (git over HTTPS and the container
|
||||
registry under `/v2/`) or on API paths. If either does, add a WAF **Skip** custom rule for those
|
||||
hostnames/paths rather than turning protection off zone-wide.
|
||||
- **Network → WebSockets: ON** (Grafana live, Headlamp-style consoles, Nextcloud, KubeVirt VNC).
|
||||
|
||||
### Step 2 — log in and create the tunnel
|
||||
|
||||
```
|
||||
cf tunnel login
|
||||
# Open the printed URL, pick the keithjackson.dev zone. Writes $CF_DIR/cert.pem — an ACCOUNT-level
|
||||
# origin certificate. Treat it as a credential.
|
||||
|
||||
cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s
|
||||
# Prints "Created tunnel home-k3s with id <UUID>". The UUID is not secret; note it.
|
||||
cf tunnel list
|
||||
```
|
||||
|
||||
### Step 3 — put the UUID into the config
|
||||
|
||||
Do this in your own worktree on a new branch, as CLAUDE.md §0 requires:
|
||||
|
||||
```
|
||||
UUID=<the id from step 2>
|
||||
sed -i "s/^tunnel: REPLACE-ME-TUNNEL-UUID$/tunnel: ${UUID}/" platform/cloudflared/config.yaml
|
||||
grep '^tunnel:' platform/cloudflared/config.yaml
|
||||
```
|
||||
|
||||
### Step 4 — encrypt the credentials into the Secret, without printing them
|
||||
|
||||
Run from that worktree. `sops set` edits the already-encrypted document in place; the JSON goes
|
||||
from the mounted file to `sops` inside the container and never reaches the terminal:
|
||||
|
||||
```
|
||||
./infra/devcontainer/dev.sh run bash -c '
|
||||
set -euo pipefail
|
||||
cd /work
|
||||
f=/root/.config/sops/cloudflared/tunnel.json
|
||||
v="$(jq -c . "$f" | tr -d "\n" | jq -Rs .)"
|
||||
sops set platform/cloudflared/cloudflared-credentials.sops.yaml \
|
||||
"[\"stringData\"][\"credentials.json\"]" "$v"
|
||||
echo "credentials written"
|
||||
'
|
||||
```
|
||||
|
||||
(From a worktree, `dev.sh` needs `DEV_KUBECONFIG=/dev/null` and the `.git` mount shim — CLAUDE.md §3.
|
||||
Or run this step from the main checkout and copy the encrypted file across.)
|
||||
|
||||
Verify — prints only `ok` or fails, never a value:
|
||||
|
||||
```
|
||||
./infra/devcontainer/dev.sh run bash -c '
|
||||
cd /work
|
||||
UUID=$(sed -n "s/^tunnel: //p" platform/cloudflared/config.yaml)
|
||||
sops -d --extract "[\"stringData\"][\"credentials.json\"]" \
|
||||
platform/cloudflared/cloudflared-credentials.sops.yaml \
|
||||
| jq -e --arg u "$UUID" ".TunnelID == \$u and (.TunnelSecret | length) > 20 and (.AccountTag | length) > 0" \
|
||||
>/dev/null && echo ok'
|
||||
```
|
||||
|
||||
Then store `tunnel.json` **and** `cert.pem` in your password manager, and delete them from disk:
|
||||
|
||||
```
|
||||
rm -f "$CF_DIR/tunnel.json" "$CF_DIR/cert.pem"
|
||||
```
|
||||
|
||||
(`cert.pem` is only needed again to inspect (`tunnel info`), create or delete a tunnel. DNS records are
|
||||
external-dns's, never `tunnel route dns`'s.)
|
||||
|
||||
### Step 5 — PR it
|
||||
|
||||
Commit `config.yaml` and the re-encrypted `cloudflared-credentials.sops.yaml`, gate, open a PR. No
|
||||
other wiring is needed: the resource line and both overlay decryption patches already exist. On
|
||||
merge Flux rolls the Deployment (the ConfigMap is content-hashed).
|
||||
|
||||
That PR touches `**/*.sops.yaml`, a sensitive path in `infra/gitea/approval-policy.json`, so
|
||||
`approval-policy` will block it until you post `/approve-override <reason>` as a PR **review** of
|
||||
type Comment (CLAUDE.md §1).
|
||||
|
||||
### Step 6 — verify
|
||||
|
||||
```
|
||||
# Both replicas Ready, on different nodes:
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n cloudflared get pods -o wide'
|
||||
|
||||
# Logs: "Registered tunnel connection connIndex=0..3" on each replica.
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n cloudflared logs -l app.kubernetes.io/name=cloudflared --tail=40'
|
||||
```
|
||||
|
||||
From the workstation: `cf tunnel info home-k3s` (needs `cert.pem` back in `$CF_DIR` temporarily)
|
||||
lists **two connectors**. In Grafana/Prometheus, `sum(cloudflared_tunnel_ha_connections)` is ~8 and
|
||||
both alerts are clear.
|
||||
|
||||
**Canary — end to end, without touching any real record.** Hand-make ONE proxied CNAME for a
|
||||
throwaway name (Cloudflare dashboard → DNS → Records → Add record):
|
||||
|
||||
Type CNAME Name tunnel-canary.home Target <UUID>.cfargotunnel.com Proxy status: Proxied
|
||||
|
||||
Do NOT use `cloudflared tunnel route dns` for it (see *When to escalate or stop*). The name hits the
|
||||
wildcard rule, so it travels edge → cloudflared → Gateway, where no HTTPRoute matches it:
|
||||
|
||||
```
|
||||
curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://tunnel-canary.home.keithjackson.dev/
|
||||
# expected: "404 0" — a 404 from the Gateway's Envoy (no route for that host) over a VALID edge
|
||||
# certificate (ssl_verify_result 0). A TLS error = step 1 is not Active yet. 502 = cloudflared
|
||||
# could not reach or verify the Gateway; check the logs and `hubble observe` above.
|
||||
```
|
||||
|
||||
**Then prove a REAL backend answers the tunnel's identity without a 403.** The canary cannot show
|
||||
this: Envoy 404s an unknown host before it resolves any backend, and the `403 Access denied`
|
||||
failure (Cilium re-checking the source pod's egress against the backend, CVE-2025-30162) only
|
||||
happens once a backend is chosen. An ephemeral container in a cloudflared pod shares that pod's
|
||||
network identity, so it takes exactly the tunnel's path to the Gateway Service:
|
||||
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c '
|
||||
POD=$(kubectl --context k3s-prod -n cloudflared get pod -l app.kubernetes.io/name=cloudflared \
|
||||
-o jsonpath="{.items[0].metadata.name}")
|
||||
kubectl --context k3s-prod -n cloudflared debug "$POD" -q -i --image=docker.io/curlimages/curl:8.11.1 \
|
||||
--target=cloudflared -- sh -c "for h in grafana homepage-unused gitea; do
|
||||
printf \"%s \" \$h; curl -sS -o /dev/null -w \"%{http_code}\n\" \
|
||||
--connect-to ::cilium-gateway-shared.gateway.svc.cluster.local:443 https://\$h.home.keithjackson.dev/; done"'
|
||||
# expected: grafana and gitea answer 200/302 (anything but 403); a 403 there means the egress
|
||||
# policy's backend selector does not cover that namespace. "homepage-unused" is a control: 404.
|
||||
# A LAN-only host (e.g. jackett) is EXPECTED to 403 here — that is the second layer working.
|
||||
```
|
||||
|
||||
(`kubectl debug` adds an ephemeral container: a cluster mutation, so an operator action. It
|
||||
disappears with the pod on the next rollout.)
|
||||
|
||||
Finally **delete the `tunnel-canary` CNAME** in the dashboard and confirm it is gone:
|
||||
`dig +short tunnel-canary.home.keithjackson.dev @1.1.1.1` returns nothing.
|
||||
|
||||
## Rotating the tunnel credential
|
||||
|
||||
Do this whenever the credentials JSON (`TunnelSecret`) or `cert.pem` may have been exposed — a
|
||||
transcript, a log, a screen share. A tunnel's secret cannot be changed in place from this
|
||||
workflow, so rotation is **a new tunnel**: the old credential stays valid until the old tunnel
|
||||
is deleted, which is why the last step is not optional.
|
||||
|
||||
1. **Create the new tunnel** (*Enrolling*, step 2's `cf` helper; `cf tunnel login` again if
|
||||
`cert.pem` is not to hand). Use a NEW name — the old one is still taken:
|
||||
```
|
||||
cf tunnel list # note the OLD tunnel's id and name
|
||||
cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s-2
|
||||
```
|
||||
2. **Swap the UUID** in your own worktree, on a new branch (CLAUDE.md §0):
|
||||
```
|
||||
OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml)
|
||||
sed -i "s/^tunnel: ${OLD}$/tunnel: <NEW UUID>/" platform/cloudflared/config.yaml
|
||||
# Phase 2: the Gateway's external-dns target carries the SAME UUID. Change both in one commit;
|
||||
# the L0 check internal-services-publish-no-public-dns (9a) fails the gate if they disagree.
|
||||
sed -i "s/${OLD}\.cfargotunnel\.com/<NEW UUID>.cfargotunnel.com/" platform/gateway/gateway.yaml
|
||||
```
|
||||
3. **Swap the encrypted credentials** with *Enrolling* step 4 — the same `sops set`, then the
|
||||
same `ok`-only verify, which now checks the JSON's `TunnelID` against the NEW UUID. Never
|
||||
`sops -d` to the terminal.
|
||||
4. **Merge** — gate, PR, `/approve-override` by the operator (it touches `**/*.sops.yaml`), merge.
|
||||
Flux rolls the Deployment because the content-hashed ConfigMap changed.
|
||||
5. **Verify** with *Enrolling* step 6: both replicas Ready, `Registered tunnel connection` in
|
||||
each replica's log, `cf tunnel info home-k3s-2` lists two connectors, and
|
||||
`cf tunnel info <old name>` lists **none**.
|
||||
6. **Only then delete the old tunnel**, which revokes the exposed credential:
|
||||
```
|
||||
cf tunnel cleanup <old name> # drops any stale connector records first
|
||||
cf tunnel delete <old name>
|
||||
cf tunnel list # the old tunnel is gone
|
||||
```
|
||||
Since Phase 2, records point at `<UUID>.cfargotunnel.com`: confirm external-dns has repointed
|
||||
every record to the NEW UUID (its logs show an UPDATE per name) before deleting, or those
|
||||
hostnames go dark.
|
||||
Then replace `tunnel.json`/`cert.pem` in the password manager and delete them from disk.
|
||||
|
||||
## Rollback
|
||||
|
||||
- **Phase 1 (no DNS on the tunnel):** revert the PR. Flux prunes the `cloudflared` namespace and
|
||||
everything in it. No user traffic is affected either way. Optionally `cf tunnel delete home-k3s`
|
||||
(needs `cert.pem`) to retire the credential.
|
||||
- **Emergency stop (operator, cluster mutation):** stop the connectors without waiting for a PR:
|
||||
```
|
||||
infra/devcontainer/dev.sh run bash -c \
|
||||
'flux --context k3s-prod suspend kustomization cloudflared &&
|
||||
kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0'
|
||||
```
|
||||
Cloudflare then answers tunnel hostnames with a 530/1033 error — **from the LAN too** for the
|
||||
names PiHole does not list (apex, grafana, crafty, seerr; since 2026-11-09). Follow up in git, then
|
||||
`flux resume kustomization cloudflared`.
|
||||
- **After the DNS flip (Phase 2):** the path back is a revert PR of `feat/dns-via-cloudflare-tunnel`
|
||||
(Gateway target back to `wan.${BASE_DOMAIN}`, drop `--cloudflare-proxied`) — external-dns then
|
||||
rewrites every record DNS-only to `wan.`. That only works if the router's WAN tcp/80+443 forwards
|
||||
are **re-opened** first (they are closed after Phase 2). **Also restore PiHole's zone wildcard**
|
||||
(`address=/${BASE_DOMAIN}/${GATEWAY_VIP}` in `catalog/pihole.yaml`, and drop the
|
||||
`pihole-lan-names` L0 assertion (a)): since 2026-11-09 the apex, grafana, crafty and seerr are not
|
||||
listed there and resolve publicly, and a public `wan.` answer does not hairpin from the LAN. Never edit the records by hand:
|
||||
external-dns owns them and `policy: sync` reverts a hand edit. Revoking the tunnel credential (`cf tunnel delete`, or rotating it
|
||||
in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.
|
||||
@@ -58,6 +58,7 @@ configMapGenerator:
|
||||
# improvised live, and the one hazard it exposed -- a StatefulSet with
|
||||
# persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 --
|
||||
# destroyed monitoring/loki's volume before anyone noticed.
|
||||
- cloudflare-tunnel=cloudflare-tunnel.md
|
||||
- cluster-maintenance-shutdown=cluster-maintenance-shutdown.md
|
||||
- cdi-import-stuck=cdi-import-stuck.md
|
||||
- cert-expiry=cert-expiry.md
|
||||
|
||||
@@ -11,6 +11,19 @@ env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway UR
|
||||
**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
|
||||
procedures with a manual last mile.
|
||||
|
||||
## The Plex web UI is LAN/VPN-only (2026-11-08)
|
||||
|
||||
When public DNS moved to the Cloudflare Tunnel (`runbooks/cloudflare-tunnel.md`, Phase 2), the
|
||||
operator kept **Plex off the tunnel**. `catalog/plex.yaml` is now `exposure: internal`, so the
|
||||
`plex.home.keithjackson.dev` route carries `external-dns ... controller: none` and its public record
|
||||
is deleted; the tunnel also 404s `plex` and `plex-vip`. On the LAN (and over WireGuard) the web UI
|
||||
at `https://plex.home.keithjackson.dev` works exactly as before through PiHole's split-horizon answer.
|
||||
|
||||
**Remote streaming is unaffected** — it never used that name. It is the path below: plex.tv hands
|
||||
clients the `plex.direct` URL (`ADVERTISE_IP`) for the L4 VIP `192.168.2.24`, reached through the
|
||||
WAN **tcp/32400** forward. That forward must stay open when the WAN 80/443 forwards are closed after
|
||||
the tunnel cut-over. Remote users use app.plex.tv, not our hostname.
|
||||
|
||||
## What was actually broken (measured 2026-10-03)
|
||||
|
||||
Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
|
||||
@@ -44,9 +57,10 @@ either way — this repo has no such vantage point and must not pretend otherwis
|
||||
router by hand, so it must never move on a reschedule.
|
||||
|
||||
While in the router, **check what else is bound to the WAN interface.** If remote management is
|
||||
listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
|
||||
Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
|
||||
though remote *streaming* works. These are two separate paths and they fail independently.
|
||||
listening on 443 (see the certificate measurement above), move it or disable it. (Historical: this
|
||||
used to shadow the Gateway's 80/443 forward for the Plex *web UI* over the internet. Since
|
||||
2026-11-08 the web UI is LAN/VPN-only and public HTTP(S) goes through the Cloudflare Tunnel, so the
|
||||
80/443 forwards are retired; only tcp/32400 matters here.)
|
||||
|
||||
Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
|
||||
|
||||
|
||||
@@ -74,7 +74,7 @@ Work through these in order; the first three are by far the most likely.
|
||||
|---|---|---|
|
||||
| `GET /user returned no login — is GITEA_TOKEN valid?` and a CrashLoop | the credentials are still the committed **placeholders**, or the token was revoked | fill them in per `platform/pm-agent/pm-agent-credentials.sops.yaml.example` |
|
||||
| `ImagePullBackOff`, digest `sha256:0000…` | (historical: before 2026-10-30 the committed digest was an all-zero placeholder) the image has never been published at the digest named | CI publishes it, not a person: check the latest `images` workflow run on `main` (it fails loudly if the `REGISTRY_PUSH_TOKEN` secret is missing; re-run it with `workflow_dispatch`), then merge Renovate's digest-bump PR, which updates **both** `deployment.yaml` and `configmap.yaml` (`PM_WORKER_IMAGE`) together — but read the row below first |
|
||||
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and gets the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
||||
| `ImagePullBackOff` with `x509: certificate is not valid for any names` | **diagnosed.** containerd pulls as a node process, resolves through the node's public upstreams, and (before the 2026-11-08 Tunnel cutover; the public answer is now Cloudflare) got the CNAME to the WAN IP — which this router does not hairpin, so it answers :443 with its own `CN=%D` placeholder certificate (no SANs at all). Nothing to do with the digest, the Gateway or the wildcard cert | the `/etc/hosts` pin for `gitea.{{ k3s_base_domain }}` → `192.168.2.20` in `infra/ansible/inventory/group_vars/k3s_prod/vars.yml` fixes it — the same pin the Keycloak issuer already has. Apply it with the project's own tooling, scoped to just that task: `cd infra/ansible && ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`. Full evidence: `decisions/2026-10-06-pm-agent-node-resolves-gitea-to-wan.md` |
|
||||
| `ImagePullBackOff` with `401 Unauthorized` / `unauthorized` | the pull credential is wrong, absent or scoped without `read:package`. The `pm-agent-registry` Secret exists and is referenced; its *value* is what starts as a placeholder | check the key names only, never values: `kubectl -n pm-agent get secret pm-agent-registry -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'` should list `.dockerconfigjson`. Then confirm the token carries `read:package`, that its account can read the org's packages (a team with the Packages unit — see the `.example`), and that the registry key inside the JSON is the bare host `gitea.home.keithjackson.dev` — with a scheme or a port it matches nothing and the pull goes out anonymously, which looks identical to a missing Secret. Steps: `platform/pm-agent/pm-agent-registry.sops.yaml.example` |
|
||||
| `GET /user` answers `403` / `token does not have at least one of required scope(s): [read:user]` at startup, then a CrashLoop | the token lacks `read:user` — Gitea 1.24.3 scopes the whole `/api/v1/user` group, and the dispatcher's first call is `GET /user` | mint a new token with all four scopes (`read:user`, `write:repository`, `write:issue`, `read:package`) and put it in **both** Secrets. Add `read:notification` if you want the cheap pull-request mention query (`platform/pm-agent/README.md`) |
|
||||
| a worker Job is refused by admission (`pm-agent-job-shape`, `imagePullSecrets`) | `PM_WORKER_IMAGE` is a digest built before the template carried `imagePullSecrets` (PR #61), so its baked template lacks the entry admission requires | pin the agent-runner digest CI published after #61 merged (Renovate's bump PR); until then keep `PM_DRY_RUN=true` |
|
||||
|
||||
@@ -252,7 +252,7 @@ cellular, with Wi-Fi off, and check all four — each proves a different hop:
|
||||
|---|---|---|
|
||||
| 1 | tunnel shows a recent handshake in the WireGuard app | the WAN forward + the server key + the peer stanza |
|
||||
| 2 | `ping 192.168.2.1` (the router) | decrypt → FORWARD → LAN, and the SNAT return path |
|
||||
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway accepting a `host`-identity source — the specific thing a pod-networked VPN would have 403'd |
|
||||
| 3 | open `https://plex.home.keithjackson.dev` | PiHole DNS over the tunnel **and** the Gateway admitting a tunnel client, which a pod-networked VPN would have 403'd. Expected source identity is `world`, like a LAN browser: measured 2026-11-09 on VIPs `.23` and `.24`, not yet observed on the Gateway's `.20` |
|
||||
| 4 | visit an IP-echo site; it should show `136.51.115.252` | full-tunnel WAN egress out the house, not the carrier |
|
||||
|
||||
Then confirm server-side:
|
||||
@@ -260,18 +260,97 @@ Then confirm server-side:
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- wg show'
|
||||
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer.
|
||||
# a `latest handshake` under ~3 min, and non-zero transfer, on the expected peer. `ds/wireguard`
|
||||
# picks ONE pod; if it reads 0, ask the pod on the node holding the forward (see Troubleshooting).
|
||||
```
|
||||
|
||||
**If check 3 fails while 2 passes**, the tunnel is fine and the problem is DNS or the Gateway, not
|
||||
WireGuard. Check the client's `DNS = 192.168.2.23` and that PiHole is up.
|
||||
|
||||
**If 1 and 2 pass but large transfers hang**, it is MTU. The profile ships `MTU = 1420` (1500 − 60
|
||||
bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
|
||||
**If 1 and 2 pass but pages or large transfers hang**, suspect the client MTU; see
|
||||
Troubleshooting.
|
||||
|
||||
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from outside —
|
||||
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
|
||||
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
|
||||
**If nothing handshakes at all**, you cannot tell a wrong forward from a wrong key from the
|
||||
device, by design. `wg show` cannot tell them apart either: it reads the same in both cases. The
|
||||
node's conntrack table can; see Troubleshooting.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
Ask the pod on **the node carrying the traffic**: the node the WAN forward names for `away`,
|
||||
`k3s-prod-1` for `home`. Sessions are per node, and the other two pods read `0`.
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress get pods -o wide'
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- wg show wg0 latest-handshakes'
|
||||
# likewise: `wg show wg0 endpoints`, `wg show wg0 transfer`
|
||||
```
|
||||
|
||||
**Use only those three subcommands to diagnose.** Never `wg showconf`, `wg show wg0 private-key` or
|
||||
`wg show wg0 preshared-keys`: they print key material into your terminal and transcript.
|
||||
|
||||
| output | reading |
|
||||
|---|---|
|
||||
| `latest-handshakes`: `<public key> 0` | no handshake with this peer since the pod started |
|
||||
| `endpoints`: `<public key> (none)` | the server has not accepted a packet from this peer since the pod started |
|
||||
| `transfer`: `<public key> <rx> <tx>` | bytes, as the server counts them |
|
||||
|
||||
### The app shows TX, no RX and no "Latest handshake"
|
||||
|
||||
**Symptom.** The app's transmit counter climbs and receive stays at zero. On the server,
|
||||
`endpoints` is `(none)` and `latest-handshakes` is `0` for that peer.
|
||||
|
||||
**Cause.** A key mismatch: the server's public key in the profile, or the device's public key in
|
||||
`peers.conf`. WireGuard drops an initiation it cannot authenticate and sends nothing back, so
|
||||
neither end logs an error. Measured 2026-11-09: a mistyped server public key in the phone's profile.
|
||||
|
||||
**Check.** First prove the packets arrive. The pod has no `conntrack` binary, but in `hostNetwork`
|
||||
it can read the host's table:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- grep dport=51820 /proc/net/nf_conntrack'
|
||||
```
|
||||
|
||||
- A line whose first tuple is `src=<the phone's public address> ... dport=51820`, marked
|
||||
`[UNREPLIED]`: the datagrams reach the node and the server stays silent. It is the keys.
|
||||
- No such line: the packets are not arriving. Look at the router forward, the DNS name, or the
|
||||
device's network.
|
||||
|
||||
Then compare the keys **by copy and paste, never by eye and never retyped**: the server's public
|
||||
half (the first command in Step 2's "Record it in your PASSWORD MANAGER") against the profile's
|
||||
`PublicKey`, and the device's public key from the app against its stanza in `peers.conf`.
|
||||
|
||||
### Works on Wi-Fi, stalls on mobile data, handshake healthy
|
||||
|
||||
**Symptom.** `latest-handshakes` is recent and some traffic flows, but names or pages fail on
|
||||
mobile data. The same device is fine on Wi-Fi.
|
||||
|
||||
**Cause.** Most likely a client MTU too large for the carrier's path: a tunnel MTU of `1420` means
|
||||
outer packets of up to 1480 bytes. This is inferred from one before-and-after, on one phone and one
|
||||
carrier (2026-11-09): `1420` failed and clearing the field fixed it. The path MTU itself was not
|
||||
measured.
|
||||
|
||||
**Check.** In the Android WireGuard app, edit the tunnel, clear the `MTU` field or set it to `1280`,
|
||||
save, then turn the tunnel off and on. A blank field means `1280` in the app's default userspace
|
||||
backend (`GoBackend`: `getMtu().orElse(1280)`). On other clients set `MTU = 1280` in the profile.
|
||||
|
||||
`new-peer.sh` now generates `MTU = 1280` unless `WG_MTU` overrides it. A profile already on a device
|
||||
does not change, so edit it by hand. If `1280` does not help, it is not the MTU: go back to
|
||||
Step 6's check 3.
|
||||
|
||||
### The `away` profile cannot connect from home
|
||||
|
||||
**Symptom.** On the home network, `away` never handshakes.
|
||||
|
||||
**Cause.** Its Endpoint is `wan.home.keithjackson.dev`, the house's WAN address, and the router has
|
||||
no hairpin NAT. WireGuard cannot use the Cloudflare Tunnel instead.
|
||||
|
||||
**Check.** Turn Wi-Fi off. If `away` handshakes on mobile data, the profile is fine: use the `home`
|
||||
tunnel indoors, as described in
|
||||
[Phone profiles: one tunnel for home, one for away](../platform/vpn-ingress/README.md#phone-profiles-one-tunnel-for-home-one-for-away).
|
||||
|
||||
## Revoking a device
|
||||
|
||||
@@ -293,10 +372,17 @@ top follow-up for this component.** Until then:
|
||||
|
||||
## Known limitations
|
||||
|
||||
* **No IPv6 in the tunnel.** A full-tunnel client with native IPv6 still reaches IPv6 sites outside
|
||||
the tunnel. Functionally harmless; a privacy leak if hiding the client's network was the point.
|
||||
* **Client IPs are not preserved** on the LAN side: everything arrives SNATed as the node's
|
||||
address. A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
|
||||
preserve them — at the cost of another hand-maintained off-repo route.
|
||||
* **No IPv6 in the tunnel.** What a client's IPv6 does then depends on the client. In the Android
|
||||
app's default `GoBackend`, a config with one peer and a default route skips `allowFamily()`, so
|
||||
Android should block IPv6 while the tunnel is up and apps fall back to IPv4. That is read from
|
||||
upstream source, not observed on a device. A `--split` profile has no default route, so its IPv6
|
||||
stays outside the tunnel. Expect `wg-quick` on a dual-stack Linux host to leave IPv6 outside too
|
||||
(not tested).
|
||||
* **Client IPs are only partly preserved.** LAN hosts and the internet see the node's address
|
||||
(MASQUERADE). A static route for `10.90.0.0/24` → the forwarded node, added on the router, would
|
||||
preserve them there, at the cost of another hand-maintained off-repo route. LoadBalancer VIPs are
|
||||
different: Cilium handles the packet before MASQUERADE, so the source stays `10.90.0.x`, identity
|
||||
`world` (BPF conntrack on prod-1, 2026-11-09, VIPs `.23` and `.24`). Any `loadBalancerSourceRanges`
|
||||
must therefore list the tunnel subnet, as PiHole's do.
|
||||
* **Rollouts briefly drop the forwarded node.** `maxUnavailable: 1`, so one node at a time; clients
|
||||
re-handshake automatically (retry every 5s, rekey every 2 min).
|
||||
@@ -82,13 +82,10 @@ platform must render, and cites the requirement each mapping satisfies. Changing
|
||||
all seven the moment it is set. Per-resource exclusion has exactly the blast radius of the
|
||||
defect. The filter stays available as hardening once those seven are adjudicated — see
|
||||
`platform/externaldns/helmrelease.yaml` and the register in the L0 check.
|
||||
- `public` renders the record but the renderer does **not** yet set a target: the record still
|
||||
resolves to the shared Gateway VIP, i.e. an RFC1918 address, so a `public` service is not in fact
|
||||
publicly reachable. Fixing that is **not** a renderer change on v0.21.0 —
|
||||
`source/gateway.go` reads `external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and
|
||||
a ListenerSet), never from an HTTPRoute, so a per-service target override is not expressible while
|
||||
the public services share the `shared` Gateway. The open decision is recorded in
|
||||
`platform/externaldns/helmrelease.yaml`.
|
||||
- `public` renders the record but the renderer does **not** set a target: external-dns reads
|
||||
`external-dns.alpha.kubernetes.io/target` only from the **Gateway** (and a ListenerSet), never from
|
||||
an HTTPRoute, so the target is set once, on the `shared` Gateway — since 2026-11-08 the Cloudflare
|
||||
Tunnel, proxied (`platform/gateway/gateway.yaml`, `platform/externaldns/README.md`).
|
||||
- Exposure does **not** relax east-west default-deny (docs/09 §"Interaction with default-deny").
|
||||
- **Requirements:** R-DNS-02/03 (docs/09), R-NET-04 (docs/03).
|
||||
|
||||
@@ -323,8 +320,9 @@ R-OBS-01 (the SLI must follow the rendered kind).
|
||||
`public` service**. An `internal` service's route is rendered with ExternalDNS's controller
|
||||
annotation set to `none`, which removes it from ExternalDNS's view entirely (see §`exposure`), so
|
||||
its `<host>` exists **only** in the internal split-horizon answer PiHole serves, never in the
|
||||
public zone. A `public` record's TARGET is still the Gateway VIP, not a routable address — open
|
||||
operator decision, see §`exposure`.
|
||||
public zone. A `public` record's TARGET is the Cloudflare Tunnel (a proxied CNAME, from the
|
||||
shared Gateway's target annotation; `platform/externaldns/README.md`), and a `public` name is
|
||||
not listed in PiHole unless it must stay on the LAN path.
|
||||
- The `host` pattern is enforced to `<label>.home.keithjackson.dev` by the schema.
|
||||
- **Requirements:** **R-TLS-01/02/03** (docs/08), **R-DNS-01/02/03** (docs/09).
|
||||
|
||||
|
||||
Reference in new issue
Block a user