Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 jakman85
|
61ea71cc35 |
No files matched your search
@@ -14,12 +14,10 @@ resources:
|
||||
- nextcloud-db-backup-credentials-prod.sops.yaml
|
||||
- nextcloud-db-backup-credentials.sops.yaml
|
||||
- nextcloud-db.yaml
|
||||
- pihole-bootstrap-secret.sops.yaml
|
||||
- pihole-dns.yaml
|
||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||
- pihole.yaml
|
||||
- plex-claim-secret.sops.yaml
|
||||
- plex-remote.yaml
|
||||
- plex.yaml
|
||||
- qbittorrent-bootstrap-secret.sops.yaml
|
||||
- qbittorrent-oauth2-proxy-credentials.sops.yaml
|
||||
|
||||
@@ -1,72 +0,0 @@
|
||||
# =================================================================================================
|
||||
# PiHole WEB/API ADMIN PASSWORD — the one piece of PiHole state that was NOT declared anywhere.
|
||||
# =================================================================================================
|
||||
# WHY THIS FILE EXISTS. Measured on live prod 2026-10-04, `pihole.toml` marks every non-default
|
||||
# setting with `### CHANGED`, and names the ones that came from the environment `### CHANGED (env)`.
|
||||
# The complete list was:
|
||||
#
|
||||
# dns.upstreams ### CHANGED (env) <- declared, catalog/pihole.yaml
|
||||
# dns.listeningMode ### CHANGED (env) <- declared, catalog/pihole.yaml
|
||||
# dns.rateLimit.count ### CHANGED (env) <- declared, catalog/pihole.yaml
|
||||
# misc.dnsmasq_lines ### CHANGED (env) <- declared, catalog/pihole.yaml
|
||||
# files.macvendor ### CHANGED <- set by the image itself, container path
|
||||
# webserver.api.pwhash ### CHANGED <- SET OUT OF BAND. Declared NOWHERE. This file.
|
||||
#
|
||||
# So exactly ONE operator-set value lived only on the Longhorn PVC: the admin password, as a
|
||||
# one-way BALLOON-SHA256 hash. Nobody could recover it (a hash does not reverse), nothing in git
|
||||
# produced it, and a rebuild from git would have come up with a DIFFERENT password — or none.
|
||||
#
|
||||
# THE REASON THIS BLOCKS HA, which is why it is fixed first. The next step is running more than one
|
||||
# PiHole replica, and a replica with a fresh PVC would come up with its own unrelated password. The
|
||||
# admin UI would then accept or reject the same credential depending on which pod answered. Declaring
|
||||
# it here makes every replica, and every rebuild, identical by construction.
|
||||
#
|
||||
# MECHANISM: `webserver.api.password` is a real, settable FTL config key — verified in the running
|
||||
# image, where `pihole setpassword` is itself just `setFTLConfigValue "webserver.api.password"` with
|
||||
# the comment "pihole-FTL will automatically hash the password". So FTL takes CLEARTEXT on this key
|
||||
# and writes the hash into `webserver.api.pwhash` itself. catalog/pihole.yaml therefore passes this
|
||||
# value as `FTLCONF_webserver_api_password`, and the hash on the PVC becomes a derived artefact
|
||||
# rather than the source of truth.
|
||||
#
|
||||
# DEFENCE IN DEPTH, NOT THE ONLY GATE. The admin UI is already behind the oauth2-proxy sidecar
|
||||
# (`--allowed-group=platform-admins`), so this password is a SECOND gate. It is kept rather than set
|
||||
# empty because the `pihole` Service still publishes port 80 (PiHole direct, bypassing the proxy on
|
||||
# :4180) and only the CiliumNetworkPolicy closes it — and the default-deny policy in this repo has a
|
||||
# documented drift history. An unauthenticated admin UI that can rewrite LAN DNS is not a good thing
|
||||
# to leave resting on one policy staying correct. Narrowing that Service port is tracked follow-up.
|
||||
#
|
||||
# ROTATION: `sops` this file in your own terminal, change the value, commit. Flux re-applies the
|
||||
# Secret, the pod restarts, FTL re-hashes. Do NOT `sops -d` it into a shared terminal or a log
|
||||
# (CLAUDE.md section 6); read it in an editor. The value below was generated randomly at authoring
|
||||
# time and has never been printed to a transcript.
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: pihole-bootstrap
|
||||
namespace: pihole
|
||||
labels:
|
||||
app.kubernetes.io/name: pihole
|
||||
type: Opaque
|
||||
stringData:
|
||||
#ENC[AES256_GCM,data:TMOuRyJeb5IVn3lZPXY7lgh1KGxu7GY9+WIOLHpMB50JqzmM2tHIaWUWKTt1JtZoX/E5/Y7WCUTnfkJLhRBN2zV2MwWRBSLYVy5CMg7xV8PTLRZJYAVUYS5Trg==,iv:q+DpfT6PQfY5ZuNsmP7Ddl+PDLjSm89fJICAhYoW64k=,tag:axijaO3DaxIZgZ3LeK2hXA==,type:comment]
|
||||
webui-password: ENC[AES256_GCM,data:7FcK/bUW79M46f/MlouAmATKbMnoUTNuMUjtqJ5vpU4=,iv:I8DBFB39uXZV42Ie2IPwSefmMgdKEtiDknfc9Tu7h5E=,tag:L423784UrDU3TvylpHc6KQ==,type:str]
|
||||
sops:
|
||||
kms: []
|
||||
gcp_kms: []
|
||||
azure_kv: []
|
||||
hc_vault: []
|
||||
age:
|
||||
- recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
|
||||
enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0MkRSNjZLZlpXYkcvaVJL
|
||||
cjlqb2wrcWREZnZLdGpsTHNPWEpseHlRZnpjCnJVM1FDQ2M2dXlaS1JBZ2k2V1BP
|
||||
Yi9QUDZSMzZsL09pTi93RTdjdGxtY2MKLS0tIDIrQTZIVWdhMGIxSCtlUFN2Mlho
|
||||
SWVybld4VjJyaHRSKzZZU0xsUUV2SEEKbDYHV/qBZVspK7DpBW3vYjRMJ/HmAN+9
|
||||
G9bOlNEp0jAtcNZEn6lpQ9ORgvmIrhJHoBQRLabdMWzNWSZsNn56tA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
lastmodified: "2026-10-04T06:01:27Z"
|
||||
mac: ENC[AES256_GCM,data:rhTMTOwLOtHXHgqI5M2ZW/6w+UpLpxBlTk7NoN0wmGSqOcE+OGrXA1DhkZrJweyQT0RSNSSBm/6QMztLP81eepOzdp2rEkwvbQPK/l6K9g16Ww9c1IatS3NQsKEB8AoMDx7jBQ6exXYgggPx3MJsNYbE+WyloQ96U9KPdrecAM8=,iv:n34v2zX9D5xsaR1PFCH94mNMiuRfhUyI2lseUCoqiTA=,tag:9YmN9nIVLXfsGYIb0bUl2w==,type:str]
|
||||
pgp: []
|
||||
encrypted_regex: ^(data|stringData)$
|
||||
version: 3.9.4
|
||||
@@ -90,21 +90,9 @@ spec:
|
||||
value: '0'
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
|
||||
- name: FTLCONF_webserver_api_password
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: pihole-bootstrap
|
||||
key: webui-password
|
||||
volumeMounts:
|
||||
- name: config
|
||||
mountPath: /data/config
|
||||
lifecycle:
|
||||
postStart:
|
||||
exec:
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n [ -n \"$url\" ] || continue\n printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n [ -n \"$url\" ] || continue\n # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n have=$(sql \"select count(*) from adlist where address='$url';\")\n if [ \"$have\" = \"0\" ]; then\n echo \"pihole-adlists: adding $url\"\n sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n CHANGED=1\n else\n en=$(sql \"select enabled from adlist where address='$url';\")\n if [ \"$en\" != \"1\" ]; then\n echo \"pihole-adlists: re-enabling $url\"\n sql \"update adlist set enabled=1 where address='$url';\"\n CHANGED=1\n fi\n fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n [ -n \"$addr\" ] || continue\n if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n echo \"pihole-adlists: disabling undeclared $addr\"\n sql \"update adlist set enabled=0 where address='$addr';\"\n echo changed >> /tmp/adlist-changed\n fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n echo \"pihole-adlists: declared set changed - running gravity update\"\n pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
|
||||
- name: oauth2-proxy
|
||||
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
|
||||
@@ -1,126 +0,0 @@
|
||||
---
|
||||
# =================================================================================================
|
||||
# PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
|
||||
# =================================================================================================
|
||||
# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
|
||||
# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
|
||||
# mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
|
||||
# schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
|
||||
# type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
|
||||
# `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
|
||||
#
|
||||
# WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
|
||||
# Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
|
||||
# returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
|
||||
# (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
|
||||
#
|
||||
# The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
|
||||
# `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
|
||||
# list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
|
||||
# publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
|
||||
# 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
|
||||
# `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
|
||||
# publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
|
||||
# file) over advertising the Gateway URL: Plex then terminates its own TLS with its
|
||||
# `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
|
||||
#
|
||||
# VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
|
||||
# lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
|
||||
# (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
|
||||
# CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
|
||||
# this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
|
||||
# port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
|
||||
# 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
|
||||
# IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
|
||||
# .22 (crafty-l4-*) and .23 (pihole-dns-*).
|
||||
#
|
||||
# ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
|
||||
# lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
|
||||
# DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
|
||||
#
|
||||
# NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
|
||||
# its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
|
||||
# resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
|
||||
# clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
|
||||
# to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
|
||||
# without a token), which is what makes an unfiltered port defensible here.
|
||||
#
|
||||
# NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
|
||||
# `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
|
||||
# Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
|
||||
# address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
|
||||
# from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
|
||||
# HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
|
||||
# through the Gateway exactly as before.
|
||||
#
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
|
||||
# -------------------------------------------------------------------------------------------------
|
||||
# `Local` is the tempting choice because it preserves the client source IP, which this service
|
||||
# actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
|
||||
# reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
|
||||
# holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
|
||||
# by a node that has no Plex pod, which black-holes the port while every object still reports
|
||||
# healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
|
||||
# scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
|
||||
#
|
||||
# KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
|
||||
# to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
|
||||
# catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
|
||||
# Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
|
||||
# `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
|
||||
# authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
|
||||
# on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
|
||||
# without also de-classifying genuine LAN clients, so it is accepted.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: plex-l4-tcp
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
annotations:
|
||||
lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
externalTrafficPolicy: Cluster
|
||||
selector:
|
||||
app.kubernetes.io/name: plex
|
||||
ports:
|
||||
- name: plex-remote
|
||||
port: 32400
|
||||
targetPort: 32400
|
||||
protocol: TCP
|
||||
---
|
||||
# INGRESS allow for the raw port (R-NET-01/02).
|
||||
#
|
||||
# The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
|
||||
# 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
|
||||
# this file adds the off-cluster case without touching those.
|
||||
#
|
||||
# `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
|
||||
# `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
|
||||
# this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
|
||||
# NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
|
||||
# `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
|
||||
# four entities are named.
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: plex-allow-remote-32400
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: plex
|
||||
ingress:
|
||||
- fromEntities:
|
||||
- world
|
||||
- cluster
|
||||
- host
|
||||
- remote-node
|
||||
toPorts:
|
||||
- ports:
|
||||
- { port: "32400", protocol: TCP }
|
||||
@@ -100,20 +100,6 @@ env:
|
||||
# while resolution for the zone does not work. Confirmed both ways against the live image.
|
||||
- name: FTLCONF_misc_dnsmasq_lines
|
||||
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
|
||||
# THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
|
||||
# apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
|
||||
# carries the full rationale and the measurement that found it.
|
||||
#
|
||||
# `webserver.api.password` takes CLEARTEXT and FTL hashes it itself into `webserver.api.pwhash`.
|
||||
# Verified in the running image, not assumed: `pihole setpassword` is itself
|
||||
# `setFTLConfigValue "webserver.api.password" "$PASSWORD"`, commented upstream as "pihole-FTL will
|
||||
# automatically hash the password". So the BALLOON-SHA256 hash on the PVC is now a DERIVED artefact
|
||||
# and this is its source of truth. Rotating the Secret rotates the password.
|
||||
- name: FTLCONF_webserver_api_password
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: pihole-bootstrap
|
||||
key: webui-password
|
||||
ingressRoutes:
|
||||
- host: pihole.${BASE_DOMAIN}
|
||||
port: 80 # admin web UI, protected at the Gateway by forward-auth (docs/09)
|
||||
@@ -128,110 +114,6 @@ auth:
|
||||
# rather than a renderer constant: the measurement that justifies the flag on two apps is
|
||||
# evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
|
||||
# the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
|
||||
lifecycle:
|
||||
# LEVEL-TRIGGERED ADLIST ASSERTION, run on EVERY container start. Same pattern, and the same
|
||||
# reasoning, as catalog/qbittorrent.yaml's postStart: desired state lives in git and is
|
||||
# RE-ASSERTED unconditionally rather than set once by hand.
|
||||
#
|
||||
# WHY THIS IS NEEDED AT ALL — the blocklists were NOT reproducible from git. Measured on live prod
|
||||
# 2026-10-04: `gravity.db` held exactly ONE adlist (StevenBlack's hosts, enabled) and 72,234
|
||||
# gravity domains, and NOTHING in this repo named it. Worse, a FRESH database does not seed it:
|
||||
# the image's own schema template `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the
|
||||
# `adlist` table and inserts NO rows into it (grepped in the running image — there is no
|
||||
# `INSERT INTO adlist` at all). So a rebuild from git, or a replica with a new PVC, would come up
|
||||
# SERVING DNS AND BLOCKING NOTHING, silently and with every object reporting healthy.
|
||||
#
|
||||
# WHY A postStart HOOK AND NOT A FILE. PiHole v6 keeps adlists as DB rows, not config. The
|
||||
# `/etc/pihole/adlists.list` file still exists in `gravity.sh` but is MIGRATION-ONLY: its import
|
||||
# sits inside `migrate_to_database()`, guarded by `if [ ! -e "$gravityDBfile" ]`, so it fires only
|
||||
# while CREATING a database and never again on an existing one. A ConfigMap mounted there would
|
||||
# therefore be silently ignored on every pod this cluster currently runs. There is also no
|
||||
# `pihole adlist` CLI subcommand (only allow/deny for domains), so SQL against the DB is the
|
||||
# actual interface. FTL ships an embedded shell (`pihole-FTL sqlite3`, SQLite 3.53.1) so no extra
|
||||
# tooling is needed in the image.
|
||||
#
|
||||
# IT RECONCILES, IT DOES NOT JUST ADD: adlists present in the DB but absent from DECLARED below are
|
||||
# DISABLED (not deleted — deleting cascades to `gravity` rows and would force a full re-download of
|
||||
# everything on a typo). So this list is the source of truth in both directions, and `pihole -g`
|
||||
# runs ONLY when something actually changed, which keeps the steady-state start free of a 70k-domain
|
||||
# rebuild.
|
||||
#
|
||||
# NO BRACED DOLLAR EXPANSION ANYWHERE IN THIS SCRIPT, DELIBERATELY — the same contract
|
||||
# catalog/qbittorrent.yaml documents at length. apps-wave-0 declares postBuild.substituteFrom
|
||||
# cluster-vars, so Flux runs envsubst over this whole manifest at reconcile time; a `${VAR}` form
|
||||
# would behave differently on the cluster than under a plain `kubectl apply` of the golden. Bare
|
||||
# `$VAR` and `$(cmd)` are untouched by envsubst and so behave identically however this is run.
|
||||
# `render.py`'s assert_substitution_safe refuses to emit a manifest that breaks this.
|
||||
postStart:
|
||||
exec:
|
||||
command:
|
||||
- /bin/bash
|
||||
- -c
|
||||
- |
|
||||
# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and
|
||||
# does NOT run a gravity rebuild.
|
||||
set -u
|
||||
GRAVITY=/etc/pihole/gravity.db
|
||||
|
||||
# THE DECLARED SET — one URL per line, the source of truth for what PiHole blocks.
|
||||
# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.
|
||||
DECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
|
||||
|
||||
# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,
|
||||
# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.
|
||||
i=0
|
||||
while [ ! -s "$GRAVITY" ] && [ "$i" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done
|
||||
if [ ! -s "$GRAVITY" ]; then
|
||||
echo "pihole-adlists: gravity.db absent after 180s - leaving adlists untouched" >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
sql() { pihole-FTL sqlite3 "$GRAVITY" "$1"; }
|
||||
CHANGED=0
|
||||
|
||||
# 1) Every declared URL must be present AND enabled.
|
||||
echo "$DECLARED" | while IFS= read -r url; do
|
||||
[ -n "$url" ] || continue
|
||||
printf '%s\n' "$url"
|
||||
done > /tmp/declared.txt
|
||||
while IFS= read -r url; do
|
||||
[ -n "$url" ] || continue
|
||||
# Single quotes inside a URL would break this SQL; adlist URLs do not contain them.
|
||||
have=$(sql "select count(*) from adlist where address='$url';")
|
||||
if [ "$have" = "0" ]; then
|
||||
echo "pihole-adlists: adding $url"
|
||||
sql "insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');"
|
||||
CHANGED=1
|
||||
else
|
||||
en=$(sql "select enabled from adlist where address='$url';")
|
||||
if [ "$en" != "1" ]; then
|
||||
echo "pihole-adlists: re-enabling $url"
|
||||
sql "update adlist set enabled=1 where address='$url';"
|
||||
CHANGED=1
|
||||
fi
|
||||
fi
|
||||
done < /tmp/declared.txt
|
||||
|
||||
# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).
|
||||
sql "select address from adlist where enabled=1;" | while IFS= read -r addr; do
|
||||
[ -n "$addr" ] || continue
|
||||
if ! grep -qxF "$addr" /tmp/declared.txt; then
|
||||
echo "pihole-adlists: disabling undeclared $addr"
|
||||
sql "update adlist set enabled=0 where address='$addr';"
|
||||
echo changed >> /tmp/adlist-changed
|
||||
fi
|
||||
done
|
||||
[ -s /tmp/adlist-changed ] && CHANGED=1
|
||||
rm -f /tmp/adlist-changed /tmp/declared.txt
|
||||
|
||||
# 3) Rebuild gravity ONLY if the declared set actually moved.
|
||||
if [ "$CHANGED" = "1" ]; then
|
||||
echo "pihole-adlists: declared set changed - running gravity update"
|
||||
pihole -g || echo "pihole-adlists: gravity update failed; existing gravity.db retained" >&2
|
||||
else
|
||||
echo "pihole-adlists: already matches the declared set - nothing to do"
|
||||
fi
|
||||
exit 0
|
||||
update:
|
||||
channel: stable # R-UPD-01
|
||||
range: ">=2024.0.0"
|
||||
|
||||
@@ -2002,24 +2002,6 @@ EXTRA_UNITS: dict = {
|
||||
"pihole-dns.yaml",
|
||||
],
|
||||
},
|
||||
# Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
|
||||
# CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
|
||||
# `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
|
||||
# own policy, applied in the same wave as the Deployment they select, exactly as the rendered
|
||||
# Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
|
||||
#
|
||||
# It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
|
||||
# express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
|
||||
# `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
|
||||
# Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
|
||||
# outside it is the exposure the schema has no word for.
|
||||
"plex-remote": {
|
||||
"anchor": "plex",
|
||||
"above": False,
|
||||
"files": [
|
||||
"plex-remote.yaml",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -100,8 +100,7 @@ data:
|
||||
# platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
|
||||
# tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
|
||||
# S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
|
||||
# GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
|
||||
# plex-remote: PLEX_L4_VIP),
|
||||
# GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
|
||||
# velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
|
||||
# VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
|
||||
# UNSUBSTITUTED path would ship the literal string into the cluster.
|
||||
@@ -176,19 +175,6 @@ data:
|
||||
# dev -> 192.168.1.56 (base default; next free after crafty .55 in dev's .53-.60 pool)
|
||||
# prod -> 192.168.2.23 (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
|
||||
PIHOLE_DNS_VIP: "192.168.1.56" # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
|
||||
# Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
|
||||
# A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
|
||||
# cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
|
||||
# server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
|
||||
# measured working) cannot put the server on a client's connection list at all.
|
||||
#
|
||||
# PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
|
||||
# address by hand. A VIP taken from the pool head would silently move on a reschedule and break
|
||||
# remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
|
||||
#
|
||||
# dev -> 192.168.1.57 (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
|
||||
# prod -> 192.168.2.24 (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
|
||||
PLEX_L4_VIP: "192.168.1.57" # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
|
||||
|
||||
# The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
|
||||
# peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
|
||||
|
||||
@@ -476,26 +476,6 @@ patches:
|
||||
- op: replace
|
||||
path: /data/PIHOLE_DNS_VIP
|
||||
value: "192.168.2.23"
|
||||
# Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
|
||||
# (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
|
||||
# 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
|
||||
# reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
|
||||
# (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
|
||||
# (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
|
||||
# the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
|
||||
# control-plane VIP (.9).
|
||||
#
|
||||
# UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
|
||||
# here, by hand, off-repo — so changing this value is a two-place edit and the router is the
|
||||
# half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
|
||||
#
|
||||
# NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
|
||||
# for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
|
||||
# this address would not have been caught. Probe before trusting it if remote access comes up
|
||||
# with a conflict.
|
||||
- op: replace
|
||||
path: /data/PLEX_L4_VIP
|
||||
value: "192.168.2.24"
|
||||
# The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
|
||||
# NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
|
||||
# It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
|
||||
|
||||
@@ -1,45 +0,0 @@
|
||||
---
|
||||
date: 2026-10-03
|
||||
branch: fix/node-keycloak-hosts-pin
|
||||
title: Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Follow-up to PR #40 (`decisions/2026-10-03-public-dns-wan-target.md`). Public DNS for
|
||||
`*.home.keithjackson.dev` now CNAMEs to the WAN IP. The prod nodes resolve through public DNS,
|
||||
and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer
|
||||
`keycloak.home.keithjackson.dev`.
|
||||
|
||||
- `roles/node_baseline`: a new `node_baseline_hosts_pins` list (default `[]`) and one
|
||||
`blockinfile` task (tags `dns`, `hosts`). It writes the pins into `/etc/hosts` as a single
|
||||
marked block. An empty list removes the block.
|
||||
- `group_vars/k3s_prod/vars.yml`: pins `keycloak.{{ k3s_base_domain }}` → `192.168.2.20`, which
|
||||
must stay equal to `GATEWAY_VIP` (parity comment added). It also replaces the stale "no
|
||||
/etc/hosts pin is needed" paragraph with the measured reason the pin is now required.
|
||||
|
||||
Dev is unaffected (empty default).
|
||||
|
||||
## What I verified
|
||||
|
||||
- After PR #40 reconciled (`platform` lastAppliedRevision `576e1ab`, Healthy=True), on
|
||||
k3s-prod-2 and k3s-prod-3: `getent hosts keycloak.home.keithjackson.dev` →
|
||||
`136.51.115.252 wan.home.keithjackson.dev`. `curl https://keycloak…/realms/homelab/.well-known/openid-configuration`
|
||||
→ exit 60, `ssl_verify_result=18` (the router's self-signed cert, not the Gateway).
|
||||
- `journalctl -u k3s` (last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds
|
||||
the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation.
|
||||
- Neither node has cloud-init (`/etc/cloud` absent), so nothing rewrites `/etc/hosts` at boot.
|
||||
- `--check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3`: each node gets exactly the
|
||||
three-line marked block `192.168.2.20 keycloak.home.keithjackson.dev`.
|
||||
- Outside-in, PR #40 works: check-host.net probes from 3 external nodes got `200 OK` from
|
||||
`https://plex.home.keithjackson.dev/identity` at `136.51.115.252`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **k3s-prod-1**: Ansible cannot reach it. SSH host key verification fails (`known_hosts:94`,
|
||||
ED25519 `SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE`), which is consistent with
|
||||
the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint
|
||||
before the pin can be applied there.
|
||||
- The playbook has not been applied for real. It is an operator run after merge:
|
||||
`ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`.
|
||||
- An end-to-end kubectl OIDC login after the pin (needs a browser).
|
||||
@@ -1,129 +0,0 @@
|
||||
---
|
||||
date: 2026-10-03
|
||||
branch: feat/plex-remote-access-l4
|
||||
title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
|
||||
rather than inferred:
|
||||
|
||||
1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
|
||||
`customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
|
||||
`PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
|
||||
Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
|
||||
2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
|
||||
`136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
|
||||
Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
|
||||
any route.
|
||||
|
||||
This branch fixes the in-cluster half. **Remote access does not work until the operator also does
|
||||
two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
|
||||
"manually specify public port" in the Plex UI.
|
||||
|
||||
- **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
|
||||
pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
|
||||
(`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
|
||||
because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
|
||||
for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
|
||||
`platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
|
||||
- **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
|
||||
so `--emit-waves` places the unit in plex's own wave.
|
||||
- **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
|
||||
substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
|
||||
- **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
|
||||
- **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
|
||||
- **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
|
||||
no alert is possible, and the two documented warts below.
|
||||
|
||||
**`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
|
||||
was the alternative — it needs no new WAN hole and the mechanism was verified working on an
|
||||
already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
|
||||
TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
|
||||
`ADVERTISE_IP` remains accurate and untouched.
|
||||
|
||||
**`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
|
||||
Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
|
||||
Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
|
||||
puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
|
||||
as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.
|
||||
|
||||
**SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
|
||||
it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
|
||||
"Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
|
||||
PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
|
||||
|
||||
## What I verified
|
||||
|
||||
All measurements against live prod, 2026-10-03, read-only.
|
||||
|
||||
- **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
|
||||
`ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
|
||||
`PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
|
||||
attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
|
||||
- **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
|
||||
`GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
|
||||
`machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
|
||||
- **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
|
||||
`/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
|
||||
Gateway.
|
||||
- **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
|
||||
logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
|
||||
allowed network.
|
||||
- **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
|
||||
cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
|
||||
`plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
|
||||
- **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
|
||||
IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
|
||||
(unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
|
||||
the kube-vip control-plane VIP (`.9`).
|
||||
- **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
|
||||
init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
|
||||
`PLEX_PREFERENCE` → **NONE**.
|
||||
- **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
|
||||
"why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
|
||||
**outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
|
||||
marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
|
||||
- **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
|
||||
`spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
|
||||
**pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
|
||||
and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
|
||||
- **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
|
||||
hand-written unit and the edited wave kustomization are mutually consistent.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
|
||||
be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
|
||||
branch removes two blockers and adds none; it does not demonstrate an outcome.
|
||||
- **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
|
||||
`:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
|
||||
prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
|
||||
forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
|
||||
(app.plex.tv from a cellular connection) is the only honest check.
|
||||
- **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
|
||||
`.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
|
||||
not have been caught.
|
||||
- **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
|
||||
LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
|
||||
the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
|
||||
`remote-node` is the correct entity set **under this SNAT path** is reasoned from
|
||||
`platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
|
||||
- **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
|
||||
no remote connection after both manual steps, this branch is necessary-but-insufficient.
|
||||
- **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
|
||||
a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
|
||||
- **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
|
||||
documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
|
||||
add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
|
||||
intentionally not onboarded in `ci/render-golden.sh`
|
||||
(`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
|
||||
deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
|
||||
any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
|
||||
`--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
|
||||
which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
|
||||
`render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
|
||||
VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
|
||||
`--emit-waves`, and I have not determined how long the drift has existed or whether either service
|
||||
is *meant* to be onboarded.
|
||||
@@ -1,73 +0,0 @@
|
||||
---
|
||||
date: 2026-10-03
|
||||
branch: feat/public-dns-wan-target
|
||||
title: Public DNS now points at the WAN — Gateway-published names CNAME to an in-cluster DDNS record
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator closed the open "public-target gap" recorded in
|
||||
`platform/externaldns/helmrelease.yaml`. The WAN NAT / port-forwarding (80/443 → Gateway VIP
|
||||
192.168.2.20) was done by the operator outside this repo.
|
||||
|
||||
Operator decisions (2026-10-03):
|
||||
|
||||
1. **Scope: every Gateway-published name goes public.** That covers the five `exposure: public`
|
||||
apps (plex, gitea, keycloak, nextcloud, seerr) and also the hand-authored admin UIs (auth,
|
||||
crafty, grafana, headlamp, the `home.keithjackson.dev` apex). `internal` services still publish
|
||||
nothing: their `controller: none` route annotation is checked before the target is read.
|
||||
2. **The WAN IP is kept current in-cluster**, not hardcoded.
|
||||
|
||||
Implementation:
|
||||
|
||||
- `platform/gateway/gateway.yaml`: `external-dns.alpha.kubernetes.io/target: wan.${BASE_DOMAIN}` on
|
||||
the shared Gateway. Every gateway-sourced record changes from `A 192.168.2.20` to
|
||||
`CNAME wan.home.keithjackson.dev`.
|
||||
- `platform/externaldns/ddns.yaml` (new): `favonia/cloudflare-ddns:1.17.1` Deployment in the
|
||||
`external-dns` namespace. It reuses `cloudflare-api-token` and keeps `wan.${BASE_DOMAIN}` an
|
||||
unproxied IPv4 A record at the WAN IP (detected via Cloudflare trace, checked every 5 min,
|
||||
`DELETE_ON_STOP=false`). external-dns has no TXT registry entry for it, so it never owns or
|
||||
deletes it.
|
||||
- The helmrelease comment block and the README now describe the decision instead of the gap.
|
||||
|
||||
Known consequence: **k3s nodes resolve through public DNS** (`node_baseline_dns_servers` = 8.8.8.8 /
|
||||
1.1.1.1). The kube-apiserver's OIDC issuer `keycloak.home.keithjackson.dev` therefore resolves to the
|
||||
WAN IP from the nodes, and kubectl SSO works only if the router hairpins WAN:443 back to the Gateway.
|
||||
A LAN-side probe of `136.51.115.252:443` on 2026-10-03 returned a **self-signed placeholder cert**,
|
||||
not the Gateway's Let's Encrypt wildcard. That suggests no hairpin. The cert-based admin kubeconfig
|
||||
is unaffected.
|
||||
|
||||
## What I verified
|
||||
|
||||
- Live prod, 2026-10-03, before the change: Gateway `gateway/shared` ADDRESS `192.168.2.20`.
|
||||
External-dns args include `--source=gateway-httproute --source=service --policy=sync
|
||||
--txt-owner-id=prod --domain-filter=keithjackson.dev`, and it logs "All records are already up to
|
||||
date".
|
||||
- Public answers at 1.1.1.1: plex/gitea/keycloak/grafana/auth/apex → `192.168.2.20`;
|
||||
`crafty-mc` → `.22` and `unifi` → `.21` (service source, not affected by this change); jackett →
|
||||
none (internal, correctly unpublished).
|
||||
- The authority (betty.ns.cloudflare.com) has no unprefixed TXT, AAAA or MX at `plex.` or the apex,
|
||||
so the A→CNAME switch has no co-existing record to collide with.
|
||||
- WAN IP from the workstation: `136.51.115.252`.
|
||||
- No CiliumNetworkPolicy or NetworkPolicy in `external-dns`. The Kyverno label and default-deny
|
||||
policies match only `app-namespace` namespaces. The vpn egress policy selects only
|
||||
`egress-class: vpn` pods.
|
||||
- Toolbox gate result: see the PR description (run after this commit).
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The outside-in path.** I have no vantage point outside the LAN. Whether
|
||||
`https://<svc>.home.keithjackson.dev` from the internet reaches the Gateway with the LE cert is
|
||||
UNVERIFIED.
|
||||
- **Hairpin / kubectl SSO after the flip.** Likely broken per the probe above. Fix candidates: a
|
||||
node `/etc/hosts` pin for `keycloak.${BASE_DOMAIN}` → Gateway VIP via `node_baseline`, or router
|
||||
NAT reflection.
|
||||
- **The record-type flip itself**, until Flux reconciles: whether Cloudflare accepts external-dns's
|
||||
create-CNAME in the same batch as the delete-A, or converges one interval later.
|
||||
- `favonia/cloudflare-ddns` under this pod securityContext. The README warns that
|
||||
`no-new-privileges` misbehaves on some Proxmox-style hosts. These are bare-metal nodes, but it is
|
||||
untested until the pod runs.
|
||||
- Whether the L4 service-source names (crafty-mc, unifi) should also go public. Not decided, not
|
||||
changed.
|
||||
- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh` header lines 39–40
|
||||
still describe the target as open. That is a sensitive path, so it is left for a follow-up.
|
||||
@@ -1,133 +0,0 @@
|
||||
---
|
||||
date: 2026-10-04
|
||||
branch: feat/pihole-declare-state
|
||||
title: PiHole's operator-set state is now declared in git — the prerequisite for running more than one replica
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Phase 1 of making LAN DNS survive a node failure. The operator asked for multi-replica PiHole;
|
||||
measurement showed that cannot be done safely first, because PiHole's important state was not
|
||||
reproducible from git. A second replica with a fresh PVC would have come up **serving DNS and
|
||||
blocking nothing, with a different admin password** — answers varying by which pod replied. This
|
||||
branch closes that; the replica work is phase 2 and is deliberately NOT here.
|
||||
|
||||
**The undeclared-state inventory, measured not guessed.** `pihole.toml` marks every non-default
|
||||
setting `### CHANGED`, and those that came from the environment `### CHANGED (env)`. The complete
|
||||
live list on 2026-10-04:
|
||||
|
||||
| Setting | Status |
|
||||
|---|---|
|
||||
| `dns.upstreams`, `dns.listeningMode`, `dns.rateLimit.count`, `misc.dnsmasq_lines` | `### CHANGED (env)` — already declared in `catalog/pihole.yaml` |
|
||||
| `files.macvendor` | `### CHANGED` — set by the image itself (container path), not operator state |
|
||||
| `webserver.api.pwhash` | **`### CHANGED` — set out of band. Declared NOWHERE.** |
|
||||
|
||||
So exactly **one** operator-set value lived only on the PVC. Separately, the **adlists** were also
|
||||
undeclared: `gravity.db` held one list (StevenBlack's hosts) and 72,234 gravity domains, with nothing
|
||||
in this repo naming it.
|
||||
|
||||
- **`apps/wave-0/pihole-bootstrap-secret.sops.yaml` (new, SOPS/age).** Secret `pihole-bootstrap`,
|
||||
key `webui-password`. Named to the `<svc>-bootstrap-secret.sops.yaml` convention so
|
||||
`render.py --emit-waves` places it with no renderer change. The value is a 32-char random password
|
||||
generated at authoring time and **never printed to a transcript**.
|
||||
- **`catalog/pihole.yaml`:** `FTLCONF_webserver_api_password` sourced from that Secret.
|
||||
`webserver.api.password` takes CLEARTEXT and FTL hashes it itself — verified in the running image,
|
||||
where `pihole setpassword` *is* `setFTLConfigValue "webserver.api.password"`, commented upstream
|
||||
"pihole-FTL will automatically hash the password". The PVC's `pwhash` becomes a derived artefact.
|
||||
- **`catalog/pihole.yaml`:** a `lifecycle.postStart` hook that **reconciles** the declared adlist set
|
||||
into `gravity.db` on every container start — adds missing, re-enables disabled, and **disables**
|
||||
(never deletes) anything enabled but undeclared, running `pihole -g` only when something moved.
|
||||
Same level-triggered idiom, and the same reasoning, as `catalog/qbittorrent.yaml`'s postStart.
|
||||
- **`apps/wave-0/pihole.yaml`, `apps/wave-0/kustomization.yaml`:** re-rendered / secret registered.
|
||||
|
||||
**Why a postStart hook and not a file.** PiHole v6 keeps adlists as DB rows. `/etc/pihole/adlists.list`
|
||||
still exists in `gravity.sh` but is **migration-only**: its import sits inside `migrate_to_database()`
|
||||
behind `if [ ! -e "$gravityDBfile" ]`, so it fires only while *creating* a database. A ConfigMap
|
||||
mounted there would be silently ignored on every pod this cluster runs today. There is also no
|
||||
`pihole adlist` CLI subcommand, so SQL is the actual interface; FTL ships an embedded shell
|
||||
(`pihole-FTL sqlite3`) so nothing had to be added to the image.
|
||||
|
||||
**The password is kept rather than set empty, deliberately.** The admin UI is already behind the
|
||||
oauth2-proxy sidecar (`--allowed-group=platform-admins`), so an empty password would have been
|
||||
simpler. But the `pihole` Service still publishes **port 80** (PiHole direct, bypassing the proxy on
|
||||
`:4180`) and only the CiliumNetworkPolicy closes it — and this repo has a documented default-deny
|
||||
drift history. An unauthenticated admin UI that can rewrite LAN DNS should not rest on one policy
|
||||
staying correct. Narrowing that Service port is follow-up, not done here.
|
||||
|
||||
**SENSITIVE PATHS — this PR adds a `**/*.sops.yaml`, so `ci/approval-policy.sh` will BLOCK it.** It
|
||||
needs the operator's `/approve-override <reason>` as a PR **review** of type "Comment". Not posted by
|
||||
me: CLAUDE.md section 1 forbids an agent clearing this on its own PR.
|
||||
|
||||
## What I verified
|
||||
|
||||
Live prod, 2026-10-04, read-only except where noted.
|
||||
|
||||
- **The undeclared-state inventory above** — read from `pihole.toml`'s own `### CHANGED` markers, with
|
||||
the password value masked. This is the measurement the whole branch rests on.
|
||||
- **A fresh `gravity.db` seeds NO adlists.** The image's own schema template
|
||||
`/etc/.pihole/advanced/Templates/gravity.db.sql` creates the `adlist` table and contains no
|
||||
`INSERT INTO adlist` at all — grepped in the running image. Confirmed empirically below (0 rows).
|
||||
This is why the adlist declaration is load-bearing and not cosmetic.
|
||||
- **The adlists.list path really is migration-only** — read the guard in `/opt/pihole/gravity.sh`
|
||||
(`migrate_to_database()`, `if [ ! -e "$gravityDBfile" ]`).
|
||||
- **`webserver.api.password` is a real settable key** — `pihole-FTL --config webserver.api.password`
|
||||
returns a masked value, and `/usr/local/bin/pihole`'s `setpassword` sets exactly that key.
|
||||
- **THE RENDERED HOOK ACTUALLY WORKS — all four paths, extracted from `apps/wave-0/pihole.yaml` and
|
||||
run in the live pod against THROWAWAY COPIES of `gravity.db` in `/tmp`** (production `gravity.db`
|
||||
was never written, and `pihole -g` was stubbed so no 70k-domain download could start):
|
||||
1. *Steady state* → `already matches the declared set - nothing to do`. No writes, no rebuild.
|
||||
2. *True fresh DB built from the image's schema template* → 0 adlists before; hook added the
|
||||
declared list and triggered the rebuild. **This is the fresh-PVC/new-replica case.**
|
||||
3. *An undeclared list enabled* → disabled it (`enabled=0`), left the declared one alone, and did
|
||||
not delete the row.
|
||||
4. *Declared list present but disabled* → re-enabled it.
|
||||
- **Required tools exist in the image:** `bash`, `expr`, `grep`, `printf`, `sleep`, `pihole-FTL`.
|
||||
- **The Secret is genuinely encrypted** to the repo's age recipient; `stringData.webui-password`
|
||||
starts `ENC[`. The gate's `no-plaintext-secrets (R-SEC-01)` L0 check passes.
|
||||
- **Schema:** `check-jsonschema` against `schemas/servicedefinition.schema.json` passes.
|
||||
- **Render determinism:** the canonical
|
||||
`render.py --all catalog -o apps --wave-layout --no-image-automation` changed **only**
|
||||
`apps/wave-0/pihole.yaml`. `render-golden` passes, so the hook's text also satisfies
|
||||
`assert_substitution_safe` (no braced dollar expansion — the Flux envsubst contract).
|
||||
- **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
|
||||
`render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two usual environmental
|
||||
skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3
|
||||
(`CLUSTER_AVAILABLE` unset). **Not green; reported as rc 3.**
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Nothing here has been reconciled.** Flux has not applied any of it, so the password has never
|
||||
been set by this mechanism and the hook has never run during a real container start. Everything
|
||||
above is the rendered artifact tested out-of-band, plus the image's own source.
|
||||
- **THE CURRENT ADMIN PASSWORD WILL STOP WORKING, and that is the intent.** FTL re-hashes
|
||||
`webserver.api.password` on start, overwriting the existing `pwhash`. Whoever knows the present
|
||||
password (nobody, which is why this started) loses it; the declared one replaces it. I have not
|
||||
observed that replacement happen.
|
||||
- **Cold-start cost is unmeasured and could matter.** `postStart` must complete before the container
|
||||
is reported Running. On a fresh PVC the hook waits for `gravity.db` (bounded, up to 180s) and then
|
||||
runs a real `pihole -g` (72k domains, network downloads). `apps-wave-0` is `wait: true` and
|
||||
`apps-wave-1` `dependsOn` it, so a slow cold start **delays the app tier**. The steady-state path
|
||||
exits in about a second (scenario 1), so this is a rebuild/first-boot cost only — but it is not
|
||||
measured, and the stub in testing means `pihole -g`'s real duration here is unknown.
|
||||
- **Whether `pihole -g` succeeds from inside the hook's environment.** Its failure is caught and the
|
||||
existing `gravity.db` retained, so a failure degrades rather than breaks — untested live.
|
||||
- **Concurrency.** Nothing serialises the hook against FTL's own scheduled gravity update. SQLite
|
||||
handles the locking, and the hook's writes are three small statements, but no interleaving was
|
||||
exercised.
|
||||
- **MULTI-REPLICA — PHASE 2, NOT STARTED.** This branch only removes the blocker. The remaining work,
|
||||
and it is substantial: a `replicas`/placement field on the ServiceDefinition schema (currently
|
||||
`additionalProperties: false` with no such field), `render.py` emitting a StatefulSet with
|
||||
`volumeClaimTemplates` (the config PVC is **ReadWriteOnce**, so a 2-replica Deployment cannot
|
||||
work), pod anti-affinity, a PodDisruptionBudget (there is none today), re-rendered goldens, and
|
||||
`conformance/**` updates — the last making phase 2 sensitive-path as well. The PVC rename a
|
||||
StatefulSet forces will discard the existing `gravity.db`, which is only safe **because** of this
|
||||
branch, and should be done only after this one is confirmed working in-cluster.
|
||||
- **The real risk the operator named is NOT fixed by phase 2 either.** Replicas address *node*
|
||||
failure. LAN DNS still depends on the cluster as a whole — etcd quorum, Cilium, Longhorn, a bad
|
||||
reconcile. Quantified on 2026-10-04: with `tolerationSeconds=300` plus node-NotReady detection
|
||||
plus a Longhorn RWO force-detach, a failure of the node running the pod is roughly **6-10 minutes
|
||||
of no LAN DNS**; a failure of the node holding the `pihole-dns-*` L2 lease (currently a
|
||||
*different* node — lease on k3s-prod-2, pod on k3s-prod-1) is tens of seconds. An out-of-cluster
|
||||
secondary resolver carrying the same `address=` lines is the only option that removes the
|
||||
dependency, and it was considered and not chosen.
|
||||
- **Port 80 on the `pihole` Service is still published** and still closed only by policy. Not narrowed.
|
||||
@@ -306,27 +306,19 @@ node_baseline_dns_servers:
|
||||
# k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
|
||||
# these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
|
||||
#
|
||||
# THE PUBLIC ANSWER IS NOW THE WAN, SO THE ISSUER IS PINNED (2026-10-03). Since
|
||||
# platform/gateway/gateway.yaml set an external-dns `target`, every public name is a CNAME
|
||||
# to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the
|
||||
# LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
|
||||
# k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18). The apiserver would then
|
||||
# be unable to fetch its issuer's discovery/JWKS, so kubectl OIDC SSO would fail on the
|
||||
# next k3s restart or Keycloak key rotation. node_baseline_hosts_pins (below) pins the
|
||||
# issuer host to the Gateway VIP in /etc/hosts, which glibc consults before these
|
||||
# resolvers. (History: before the 2026-09-04 subdomain split the public answer was DEV's
|
||||
# VIP; after it, the public answer was prod's own VIP and no pin was needed.)
|
||||
# THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
|
||||
# authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
|
||||
# publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
|
||||
# kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
|
||||
# keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
|
||||
# PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
|
||||
# and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
|
||||
#
|
||||
# The cert-based admin kubeconfig never depends on any of this. It is the break-glass path.
|
||||
# STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
|
||||
# external-dns has actually published the record. Until then the issuer does not resolve
|
||||
# from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
|
||||
# path, and it always works. Ordering, not a design gap.
|
||||
node_baseline_dns_search: []
|
||||
# /etc/hosts pins: node-process names that must reach the in-cluster Gateway directly rather
|
||||
# than via the public (WAN, non-hairpinned) answer. See "THE PUBLIC ANSWER IS NOW THE WAN" above.
|
||||
# [parity] the IP MUST equal GATEWAY_VIP in clusters/prod/kustomization.yaml (192.168.2.20),
|
||||
# and the name MUST follow k3s_oidc_issuer_url's host (roles/k3s_server/defaults/main.yml).
|
||||
node_baseline_hosts_pins:
|
||||
- ip: "192.168.2.20"
|
||||
names:
|
||||
- "keycloak.{{ k3s_base_domain }}"
|
||||
|
||||
# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
|
||||
# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
|
||||
|
||||
@@ -242,9 +242,3 @@ node_baseline_dns_search:
|
||||
node_baseline_dns_options:
|
||||
- timeout:2
|
||||
- attempts:2
|
||||
# Static /etc/hosts entries (tags: dns, hosts), as a list of {ip, names}. EMPTY by default:
|
||||
# dev needs none. Prod pins its OIDC issuer host to the Gateway VIP because the public
|
||||
# answer for it is the non-hairpinned WAN IP (inventory/group_vars/k3s_prod/vars.yml).
|
||||
# Written as ONE marked block, so removing an entry here removes it from the node, and
|
||||
# an empty list removes the block entirely.
|
||||
node_baseline_hosts_pins: []
|
||||
@@ -1090,22 +1090,3 @@
|
||||
{% endif %}
|
||||
when: node_baseline_dns_mechanism == 'resolv-conf-file'
|
||||
tags: [dns]
|
||||
|
||||
# /etc/hosts pins — tag `dns` (and `hosts` on its own). Some node processes must reach a
|
||||
# public hostname at the in-cluster Gateway, not at its public answer. On prod, the
|
||||
# kube-apiserver's OIDC issuer resolves publicly to the WAN IP, which the router does not
|
||||
# hairpin. glibc reads /etc/hosts before any nameserver (nsswitch `files dns`), so a pin wins
|
||||
# regardless of which resolver mechanism above is in effect. One marked block: entries
|
||||
# dropped from node_baseline_hosts_pins disappear from the node, and an empty list
|
||||
# removes the block.
|
||||
- name: Pin node-process hostnames in /etc/hosts
|
||||
ansible.builtin.blockinfile:
|
||||
path: /etc/hosts
|
||||
marker: "# {mark} node_baseline hosts pins (Ansible, tags: hosts)"
|
||||
block: |
|
||||
{% for pin in node_baseline_hosts_pins %}
|
||||
{{ pin.ip }} {{ pin.names | join(' ') }}
|
||||
{% endfor %}
|
||||
state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
|
||||
backup: true
|
||||
tags: [dns, hosts]
|
||||
@@ -20,30 +20,6 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
|
||||
its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
|
||||
Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.
|
||||
|
||||
## Public target + in-cluster DDNS (2026-10-03)
|
||||
|
||||
Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
|
||||
|
||||
```
|
||||
plex.home.keithjackson.dev CNAME wan.home.keithjackson.dev <- external-dns (Gateway `target` annotation)
|
||||
wan.home.keithjackson.dev A <current WAN IP> <- cloudflare-ddns (ddns.yaml), every 5m
|
||||
```
|
||||
|
||||
- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
|
||||
applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
|
||||
nothing.
|
||||
- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
|
||||
It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
|
||||
`egress-class: vpn`.
|
||||
- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
|
||||
(split-horizon) and never use the public path.
|
||||
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
|
||||
OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
|
||||
the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
|
||||
|
||||
Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
|
||||
`kubectl -n external-dns logs deploy/cloudflare-ddns`.
|
||||
|
||||
## Secrets
|
||||
|
||||
Cloudflare API token referenced **by name** (`cloudflare-api-token`, shared with cert-manager),
|
||||
|
||||
@@ -1,87 +0,0 @@
|
||||
---
|
||||
# In-cluster dynamic DNS for the home WAN (2026-10-03).
|
||||
#
|
||||
# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
|
||||
# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
|
||||
# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
|
||||
# moves all of them through this one record, with no Git change and no external-dns involvement.
|
||||
#
|
||||
# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
|
||||
# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
|
||||
# Conversely this updater manages only the names in DOMAINS and never touches external-dns's records.
|
||||
#
|
||||
# IP DETECTION. Default `cloudflare.trace`: an HTTPS request to Cloudflare's trace endpoint, which
|
||||
# reports the address the request arrived from. Correct here only because this pod is NOT in the
|
||||
# `vpn` egress class — the CiliumEgressGatewayPolicy (platform/vpn-egress) selects pods by
|
||||
# `homelab.keithjackson.dev/egress-class: vpn` alone, cluster-wide. Labelling this pod `vpn` would
|
||||
# publish the VPN exit IP instead of the WAN IP.
|
||||
#
|
||||
# TOKEN. Reuses this namespace's `cloudflare-api-token` (same Zone:DNS:Edit token as external-dns
|
||||
# and cert-manager) — no new secret.
|
||||
#
|
||||
# TIER NOTE. This lives in the gating `platform` Flux Kustomization (wait: true) next to
|
||||
# external-dns. If it ever crashloops, `platform` goes not-Ready and apps stop reconciling (no
|
||||
# outage). Kept here rather than in its own non-gating Kustomization to avoid a clusters/** change.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: cloudflare-ddns
|
||||
namespace: external-dns
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflare-ddns
|
||||
spec:
|
||||
replicas: 1 # exactly one writer for the record
|
||||
revisionHistoryLimit: 3
|
||||
strategy:
|
||||
type: Recreate # never two updaters at once
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: cloudflare-ddns
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflare-ddns
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: cloudflare-ddns
|
||||
image: favonia/cloudflare-ddns:1.17.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
env:
|
||||
- name: CLOUDFLARE_API_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: cloudflare-api-token
|
||||
key: api-token
|
||||
- name: DOMAINS
|
||||
value: "wan.${BASE_DOMAIN}"
|
||||
- name: IP6_PROVIDER
|
||||
value: "none" # IPv4 WAN only; leaves any AAAA alone
|
||||
- name: PROXIED
|
||||
value: "false" # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
|
||||
- name: UPDATE_CRON
|
||||
value: "@every 5m"
|
||||
- name: DELETE_ON_STOP
|
||||
value: "false" # a pod restart must not blank the record every CNAME depends on
|
||||
- name: RECORD_COMMENT
|
||||
value: "managed by k3s cloudflare-ddns (platform/externaldns/ddns.yaml)"
|
||||
- name: EMOJI
|
||||
value: "false"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 5m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
memory: 64Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
@@ -81,29 +81,38 @@ spec:
|
||||
# what this instance can see or own; per-resource exclusion only ever REMOVES an object from
|
||||
# this instance's view, so it cannot make prod capable of deleting a record it does not own.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
|
||||
# THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
|
||||
#
|
||||
# Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
|
||||
# RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
|
||||
# fix, and both still hold:
|
||||
# * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
|
||||
# from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
||||
# gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
|
||||
# shared Gateway therefore moves EVERY gateway-sourced record at once.
|
||||
# * The home WAN IP is residential and can change, so a literal IP in Git would rot.
|
||||
# The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
|
||||
# that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
|
||||
# actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
|
||||
# this change, for two independent reasons:
|
||||
#
|
||||
# The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
|
||||
# UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
|
||||
# (b) the address is kept current IN-CLUSTER. Implementation:
|
||||
# * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
|
||||
# each published name becomes a CNAME to that one hostname;
|
||||
# * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
|
||||
# record at the current WAN IP. That record has no external-dns TXT registry entry, so this
|
||||
# instance never owns or garbage-collects it.
|
||||
# Internal services are unchanged: their routes carry `controller: none` (above), which is
|
||||
# checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
|
||||
# LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
|
||||
# LB VIP. Whether those should go public is a separate, still-open question.
|
||||
# 1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
|
||||
# (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
|
||||
# points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
|
||||
# certificate. Whether prod's public services belong behind it, behind a new forward, or
|
||||
# are not meant to be public at all is an operator decision. Breaking five services' DNS
|
||||
# on a guess is worse than leaving a known-wrong-but-stable state.
|
||||
# 2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
|
||||
# real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
|
||||
# GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
||||
# gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
|
||||
# no per-service target override while every public service shares the `shared` Gateway.
|
||||
# Annotating the Gateway would move EVERY gateway-sourced record, which after this change
|
||||
# still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
|
||||
# routable address for admin UIs. That is strictly worse than the status quo.
|
||||
#
|
||||
# SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
|
||||
# is answered, because a knob that cannot take effect invites someone to set it and expect one:
|
||||
# (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
|
||||
# port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
|
||||
# (b) HOW should they be separated from the non-public names that share the Gateway? Options,
|
||||
# roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
|
||||
# annotation, with only the public services' routes attached; or publishing the five names
|
||||
# from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
|
||||
# which restores per-service targeting without a second Gateway.
|
||||
# Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
|
||||
#
|
||||
|
||||
@@ -8,5 +8,3 @@ resources:
|
||||
# SOPS-encrypted Cloudflare token (env CF_API_TOKEN) — SAME token as cert-manager, second copy in
|
||||
# the external-dns namespace. Flux decrypts at reconcile.
|
||||
- cloudflare-api-token.sops.yaml
|
||||
# DDNS for `wan.${BASE_DOMAIN}` — the CNAME target every Gateway-published record points at.
|
||||
- ddns.yaml
|
||||
@@ -7,23 +7,6 @@ kind: Gateway
|
||||
metadata:
|
||||
name: shared
|
||||
namespace: gateway
|
||||
annotations:
|
||||
# PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
|
||||
# route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
|
||||
# unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
|
||||
# becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
|
||||
# pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
|
||||
#
|
||||
# Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
|
||||
# admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
|
||||
# `exposure: public` apps. `internal` services stay unpublished via their
|
||||
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
|
||||
# BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
|
||||
# from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
|
||||
#
|
||||
# LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
|
||||
# ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
|
||||
external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
|
||||
spec:
|
||||
gatewayClassName: cilium
|
||||
listeners:
|
||||
|
||||
@@ -1,148 +0,0 @@
|
||||
# plex-remote-access — getting Plex reachable from outside the LAN, and proving it
|
||||
|
||||
**Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
|
||||
NOT work until both are done.** Neither can live in this repo.
|
||||
**Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
|
||||
**operator procedure**, not an incident page.
|
||||
**Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
|
||||
`clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
|
||||
`catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
|
||||
env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
|
||||
**Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
|
||||
procedures with a manual last mile.
|
||||
|
||||
## What was actually broken (measured 2026-10-03)
|
||||
|
||||
Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
|
||||
clients:
|
||||
|
||||
1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
|
||||
`customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
|
||||
`PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
|
||||
nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
|
||||
lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
|
||||
a mapping to the pod IP would be meaningless if it could.
|
||||
2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
|
||||
`136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
|
||||
Let's Encrypt wildcard.
|
||||
|
||||
The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.
|
||||
|
||||
**Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
|
||||
the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
|
||||
failure. But a router certificate answering 443 means the router's own management UI is bound to the
|
||||
WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
|
||||
real outside vantage point** (phone on cellular, or an external port checker) before concluding
|
||||
either way — this repo has no such vantage point and must not pretend otherwise.
|
||||
|
||||
## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
|
||||
|
||||
WAN tcp/32400 -> 192.168.2.24 (${PLEX_L4_VIP}) port 32400
|
||||
|
||||
`192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
|
||||
`CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
|
||||
router by hand, so it must never move on a reschedule.
|
||||
|
||||
While in the router, **check what else is bound to the WAN interface.** If remote management is
|
||||
listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
|
||||
Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
|
||||
though remote *streaming* works. These are two separate paths and they fail independently.
|
||||
|
||||
Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
|
||||
|
||||
`EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
|
||||
is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.
|
||||
|
||||
**`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
|
||||
(IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
|
||||
LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.
|
||||
|
||||
## Step 2 — tick "manually specify public port" in Plex (operator, UI only)
|
||||
|
||||
**There is no GitOps path to this and no environment variable for it.** Measured in the running
|
||||
image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
|
||||
`/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
|
||||
only six preferences, and this is not one of them. It is the same class of manual last mile as the
|
||||
`HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.
|
||||
|
||||
In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):
|
||||
|
||||
**Settings → Remote Access**
|
||||
- Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
|
||||
- Tick **Manually specify public port** and set it to **32400**.
|
||||
- **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.
|
||||
|
||||
Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
|
||||
connection, so the port-forward in step 1 sits there unused.
|
||||
|
||||
## How to verify
|
||||
|
||||
Do these in order; each one isolates a different half.
|
||||
|
||||
**1. The port is open from outside.** From a phone on cellular, or any off-LAN host:
|
||||
|
||||
curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
|
||||
|
||||
`200` means the forward and the policy are both good. A hang or refusal means step 1.
|
||||
|
||||
**2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
|
||||
outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
|
||||
was not applied.
|
||||
|
||||
**3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
|
||||
actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
|
||||
the server appears and plays. A client finds this server through plex.tv's connection list, never
|
||||
through DNS.
|
||||
|
||||
**4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
|
||||
CiliumNetworkPolicy is the next suspect:
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
|
||||
|
||||
See `runbooks/denied-flows.md` for reading actual drops.
|
||||
|
||||
## Why no alert
|
||||
|
||||
Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
|
||||
pod is `Running`, the policy is applied, and `plex-down` watches
|
||||
`kube_deployment_status_replicas_available` — none of which can see a router with a missing
|
||||
port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
|
||||
not have. **Test 3 above is the only honest check, and it is manual.**
|
||||
|
||||
## Why not the Gateway URL (`ADVERTISE_IP`)
|
||||
|
||||
The alternative was to leave Plex on `ClusterIP` and set
|
||||
`ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
|
||||
preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
|
||||
hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
|
||||
line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
|
||||
lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
|
||||
itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
|
||||
endpoint.
|
||||
|
||||
**The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
|
||||
its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
|
||||
of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
|
||||
authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
|
||||
the deferral note there is still accurate.
|
||||
|
||||
## Known wart: remote clients are classified as LAN
|
||||
|
||||
`apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
|
||||
ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
|
||||
`ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
|
||||
which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.
|
||||
|
||||
This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
|
||||
show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
|
||||
|
||||
`Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
|
||||
elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
|
||||
announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
|
||||
exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
|
||||
top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
|
||||
node mismatch is the normal case here, not a rare one.
|
||||
Reference in new issue
Block a user