jakman85andClaude Opus 5
99560c2d22
PR #28 fixed the cookie collision, the 5-minute session and the missing metrics
listener in one renderer function, and deliberately declined `--api-route`
because the config model could not express the difference between two apps.
This adds the hook it was missing and sets the flag only where a measurement
says to.
THE DEFECT. oauth2-proxy answers an unauthenticated request with a 302 to
Keycloak. Its AJAX guard would send 401 instead, but it fires only on an EXACT
`Accept: application/json`, which these apps' JavaScript does not consistently
send. So a browser XHR gets a 302, FOLLOWS IT INVISIBLY, has it satisfied
silently by the live SSO session, and resolves 200 carrying the app's HTML where
the JavaScript expected JSON. On a write it is worse: a browser following a 302
re-issues a POST as a GET and DROPS THE BODY, so the operator's change is
discarded and the UI reports success.
MEASURED, on two apps and two different paths -- which is the whole argument:
jackett 2026-09-30: 33 x 302 vs 2 x 401 on /api/ paths, and 4 of 6 POSTs
answered by the proxy and never reaching Jackett (upstream `-`).
radarr 2026-10-01, 23h after PR #28 deployed (pod radarr-7d55cbffb9-42hmh,
0 restarts): 232 unauthenticated requests in the pod's whole life and
ALL 232 are the same call -- POST /signalr/messages/negotiate from a
browser, 302, upstream `-`. A left-open tab whose session the cookie
rename orphaned, polling forever, silently redirected every time and
NEVER re-prompting. The reported symptom, still in production.
So radarr's broken path is /signalr/ and jackett's is /api/, and PIHOLE SERVES
NO /api/ AT ALL. One function renders all five sidecars, so the cheap fix is a
constant -- and a constant applies one app's measurement to four apps nobody
measured. `auth` was `additionalProperties: false`, so there was no per-app
hook. Adding one named, documented key is the change; the block stays closed.
schemas/servicedefinition.schema.json 1.8.0 -> 1.9.0: optional
`auth.apiRoutes`, a list (radarr needs two, and --api-route is natively
repeatable) of anchored path regexes. Plus the shape invariant that a
non-empty list requires mode: forward-auth, since no sidecar is rendered
for oidc/native/none and declaring it there would be a SILENT NO-OP.
catalog/render.py emits one --api-route per entry, in order, and RAISES on
an unanchored entry. The flag is SINGULAR -- verified against the pinned
v7.8.1 binary's own --help; --api-routes does not exist and the proxy
refuses to start with it.
catalog/{jackett,radarr}.yaml the two measured apps.
catalog/{pihole,sonarr,qbittorrent}.yaml comments recording that they get
NOTHING, and why. Sonarr and qBittorrent were measured (0 unauthenticated
requests in 23h) and left alone: that is evidence of no TRAFFIC, not of no
need, and sonarr being nearly radarr's codebase is what makes the guess
tempting and still a guess.
apps/wave-0/jackett.yaml, apps/wave-1/radarr.yaml regenerated. Only TWO
goldens change; pihole/qbittorrent/sonarr are byte-identical to main,
which is the per-app property proving itself on the real tree.
BREADTH IS THE DANGEROUS DIRECTION AND IT IS NOT SYMMETRIC. A missing
--api-route is a silent 302. A too-broad one answers a plain PAGE LOAD with
401, so a logged-out operator gets a bare error page and is never offered a
sign-in -- the service looks permanently broken. Each entry must therefore
match `^\^/[A-Za-z0-9._~/-]+(\.\*|\$)?$`: anchored at ^/ then literal path
characters. `^/`, `^/.*`, `.*`, `/api/` and `^/api/|^/` are all rejected, all
five provoked. Alternation is refused rather than supported because the flag is
repeatable, so one entry per route costs nothing.
NEW GATE CHECK: conformance/tests/l0-static/oauth2-proxy-api-routes-per-app
(whole-repo scope, so ci/l0.sh's --include-scope union runs it on every
invocation; no bespoke stage, which whole-repo-invariants-wired would fail).
It is a SIBLING of oauth2-proxy-cookie-names-distinct, not an extension of it:
that check's subject is cookie IDENTITY and shares neither inputs nor failure
mode with this, so bundling them would make its name a lie and let a future
api-route decision be expressed only by weakening a COOKIE check.
The check exists for one reason the other gates cannot cover: render-golden
compares the render to the goldens, so a RENDERER CONSTANT plus regenerated
goldens AGREE WITH EACH OTHER while both being wrong. Provoked: the regen exits
0 -- render-golden is happy -- and this check exits 1 with 15 failures naming
pihole, qbittorrent, radarr and sonarr individually. It also asserts the
anchoring pattern is the SAME in the schema and the renderer, because three
surfaces guard this and if they drift the loosest wins silently.
Tested in both directions on throwaway trees outside the repo: compliant tree
rc 0 / 43 assertions / 6 sidecars; pre-change tree rc 1 / 11 failures; and four
targeted mutations (hand-edited golden, widened regex, renderer constant,
deleted schema pattern) each firing their own assertion class and nothing else.
ALSO FOLDED IN (coordinator's request; defect is in the file I own). _selftest()
reached _cookie_name() through a bare `next(genexpr)` with NO default, so
deleting the --cookie-name emission -- the exact mutation that assertion exists
to catch -- raised StopIteration, which escaped _selftest() BEFORE its collected
`SELFTEST FAIL:` lines were printed. Right verdict, zero diagnostics, and it
masked every other assertion the same mutation broke (found by Desmond Oyelaran
while wiring the selftest into the render-golden stage; his PR #33 is not in this
base -- origin/main @ cdf3e14 drives the selftest from no stage, checked). Fixed
with `next(..., None)` plus an explicit check(), so a missing flag reports as a
NAMED assertion. Provoking it exposed three more instances of the same class in
the same block -- `side = fa_containers[1]`, `side["env"][0]`, and the fa_svc
lookup all raise rather than assert when the sidecar is absent -- so all four are
now None-tolerant, and the two fixtures added for --api-route were written that
way from the start. Measured on throwaway copies:
--cookie-name deleted: before rc 1 / 0 FAIL lines / StopIteration
after rc 1 / 3 FAIL lines / no traceback
sidecar not appended: before rc 1 / 0 FAIL lines / IndexError
after rc 1 / 20 FAIL lines / no traceback
Goldens are unchanged by this part; it touches _selftest() only.
NOT verified, in full in the handoff entry: no browser round-trip by anyone for
this flag -- that a 401 REPAIRS the operator's experience is reasoned from HTTP
semantics, not watched. `^/api/` on radarr is PR #30's measurement of radarr's
pre-fix pod, not mine; the current pod has zero /api/ traffic. Sonarr and
qBittorrent are UNMEASURED, not cleared. --cookie-refresh is still unexercised
(0 AuthSuccess on all five pods). Nothing was applied; the live pods are still
the pre-change spec.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
128 lines
9.1 KiB
YAML
128 lines
9.1 KiB
YAML
# ServiceDefinition: pihole
|
|
# Source of truth: spec docs/13-service-catalog.md, row 6.
|
|
# internal | egress internet | config on Longhorn (DR-critical config) | auth forward-auth.
|
|
# Also a platform DNS dependency (docs/09). Validates against schemas/servicedefinition.schema.json.
|
|
name: pihole
|
|
kind: container
|
|
source:
|
|
image: pihole/pihole
|
|
exposure: internal # LAN-only admin UI (docs/13 row 6; R-DNS/R-NET)
|
|
egress: internet # normal WAN egress (docs/03; R-NET-05)
|
|
storage:
|
|
- name: config
|
|
size: 2Gi
|
|
class: longhorn # replicated config state; DR-critical (docs/04 line 118; R-STATE-02/03)
|
|
dependencies: [] # default-deny: reaches only DNS; it PROVIDES DNS (docs/03; R-NET-01/02)
|
|
# ------------------------------------------------------------------------------------------------
|
|
# LAN SPLIT-HORIZON DNS — the half of R-DNS-03 that platform/coredns/coredns-custom.yaml defers to.
|
|
# ------------------------------------------------------------------------------------------------
|
|
# coredns-custom answers the ${BASE_DOMAIN} zone for PODS ONLY and says so in its own header ("LAN
|
|
# clients keep their own answers (PiHole split-horizon)"). Until now PiHole served no DNS at all —
|
|
# its Service was ClusterIP :80/:4180, no port 53, no LoadBalancer — so the LAN half did not exist
|
|
# and LAN clients depended on hand-maintained per-client dnsmasq files. One went stale after the dev
|
|
# decommission and cost the operator sonarr.${BASE_DOMAIN}. Upstream cannot cover for it either: the
|
|
# SOHO router that owns this LAN strips RFC1918 answers for public domains as rebind protection and
|
|
# returns NOERROR / ANSWER 0. apps/wave-0/pihole-dns.yaml exposes udp/53 + tcp/53 on
|
|
# ${PIHOLE_DNS_VIP}; these env vars are what make the ANSWERS declarative.
|
|
#
|
|
# WHY env AND NOT A RECONCILER: the repo's level-triggered CronJob reconcilers
|
|
# (platform/keycloak/client-secret-reconciler.yaml, the CNPG grant reconciler in
|
|
# apps/wave-0/nextcloud-db.yaml) exist because their state has NO declarative entry point — a
|
|
# Keycloak client secret lives in Keycloak's own database and a Postgres GRANT lives in the
|
|
# catalog, and nothing in a manifest can express either. PiHole v6 is not that case. Every FTL
|
|
# setting has an `FTLCONF_<section>_<key>` environment variable, and a value supplied that way is
|
|
# ENV-FORCED: FTL logs it as "forced through environment", applies it on every start ahead of
|
|
# /etc/pihole/pihole.toml, and marks it read-only in the admin UI so a human cannot drift it. That
|
|
# is strictly better than a reconciler — it is enforced at t=0 on a cold start rather than at the
|
|
# next cron tick, it cannot be defeated by the UI, and it carries no extra RBAC, image or Job. A
|
|
# CronJob here would be ceremony around a knob the app already exposes. (Verified on the live prod
|
|
# image, PiHole Core v6.4.3 / FTL v6.7, 2026-09-14.)
|
|
#
|
|
# THE PVC IS NOT THE MECHANISM AND MUST NOT BE. The `config` volume below mounts at /data/config;
|
|
# PiHole's actual state directory is /etc/pihole, which is EPHEMERAL in this Deployment. Anything
|
|
# configured through the UI is lost on the next reschedule — which is exactly what happened to the
|
|
# upstream servers: the live pod's pihole.toml carried a hand-set `upstreams` marked "### CHANGED"
|
|
# that no manifest declared, so a restart would have left PiHole with upstreams = [] and no
|
|
# recursion at all. Declaring upstreams here closes that too: PiHole must stay a usable GENERAL
|
|
# resolver, not only authoritative for this zone, or pointing LAN clients at it breaks the internet.
|
|
env:
|
|
# Default is LOCAL, which answers only clients on the same subnet as the listening interface. The
|
|
# pod's interface is on the CLUSTER pod CIDR, so every LAN client would be refused. ALL is
|
|
# required for a LoadBalancer-fronted resolver; what keeps PiHole from being an OPEN resolver is
|
|
# the exposure, not this setting — `loadBalancerSourceRanges: [${LAN_CIDR}]` on the DNS Services,
|
|
# enforced by Cilium in the datapath before load-balancing, plus a CiliumNetworkPolicy that admits
|
|
# nothing but udp/53 and tcp/53. See apps/wave-0/pihole-dns.yaml.
|
|
- name: FTLCONF_dns_listeningMode
|
|
value: "ALL"
|
|
# Recursion for everything that is NOT this zone. Public resolvers on purpose, NOT the SOHO
|
|
# router at the LAN gateway: that router is the box whose rebind protection strips RFC1918
|
|
# answers, and forwarding to a device that may forward back here invites a loop.
|
|
- name: FTLCONF_dns_upstreams
|
|
value: "8.8.8.8;8.8.4.4"
|
|
# Per-client query rate limit OFF (default: 1000 queries per 60s, then REFUSED). It has to be off
|
|
# because of how the DNS Services are exposed: apps/wave-0/pihole-dns.yaml uses
|
|
# externalTrafficPolicy Cluster — it must, or LAN resolution breaks whenever Cilium's L2
|
|
# announcement lease and the pihole pod land on different nodes, which is most of the time — and
|
|
# Cilium runs in SNAT mode here, so a query forwarded between nodes reaches PiHole carrying the
|
|
# INGRESS NODE's address instead of the client's. Every LAN device therefore counts against ONE
|
|
# apparent client's budget, and a normal evening of browsing from a handful of machines would trip
|
|
# a limit meant to catch one misbehaving host. Leaving it at the default turns the whole LAN's DNS
|
|
# off for 60 seconds at a time, intermittently, which is the worst possible failure shape. The
|
|
# limit becomes meaningful again — and should be restored — if Cilium is ever switched to
|
|
# `loadBalancer.mode: dsr`, which preserves the client address.
|
|
- name: FTLCONF_dns_rateLimit_count
|
|
value: "0"
|
|
# THE SPLIT-HORIZON ANSWERS. A dnsmasq `address=/<domain>/<ip>` entry answers that domain AND
|
|
# every subdomain of it, and dnsmasq resolves overlapping entries by LONGEST MATCHING DOMAIN —
|
|
# so the first line is a wildcard for the whole zone and each later line is a specific override.
|
|
# PRECEDENCE IS THEREFORE STRUCTURAL, not order-dependent: `unifi.${BASE_DOMAIN}` matches both
|
|
# line 1 and line 2, line 2 is the longer domain, line 2 wins. Rewriting these in any order
|
|
# changes nothing.
|
|
#
|
|
# A NEWLY ONBOARDED SERVICE NEEDS NO CHANGE HERE. Every catalog ingressRoute host is
|
|
# `<label>.${BASE_DOMAIN}` served by the shared Gateway, so the wildcard already answers it with
|
|
# ${GATEWAY_VIP} — the same guarantee, and the same reasoning, as the `template IN A` block in
|
|
# platform/coredns/coredns-custom.yaml. Add a line here ONLY for a service that does not live
|
|
# behind the Gateway: an L4 LoadBalancer on its own pinned VIP. Those are enumerated from the
|
|
# repo, not from memory — grep `external-dns.alpha.kubernetes.io/hostname` on a Service, which
|
|
# is the annotation that means "this Service publishes its own hostname at its own address":
|
|
# apps/vms/unifi/50-service-l4.yaml (unifi -> ${UNIFI_L4_VIP}) and
|
|
# platform/crafty/service-l4.yaml (crafty-mc -> ${CRAFTY_L4_VIP}). Those two, plus PiHole's own
|
|
# resolver address, are the complete set of exceptions today.
|
|
#
|
|
# AAAA needs nothing: dnsmasq answers a AAAA query for an `address=`d name with NODATA
|
|
# (NOERROR, no answer), so IPv6-capable clients fall back to the A record instead of getting
|
|
# NXDOMAIN — the same behaviour coredns-custom hand-writes a `template IN AAAA` block for.
|
|
#
|
|
# Array-valued FTLCONF variables are SEMICOLON-separated. A JSON-looking list is accepted and
|
|
# then silently wrong: FTL stores `["a","b"]` as ONE array element containing those literal
|
|
# characters, writes that single bogus line into /etc/pihole/dnsmasq.conf and reports "0 invalid"
|
|
# while resolution for the zone does not work. Confirmed both ways against the live image.
|
|
- name: FTLCONF_misc_dnsmasq_lines
|
|
value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
|
|
ingressRoutes:
|
|
- host: pihole.${BASE_DOMAIN}
|
|
port: 80 # admin web UI, protected at the Gateway by forward-auth (docs/09)
|
|
auth:
|
|
mode: forward-auth # oauth2-proxy -> Keycloak; bare UI, no native OIDC (docs/15; R-IDP-02)
|
|
groups:
|
|
- platform-admins # sensitive admin UI: authenticated AND in group (R-IDP-06)
|
|
# NO `apiRoutes` HERE, DELIBERATELY. PiHole's admin UI serves no /api/ path, so the `^/api/`
|
|
# that jackett and radarr need would match nothing on a good day — and a BROADER guess would
|
|
# answer PiHole's own page loads with 401, leaving a logged-out operator on a bare error page
|
|
# with no sign-in offered. This app is the reason auth.apiRoutes is per-service (schema 1.9.0)
|
|
# rather than a renderer constant: the measurement that justifies the flag on two apps is
|
|
# evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
|
|
# the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
|
|
update:
|
|
channel: stable # R-UPD-01
|
|
range: ">=2024.0.0"
|
|
cadence: "0 4 * * *"
|
|
autoRollbackOnFailedHealth: true # R-UPD-03
|
|
observability:
|
|
metrics: true
|
|
slis:
|
|
- up # R-OBS-01
|
|
backup:
|
|
policy: daily-offcluster # PiHole config is DR-critical (docs/04 line 118; R-STATE-03)
|