jakman85andClaude Opus 5
c8c1d2935d
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
platform/oauth2-proxy
oauth2-proxy — forward-auth at the Gateway for apps without usable OIDC, authenticating against Keycloak.
- Governing ADR: ADR-0014 — IdP: Keycloak
- Spec: docs/15 — Identity & Access
- Serves: R-IDP-01..06 (esp. R-IDP-02: management UIs never unauthenticated)
- Owner: WS-SEC (Gateway wiring shared with WS-NET)
What this stub is
Flux HelmRelease for oauth2-proxy as the forward-auth provider for non-OIDC apps (PiHole, *arr UIs,
qBittorrent, Homepage, Prometheus, Alertmanager, Hubble UI — ADR-0014). Replica count is an overlay
knob (${OAUTH2_PROXY_REPLICAS}, R-ENV-02). Client/cookie secrets referenced by name,
SOPS-encrypted (WS-SEC).
Contract (SSO/auth + Gateway)
Pairs with ../keycloak to realize the "SSO/auth contract": a ServiceDefinition auth: forward-auth service is protected by oauth2-proxy at the Gateway (ext-auth); auth: oidc services go
native to Keycloak. Gateway ext-auth wiring is a WS-NET contract.
Enforcement (ext_authz forward-auth)
The authenticator (oauth2-proxy) is deployed and wired to Keycloak: provider=oidc,
oidc-issuer-url=.../realms/homelab, --reverse-proxy, --upstream=static://202,
--set-xauthrequest, cookie/redirect domain .home.keithjackson.dev, and — added here —
--allowed-groups=platform-admins (R-IDP-06 floor) + --skip-provider-button (302 straight to
Keycloak). The Keycloak oauth2-proxy client already carries the auth.home…/oauth2/callback
redirect URI and a groups claim with full.path=false (bare platform-admins) — no realm change
needed.
Mechanism. Forward-auth is an Envoy ext_authz HTTP filter that calls oauth2-proxy's
/oauth2/auth decision endpoint: 202 for a valid session in platform-admins (request passes to
the upstream, X-Auth-Request-* identity headers forwarded), 401/403 otherwise. With
failure_mode_allow: false the gateway fails closed — a denied or errored check never reaches the
upstream UI (R-IDP-02). The filter + oauth2-proxy cluster live in
forward-auth-cec.yaml.
Why this is NOT wired into the shared Gateway (Cilium 1.16.5 constraint)
The shared Gateway's Envoy program is a machine-owned, constantly-regenerated
CiliumEnvoyConfig (gateway/cilium-gateway-shared, ownerReferences->Gateway, regenerated on every
HTTPRoute change). Cilium 1.16.5 exposes no supported hook to inject an ext_authz filter into
that generated listener, and native Gateway-API ExternalAuth (GEP-1494) ships in no released Cilium
version yet (PoC only). So there is no clean per-route ext_authz on the shared Gateway today.
Options (PM decision):
| # | Approach | Pros | Cons |
|---|---|---|---|
| A | Interim self-authored CEC — a dedicated ext_authz HTTPS listener on its own L2 VIP (forward-auth-cec.yaml), protected hosts moved off the shared Gateway and DNS repointed. |
Works on 1.16.5 today; per-route opt-in; reuses the wildcard TLS via SDS; verified below. | Hand-authored TLS+routes; a 2nd ingress VIP + DNS; per-app maintenance; not Gateway-API-native. |
| B | oauth2-proxy in reverse-proxy mode per app — app HTTPRoute backendRef -> a per-app oauth2-proxy (--upstream=http://app.svc). |
Fully Gateway-API-native (just a backendRef swap); robust; no CEC. | One oauth2-proxy instance/config per app; more pods; not the single-proxy ext_authz model. |
| C | Upgrade Cilium to a release with native ExternalAuth (GEP-1494) and gate routes with an HTTPRoute filter. |
Cleanest, truly per-route, Crafty-friendly. | Not released yet; large jump from 1.16.5; must keep the Ansible-bootstrap/Flux parity contract in lock-step. |
Recommendation: ship the oauth2-proxy hardening now (done); adopt A as the interim enforcement for the sensitive UIs when the Phase-2 L2 VIP pool lands; track C as the long-term target and drop A once released.
Opt-in (per route)
A route is protected only by being listed as a virtualHost on the ext_authz listener — apps not
listed keep hitting the shared Gateway. To protect an app (pihole today; Crafty next): add its
Service + EDS cluster + virtualHost to forward-auth-cec.yaml, remove its HTTPRoute from the shared
Gateway, and repoint its DNS at the ext_authz VIP. The ServiceDefinition already declares
auth.mode: forward-auth + groups: [platform-admins] (rendered onto the HTTPRoute annotations) — the
opt-in signal. Ready to extend to Crafty and the other non-OIDC UIs (PiHole/*arr/qBittorrent/Homepage).
Proof (in-cluster, cilium-envoy v1.30.8, 2026-08-28)
A throwaway standalone cilium-envoy pod ran the exact ext_authz stanza from forward-auth-cec.yaml
against the live oauth2-proxy Service, with a no-filter control route on the same backend:
- Control (
:8081, no ext_authz) -> 200CONTROL-OK. - Protected (
:8080, ext_authz), unauthenticated -> 403; body is oauth2-proxy's sign-in page, not the upstream (UPSTREAM-OKnever served). oauth2-proxy's own access log recorded the/oauth2/authcall from the envoy pod.failure_mode_allow: falseconfirmed fail-closed.
This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on this exact Envoy. The
allow path (valid platform-admins session -> 202 -> upstream) needs a real browser round-trip.
Browser test (for the human, once forward-auth-cec.yaml is deployed)
- Incognito ->
https://pihole.home.keithjackson.dev-> expect a 302 tohttps://keycloak.home.keithjackson.dev/realms/homelab/...(not the PiHole UI). - Log in as a platform-admins user (MFA/OTP enforced) -> land back on the PiHole UI.
- Log in as a non-platform-admins user -> expect 403 (authenticated but not authorized).
- Confirm the session cookie is
.home.keithjackson.dev-scoped (single sign-on across the other protected UIs).