Compare commits
11
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
28d0e0a0f2 | ||
|
|
b633b7fb08 | ||
|
|
c68349f4ab | ||
|
|
6770087d68 | ||
|
|
7598cff945 | ||
|
|
0ba06ec25b | ||
|
|
95b4b6af21 | ||
|
|
a7fc19fad5 | ||
|
|
70a98675ac | ||
|
|
cc1fb1d3d5 | ||
|
|
fb153f0bb5 |
No files matched your search
+19
-2
@@ -26,8 +26,11 @@ two things (e.g. "VM"), this file disambiguates.
|
||||
the port level (R-NET-01).
|
||||
- **Egress class** — a service's declared outbound path: `internet` (standard WAN) or `vpn`
|
||||
(commercial VPN tunnel). See [`docs/03-networking-isolation-and-egress.md`](docs/03-networking-isolation-and-egress.md).
|
||||
- **Exposure class** — how a service is reachable: `internal` (LAN-only) or `public` (edge
|
||||
port-forwarded).
|
||||
- **Exposure class** — how a service is reachable: `internal` (LAN/VPN-only) or `public` (published
|
||||
through the Cloudflare Tunnel, ADR-0016).
|
||||
- **Cloudflare Tunnel** — the outbound-only connection from in-cluster cloudflared to Cloudflare that
|
||||
carries all public HTTP(S) ingress; public records are proxied CNAMEs to `<uuid>.cfargotunnel.com`
|
||||
(ADR-0016).
|
||||
- **Kill-switch** — the guarantee that a `vpn`-class workload leaks nothing to the plain internet if
|
||||
the tunnel drops (R-NET-06); fail-closed.
|
||||
- **Multi-WAN** — multiple upstream internet uplinks with routing-rule-based selection; physical WAN
|
||||
@@ -58,6 +61,20 @@ two things (e.g. "VM"), this file disambiguates.
|
||||
- **cert-manager / ACME DNS-01 / Let's Encrypt** — the trusted, automated TLS pipeline (R-TLS-*).
|
||||
- **ExternalDNS** — controller that publishes DNS records from declared service intent.
|
||||
|
||||
## Identity & access
|
||||
- **IdP (Identity Provider)** — the central authority for user identity and login. Here: **Keycloak**
|
||||
([`docs/15-identity-and-access.md`](docs/15-identity-and-access.md)).
|
||||
- **SSO (Single Sign-On)** — log in once, reach every integrated service without re-authenticating.
|
||||
- **OIDC (OpenID Connect)** — the token-based auth protocol services use to federate to Keycloak
|
||||
natively (a Keycloak **client**).
|
||||
- **Forward-auth** — an identity-aware proxy (**oauth2-proxy**) at the Gateway that authenticates a
|
||||
request against Keycloak before forwarding it — used for apps without native OIDC.
|
||||
- **Realm** — a Keycloak tenant/namespace holding the platform's users, groups, and clients.
|
||||
- **Group** — a Keycloak grouping that drives per-service authorization (`auth.groups` in a
|
||||
ServiceDefinition); the basis for least-privilege access (R-IDP-06).
|
||||
- **Break-glass** — a documented emergency access path (e.g. a local admin on Gitea) so an IdP outage
|
||||
doesn't lock the operator out of recovery tools.
|
||||
|
||||
## Environments
|
||||
- **dev/test** — local KVM cluster; the active build target where everything is proven.
|
||||
- **prod** — 3× mini-PC bare-metal cluster; design-only until roadmap Phase 7.
|
||||
|
||||
@@ -78,7 +78,8 @@ home-network-k3s/
|
||||
│ ├── 11-automated-updates.md
|
||||
│ ├── 12-command-center-ui.md
|
||||
│ ├── 13-service-catalog.md
|
||||
│ └── 14-testing-and-conformance.md
|
||||
│ ├── 14-testing-and-conformance.md
|
||||
│ └── 15-identity-and-access.md
|
||||
├── decisions/ ← ADRs (one decision per file) + index/template
|
||||
├── roadmap/ ← phases.md, workstreams.md
|
||||
└── agents/ ← roles, coordination protocol, DoD, task board
|
||||
|
||||
@@ -53,7 +53,10 @@ screenshot, or "it reconciles" is **not** done (R-TEST-01..04, [`../docs/14-test
|
||||
### DNS & ingress (R-DNS-*)
|
||||
- [ ] `<service>.home.keithjackson.dev` record appears automatically and routes correctly.
|
||||
- [ ] `internal` is unreachable publicly; `public` is reachable + TLS-valid.
|
||||
- [ ] Internal clients resolve to the internal Gateway (no public hairpin).
|
||||
- [ ] Internal clients resolving via Pi-hole reach the internal Gateway; via a public resolver, a
|
||||
`public` service still works through the tunnel (R-DNS-04).
|
||||
- [ ] Non-`public` hostnames get 404 at the tunnel and have no public record (R-DNS-05); the
|
||||
Kubernetes API is not publicly reachable (R-SEC-04).
|
||||
|
||||
### Automated updates (R-UPD-*)
|
||||
- [ ] Advances to a newer **stable** version automatically on schedule with **no human/AI edit**.
|
||||
|
||||
@@ -50,8 +50,8 @@ Seeded from [`../roadmap/phases.md`](../roadmap/phases.md) + [`../roadmap/workst
|
||||
|---|---|---|---|---|---|---|
|
||||
| T-3.0 | L3 fault-injection harness — **in-cluster Chainsaw Job** (CI-triggered, scoped test SA) using `script`/`try`/`finally` (force tunnel down, kill node, inject bad image, trigger restore) | WS-TEST | R-TEST-05, R-SEC-02 | T-0.4 | — | todo |
|
||||
| T-3.1 | Egress classes; Cilium Egress Gateway wiring (steer `vpn` pods → gateway) | WS-NET | R-NET-04/05 | T-2.2 | — | todo |
|
||||
| T-3.2p | **Prereq:** provide **PIA username/password**; store SOPS-encrypted (PIA purchased) | WS-SEC | R-NET-05 | — | — | todo |
|
||||
| T-3.2 | **Single shared in-cluster PIA/Gluetun gateway** (WireGuard, active/standby) + built-in kill-switch + native PF (`qbittorrent-natmap` port-sync) (ADR-0013) | WS-NET | R-NET-05/06 | T-3.1, T-3.2p | — | todo |
|
||||
| T-3.2p | **Prereq:** provide **PIA username/password + `.ovpn`**; store in **Ansible vault** on the egress-gateway node(s) (ADR-0015) | WS-SEC | R-NET-05 | — | — | todo |
|
||||
| T-3.2 | **Node-level PIA/OpenVPN tunnel** (host systemd, Ansible, split-tunnel) on egress-gateway node(s) + **CiliumEgressGatewayPolicy** steering `egress-class=vpn` + **host nftables kill-switch** + host PF helper for qBittorrent (ADR-0015) | WS-NET | R-NET-05/06 | T-3.1, T-3.2p | — | todo |
|
||||
| ~~T-3.3~~ | ~~Edge-router multi-WAN split~~ — **out of scope** (single WAN, dumb SOHO edge; OQ-1) | WS-NET | R-NET-04 | — | — | won't-do |
|
||||
| T-3.4 | Prove: VPN IP egress, **kill-switch fail-closed (central)**, egress-class flip via git | WS-NET | R-NET-05/06/07 | T-3.0, T-3.2 | — | todo |
|
||||
|
||||
@@ -78,6 +78,8 @@ Seeded from [`../roadmap/phases.md`](../roadmap/phases.md) + [`../roadmap/workst
|
||||
| T-6.1 | Onboard **pihole** (DNS underpinning) | WS-APPS | R-DNS-03 | T-2.5 | — | todo |
|
||||
| T-6.1a | Onboard **gitea** early (bootstrapping-critical; self-hosts GitOps source + CI/CD) | WS-APPS | R-STATE-03 | T-4.4 | — | todo |
|
||||
| T-6.1b | Migrate GitOps source local→Gitea; move CI to **Gitea Actions** (Kyverno gate + Chainsaw Job trigger; PR merge gate); off-cluster repo mirror | WS-GITOPS | R-GITOPS-01, R-TEST-03 | T-6.1a | — | todo |
|
||||
| T-6.1c | Onboard **Keycloak** (IdP) + PostgreSQL (Longhorn, DR-critical) + **oauth2-proxy** forward-auth at Gateway; declarative realm/clients/groups; MFA; break-glass path | WS-SEC | R-IDP-01..06 | T-2.5, T-2.6 | — | todo |
|
||||
| T-6.1d | Render ServiceDefinition `auth` block → Keycloak OIDC client / oauth2-proxy protection | WS-SEC/WS-GITOPS | R-IDP-02/03 | T-0.3, T-6.1c | — | todo |
|
||||
| T-6.2 | Onboard **plex** (bulk tier + public exposure) | WS-APPS | R-STATE-04 | T-2.6, T-6.1 | — | todo |
|
||||
| T-6.3 | Onboard **qbittorrent, jackett** (VPN + kill-switch) | WS-APPS | R-NET-05/06 | T-3.4 | — | todo |
|
||||
| T-6.4 | Onboard **sonarr, radarr** (VPN + arr↔downloader allows) | WS-APPS | R-NET-02/05 | T-6.3 | — | todo |
|
||||
@@ -100,6 +102,6 @@ Seeded from [`../roadmap/phases.md`](../roadmap/phases.md) + [`../roadmap/workst
|
||||
|---|---|---|---|
|
||||
| OQ-1 | WAN count + edge router | — | ✅ resolved: 1 WAN, dumb SOHO edge; multi-WAN out of scope; VPN + segmentation in-cluster |
|
||||
| OQ-2 | Bulk-media home | — | ✅ resolved: media outside cluster, backing TBD later (not blocking) |
|
||||
| OQ-3 | VPN provider/protocol | — | ✅ resolved: **PIA** via Gluetun, single shared in-cluster tunnel, **WireGuard** + native PF (ADR-0013, supersedes 0012) |
|
||||
| OQ-3 | VPN provider/protocol + egress steering | — | ✅ resolved: **PIA over OpenVPN-UDP** (ADR-0013); steering = **node-level host tunnel + Cilium Egress Gateway + host nftables kill-switch** (ADR-0015, supersedes the in-cluster-Gluetun steering — a Gluetun pod's tun0 can't be steered by Cilium egress gateway) |
|
||||
| OQ-4 | DNS-01 challenge access | — | ✅ resolved: scoped **Cloudflare API token** (Zone·DNS·Edit), SOPS-encrypted; drives cert-manager + ExternalDNS. Operator provides token (T-2.4p) |
|
||||
| **OQ-BKP** | **Off-cluster DR backup target** (S3/NFS endpoint) | T-4.3 | ⛔ **OPEN** — needed by Phase 4, not Phase 2 |
|
||||
+4
-1
@@ -54,4 +54,7 @@ serves. **Do not edit an accepted decision in place** — supersede it with a ne
|
||||
| [0010](adr-0010-conformance-testing.md) | Conformance testing: automated, reproducible, behavior-based | Accepted | R-TEST-01..06 |
|
||||
| [0011](adr-0011-test-tooling-kyverno-chainsaw.md) | Conformance test tooling: Kyverno + Chainsaw | Accepted | R-TEST-01..05 |
|
||||
| [0012](adr-0012-vpn-egress-single-expressvpn-gateway.md) | VPN egress: single shared in-cluster ExpressVPN gateway | ~~Superseded by 0013~~ | R-NET-05/06/07 |
|
||||
| [0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md) | VPN provider: PIA via Gluetun (WireGuard + port forwarding) | Accepted | R-NET-05/06/07 |
|
||||
| [0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md) | VPN provider: PIA over OpenVPN-UDP + port forwarding | Accepted — provider/protocol only (client+steering superseded by 0015) | R-NET-05/06/07 |
|
||||
| [0014](adr-0014-idp-keycloak.md) | Identity Provider: Keycloak (SSO via OIDC + forward-auth) | Accepted | R-IDP-01..06 |
|
||||
| [0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md) | VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch) | Accepted | R-NET-05/06/07 |
|
||||
| [0016](adr-0016-public-ingress-cloudflare-tunnel.md) | Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding | Proposed | R-DNS-01..05, R-TLS-01/02, R-SEC-02/04 |
|
||||
@@ -1,9 +1,34 @@
|
||||
# ADR-0013 — VPN provider: PIA via Gluetun (WireGuard + native port forwarding)
|
||||
- Status: Accepted
|
||||
# ADR-0013 — VPN provider: PIA via Gluetun (OpenVPN-UDP + native port forwarding)
|
||||
- Status: Accepted — provider/protocol only (amended 2026-08-22 protocol→OpenVPN-UDP; Gluetun client + egress
|
||||
steering superseded by [ADR-0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md) 2026-08-23)
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-NET-05, R-NET-06, R-NET-07
|
||||
- Supersedes: [ADR-0012](adr-0012-vpn-egress-single-expressvpn-gateway.md)
|
||||
|
||||
> **Amendment — 2026-08-22 (protocol: WireGuard → OpenVPN-UDP).** The original decision specified
|
||||
> PIA over **WireGuard**. This is **not implementable with Gluetun**: Gluetun does not support PIA
|
||||
> over WireGuard — upstream states native support "cannot be added… slow work in progress," and it
|
||||
> was confirmed live rejecting `private internet access` as a WireGuard provider on v3.40 / v3.41 /
|
||||
> latest. **PIA in Gluetun is OpenVPN-only.** The decision is amended to **PIA over OpenVPN-UDP via
|
||||
> Gluetun**; everything else in this ADR stands (single shared gateway, central kill-switch /
|
||||
> fail-closed, native port forwarding, egress steering, active/standby). OpenVPN-UDP satisfies
|
||||
> R-NET's "non-TCP, UDP or better" requirement, and Gluetun's HTTP control API (health +
|
||||
> forwarded-port lookup) is what the shared-gateway topology relies on — a file-based PF handoff
|
||||
> (the WireGuard-container alternative) would pull toward the sidecar model ADR-0008 rejected.
|
||||
> **WireGuard fallback (documented, not primary):** if OpenVPN-UDP throughput ever proves a
|
||||
> bottleneck, `thrnz/docker-wireguard-pia` provides PIA-over-WireGuard as a drop-in gateway — a
|
||||
> contained swap, since the egress-steering design is protocol-agnostic. The filename retains the
|
||||
> `-wireguard` slug as a stable historical identifier.
|
||||
|
||||
> **Superseded in part — 2026-08-23 by [ADR-0015](adr-0015-vpn-egress-steering-node-tunnel-cilium.md).** The
|
||||
> **provider + protocol** here (PIA over OpenVPN-UDP, native port forwarding) STANDS. What is superseded: the
|
||||
> **client + placement** (a single in-cluster **Gluetun** pod) and the deferred **steering** mechanism. Live
|
||||
> validation confirmed the in-cluster Gluetun tunnel works (PIA egress) but **cannot be steered by Cilium
|
||||
> Egress Gateway** — Gluetun's `tun0` is in the pod netns, and `hostNetwork` Gluetun hijacks the node's own
|
||||
> default route (breaks k3s/etcd). ADR-0015 moves the tunnel to a **host OpenVPN service on egress-gateway
|
||||
> node(s)**, steered by **Cilium Egress Gateway** with a **host nftables kill-switch**. The same PIA `.ovpn`
|
||||
> transfers over.
|
||||
|
||||
## Context
|
||||
ADR-0012 kept the operator's required model — a single shared in-cluster VPN gateway with rules
|
||||
enforced outside the service — but named **ExpressVPN** as the provider, which forced a custom
|
||||
@@ -13,13 +38,16 @@ which *is* first-class in Gluetun. The egress **model is unchanged**; only the p
|
||||
protocol change — so ADR-0012 is superseded.
|
||||
|
||||
## Decision
|
||||
The single shared in-cluster VPN gateway runs **Gluetun** configured for **PIA** over **WireGuard**,
|
||||
The single shared in-cluster VPN gateway runs **Gluetun** configured for **PIA** over **OpenVPN-UDP**,
|
||||
with Gluetun's **native automated port forwarding** for qBittorrent:
|
||||
|
||||
- **Provider/client:** PIA via the **Gluetun** container as the gateway VPN client (Gluetun provides
|
||||
the built-in **kill-switch/firewall**, health checks, and auto-reconnect).
|
||||
- **Protocol:** **WireGuard** (UDP) — satisfies "non-TCP, UDP or better" and is a strict upgrade over
|
||||
ExpressVPN's OpenVPN-UDP.
|
||||
- **Protocol:** **OpenVPN-UDP** — satisfies "non-TCP, UDP or better." (WireGuard was the original
|
||||
choice but is not supported for PIA in Gluetun — see the amendment above.) Unlike the ExpressVPN
|
||||
plan this needs **no bespoke OpenVPN build**: PIA is a first-class Gluetun provider, so it is
|
||||
stock env-var config (`VPN_TYPE=openvpn`), and Gluetun exposes the tunnel state + forwarded port
|
||||
over its HTTP control API.
|
||||
- **Credentials:** just the **PIA username/password**, stored **SOPS-encrypted** ([ADR-0007](adr-0007-secrets-sops-age.md)).
|
||||
No manual `.ovpn` extraction (the ExpressVPN prerequisite is gone).
|
||||
- **Port forwarding:** Gluetun native (`VPN_PORT_FORWARDING=on`, provider `private internet
|
||||
@@ -32,8 +60,13 @@ with Gluetun's **native automated port forwarding** for qBittorrent:
|
||||
## Alternatives considered
|
||||
- **ExpressVPN (ADR-0012)** — dropped: not Gluetun-native, needed a custom OpenVPN workaround; and the
|
||||
operator's subscription is lapsing.
|
||||
- **ProtonVPN** — equally good fit (Gluetun-native NAT-PMP PF, WireGuard); viable if PIA
|
||||
disappoints, but the operator has purchased PIA.
|
||||
- **PIA over WireGuard via a dedicated container (`thrnz/docker-wireguard-pia`)** — the only way to
|
||||
keep **WireGuard** with PIA, since Gluetun cannot. Rejected as primary: it exposes the forwarded
|
||||
port as a shared **file** (`/pia-shared/port.dat`) rather than an HTTP API, which fits a same-netns
|
||||
sidecar (the model ADR-0008 rejected) better than a shared gateway, plus script-based health and a
|
||||
smaller single-maintainer image. Retained as the WireGuard **fallback** (see amendment).
|
||||
- **ProtonVPN** — equally good fit (Gluetun-native NAT-PMP PF, and Gluetun **does** support ProtonVPN
|
||||
over WireGuard); viable if PIA disappoints, but the operator has purchased PIA.
|
||||
- **AirVPN** — WireGuard works in Gluetun but automated PF is not integrated (static port only);
|
||||
less hands-off.
|
||||
- **Mullvad** — no port forwarding since 2023 → degraded qBittorrent seeding; rejected for this
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# ADR-0014 — Identity Provider: Keycloak (SSO via OIDC + forward-auth)
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-15
|
||||
- Serves: R-IDP-01, R-IDP-02, R-IDP-03, R-IDP-04, R-IDP-05, R-IDP-06
|
||||
- Supersedes: —
|
||||
|
||||
## Context
|
||||
The platform needs one identity and single sign-on: a central user directory, group-based
|
||||
authorization, MFA, and a way to protect both OIDC-capable apps and bare management UIs — without
|
||||
per-app passwords scattered across services. This also resolves the previously-open C2 auth question
|
||||
([12](../docs/12-command-center-ui.md)).
|
||||
|
||||
## Decision
|
||||
Use **Keycloak** as the platform IdP, integrated two ways behind one identity:
|
||||
- **Native OIDC** for apps that support it (Gitea, Nextcloud, Grafana, Kasm, Semaphore, Headlamp).
|
||||
- **Forward-auth proxy (oauth2-proxy → Keycloak)** at the Gateway for apps without usable OIDC
|
||||
(PiHole, *arr UIs, qBittorrent, Homepage, Prometheus, Alertmanager, Hubble UI).
|
||||
|
||||
Supporting decisions:
|
||||
- **Config as code**: realms/clients/groups are declarative and GitOps-reconciled (Keycloak Operator/
|
||||
Helm for the instance; `keycloak-config-cli` or realm-import for config). Secrets SOPS-encrypted.
|
||||
- **Authorization by groups**; **MFA** required for privileged groups.
|
||||
- **PostgreSQL on Longhorn**, DR-critical, backed up off-cluster; Keycloak rebuilds from DB + git.
|
||||
- **`public` exposure** (hardened) so login flows work for internet-exposed services.
|
||||
- **Exceptions**: Plex/Seerr use Plex auth; Minecraft uses Mojang/Microsoft; UniFi keeps native auth.
|
||||
|
||||
## Alternatives considered
|
||||
- **Authelia / Authentik** — lighter, and Authelia pairs cleanly with forward-auth; but Keycloak is
|
||||
the operator's explicit choice and offers the broadest OIDC/SAML, realm, and federation feature set
|
||||
for heterogeneous apps. (Chosen.)
|
||||
- **Per-app local auth only** — rejected: no SSO, password sprawl, weak central policy/MFA (fails
|
||||
R-IDP-01/06).
|
||||
- **Cloud IdP (e.g. hosted OIDC)** — rejected: this is a self-hosted, offline-capable home platform;
|
||||
identity should not depend on an external SaaS.
|
||||
|
||||
## Consequences
|
||||
- One login, one directory, central MFA and group policy; management UIs are never unauthenticated on
|
||||
the network (R-IDP-02).
|
||||
- **Keycloak becomes a critical dependency**: an IdP outage impairs SSO logins → DR-critical storage
|
||||
+ a documented **break-glass** local-admin path (e.g. on Gitea) so recovery tools stay reachable.
|
||||
- New platform components to run and secure: Keycloak, PostgreSQL, oauth2-proxy — all under
|
||||
default-deny with explicit allows and trusted TLS.
|
||||
- Onboarded **early** (with PiHole/Gitea) so later services integrate SSO as they come up.
|
||||
- Adds an `auth` block to the ServiceDefinition ([13](../docs/13-service-catalog.md)) so SSO mode +
|
||||
allowed groups are declarative per service.
|
||||
@@ -0,0 +1,99 @@
|
||||
# ADR-0015 — VPN egress steering: node-level tunnel + Cilium Egress Gateway (host kill-switch)
|
||||
- Status: Accepted
|
||||
- Date: 2026-08-23
|
||||
- Serves: R-NET-05, R-NET-06, R-NET-07
|
||||
- Supersedes (egress-steering + tunnel placement of): [ADR-0013](adr-0013-vpn-provider-pia-gluetun-wireguard.md)
|
||||
— which now stands only for the **provider/protocol** (PIA over OpenVPN-UDP + port forwarding). By
|
||||
extension this also closes the steering question left open since [ADR-0008](adr-0008-vpn-egress-model.md).
|
||||
|
||||
## Context
|
||||
ADR-0013 placed the VPN in a **single shared in-cluster Gluetun pod** and *deferred* the egress-steering
|
||||
mechanism (how `egress-class=vpn` pods actually route through it), naming "Cilium Egress Gateway → the
|
||||
gateway's tunnel interface" as the recommended-but-unvalidated option.
|
||||
|
||||
Implementation (2026-08) proved that option **non-viable**, and validated the tunnel itself:
|
||||
- The in-cluster Gluetun tunnel **works** — PIA Stockholm egress confirmed, stable.
|
||||
- But `CiliumEgressGatewayPolicy` SNATs matched pods out of a **node** interface; Gluetun-in-a-pod's `tun0`
|
||||
lives in the *pod* netns, invisible to the node datapath. Running Gluetun `hostNetwork` to put `tun0` on
|
||||
the node **breaks the node**: Gluetun owns its netns routing and on connect redirects `0.0.0.0/0` via the
|
||||
tunnel — in host netns that hijacks the **node's own default route**, cutting k3s/etcd/kubelet off. Gluetun
|
||||
has no "create `tun0` but don't touch the default route" mode.
|
||||
- The accepted community alternative — the **pod-gateway** pattern (VXLAN overlay + admission-webhook-injected
|
||||
client routing + gateway NAT, e.g. `angelnu/pod-gateway`) — works but is heavy, mutates client pods, and
|
||||
side-steps Cilium's datapath.
|
||||
|
||||
The clean fix is to move the tunnel to where routing is fully controllable — the **node** — and use Cilium's
|
||||
native egress gateway to steer.
|
||||
|
||||
## Decision
|
||||
The `egress: vpn` contract is realized by a **node-level VPN tunnel steered by Cilium Egress Gateway, with a
|
||||
host-level kill-switch**:
|
||||
|
||||
- **Tunnel — host service, not a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN
|
||||
systemd service** (PIA over OpenVPN-UDP — provider/protocol per ADR-0013, same `.ovpn`). It is **Ansible-
|
||||
managed host infrastructure** (like k3s itself), not a Flux workload. It is configured **split-tunnel**:
|
||||
it creates `tun0` but does **not** set the node default route (`route-noexec` + policy routing), so the
|
||||
node's own traffic (control plane, image pulls, other pods) stays on the WAN. Only traffic Cilium points
|
||||
at `tun0` uses the VPN.
|
||||
- **Steering — Cilium Egress Gateway.** A `CiliumEgressGatewayPolicy` selects pods labeled
|
||||
`homelab.keithjackson.dev/egress-class=vpn` and egresses them via the gateway node's `tun0` (SNAT to the
|
||||
tunnel address). Native Cilium eBPF — **no client-pod changes, no sidecar, no overlay**. (`egressGateway`,
|
||||
`bpf.masquerade`, `kubeProxyReplacement` are already enabled per ADR-0002.)
|
||||
- **Kill-switch (fail-closed, R-NET-06) — host nftables.** An **nftables rule on the gateway node** DROPS the
|
||||
steered (`egress-class=vpn`) traffic whenever `tun0` is absent/down — a host backstop **independent of**
|
||||
Cilium egress-gateway fail-open edge cases. This is a *stronger* guarantee than either pod-based option.
|
||||
- **Resilience.** Two egress-gateway nodes + **Cilium Egress Gateway HA** (gateway failover) provide the
|
||||
active/standby the earlier ADRs asked for.
|
||||
|
||||
The user-facing contract is unchanged: services are labeled `egress-class=vpn` (Kyverno-enforced), have **no
|
||||
independent egress path**, and the rules are **enforced outside the service** — now via a Cilium policy + host
|
||||
nft rather than an in-cluster gateway pod.
|
||||
|
||||
### Invariants (NON-NEGOTIABLE — implementation must satisfy all four)
|
||||
The two egress classes have **opposite fail-safe directions**; getting this wrong either leaks VPN traffic or
|
||||
breaks the cluster. These are hard requirements, not preferences:
|
||||
|
||||
1. **Default is the local/private WAN path — for everything.** Every pod (regardless of namespace) AND the
|
||||
nodes themselves egress via the normal private-network/WAN stack by default. Traffic is on the VPN **only**
|
||||
if a service explicitly opts in with `egress-class=vpn`.
|
||||
2. **The VPN tunnel is never a default route.** Not for the node, not for any pod. `tun0` carries **only** the
|
||||
flows a `CiliumEgressGatewayPolicy` explicitly selects (`egress-class=vpn`). The host tunnel is split-tunnel
|
||||
(`route-noexec`/policy routing) precisely to guarantee this.
|
||||
3. **The kill-switch blackhole is scoped to vpn-class only.** When `tun0` is down, the nftables rule drops
|
||||
**only** the `egress-class=vpn` (Cilium-egress-gateway-marked) traffic. It MUST NOT match or drop default/
|
||||
`internet`-class traffic or node traffic. `internet`-class egress is unaffected whether the tunnel is up
|
||||
or down.
|
||||
4. **Opposite fail directions.** Default/`internet` traffic **fails open** (toward the WAN — never blackholed).
|
||||
`egress-class=vpn` traffic **fails closed** (toward blackhole — never leaks to the WAN). A misconfiguration
|
||||
or partial failure must degrade in these directions, never the reverse.
|
||||
|
||||
These are gated by the `egress-class` (default/internet shows WAN IP; vpn shows VPN IP) and `vpn-killswitch`
|
||||
(tunnel down ⇒ vpn-class blackholed, internet-class unaffected) conformance tests.
|
||||
|
||||
## Consequences
|
||||
- Steering is **Cilium-native and declarative** (a `CiliumEgressGatewayPolicy` in Git) and touches no app pods
|
||||
— flipping a service's `egress-class` label re-routes it with no workload rearchitecting.
|
||||
- **Two-layer ownership:** the tunnel is host infra (**Ansible** role on the gateway nodes); the steering
|
||||
policy is **Flux/GitOps**. Consistent with how k3s is provisioned. This is the intended split, not a leak.
|
||||
- **Secrets move host-side:** PIA credentials + `.ovpn` live on the gateway node(s) via **Ansible vault**, not
|
||||
a cluster SOPS Secret — see docs/07. (SOPS still governs in-cluster secrets.)
|
||||
- **Gluetun is dropped** as the gateway. Its validated custom-mode config + the working PIA `.ovpn` transfer
|
||||
directly to the host OpenVPN service, so the connectivity work is retained; only the packaging changes.
|
||||
- **Port forwarding** (PIA NAT-PMP for qBittorrent) must be re-solved host-side (a small pf/natpmp helper that
|
||||
publishes the assigned port to the app) — Phase 6, as before.
|
||||
- **SPOF** on a single gateway node unless run HA (two nodes + Cilium failover).
|
||||
- Validated end-to-end by the **`egress-class`** (R-NET-05/07) and **`vpn-killswitch`** (R-NET-06) conformance
|
||||
tests: vpn-class egress shows the PIA IP, internet-class shows the WAN IP, a git label flip re-routes, and
|
||||
forcing `tun0` down drops vpn-class egress (no leak) while internet-class is unaffected.
|
||||
|
||||
## Alternatives considered
|
||||
- **In-cluster Gluetun pod + Cilium Egress Gateway → its `tun0`** (ADR-0013's deferred option 1) — NON-VIABLE:
|
||||
`hostNetwork` Gluetun hijacks the node default route.
|
||||
- **pod-gateway pattern** (VXLAN overlay + admission webhook + gateway NAT) — the accepted *in-cluster*
|
||||
alternative; works but heavier, mutates client pods, bypasses Cilium's datapath. Rejected for a Cilium
|
||||
cluster in favor of the native node-level egress gateway.
|
||||
- **Per-pod Gluetun sidecar** (ADR-0008 Pattern A) — rejected earlier: privilege colocation with the workload
|
||||
and no declarative east-west policy.
|
||||
- **Keeping the working in-cluster Gluetun pod for its OWN egress only** — retained as the fallback if the
|
||||
node-level tunnel proves troublesome, but it cannot serve *other* pods, which is the whole `egress: vpn`
|
||||
requirement.
|
||||
@@ -0,0 +1,112 @@
|
||||
# ADR-0016 — Public ingress: Cloudflare Tunnel (outbound-only) instead of edge port-forwarding
|
||||
- Status: Proposed (decided by the operator 2026-11-08; set Accepted on merge)
|
||||
- Date: 2026-11-08
|
||||
- Serves: R-DNS-01, R-DNS-02, R-DNS-03, R-DNS-04, R-DNS-05, R-TLS-01, R-TLS-02, R-SEC-02, R-SEC-04
|
||||
- Supersedes: — (amends the **ingress** part of the OQ-1 resolution, "edge does NAT + port-forward
|
||||
only", in [00](../docs/00-vision-and-scope.md), [03](../docs/03-networking-isolation-and-egress.md)
|
||||
and [09](../docs/09-dns-and-ingress.md). [ADR-0002](adr-0002-cni-cilium.md) (Gateway API on Cilium)
|
||||
and [ADR-0006](adr-0006-tls-cert-manager-letsencrypt.md) (cert-manager + Let's Encrypt DNS-01) are
|
||||
unchanged: the Gateway still terminates origin TLS with the LE wildcard.)
|
||||
|
||||
## Context
|
||||
The OQ-1 resolution routed public traffic as: public DNS → WAN IP → **edge router port-forward
|
||||
80/443** → Cilium Gateway VIP. LAN clients were expected to avoid the WAN path through **Pi-hole
|
||||
split-horizon DNS** (R-DNS-03).
|
||||
|
||||
That design fails in this environment:
|
||||
- The ISP router (GFiber) **cannot hairpin NAT**. A LAN client that resolves a service to the WAN IP
|
||||
cannot reach it.
|
||||
- The same router **advertises Google IPv6 DNS servers via Router Advertisement**, and the operator
|
||||
cannot turn that off. LAN clients (Android in particular) therefore use a public resolver
|
||||
alongside, or instead of, Pi-hole. They get the WAN answer and fail.
|
||||
|
||||
So split-horizon DNS cannot be *relied on* to keep LAN clients off the WAN path. Correctness for
|
||||
public web apps has to come from the public answer itself working from the LAN.
|
||||
|
||||
## Decision
|
||||
**Public HTTP(S) ingress goes through a Cloudflare Tunnel.**
|
||||
|
||||
- **cloudflared runs in-cluster** (a Flux-managed workload) and holds an **outbound-only**
|
||||
connection to Cloudflare. No inbound connection to the cluster is needed for public web traffic.
|
||||
- **Public records** for `exposure: public` services are **proxied CNAMEs to
|
||||
`<tunnel-uuid>.cfargotunnel.com`**, managed by **ExternalDNS** from declared service intent
|
||||
(R-DNS-02). No record is hand-edited.
|
||||
- **The edge router's 80/443 forwards are closed** after cutover.
|
||||
- **Hostnames are unchanged:** `<service>.home.keithjackson.dev` and the apex
|
||||
`home.keithjackson.dev` (R-DNS-01, R-C2-01).
|
||||
- **Edge TLS** is Cloudflare's: **Advanced Certificate Manager / Total TLS**, because Universal SSL
|
||||
covers only one subdomain level (`*.keithjackson.dev`) and not `<svc>.home.keithjackson.dev`. The
|
||||
edge certificate is publicly trusted and auto-renewed (R-TLS-01/02).
|
||||
- **Origin TLS stays at the Gateway** with the cert-manager Let's Encrypt wildcard (ADR-0006).
|
||||
cloudflared connects to the Gateway over HTTPS and **validates the origin certificate**, using
|
||||
the service hostname as SNI. It does not skip verification (`noTLSVerify` is not used).
|
||||
- **Only `exposure: public` services have a tunnel route.** The tunnel's ingress rules end in a
|
||||
**catch-all `http_status:404`**, so any other hostname presented at the tunnel is refused
|
||||
(R-DNS-05).
|
||||
- **The Kubernetes API is never reachable through the tunnel**, and is never published by any other
|
||||
path. It is LAN/VPN-only (R-SEC-04).
|
||||
|
||||
**Never published (LAN/VPN-only, `exposure: internal`):** jackett, pihole, semaphore, qbittorrent,
|
||||
radarr, sonarr, **headlamp** (a front end to the Kubernetes API), and the non-HTTP L4 services
|
||||
**unifi** and **crafty-mc** (Minecraft). They get no public DNS record and no tunnel route.
|
||||
|
||||
**Plex stays off the tunnel.** Cloudflare's terms restrict proxying video, so Plex remote streaming
|
||||
keeps using **plex.tv + WAN tcp/32400** (an edge forward that stays). The `plex.home` web UI becomes
|
||||
**LAN-only**: no tunnel route and no proxied record.
|
||||
|
||||
**Unchanged:**
|
||||
- **WireGuard remote access**: udp/51820, reached through the DNS-only (unproxied) `wan.` record,
|
||||
which cloudflare-ddns keeps current. The WAN IP therefore **stays public**. The tunnel does not
|
||||
hide the origin; it only removes 80/443 from the edge.
|
||||
- **Pi-hole keeps serving the LAN**, but split DNS is no longer *required* for public web apps to
|
||||
work. It keeps explicit local records for:
|
||||
- the internal-only names;
|
||||
- the L4 VIPs;
|
||||
- the names that must avoid Cloudflare's **100 MB request-body** and **100 s response-timeout**
|
||||
limits, or must keep working during an internet outage: **gitea**, **nextcloud**, and
|
||||
**keycloak/auth**.
|
||||
- **In-cluster CoreDNS split horizon** and the **node `/etc/hosts` pins** (the OIDC issuer and the
|
||||
Gitea registry) stay. Cluster-internal callers never go through Cloudflare.
|
||||
|
||||
## Alternatives considered
|
||||
- **Keep the port-forward and fix split horizon.** Rejected: the router's RA-advertised Google IPv6
|
||||
DNS cannot be disabled, so some LAN clients will always get the public answer, and the router
|
||||
cannot hairpin it.
|
||||
- **Replace the ISP router (or put a router capable of hairpin NAT and RA control behind it).**
|
||||
Rejected for now: it adds hardware and a non-IaC edge device, which contradicts the "dumb SOHO
|
||||
edge" premise of OQ-1, and the GFiber RA behaviour would still need to be contained. It can be
|
||||
reopened by a future ADR.
|
||||
- **Block or override the Google DNS on the LAN** (firewall DoH/DNS egress, or intercept port 53).
|
||||
Rejected: the SOHO edge has no such controls, and client-side DoH/hard-coded resolvers make it
|
||||
incomplete anyway.
|
||||
- **Tailscale Funnel / ngrok / another reverse-tunnel provider.** Rejected: Cloudflare is already the
|
||||
authoritative DNS and token holder (OQ-4), so its tunnel integrates with ExternalDNS and the
|
||||
existing scoped token with no new vendor.
|
||||
- **Self-hosted reverse tunnel on a VPS** (WireGuard + reverse proxy). Rejected: it adds a host to
|
||||
run, patch and secure, outside the cluster's GitOps model.
|
||||
|
||||
## Consequences
|
||||
- **Positive:** LAN clients reach public web apps correctly whichever resolver answers. No NAT
|
||||
hairpin is needed. Inbound 80/443 is no longer open on the WAN IP. Publishing is an explicit,
|
||||
declared, per-hostname act (tunnel route + record), and anything else 404s at the edge.
|
||||
- **Dependence on Cloudflare.** A Cloudflare or tunnel outage takes public access down. The LAN keeps
|
||||
working through Pi-hole's local records for gitea, nextcloud and keycloak/auth and the
|
||||
internal-only names. Public apps without a local record are unreachable from the LAN during such an
|
||||
outage too. Tunnel health becomes an alert condition (R-OBS-04).
|
||||
- **Cloudflare sees plaintext.** TLS terminates at Cloudflare's edge, so Cloudflare can read the
|
||||
request and response content of every tunnelled service. Re-encryption to the origin protects
|
||||
only the Cloudflare → cluster leg.
|
||||
- **Upload and timeout limits:** 100 MB request bodies and 100 s responses through the proxy. Large
|
||||
uploads (Nextcloud, Gitea pushes and registry blobs) must use the LAN path, which is why those names
|
||||
keep local records.
|
||||
- **Client IP** now arrives as `CF-Connecting-IP` / `X-Forwarded-For` from cloudflared, not as the
|
||||
TCP source. Logging, rate-limiting and any IP-based policy must read those headers, and trust them
|
||||
only from cloudflared.
|
||||
- **The WAN IP stays public** (WireGuard, Plex 32400). The tunnel is not origin cloaking.
|
||||
- **TLS certificates:** two publicly trusted certificates are in play per public name (Cloudflare edge
|
||||
+ LE origin). Both auto-renew. ACM/Total TLS is a paid Cloudflare feature and an operator-held
|
||||
prerequisite.
|
||||
- **Follow-ups (implementation repo):** deploy cloudflared with a SOPS-encrypted tunnel credential;
|
||||
move ExternalDNS public records to proxied CNAMEs for `public` services only; add conformance tests
|
||||
for R-DNS-04/05 and R-SEC-04; alert on tunnel health; then close the 80/443 edge forwards (a
|
||||
documented manual edge step, like the original forwards).
|
||||
@@ -40,7 +40,9 @@ The platform is delivered in two stages:
|
||||
|
||||
## The domain & DNS model
|
||||
|
||||
- Public apex `*.keithjackson.dev` is available and its records resolve to the network's public IP.
|
||||
- Public apex `*.keithjackson.dev` is available in Cloudflare DNS. `public` services under
|
||||
`*.home.keithjackson.dev` are published through a **Cloudflare Tunnel** (proxied records, no inbound
|
||||
80/443 at the edge — ADR-0016); the network's public IP stays in use for WireGuard and Plex only.
|
||||
- All platform services live under `*.home.keithjackson.dev`.
|
||||
- Each service is reachable at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`).
|
||||
- The **root** `home.keithjackson.dev` serves the **Command & Control (C2) center** UI.
|
||||
@@ -117,6 +119,12 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
- **R-DNS-02** — DNS records MUST be managed automatically from declared service intent.
|
||||
- **R-DNS-03** — Internal clients MUST resolve service names correctly (split-horizon where public
|
||||
and internal answers differ).
|
||||
- **R-DNS-04** — `public` HTTP(S) services MUST be reachable from the internet **without inbound
|
||||
80/443 port-forwarding at the edge**, and LAN clients MUST reach them correctly **whichever resolver
|
||||
answers** (public or internal) — no dependency on NAT hairpin (ADR-0016).
|
||||
- **R-DNS-05** — Only services declared `exposure: public` MAY be reachable through the public
|
||||
ingress path. A request for any other hostname at that path MUST be refused (HTTP 404), and
|
||||
`internal` services MUST have no public DNS record (ADR-0016).
|
||||
|
||||
### Observability & alerting — `R-OBS-*`
|
||||
- **R-OBS-01** — Services MUST be monitored (metrics, logs, health).
|
||||
@@ -137,6 +145,25 @@ mandatory; `SHOULD` is strongly preferred and requires an ADR to deviate.
|
||||
- **R-SEC-02** — The platform MUST follow least-privilege for workload identities and network reach.
|
||||
- **R-SEC-03** — All the above properties MUST be demonstrable/auditable (an operator can prove
|
||||
isolation, egress path, cert validity, and backup integrity).
|
||||
- **R-SEC-04** — The Kubernetes API MUST NOT be reachable from the public internet by any path (no
|
||||
tunnel route, no edge forward, no public DNS record); it is LAN/VPN-only (ADR-0016).
|
||||
|
||||
### Identity & access (SSO) — `R-IDP-*`
|
||||
- **R-IDP-01** — The platform MUST provide a central Identity Provider (SSO) as the single user
|
||||
directory and login for platform services.
|
||||
- **R-IDP-02** — Services MUST authenticate against the IdP — natively via OIDC where supported, or
|
||||
behind an identity-aware (forward-auth) proxy otherwise. No management UI may be exposed
|
||||
unauthenticated on the network. (Documented exceptions: Plex/Seerr use Plex auth; Minecraft uses
|
||||
Mojang/Microsoft.)
|
||||
- **R-IDP-03** — Identity configuration (realms, clients, groups, roles) MUST be declarative and
|
||||
GitOps-managed, not console-only.
|
||||
- **R-IDP-04** — IdP secrets (admin credential, OIDC client secrets) MUST be SOPS-encrypted; MFA MUST
|
||||
be available to all users and required for privileged access.
|
||||
- **R-IDP-05** — The IdP MUST be treated as DR-critical (replicated + off-cluster-backed datastore)
|
||||
with a documented **break-glass** path so an IdP outage does not lock the operator out of recovery
|
||||
tools.
|
||||
- **R-IDP-06** — Access MUST be least-privilege via groups/roles; sensitive surfaces (C2, cluster
|
||||
console, alerting) require an authenticated user in an authorized group, not merely "logged in."
|
||||
|
||||
### Environments — `R-ENV-*`
|
||||
- **R-ENV-01** — dev/test MUST be provisioned as local KVMs via IaC: created, baselined, cluster-
|
||||
@@ -187,14 +214,15 @@ implementation.
|
||||
- **OQ-1 (WAN / edge)** — **RESOLVED**: one commercial WAN, no multi-ISP plans; dumb SOHO edge (no
|
||||
VLAN/VPN/services). All segmentation + VPN in-cluster; edge does NAT + port-forward only. Physical
|
||||
multi-WAN out of scope. See [`03`](03-networking-isolation-and-egress.md), [`09`](09-dns-and-ingress.md).
|
||||
**Amended 2026-11-08 (ADR-0016):** public web ingress moved to an outbound-only **Cloudflare Tunnel**;
|
||||
the edge no longer forwards 80/443. Its remaining forwards are WireGuard udp/51820 and Plex tcp/32400.
|
||||
- **OQ-2 (bulk media)** — **RESOLVED (deferred backing)**: Plex bulk media lives **outside the
|
||||
cluster**; the specific backing is **TBD later** and does not block the core platform. Plex's
|
||||
metadata DB still lives on replicated Longhorn. The **off-cluster DR backup target** (for app
|
||||
config, Nextcloud, Gitea, etc.) remains to be chosen at Phase 4. See [`04`](04-storage-and-state.md).
|
||||
- **OQ-3 (VPN)** — **RESOLVED**: **PIA** via Gluetun, cluster-provisioned, **single shared
|
||||
in-cluster tunnel** over **WireGuard** (non-TCP), native port forwarding, rules enforced outside
|
||||
the service (ADR-0013, supersedes ADR-0012's ExpressVPN choice). See
|
||||
[`03`](03-networking-isolation-and-egress.md).
|
||||
- **OQ-3 (VPN)** — **RESOLVED**: **PIA over OpenVPN-UDP** (non-TCP; ADR-0013). Egress is a **node-level
|
||||
host tunnel steered by Cilium Egress Gateway** with a host nftables kill-switch (ADR-0015) — `vpn`-class
|
||||
pods routed through it, rules enforced outside the service. See [`03`](03-networking-isolation-and-egress.md).
|
||||
|
||||
- **OQ-4 (TLS/DNS challenge access)** — **RESOLVED**: a **scoped Cloudflare API token**
|
||||
(`Zone·DNS·Edit`, `keithjackson.dev`), SOPS-encrypted, drives cert-manager DNS-01 and ExternalDNS.
|
||||
|
||||
@@ -15,14 +15,15 @@ choice here without a superseding ADR.**
|
||||
| VM workloads (IaaS) | **KubeVirt** (+ CDI) | R-PLAT-03 | [ADR-0005](../decisions/adr-0005-vm-workloads-kubevirt.md) | [06](06-vm-workloads-kubevirt.md) |
|
||||
| TLS | **cert-manager** + Let's Encrypt (DNS-01) | R-TLS-01..03 | [ADR-0006](../decisions/adr-0006-tls-cert-manager-letsencrypt.md) | [08](08-tls-and-certificates.md) |
|
||||
| Secrets | **SOPS + age** (Flux-native) | R-SEC-01 | [ADR-0007](../decisions/adr-0007-secrets-sops-age.md) | [07](07-security-and-secrets.md) |
|
||||
| VPN egress | **Single shared in-cluster PIA/Gluetun gateway** (WireGuard + native PF) + Cilium Egress Gateway | R-NET-05/06/07 | [ADR-0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md) (supersedes 0012/0008) | [03](03-networking-isolation-and-egress.md) |
|
||||
| VPN egress | **Node-level PIA/OpenVPN tunnel** steered by **Cilium Egress Gateway** (host nftables kill-switch) | R-NET-05/06/07 | [ADR-0015](../decisions/adr-0015-vpn-egress-steering-node-tunnel-cilium.md) (provider [0013](../decisions/adr-0013-vpn-provider-pia-gluetun-wireguard.md)) | [03](03-networking-isolation-and-egress.md) |
|
||||
| Automated updates | **Flux image-update-automation** (images) + **Renovate** (charts) | R-UPD-01..03 | [ADR-0009](../decisions/adr-0009-automated-image-updates.md) | [11](11-automated-updates.md) |
|
||||
| DNS records | **ExternalDNS** (public via Cloudflare) + **PiHole** (internal) | R-DNS-01..03 | — | [09](09-dns-and-ingress.md) |
|
||||
| Ingress (north-south) | **Gateway API** implemented by Cilium | R-DNS-01, R-TLS-01 | [ADR-0002](../decisions/adr-0002-cni-cilium.md) | [09](09-dns-and-ingress.md) |
|
||||
| Observability | **kube-prometheus-stack** + **Loki** + **Grafana Alloy** + Alertmanager→SMTP | R-OBS-01..04 | — | [10](10-observability-and-alerting.md) |
|
||||
| Backup/DR | **Velero** + Longhorn backups → off-cluster S3/NFS | R-STATE-03 | [ADR-0003](../decisions/adr-0003-storage-longhorn.md) | [04](04-storage-and-state.md) |
|
||||
| Node/infra provisioning | **OpenTofu** (libvirt) + **cloud-init** + **Ansible** (k3s bootstrap) | R-ENV-01..03 | [ADR-0001](../decisions/adr-0001-kubernetes-distribution-k3s.md) | [02](02-environments-and-provisioning.md) |
|
||||
| Command center UI | **Homepage** + **Headlamp** + **Grafana** panels | R-C2-01/02 | — | [12](12-command-center-ui.md) |
|
||||
| Identity / SSO | **Keycloak** (OIDC + SAML) + **oauth2-proxy** forward-auth | R-IDP-01..06 | [ADR-0014](../decisions/adr-0014-idp-keycloak.md) | [15](15-identity-and-access.md) |
|
||||
| Command center UI | **Homepage** + **Headlamp** + **Grafana** panels (behind Keycloak SSO) | R-C2-01/02 | — | [12](12-command-center-ui.md) |
|
||||
| Conformance testing | **Kyverno** (L0 policy) + **Chainsaw** (L1/L2/L3 behavior + fault-injection) | R-TEST-01..06 | [ADR-0010](../decisions/adr-0010-conformance-testing.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md) | [14](14-testing-and-conformance.md) |
|
||||
|
||||
## Layer cake
|
||||
@@ -38,13 +39,14 @@ choice here without a superseding ADR.**
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Platform services │
|
||||
│ Flux (GitOps) · cert-manager (TLS) · ExternalDNS · Gateway API │
|
||||
│ Keycloak (SSO/IdP) + oauth2-proxy (forward-auth) │
|
||||
│ kube-prometheus-stack + Loki + Alloy (observability) │
|
||||
│ Velero (backup) · Renovate (chart updates) · SOPS (secrets) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Storage: Longhorn (replicated block) + bulk media store (see ADR-0003) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Network: Cilium (default-deny NetworkPolicy · Egress Gateway · Hubble) │
|
||||
│ VPN egress (Gluetun sidecars / VPN gateway) · Multi-WAN routing │
|
||||
│ VPN egress (node tunnel + Cilium egress GW) · Multi-WAN routing │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Kubernetes: k3s (3 nodes, HA control plane, embedded etcd) │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
@@ -52,8 +54,8 @@ choice here without a superseding ADR.**
|
||||
│ dev/test: libvirt KVMs (OpenTofu + cloud-init + Ansible) │
|
||||
│ prod: 3× mini-PC bare metal, local disks │
|
||||
├──────────────────────────────────────────────────────────────────────────┤
|
||||
│ Edge: router/firewall — multi-WAN uplinks, port-forward to Gateway, │
|
||||
│ VPN uplink(s), split-horizon DNS │
|
||||
│ Edge: SOHO router — single WAN, NAT; forwards only WireGuard + Plex │
|
||||
│ Public web ingress: Cloudflare Tunnel (outbound-only cloudflared, ADR-0016)│
|
||||
└──────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
|
||||
@@ -48,6 +48,15 @@ Never fork the base config per environment.
|
||||
- Nodes are both control-plane and workload (3-node converged cluster); with only 3 nodes we do not
|
||||
isolate control-plane nodes.
|
||||
|
||||
### Cilium install ownership (Ansible bootstrap ↔ Flux)
|
||||
There is a deliberate hand-off between the two control layers: **Ansible installs Cilium at cluster
|
||||
bootstrap** (the CNI must exist before Flux can run any workload), and thereafter the **Flux
|
||||
HelmRelease adopts and manages Cilium's configuration** ([05](05-gitops-and-delivery.md)). This
|
||||
resolves the apparent overlap between "Ansible installs the CNI" (here) and "Flux manages
|
||||
`platform/`" (docs/05). The contract: the bootstrap values and the Flux HelmRelease values MUST stay
|
||||
in **parity** for the settings that define cluster networking — notably `kubeProxyReplacement` and
|
||||
`k8sServiceHost`/`k8sServicePort` — or the two layers will fight over the CNI config.
|
||||
|
||||
### Bare-metal specifics (prod, design-only for now)
|
||||
- Each mini-PC contributes local NVMe/SSD to Longhorn; document the disk layout (OS vs. Longhorn
|
||||
data device) before prod build.
|
||||
|
||||
@@ -35,6 +35,13 @@ Cilium replaces the default k3s CNI and provides:
|
||||
- **Platform namespace(s)** (Flux, cert-manager, monitoring, storage) are isolated from application
|
||||
namespaces; only the specific flows they need (e.g. metrics scraping) are opened.
|
||||
|
||||
**Platform label-key convention.** The spec fixes label *values* (e.g. `internet|vpn`,
|
||||
`oidc|forward-auth|native|none`); the label *keys* the platform uses are:
|
||||
`homelab.keithjackson.dev/egress-class`, `homelab.keithjackson.dev/auth-mode`, and the supporting
|
||||
`homelab.keithjackson.dev/{app-namespace,baseline}`. These keys are the contract that network
|
||||
policies, Kyverno rules, and renderers key off (values per this doc, [13](13-service-catalog.md),
|
||||
and [15](15-identity-and-access.md)).
|
||||
|
||||
### Default-deny baseline (R-NET-01)
|
||||
Every namespace gets a baseline policy set applied by GitOps at namespace creation:
|
||||
- **Deny all ingress** and **deny all egress** by default.
|
||||
@@ -75,45 +82,61 @@ as multi-ISP routing but as **two egress paths selected by routing rules**:
|
||||
route across; the `internet` path is simply the default route out.
|
||||
- **The edge router is a dumb SOHO device** — no VLANs, no VPN, no services, no API/IaC. Therefore
|
||||
**all segmentation and all VPN handling live in-cluster** (Cilium policy + the VPN gateway), not on
|
||||
the edge. The edge's only jobs are NAT out and **port-forwarding 80/443 (and specific service
|
||||
ports) to the ingress VIP** for `public` services — a documented manual step ([09](09-dns-and-ingress.md)).
|
||||
the edge. The edge's only jobs are NAT out and a few **manual port-forwards that are not web
|
||||
ingress**: WireGuard udp/51820 and Plex tcp/32400 ([09](09-dns-and-ingress.md)). Public web ingress
|
||||
uses an outbound-only **Cloudflare Tunnel** from in-cluster cloudflared (ADR-0016), so the edge does
|
||||
not forward 80/443.
|
||||
- The **service-facing contract is unchanged**: a service still just declares `egress: internet|vpn`.
|
||||
|
||||
## VPN egress model: single shared PIA/Gluetun gateway (R-NET-05/06 — ADR-0013, supersedes ADR-0012/0008)
|
||||
## VPN egress model: node-level tunnel + Cilium Egress Gateway (R-NET-05/06 — ADR-0015; provider ADR-0013)
|
||||
|
||||
VPN-class services: **sonarr, radarr, jackett, qbittorrent**. Per OQ-3 the design is a **single,
|
||||
shared VPN connection** — not a per-service sidecar — so that **network rules are enforced outside
|
||||
the service**:
|
||||
the service**. The steering mechanism was settled by **ADR-0015** after the original in-cluster-Gluetun
|
||||
approach proved un-steerable (a Gluetun *pod*'s `tun0` is invisible to the node datapath, and
|
||||
`hostNetwork` Gluetun hijacks the node's own default route).
|
||||
|
||||
- **One in-cluster VPN egress gateway** (an HA-capable Deployment) runs **Gluetun** configured for
|
||||
**PIA over WireGuard** (UDP — "non-TCP or better"). The cluster provisions and controls the tunnel
|
||||
(nothing on the edge router). Gluetun provides the built-in **kill-switch/firewall**, health
|
||||
checks, and auto-reconnect.
|
||||
- **`vpn`-class pods are steered through the gateway** (Cilium Egress Gateway / routing); they have
|
||||
**no independent egress path**. The kill-switch and allow rules are enforced at the **gateway +
|
||||
namespace/policy level**, i.e. *outside* each service — the service is just labeled `egress: vpn`
|
||||
and cannot leak on its own.
|
||||
- **Kill-switch (fail-closed) is mandatory and enforced centrally:** if the tunnel is down, `vpn`
|
||||
pods have **no route to the plain internet**; only the tunnel path exists. `internet`-class
|
||||
services are unaffected.
|
||||
- **Port forwarding for qBittorrent:** Gluetun **native PF** (`VPN_PORT_FORWARDING=on`) on a
|
||||
**PF-enabled PIA region**; a **port-sync helper** (e.g. `qbittorrent-natmap`) writes the assigned
|
||||
port into qBittorrent so it tracks reconnects — kept hands-off. Alert if PF acquisition fails.
|
||||
- **Trade-off (documented):** one shared tunnel = one shared egress IP and a single failure domain
|
||||
for `vpn` services → run the gateway **active/standby** for resilience; alert on tunnel state
|
||||
([10](10-observability-and-alerting.md)).
|
||||
- **Prerequisite (simple):** just the **PIA username/password**, stored SOPS-encrypted
|
||||
([07](07-security-and-secrets.md)) — no manual `.ovpn` extraction. Validate end-to-end (tunnel +
|
||||
kill-switch + PF) in dev/test.
|
||||
**Default is the WAN; the VPN is strictly opt-in.** Every pod — and the nodes themselves — egress the
|
||||
local/private WAN by default and **never** the VPN. Only services labeled `egress-class=vpn` are steered to
|
||||
the tunnel, and the kill-switch blackholes **only** that class. The two classes fail in **opposite**
|
||||
directions: `internet` fails **open** (WAN, never blackholed); `vpn` fails **closed** (blackhole, never leaks
|
||||
to WAN). See the ADR-0015 invariants — implementation must not route the default through the VPN or blackhole
|
||||
the wrong class.
|
||||
|
||||
The model:
|
||||
|
||||
- **Tunnel on the node, not in a pod.** One or two labeled **egress-gateway nodes** run a **host OpenVPN
|
||||
systemd service** — **PIA over OpenVPN-UDP** (provider/protocol per ADR-0013; "non-TCP or better"),
|
||||
**Ansible-managed host infra** (like k3s itself), NOT a Flux workload. It is **split-tunnel**: it
|
||||
creates `tun0` but does **not** set the node's default route (`route-noexec` + policy routing), so the
|
||||
node's own traffic stays on the WAN — only what Cilium points at `tun0` uses the VPN. The tunnel is
|
||||
on the cluster nodes, not the edge router (dumb SOHO edge, per OQ-1).
|
||||
- **`vpn`-class pods are steered by Cilium Egress Gateway.** A `CiliumEgressGatewayPolicy` selects pods
|
||||
labeled `egress-class=vpn` and egresses them via the gateway node's `tun0` (native eBPF SNAT to the
|
||||
tunnel IP). **No client-pod changes, no sidecar, no overlay.** They have **no independent egress path**;
|
||||
the service is just labeled `egress: vpn` and cannot leak on its own.
|
||||
- **Kill-switch (fail-closed) — host nftables.** An **nftables rule on the gateway node** DROPS steered
|
||||
(`egress-class=vpn`) traffic whenever `tun0` is down — a host backstop independent of any Cilium
|
||||
egress-gateway fail-open edge case. If the tunnel is down, `vpn` pods have **no route to the plain
|
||||
internet**; `internet`-class services are unaffected.
|
||||
- **Port forwarding for qBittorrent:** PIA NAT-PMP, re-solved **host-side** (a small pf/natpmp helper on
|
||||
the gateway node that publishes the assigned forwarded port into qBittorrent) — kept hands-off. Alert
|
||||
if PF acquisition fails.
|
||||
- **Resilience:** two egress-gateway nodes + **Cilium Egress Gateway HA** (gateway failover) give the
|
||||
active/standby resilience; alert on tunnel state ([10](10-observability-and-alerting.md)).
|
||||
- **Prerequisite:** the **PIA username/password + `.ovpn`** on the gateway node(s) via **Ansible vault**
|
||||
([07](07-security-and-secrets.md)) — the tunnel is host infra, so its creds are host-side (not a cluster
|
||||
SOPS Secret). Validate end-to-end (tunnel + steering + kill-switch + PF) in dev/test.
|
||||
|
||||
**Kill-switch is an acceptance gate:** with the tunnel forcibly down, every `vpn` service must fail
|
||||
closed (no packets to the plain internet). Tested, not assumed.
|
||||
closed (no packets to the plain internet), while `internet`-class is unaffected. Tested, not assumed.
|
||||
|
||||
## Ingress (north-south) — summary
|
||||
|
||||
North-south exposure is handled by **Gateway API implemented by Cilium**, with TLS from
|
||||
cert-manager and DNS from ExternalDNS. Detailed in [`09-dns-and-ingress.md`](09-dns-and-ingress.md).
|
||||
Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forwarded at the edge).
|
||||
Exposure is a declared property: `internal` (LAN/VPN-only) vs `public` (published through the
|
||||
Cloudflare Tunnel, ADR-0016).
|
||||
|
||||
## Observability of the network (supports R-SEC-03, R-OBS-04)
|
||||
- **Hubble** provides flow visibility to verify default-deny and troubleshoot allow rules.
|
||||
@@ -131,8 +154,10 @@ Exposure is a declared property: `internal` (LAN-only) vs `public` (port-forward
|
||||
## Resolved
|
||||
- **OQ-1** — one commercial WAN, no multi-ISP plans; dumb SOHO edge (no VLAN/VPN/services). All
|
||||
segmentation + VPN in-cluster; edge does NAT + port-forward only. Multi-ISP routing out of scope.
|
||||
- **OQ-3** — **PIA** via Gluetun, single shared in-cluster tunnel over **WireGuard**, native port
|
||||
forwarding, rules enforced outside the service (ADR-0013, supersedes ADR-0012's ExpressVPN choice).
|
||||
Amended 2026-11-08 (ADR-0016): 80/443 are no longer forwarded; public web ingress is a Cloudflare
|
||||
Tunnel.
|
||||
- **OQ-3** — **PIA over OpenVPN-UDP** (ADR-0013); egress is a **node-level host tunnel** steered by **Cilium
|
||||
Egress Gateway** with a host nftables kill-switch (ADR-0015), rules enforced outside the service.
|
||||
|
||||
## Open questions
|
||||
- Exact steering mechanism for `vpn` pods → gateway (Cilium Egress Gateway vs routing/SNAT) —
|
||||
|
||||
@@ -16,12 +16,23 @@ Defense in depth with declarative, auditable controls:
|
||||
- **TLS**: trusted, auto-managed certificates end to end ([08](08-tls-and-certificates.md), R-TLS-*).
|
||||
- **Supply chain**: pin images by digest where automation allows; prefer known-good upstreams;
|
||||
updates flow through policy, not ad-hoc pulls ([11](11-automated-updates.md)).
|
||||
- **Identity / SSO**: a central IdP (**Keycloak**) provides single sign-on, one user directory,
|
||||
group-based authorization, and MFA; services authenticate via native OIDC or an oauth2-proxy
|
||||
forward-auth layer so no management UI is unauthenticated ([15](15-identity-and-access.md),
|
||||
[ADR-0014](../decisions/adr-0014-idp-keycloak.md), R-IDP-*). This is the primary *authentication*
|
||||
control, complementing network default-deny (the primary *isolation* control).
|
||||
- **Policy enforcement (admission)**: **Kyverno** runs in-cluster as an admission controller and
|
||||
rejects non-conforming resources at reconcile time — e.g. a namespace missing its default-deny
|
||||
baseline, a plaintext Secret, a workload without a required egress-class label. The *same* policies
|
||||
baseline, or a workload without a required egress-class/auth-mode label. The *same* policies
|
||||
gate CI ([14](14-testing-and-conformance.md), [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)),
|
||||
so invariants are enforced both before merge and at admission (defense in depth). Enforcement
|
||||
complements — never replaces — the runtime behavior tests.
|
||||
- **Plaintext-secret enforcement is split by layer (do not conflate):** the "no plaintext Secret"
|
||||
invariant is a **CI-time** check (the committed Secret must carry SOPS ciphertext). It **cannot**
|
||||
be an admission rule, because Flux **decrypts SOPS before** the object reaches the API server —
|
||||
an admitted Secret never carries ciphertext, so an admission "reject plaintext" rule would reject
|
||||
*every* real Secret. Admission instead enforces **SOPS provenance** (the Secret was reconciled
|
||||
through the Flux/SOPS path), not ciphertext presence.
|
||||
- **Auditability**: isolation, egress path, cert validity, and backup integrity are all
|
||||
demonstrable (R-SEC-03) via Hubble, cert status, and DR drills.
|
||||
|
||||
@@ -36,8 +47,11 @@ Defense in depth with declarative, auditable controls:
|
||||
[AGENTS.md §5](../AGENTS.md)).
|
||||
|
||||
### Secret categories to manage
|
||||
- VPN credentials (PIA username/password; WireGuard) for the egress model (ADR-0013).
|
||||
- VPN credentials (PIA username/password + `.ovpn`; OpenVPN-UDP) — held **host-side on the egress-gateway
|
||||
node(s) via Ansible vault** (the tunnel is host infra, not a cluster pod; ADR-0015), NOT a cluster SOPS Secret.
|
||||
- Keycloak admin credential and per-service **OIDC client secrets** ([15](15-identity-and-access.md)).
|
||||
- DNS provider API token (Cloudflare) for ACME DNS-01 + ExternalDNS.
|
||||
- Cloudflare Tunnel credential/token for in-cluster cloudflared (SOPS-encrypted; ADR-0016).
|
||||
- SMTP credentials for Alertmanager email.
|
||||
- Service credentials (Nextcloud, Gitea admin, UniFi, Semaphore, Kasm, DB passwords).
|
||||
- Off-cluster backup target credentials (S3/NFS) for Velero/Longhorn.
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# 08 — TLS & Certificates
|
||||
|
||||
Serves: **R-TLS-01/02/03**, **R-DNS-01**. Related: [ADR-0006 (cert-manager + Let's Encrypt)](../decisions/adr-0006-tls-cert-manager-letsencrypt.md).
|
||||
Serves: **R-TLS-01/02/03**, **R-DNS-01**. Related: [ADR-0006 (cert-manager + Let's Encrypt)](../decisions/adr-0006-tls-cert-manager-letsencrypt.md),
|
||||
[ADR-0016 (Cloudflare Tunnel public ingress)](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md).
|
||||
|
||||
## Requirement recap
|
||||
|
||||
@@ -30,8 +31,8 @@ services (R-TLS-01/02/03).
|
||||
|
||||
## DNS-01 challenge access (OQ-4 — RESOLVED: scoped Cloudflare API token)
|
||||
|
||||
Cloudflare already has a wildcard `*.keithjackson.dev` → public-IP record, which covers
|
||||
*reachability* but not certificate issuance (ACME DNS-01 must **write `_acme-challenge` TXT records**,
|
||||
Cloudflare already has a wildcard `*.keithjackson.dev` → public-IP record, which does not cover
|
||||
certificate issuance (ACME DNS-01 must **write `_acme-challenge` TXT records**,
|
||||
and that wildcard does not match two-label-deep names like `plex.home.keithjackson.dev`).
|
||||
|
||||
**Decision (OQ-4):** use a **scoped Cloudflare API token** — `Zone·DNS·Edit` limited to the
|
||||
@@ -55,8 +56,15 @@ DNS edit only) so its blast radius is bounded (R-SEC-02). Rotation = re-encrypt
|
||||
## Where TLS terminates
|
||||
|
||||
- **North-south**: at the **Gateway API** listener (implemented by Cilium, [09](09-dns-and-ingress.md)),
|
||||
using the cert-manager-issued cert. Public services are reached via the edge router forwarding to
|
||||
the Gateway; internal services resolve to the Gateway on the LAN.
|
||||
using the cert-manager-issued cert. Internal services resolve to the Gateway on the LAN.
|
||||
- **Public services via the Cloudflare Tunnel (ADR-0016)** are TLS twice:
|
||||
- **Edge:** Cloudflare terminates the client's TLS with an ACM/Total TLS certificate. It is
|
||||
publicly trusted and auto-renewed (R-TLS-01/02). Universal SSL is not enough, because it covers
|
||||
only one subdomain level.
|
||||
- **Origin:** cloudflared re-encrypts to the Gateway and **validates** the Let's Encrypt wildcard,
|
||||
using the service hostname as SNI. Verification is never disabled.
|
||||
|
||||
Cloudflare sees plaintext between the two legs, a trade-off accepted in ADR-0016.
|
||||
- Internal service-to-service traffic relies primarily on network policy for security; mTLS between
|
||||
workloads is **optional** future hardening (could be added via Cilium/service mesh) and is not a
|
||||
launch requirement — record as a possible future ADR.
|
||||
|
||||
+73
-24
@@ -1,14 +1,16 @@
|
||||
# 09 — DNS & Ingress
|
||||
|
||||
Serves: **R-DNS-01/02/03**, **R-C2-01**, works with **R-TLS-***. Related:
|
||||
[ADR-0002 (Cilium/Gateway API)](../decisions/adr-0002-cni-cilium.md).
|
||||
Serves: **R-DNS-01/02/03/04/05**, **R-C2-01**, **R-SEC-04**, works with **R-TLS-***. Related:
|
||||
[ADR-0002 (Cilium/Gateway API)](../decisions/adr-0002-cni-cilium.md),
|
||||
[ADR-0016 (Cloudflare Tunnel public ingress)](../decisions/adr-0016-public-ingress-cloudflare-tunnel.md).
|
||||
|
||||
## Naming model
|
||||
|
||||
- All services live at `<service>.home.keithjackson.dev` (e.g. `plex.home.keithjackson.dev`).
|
||||
- The **root** `home.keithjackson.dev` serves the **Command & Control UI** ([12](12-command-center-ui.md)).
|
||||
- Public apex `*.keithjackson.dev` resolves to the network's public IP; `*.home.keithjackson.dev`
|
||||
is the platform's zone.
|
||||
- `*.home.keithjackson.dev` is the platform's zone, in Cloudflare DNS. `public` services are
|
||||
published as **proxied CNAMEs to the Cloudflare Tunnel** (`<tunnel-uuid>.cfargotunnel.com`,
|
||||
ADR-0016), not as records pointing at the network's public IP.
|
||||
|
||||
## DNS automation (R-DNS-02)
|
||||
|
||||
@@ -16,16 +18,32 @@ Serves: **R-DNS-01/02/03**, **R-C2-01**, works with **R-TLS-***. Related:
|
||||
authoritative DNS provider (Cloudflare, per OQ-4) automatically — a new service's DNS appears from
|
||||
its declared intent, no manual record editing.
|
||||
- Records for `*.home.keithjackson.dev` are created/updated/removed as services come and go.
|
||||
- Public records are created **only for `exposure: public` services** (R-DNS-05). The DNS-only
|
||||
(unproxied) `wan.` record for WireGuard is kept current by cloudflare-ddns, outside this flow.
|
||||
|
||||
## Split-horizon resolution (R-DNS-03)
|
||||
|
||||
Internal and external answers differ:
|
||||
- **External** (`*.home.keithjackson.dev` from the internet): resolves to the public IP; reaches
|
||||
the edge router, which forwards `public`-class services to the in-cluster Gateway.
|
||||
- **Internal** (LAN clients): resolves `<service>.home.keithjackson.dev` to the **internal Gateway
|
||||
address** directly, avoiding hairpinning through the public IP.
|
||||
- **PiHole** (itself an onboarded service) provides internal LAN DNS and the split-horizon answers
|
||||
for `.home.keithjackson.dev`; the trusted Let's Encrypt cert is served either way ([08](08-tls-and-certificates.md)).
|
||||
- **External** (`<service>.home.keithjackson.dev` from the internet): a `public` service resolves to
|
||||
**Cloudflare's edge** (proxied CNAME to the tunnel). Cloudflare forwards the request down the
|
||||
outbound-only tunnel to cloudflared in the cluster, which forwards it to the Gateway (ADR-0016).
|
||||
`internal` services have no public answer.
|
||||
- **Internal** (LAN clients): Pi-hole answers with the **internal Gateway address** (or the service's
|
||||
L4 VIP) for the names it holds local records for.
|
||||
- **Correctness does not depend on split horizon for `public` web apps** (R-DNS-04). The ISP router
|
||||
cannot hairpin NAT and advertises a public IPv6 resolver via RA, so some LAN clients will get the
|
||||
public answer. That answer goes through Cloudflare and works from the LAN too.
|
||||
- **PiHole** (itself an onboarded service) provides internal LAN DNS. It keeps explicit local records
|
||||
for:
|
||||
- the `internal` names;
|
||||
- the L4 VIPs;
|
||||
- the names that must avoid Cloudflare's 100 MB body / 100 s timeout limits or must work during an
|
||||
internet outage: **gitea**, **nextcloud**, **keycloak/auth**.
|
||||
|
||||
A trusted certificate is served either way: Cloudflare's edge certificate on the tunnel path, the
|
||||
Let's Encrypt wildcard on the LAN path ([08](08-tls-and-certificates.md)).
|
||||
- **In-cluster** resolution keeps its own split horizon (CoreDNS) and the node `/etc/hosts` pins (OIDC
|
||||
issuer, Gitea registry), so cluster-internal callers never traverse Cloudflare.
|
||||
|
||||
> DNS is thus both a platform capability *and* a hosted service. Bootstrapping order matters: the
|
||||
> platform needs working DNS before PiHole is fully managed — handled by the bring-up sequence in
|
||||
@@ -40,19 +58,41 @@ Internal and external answers differ:
|
||||
- Each service contributes an **HTTPRoute** mapping its hostname to its Service — rendered from the
|
||||
ServiceDefinition, not hand-authored.
|
||||
- **Exposure class** is declared per service:
|
||||
- `internal` — reachable only on the LAN (no edge port-forward; internal DNS only).
|
||||
- `public` — additionally reachable from the internet via the edge router forwarding 80/443 to the
|
||||
Gateway.
|
||||
- `internal` — reachable only on the LAN/VPN (no tunnel route, no public record; internal DNS only).
|
||||
- `public` — additionally reachable from the internet through the **Cloudflare Tunnel**:
|
||||
cloudflared → the Gateway over HTTPS (ADR-0016).
|
||||
- **Never `public`:** jackett, pihole, semaphore, qbittorrent, radarr, sonarr, **headlamp** (a front
|
||||
end to the Kubernetes API), plex's web UI, and the non-HTTP L4 services unifi and crafty-mc.
|
||||
- **The Kubernetes API is never published** — no tunnel route, no edge forward, no public record
|
||||
(R-SEC-04).
|
||||
|
||||
## Public ingress: Cloudflare Tunnel (ADR-0016)
|
||||
|
||||
- **cloudflared** runs in-cluster as a Flux-managed workload and holds an **outbound-only**
|
||||
connection to Cloudflare. No inbound 80/443 reaches the WAN IP.
|
||||
- Its ingress rules list one route per `public` hostname → the Gateway (HTTPS, origin certificate
|
||||
validated with the hostname as SNI). The rules end in a **catch-all `http_status:404`**, so an
|
||||
unlisted hostname is refused at the tunnel (R-DNS-05).
|
||||
- **Edge TLS** is Cloudflare ACM/Total TLS (Universal SSL does not cover two-label-deep names);
|
||||
**origin TLS** is the Let's Encrypt wildcard at the Gateway ([08](08-tls-and-certificates.md)).
|
||||
- **Limits and trade-offs:**
|
||||
- 100 MB request bodies and 100 s responses through the proxy;
|
||||
- Cloudflare terminates TLS and sees plaintext;
|
||||
- a Cloudflare outage takes public access down (the LAN keeps working through Pi-hole's local
|
||||
records);
|
||||
- the client IP arrives as `CF-Connecting-IP` / `X-Forwarded-For`, not as the TCP source.
|
||||
|
||||
## Edge integration (single SOHO router — OQ-1 resolved)
|
||||
|
||||
- The environment has **one SOHO edge router**: no VLANs, no VPN, no services, no API/IaC. Its only
|
||||
ingress job is **manual port-forwarding of 80/443 (and specific service ports, e.g. Plex 32400,
|
||||
Minecraft 25565) to the Gateway VIP** for `public` services. This is a documented manual edge step
|
||||
(the SOHO device has no automation) — the analogue of the old pfSense manual-config exception.
|
||||
Confirm the router can port-forward (basic NAT; almost all can).
|
||||
- The environment has **one SOHO edge router**: no VLANs, no VPN, no services, no API/IaC. It
|
||||
**does not forward 80/443** (public web ingress is the tunnel, ADR-0016). Its only remaining
|
||||
inbound forwards are **WireGuard udp/51820** (remote access, via the DNS-only `wan.` record) and
|
||||
**Plex tcp/32400** (remote streaming via plex.tv; Cloudflare's terms restrict proxied video). This
|
||||
is a documented manual edge step (the SOHO device has no automation), the analogue of the old
|
||||
pfSense manual-config exception. The WAN IP therefore stays public.
|
||||
- No multi-WAN inbound rules — single WAN ([03](03-networking-isolation-and-egress.md)).
|
||||
- `internal` services are never port-forwarded; they rely on internal DNS + Gateway on the LAN.
|
||||
- `internal` services are never port-forwarded or tunnelled; they rely on internal DNS + Gateway (or
|
||||
their L4 VIP) on the LAN, or WireGuard from outside.
|
||||
- **Ingress VIP mechanism — resolved: Cilium L2 announcement (ARP/NDP)** on the flat LAN. BGP is not
|
||||
available on a SOHO router, and a single flat subnet makes L2 announcement the natural choice.
|
||||
|
||||
@@ -62,9 +102,10 @@ A wildcard `*.keithjackson.dev` matches only **one** label deep — it covers `h
|
||||
but **not** `plex.home.keithjackson.dev`. Consequences:
|
||||
- **`internal` services need no public record** — PiHole resolves `<svc>.home.keithjackson.dev` to
|
||||
the internal Gateway VIP (split-horizon). This is the majority of services.
|
||||
- **`public` services** need a public record for `<svc>.home.keithjackson.dev`. Per OQ-4 (resolved),
|
||||
**ExternalDNS manages these automatically** via the scoped Cloudflare API token — no manual record
|
||||
editing (see [08](08-tls-and-certificates.md)).
|
||||
- **`public` services** need a public record for `<svc>.home.keithjackson.dev`: a **proxied CNAME to
|
||||
the tunnel** (ADR-0016). Per OQ-4 (resolved), **ExternalDNS manages these automatically** via the
|
||||
scoped Cloudflare API token — no manual record editing (see [08](08-tls-and-certificates.md)). The
|
||||
same depth limit is why edge TLS needs ACM/Total TLS rather than Universal SSL.
|
||||
|
||||
## Interaction with default-deny
|
||||
|
||||
@@ -76,10 +117,18 @@ not let other in-cluster services reach it unless an explicit allow rule exists.
|
||||
- A new service's `<service>.home.keithjackson.dev` record appears automatically and routes to it.
|
||||
- `internal` services are unreachable from the public internet; `public` services are reachable and
|
||||
TLS-valid.
|
||||
- Internal clients resolve to the internal Gateway (no public hairpin) and get a trusted cert.
|
||||
- Internal clients resolving through Pi-hole reach the internal Gateway for the names it holds local
|
||||
records for, and get a trusted cert.
|
||||
- A LAN client resolving a `public` name through a **public** resolver reaches it, TLS-valid
|
||||
(R-DNS-04). The WAN IP does not answer on 80/443.
|
||||
- A request at the tunnel for any hostname that is not a `public` service returns **404**, and no
|
||||
`internal` service has a public DNS record (R-DNS-05).
|
||||
- The Kubernetes API is unreachable from the internet: no tunnel route, no edge forward, no public
|
||||
record (R-SEC-04).
|
||||
- The root `home.keithjackson.dev` serves the C2 UI.
|
||||
|
||||
## Resolved
|
||||
- **OQ-1** — single SOHO edge (manual port-forward only); ingress VIP via **Cilium L2 announcement**.
|
||||
- **OQ-1** — single SOHO edge; ingress VIP via **Cilium L2 announcement**. Public web ingress amended
|
||||
2026-11-08 to a **Cloudflare Tunnel** (ADR-0016); the edge keeps only udp/51820 and tcp/32400.
|
||||
- **OQ-4** — scoped **Cloudflare API token** drives ExternalDNS (records) and cert-manager DNS-01
|
||||
(certs) automatically; see [08](08-tls-and-certificates.md).
|
||||
@@ -27,6 +27,8 @@ Platform health (R-OBS-04) — first-class alerts:
|
||||
NetworkPolicy reconcile errors.
|
||||
- **Backups (Velero/Longhorn)**: backup job failure, missed schedule, restore-test failures.
|
||||
- **KubeVirt**: VM crash-loops / failed launches.
|
||||
- **Identity (Keycloak)**: IdP down/unhealthy or its DB unavailable (SSO is a critical dependency —
|
||||
[15](15-identity-and-access.md)); certificate/login-endpoint failures.
|
||||
|
||||
## Alert delivery (R-OBS-02)
|
||||
|
||||
|
||||
@@ -49,8 +49,10 @@ kubeconfig in a browser:
|
||||
|
||||
- The C2 root and its sub-UIs are `internal` exposure by default (LAN-only) unless the operator
|
||||
explicitly makes a component `public`.
|
||||
- Authentication in front of the C2 (and sensitive sub-UIs like Headlamp) — SSO/reverse-proxy auth —
|
||||
is required; the mechanism (e.g. an identity-aware proxy) is pinned during the C2 workstream.
|
||||
- Authentication is via **Keycloak SSO** ([15](15-identity-and-access.md), [ADR-0014](../decisions/adr-0014-idp-keycloak.md)):
|
||||
Headlamp/Grafana use **native OIDC**; UIs without OIDC (Homepage, Prometheus, Alertmanager, Hubble)
|
||||
sit behind the **oauth2-proxy forward-auth** at the Gateway. Access to the C2 and sensitive
|
||||
surfaces requires an authenticated user **in an authorized group** with **MFA** (R-IDP-06).
|
||||
- Trusted TLS on the root and all sub-UIs ([08](08-tls-and-certificates.md)).
|
||||
|
||||
## Acceptance gates
|
||||
@@ -59,6 +61,9 @@ kubeconfig in a browser:
|
||||
- Platform health (nodes, storage, Flux, certs, backups, VPN) is visible without leaving the C2.
|
||||
- Safe operational actions work and are access-controlled + audited; destructive actions confirm.
|
||||
|
||||
## Resolved
|
||||
- Auth mechanism — **Keycloak SSO** (native OIDC + oauth2-proxy forward-auth), [15](15-identity-and-access.md) /
|
||||
[ADR-0014](../decisions/adr-0014-idp-keycloak.md).
|
||||
|
||||
## Open questions
|
||||
- Auth mechanism for the C2 (identity-aware proxy vs per-app auth) — pin in the C2 workstream.
|
||||
- Custom portal vs. Homepage-as-root long term (ADR if we diverge from Homepage).
|
||||
@@ -19,7 +19,7 @@ kind: container | vm # PaaS (default) or IaaS via KubeVirt (R-PLAT
|
||||
source:
|
||||
image: <repo/image> # container: image ref
|
||||
# or vmTemplate: <image/source> # vm: CDI source image
|
||||
exposure: internal | public # LAN-only vs edge port-forwarded (R-DNS/R-NET)
|
||||
exposure: internal | public # LAN/VPN-only vs Cloudflare Tunnel (R-DNS-04/05, ADR-0016)
|
||||
egress: internet | vpn # egress class (R-NET-05); vpn ⇒ kill-switch (R-NET-06)
|
||||
storage:
|
||||
- name: config
|
||||
@@ -34,6 +34,13 @@ dependencies: # explicit allow rules; everything else denie
|
||||
ingressRoutes: # hostnames/paths to expose (rendered to HTTPRoute + TLS + DNS)
|
||||
- host: <name>.home.keithjackson.dev
|
||||
port: <servicePort>
|
||||
auth: # SSO integration (R-IDP-02; see 15-identity-and-access.md)
|
||||
mode: oidc | forward-auth | native | none
|
||||
# oidc → app configured as a Keycloak OIDC client (does its own login)
|
||||
# forward-auth → protected by oauth2-proxy → Keycloak at the Gateway (for UIs w/o OIDC)
|
||||
# native → app manages its own auth (e.g. Plex/Seerr via Plex) — documented exception
|
||||
# none → no auth (internal-only, discouraged)
|
||||
groups: [ <keycloak-group>, ... ] # groups allowed access (R-IDP-06)
|
||||
update:
|
||||
channel: stable # never pre-release unless opted in (R-UPD-01)
|
||||
range: <semver range | tag regex># acceptable versions
|
||||
@@ -53,10 +60,12 @@ backup:
|
||||
## Default posture for every service (unless overridden)
|
||||
- `kind: container`, `exposure: internal`, `egress: internet`.
|
||||
- **Default-deny**: no `dependencies` ⇒ can reach only DNS ([03](03-networking-isolation-and-egress.md)).
|
||||
- **Auth default `forward-auth`**: any exposed UI is protected by Keycloak SSO unless it declares
|
||||
native OIDC (`oidc`) or a documented exception (`native`) ([15](15-identity-and-access.md), R-IDP-02).
|
||||
- Longhorn-backed config PVC; off-cluster backup; trusted TLS; auto-DNS; monitored; auto-updated on
|
||||
`stable`.
|
||||
|
||||
## The initial catalog (13 services)
|
||||
## The initial catalog (14 services)
|
||||
|
||||
Egress and dependencies below are **normative intent**; implementers render the exact policies.
|
||||
"Bulk" = large media tier ([04](04-storage-and-state.md)). All get trusted TLS + auto-DNS + monitoring
|
||||
@@ -64,7 +73,8 @@ Egress and dependencies below are **normative intent**; implementers render the
|
||||
|
||||
| # | Service | kind | exposure | egress | Storage | Key dependencies (explicit allows) | Notes |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| 1 | **plex** | container | public | **internet** | config: longhorn (small, replicated); media: **bulk** | — (clients reach it; Seerr → Plex) | Largest service; media on bulk tier, metadata DB on Longhorn ([04](04-storage-and-state.md)) |
|
||||
| 0 | **keycloak** | container | public | internet | db (PostgreSQL): longhorn (irreplaceable → backed up) | → its DB; ← all SSO clients + oauth2-proxy | **IdP/SSO** ([15](15-identity-and-access.md)); **foundational**, onboarded **early**; DR-critical + break-glass path; hardened public exposure |
|
||||
| 1 | **plex** | container | public | **internet** | config: longhorn (small, replicated); media: **bulk** | — (clients reach it; Seerr → Plex) | Largest service; media on bulk tier, metadata DB on Longhorn ([04](04-storage-and-state.md)); auth: **native** (Plex) |
|
||||
| 2 | **sonarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch (R-NET-06) |
|
||||
| 3 | **radarr** | container | internal | **vpn** | config: longhorn; media/downloads: bulk | → qbittorrent, → jackett; ← seerr | VPN + kill-switch |
|
||||
| 4 | **jackett** | container | internal | **vpn** | config: longhorn | ← sonarr, ← radarr | VPN + kill-switch (indexer) |
|
||||
@@ -90,17 +100,48 @@ radarr → qbittorrent, jackett
|
||||
(everything → DNS only, by default)
|
||||
```
|
||||
|
||||
**SSO integration (`auth.mode`, R-IDP-02; see [15](15-identity-and-access.md)):**
|
||||
- **oidc** (native Keycloak client): gitea, nextcloud, semaphore, kasm, grafana*, headlamp*
|
||||
(*platform UIs, not in the numbered app list).
|
||||
- **forward-auth** (oauth2-proxy → Keycloak): pihole, sonarr, radarr, jackett, qbittorrent, unifi,
|
||||
and platform UIs (Homepage, Prometheus, Alertmanager, Hubble).
|
||||
- **native** (own auth, documented exceptions): plex, seerr (Plex accounts); minecraft
|
||||
(Mojang/Microsoft).
|
||||
- **keycloak** itself is the IdP — its own admin login is local + MFA (the root of trust).
|
||||
|
||||
**"…but not limited to":** the catalog is open-ended. New services follow the same ServiceDefinition
|
||||
and inherit all guarantees automatically — that is the point of the abstraction.
|
||||
|
||||
## Contract clarifications (v1.1 — resolved 2026-08-16)
|
||||
|
||||
Points the schema left ambiguous, now pinned so the JSON Schema, Kyverno policies, and renderers agree:
|
||||
- **`source` is exactly-one-of** — `image` for `kind: container`, `vmTemplate` for `kind: vm`
|
||||
(schema-enforced conditional).
|
||||
- **`storage[].class: longhorn ⇒ size` is required** — enforced by an **L0 Kyverno policy** (a
|
||||
replicated volume with no size is a defect). `bulk` size may be omitted (external mount).
|
||||
- **`backup` is a reference** — `backup: {policy: <ref>}` where `policy` is either the literal
|
||||
`reacquirable` (bulk/re-acquirable data) or a named off-cluster backup schedule. Cadence/retention
|
||||
live in the referenced schedule object, **not** inline in the ServiceDefinition.
|
||||
- **`update.channel` is a free string defaulting to `stable`** — not a closed enum, so the sanctioned
|
||||
"opt-in to a non-stable channel" (R-UPD-01) is expressible per service.
|
||||
- **`auth.groups` is required for `auth.mode ∈ {oidc, forward-auth}`** — access must be scoped to at
|
||||
least one Keycloak group (R-IDP-06); optional for `native`/`none`.
|
||||
- **`exposure: public ⇒ auth.mode ≠ none`** — a public service may not be unauthenticated
|
||||
(docs/15, R-IDP-02); **schema-enforced** invariant.
|
||||
- **Irreplaceable ⇒ off-cluster backup** — DR-critical services MUST NOT be `reacquirable`. This
|
||||
depends on per-service criticality the schema can't see, so it is enforced by an **L0 Kyverno
|
||||
policy** keyed on the service's criticality signal, not by the JSON Schema.
|
||||
|
||||
## Cross-cutting notes
|
||||
- **Plex media** is the R-STATE-04 case: keep the metadata DB on replicated Longhorn; keep the large
|
||||
library on the bulk tier; downloads stage on bulk and are handed off by the arr stack.
|
||||
- **DR-critical data** (Nextcloud, Gitea, Minecraft worlds, PiHole config, service DBs) MUST have
|
||||
off-cluster backups ([04](04-storage-and-state.md)); bulk media may be `reacquirable` per operator
|
||||
choice (OQ-2).
|
||||
- **Public exposure** (Plex, Seerr, Nextcloud, Minecraft, Gitea) requires edge port-forwarding
|
||||
([09](09-dns-and-ingress.md)); everything else stays `internal`.
|
||||
- **Public exposure** (Seerr, Nextcloud, Gitea, …) is through the **Cloudflare Tunnel**
|
||||
([09](09-dns-and-ingress.md), ADR-0016). Everything else stays `internal`. **Plex** remote
|
||||
streaming uses plex.tv + an edge forward of tcp/32400 (not the tunnel), and its web UI is LAN-only.
|
||||
**Minecraft (crafty-mc)** and other L4 services are LAN/VPN-only.
|
||||
- **Kasm** is resource-heavy and may be a candidate for `kind: vm` if container mode proves
|
||||
insufficient — the ServiceDefinition makes that a one-field change.
|
||||
- **Gitea is bootstrapping-critical.** The platform self-hosts its git and CI/CD, so Gitea is brought
|
||||
|
||||
@@ -82,6 +82,11 @@ Each maps to requirement IDs; phrased as Given/When/Then intent.
|
||||
- **DNS/exposure (R-DNS-*):** *Given* a service, *When* resolved, *Then* the record exists and
|
||||
internal clients hit the internal Gateway; `internal` is unreachable publicly, `public` is
|
||||
reachable + TLS-valid.
|
||||
- **Public ingress path (R-DNS-04/05, R-SEC-04):** *Given* the tunnel ingress config, *When* it is
|
||||
rendered, *Then* every route is a `public` service, the last rule is `http_status:404`, and no rule
|
||||
targets the Kubernetes API (L0). *When* a non-`public` hostname is requested at the tunnel, *Then*
|
||||
it gets 404. *When* a `public` name is resolved via a public resolver from the LAN, *Then* it is
|
||||
reachable + TLS-valid (L2).
|
||||
- **Hands-off update (R-UPD-01/02, L3):** *Given* an out-of-date service under a stable policy,
|
||||
*When* the scan interval elapses, *Then* it advances with no human/AI edit; *Given* a deliberately
|
||||
bad update, *Then* health-gating auto-rolls-back and an email alert fires; *Given* a pinned
|
||||
@@ -122,6 +127,12 @@ from the **Kyverno ecosystem**:
|
||||
behavior tests**: a policy admitting a manifest is not proof the runtime behaves (R-TEST-01), so
|
||||
L2/L3 Chainsaw tests remain mandatory. Policies are authored once and shared between the CLI gate
|
||||
and the admission controller.
|
||||
- **ServiceDefinition validation is two complementary gates, not one** (do not consolidate): the
|
||||
**JSON Schema** (`schemas/servicedefinition.schema.json`) checks *shape* — types, enums, required
|
||||
fields, exactly-one-of `source`, `public ⇒ auth.mode≠none` — as a fast, cluster-free PR check;
|
||||
**Kyverno `kyverno-json`** enforces the *cross-cutting invariants* the schema can't express
|
||||
(`longhorn ⇒ size`, irreplaceable⇒off-cluster-backup, label conventions). Both are L0 and run in
|
||||
CI; they overlap by design (defense in depth), not by redundancy.
|
||||
- **Chainsaw** — **L1 rendering, L2 behavioral, and L3 fault-injection.** Declarative steps
|
||||
(apply/assert/error, `script`/`command`, `try`/`catch`/`finally`), bounded polling with explicit
|
||||
timeouts, per-test namespace isolation, and automatic cleanup satisfy the reproducibility contract.
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
# 15 — Identity & Access (SSO)
|
||||
|
||||
Serves: **R-IDP-01..06**, supports **R-C2-02**, **R-SEC-02/03**. Related:
|
||||
[ADR-0014 (Keycloak)](../decisions/adr-0014-idp-keycloak.md).
|
||||
|
||||
## Purpose
|
||||
|
||||
One place to log in. A central **Identity Provider (IdP)** gives the platform single sign-on (SSO),
|
||||
one user directory, group-based authorization, and MFA — so every service is protected by the same
|
||||
identity, and no management UI sits unauthenticated on the network.
|
||||
|
||||
## Choice: Keycloak
|
||||
|
||||
**Keycloak** is the platform IdP ([ADR-0014](../decisions/adr-0014-idp-keycloak.md)): a mature,
|
||||
self-hostable OpenID Connect (OIDC) + SAML provider with users, groups, roles, clients, MFA
|
||||
(TOTP/WebAuthn), and brute-force protection.
|
||||
|
||||
## Two integration patterns (behind one identity)
|
||||
|
||||
Every service authenticates against Keycloak, via whichever pattern it supports:
|
||||
|
||||
1. **Native OIDC** — the app is configured as a Keycloak OIDC **client** and does its own login
|
||||
redirect. Preferred where available. Initial set: **Gitea, Nextcloud, Grafana, Kasm, Semaphore,
|
||||
Headlamp**.
|
||||
2. **Forward-auth proxy** — for apps with **no** usable OIDC, an identity-aware proxy
|
||||
(**oauth2-proxy**) sits in front at the Gateway, authenticates the user against Keycloak, and only
|
||||
then forwards the request. Used for: **PiHole admin, the *arr web UIs (Sonarr/Radarr/Jackett),
|
||||
qBittorrent UI, Homepage, Prometheus, Alertmanager, Hubble UI**, and any other bare UI.
|
||||
|
||||
> This resolves the C2 auth open question ([12](12-command-center-ui.md)): it is **both** — native
|
||||
> OIDC where supported, forward-auth everywhere else, all backed by Keycloak.
|
||||
|
||||
### Exceptions (documented, not oversights)
|
||||
- **Plex** and **Seerr** use **Plex's own account system** (Plex OAuth) — they do not federate to
|
||||
Keycloak.
|
||||
- **Minecraft** authenticates players via Mojang/Microsoft — N/A.
|
||||
- **UniFi** keeps its native account model (limited SSO support); protect its UI via forward-auth if
|
||||
desired.
|
||||
|
||||
## Authorization model (R-IDP-06)
|
||||
|
||||
- **Groups drive access.** Keycloak groups (e.g. `platform-admins`, `media-users`, `viewers`) map to
|
||||
per-service access and roles. A service's ServiceDefinition declares which groups may reach it
|
||||
([13](13-service-catalog.md), `auth.groups`).
|
||||
- **Least privilege**: the C2 console, Headlamp, Alertmanager, and other sensitive UIs require an
|
||||
authenticated user **in an authorized group** — not merely "logged in."
|
||||
- **MFA**: TOTP/WebAuthn available for all users and **required for privileged groups**
|
||||
(`platform-admins`).
|
||||
|
||||
## Identity config as code (R-IDP-03)
|
||||
|
||||
Realms, clients, groups, and roles are **declarative and GitOps-managed** — not clicked into the
|
||||
admin console and forgotten:
|
||||
- The Keycloak instance is deployed via the **Keycloak Operator** (or Helm), and **realm/client/group
|
||||
configuration is reconciled declaratively** (e.g. `keycloak-config-cli` or realm-import CRs). Exact
|
||||
tool is an implementation detail; the requirement is that identity config lives in git and
|
||||
reconciles like everything else.
|
||||
- **OIDC client secrets and the Keycloak admin credential are SOPS-encrypted** ([07](07-security-and-secrets.md)).
|
||||
|
||||
## State, exposure & resilience
|
||||
|
||||
- **Datastore**: Keycloak runs on **PostgreSQL**; the DB lives on **3-replica Longhorn** and is
|
||||
**DR-critical → backed up off-cluster** ([04](04-storage-and-state.md)). Keycloak is stateless
|
||||
itself and rebuilds from the DB + git-managed realm config.
|
||||
- **Exposure**: `keycloak.home.keithjackson.dev` is **`public`** so it can serve login/redirect flows
|
||||
for the internet-exposed services (Nextcloud, Gitea) — but it is **security-sensitive** and
|
||||
hardened: brute-force detection on, strong admin auth + MFA, trusted TLS ([08](08-tls-and-certificates.md)),
|
||||
minimal exposed endpoints. If the operator prefers internal-only, public services fall back to
|
||||
their native auth.
|
||||
- **HA**: a single replica with DB on Longhorn + backups is acceptable for the home scale; note
|
||||
clustered Keycloak as a future hardening. **Availability caveat:** SSO becomes a dependency for
|
||||
logging into federated services — document a break-glass path (e.g. a local admin on Gitea) so an
|
||||
IdP outage doesn't lock the operator out of recovery tools.
|
||||
|
||||
## Interaction with default-deny & the gateway
|
||||
|
||||
- Keycloak and oauth2-proxy are ordinary workloads under **default-deny** ([03](03-networking-isolation-and-egress.md));
|
||||
the specific flows they need (app → Keycloak OIDC, proxy → Keycloak, Keycloak → its DB) are
|
||||
explicit allow rules.
|
||||
- Forward-auth is enforced at the **Gateway** ([09](09-dns-and-ingress.md)) so protection happens
|
||||
before traffic reaches the app.
|
||||
|
||||
## Bootstrapping & order
|
||||
|
||||
- Keycloak is **foundational** and onboarded **early** (with PiHole/Gitea, [13](13-service-catalog.md)) so
|
||||
later services can integrate SSO as they come up.
|
||||
- No hard circular dependency at bring-up: Gitea can start with a local admin and add OIDC once
|
||||
Keycloak is live; the C2 and sensitive UIs gain SSO as soon as the proxy + realm exist.
|
||||
|
||||
## Acceptance gates
|
||||
- A user logs in once at Keycloak and reaches all OIDC-integrated services without re-entering
|
||||
credentials (SSO proven).
|
||||
- A forward-auth-protected UI (e.g. PiHole admin) is **unreachable without a valid Keycloak session**
|
||||
and reachable with one, restricted to the allowed group.
|
||||
- Realm/client/group config applied purely from git reproduces a working SSO setup (no console
|
||||
clicking); destroying and rebuilding Keycloak from DB + git restores identity.
|
||||
- Privileged access (C2/Headlamp) requires MFA and group membership; a non-member is denied.
|
||||
- Client secrets and admin credentials exist only SOPS-encrypted.
|
||||
|
||||
## Open questions
|
||||
- Final declarative-config tool (`keycloak-config-cli` vs operator realm-import CRs) — implementation
|
||||
detail for WS-SEC.
|
||||
- Confirm `public` exposure for Keycloak (recommended) vs internal-only with native-auth fallback for
|
||||
public services.
|
||||
+4
-2
@@ -76,8 +76,10 @@ survives rebuild; default-deny + `egress` apply to the VM; guest auto-updates.
|
||||
|
||||
## Phase 6 — Service Catalog Onboarding + C2 UI
|
||||
**Entry:** Phases 2–4 done.
|
||||
**Do:** onboard the 13 services via ServiceDefinitions in dependency order (see workstream WS-APPS);
|
||||
**bring Gitea online early in this phase** and cut the GitOps source + CI/CD over to it (Gitea
|
||||
**Do:** onboard the 14 services via ServiceDefinitions in dependency order (see workstream WS-APPS);
|
||||
**bring Gitea and Keycloak online early in this phase** — Keycloak is the foundational IdP so
|
||||
services adopt SSO (native OIDC / forward-auth) as they onboard ([`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md));
|
||||
and cut the GitOps source + CI/CD over to Gitea (Gitea
|
||||
Actions running the Kyverno gate + Chainsaw conformance Job; PRs as the enforced merge gate —
|
||||
[`../docs/05-gitops-and-delivery.md`](../docs/05-gitops-and-delivery.md), [`../CONTRIBUTING.md`](../CONTRIBUTING.md));
|
||||
stand up the C2 UI (Homepage root + Grafana + Headlamp + Flux status).
|
||||
|
||||
+14
-5
@@ -11,7 +11,7 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol.
|
||||
| **WS-GITOPS** | GitOps, CI, catalog renderer | [05](../docs/05-gitops-and-delivery.md), [13](../docs/13-service-catalog.md) | R-GITOPS-*, R-UPD-02 | WS-INFRA |
|
||||
| **WS-NET** | Networking, isolation, egress, ingress | [03](../docs/03-networking-isolation-and-egress.md), [09](../docs/09-dns-and-ingress.md) | R-NET-*, R-DNS-* | WS-INFRA |
|
||||
| **WS-STORAGE** | Storage, state, backup/DR | [04](../docs/04-storage-and-state.md) | R-STATE-* | WS-INFRA |
|
||||
| **WS-SEC** | Secrets, TLS, RBAC, auditability | [07](../docs/07-security-and-secrets.md), [08](../docs/08-tls-and-certificates.md) | R-SEC-*, R-TLS-* | WS-INFRA, WS-NET |
|
||||
| **WS-SEC** | Secrets, TLS, RBAC, **identity/SSO**, auditability | [07](../docs/07-security-and-secrets.md), [08](../docs/08-tls-and-certificates.md), [15](../docs/15-identity-and-access.md) | R-SEC-*, R-TLS-*, R-IDP-* | WS-INFRA, WS-NET |
|
||||
| **WS-OBS** | Observability & alerting | [10](../docs/10-observability-and-alerting.md) | R-OBS-* | WS-INFRA, WS-STORAGE |
|
||||
| **WS-UPDATE** | Automated updates | [11](../docs/11-automated-updates.md) | R-UPD-* | WS-GITOPS, WS-OBS |
|
||||
| **WS-VM** | KubeVirt IaaS | [06](../docs/06-vm-workloads-kubevirt.md) | R-PLAT-03 | WS-NET, WS-STORAGE |
|
||||
@@ -34,6 +34,9 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol.
|
||||
dashboards. Consumed by WS-APPS/WS-VM.
|
||||
- **Update policy contract** (WS-UPDATE) — ImagePolicy/Renovate conventions a ServiceDefinition's
|
||||
`update` block maps to.
|
||||
- **SSO/auth contract** (WS-SEC) — how a ServiceDefinition's `auth` block (oidc / forward-auth /
|
||||
native / groups) maps to a Keycloak client or oauth2-proxy protection ([15](../docs/15-identity-and-access.md)).
|
||||
Consumed by WS-APPS/WS-C2.
|
||||
- **Conformance harness + gate** (WS-TEST) — the reproducible test framework (**Kyverno** for L0
|
||||
policy, **Chainsaw** for L1/L2/L3 behavior + fault-injection, [ADR-0011](../decisions/adr-0011-test-tooling-kyverno-chainsaw.md)),
|
||||
requirement-coverage report, and CI/phase gates every workstream's tests plug into. Each workstream
|
||||
@@ -55,10 +58,16 @@ only via [`../agents/coordination-protocol.md`](../agents/coordination-protocol.
|
||||
and CI/CD; after it is stable, WS-GITOPS cuts Flux's source over to Gitea and moves CI to Gitea
|
||||
Actions ([`../docs/05-gitops-and-delivery.md`](../docs/05-gitops-and-delivery.md),
|
||||
[`../CONTRIBUTING.md`](../CONTRIBUTING.md)). DR-critical → off-cluster backup + mirror. →
|
||||
3. **plex** (largest; validates bulk tier + public exposure) →
|
||||
4. **qbittorrent, jackett** (VPN + kill-switch proof) → 5. **sonarr, radarr** (VPN + arr↔downloader
|
||||
allows) → 6. **seerr** (cross-service allows to arr + plex) → 7. **nextcloud** (DR-critical data +
|
||||
backups) → 8. **minecraft, unifi, semaphore, kasm** (remaining).
|
||||
3. **keycloak (+ oauth2-proxy)** — **foundational IdP/SSO**, brought online early so later services
|
||||
integrate SSO as they onboard ([`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md)).
|
||||
DR-critical + break-glass path. →
|
||||
4. **plex** (largest; validates bulk tier + public exposure) →
|
||||
5. **qbittorrent, jackett** (VPN + kill-switch proof) → 6. **sonarr, radarr** (VPN + arr↔downloader
|
||||
allows) → 7. **seerr** (cross-service allows to arr + plex) → 8. **nextcloud** (DR-critical data +
|
||||
backups) → 9. **minecraft, unifi, semaphore, kasm** (remaining).
|
||||
|
||||
As each service onboards, it adopts SSO per its `auth` mode (native OIDC or forward-auth) —
|
||||
[`../docs/15-identity-and-access.md`](../docs/15-identity-and-access.md).
|
||||
|
||||
**GitOps source migration** (WS-GITOPS, tied to step 2): local bootstrap source → self-hosted Gitea;
|
||||
CI (Kyverno gate + Chainsaw conformance Job trigger) moves to Gitea Actions; PRs become the enforced
|
||||
|
||||
Reference in new issue
Block a user