Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5.5
|
9a69f6906a | ||
|
jakman85
|
7ef2af205d | ||
|
jakman85andClaude Opus 5.5
|
788eaeae62 | ||
|
jakman85
|
576e1abb19 | ||
|
jakman85andClaude Opus 5.5
|
a1e808e697 | ||
|
jakman85
|
bcb8931b4f | ||
|
jakman85
|
0d7f94098b |
No files matched your search
@@ -0,0 +1,76 @@
|
||||
---
|
||||
date: 2026-10-03
|
||||
branch: fix/cilium-cgroup-hostroot
|
||||
title: Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`cgroup.hostRoot` changes from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity
|
||||
copies: `platform/cilium/helmrelease.yaml` (authoritative), `platform/cilium/values.yaml`
|
||||
(mirror) and `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` (bootstrap).
|
||||
`autoMount` stays `false`.
|
||||
|
||||
**The defect.** With `autoMount: false`, the Cilium chart only bind-mounts whatever exists at
|
||||
`hostRoot`, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml).
|
||||
The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for
|
||||
`/sys/fs/cgroup`. Nothing mounts cgroup2 at `/run/cilium/cgroupv2` on these nodes; it is an
|
||||
empty `DirectoryOrCreate` hostPath. So the agent mounted cgroup2 there itself, from inside its
|
||||
own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup.
|
||||
The socket-LB programs (`cil_sock4_connect` etc.) were attached to it and covered only the
|
||||
agent.
|
||||
|
||||
**The consequence.** Every other host-netns process got no Service/LB translation:
|
||||
- Service ClusterIPs were unreachable from every node's host.
|
||||
- The node holding the Gateway's L2 lease could not reach the Gateway VIP (`no route to host`).
|
||||
Other nodes reached it only because their packets crossed the wire to the lease holder, where
|
||||
the tc datapath handles them.
|
||||
- k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach
|
||||
the OIDC issuer, logging `oidc: authenticator not initialized` since at least 2026-09-12
|
||||
(899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster
|
||||
OIDC clients failed on whichever calls landed on that apiserver.
|
||||
- Webhooks and aggregated APIs were unaffected only because k3s runs
|
||||
`--enable-aggregator-routing=true` (endpoint IPs, not ClusterIPs).
|
||||
|
||||
Cilium's own source (v1.20.1 `bpf/bpf_sock.c`, `__sock4_xlate_fwd`) handles exactly this case
|
||||
when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to
|
||||
`127.0.0.1:<l7_lb_proxy_port>`, the local Envoy listener. The fix is to make the hook cover the
|
||||
host.
|
||||
|
||||
## What I verified
|
||||
|
||||
- `cilium-dbg status --verbose` on k3s-prod-1: `Socket LB: Enabled`, `Coverage: Hostns-only`.
|
||||
The live cilium-config shows `bpf-lb-sock=false`, but socket LB is implied by kube-proxy
|
||||
replacement. The Gateway Service frontends `192.168.2.20:{80,443}` and
|
||||
`10.43.225.79:{80,443}` map to `127.0.0.1:14337`.
|
||||
- Host: `findmnt /run/cilium/cgroupv2` is empty on all three nodes, and `/sys/fs/cgroup` is
|
||||
`cgroup2`.
|
||||
- In the agent: `/run/cilium/cgroupv2` is a `cgroup2 none` mount, the agent's own cgroup reads
|
||||
`0::/`, the mount has 0 child cgroups, and `bpftool cgroup show /run/cilium/cgroupv2` lists all
|
||||
`cil_sock*` programs. k3s runs in `0::/system.slice/k3s.service`, and the agent container in
|
||||
`/kubepods.slice/…/cri-containerd-3ad04cde….scope`.
|
||||
- **The decisive test**, run on k3s-prod-1 in the same netns: raw TCP connect to
|
||||
`192.168.2.20:443`, `10.43.225.79:443` and `10.43.59.141:8080` CONNECTED from inside the agent
|
||||
container (covered cgroup) and FAILED from the host (`No route to host` / timeout).
|
||||
- Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via
|
||||
`192.168.2.9:6443`, not a Service), kube-vip (`127.0.0.1:6443`, `svc_enable=false`) and
|
||||
node-exporter (no outbound). Nothing depends on host Service translation being absent; the
|
||||
change only makes Service/LB addresses reachable from the host where they currently fail.
|
||||
- `helm template cilium/cilium --version 1.20.1` with the new values, in the toolbox: the
|
||||
DaemonSet bind-mounts hostPath `/sys/fs/cgroup` into `cilium-agent` and `clean-cilium-state`,
|
||||
the cilium-config shows `cgroup-root: "/sys/fs/cgroup"`, and the init containers are unchanged
|
||||
(no `mount-cgroup`).
|
||||
- Toolbox gate: see the PR description.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The rollout itself.** After merge, the agent DaemonSet rolls one node at a time
|
||||
(`maxUnavailable: 1`). On each node, afterwards: host TCP connects to the three targets above
|
||||
must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging
|
||||
`authenticator not initialized` with no k3s restart.
|
||||
- That the old programs, attached to the previous agent container's cgroup, disappear with it.
|
||||
Expected, because cgroup BPF attachments die with the cgroup, but not observed.
|
||||
- An end-to-end kubectl OIDC login (needs a browser).
|
||||
- No regression test was added. A natural one is an L2 conformance check that a host-netns
|
||||
connect to the Gateway VIP succeeds on the lease holder, but `conformance/**` is a sensitive
|
||||
path. Left as a follow-up.
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
date: 2026-10-03
|
||||
branch: fix/node-keycloak-hosts-pin
|
||||
title: Pin the OIDC issuer to the Gateway VIP on prod nodes — the WAN answer does not hairpin
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Follow-up to PR #40 (`decisions/2026-10-03-public-dns-wan-target.md`). Public DNS for
|
||||
`*.home.keithjackson.dev` now CNAMEs to the WAN IP. The prod nodes resolve through public DNS,
|
||||
and the router does not hairpin, so the kube-apiserver could no longer reach its OIDC issuer
|
||||
`keycloak.home.keithjackson.dev`.
|
||||
|
||||
- `roles/node_baseline`: a new `node_baseline_hosts_pins` list (default `[]`) and one
|
||||
`blockinfile` task (tags `dns`, `hosts`). It writes the pins into `/etc/hosts` as a single
|
||||
marked block. An empty list removes the block.
|
||||
- `group_vars/k3s_prod/vars.yml`: pins `keycloak.{{ k3s_base_domain }}` → `192.168.2.20`, which
|
||||
must stay equal to `GATEWAY_VIP` (parity comment added). It also replaces the stale "no
|
||||
/etc/hosts pin is needed" paragraph with the measured reason the pin is now required.
|
||||
|
||||
Dev is unaffected (empty default).
|
||||
|
||||
## What I verified
|
||||
|
||||
- After PR #40 reconciled (`platform` lastAppliedRevision `576e1ab`, Healthy=True), on
|
||||
k3s-prod-2 and k3s-prod-3: `getent hosts keycloak.home.keithjackson.dev` →
|
||||
`136.51.115.252 wan.home.keithjackson.dev`. `curl https://keycloak…/realms/homelab/.well-known/openid-configuration`
|
||||
→ exit 60, `ssl_verify_result=18` (the router's self-signed cert, not the Gateway).
|
||||
- `journalctl -u k3s` (last 20 min) showed no OIDC/JWKS errors yet. The apiserver still holds
|
||||
the keys it cached before the flip, so the failure is latent until a k3s restart or key rotation.
|
||||
- Neither node has cloud-init (`/etc/cloud` absent), so nothing rewrites `/etc/hosts` at boot.
|
||||
- `--check --diff --tags hosts --limit k3s-prod-2,k3s-prod-3`: each node gets exactly the
|
||||
three-line marked block `192.168.2.20 keycloak.home.keithjackson.dev`.
|
||||
- Outside-in, PR #40 works: check-host.net probes from 3 external nodes got `200 OK` from
|
||||
`https://plex.home.keithjackson.dev/identity` at `136.51.115.252`.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **k3s-prod-1**: Ansible cannot reach it. SSH host key verification fails (`known_hosts:94`,
|
||||
ED25519 `SHA256:zgUS8OeWznzCX9hlN7XdZ8iLNkU1G2loXjtJFfEUYpE`), which is consistent with
|
||||
the 2026-09-11 SSD replacement and reinstall. The operator must confirm the fingerprint
|
||||
before the pin can be applied there.
|
||||
- The playbook has not been applied for real. It is an operator run after merge:
|
||||
`ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags hosts`.
|
||||
- An end-to-end kubectl OIDC login after the pin (needs a browser).
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
date: 2026-10-03
|
||||
branch: feat/public-dns-wan-target
|
||||
title: Public DNS now points at the WAN — Gateway-published names CNAME to an in-cluster DDNS record
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator closed the open "public-target gap" recorded in
|
||||
`platform/externaldns/helmrelease.yaml`. The WAN NAT / port-forwarding (80/443 → Gateway VIP
|
||||
192.168.2.20) was done by the operator outside this repo.
|
||||
|
||||
Operator decisions (2026-10-03):
|
||||
|
||||
1. **Scope: every Gateway-published name goes public.** That covers the five `exposure: public`
|
||||
apps (plex, gitea, keycloak, nextcloud, seerr) and also the hand-authored admin UIs (auth,
|
||||
crafty, grafana, headlamp, the `home.keithjackson.dev` apex). `internal` services still publish
|
||||
nothing: their `controller: none` route annotation is checked before the target is read.
|
||||
2. **The WAN IP is kept current in-cluster**, not hardcoded.
|
||||
|
||||
Implementation:
|
||||
|
||||
- `platform/gateway/gateway.yaml`: `external-dns.alpha.kubernetes.io/target: wan.${BASE_DOMAIN}` on
|
||||
the shared Gateway. Every gateway-sourced record changes from `A 192.168.2.20` to
|
||||
`CNAME wan.home.keithjackson.dev`.
|
||||
- `platform/externaldns/ddns.yaml` (new): `favonia/cloudflare-ddns:1.17.1` Deployment in the
|
||||
`external-dns` namespace. It reuses `cloudflare-api-token` and keeps `wan.${BASE_DOMAIN}` an
|
||||
unproxied IPv4 A record at the WAN IP (detected via Cloudflare trace, checked every 5 min,
|
||||
`DELETE_ON_STOP=false`). external-dns has no TXT registry entry for it, so it never owns or
|
||||
deletes it.
|
||||
- The helmrelease comment block and the README now describe the decision instead of the gap.
|
||||
|
||||
Known consequence: **k3s nodes resolve through public DNS** (`node_baseline_dns_servers` = 8.8.8.8 /
|
||||
1.1.1.1). The kube-apiserver's OIDC issuer `keycloak.home.keithjackson.dev` therefore resolves to the
|
||||
WAN IP from the nodes, and kubectl SSO works only if the router hairpins WAN:443 back to the Gateway.
|
||||
A LAN-side probe of `136.51.115.252:443` on 2026-10-03 returned a **self-signed placeholder cert**,
|
||||
not the Gateway's Let's Encrypt wildcard. That suggests no hairpin. The cert-based admin kubeconfig
|
||||
is unaffected.
|
||||
|
||||
## What I verified
|
||||
|
||||
- Live prod, 2026-10-03, before the change: Gateway `gateway/shared` ADDRESS `192.168.2.20`.
|
||||
External-dns args include `--source=gateway-httproute --source=service --policy=sync
|
||||
--txt-owner-id=prod --domain-filter=keithjackson.dev`, and it logs "All records are already up to
|
||||
date".
|
||||
- Public answers at 1.1.1.1: plex/gitea/keycloak/grafana/auth/apex → `192.168.2.20`;
|
||||
`crafty-mc` → `.22` and `unifi` → `.21` (service source, not affected by this change); jackett →
|
||||
none (internal, correctly unpublished).
|
||||
- The authority (betty.ns.cloudflare.com) has no unprefixed TXT, AAAA or MX at `plex.` or the apex,
|
||||
so the A→CNAME switch has no co-existing record to collide with.
|
||||
- WAN IP from the workstation: `136.51.115.252`.
|
||||
- No CiliumNetworkPolicy or NetworkPolicy in `external-dns`. The Kyverno label and default-deny
|
||||
policies match only `app-namespace` namespaces. The vpn egress policy selects only
|
||||
`egress-class: vpn` pods.
|
||||
- Toolbox gate result: see the PR description (run after this commit).
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **The outside-in path.** I have no vantage point outside the LAN. Whether
|
||||
`https://<svc>.home.keithjackson.dev` from the internet reaches the Gateway with the LE cert is
|
||||
UNVERIFIED.
|
||||
- **Hairpin / kubectl SSO after the flip.** Likely broken per the probe above. Fix candidates: a
|
||||
node `/etc/hosts` pin for `keycloak.${BASE_DOMAIN}` → Gateway VIP via `node_baseline`, or router
|
||||
NAT reflection.
|
||||
- **The record-type flip itself**, until Flux reconciles: whether Cloudflare accepts external-dns's
|
||||
create-CNAME in the same batch as the delete-A, or converges one interval later.
|
||||
- `favonia/cloudflare-ddns` under this pod securityContext. The README warns that
|
||||
`no-new-privileges` misbehaves on some Proxmox-style hosts. These are bare-metal nodes, but it is
|
||||
untested until the pod runs.
|
||||
- Whether the L4 service-source names (crafty-mc, unifi) should also go public. Not decided, not
|
||||
changed.
|
||||
- `conformance/tests/l0-static/internal-services-publish-no-public-dns/check.sh` header lines 39–40
|
||||
still describe the target as open. That is a sensitive path, so it is left for a follow-up.
|
||||
@@ -306,19 +306,27 @@ node_baseline_dns_servers:
|
||||
# k3s_base_domain, below). It reaches the issuer through the NODE resolver, i.e. through
|
||||
# these PUBLIC upstreams — not through CoreDNS's split-horizon answer.
|
||||
#
|
||||
# THIS USED TO BE BROKEN, AND THE 2026-09-04 SUBDOMAIN SPLIT FIXED IT. While prod was
|
||||
# authored to serve dev's hostnames, a prod node resolving keycloak.home.keithjackson.dev
|
||||
# publicly got DEV's Gateway VIP (192.168.1.53) — the wrong cluster, on the wrong LAN, so
|
||||
# kubectl OIDC SSO on prod could not work at all. Now prod's issuer is
|
||||
# keycloak.prod.home.keithjackson.dev, and the public record for that name is published by
|
||||
# PROD's own external-dns pointing at PROD's Gateway VIP (192.168.2.20). The public answer
|
||||
# and the split-horizon answer therefore AGREE, and no /etc/hosts pin is needed.
|
||||
# THE PUBLIC ANSWER IS NOW THE WAN, SO THE ISSUER IS PINNED (2026-10-03). Since
|
||||
# platform/gateway/gateway.yaml set an external-dns `target`, every public name is a CNAME
|
||||
# to wan.home.keithjackson.dev = the home WAN IP. The SOHO router does NOT hairpin: from the
|
||||
# LAN, WAN:443 answers with the router's own self-signed placeholder cert (measured from
|
||||
# k3s-prod-2/-3 2026-10-03: `curl` exit 60, ssl_verify_result=18). The apiserver would then
|
||||
# be unable to fetch its issuer's discovery/JWKS, so kubectl OIDC SSO would fail on the
|
||||
# next k3s restart or Keycloak key rotation. node_baseline_hosts_pins (below) pins the
|
||||
# issuer host to the Gateway VIP in /etc/hosts, which glibc consults before these
|
||||
# resolvers. (History: before the 2026-09-04 subdomain split the public answer was DEV's
|
||||
# VIP; after it, the public answer was prod's own VIP and no pin was needed.)
|
||||
#
|
||||
# STILL TO VERIFY on prod bring-up (PROD-PROVISIONING.md §10): this holds only once prod's
|
||||
# external-dns has actually published the record. Until then the issuer does not resolve
|
||||
# from the nodes and kubectl SSO fails — the cert-based admin kubeconfig is the break-glass
|
||||
# path, and it always works. Ordering, not a design gap.
|
||||
# The cert-based admin kubeconfig never depends on any of this. It is the break-glass path.
|
||||
node_baseline_dns_search: []
|
||||
# /etc/hosts pins: node-process names that must reach the in-cluster Gateway directly rather
|
||||
# than via the public (WAN, non-hairpinned) answer. See "THE PUBLIC ANSWER IS NOW THE WAN" above.
|
||||
# [parity] the IP MUST equal GATEWAY_VIP in clusters/prod/kustomization.yaml (192.168.2.20),
|
||||
# and the name MUST follow k3s_oidc_issuer_url's host (roles/k3s_server/defaults/main.yml).
|
||||
node_baseline_hosts_pins:
|
||||
- ip: "192.168.2.20"
|
||||
names:
|
||||
- "keycloak.{{ k3s_base_domain }}"
|
||||
|
||||
# --- platform hostname suffix, ANSIBLE SIDE (R-ENV-02, R-DNS-01) --------------------
|
||||
# PROD SERVES ITS OWN SUBDOMAIN. [parity] MUST equal BASE_DOMAIN in clusters/prod/
|
||||
|
||||
@@ -51,11 +51,14 @@ hubble:
|
||||
l2announcements:
|
||||
enabled: true
|
||||
|
||||
# k3s ships its own cgroup mount; point Cilium at it instead of auto-mounting.
|
||||
# Point Cilium at the host's real cgroup2 root (/sys/fs/cgroup, mounted by systemd) instead of
|
||||
# auto-mounting. NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so the agent mounted its
|
||||
# OWN namespaced cgroup and socket-LB covered only itself — host processes (the apiserver) lost
|
||||
# Service/LB translation. [parity] platform/cilium/helmrelease.yaml explains (2026-10-03).
|
||||
cgroup:
|
||||
autoMount:
|
||||
enabled: false
|
||||
hostRoot: /run/cilium/cgroupv2
|
||||
hostRoot: /sys/fs/cgroup
|
||||
|
||||
# Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
|
||||
# platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a
|
||||
|
||||
@@ -242,3 +242,9 @@ node_baseline_dns_search:
|
||||
node_baseline_dns_options:
|
||||
- timeout:2
|
||||
- attempts:2
|
||||
# Static /etc/hosts entries (tags: dns, hosts), as a list of {ip, names}. EMPTY by default:
|
||||
# dev needs none. Prod pins its OIDC issuer host to the Gateway VIP because the public
|
||||
# answer for it is the non-hairpinned WAN IP (inventory/group_vars/k3s_prod/vars.yml).
|
||||
# Written as ONE marked block, so removing an entry here removes it from the node, and
|
||||
# an empty list removes the block entirely.
|
||||
node_baseline_hosts_pins: []
|
||||
@@ -1090,3 +1090,22 @@
|
||||
{% endif %}
|
||||
when: node_baseline_dns_mechanism == 'resolv-conf-file'
|
||||
tags: [dns]
|
||||
|
||||
# /etc/hosts pins — tag `dns` (and `hosts` on its own). Some node processes must reach a
|
||||
# public hostname at the in-cluster Gateway, not at its public answer. On prod, the
|
||||
# kube-apiserver's OIDC issuer resolves publicly to the WAN IP, which the router does not
|
||||
# hairpin. glibc reads /etc/hosts before any nameserver (nsswitch `files dns`), so a pin wins
|
||||
# regardless of which resolver mechanism above is in effect. One marked block: entries
|
||||
# dropped from node_baseline_hosts_pins disappear from the node, and an empty list
|
||||
# removes the block.
|
||||
- name: Pin node-process hostnames in /etc/hosts
|
||||
ansible.builtin.blockinfile:
|
||||
path: /etc/hosts
|
||||
marker: "# {mark} node_baseline hosts pins (Ansible, tags: hosts)"
|
||||
block: |
|
||||
{% for pin in node_baseline_hosts_pins %}
|
||||
{{ pin.ip }} {{ pin.names | join(' ') }}
|
||||
{% endfor %}
|
||||
state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
|
||||
backup: true
|
||||
tags: [dns, hosts]
|
||||
@@ -106,14 +106,33 @@ spec:
|
||||
# CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
|
||||
l2announcements:
|
||||
enabled: true
|
||||
# --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount ---
|
||||
# [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS
|
||||
# part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount
|
||||
# cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption.
|
||||
# --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
|
||||
# [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
|
||||
#
|
||||
# FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
|
||||
# EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
|
||||
# at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
|
||||
# cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
|
||||
# /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
|
||||
# mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
|
||||
# agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
|
||||
# only the agent. Every other host-netns process (the k3s apiserver in
|
||||
# /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
|
||||
# every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
|
||||
# ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
|
||||
# initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
|
||||
# Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
|
||||
# and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
|
||||
# FAILED from the host, same node, same netns. Full write-up:
|
||||
# decisions/2026-10-03-cilium-cgroup-hostroot.md.
|
||||
#
|
||||
# /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
|
||||
# with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
|
||||
# stays off because the mount it would create already exists.
|
||||
cgroup:
|
||||
autoMount:
|
||||
enabled: false
|
||||
hostRoot: /run/cilium/cgroupv2
|
||||
hostRoot: /sys/fs/cgroup
|
||||
# --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
|
||||
# [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
|
||||
# The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
|
||||
|
||||
@@ -41,11 +41,13 @@ hubble:
|
||||
# [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
|
||||
l2announcements:
|
||||
enabled: true
|
||||
# [parity] k3s provides the cgroupv2 mount; do not auto-mount
|
||||
# [parity] the host's cgroup2 root is /sys/fs/cgroup (systemd mounts it); do not auto-mount.
|
||||
# NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so socket-LB attached to the agent's own
|
||||
# cgroup and host processes lost Service/LB translation (2026-10-03 — see helmrelease.yaml).
|
||||
cgroup:
|
||||
autoMount:
|
||||
enabled: false
|
||||
hostRoot: /run/cilium/cgroupv2
|
||||
hostRoot: /sys/fs/cgroup
|
||||
# [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
|
||||
# default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
|
||||
# 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the
|
||||
|
||||
@@ -20,6 +20,30 @@ A wildcard `*.keithjackson.dev` matches only one label deep, so `<svc>.home.keit
|
||||
its own record. **`internal` services need NO public record** — PiHole resolves them to the internal
|
||||
Gateway VIP (split-horizon). ExternalDNS manages **`public`** services' records only.
|
||||
|
||||
## Public target + in-cluster DDNS (2026-10-03)
|
||||
|
||||
Gateway-published records are **CNAMEs to `wan.home.keithjackson.dev`**, not the Gateway VIP:
|
||||
|
||||
```
|
||||
plex.home.keithjackson.dev CNAME wan.home.keithjackson.dev <- external-dns (Gateway `target` annotation)
|
||||
wan.home.keithjackson.dev A <current WAN IP> <- cloudflare-ddns (ddns.yaml), every 5m
|
||||
```
|
||||
|
||||
- The `target` annotation is on the **shared Gateway** (`platform/gateway/gateway.yaml`), so it
|
||||
applies to every route that publishes at all. `internal` routes (`controller: none`) still publish
|
||||
nothing.
|
||||
- `ddns.yaml` runs `favonia/cloudflare-ddns` in this namespace with the same `cloudflare-api-token`.
|
||||
It detects the WAN IP through Cloudflare's trace endpoint, so the pod **must not** carry
|
||||
`egress-class: vpn`.
|
||||
- LAN and pods resolve `*.home.keithjackson.dev` to the Gateway VIP via PiHole and CoreDNS
|
||||
(split-horizon) and never use the public path.
|
||||
- **Nodes resolve through public DNS** (`node_baseline_dns_servers`), so from a node, the apiserver's
|
||||
OIDC issuer `keycloak.home.keithjackson.dev` resolves to the WAN IP. kubectl SSO therefore relies on
|
||||
the router hairpinning (NAT reflection) WAN:443 back to the Gateway.
|
||||
|
||||
Check: `dig +short @1.1.1.1 plex.home.keithjackson.dev` → `wan.home.keithjackson.dev.` then the WAN IP;
|
||||
`kubectl -n external-dns logs deploy/cloudflare-ddns`.
|
||||
|
||||
## Secrets
|
||||
|
||||
Cloudflare API token referenced **by name** (`cloudflare-api-token`, shared with cert-manager),
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
# In-cluster dynamic DNS for the home WAN (2026-10-03).
|
||||
#
|
||||
# Keeps ONE record, `wan.${BASE_DOMAIN}` (A, IPv4 only, not proxied), pointed at the home WAN's
|
||||
# current public IP. Every public name external-dns publishes from the shared Gateway is a CNAME to
|
||||
# it (platform/gateway/gateway.yaml `external-dns.alpha.kubernetes.io/target`), so a WAN IP change
|
||||
# moves all of them through this one record, with no Git change and no external-dns involvement.
|
||||
#
|
||||
# OWNERSHIP SPLIT WITH EXTERNAL-DNS. This record carries no external-dns TXT registry entry, so
|
||||
# external-dns (policy: sync, owner `prod`) never treats it as its own and never deletes it.
|
||||
# Conversely this updater manages only the names in DOMAINS and never touches external-dns's records.
|
||||
#
|
||||
# IP DETECTION. Default `cloudflare.trace`: an HTTPS request to Cloudflare's trace endpoint, which
|
||||
# reports the address the request arrived from. Correct here only because this pod is NOT in the
|
||||
# `vpn` egress class — the CiliumEgressGatewayPolicy (platform/vpn-egress) selects pods by
|
||||
# `homelab.keithjackson.dev/egress-class: vpn` alone, cluster-wide. Labelling this pod `vpn` would
|
||||
# publish the VPN exit IP instead of the WAN IP.
|
||||
#
|
||||
# TOKEN. Reuses this namespace's `cloudflare-api-token` (same Zone:DNS:Edit token as external-dns
|
||||
# and cert-manager) — no new secret.
|
||||
#
|
||||
# TIER NOTE. This lives in the gating `platform` Flux Kustomization (wait: true) next to
|
||||
# external-dns. If it ever crashloops, `platform` goes not-Ready and apps stop reconciling (no
|
||||
# outage). Kept here rather than in its own non-gating Kustomization to avoid a clusters/** change.
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: cloudflare-ddns
|
||||
namespace: external-dns
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflare-ddns
|
||||
spec:
|
||||
replicas: 1 # exactly one writer for the record
|
||||
revisionHistoryLimit: 3
|
||||
strategy:
|
||||
type: Recreate # never two updaters at once
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: cloudflare-ddns
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflare-ddns
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: cloudflare-ddns
|
||||
image: favonia/cloudflare-ddns:1.17.1
|
||||
imagePullPolicy: IfNotPresent
|
||||
env:
|
||||
- name: CLOUDFLARE_API_TOKEN
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
name: cloudflare-api-token
|
||||
key: api-token
|
||||
- name: DOMAINS
|
||||
value: "wan.${BASE_DOMAIN}"
|
||||
- name: IP6_PROVIDER
|
||||
value: "none" # IPv4 WAN only; leaves any AAAA alone
|
||||
- name: PROXIED
|
||||
value: "false" # DNS-only: Plex/Gitea/etc. terminate TLS on the Gateway
|
||||
- name: UPDATE_CRON
|
||||
value: "@every 5m"
|
||||
- name: DELETE_ON_STOP
|
||||
value: "false" # a pod restart must not blank the record every CNAME depends on
|
||||
- name: RECORD_COMMENT
|
||||
value: "managed by k3s cloudflare-ddns (platform/externaldns/ddns.yaml)"
|
||||
- name: EMOJI
|
||||
value: "false"
|
||||
resources:
|
||||
requests:
|
||||
cpu: 5m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
memory: 64Mi
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
@@ -81,38 +81,29 @@ spec:
|
||||
# what this instance can see or own; per-resource exclusion only ever REMOVES an object from
|
||||
# this instance's view, so it cannot make prod capable of deleting a record it does not own.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# THE PUBLIC-TARGET GAP — STILL OPEN, AND NOT FIXABLE HERE ON v0.21.0 (operator decision)
|
||||
# THE PUBLIC TARGET — DECIDED 2026-10-03 (operator): every Gateway-published name goes public
|
||||
#
|
||||
# The five `exposure: public` services (plex, gitea, seerr, keycloak, nextcloud) publish records
|
||||
# that point at 192.168.2.20 — the Gateway VIP, an unroutable RFC1918 address — so they are not
|
||||
# actually reachable from the internet either. Their records are LEFT EXACTLY AS THEY ARE by
|
||||
# this change, for two independent reasons:
|
||||
# Until 2026-10-03 every gateway-sourced record answered with the Gateway VIP (192.168.2.20,
|
||||
# RFC1918), so nothing published here was reachable from the internet. Two facts constrained the
|
||||
# fix, and both still hold:
|
||||
# * `external-dns.alpha.kubernetes.io/target` is read in v0.21.0's gateway-httproute source ONLY
|
||||
# from the GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
||||
# gw.Annotations)`) and from a ListenerSet listener — NEVER from an HTTPRoute. A target on the
|
||||
# shared Gateway therefore moves EVERY gateway-sourced record at once.
|
||||
# * The home WAN IP is residential and can change, so a literal IP in Git would rot.
|
||||
#
|
||||
# 1. THE VALUE IS NOT ESTABLISHED. The one public IP this repo knows is 136.51.115.252
|
||||
# (clusters/prod/kustomization.yaml, where the zone's `*.keithjackson.dev` wildcard also
|
||||
# points). It belongs to the LEGACY non-k3s stack and currently serves a placeholder
|
||||
# certificate. Whether prod's public services belong behind it, behind a new forward, or
|
||||
# are not meant to be public at all is an operator decision. Breaking five services' DNS
|
||||
# on a guess is worse than leaving a known-wrong-but-stable state.
|
||||
# 2. THE MECHANISM CANNOT BE SCOPED TO THEM. `external-dns.alpha.kubernetes.io/target` is a
|
||||
# real override, but in v0.21.0's gateway-httproute source it is read ONLY from the
|
||||
# GATEWAY (source/gateway.go: `overrides: annotations.TargetsFromTargetAnnotation(
|
||||
# gw.Annotations)`) and from a ListenerSet listener — NEVER from the HTTPRoute. So there is
|
||||
# no per-service target override while every public service shares the `shared` Gateway.
|
||||
# Annotating the Gateway would move EVERY gateway-sourced record, which after this change
|
||||
# still includes auth, crafty, grafana, headlamp and the apex — i.e. it would publish a
|
||||
# routable address for admin UIs. That is strictly worse than the status quo.
|
||||
#
|
||||
# SO THE DECISION THE OPERATOR OWNS IS TWO-PART, and no knob is added here until the first half
|
||||
# is answered, because a knob that cannot take effect invites someone to set it and expect one:
|
||||
# (a) WHAT address should the five public names answer with — 136.51.115.252 as-is, a new
|
||||
# port-forward/WAN address, or nothing (drop `exposure: public` to `internal` instead)?
|
||||
# (b) HOW should they be separated from the non-public names that share the Gateway? Options,
|
||||
# roughly in ascending cost: a SECOND Gateway (or a ListenerSet) carrying the target
|
||||
# annotation, with only the public services' routes attached; or publishing the five names
|
||||
# from the `crd` source (DNSEndpoint objects the renderer emits for `public` services),
|
||||
# which restores per-service targeting without a second Gateway.
|
||||
# Only (b) is a manifest change; (a) is a fact about the home WAN that has to be checked.
|
||||
# The operator's answers: (a) yes, ALL of them — the five `exposure: public` apps AND the admin
|
||||
# UIs (auth, crafty, grafana, headlamp, the apex), which sit behind Keycloak / oauth2-proxy;
|
||||
# (b) the address is kept current IN-CLUSTER. Implementation:
|
||||
# * platform/gateway/gateway.yaml sets `target: wan.${BASE_DOMAIN}` on the shared Gateway, so
|
||||
# each published name becomes a CNAME to that one hostname;
|
||||
# * platform/externaldns/ddns.yaml (favonia/cloudflare-ddns) keeps `wan.${BASE_DOMAIN}` an A
|
||||
# record at the current WAN IP. That record has no external-dns TXT registry entry, so this
|
||||
# instance never owns or garbage-collects it.
|
||||
# Internal services are unchanged: their routes carry `controller: none` (above), which is
|
||||
# checked BEFORE the target is read, so they still publish nothing. The `service` source (the L4
|
||||
# LoadBalancers crafty-mc, unifi) does not read the Gateway's annotation and still publishes its
|
||||
# LB VIP. Whether those should go public is a separate, still-open question.
|
||||
# ---------------------------------------------------------------------------------------------
|
||||
# CROSS-CLUSTER DNS OWNERSHIP. Read before changing either knob below.
|
||||
#
|
||||
|
||||
@@ -8,3 +8,5 @@ resources:
|
||||
# SOPS-encrypted Cloudflare token (env CF_API_TOKEN) — SAME token as cert-manager, second copy in
|
||||
# the external-dns namespace. Flux decrypts at reconcile.
|
||||
- cloudflare-api-token.sops.yaml
|
||||
# DDNS for `wan.${BASE_DOMAIN}` — the CNAME target every Gateway-published record points at.
|
||||
- ddns.yaml
|
||||
@@ -7,6 +7,23 @@ kind: Gateway
|
||||
metadata:
|
||||
name: shared
|
||||
namespace: gateway
|
||||
annotations:
|
||||
# PUBLIC DNS TARGET (2026-10-03, operator decision). Without this, external-dns publishes every
|
||||
# route on this Gateway with the Gateway's OWN address (${GATEWAY_VIP}, RFC1918), which is
|
||||
# unreachable from the internet. With it, every published `<svc>.${BASE_DOMAIN}` (and the apex)
|
||||
# becomes a CNAME to `wan.${BASE_DOMAIN}`, an A record that platform/externaldns/ddns.yaml keeps
|
||||
# pointed at the home WAN's current public IP. The WAN forwards 80/443 to ${GATEWAY_VIP}.
|
||||
#
|
||||
# Scope is EVERY route on this Gateway that publishes at all — the operator chose to make the
|
||||
# admin UIs (auth, grafana, headlamp, crafty, apex) publicly resolvable alongside the five
|
||||
# `exposure: public` apps. `internal` services stay unpublished via their
|
||||
# `external-dns.alpha.kubernetes.io/controller: none` route annotation, which external-dns checks
|
||||
# BEFORE it reads this target. external-dns v0.21.0 reads `target` from the Gateway only, never
|
||||
# from an HTTPRoute — see platform/externaldns/helmrelease.yaml.
|
||||
#
|
||||
# LAN and in-cluster clients are unaffected: PiHole and CoreDNS answer ${BASE_DOMAIN} with
|
||||
# ${GATEWAY_VIP} directly (split-horizon), so they never take this public path.
|
||||
external-dns.alpha.kubernetes.io/target: "wan.${BASE_DOMAIN}"
|
||||
spec:
|
||||
gatewayClassName: cilium
|
||||
listeners:
|
||||
|
||||
Reference in new issue
Block a user