Cilium was one of ~12 components inside a single `platform` Kustomization with wait:true,
so the CNI's health was indistinguishable from eleven unrelated components in the tier's
Ready condition, and the only emergency valve was freezing the whole platform tier. This
repo already set the precedent in reverse (oauth2-proxy and headlamp were split OUT so
unhealthy leaves would not gate a tier); Cilium is the mirror image — the one component
that legitimately SHOULD gate everything.
New clusters/base/cilium.yaml: path ./platform/cilium, wait:true, prune:true, interval 1h
(a CNI does not need a 10m re-render cadence; kustomize-controller is source-triggered so
a push still reconciles immediately — the interval is only the drift-correction floor),
timeout 10m (a 3-node DaemonSet rollout with wait exceeds the 5m default),
dependsOn:[flux-system], and postBuild.substituteFrom: cluster-vars — the last is
LOAD-BEARING, not decorative: helmrelease.yaml references ${K3S_API_HOST} and
kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so
without it this Kustomization would fail immediately.
*** THE HAZARD WAS FAR WORSE THAN "the CNI gets pruned" ***
platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system anywhere
under ./platform, and the live object carries kustomize.toolkit.fluxcd.io/name=platform.
With prune:true, the moment ./platform/cilium leaves platform's path the stale set is not
"the Cilium HelmRelease" — it is `kubectl delete namespace kube-system`, taking coredns,
metrics-server, local-path AND Cilium. A full cluster kill, not a CNI kill.
TWO INDEPENDENT PROTECTIONS, deliberately not trusting either alone:
1. prune:false on `platform` for this transition merge — Flux's documented Procedure 1 for
moving resources between Kustomizations. Chosen over the per-resource
`kustomize.toolkit.fluxcd.io/prune: disabled` annotation because that needs THREE merges
and leaves the new Kustomization owning resources it can never prune.
2. The pruner's own owner-label filter. In v1.4.0, prune() passes Inclusions:
GetOwnerLabels(obj.Name...), so platform only deletes objects whose LIVE labels still
say name=platform. The new cilium Kustomization relabels all three to name=cilium on
apply, and platform.dependsOn:[cilium] is REVISION-GATED, so platform provably cannot
reconcile revision N until cilium has APPLIED revision N. The relabel deterministically
precedes the prune.
Protection 2 alone would probably have sufficed. Not betting kube-system on "probably".
NOTED FOR THE FUTURE, in-file: anyone moving a platform/* component must first check
whether it is the sole declarer of a shared Namespace — that turns a component-scoped
prune into a cluster-scoped one.
Verified: `kubectl kustomize platform` goes 88 -> 85 documents, 3 removed / 0 added /
0 CHANGED, and `kubectl kustomize platform/cilium` reproduces those 3 byte-identical
(sha256-matched). A pure ownership move; no Cilium value touched. L0 substitution gate
103 assertions (was 102), new cilium row, 0 hazards. Server-side dry-run against the live
cluster accepted both CRs. clusters/prod renders clean off the same base.
MERGE 2 (restore prune:true) follows only after verifying: 13/13 Ready; kube-system, the
HelmRelease and the HelmRepository all relabelled to name=cilium; platform's inventory no
longer lists them; and Cilium pod ages/restarts UNCHANGED from
cilium-2f8h8 r=0 2026-09-01T05:55:27Z / cilium-425m8 r=1 2026-08-30T13:25:10Z /
cilium-54cww r=0 2026-09-01T05:55:26Z, HelmRelease v11.
Serves: R-PLAT-01, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Headlamp's HelmRelease sat in the wait:true `platform` Flux tier. Its pod
crashlooped (`flag provided but not defined: -session-ttl` — chart 0.40.1
renders a container arg the appVersion v0.40.1 backend lacks; upstream
kubernetes-sigs/headlamp#4883), and because it gated `platform`, the whole
chain stalled: platform Ready=Unknown -> platform-config, oauth2-proxy,
apps-wave-0/1/2 and apps-vms all Ready=False "dependency not ready".
Part A (unblock + durable lesson): a leaf UI must never gate the chain. Move
all Headlamp resources into a new non-gating Flux Kustomization
clusters/base/headlamp.yaml (dependsOn platform-config, wait:false, SOPS
decryption patched per-env in clusters/dev) — mirroring oauth2-proxy exactly.
The new Kustomization owns the whole platform/headlamp component; its HTTPRoute
moves from platform-config into platform/headlamp/kustomization.yaml. Removed
headlamp from platform/kustomization.yaml and its httproute from
platform-config/kustomization.yaml. Now a crashlooping Headlamp can never stall
platform/platform-config/apps.
Part B (make it start): pin the chart to 0.41.0 (`>=0.41.0 <0.42.0`). The
backend added `f.Int("session-ttl", …)` in v0.41.0 and chart 0.41.0 pins that
same image (appVersion == chart version), so chart+image realign; every arg the
chart renders for our values is supported. OIDC config + per-user RBAC
(clusterRoleBinding.create:false) unchanged.
Gate: `make -C ci gate` PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add Headlamp (kubernetes-sigs/headlamp Helm chart) at
headlamp.home.keithjackson.dev with native Keycloak OIDC login and PER-USER
RBAC: Headlamp forwards each user's id_token to the k3s apiserver, so the
existing oidc:* group bindings (platform-admins->cluster-admin, viewers->view)
apply per user. The ServiceAccount is deliberately NOT bound to cluster-admin
(clusterRoleBinding.create=false), since user requests ride the forwarded token.
Key integration nuance: the apiserver is configured oidc-client-id=kubernetes,
so it only accepts tokens whose aud contains `kubernetes`. The new `headlamp`
Keycloak client carries an oidc-audience-mapper (included.client.audience=
kubernetes) so tokens carry aud=[headlamp, kubernetes] -- accepted by both
Headlamp's own verifier and the apiserver, leaving the apiserver config
untouched. A groups mapper emits the `groups` claim the apiserver maps to RBAC.
- platform/headlamp: namespace, HelmRepository/HelmRelease, SOPS OIDC secret
(externalSecret envFrom), default-deny NetworkPolicy + Cilium ingress
(gateway)/egress (apiserver+Keycloak) policies, README. Wired into
platform/kustomization.yaml (platform tier).
- platform/headlamp/httproute.yaml wired into platform-config (Gateway API CR,
dependsOn platform), modeled on grafana-httproute.
- keycloak-realm-import: add `headlamp` confidential client + groups/audience
mappers + HEADLAMP_CLIENT_SECRET placeholder. Client also created LIVE via
kcadm (realm import no-ops on the existing realm).
- keycloak-realm-clients.sops: add headlamp-client-secret (mirrored into
headlamp-oidc-credentials clientSecret). Both encrypted to the repo age key.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy is a leaf forward-auth service that depends on Keycloak +
trusted wildcard TLS + split-horizon DNS. It lived in the gating `platform`
Flux Kustomization (wait:true), so on a cold start it Failed until those
dependencies existed, keeping the whole `platform` tier not-Ready. That in
turn blocked `platform-config` — which brings up the very Keycloak instance,
wildcard Certificate and L2 VIP oauth2-proxy needs (and previously carried
oauth2-proxy's own httproute/servicemonitor) — from ever reconciling. A
self-gating cold-start deadlock, twice-proven (DECISIONS-LOG sessions 7 & 9).
Break the cycle by moving the whole oauth2-proxy component out of the
`platform` tier into its own non-gating post-platform Flux Kustomization:
- clusters/base/oauth2-proxy.yaml: new Kustomization -> ./platform/oauth2-proxy,
dependsOn platform-config, wait:false, substituteFrom cluster-vars.
- clusters/base/kustomization.yaml: reference the new Kustomization.
- platform/kustomization.yaml: drop the oauth2-proxy component (no longer in
the gating tier).
- platform/oauth2-proxy/kustomization.yaml: pull in httproute + servicemonitor
so the full component travels together.
- platform-config/kustomization.yaml: drop the in-place oauth2-proxy
httproute/servicemonitor refs (moved with the controller).
- clusters/dev/kustomization.yaml: add the SOPS decryption patch for the new
oauth2-proxy Kustomization (it now owns the encrypted credentials Secret that
the `platform` Kustomization used to decrypt).
Now `platform` reaches Ready without oauth2-proxy; platform-config brings up
Keycloak/TLS/DNS; then oauth2-proxy reconciles against dependencies that
already exist. wait:false guarantees its health can never gate anything above
it. No config of oauth2-proxy itself changed — only its reconciliation tier.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds the VM/CDI storage tier approved by the PM (RWX for live-migration):
- platform/external-snapshotter (NEW, controllers pass): the cluster-singleton
snapshot.storage.k8s.io control plane (CRDs + snapshot-controller in
kube-system), pinned v8.2.0 (Kubernetes deps aligned to 1.31 since v8.1.0;
this cluster is k3s v1.31.5). These CRDs were absent — only snapshots.longhorn.io
and kubevirt.io snapshot CRDs existed.
- platform/longhorn/volumesnapshotclass.yaml (NEW, platform-config): the
driver.longhorn.io VolumeSnapshotClass (type: snap, in-cluster).
- platform/kubevirt/cdi-storageprofiles.yaml (NEW, platform-config): StorageProfile
longhorn -> cloneStrategy: snapshot; StorageProfile longhorn-vm -> snapshot clone
+ claimPropertySets RWX/Block. Non-degraded default virt class clears
CDIDefaultStorageClassDegraded.
- platform/longhorn/storageclasses.yaml: NEW longhorn-vm StorageClass —
migratable: "true" (Longhorn's KubeVirt-live-migration knob; RWX/Block-only on
1.7.3), is-default-virt-class (NOT the k8s default; longhorn stays RWO default).
- platform/kubevirt/kubevirt-cr.yaml: comment only — LiveMigration is GA on v1.9
(no feature gate); no cluster-wide evictionStrategy (would wedge RWO-disk drains);
live-migration stays per-VM opt-in on a longhorn-vm disk.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 3x 6GB nodes are OOM (k3s-dev-2's kubelet stopped; dev-1/dev-3 <600MB free). KubeVirt runs
a full control plane (~1.5-2GB) for ZERO running VMs — its IaaS conformance (R-PLAT-03/04,
R-NET-01, R-STATE-01) is already certified, so disabling the running pods loses no delivered
capability. Re-enable (uncomment kubevirt in platform + platform-config) once the VMs' RAM is
grown. Manifests retained; reversible.
In-cluster clients resolving a platform hostname over its PUBLIC name got the
off-cluster answer, so pod->keycloak.home.keithjackson.dev hit a path that does not
serve the *.home.keithjackson.dev SNI -> `tls: unrecognized name` (oauth2-proxy OIDC
discovery crash; Grafana generic_oauth would hit the same wall).
Add a GitOps-managed coredns-custom ConfigMap (kube-system) that k3s CoreDNS imports as
a `*.server` block. The template plugin answers the whole home.keithjackson.dev zone with
${GATEWAY_VIP} (the shared Gateway VIP), so pod->public-hostname traffic hairpins through
the Gateway with the correct SNI -> wildcard cert validates. CoreDNS reloads the ConfigMap
live (no pod restart). Wired into platform/kustomization.yaml (Phase 2 network core).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Generate the 3 OIDC client secrets (SOPS-encrypted, minimal manifests) so the realm
import can inject the grafana + oauth2-proxy client secrets and both consumers match:
- platform/keycloak/keycloak-realm-clients.sops.yaml (grafana + oauth2-proxy client secrets)
- platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml (client-id/secret/cookie)
- platform/monitoring/grafana-oidc-credentials.sops.yaml (grafana client secret)
Wire each into its component kustomization and re-enable oauth2-proxy in platform +
platform-config (reverses the temp deferral now that its secret exists).
Values generated by PM per operator instruction (random; recorded out-of-repo at
~/.config/home-k3s/keycloak-sso-secrets.txt for later rotation). Encrypted via sops in the
dev container to the age recipient in .sops.yaml; plaintext never committed.
Serves: R-IDP-01, R-IDP-02, R-IDP-04, R-SEC-01.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy's HelmRelease mounts existingSecret oauth2-proxy-credentials, an operator
hand-off not yet filled. Under the unified graph (platform wait:true), the not-ready HR
blocks platform -> platform-config -> the whole downstream graph including KubeVirt.
Deferring it (comment its line in platform/kustomization.yaml + its two entries in
platform-config) lets the no-secret components (KubeVirt/CDI, CNPG, Keycloak operator +
instance) reconcile and be certified now. Re-enable both together once the operator fills
platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml.
Serves: R-IDP-02 (deferred). PM sequencing correction — enable per phase AND readiness.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The in-cluster Gluetun gateway is retired; VPN egress is now a node-level host
OpenVPN split-tunnel (Ansible role vpn_egress_gateway) steered by a Cilium
Egress Gateway policy with a host nftables kill-switch (ADR-0015). Credentials
moved host-side to the Ansible vault. Update docs/comments only:
- platform/vpn-egress/README.md: rewrite for ADR-0015 (live + conformance-green);
drop the resolved "option 1 not viable with Gluetun" finding to a brief history note.
- platform/kustomization.yaml: Phase 3 comment -> node tunnel + Cilium steering.
- platform/kyverno/.../require-egress-class-label.yaml: ADR-0013 -> ADR-0015 steering.
- schemas/servicedefinition.schema.json: egress=vpn description -> node tunnel + Cilium.
- clusters/base/cluster-vars.yaml: reframe PIA_REGION / CLUSTER_POD_CIDR WHY comments
to ADR-0015 (Cilium excludedCIDRs); both keys retained (still referenced/documented).
- platform/vpn-egress/namespace.yaml: no privileged gateway pod under ADR-0015.
- pia-credentials.sops.yaml / pia-custom-ovpn.sops.yaml: comment headers only note
they are now UNUSED (creds host-side), retained pending cleanup; payloads untouched.
No runtime manifest behavior, alert exprs, kustomization resources, or SOPS payloads
changed. No spec-repo edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CORRECTION: the connection failures were gluetun dialing RETIRED PIA IPs (46.246.x
= dead → timeouts, which I mis-read as a network block). A fresh PIA .ovpn connects
from the same host, and PIA's current SE Stockholm servers (158.173.x) are reachable.
Cilium was independently cleared (default-allow, host-fw off, Hubble FORWARDED).
Fix: UPDATER_PERIOD=24h so gluetun fetches PIA's current servers; keep
OPENVPN_MTU/MSSFIX=1350 for the real Cilium-VXLAN 1450 pod MTU (data channel);
PIA default UDP port. Re-enable vpn-egress. Full correction in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Decisive diagnosis: OpenVPN's outbound init packets are 14 bytes with zero
replies (so MTU was never the blocker), pod UDP egress is healthy (NTP replies),
but TCP 501/443 to PIA IPs time out while Google/Cloudflare are OPEN — across many
PIA servers. This 192.168.2.x WiFi segment blackholes PIA's IP ranges (and
intercepts DNS). Not code-fixable. Reverted the port/MTU red-herring env to PIA
defaults; disabled vpn-egress pending an operator network fix (move to the
unfiltered 192.168.1.x LAN, or unblock PIA on the router). Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Actual root cause (operator-directed layered MTU check): Cilium runs VXLAN
(routing-mode tunnel) so pods get MTU 1450 (1500-50 encap) while node, hypervisor,
and the Google Fiber WAN are all 1500. OpenVPN's ~1500-byte handshake packets
exceed 1450 and are dropped (PMTUD broken). The ports were a red herring.
Fix: OPENVPN_MTU=1350 + OPENVPN_MSSFIX=1350 so encrypted packets (~1402) fit under
1450. Keeps OpenVPN + VXLAN unchanged. Re-enable vpn-egress. Full layered trace
(pod/node/host) in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
OpenVPN can't connect on this network (each port fails differently, so it's
OpenVPN-specific not a UDP block — NTP/123 works): 1197 = home OpenVPN-server
conflict; 53 = DNS interception; 8080 = TLS handshake fails across servers (MTU
on WiFi+triple-NAT). Recommend pivoting to WireGuard via thrnz/docker-wireguard-pia
(UDP 1337, MTU-robust, ADR-0013 fallback, operator's original preference) — a
spec-level (ADR-0013) change, so escalated. Disabled meanwhile. Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Port 53 also got no PIA replies: this network transparently intercepts outbound
:53, so dig works but OpenVPN-over-53 is swallowed. An NTP/123 probe from the host
returned fine, proving arbitrary non-53 UDP works bidirectionally (so it's
port-specific: 1197 conflict + 53 DNS-intercept, not a UDP block). Move to 8080
(community-verified for PIA, non-DNS, non-1197).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Wiring the port via "${PIA_OPENVPN_PORT}" made Flux/kustomize emit an unquoted
53 (int) after substitution, which k8s rejects for an env value ("expected
string, got 53"). Same gotcha as the Longhorn numberOfReplicas fix. Hardcode the
quoted string; cluster-var removed with a NOTE explaining why.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator identified the block: the home network NATs its own OpenVPN server on
WAN:1197, colliding with the cluster's outbound PIA-on-1197 flow (port-specific,
not a general UDP block — UDP/53 worked in tcpdump). PIA accepts OpenVPN on
53/1194/1197/1198/8080/9201; chose 53 (empirically proven to traverse this net).
- cluster-vars: add PIA_OPENVPN_PORT="53"
- gluetun: OPENVPN_ENDPOINT_PORT=${PIA_OPENVPN_PORT}
- re-enable vpn-egress; DECISIONS-LOG resolution noted
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Phase-3 VPN config is complete + correct (OpenVPN-UDP, SE Stockholm), but the
OpenVPN UDP/1197 handshake to PIA never gets a reply. Traced with tcpdump: packets
leave pod→node→host→WiFi (SNAT 192.168.2.103) fine, but 0 replies return to the
host, while host UDP/53 works. The 192.168.2.x home router/ISP drops OpenVPN UDP —
upstream of everything in this repo. Reverted to disabled to avoid a crashloop
firing false KubePodCrashLooping alerts; one-line re-enable once the network is
fixed. Full trace in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gluetun cannot do PIA over WireGuard (upstream unsupported; live "provider name
is not valid for Wireguard"). Operator approved OpenVPN-UDP and ADR-0013 was
amended in the spec. This aligns the impl:
- platform/kustomization.yaml: re-enable `- vpn-egress` (Phase 3, OpenVPN-UDP)
- gluetun-gateway.yaml: clean stale WireGuard comments (header, image note,
VPN_TYPE/creds comments); config already VPN_TYPE=openvpn
- README.md: PIA-over-OpenVPN, status=enabled, steering iface wg0 -> tun0
- DECISIONS-LOG.md: disk/drive incident (k3s -> NVMe; failing Seagate) + Phase-3
OpenVPN resolution
Stale WireGuard comments inside the operator's encrypted pia-credentials.sops.yaml
header left untouched (editing breaks the SOPS MAC; cosmetic).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
PIA creds provisioned (SOPS) + wired. Enable vpn-egress in platform/kustomization.yaml
to deploy the shared Gluetun gateway (PIA over WireGuard, Sweden region, kill-switch).
Egress-class steering (vpn-class pods -> tunnel) still to be validated against the live
tunnel next. Gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Enable monitoring in platform/kustomization.yaml (Phase 4). Grafana admin +
Alertmanager SMTP secrets are provisioned (SOPS) and wired; Longhorn SC is up for
Prometheus/Loki PVCs. Set the Alertmanager receiver 'to:' to a valid inbox
([email protected]) so the config loads (REPLACE-ME is not a valid address);
operator can change it. NOTE: smtp_from ([email protected]) differs from
the Gmail auth user — Gmail may reject/rewrite unless a send-as alias is configured;
that only affects email delivery at alert time, not stack health.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The prior integration commit enabled kyverno in the platform render, but (a) the
admission bundle used ../policies refs that fail default kustomize load
restrictions, and (b) admission enforcement is Phase 2 (T-2.7), not Phase 0.
Fix: platform/kyverno/kustomization.yaml now lists resources with downward paths
(builds standalone); platform/kustomization.yaml re-defers kyverno with a Phase-2
note. All Phase-0 gates green: kustomize (platform, platform/kyverno, dev),
schema validation, conformance scaffold.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add platform/kyverno/kustomization.yaml (renders the admission enforcing bundle)
and enable the kyverno component in platform/kustomization.yaml (first, so
admission policies precede other components). kustomize build verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 5