Headlamp's HelmRelease sat in the wait:true `platform` Flux tier. Its pod
crashlooped (`flag provided but not defined: -session-ttl` — chart 0.40.1
renders a container arg the appVersion v0.40.1 backend lacks; upstream
kubernetes-sigs/headlamp#4883), and because it gated `platform`, the whole
chain stalled: platform Ready=Unknown -> platform-config, oauth2-proxy,
apps-wave-0/1/2 and apps-vms all Ready=False "dependency not ready".
Part A (unblock + durable lesson): a leaf UI must never gate the chain. Move
all Headlamp resources into a new non-gating Flux Kustomization
clusters/base/headlamp.yaml (dependsOn platform-config, wait:false, SOPS
decryption patched per-env in clusters/dev) — mirroring oauth2-proxy exactly.
The new Kustomization owns the whole platform/headlamp component; its HTTPRoute
moves from platform-config into platform/headlamp/kustomization.yaml. Removed
headlamp from platform/kustomization.yaml and its httproute from
platform-config/kustomization.yaml. Now a crashlooping Headlamp can never stall
platform/platform-config/apps.
Part B (make it start): pin the chart to 0.41.0 (`>=0.41.0 <0.42.0`). The
backend added `f.Int("session-ttl", …)` in v0.41.0 and chart 0.41.0 pins that
same image (appVersion == chart version), so chart+image realign; every arg the
chart renders for our values is supported. OIDC config + per-user RBAC
(clusterRoleBinding.create:false) unchanged.
Gate: `make -C ci gate` PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add Headlamp (kubernetes-sigs/headlamp Helm chart) at
headlamp.home.keithjackson.dev with native Keycloak OIDC login and PER-USER
RBAC: Headlamp forwards each user's id_token to the k3s apiserver, so the
existing oidc:* group bindings (platform-admins->cluster-admin, viewers->view)
apply per user. The ServiceAccount is deliberately NOT bound to cluster-admin
(clusterRoleBinding.create=false), since user requests ride the forwarded token.
Key integration nuance: the apiserver is configured oidc-client-id=kubernetes,
so it only accepts tokens whose aud contains `kubernetes`. The new `headlamp`
Keycloak client carries an oidc-audience-mapper (included.client.audience=
kubernetes) so tokens carry aud=[headlamp, kubernetes] -- accepted by both
Headlamp's own verifier and the apiserver, leaving the apiserver config
untouched. A groups mapper emits the `groups` claim the apiserver maps to RBAC.
- platform/headlamp: namespace, HelmRepository/HelmRelease, SOPS OIDC secret
(externalSecret envFrom), default-deny NetworkPolicy + Cilium ingress
(gateway)/egress (apiserver+Keycloak) policies, README. Wired into
platform/kustomization.yaml (platform tier).
- platform/headlamp/httproute.yaml wired into platform-config (Gateway API CR,
dependsOn platform), modeled on grafana-httproute.
- keycloak-realm-import: add `headlamp` confidential client + groups/audience
mappers + HEADLAMP_CLIENT_SECRET placeholder. Client also created LIVE via
kcadm (realm import no-ops on the existing realm).
- keycloak-realm-clients.sops: add headlamp-client-secret (mirrored into
headlamp-oidc-credentials clientSecret). Both encrypted to the repo age key.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy is a leaf forward-auth service that depends on Keycloak +
trusted wildcard TLS + split-horizon DNS. It lived in the gating `platform`
Flux Kustomization (wait:true), so on a cold start it Failed until those
dependencies existed, keeping the whole `platform` tier not-Ready. That in
turn blocked `platform-config` — which brings up the very Keycloak instance,
wildcard Certificate and L2 VIP oauth2-proxy needs (and previously carried
oauth2-proxy's own httproute/servicemonitor) — from ever reconciling. A
self-gating cold-start deadlock, twice-proven (DECISIONS-LOG sessions 7 & 9).
Break the cycle by moving the whole oauth2-proxy component out of the
`platform` tier into its own non-gating post-platform Flux Kustomization:
- clusters/base/oauth2-proxy.yaml: new Kustomization -> ./platform/oauth2-proxy,
dependsOn platform-config, wait:false, substituteFrom cluster-vars.
- clusters/base/kustomization.yaml: reference the new Kustomization.
- platform/kustomization.yaml: drop the oauth2-proxy component (no longer in
the gating tier).
- platform/oauth2-proxy/kustomization.yaml: pull in httproute + servicemonitor
so the full component travels together.
- platform-config/kustomization.yaml: drop the in-place oauth2-proxy
httproute/servicemonitor refs (moved with the controller).
- clusters/dev/kustomization.yaml: add the SOPS decryption patch for the new
oauth2-proxy Kustomization (it now owns the encrypted credentials Secret that
the `platform` Kustomization used to decrypt).
Now `platform` reaches Ready without oauth2-proxy; platform-config brings up
Keycloak/TLS/DNS; then oauth2-proxy reconciles against dependencies that
already exist. wait:false guarantees its health can never gate anything above
it. No config of oauth2-proxy itself changed — only its reconciliation tier.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds the VM/CDI storage tier approved by the PM (RWX for live-migration):
- platform/external-snapshotter (NEW, controllers pass): the cluster-singleton
snapshot.storage.k8s.io control plane (CRDs + snapshot-controller in
kube-system), pinned v8.2.0 (Kubernetes deps aligned to 1.31 since v8.1.0;
this cluster is k3s v1.31.5). These CRDs were absent — only snapshots.longhorn.io
and kubevirt.io snapshot CRDs existed.
- platform/longhorn/volumesnapshotclass.yaml (NEW, platform-config): the
driver.longhorn.io VolumeSnapshotClass (type: snap, in-cluster).
- platform/kubevirt/cdi-storageprofiles.yaml (NEW, platform-config): StorageProfile
longhorn -> cloneStrategy: snapshot; StorageProfile longhorn-vm -> snapshot clone
+ claimPropertySets RWX/Block. Non-degraded default virt class clears
CDIDefaultStorageClassDegraded.
- platform/longhorn/storageclasses.yaml: NEW longhorn-vm StorageClass —
migratable: "true" (Longhorn's KubeVirt-live-migration knob; RWX/Block-only on
1.7.3), is-default-virt-class (NOT the k8s default; longhorn stays RWO default).
- platform/kubevirt/kubevirt-cr.yaml: comment only — LiveMigration is GA on v1.9
(no feature gate); no cluster-wide evictionStrategy (would wedge RWO-disk drains);
live-migration stays per-VM opt-in on a longhorn-vm disk.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 3x 6GB nodes are OOM (k3s-dev-2's kubelet stopped; dev-1/dev-3 <600MB free). KubeVirt runs
a full control plane (~1.5-2GB) for ZERO running VMs — its IaaS conformance (R-PLAT-03/04,
R-NET-01, R-STATE-01) is already certified, so disabling the running pods loses no delivered
capability. Re-enable (uncomment kubevirt in platform + platform-config) once the VMs' RAM is
grown. Manifests retained; reversible.
In-cluster clients resolving a platform hostname over its PUBLIC name got the
off-cluster answer, so pod->keycloak.home.keithjackson.dev hit a path that does not
serve the *.home.keithjackson.dev SNI -> `tls: unrecognized name` (oauth2-proxy OIDC
discovery crash; Grafana generic_oauth would hit the same wall).
Add a GitOps-managed coredns-custom ConfigMap (kube-system) that k3s CoreDNS imports as
a `*.server` block. The template plugin answers the whole home.keithjackson.dev zone with
${GATEWAY_VIP} (the shared Gateway VIP), so pod->public-hostname traffic hairpins through
the Gateway with the correct SNI -> wildcard cert validates. CoreDNS reloads the ConfigMap
live (no pod restart). Wired into platform/kustomization.yaml (Phase 2 network core).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Generate the 3 OIDC client secrets (SOPS-encrypted, minimal manifests) so the realm
import can inject the grafana + oauth2-proxy client secrets and both consumers match:
- platform/keycloak/keycloak-realm-clients.sops.yaml (grafana + oauth2-proxy client secrets)
- platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml (client-id/secret/cookie)
- platform/monitoring/grafana-oidc-credentials.sops.yaml (grafana client secret)
Wire each into its component kustomization and re-enable oauth2-proxy in platform +
platform-config (reverses the temp deferral now that its secret exists).
Values generated by PM per operator instruction (random; recorded out-of-repo at
~/.config/home-k3s/keycloak-sso-secrets.txt for later rotation). Encrypted via sops in the
dev container to the age recipient in .sops.yaml; plaintext never committed.
Serves: R-IDP-01, R-IDP-02, R-IDP-04, R-SEC-01.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy's HelmRelease mounts existingSecret oauth2-proxy-credentials, an operator
hand-off not yet filled. Under the unified graph (platform wait:true), the not-ready HR
blocks platform -> platform-config -> the whole downstream graph including KubeVirt.
Deferring it (comment its line in platform/kustomization.yaml + its two entries in
platform-config) lets the no-secret components (KubeVirt/CDI, CNPG, Keycloak operator +
instance) reconcile and be certified now. Re-enable both together once the operator fills
platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml.
Serves: R-IDP-02 (deferred). PM sequencing correction — enable per phase AND readiness.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The in-cluster Gluetun gateway is retired; VPN egress is now a node-level host
OpenVPN split-tunnel (Ansible role vpn_egress_gateway) steered by a Cilium
Egress Gateway policy with a host nftables kill-switch (ADR-0015). Credentials
moved host-side to the Ansible vault. Update docs/comments only:
- platform/vpn-egress/README.md: rewrite for ADR-0015 (live + conformance-green);
drop the resolved "option 1 not viable with Gluetun" finding to a brief history note.
- platform/kustomization.yaml: Phase 3 comment -> node tunnel + Cilium steering.
- platform/kyverno/.../require-egress-class-label.yaml: ADR-0013 -> ADR-0015 steering.
- schemas/servicedefinition.schema.json: egress=vpn description -> node tunnel + Cilium.
- clusters/base/cluster-vars.yaml: reframe PIA_REGION / CLUSTER_POD_CIDR WHY comments
to ADR-0015 (Cilium excludedCIDRs); both keys retained (still referenced/documented).
- platform/vpn-egress/namespace.yaml: no privileged gateway pod under ADR-0015.
- pia-credentials.sops.yaml / pia-custom-ovpn.sops.yaml: comment headers only note
they are now UNUSED (creds host-side), retained pending cleanup; payloads untouched.
No runtime manifest behavior, alert exprs, kustomization resources, or SOPS payloads
changed. No spec-repo edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CORRECTION: the connection failures were gluetun dialing RETIRED PIA IPs (46.246.x
= dead → timeouts, which I mis-read as a network block). A fresh PIA .ovpn connects
from the same host, and PIA's current SE Stockholm servers (158.173.x) are reachable.
Cilium was independently cleared (default-allow, host-fw off, Hubble FORWARDED).
Fix: UPDATER_PERIOD=24h so gluetun fetches PIA's current servers; keep
OPENVPN_MTU/MSSFIX=1350 for the real Cilium-VXLAN 1450 pod MTU (data channel);
PIA default UDP port. Re-enable vpn-egress. Full correction in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Decisive diagnosis: OpenVPN's outbound init packets are 14 bytes with zero
replies (so MTU was never the blocker), pod UDP egress is healthy (NTP replies),
but TCP 501/443 to PIA IPs time out while Google/Cloudflare are OPEN — across many
PIA servers. This 192.168.2.x WiFi segment blackholes PIA's IP ranges (and
intercepts DNS). Not code-fixable. Reverted the port/MTU red-herring env to PIA
defaults; disabled vpn-egress pending an operator network fix (move to the
unfiltered 192.168.1.x LAN, or unblock PIA on the router). Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Actual root cause (operator-directed layered MTU check): Cilium runs VXLAN
(routing-mode tunnel) so pods get MTU 1450 (1500-50 encap) while node, hypervisor,
and the Google Fiber WAN are all 1500. OpenVPN's ~1500-byte handshake packets
exceed 1450 and are dropped (PMTUD broken). The ports were a red herring.
Fix: OPENVPN_MTU=1350 + OPENVPN_MSSFIX=1350 so encrypted packets (~1402) fit under
1450. Keeps OpenVPN + VXLAN unchanged. Re-enable vpn-egress. Full layered trace
(pod/node/host) in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
OpenVPN can't connect on this network (each port fails differently, so it's
OpenVPN-specific not a UDP block — NTP/123 works): 1197 = home OpenVPN-server
conflict; 53 = DNS interception; 8080 = TLS handshake fails across servers (MTU
on WiFi+triple-NAT). Recommend pivoting to WireGuard via thrnz/docker-wireguard-pia
(UDP 1337, MTU-robust, ADR-0013 fallback, operator's original preference) — a
spec-level (ADR-0013) change, so escalated. Disabled meanwhile. Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Port 53 also got no PIA replies: this network transparently intercepts outbound
:53, so dig works but OpenVPN-over-53 is swallowed. An NTP/123 probe from the host
returned fine, proving arbitrary non-53 UDP works bidirectionally (so it's
port-specific: 1197 conflict + 53 DNS-intercept, not a UDP block). Move to 8080
(community-verified for PIA, non-DNS, non-1197).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Wiring the port via "${PIA_OPENVPN_PORT}" made Flux/kustomize emit an unquoted
53 (int) after substitution, which k8s rejects for an env value ("expected
string, got 53"). Same gotcha as the Longhorn numberOfReplicas fix. Hardcode the
quoted string; cluster-var removed with a NOTE explaining why.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator identified the block: the home network NATs its own OpenVPN server on
WAN:1197, colliding with the cluster's outbound PIA-on-1197 flow (port-specific,
not a general UDP block — UDP/53 worked in tcpdump). PIA accepts OpenVPN on
53/1194/1197/1198/8080/9201; chose 53 (empirically proven to traverse this net).
- cluster-vars: add PIA_OPENVPN_PORT="53"
- gluetun: OPENVPN_ENDPOINT_PORT=${PIA_OPENVPN_PORT}
- re-enable vpn-egress; DECISIONS-LOG resolution noted
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Phase-3 VPN config is complete + correct (OpenVPN-UDP, SE Stockholm), but the
OpenVPN UDP/1197 handshake to PIA never gets a reply. Traced with tcpdump: packets
leave pod→node→host→WiFi (SNAT 192.168.2.103) fine, but 0 replies return to the
host, while host UDP/53 works. The 192.168.2.x home router/ISP drops OpenVPN UDP —
upstream of everything in this repo. Reverted to disabled to avoid a crashloop
firing false KubePodCrashLooping alerts; one-line re-enable once the network is
fixed. Full trace in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gluetun cannot do PIA over WireGuard (upstream unsupported; live "provider name
is not valid for Wireguard"). Operator approved OpenVPN-UDP and ADR-0013 was
amended in the spec. This aligns the impl:
- platform/kustomization.yaml: re-enable `- vpn-egress` (Phase 3, OpenVPN-UDP)
- gluetun-gateway.yaml: clean stale WireGuard comments (header, image note,
VPN_TYPE/creds comments); config already VPN_TYPE=openvpn
- README.md: PIA-over-OpenVPN, status=enabled, steering iface wg0 -> tun0
- DECISIONS-LOG.md: disk/drive incident (k3s -> NVMe; failing Seagate) + Phase-3
OpenVPN resolution
Stale WireGuard comments inside the operator's encrypted pia-credentials.sops.yaml
header left untouched (editing breaks the SOPS MAC; cosmetic).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
PIA creds provisioned (SOPS) + wired. Enable vpn-egress in platform/kustomization.yaml
to deploy the shared Gluetun gateway (PIA over WireGuard, Sweden region, kill-switch).
Egress-class steering (vpn-class pods -> tunnel) still to be validated against the live
tunnel next. Gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Enable monitoring in platform/kustomization.yaml (Phase 4). Grafana admin +
Alertmanager SMTP secrets are provisioned (SOPS) and wired; Longhorn SC is up for
Prometheus/Loki PVCs. Set the Alertmanager receiver 'to:' to a valid inbox
([email protected]) so the config loads (REPLACE-ME is not a valid address);
operator can change it. NOTE: smtp_from ([email protected]) differs from
the Gmail auth user — Gmail may reject/rewrite unless a send-as alias is configured;
that only affects email delivery at alert time, not stack health.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The prior integration commit enabled kyverno in the platform render, but (a) the
admission bundle used ../policies refs that fail default kustomize load
restrictions, and (b) admission enforcement is Phase 2 (T-2.7), not Phase 0.
Fix: platform/kyverno/kustomization.yaml now lists resources with downward paths
(builds standalone); platform/kustomization.yaml re-defers kyverno with a Phase-2
note. All Phase-0 gates green: kustomize (platform, platform/kyverno, dev),
schema validation, conformance scaffold.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add platform/kyverno/kustomization.yaml (renders the admission enforcing bundle)
and enable the kyverno component in platform/kustomization.yaml (first, so
admission policies precede other components). kustomize build verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 4.8