home-network-k3s-impl

0 Fork 0

Author	SHA1	Message	Date
jakman85andClaude Opus 5	1a67fc1b3c	feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers) gate / gate (pull_request) Failing after 1m48s Details gate / conformance (pull_request) Has been skipped Details Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and .home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/* , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-05 22:06:18 -05:00
jakman85andClaude Opus 5	0aa1a0a78a	feat(plex): one advertised hostname for local+remote, its own LE cert, and a LAN definition gate / gate (pull_request) Successful in 1m49s Details gate / conformance (pull_request) Has been skipped Details Two symptoms after the publishing fix landed: 1. LAN clients were reported as Remote. Measured root cause: LanNetworksBandwidth was ABSENT, so Plex had no notion of a local network and classified everything WAN -- 468 requests logged "Allowed Network (WAN)", ZERO "(LAN)", including Seerr from inside the cluster. allowedNetworks only waives AUTH; it does not make a client local. Two settings, and only one has an env hook. 2. The advertised URL hard-coded both the VIP and the server's cert hash into a cluster-var -- two values nothing kept in sync. Operator design: ONE hostname, plex-vip.${BASE_DOMAIN}, split-horizon -- PiHole answers it with ${PLEX_L4_VIP} on the LAN, Cloudflare answers it publicly with the WAN address NATed to the same VIP. - catalog/plex.yaml: ADVERTISE_IP -> https://plex-vip.${BASE_DOMAIN}:32400, plus a level-triggered postStart hook (pihole/qbittorrent idiom) that re-asserts LanNetworksBandwidth and the three customCertificate* prefs every start, converting PEM -> PKCS#12 so cert renewals need no extra part. - apps/wave-0/plex-cert.yaml: cert-manager Certificate for exactly that name from letsencrypt-prod, so the advertised https URL is servable. - schema 1.11.0: `optional` on secretKeyRef/configMapKeyRef. Load-bearing -- the cert is issued INTO the plex namespace, which the rendered plex.yaml itself creates, so it lands in the same reconcile; hard refs would have held Plex in CreateContainerConfigError and, since apps-wave-0 is wait:true, stalled the whole app tier on a slow DNS01 challenge. - catalog/pihole.yaml: the plex-vip split-horizon answer. Required -- measured, the name resolved to the Gateway VIP where nothing listens on 32400 (http=000). - PLEX_ADVERTISE_URL deleted; the hostname removes the cert-hash fragility. Corrects my own earlier claim that a SAN match is REQUIRED: it is not (httpsRequired=0, secureConnections defaults to Preferred, so clients can fall back to cleartext). A matching cert keeps the remote leg ENCRYPTED. Hence: obtain a real cert for the advertised name rather than pick the name to suit Plex's built-in one. UNVERIFIED, and stated plainly in the handoff: nothing here has run. Cert issuance for this name is inferred from the wildcard's success, and whether Plex accepts an OpenSSL-3 PKCS#12 bundle is untested -- the largest risk. Touches clusters/** -> sensitive paths -> needs a non-author approval or the operator's /approve-override; deliberately not self-posted (CLAUDE.md 1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 03:37:57 -05:00
jakman85andClaude Opus 5	1f4552d409	fix(plex): advertise the plex.direct name, not the bare VIP (server vanished from the plex.tv account) gate / gate (pull_request) Successful in 1m48s Details gate / conformance (pull_request) Has been skipped Details customConnections held two hand-typed URLs and neither could ever complete, so Plex had nothing publishable and the server disappeared from the account: - https://192.168.2.24:32400 -- the server's cert is .130072f20c334d6c8f63d0dfef87309b.plex.direct with NO IP SAN, so bare-IP https fails TLS for every client. Measured: a continuous storm of "CERT: incomplete TLS handshake ... certificate unknown" from real LAN clients arriving through the plex-l4-tcp VIP. - https://plex.home.keithjackson.dev:32400 -- nothing listens; the Gateway serves that host on 443 (measured http=000 against 192.168.2.20:32400). With no completable connection plex.tv published nothing ("Mapped - Not Published"), Plex fell back to Relay, and clients reported "not authorized". Fix: advertise <dashed-ip>.<serverhash>.plex.direct -- public DNS that resolves to the embedded address, so it points at the LAN VIP and matches the wildcard cert. Verified with full TLS validation (no curl -k): 200 + verify OK, where the bare-IP form fails to connect. Declared as ADVERTISE_IP so git owns the value instead of a web form: the image's 40-plex-first-run writes it into customConnections on every pod start, outside the claim guard. Does NOT fix off-LAN access -- the advertised address is RFC1918 and the WAN tcp/32400 forward remains unverified. ManualPortMappingMode=1 with an absent port is left as-is (no GitOps path exists for it). Touches clusters/* -> sensitive paths -> needs the operator's /approve-override, deliberately not self-posted (CLAUDE.md 1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 02:53:01 -05:00
jakman85andClaude Opus 5	fbaeb3e038	feat(plex): expose tcp/32400 for remote access via a pinned LoadBalancer VIP gate / gate (pull_request) Successful in 1m49s Details gate / conformance (pull_request) Has been skipped Details Remote Plex clients had no path in, for two measured reasons: Plex advertised no remote connection at all (customConnections and ManualPortMappingMode both ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so 32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this server on a client's connection list. Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS with .plex.direct and streams bypass the Envoy L7 hop. Adds a hand-written non-catalog unit (the SD schema cannot express a LoadBalancer Service or a pinned lbipam VIP), registered in render.py's EXTRA_UNITS so --emit-waves places it in plex's own wave. externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather than re-deriving it. Known consequence, documented: SNAT puts remote clients inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change. Remote access does NOT work until the operator does two manual steps that cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually specify public port" toggle (no env var exists for it; verified NONE in the image's init surface). runbooks/plex-remote-access.md is the procedure. Touches clusters/* -- a sensitive path -- so the approval-policy stage will block pending an operator /approve-override. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-03 23:30:46 -05:00
jakman85andClaude Opus 5	4e1b169256	feat(gpu): Intel GPU device plugin v0.37.0, in its own Flux Kustomization gate / gate (pull_request) Successful in 1m41s Details gate / conformance (pull_request) Has been skipped Details Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured fix for the Plex hardware-transcoding blocker, and it only makes the resource EXIST — no consumer is switched over here. WHY. The `/dev/dri` hostPath merged on main @ `cdf3e14` makes the device node visible and not openable. Measured in the running pod on k3s-prod-2: uid 0, O_RDWR -> EPERM "Operation not permitted" uid 1000, O_RDWR -> EPERM "Operation not permitted" EPERM from the device cgroup, not EACCES from file permissions — root on the HOST opens the same device rc=0, the node is unified cgroup2fs, and the container's runc config.json carries one deny-all rule with `linux.devices: None`. A hostPath is a filesystem operation: it makes the device node appear and adds no cgroup allow rule, because Kubernetes cannot infer one from a volume. A device plugin's Allocate response is what makes kubelet populate `linux.devices` AND the matching cgroup allow rule — verified in the v0.37.0 source, which emits `DeviceSpec{HostPath, ContainerPath, Permissions: "rw"}` per DRM node. It is the only non-privileged way to get there; `privileged: true` is blocked by Kyverno restrict-host-namespaces under Enforce in app namespaces, and `plex` is labelled app-namespace=true. PLACEMENT: its own Kustomization (dependsOn cilium, wait:true), not a line in platform/kustomization.yaml. The shared tier was cheaper and needed no human reviewer on approval-policy.json's terms, but it is wait:true over eleven-plus components and the repo already argued this when Cilium was split out: its Ready condition cannot tell "the plugin is wedged" from "longhorn is slow", the only emergency valve is freezing the whole tier, and the tier's 10m/5m interval/timeout are called out there as wrong for a 3-node DaemonSet rollout — which is exactly what this is. Worse here than average: platform gates platform-config gates apps-wave-0 (wait:true) gates apps-wave-1, so a plugin that never went healthy inside it would stall every app — a component added to fix a silent no-op would have gained the power to cause a real outage. Touching clusters/** costs nothing extra: required_approvals:1 with block_admin_merge_override:true applies to every PR here regardless of path. wait:true is a health SIGNAL, not a gate — nothing in the rendered graph dependsOn this, so it gates nothing, while making `flux get kustomization intel-gpu-plugin` answer "did it come up on all three nodes". NON-CHOICES, each argued in the README: standalone DaemonSet over the operator (no CRD, controller-manager, webhook or second namespace to keep healthy); `--shared-dev-num` left at the upstream default of 1 (a higher count is a lie the scheduler believes — no GPU-side partitioning backs it, so it converts a loud Unschedulable into a quiet "transcodes are slow"); `--health-management` left OFF, the one flag that would open a device and reintroduce this EPERM; no NFD overlay (all three nodes identical); no supplementalGroups (the pms-docker cont-init hook already adds `plex` to each device's group, measured live as Groups: 44 100 992 1000). The DaemonSet is vendored as `kubectl kustomize` output of upstream v0.37.0's gpu_plugin/base, rather than referenced as a remote kustomize base, for the reason external-snapshotter records: a remote base is git-fetched on every reconcile and those fetches have flaked a whole tier. It is generated rather than byte-copied because upstream mixes YAML sequence-indentation styles within the one document and ci/yamllint.yaml sets `indent-sequences: consistent`, so a verbatim copy is a hard lint ERROR — measured, not guessed: the first gate run of this branch returned rc 1 BLOCKED on exactly that. The transform is proven formatting-only (`yq -P 'sort_keys(..)'` of both diffs clean). Image pinned to docker.io/intel/intel-gpu-plugin:0.37.0 (digest verified present: sha256:7001200071db10587fc31184eedda4f5cf291efba737adf85a8c80539f7fa5e1). ORDERING — DO NOT REORDER. catalog/render.py, catalog/plex.yaml and apps/wave-0/plex.yaml are untouched. The renderer change that emits `gpu.intel.com/i915: 1` instead of the hostPath must NOT land until this plugin is merged AND confirmed advertising the resource on all three nodes: a request for a resource no node advertises makes the Plex pod Unschedulable, and apps-wave-0 is wait:true with apps-wave-1 depending on it. VERIFIED: manifests render offline (component, clusters/base, both overlays); no node currently advertises gpu.intel.com/* (queried — "NONE"); the plugin only lists/stats devices and never opens one (no os.Open/OpenFile/ioctl/syscall anywhere in its 908-line source); all four app-scoped Kyverno policies are gated on app-namespace=true and the new namespace is labelled "false". NOT VERIFIED: anything runtime. The plugin has never executed, the resource has never been advertised, and the open() test has not been re-run. Gate returned rc 3 (UNDETERMINED) with exactly the three expected skips — not green. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-01 22:25:33 -05:00
jakman85andClaude Opus 5	8eb5baffa4	feat(sec): constrain the conformance CI trigger at admission (R-SEC-02) gate / gate (pull_request) Successful in 1m38s Details gate / conformance (pull_request) Has been skipped Details Implements the credential model the operator chose for wiring L2/L3 into CI: a narrow trigger SA bounded by a fail-closed Kyverno admission policy, rather than the zero-credential GitOps-native alternative (plan section 4, primary model). THE ESCALATION THIS CLOSES. Kubernetes places NO `use` check on a Job's serviceAccountName. Any principal that can `create jobs` in ns conformance can create a Job running as conformance-runner -- nodes/patch, pods/exec, namespaces/patch -- with an arbitrary command. A CI token with plain `create jobs` IS that privileged identity no matter how narrow its own Role looks. RBAC cannot fix this; only constraining the SHAPE of what the trigger may create can. Adds: * SA/Role/RoleBinding conformance-trigger -- create+get+list+watch jobs, get pods and pods/log, nothing else. Namespaced. No delete (a trigger that can delete can hide a failed run), no exec, no secrets. Results come back as a tarball on stdout read via pods/log, so no execution channel into the runner pod is ever granted. * ClusterPolicy conformance-trigger-job-shape -- Enforce, failurePolicy: Fail. It does NOT use ${KYVERNO_VALIDATION_ACTION}: a security boundary an overlay can downgrade to Audit is not a boundary. It constrains ONLY conformance-trigger, so a mistake here cannot wedge the rest of the cluster. * Flux Kustomization `conformance` -> ./conformance/incluster, wait:false and non-gating like apps-vms, so a test identity can never stall the platform or app tiers. * A kyverno test with seven cases, six of them real escalation paths. VERIFIED by `kyverno apply` (pass: 10, fail: 6, error: 0), not assumed: evil-command DENIED at /spec/template/spec/containers/0/command/ <- THE attack evil-image DENIED at .../image/ tag-not-digest DENIED at .../image/ (mutable tag is republishable) host-namespaces DENIED at /spec/template/spec/hostPID/ extra-volume DENIED by the volumes foreach injected-args DENIED by the args foreach good-run ADMITTED ALSO FIXES A FAIL-OPEN IN THE GATE ITSELF. `kyverno test` does NOT fail when a policy cannot LOAD: every row becomes "Skip / Invalid Policy" and it still prints "N tests passed" and exits 0. Measured while writing this policy -- a schema error (subjects in a separate match.all entry from resources) produced exactly that, and ci/l0.sh would have certified a security policy that does not load as green. This is not specific to this policy: a typo in restrict-host-namespaces or secret-provenance would silently disable it with the gate still green. ci/l0.sh now scans for "Invalid Policy" and fails. Proven: broken policy -> l0 rc=1, restored -> rc=0. IMAGE PINNING, HONEST LIMIT: the plan specifies pinning the exact runner image digest. That image does not exist yet, so this enforces the enforceable half -- the image must be under the conformance repository path AND digest-pinned, never a mutable tag. Residual surface: anyone who can PUSH to that path could publish a different digest. Tighten to the literal digest when the image is built; it is a one-line change to the pattern, reviewed in a PR, which is where a change to a security boundary belongs. NOT YET WIRED, deliberately: no runner image, no trigger kubeconfig, no CLUSTER_AVAILABLE. The identities here are inert until those exist, and PR #12 makes CLUSTER_AVAILABLE=true block rather than fake a green in the meantime. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-29 03:22:41 -05:00
jakman85andClaude Opus 5	fd43f332d6	feat(pihole): authoritative LAN DNS for the zone on udp/53 + tcp/53 at a pinned VIP platform/coredns/coredns-custom.yaml has always said its split-horizon is for PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS: its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03 that docs/09 and the README's Phase 2 row both claim did not exist, so LAN clients ran on hand-maintained per-client dnsmasq files - one went stale after the dev decommission and cost the operator sonarr.home.keithjackson.dev. The SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips RFC1918 answers for public domains as rebind protection and returns NOERROR with no answer. EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason apps/vms/unifi and platform/crafty hand-write theirs): - two Services, udp/53 and tcp/53, sharing one pinned VIP ${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the .20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22 crafty; arping from a LAN host got no reply). - no external-dns hostname annotation: nothing public is created, and platform/externaldns/ is untouched. ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes every FTL setting as an FTLCONF_ environment variable and treats it as env-forced, applied ahead of pihole.toml on every cold start and read-only in the UI. That is a real declarative entry point, which the Keycloak and CNPG CronJob reconcilers exist precisely because their state lacks, so a reconciler here would be ceremony. misc.dnsmasq_lines carries a wildcard address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service (unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching domain, so precedence is structural and a newly onboarded Gateway-fronted service needs no DNS change. dns.upstreams is declared too - the live pod carried hand-set upstreams that no manifest knew about and that every restart discarded. externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not filter lease candidates by which nodes hold a backend - the two Services landed on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps renewing after the pod moves away, so with Local a reschedule leaves LAN DNS dead indefinitely. Cluster serves the VIP from any node; the LAN restriction moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP before load-balancing (verified by narrowing the range to exclude the test client and back). The cost is SNAT: forwarded queries reach PiHole from the ingress node, so PiHole's per-client rate limit is disabled in the catalog or the whole LAN would count as one client against its 1000-per-60s default. Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex -> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion (cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA NODATA, NXDOMAIN outside the zone, and still answering after a forced prod-1 -> prod-2 reschedule. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-14 00:57:58 -05:00
jakman85andClaude Opus 5	02c58aae15	fix(homepage): the C2 root hid a running service — Crafty had no tile The R-C2-01 check added in the previous commit is RED on main for a real reason. platform/crafty/httproute.yaml serves crafty.${BASE_DOMAIN} on the shared Gateway, clusters/base/crafty.yaml reconciles it (dependsOn platform-config, wait:false), the web UI is Keycloak-gated by a per-pod oauth2-proxy sidecar — and nothing on the operator's front door linked to it. The service was running and the root did not admit it existed. R-C2-01: "The root `home.keithjackson.dev` MUST serve a UI that surfaces the services running in the cluster and links to them." An unsurfaced route is a requirement gap, not a missing nicety, and it is the exact failure mode nothing else in the gate notices: every cluster test stays green, the apex answers 200, and the estate is one service larger than the dashboard says. This is the SAME defect the unmerged 49c160d found on 2026-09-07 (feat/coverage-debt- paydown, ~107 commits behind main). That branch was never merged, so the gap survived — re-fixed here against today's tree, in token form (`crafty.${BASE_DOMAIN}`, not the literal hostname that branch predated). Grouped under Media because Crafty manages game servers; Platform would be equally defensible and purely cosmetic. The tile also does duty for the L4 game ports (crafty-mc.${BASE_DOMAIN}, TCP 25565-25567 / UDP 19132-19134), which have no browsable URL of their own — the check reports those as an advisory rather than requiring a link. ALSO: clusters/base/homepage.yaml's postBuild comment claimed "Homepage references no ${VARS} today; the substitution is retained for parity". That is no longer true and the claim is dangerous, because it invites someone to delete the substitution as dead weight. This component now references ${BASE_DOMAIN} in httproute.yaml's apex hostname AND in every tile href; without substitution the token is applied literally, the apex route binds a host matching no listener, every link on the root is dead, and Flux still reports Ready. Comment corrected, and the property is asserted by c2-root-surface (5) so it cannot rot back. Verified: conformance/tests/l0-static/c2-root-surface -> check OK, 49 assertions. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-13 16:28:21 -05:00
jakman85andClaude Opus 5	44983095e1	feat(runbooks): make the 42 alert runbook_url annotations resolve Every platform-health alert rule carries `runbook_url: "https://${BASE_DOMAIN}/runbooks/<name>"` — 42 annotations over 34 distinct names, across platform/monitoring/alerts.yaml, platform/monitoring/pvc-capacity-alerts.yaml, platform/cnpg/alerts.yaml and platform/kubevirt/alerts.yaml. There was no runbooks/ directory and no HTTPRoute serving that path: an in-cluster probe of https://home.keithjackson.dev/runbooks/node-down returned HTTP 404. Every alert that reached a human handed them a dead link at the exact moment the documentation is read, and the annotation made it look solved — the same green-surface-over-absent-mechanism class as the import Job that reported success while storing placeholder secrets. No alerts.yaml is touched and no runbook_url value changes. The existing URL scheme is made to resolve instead. WHAT LANDS - runbooks/<34>.md — one per name, each grounded in this cluster's real topology and, where one exists, its real incident history: the 2026-09-07 k3s-prod-1 etcd I/O starvation (measured PSI 11.36 vs 0.75/1.29, the exact journal lines, the four mechanisms proven inert), the Longhorn reboot tax and the freeze/thaw of concurrent-replica-rebuild-per-node-limit, Cilium's once-resolved egress interface cache (Ifindex 3 against a live 1780) and the absence of OSS failover, the CNPG 38-hour ContinuousArchiving=False with its measured 6.8 MiB/h disk-full clock and the 2026-09-12 aggregation-scope fix, the 2026-09-06 Keycloak lockout in which all 12 clients held literal placeholder secrets behind an entirely green cluster, the Velero R2 x-amz-tagging NotImplemented incompatibility and its v1.14.3-rc.2 pin, and swap re-enabling itself across reboots via systemd-gpt-auto-generator. Each page states what the alert means HERE, the first diagnostic commands, likely causes ordered by probability with a remediation each, and when to escalate or stop. - Every page uses FULLY-QUALIFIED resource names and says why: three CRDs register the kind `Backup` here, the bare short name resolves to Longhorn's (zero objects), so `kubectl get backup -A` prints "No resources found" on a cluster holding 6 Velero and 27 CNPG backups. Acting on that false negative is how a CLI quirk becomes data loss. - Every page states that kubectl only works via `infra/devcontainer/dev.sh run bash -c '...'` with an explicit `--context k3s-prod`, because dev.sh defaults its current-context to k3s-dev. - Pages also record which alert groups are INERT today (measured 2026-09-13: no Longhorn, cert-manager, Flux-controller or Hubble scrape job exists, so 13 rules evaluate against nothing) so nobody reads their silence as health. SERVING — runbooks/serving/ plus clusters/base/runbooks.yaml A two-replica nginx-unprivileged serving a ConfigMap generated from runbooks/*.md, at the apex `/runbooks` prefix, in its own non-gating post-platform Flux tier (dependsOn platform-config, wait:false) — the platform/homepage pattern, file for file. - The service catalog cannot express this route: ingressRoutes[].host is pattern-bound to a single DNS label plus the apex, so the bare apex is unrepresentable, and there is no path field. - Rejected: redirecting to the Gitea web UI. A runbook is read when the platform is broken, and Gitea is an app-tier workload whose own dependencies several of these runbooks document; Gateway API's RequestRedirect also has no "append .md" modifier, so it would need 34 hand-maintained ReplaceFullPath rules. - configMapGenerator maps each file to an EXTENSIONLESS key (node-down=node-down.md), so /runbooks/node-down is a plain static lookup with no rewrite rule. Served as text/plain: the namespace has DNS-only egress, so a CDN markdown renderer could not load anyway. - The tier declares NO postBuild.substituteFrom, deliberately: envsubst would run over the prose and silently empty every ${...} shell example (it fails open — the 2026-09-01 emptied Cilium k8sServiceHost). The HTTPRoute omits `hostnames:` and inherits the https-apex listener's, so nothing needs substituting. AUTH — PUBLIC, deliberately auth-mode=none. Gating these pages behind oauth2-proxy/Keycloak would make runbooks/keycloak-down unreadable when Keycloak is down and runbooks/keycloak-client-secrets unreadable during the exact 2026-09-06 failure mode. An operator who cannot log in is this route's primary reader. Content is operational procedure only, no secrets. The rationale is recorded in an annotation on the HTTPRoute so the route is left OUT of the Gateway ext-authz attachment when forward-auth enforcement lands. PROVEN — in-cluster probe through the Gateway on prod /runbooks/node-down HTTP 200 8613 bytes text/plain; charset=utf-8 /runbooks/keycloak-down HTTP 200 9378 bytes text/plain; charset=utf-8 /runbooks/cnpg-archiving HTTP 200 11729 bytes text/plain; charset=utf-8 /runbooks/longhorn-faulted HTTP 200 9135 bytes text/plain; charset=utf-8 /runbooks/vpn-down HTTP 200 11092 bytes text/plain; charset=utf-8 /runbooks/backup-failure HTTP 200 9604 bytes text/plain; charset=utf-8 HTTPRoute Accepted by io.cilium/gateway-controller on sectionName https-apex; the `/runbooks` prefix wins over homepage's apex `/` by Gateway API longest-path precedence. GUARD — conformance/tests/l0-static/alert-runbooks-resolvable (R-OBS-04) 363 offline assertions. It checks BOTH halves, because the regression is silent in both directions: an alert with a new runbook_url and no runbook links to a 404, AND a runbook committed without its configMapGenerator entry is in git, reviewed, merged, and still 404s in the cluster. Also asserts a quality floor so a stub cannot make the link look fixed, and that route/listener/nginx prefix agree with the annotations. Proven to fail, then reverted: a bogus runbook_url (2 FAIL), a runbook removed from the generator (1 FAIL), and a 31-byte stub (7 FAIL). Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-12 23:49:58 -05:00
jakman85andClaude Opus 5	2590bc4317	feat(env): BULK_NFS_SERVER — the one knob prod cannot answer yet, and why it is a placeholder The bulk tier's PersistentVolumes need a server address, and there is no single correct one: the two clusters sit on OPPOSITE SIDES of pfSense from the storage host, which has exactly one interface (vmbr0 192.168.1.9/24, on pfSense's LAN). dev 192.168.1.40/.41/.42 — the SAME segment. Reaches it directly, and already holds one NFS export from it (/mnt/Storage/k8s-volumes, in the host's /etc/exports). Proven path, so base's default is the dev value per this file's existing convention, and dev restates it. prod 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does not traverse that today; 22 and 8006 answer only because pfSense port-forwards them via 192.168.2.102. Opening the storage path is a PENDING operator decision whose two candidate outcomes give DIFFERENT values here: 192.168.2.102 (another port-forward) or 192.168.1.9 (a route). So prod holds TBD-PROD-BULK-NFS-SERVER, this repo's established loud-placeholder convention. Two findings recorded in the prod overlay because they change what the operator has to do: * THE EXPORT'S CLIENT LIST IS THE SAME UNDER BOTH OUTCOMES. Measured today from this workstation (192.168.2.132) through the existing port-forward: the storage host logged the connection as coming from 192.168.2.132 — a WAN-side address — not from a pfSense LAN address (`ss -tn` peer 192.168.2.132; the host's only route off-LAN is its default via 192.168.1.1). pfSense DNATs WITHOUT source NAT, so the NFS server sees prod's REAL node addresses either way. The `zfs set sharenfs=` work therefore does not have to wait for the routing decision. * ONE PORT, NOT FOUR. The PVs mount nfsvers=4.1, and NFSv4 carries mount and lock in-protocol — no rpcbind/111, mountd or statd. Whichever outcome is chosen, it opens tcp/2049 and nothing else. NFSv3 would have needed four ports, three of them on ephemeral numbers. * AND ONE HONEST WEAKNESS, stated in the overlay where an operator will hit it. * Unlike the LAN knobs above it, this placeholder does NOT fail at apply time. A PV's spec.nfs.server is an unvalidated string: the API server accepts it, the PV is created, and the PVC BINDS (binding is pure control-plane). The failure appears only when a pod mounts — MountVolume.SetUp "Failed to resolve server TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr and nextcloud would sit in ContainerCreating on prod. There is no CRD validation to lean on here, so the guard has to be procedural: set the value before prod reconciles a revision carrying the bulk volumes. Dev renders a working value and is where these manifests are meant to be proven. Related finding, NOT fixed here because the file does not exist on main: the static guard this convention is supposed to lean on, `bootstrap/standup.sh`'s `s3_is_placeholder` family, is absent from main (it lives on unmerged prod branches), so conformance/tests/l0-static/placeholder-guard-coupling exits 3 SKIPPED and nothing in the gate currently refuses a placeholder-shaped overlay value. When standup.sh lands, its guard needs an arm for this knob and a consumer that calls it — assertions (2) and (3) of that check then cover this value automatically, since the detector's SEED vocabulary already matches "TBD". Evidence: `kubectl kustomize clusters/dev` -> BULK_NFS_SERVER: 192.168.1.9; `kubectl kustomize clusters/prod` -> BULK_NFS_SERVER: TBD-PROD-BULK-NFS-SERVER. Both exit 0. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 23:47:39 -05:00
jakman85andClaude Opus 5	161430d915	feat(backup): the overlay picks the credential — BACKUP_CREDENTIALS_SUFFIX The previous commit created the two halves. This one wires the choice, so dev renders MinIO credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the overlay patch is the entire difference between environments; no base manifest is forked). cluster-vars BACKUP_CREDENTIALS_SUFFIX base "" \| dev "" \| prod "-prod" velero HelmRelease credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX} 3x CNPG Cluster s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX} ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters of the way, leaving one consumer reading the OTHER environment's key with nothing rendering wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED variable, never on a defined-but-stale one. With one suffix the four move together or not at all. The cost is that the rendered Secret name is not greppable from the consumer, so both kustomizations spell the two names out in full beside the resources. THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects that exist in every render, and that dev is running today. The alternative failure mode is strictly worse than the shared-Secret state this replaces: velero's pod hangs in ContainerCreating forever when `existingSecret` names a missing Secret (the volume never mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the value explicitly anyway, so neither inherits it silently. Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is untouched (it stays decryption-only, and velero stays suspended on prod). VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform, platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with that environment's variables: strict envsubst PASSES for all 8 (env x tier) combinations dev velero existingSecret velero-cloud-credentials dev keycloak/nextcloud/semaphore-db <db>-backup-credentials (both key refs each) prod velero existingSecret velero-cloud-credentials-prod prod keycloak/nextcloud/semaphore-db <db>-backup-credentials-prod (both key refs each) both all EIGHT Secret objects present in the rendered output (dev and prod alike) no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell escapes in apps/wave-0, which are pre-existing and unrelated. Read alongside the endpoints the same overlays already set, the two environments are now isolated on credential AND target: dev velero s3://velero (root) + cnpg s3://cnpg @ http://192.168.1.9:9000 region us-east-1 prod velero k3s-prod-backup/velero + barman prefix @ https://<acct>.r2.cloudflarestorage.com region auto Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 01:41:56 -05:00
jakman85andClaude Opus 5	0a0527e7db	feat(backup): prod's off-cluster DR lands on R2 — and stops the WAL clock Prod has been archiving WAL to `TBD-PROD-S3-BACKUP-ENDPOINT` since bootstrap. Verified live 2026-09-07 on all three CNPG Clusters: ContinuousArchiving=False, "unexpected failure invoking barman-cloud-wal-archive: exit status 4". Because archive_mode is ON, that is not a degraded backup — it is a DISK-FULL CLOCK. Postgres will not recycle a segment it has not archived, so keycloak-db's pgdata PVC is at 579 MiB used / 4360 MiB free and climbing monotonically. When it fills, the primary PANICs and takes SSO down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Wiring the endpoint is what lets Postgres drain the whole accumulated backlog in one pass. THE KNOB LAYOUT (three new, one repurposed) and why it is shaped this way. The operator created exactly ONE R2 bucket, `k3s-prod-backup`, but the two backup tools want differently-SHAPED values: the Velero chart takes `bucket` and `prefix` as separate fields, while CNPG's barmanObjectStore.destinationPath is one full s3:// URL whose first path segment IS the bucket. So: S3_BACKUP_ENDPOINT dev http://192.168.1.9:9000 prod https://<acct>.r2... S3_BACKUP_REGION dev us-east-1 prod auto VELERO_BACKUP_BUCKET dev velero prod k3s-prod-backup VELERO_BACKUP_PREFIX dev "" prod velero CNPG_BACKUP_PATH dev s3://cnpg prod s3://k3s-prod-backup/barman There is deliberately NO separate `S3_BACKUP_BUCKET`: CNPG's bucket already lives inside CNPG_BACKUP_PATH, and a second knob holding the same bucket name twice is a rotation waiting to drift. Three knobs, each shaped for its own consumer, and no two of them can contradict each other. TENANCY. Inside k3s-prod-backup, velero and barman get disjoint TOP-LEVEL prefixes — s3://k3s-prod-backup/velero/ and s3://k3s-prod-backup/barman/<cluster>/ — with the bucket ROOT left empty so a misrendered prefix lands somewhere obviously wrong instead of silently colliding. This is not tidiness: Velero ADOPTS every backup under its location and TTL-DELETES it on retention, so an overlapping prefix would let it reap WAL segments; and interleaving two Postgres servers' WAL into one prefix corrupts the chain outright. Dev↔prod isolation is doubly independent — different endpoint AND different bucket names — so neither cluster can reach the other's objects even with a mis-substituted prefix. DEV IS UNCHANGED, BYTE-FOR-BYTE, and that is asserted rather than assumed. Base defaults are dev's LIVE values (verified against the running cluster: BSL bucket=velero with NO prefix, region us-east-1; all three CNPG Clusters ContinuousArchiving=True at s3://cnpg on the MinIO), and the dev overlay now restates all five explicitly for the same reason LONGHORN_DATA_PATH is restated — a silent drift in base would ORPHAN the backups dev's PROVEN restore depends on. The empty-prefix case was the one real risk, and it is settled empirically: with prefix "" the chart omits objectStorage.prefix from the rendered BackupStorageLocation ENTIRELY (helm template, chart 12.1.0), so dev's live BSL is unchanged, not merely equivalent. R2's TWO GOTCHAS, both wired up front: - region must be exactly `auto`. head-bucket answers {"BucketRegion":"WNAM"}, which is a PHYSICAL LOCATION HINT, not a signing region. This was hardcoded us-east-1 inside the Velero HelmRelease — correct-by-accident for MinIO, wrong for anything else — so it is a knob now. - checksumAlgorithm: "". velero-plugin-for-aws v1.14.2 bundles an AWS SDK with flexible checksums on by default and sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT; R2 rejects those with HTTP 400 InvalidArgument, failing every backup and every kopia blob upload. It must be PRESENT-but-empty (the plugin branches on the key existing), and the chart does emit an empty value — verified by rendering it. Harmless against MinIO, so it is unconditional rather than a fourth knob. WHAT THIS DOES NOT DO. Velero stays SUSPENDED on prod. `spec.suspend: true` is owned by field manager `flux` (verified: kustomize-controller's managedFields do not include f:suspend), so server-side apply will not strip it, and nothing here declares suspension — which would also trip R-ENV-03, since a prod overlay may patch only cluster-vars and a Flux Kustomization's `decryption`. Unsuspending is the operator's, after `standup.sh --s3-endpoint <r2> s3check` proves a round trip. NO DNS KNOB IS TOUCHED. BASE_DOMAIN stays prod.home.keithjackson.dev and every EXTERNALDNS_* value is untouched. The apex inversion on work/prod-standup-campaign was deliberately NOT carried: this branch ports that branch's R2 reasoning knob by knob, and merges nothing. VERIFIED: ci/l0.sh green (flux-postbuild-substitution 272 assertions, 0 hazards, 33 knobs defined / 28 referenced, every new token resolved in BOTH overlays); render-golden, lint, schema, spec-immutability green; touched requirement R-STATE-03 covered. Refs R-STATE-03, R-IDP-05, R-ENV-02, R-ENV-03. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-07 01:14:29 -05:00
jakman85andClaude Opus 5	d1a6628a97	feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach the cluster and provide fault tolerant HA. Without a change it will not work in HA." THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live before changing anything: all three prod cilium-agent pods carried KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated that stale comment as an open question when the evidence already settled it. SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs: k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join target and a tls-san. infra/tofu/outputs.tf GENERATES this name, so renaming would ripple into tofu and the runbooks. k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium consumes. Derived, not restated, so an environment overrides ONE line and the VIP, the SAN and the CNI endpoint move together. Prod needed no new override. THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran node-baseline -> k3s -> cilium, so Cilium would install against an address nothing announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value: site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier (kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so a cold run stops THERE with a named cause instead of surfacing as unreachable agents. Also fixed a pre-existing bug that only mattered once this play joined site.yml: the role's read-only interface probe ran without `check_mode: false`, so under --check it was skipped and the assert fired with a misleading "set kube_vip_interface explicitly". VERIFIED BY ME, not just reported: renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP) cilium template consumes k3s_apiserver_endpoint, not k3s_api_host cold join still `--server https://{{ k3s_api_host }}:6443` — the founder order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux gate full CI gate PASS Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders ['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved — .9 already answers under full CA verification; and --check --diff of kube-vip.yml against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the kube-vip prod already runs. ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine checks: the Cilium template must use the endpoint var; the endpoint must derive from kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal that environment's VIP and be none of its node IPs (the [parity] contract, now machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium. Every new assertion mutation-tested and bites. One honest negative recorded: reverting only the BASE default is not caught, because both overlays patch the key — the check reads effective per-environment values, so the base default is unguarded. APPLY ORDER, and this is the operator's step — nothing was applied: PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01 incident), it will not proceed past an unready pod, existing datapath survives an agent restart, and .9 is already live before the change lands. DEV SEPARATELY, operator present. Same mechanics but live dev agents dial 192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that. ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook. NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that test's gap list. Refs R-PLAT-01, R-NET-01, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-06 22:17:30 -05:00
jakman85andClaude Opus 5	f21e89d9d2	fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details `platform-config` applied the CDI CR and cdi-storageprofiles.yaml in ONE pass. StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and Flux server-side dry-runs every object before applying any — so the unknown kind aborted the tier, the CR was never applied, the CRD was never created, and the tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI CR, no storageprofiles CRD, every downstream tier blocked. The profiles now live in their own tier ./platform-config-cdi with dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated and platform-config runs wait: true, the CDI CR must be applied AND healthy — exactly the event that creates the CRD — before the profiles are applied. Cold clusters order themselves. `standup.sh crds` is retained as a fallback for clusters already stuck, and because it is idempotent. THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is created by a CONTROLLER AT RUNTIME is second-order and needs its own pass. cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles, datavolumes, datasources and dataimportcrons appear only after it reconciles the CDI CR. An audit found StorageProfile is the only second-order kind in platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster, VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium L2 CRs all have their CRD in git or in a chart. Moving the CDI CR up into `platform` was rejected: that would put CDI's health on the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and headlamp splits exist to undo. apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic: apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and its comment claimed platform-config guaranteed the profiles — which this move would have silently falsified. THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has prune: true, so it deletes both profiles once, CDI recreates them with empty spec, and the new tier re-applies the overrides ~30-60s later. Compared our override against CDI's unaided inference for the same Longhorn provisioner on dev: longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block longhorn-static (no override) spec unset -> status ReadWriteOnce/Block Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can no longer cause it (apps-vms gates on the new tier); a human must not do it by hand. Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated — CDI creating and inferring them with no Flux involvement. MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the target first. If it is not, the profiles are pruned and sit at CDI defaults until it recovers — and prod was in that state hours earlier. Fallback is the two-step the Cilium move used (merge with prune: false, verify the ownership label moved, restore prune: true). substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN}, deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits paths whose Kustomization declares it, so omitting it would move the file OUT of the audited set — a blind spot for a future literal-token ship. Gate confirms it stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS decryption stanza: the moved file is two plain CRs, so no overlay change was needed. VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the two profiles; the graph is byte-identical across dev and prod with apps-vms carrying both edges and no cycle. Full CI gate PASS. KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include platform-config-cdi, so `resume` will not wait on it. Another agent holds that file; the PM will add it. Refs R-GITOPS-01, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-06 22:14:19 -05:00
jakman85andClaude Opus 5	c8c1d2935d	feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources, and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled to 0; verified restored against Cloudflare directly (my first check went through a pfSense override and falsely showed healthy — local resolvers are not evidence about public DNS). THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put "never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified --limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored manifests reached across a third party and destroyed dev's state. Cluster-to-cluster isolation was verified; the dependency that mattered was cluster-to-Cloudflare. Operator's architecture: dev -> .dev.home.keithjackson.dev, prod -> .prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home. PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. + apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and 12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every user, and moves gitea.home. — which is BOTH the GitOps source and the current push target. BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs. Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles" has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized: Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15 links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove --oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11 catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose occurrences and non-reconciled paths. TWO THAT CANNOT BE TOKENIZED, flagged not forced: - headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt), so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy anyway — client secrets are realm-scoped. - kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url, overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets prod.home. EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment — dev renders exactly `default` (the chart default it was already using, so its existing records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL, deliberately not `default`, so a future environment that forgets the knob owns nothing and can delete nothing rather than silently inheriting dev's identity. I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE — ExternalDNS generates the record then skips it with "no hosted zone matching record DNS Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev. Isolation comes from the subdomain split plus txtOwnerId, which is what the incident actually needed and what a domain filter cannot provide. DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly. POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12 hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod., and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's. L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS. PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied. Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-04 22:43:54 -05:00
jakman85andClaude Opus 5	05baae90ec	feat(prod): kernel fix + disk layout — prod nodes ready for k3s Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled changes, all applied and verified on all three nodes. === 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have === A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on. roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's Debian release, writes a deb822 trixie-backports source, installs, reads back the exact version, writes a provenance/rollback note to the node. Scoped by node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen over a hostname test so a fourth prod node inherits it from group membership. Verified: --check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest` not `present` (linux-image-amd64 was already installed from main, so `present` would have silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic + ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF automatic security upgrades for the very kernel being installed). Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable. THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it 2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same address, handled correctly; not a lucky draw. §0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was corrected: >=6.13 has three rejection gates and only the third prints that line; all three nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing /boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither exists in 6.12. TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown: - MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node 2's real value was -1569980416, which did not match, so state fell through to `clean` and the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the exact bug the check exists to find. Now (-?\d+). - bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever. Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above. === 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded === Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe). Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory. Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on the first re-image and the stale value's failure mode is mkfs on the WRONG partition. Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr /var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat. /srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used" was ext4's default 5% root reserve). === 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd === Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB. Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled /var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster death, not degradation, arriving during bring-up. Invisible when the disk decision was made, because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var. Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and "unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make a stranger's `data` volume look already-claimed and weaken an assert. The transition off the existing LABEL=longhorn mount is DECLARED, not inferred (node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness assert fails closed on dev's 18 volumes even if both were defeated. k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back on the OS filesystem invisibly until the disk filled. LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn, prod /data/longhorn. DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn volumes on that path. All 12 substituting Flux paths rendered through `kubectl kustomize \| flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing the same harness detects prod changing on exactly one line — an all-identical result is worthless if the harness cannot see change. Live check after: dev's default-data-path setting still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready. Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks` selected the include and ran nothing inside it while reporting success. OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node 3's decided resolvers will silently revert. Gate: MERGE GATE: PASS. L0 substitution check 106 assertions. Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-04 00:39:10 -05:00
jakman85andClaude Opus 5	22174186ff	chore(flux): restore platform prune:true — MERGE 2 of 2 (Cilium split cleanup) gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details Closes the transition opened in merge 1 (`63f98aa`). `prune: true` is restored on the `platform` Kustomization now that the ownership transfer is VERIFIED on the live cluster, not merely expected: 13/13 Kustomizations Ready, `cilium` among them, at main@sha1:23aa2854 Namespace/kube-system label kustomize.toolkit.fluxcd.io/name = cilium HelmRelease/cilium label ... = cilium HelmRepository/cilium label ... = cilium platform inventory: 0 entries matching kube-system__Namespace \| kube-system_cilium_* cilium inventory: exactly those three objects Cilium pods UNCHANGED from the pre-merge baseline — cilium-2f8h8 r=0 2026-09-01T05:55:27Z cilium-425m8 r=1 2026-08-30T13:25:10Z cilium-54cww r=0 2026-09-01T05:55:26Z HelmRelease still v11 So the relabel deterministically preceded any prune, exactly as the revision-gated dependsOn edge predicted, and the ownership move cost zero pod restarts. The in-file comment now records the lesson rather than the temporary state: before moving ANY platform/* component in future, check whether it is the sole declarer of a shared Namespace. platform/cilium/namespace.yaml was the only declaration of Namespace/kube-system under ./platform, which would have turned a component-scoped prune into `delete namespace kube-system` — taking coredns, metrics-server, local-path and Cilium. L0 substitution gate: 103 assertions, 0 hazards. Serves: R-PLAT-01, R-GITOPS-01. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 18:39:20 -05:00
jakman85andClaude Opus 5	63f98aab26	feat(flux): give Cilium its own Kustomization — MERGE 1 of 2 (transition) Cilium was one of ~12 components inside a single `platform` Kustomization with wait:true, so the CNI's health was indistinguishable from eleven unrelated components in the tier's Ready condition, and the only emergency valve was freezing the whole platform tier. This repo already set the precedent in reverse (oauth2-proxy and headlamp were split OUT so unhealthy leaves would not gate a tier); Cilium is the mirror image — the one component that legitimately SHOULD gate everything. New clusters/base/cilium.yaml: path ./platform/cilium, wait:true, prune:true, interval 1h (a CNI does not need a 10m re-render cadence; kustomize-controller is source-triggered so a push still reconciles immediately — the interval is only the drift-correction floor), timeout 10m (a 3-node DaemonSet rollout with wait exceeds the 5m default), dependsOn:[flux-system], and postBuild.substituteFrom: cluster-vars — the last is LOAD-BEARING, not decorative: helmrelease.yaml references ${K3S_API_HOST} and kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so without it this Kustomization would fail immediately. * THE HAZARD WAS FAR WORSE THAN "the CNI gets pruned" * platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system anywhere under ./platform, and the live object carries kustomize.toolkit.fluxcd.io/name=platform. With prune:true, the moment ./platform/cilium leaves platform's path the stale set is not "the Cilium HelmRelease" — it is `kubectl delete namespace kube-system`, taking coredns, metrics-server, local-path AND Cilium. A full cluster kill, not a CNI kill. TWO INDEPENDENT PROTECTIONS, deliberately not trusting either alone: 1. prune:false on `platform` for this transition merge — Flux's documented Procedure 1 for moving resources between Kustomizations. Chosen over the per-resource `kustomize.toolkit.fluxcd.io/prune: disabled` annotation because that needs THREE merges and leaves the new Kustomization owning resources it can never prune. 2. The pruner's own owner-label filter. In v1.4.0, prune() passes Inclusions: GetOwnerLabels(obj.Name...), so platform only deletes objects whose LIVE labels still say name=platform. The new cilium Kustomization relabels all three to name=cilium on apply, and platform.dependsOn:[cilium] is REVISION-GATED, so platform provably cannot reconcile revision N until cilium has APPLIED revision N. The relabel deterministically precedes the prune. Protection 2 alone would probably have sufficed. Not betting kube-system on "probably". NOTED FOR THE FUTURE, in-file: anyone moving a platform/* component must first check whether it is the sole declarer of a shared Namespace — that turns a component-scoped prune into a cluster-scoped one. Verified: `kubectl kustomize platform` goes 88 -> 85 documents, 3 removed / 0 added / 0 CHANGED, and `kubectl kustomize platform/cilium` reproduces those 3 byte-identical (sha256-matched). A pure ownership move; no Cilium value touched. L0 substitution gate 103 assertions (was 102), new cilium row, 0 hazards. Server-side dry-run against the live cluster accepted both CRs. clusters/prod renders clean off the same base. MERGE 2 (restore prune:true) follows only after verifying: 13/13 Ready; kube-system, the HelmRelease and the HelmRepository all relabelled to name=cilium; platform's inventory no longer lists them; and Cilium pod ages/restarts UNCHANGED from cilium-2f8h8 r=0 2026-09-01T05:55:27Z / cilium-425m8 r=1 2026-08-30T13:25:10Z / cilium-54cww r=0 2026-09-01T05:55:26Z, HelmRelease v11. Serves: R-PLAT-01, R-GITOPS-01. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 18:33:20 -05:00
jakman85andClaude Opus 5	666a60776a	fix(flux): close the substitution race — platform dependsOn root + strict substitution (P0/P1) P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix for the incident, not the feature gate. Verified against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION- gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision N is exactly what writes a newly-added cluster-vars knob. Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all 11 substituting Kustomizations chain transitively through it (platform-config -> platform; the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config). The "not chaining through platform" set is EMPTY, so this one edge protects the whole graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/ sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready on apply success without health-gating its children. Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest anywhere under clusters/<env> — including clusters/base — now stalls platform and everything downstream, where today it would not. Fail-closed, per doctrine. P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README). There is no gotk-components.yaml in git, and the live kustomize-controller Deployment carries NO Flux ownership labels — it was installed imperatively by `flux install` from bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT the controllers. Nothing under clusters/** can reach a controller flag. So it uses Flux's documented mechanism for hand-installed clusters: a new bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a container's args is atomic and a merge patch would replace all five install-generated args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the patch, ASSERT the gate string is present so a future Flux bump that breaks the patch target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is vendored, so components can never drift from the pinned FLUX_VERSION. Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial Deployment as a Flux-managed resource (a future prune would delete kustomize-controller). ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive on a git push — the controller Deployment is outside Flux's inventory, so an operator must re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race turns a transient empty render into a transient hard Kustomization failure. ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left platform blocked forever on a flux-system Kustomization that does not exist, and nothing downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev. Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7): prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs the prod age-key decision, a security choice not a copy), and no GitRepository source URL patch (needs a decision on whether prod tracks the same branch as dev, which would make a dev-facing merge reconcile prod). Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS. Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-01 01:45:04 -05:00
jakman85andClaude Opus 5	3f980d2a17	feat(env): tokenize hardcoded dev LAN addresses into cluster-vars (R-ENV-02) R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x). New cluster-vars knobs (base default = dev literal, per the file's existing convention): K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror) CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml CILIUM_LB_POOL_STOP " UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml CRAFTY_L4_VIP platform/crafty/service-l4.yaml S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud, semaphore}-db.yaml, and platform/velero/helmrelease.yaml Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address 10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely. clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming Kustomization without it, so a ${TOKEN} there would have shipped as a literal string). The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR}; $VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in the unifi cloud-init are escaped accordingly. PROD ADDRESS PLAN (operator, 2026-08-31): nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20 Cilium LB pool .20-.30 CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM gives it the first free address of a matching pool block. An initial prod proposal put .20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon coredns-custom kept answering .home.keithjackson.dev with .20 — an address nothing announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so kube-vip and LB-IPAM can never contend. Values prod still cannot know stay INVALID `TBD-PROD-` placeholders on purpose: base defaults are dev values, so an omitted patch means prod silently inherits a dev address — far worse than a loud failed apply. DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The only deltas are the intended additive keys in the overlay ConfigMap itself. No unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean. Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-08-31 22:43:36 -05:00
jakman85andClaude Opus 4.8	d1248e6778	feat(backup): wire Velero + CNPG Barman to the live MinIO S3 target Author the GitOps side of off-cluster backups against the on-Proxmox MinIO (http://192.168.1.9:9000, path-style; buckets velero + cnpg). Velero (R-STATE-03): - HelmRelease (chart 8.x / Velero 1.15) + velero-plugin-for-aws init container + node-agent kopia File System Backup (deployNodeAgent, uploaderType=kopia, snapshotsEnabled=false). BSL default -> bucket velero, s3ForcePathStyle. - Two daily Schedules: `daily` backs up the whole cluster's resources+PVC data (defaultVolumesToFsBackup opt-out) EXCEPT monitoring+unifi; `daily-resources-only` captures monitoring+unifi RESOURCES only (Prometheus TSDB, Loki 72h logs, and the ~7.4GiB UniFi VM disk are reconstructable — excluded from volume backup). - velero-cloud-credentials SOPS Secret (cloud INI key, ns velero). - Own non-gating Flux tier clusters/base/backup.yaml (dependsOn platform-config, wait:false) + clusters/dev SOPS decryption patch — mirrors headlamp. CNPG native Barman PITR (R-IDP-05 / R-STATE-03): - keycloak-db, nextcloud-db, semaphore-db gain spec.backup.barmanObjectStore (s3://cnpg/<cluster>, endpointURL MinIO, gzip WAL+data, 30d retention) + a daily ScheduledBackup (6-field cron, immediate) each. - Per-namespace scoped-key SOPS Secrets (<db>-backup-credentials), same-ns only. - render.py: emit-waves now recognizes <svc>-db-backup-credentials.sops.yaml as a DB-wave sidecar (+ self-test); wave-0 kustomization regenerated. Creds mirror platform/backup/minio-credentials.sops.yaml (scoped key k3s-backup), SOPS round-trip verified with the cluster age key. gate: green. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-28 22:37:17 -05:00
jakman85andClaude Opus 4.8	0437871807	Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI Add Crafty Controller 4 (registry.gitlab.com/crafty-controller/crafty-4:4.10.8) as a platform component: a web panel that runs Java + Bedrock Minecraft servers as subprocesses in one pod + a 20Gi Longhorn PVC (mounted 5x by subPath). Auth = Option B (per-app oauth2-proxy reverse-proxy sidecar), since Cilium 1.16.5 cannot do ext_authz on the shared Gateway: - oauth2-proxy v7.8.1 sidecar (:4180) reverse-proxies to Crafty's self-signed HTTPS UI (127.0.0.1:8443), gated to Keycloak homelab realm group platform-admins. - Confidential `crafty` client provisioned live via kcadm + captured in the realm import; client secret + cookie secret via SOPS/age (mirrored in realm-clients). - Web UI: HTTPRoute crafty.home.keithjackson.dev -> Service :80 -> sidecar :4180. - Game ports: Cilium L2 LoadBalancer on VIP 192.168.1.55 (next free after .53/.54), starter set TCP 25565-25567 (Java) + UDP 19132-19134 (Bedrock), NOT auth-gated. - fsGroup:0 / runAsGroup:0 / runAsUser:1000 for the image's crafty:root identity on the root:root Longhorn PVC (the seerr fsGroup lesson). - Own non-gating Flux tier clusters/base/crafty.yaml (dependsOn platform-config, wait:false) + sops-decryption patch in clusters/dev — a leaf UI must never gate the platform. - Explicit default-deny NetworkPolicy + CiliumNetworkPolicies (platform namespace, not app-namespace, so Kyverno does not auto-generate the baseline). make -C ci gate green; kustomize build of platform/crafty, clusters/base, clusters/dev clean. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-28 15:39:51 -05:00
jakman85andClaude Opus 4.8	eb31d2ef58	fix(monitoring): cap Prometheus TSDB by size to stop false absent()/==0 alerts The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown, KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown, CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches: the Prometheus PVC is 100% full ("no space left on device" on every WAL commit). Evidence from the live dev cluster: - /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit failed ... write to WAL: no space left on device". Scrapes still return 200 (targets show healthy, lastScrape=now) but samples are never stored, so the TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC. - Instant queries at "now" therefore return an EMPTY vector, so every absent()/==0 platform rule trips. Evaluated at the last-good-sample time each expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default present) — the rules are correct; the monitor was simply blind. `retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the dev PVC 5Gi->8Gi for real WAL/head/compaction headroom. - platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob} - clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC) - clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB - clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC) - platform/monitoring/README.md: knob table + rationale Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion (operator PVC expansion / manual patch); once free space returns, retentionSize keeps it healthy. Watchdog left untouched. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-28 14:38:12 -05:00
jakman85andClaude Opus 4.8	1749603f87	feat(homepage): apex landing dashboard at home.keithjackson.dev Deploy Homepage (gethomepage) as the apex landing dashboard aggregating links to cluster services, in its OWN non-gating post-platform Flux Kustomization (clusters/base/homepage.yaml, dependsOn platform-config, wait:false) — mirrors headlamp/oauth2-proxy so a leaf UI can never gate the platform chain. - platform/homepage/: plain manifests (ns, config ConfigMap, Deployment, Service, default-deny+Cilium ingress NP, apex HTTPRoute). Non-root pod, HOMEPAGE_ALLOWED_HOSTS, bundled mdi-* icons (DNS-only egress). - gateway.yaml: add https-apex + http-apex-redirect listeners for the bare apex (the *.home wildcard listener does NOT match it). Apex is already a SAN on the wildcard cert — no new Certificate. - Forward-auth annotated (auth-mode=forward-auth) but NOT enforced: the Gateway ext-authz mechanism is an unbuilt Phase-6 WS-NET TODO (same status as all other forward-auth apps). Flagged as PM follow-up. Gate: lint/l0/render-golden(10/10)/l1 green; conformance skipped (no cluster). Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-28 12:17:25 -05:00
jakman85andClaude Opus 4.8	ebe0371b1b	platform: move Headlamp out of the gating platform tier + fix -session-ttl crashloop Headlamp's HelmRelease sat in the wait:true `platform` Flux tier. Its pod crashlooped (`flag provided but not defined: -session-ttl` — chart 0.40.1 renders a container arg the appVersion v0.40.1 backend lacks; upstream kubernetes-sigs/headlamp#4883), and because it gated `platform`, the whole chain stalled: platform Ready=Unknown -> platform-config, oauth2-proxy, apps-wave-0/1/2 and apps-vms all Ready=False "dependency not ready". Part A (unblock + durable lesson): a leaf UI must never gate the chain. Move all Headlamp resources into a new non-gating Flux Kustomization clusters/base/headlamp.yaml (dependsOn platform-config, wait:false, SOPS decryption patched per-env in clusters/dev) — mirroring oauth2-proxy exactly. The new Kustomization owns the whole platform/headlamp component; its HTTPRoute moves from platform-config into platform/headlamp/kustomization.yaml. Removed headlamp from platform/kustomization.yaml and its httproute from platform-config/kustomization.yaml. Now a crashlooping Headlamp can never stall platform/platform-config/apps. Part B (make it start): pin the chart to 0.41.0 (`>=0.41.0 <0.42.0`). The backend added `f.Int("session-ttl", …)` in v0.41.0 and chart 0.41.0 pins that same image (appVersion == chart version), so chart+image realign; every arg the chart renders for our values is supported. OIDC config + per-user RBAC (clusterRoleBinding.create:false) unchanged. Gate: `make -C ci gate` PASS. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-28 12:01:01 -05:00
jakman85andClaude Opus 4.8	960cab937b	fix(flux): move oauth2-proxy to a post-platform tier (dependsOn platform-config) oauth2-proxy is a leaf forward-auth service that depends on Keycloak + trusted wildcard TLS + split-horizon DNS. It lived in the gating `platform` Flux Kustomization (wait:true), so on a cold start it Failed until those dependencies existed, keeping the whole `platform` tier not-Ready. That in turn blocked `platform-config` — which brings up the very Keycloak instance, wildcard Certificate and L2 VIP oauth2-proxy needs (and previously carried oauth2-proxy's own httproute/servicemonitor) — from ever reconciling. A self-gating cold-start deadlock, twice-proven (DECISIONS-LOG sessions 7 & 9). Break the cycle by moving the whole oauth2-proxy component out of the `platform` tier into its own non-gating post-platform Flux Kustomization: - clusters/base/oauth2-proxy.yaml: new Kustomization -> ./platform/oauth2-proxy, dependsOn platform-config, wait:false, substituteFrom cluster-vars. - clusters/base/kustomization.yaml: reference the new Kustomization. - platform/kustomization.yaml: drop the oauth2-proxy component (no longer in the gating tier). - platform/oauth2-proxy/kustomization.yaml: pull in httproute + servicemonitor so the full component travels together. - platform-config/kustomization.yaml: drop the in-place oauth2-proxy httproute/servicemonitor refs (moved with the controller). - clusters/dev/kustomization.yaml: add the SOPS decryption patch for the new oauth2-proxy Kustomization (it now owns the encrypted credentials Secret that the `platform` Kustomization used to decrypt). Now `platform` reaches Ready without oauth2-proxy; platform-config brings up Keycloak/TLS/DNS; then oauth2-proxy reconciles against dependencies that already exist. wait:false guarantees its health can never gate anything above it. No config of oauth2-proxy itself changed — only its reconciliation tier. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-27 18:12:43 -05:00
jakman85andClaude Opus 4.8	da1f833de7	fix(vms): drop postBuild substituteFrom on apps-vms — it corrupted VM cloud-init Flux's postBuild.substituteFrom runs envsubst over the WHOLE manifest, including the VM cloud-init userData. It was substituting every shell `${...}` in the UOS install script (${UOS_URL}, the API-resolver ${...}) to empty — which is why the first UOS install silently did nothing — and ${PIPESTATUS[0]} then broke the build entirely ("missing closing brace"), blocking apps-vms reconcile. VM manifests use no cluster-vars, so substitution is unneeded. Removed it; the cloud-init shell now reaches the guest intact. (A future VM needing a cluster-var must escape its shell `${...}` as `$${...}`.) Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-26 16:26:27 -05:00
jakman85andClaude Opus 4.8	317b3f02f1	feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track First KubeVirt VM app + first real CDI http import. UniFi OS Server (UOS) is Ubiquiti's current self-host product (the .deb is deprecated); it bundles UniFi OS + Network app + its own datastore, so no MongoDB companion. - apps/vms/unifi: CDI DataVolume (Ubuntu 24.04 cloud image -> longhorn-vm RWX/Block, live-migratable) + VirtualMachine (evictionStrategy: LiveMigrate) with cloud-init that installs podman/slirp4netns/jq and runs the UOS installer NON-INTERACTIVELY. The installer URL is resolved at boot from Ubiquiti's public firmware API (fw-update.ui.com/api/firmware-latest) — always-latest, no pinned URL to maintain (currently UOS v5.1.37). - L4 exposure: two LoadBalancer Services (TCP+UDP) sharing a pinned, L2-announced Cilium VIP 192.168.1.54 (device inform/adoption + STUN can't ride the L7 Gateway); external-dns publishes unifi.home.keithjackson.dev. UI on :11443. - default-deny + additive CNPs (controller ingress, egress, and CDI-importer egress for the image pull). - clusters/base/apps-vms.yaml: NON-GATING Flux Kustomization (wait:false, dependsOn platform-config) — VMs are a separate lifecycle from the app waves, so a slow VM import/install can't stall the app tier. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-26 15:05:05 -05:00
jakman85andClaude Opus 4.8	36a1a4e1f9	refactor(apps): split monolithic apps Kustomization into dependency waves Replaces the single `apps` Flux Kustomization with dependency-ordered waves (apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream instead of stalling the entire app tier. Parallel within a wave, ordered across. - catalog/render.py: compute waves by longest dependency-path depth over the ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its consumer lands strictly higher — the "DB below consumer" rule falls out of longest-path, no special case). `--emit-waves` generates the per-wave kustomizations + Flux CRs; `--wave-layout` places renders into the right dir. wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db wave 1: sonarr radarr nextcloud semaphore gitea-actions wave 2: seerr - apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact). - clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each). - clusters/dev: one SOPS decryption patch per wave (load-bearing). - DESIGN-waves.md: wave table, topology, cutover runbook. Cutover: preceded by disabling prune on `apps` (`1bfb0db`) so removing that object here does not GC its resources; the wave Kustomizations SSA-adopt the byte-identical manifests in place (no restart, no PVC churn). Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-26 12:19:00 -05:00
jakman85andClaude Opus 4.8	1bfb0db6f1	chore(flux): cutover step 1 — disable prune on `apps` before wave split Temporary: with prune:false, removing the `apps` Kustomization object in the follow-up wave-split commit will NOT garbage-collect its owned resources (CNPG Clusters -> PVCs). The apps-wave-* Kustomizations adopt them in place. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-26 12:14:59 -05:00
jakman85andClaude Opus 4.8	bba67d03ce	feat(dns): add GATEWAY_VIP cluster-var for split-horizon target (R-DNS-03) Parameterize the in-cluster split-horizon target as GATEWAY_VIP (R-ENV-02): base default = dev L2-announced Gateway VIP (10.73.0.240, first IP of the Cilium LB pool). coredns-custom answers *.home.keithjackson.dev with ${GATEWAY_VIP} so the unified graph holds across dev/prod via cluster-vars only. Prod overlay carries a TODO to set the prod LAN VIP alongside the prod L2 pool range when hardware is built. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-24 22:28:37 -05:00
jakman85andClaude Opus 4.8	aafb4ca378	docs(vpn-egress): reflect ADR-0015 node-tunnel egress, retire gluetun narrative The in-cluster Gluetun gateway is retired; VPN egress is now a node-level host OpenVPN split-tunnel (Ansible role vpn_egress_gateway) steered by a Cilium Egress Gateway policy with a host nftables kill-switch (ADR-0015). Credentials moved host-side to the Ansible vault. Update docs/comments only: - platform/vpn-egress/README.md: rewrite for ADR-0015 (live + conformance-green); drop the resolved "option 1 not viable with Gluetun" finding to a brief history note. - platform/kustomization.yaml: Phase 3 comment -> node tunnel + Cilium steering. - platform/kyverno/.../require-egress-class-label.yaml: ADR-0013 -> ADR-0015 steering. - schemas/servicedefinition.schema.json: egress=vpn description -> node tunnel + Cilium. - clusters/base/cluster-vars.yaml: reframe PIA_REGION / CLUSTER_POD_CIDR WHY comments to ADR-0015 (Cilium excludedCIDRs); both keys retained (still referenced/documented). - platform/vpn-egress/namespace.yaml: no privileged gateway pod under ADR-0015. - pia-credentials.sops.yaml / pia-custom-ovpn.sops.yaml: comment headers only note they are now UNUSED (creds host-side), retained pending cleanup; payloads untouched. No runtime manifest behavior, alert exprs, kustomization resources, or SOPS payloads changed. No spec-repo edits. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-24 14:50:08 -05:00
jakman85andClaude Opus 4.8	7d2cea8402	fix(vpn): narrow FIREWALL_OUTBOUND_SUBNETS to 10.0.0.0/16 (PF gateway overlap) PIA's PF gateway 10.31.128.1 sits inside the /8 cluster-pool, so FIREWALL_OUTBOUND_SUBNETS=10.0.0.0/8 made gluetun route it around the tunnel and port forwarding timed out (getSignature to 10.31.128.1:19999). /16 covers actual pod allocations and excludes PIA's 10.31.x. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-22 23:36:51 -05:00
jakman85andClaude Opus 4.8	457edb7a38	fix(vpn): enable gluetun server-list updater (bundled list was stale) + re-enable CORRECTION: the connection failures were gluetun dialing RETIRED PIA IPs (46.246.x = dead → timeouts, which I mis-read as a network block). A fresh PIA .ovpn connects from the same host, and PIA's current SE Stockholm servers (158.173.x) are reachable. Cilium was independently cleared (default-allow, host-fw off, Hubble FORWARDED). Fix: UPDATER_PERIOD=24h so gluetun fetches PIA's current servers; keep OPENVPN_MTU/MSSFIX=1350 for the real Cilium-VXLAN 1450 pod MTU (data channel); PIA default UDP port. Re-enable vpn-egress. Full correction in DECISIONS-LOG. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-22 23:08:18 -05:00
jakman85andClaude Opus 4.8	95a55952cc	fix(vpn): disable vpn-egress — network blackholes PIA IPs (not ports/MTU/cluster) Decisive diagnosis: OpenVPN's outbound init packets are 14 bytes with zero replies (so MTU was never the blocker), pod UDP egress is healthy (NTP replies), but TCP 501/443 to PIA IPs time out while Google/Cloudflare are OPEN — across many PIA servers. This 192.168.2.x WiFi segment blackholes PIA's IP ranges (and intercepts DNS). Not code-fixable. Reverted the port/MTU red-herring env to PIA defaults; disabled vpn-egress pending an operator network fix (move to the unfiltered 192.168.1.x LAN, or unblock PIA on the router). Full trace in DECISIONS-LOG. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-22 18:15:11 -05:00
jakman85andClaude Opus 4.8	4cde690008	fix(vpn): PIA OpenVPN port 53 -> 8080 (53 is DNS-intercepted on this network) Port 53 also got no PIA replies: this network transparently intercepts outbound :53, so dig works but OpenVPN-over-53 is swallowed. An NTP/123 probe from the host returned fine, proving arbitrary non-53 UDP works bidirectionally (so it's port-specific: 1197 conflict + 53 DNS-intercept, not a UDP block). Move to 8080 (community-verified for PIA, non-DNS, non-1197). Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-22 16:39:30 -05:00
jakman85andClaude Opus 4.8	734093cf92	fix(vpn): hardcode OPENVPN_ENDPOINT_PORT "53" as string (Flux var int-coercion) Wiring the port via "${PIA_OPENVPN_PORT}" made Flux/kustomize emit an unquoted 53 (int) after substitution, which k8s rejects for an env value ("expected string, got 53"). Same gotcha as the Longhorn numberOfReplicas fix. Hardcode the quoted string; cluster-var removed with a NOTE explaining why. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-22 16:33:38 -05:00
jakman85andClaude Opus 4.8	ec5ceb328f	fix(vpn): PIA OpenVPN on UDP/53 (avoid home WAN:1197 conflict); re-enable vpn-egress Operator identified the block: the home network NATs its own OpenVPN server on WAN:1197, colliding with the cluster's outbound PIA-on-1197 flow (port-specific, not a general UDP block — UDP/53 worked in tcpdump). PIA accepts OpenVPN on 53/1194/1197/1198/8080/9201; chose 53 (empirically proven to traverse this net). - cluster-vars: add PIA_OPENVPN_PORT="53" - gluetun: OPENVPN_ENDPOINT_PORT=${PIA_OPENVPN_PORT} - re-enable vpn-egress; DECISIONS-LOG resolution noted Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-22 16:28:37 -05:00
jakman85andClaude Opus 4.8	9a108443e8	fix(vpn): PIA_REGION Sweden -> "SE Stockholm" (Gluetun OpenVPN region name) Gluetun's PIA OpenVPN provider rejects "Sweden"; the valid PF-capable Sweden region identifier is "SE Stockholm" ("SE Streaming Optimized" lacks port forwarding). Preserves the operator's Sweden choice. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-22 14:51:29 -05:00
jakman85andClaude Opus 4.8	0d55c793f3	fix(vpn): default PIA_REGION to Sweden (operator choice, PF-capable) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>	2026-08-21 23:00:51 -05:00
jakman85andClaude Opus 4.8	6591e11ad1	fix(vpn): define PIA_REGION + CLUSTER_POD_CIDR cluster-vars; correct pod CIDR + doc where gluetun-gateway.yaml referenced ${PIA_REGION}/${CLUSTER_POD_CIDR} but they were never defined in cluster-vars, and the README didn't say where to set them. Add both to clusters/base/cluster-vars.yaml (ConfigMap data:, string values, Flux postBuild substitution). CLUSTER_POD_CIDR is 10.0.0.0/8 (Cilium cluster-pool — pods get 10.x, confirmed live), NOT the k3s 10.42.0.0/16 I'd assumed. README now names the file + format + per-env override + PF-region note. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>	2026-08-21 22:56:14 -05:00
jakman85andClaude Opus 4.8	20de78f5eb	feat(conformance): author R-TLS + R-DNS L2 tests; fix ExternalDNS zone filter Phase-2 exit tests (were Phase-0 scaffolds), authored against the live platform: - trusted-tls (R-TLS-01/03): in-cluster probe dials the gateway Service (env-agnostic via --connect-to) with wildcard SNI and verifies the served cert CHAINS TO THE LE STAGING ROOT (operator decision A; dev uses LE-staging per ADR-0006). Fixture embeds the '(STAGING) Pretend Pear X1' root. Probes public + internal-only hostnames (R-TLS-03). PASSES. - dns-exposure (R-DNS-02): a LoadBalancer Service with an external-dns hostname; polls public DNS until ExternalDNS auto-publishes the record; teardown + policy=sync removes it. ExternalDNS fix (root cause of the DNS test): domain-filter was the platform SUBDOMAIN home.keithjackson.dev, which filters OUT the actual Cloudflare zone keithjackson.dev ('no hosted zone matching record DNS Name' — record generated but skipped). Add a DNS_ZONE cluster-var (keithjackson.dev) and set domainFilters to it. policy=sync + txt-registry still only manage ExternalDNS-owned records (the pre-existing wildcard is untouched). Resolves the file's WS-NET R-DNS-02 TODO. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>	2026-08-17 08:24:51 -05:00
jakman85andClaude Opus 4.8	c6331c84bf	fix(kyverno): correct chart webhook schema (map + forceFailurePolicyIgnore) Kyverno HelmRelease install failed at Helm templating: config.webhooks was a LIST [{failurePolicy, namespaceSelector}], but the kyverno 3.3.9 chart expects config.webhooks to be a MAP (webhooks.namespaceSelector) — the chart helper reads .namespaceSelector off the value directly ('can't evaluate field namespaceSelector in type []interface {}'). Also, failurePolicy is not a per-webhook field here; the chart's global switch is config.forceFailurePolicyIgnore.enabled (bool). Fix: config.webhooks -> map with namespaceSelector (system+platform excludes); add config.forceFailurePolicyIgnore.enabled = ${KYVERNO_FORCE_FAILURE_POLICY_IGNORE}. Renamed the cluster-var KYVERNO_WEBHOOK_FAILURE_POLICY (Fail/Ignore) -> KYVERNO_FORCE_FAILURE_POLICY_IGNORE (false base / true dev). Verified with helm template (renders 73 objects, no error); gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>	2026-08-17 01:26:04 -05:00
jakman85andClaude Opus 4.8	ca982864fc	fix(platform): controllers/configs split so CRDs precede their CRs Un-suspend deadlocked: the single platform Kustomization applies HelmReleases (which register CRDs) AND custom resources of those CRDs (ClusterIssuer, Certificate, Kyverno ClusterPolicies) in ONE pass. Flux's server-side apply fails atomically ('no matches for kind ClusterIssuer') → nothing installs, so the CRD-registering HelmReleases never run either. Classic ordering deadlock. Fix (flux2 controllers/configs pattern, applies to dev+prod equally — no R-ENV-02 divergence): add a platform-config Flux Kustomization (dependsOn platform, wait) carrying the CRs; platform keeps only the controllers. - New clusters/base/platform-config.yaml (path ./platform-config, substituteFrom cluster-vars, no decryption — no SOPS content); wired into base + apps now dependsOn platform-config (transitively waits for platform). - platform-config/ references the CR files IN PLACE via ../platform/... (Flux builds LoadRestrictionsNone; Kyverno policies must stay under platform/kyverno/ for the CI L0 gate). CR resources dropped from the platform component kustomizations (cert-manager, gateway, kyverno). - Authored the missing Cilium L2 resources (were only a TODO): CiliumLoadBalancerIPPool (dev NAT 10.73.0.240-.250) + CiliumL2AnnouncementPolicy (ens*), so Gateway LoadBalancer listeners get a VIP. Prod range must be parameterized later. Verified: platform build has zero CRs; platform-config build has 2 ClusterIssuers, 6 ClusterPolicies, Certificate, Gateway, both Cilium L2 CRs; dev/prod build; gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>	2026-08-17 01:06:23 -05:00
jakman85andClaude Opus 4.8	a108d5ab98	correction: credential-forward + unified graph; activate SOPS (author-first) Operator directive: don't diverge architecture to avoid asking for credentials; provision them upfront via SOPS and keep the existing (unified) graph. Roadmap reconciliation: Phase 2 = Network/Storage/TLS/DNS/Ingress, so cert-manager/externaldns/gateway (the credential-gated pieces) ARE Phase 2; monitoring is Phase 4. Earlier A2/rollout scope was inverted by credential-avoidance. Reverts (credential-driven): remove clusters/dev/rollout/ + the $patch:delete's; restore unified base graph (dev = ../base + cluster-vars + SOPS decryption, R-ENV-02 intact); restore monitoring files (Phase 4). Kept non-credential improvements: tofu for_each/rejoin, Kyverno parametrization + webhook excludes. platform/kustomization.yaml now phased (single shared list): Phase-2 set = kyverno, cilium, longhorn, cert-manager, externaldns, gateway; later components commented with phase markers. SOPS activated: .sops.yaml at repo root; bootstrap/sops/secret-templates/*.yaml.example for all secrets (cloudflare x2, grafana, alertmanager-smtp, keycloak, oauth2-proxy, pia); bootstrap/sops/CREDENTIALS.md checklist; pre-wired commented cloudflare-api-token lines in cert-manager + externaldns; filled non-secret ACME email. Agents never touch real secret values (operator encrypts). Validated: kubectl kustomize dev(unified)/prod/platform(Phase-2 only), templates inert, make -C ci gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>	2026-08-16 22:09:20 -05:00
jakman85andClaude Opus 4.8	d9a80e5350	phase2-prep: per-component no-secret rollout + component hardening (author-first) Prep only — nothing deployed; running cluster untouched. Reconciliation graph (operator-approved Option A): author clusters/dev/rollout/ = 4 per-component Flux Kustomizations (kyverno->cilium->longhorn->monitoring, monitoring dependsOn longhorn). Dev overlay $patch:delete's the monolithic base platform/apps and adds ./rollout, so only the no-secret tier reconciles; the secret-blocked components are simply not created until hand-offs arrive. The no-secret tier carries no SOPS content, so it needs no sops-age key. prod is untouched (still builds the monolith). Convergence path coded in (delete rollout/ + the two deletes) — R-ENV-02 divergence flagged for architect review. Component hardening via cluster-vars (prod keeps strict posture): - KYVERNO_VALIDATION_ACTION Enforce->Audit(dev), _WEBHOOK_FAILURE_POLICY Fail->Ignore(dev), _ADMISSION_REPLICAS 3->1(dev); webhook namespaceSelector excludes system+platform namespaces. - MONITORING_STORAGE_CLASS var; GRAFANA_ADMIN_SECRET var (dev ''=chart default); Alertmanager SMTP/secret deferred to Phase-4 with verbatim re-enable markers. Cilium parity verified against the live CNI (no-op adopt). Validated: kubectl kustomize dev/prod/all-components, envsubst->YAML parse, make -C ci gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>	2026-08-16 21:34:57 -05:00
jakman85andClaude Opus 4.8	9743b50c3b	fix(K1): uppercase cluster-vars keys to match component substitution refs Flux postBuild.substituteFrom is case-sensitive; platform/ components reference ${UPPERCASE} tokens but cluster-vars keys were lowercase -> none would resolve (literal ${...} would reach the HelmReleases). Uppercased the ConfigMap data keys + the dev/prod overlay patch paths. Every component ref now resolves; clusters/dev & clusters/prod build; gate green. Clears the K1 landmine before any platform un-suspend. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>	2026-08-16 15:16:30 -05:00
jakman85andClaude Opus 4.8	af4f6c07ae	T-0.1 Flux GitOps layout + dev/prod overlays + platform component skeletons (R-GITOPS-01/03, R-ENV-02, R-PLAT-01) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>	2026-08-16 00:08:31 -05:00