home-network-k3s-impl

0 Fork 0

Author	SHA1	Message	Date
jakman85andClaude Opus 5	d1a6628a97	feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach the cluster and provide fault tolerant HA. Without a change it will not work in HA." THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live before changing anything: all three prod cilium-agent pods carried KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated that stale comment as an open question when the evidence already settled it. SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs: k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join target and a tls-san. infra/tofu/outputs.tf GENERATES this name, so renaming would ripple into tofu and the runbooks. k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium consumes. Derived, not restated, so an environment overrides ONE line and the VIP, the SAN and the CNI endpoint move together. Prod needed no new override. THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran node-baseline -> k3s -> cilium, so Cilium would install against an address nothing announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value: site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier (kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so a cold run stops THERE with a named cause instead of surfacing as unreachable agents. Also fixed a pre-existing bug that only mattered once this play joined site.yml: the role's read-only interface probe ran without `check_mode: false`, so under --check it was skipped and the assert fired with a misleading "set kube_vip_interface explicitly". VERIFIED BY ME, not just reported: renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP) cilium template consumes k3s_apiserver_endpoint, not k3s_api_host cold join still `--server https://{{ k3s_api_host }}:6443` — the founder order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux gate full CI gate PASS Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders ['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved — .9 already answers under full CA verification; and --check --diff of kube-vip.yml against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the kube-vip prod already runs. ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine checks: the Cilium template must use the endpoint var; the endpoint must derive from kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal that environment's VIP and be none of its node IPs (the [parity] contract, now machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium. Every new assertion mutation-tested and bites. One honest negative recorded: reverting only the BASE default is not caught, because both overlays patch the key — the check reads effective per-environment values, so the base default is unguarded. APPLY ORDER, and this is the operator's step — nothing was applied: PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01 incident), it will not proceed past an unready pod, existing datapath survives an agent restart, and .9 is already live before the change lands. DEV SEPARATELY, operator present. Same mechanics but live dev agents dial 192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that. ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook. NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that test's gap list. Refs R-PLAT-01, R-NET-01, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-06 22:17:30 -05:00
jakman85andClaude Opus 5	f178d85c71	fix(kyverno): unblock CDI imports by SATISFYING the label requirement, not exempting it Prod's UniFi VM has no disk. `DataVolume unifi/unifi-rootdisk` stuck in ImportScheduled for hours, because Kyverno denied CDI's own generated pod: validate.kyverno.svc-fail denied Pod/unifi/importer-prime-<uid> require-auth-mode-label (R-IDP-02) require-egress-class-label (R-NET-05) Both policies match any Pod in an `app-namespace: "true"` namespace with no exclusion, and CDI generates `importer-` itself — it cannot carry repo labels. A COLD-START defect, not a prod quirk: dev's identical import Succeeded 9 days ago, before the policies enforced, so dev has never exercised it and would hit it on any rebuild. Fourth bug of this shape in two days. MECHANISM: a MUTATE ClusterPolicy that ADDS `egress-class: internet` + `auth-mode: none` (add-if-absent anchor) to Pods on CREATE in app namespaces WHEN the requesting subject is ServiceAccount `cdi/cdi-sa`. Both validate policies are untouched — no exclude, no exception, `background: true` preserved on them. Why mutate beats exempting: the requirement stays literally true ("every pod in an app namespace declares both labels") and the values are HONEST — the importer does pull over the default WAN path, and `internet` is the least-reach class (it does not grant the VPN path); a transient worker pod with no UI genuinely is `auth-mode: none`. THE MATCH IS IDENTITY, NOT A NAME. `subjects: ServiceAccount cdi/cdi-sa` is the AdmissionReview userInfo, taken from the real prod denial (`system:serviceaccount:cdi:cdi-sa`, name importer-prime-…, op CREATE). A name prefix is attacker-choosable; a requesting ServiceAccount is not — and the test proves it, with a fixture `importer-prime-forged-by-an-app` that copies both the name prefix AND CDI's labels and is still DENIED. Covers every CDI worker kind (importer/uploader/cloner) without enumerating names that change per release. Confirmed on prod that cdi-sa is the only cdi-namespace SA that can create pods. REJECTED, and the first one is a trap worth recording: PolicyException — the CRD EXISTS (v2/v2beta1, from the pre-applied bundle) but all controllers run `--enablePolicyException=false` (verified live on prod: admission, background and reports controllers all carry the flag). An exception would apply cleanly through Flux and be SILENTLY IGNORED. Enabling the gate is a cluster-wide capability change for no gain over the mutate. * `exclude` scoped to cdi-sa — sound identity, worse consequences: any userInfo match forces `background: false`, which would drop background scanning and policy-report coverage of R-NET-05/R-IDP-02 for ALL EXISTING PODS to fix one pod class. It also leaves the importer with no labels at all — the unlabelled fall-through that require-egress-class-label's own header calls a fail-open leak. * name-prefix / CDI-label matching — attacker-choosable, and proven walkable. TWO-WAY PROOF, non-vacuous by construction — same resource, different requester, opposite outcomes (verified by me, not just reported): userinfo cdi-sa -> mutate pass, require-egress-class pass, require-auth-mode pass userinfo ordinary -> mutate SKIP, require-egress-class FAIL, require-auth-mode FAIL Kyverno L0 went 17 -> 29 tests passed, 0 failed. The mutate policy is deliberately kept OUT of the existing tests/kyverno-test.yaml: that suite declares no userinfo, and without userinfo the offline engine does not discriminate on subjects — it mutates everything, which would have silently turned its negative rows into passes. platform-config/kustomization.yaml is the live apply path for these ClusterPolicies; without it the policy never reaches a cluster. RESIDUAL TRADE-OFF, reversible by deleting one file: anything created by cdi-sa in an app namespace now has these labels supplied rather than being rejected for not declaring them — including a pod that is not a CDI worker. That identity already holds `pods create/delete` cluster-wide via ClusterRole `cdi`, so it is not a new capability; what changes is that for that one identity the declaration is assumed rather than demanded. restrict-host-namespaces, the namespace default-deny baseline and the CiliumNetworkPolicies all still apply. FIRST resource-mutating policy on this cluster: kyverno-resource-mutating-webhook-cfg holds 0 webhooks today; with autoUpdateWebhooks=true Kyverno registers mutate.kyverno.svc-fail for Pods within seconds of this applying. Prod's forceFailurePolicyIgnore=false makes it failurePolicy: Fail — same posture as the validating webhook already in that path. CDI retries importer creation on a loop, so the stuck import should proceed once this lands. The detached prime-be3f9d1c-… 40 GiB scratch PVC is separate operator cleanup; untouched, as is the DataVolume. VERIFIED: full CI gate PASS. Nothing applied to any cluster. Refs R-IDP-02, R-NET-05, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-06 22:17:30 -05:00
jakman85andClaude Opus 5	f21e89d9d2	fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details `platform-config` applied the CDI CR and cdi-storageprofiles.yaml in ONE pass. StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and Flux server-side dry-runs every object before applying any — so the unknown kind aborted the tier, the CR was never applied, the CRD was never created, and the tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI CR, no storageprofiles CRD, every downstream tier blocked. The profiles now live in their own tier ./platform-config-cdi with dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated and platform-config runs wait: true, the CDI CR must be applied AND healthy — exactly the event that creates the CRD — before the profiles are applied. Cold clusters order themselves. `standup.sh crds` is retained as a fallback for clusters already stuck, and because it is idempotent. THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is created by a CONTROLLER AT RUNTIME is second-order and needs its own pass. cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles, datavolumes, datasources and dataimportcrons appear only after it reconciles the CDI CR. An audit found StorageProfile is the only second-order kind in platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster, VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium L2 CRs all have their CRD in git or in a chart. Moving the CDI CR up into `platform` was rejected: that would put CDI's health on the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and headlamp splits exist to undo. apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic: apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and its comment claimed platform-config guaranteed the profiles — which this move would have silently falsified. THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has prune: true, so it deletes both profiles once, CDI recreates them with empty spec, and the new tier re-applies the overrides ~30-60s later. Compared our override against CDI's unaided inference for the same Longhorn provisioner on dev: longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block longhorn-static (no override) spec unset -> status ReadWriteOnce/Block Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can no longer cause it (apps-vms gates on the new tier); a human must not do it by hand. Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated — CDI creating and inferring them with no Flux involvement. MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the target first. If it is not, the profiles are pruned and sit at CDI defaults until it recovers — and prod was in that state hours earlier. Fallback is the two-step the Cilium move used (merge with prune: false, verify the ownership label moved, restore prune: true). substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN}, deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits paths whose Kustomization declares it, so omitting it would move the file OUT of the audited set — a blind spot for a future literal-token ship. Gate confirms it stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS decryption stanza: the moved file is two plain CRs, so no overlay change was needed. VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the two profiles; the graph is byte-identical across dev and prod with apps-vms carrying both edges and no cycle. Full CI gate PASS. KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include platform-config-cdi, so `resume` will not wait on it. Another agent holds that file; the PM will add it. Refs R-GITOPS-01, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-06 22:14:19 -05:00
jakman85andClaude Opus 5	2d7d302c7e	feat(prod): cut prod's GitOps source over to prod's OWN in-cluster Gitea gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details Prod reconciled from http://192.168.2.132:3000 — a podman container on the operator's WORKSTATION, holding a DHCP lease on WIFI. Production's source of truth was a laptop: a lease change or a closed lid stalls all reconciliation. Dev was pointed there too (deliberately, to break a strand risk during its DNS migration), so BOTH clusters depended on it simultaneously. Prod now reconciles from https://gitea.prod.home.keithjackson.dev/... — the Gitea prod itself runs, which apps-wave-0 has been serving for 28h. THE SELF-REFERENCE IS THE WHOLE DIFFICULTY, and it is why this is a commit rather than a kubectl patch. The root Kustomization renders ./clusters/prod, which CONTAINS the patch that sets GitRepository.spec.url — so a live patch is reverted on the next reconcile unless the AUTHORED value moves with it. Dev hit the identical trap on 2026-09-05. The ordered cutover: 1. push the repo to prod's Gitea (18 branches, refs verified identical, 499 commits of history — a real move, not a shallow copy); 2. fast-forward prod Gitea's `main` to `c8c1d29`, the EXACT commit prod was already running, so swapping the source changed nothing else. Its main had been at `4809199`, and repointing to that would have ROLLED PROD BACKWARDS, undoing the per-environment BASE_DOMAIN work and sending prod at the apex — the collision with live dev that two days of work exists to prevent; 3. live-patch spec.url to open a window (source changed, content identical: Ready=True at the same revision on the first poll); 4. land THIS commit through that window so the authored value agrees. VERIFIED FROM INSIDE PROD, not from the workstation: a throwaway pod resolved gitea.prod.home.keithjackson.dev through in-cluster CoreDNS to the Gateway VIP (192.168.2.20) and cloned main over the public Let's Encrypt cert with NO CA supplied. That exercises split-horizon DNS, the Gateway, Gitea and TLS on the pod network — the path Flux actually uses. A workstation clone proves none of it. The bootstrap Gitea is retained as a recovery path, not deleted (bootstrap/gitea/bootstrap-gitea.sh). Dev still reconciles from it on branch `dev-live` and returns to its own Gitea at step D4 of the §10a migration runbook. Refs R-GITOPS-01, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-06 22:04:07 -05:00

1 changed files with 19 additions and 3 deletions

									
										clusters/prod/kustomization.yaml
									
		+19
		-3
	
												View File
												
				@@ -185,9 +185,25 @@ patches:

				        name: flux-system

				        namespace: flux-system

				      spec:

				        # TEMPORARY bootstrap source — workstation container, NOT in any cluster. See the block

				        # above for the cutover to https://gitea.home.keithjackson.dev/... . Do not treat as final.

				        url: http://192.168.2.132:3000/Home-Network/home-network-k3s-impl.git

				        # CUTOVER DONE 2026-09-06 (operator direction: "move the repo to prod"). Prod now reconciles

				        # from PROD'S OWN in-cluster Gitea. The workstation bootstrap container

				        # (http://192.168.2.132:3000) is retired as prod's source — it was a podman container on the

				        # operator's workstation holding a DHCP lease on WIFI, i.e. production's source of truth was a

				        # laptop. It is kept only as a recovery path (bootstrap/gitea/bootstrap-gitea.sh).

				        #

				        # THIS LINE IS SELF-REFERENTIAL AND THAT IS THE WHOLE DIFFICULTY. The root Kustomization

				        # renders ./clusters/prod, which contains this patch — so a live `kubectl patch` of

				        # GitRepository.spec.url is REVERTED on the next reconcile unless the authored value moves

				        # too. The cutover therefore had to be: (1) push the repo to prod's Gitea, (2) fast-forward

				        # its `main` to the exact commit prod was already running so the source swap changed nothing

				        # else, (3) live-patch the url to open a window, (4) land THIS commit through that window so

				        # the authored value agrees. Dev hit the identical trap on 2026-09-05

				        # (infra/PROD-PROVISIONING.md §10a "D0/D1 as executed").

				        #

				        # Verified before the switch: a throwaway pod INSIDE prod resolved this host through

				        # in-cluster CoreDNS to the Gateway VIP and cloned over the public LE cert with no CA

				        # supplied. The workstation being able to reach it proves nothing about pod egress.

				        url: https://gitea.prod.home.keithjackson.dev/Home-Network/home-network-k3s-impl.git

				        ref:

				          branch: main

				  - target:

Compare commits