2 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	6f61c9622d	feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite ... OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach the cluster and provide fault tolerant HA. Without a change it will not work in HA." THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live before changing anything: all three prod cilium-agent pods carried KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated that stale comment as an open question when the evidence already settled it. SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs: k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join target and a tls-san. infra/tofu/outputs.tf GENERATES this name, so renaming would ripple into tofu and the runbooks. k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium consumes. Derived, not restated, so an environment overrides ONE line and the VIP, the SAN and the CNI endpoint move together. Prod needed no new override. THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran node-baseline -> k3s -> cilium, so Cilium would install against an address nothing announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value: site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier (kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so a cold run stops THERE with a named cause instead of surfacing as unreachable agents. Also fixed a pre-existing bug that only mattered once this play joined site.yml: the role's read-only interface probe ran without `check_mode: false`, so under --check it was skipped and the assert fired with a misleading "set kube_vip_interface explicitly". VERIFIED BY ME, not just reported: renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP) cilium template consumes k3s_apiserver_endpoint, not k3s_api_host cold join still `--server https://{{ k3s_api_host }}:6443` — the founder order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux gate full CI gate PASS Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders ['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved — .9 already answers under full CA verification; and --check --diff of kube-vip.yml against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the kube-vip prod already runs. ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine checks: the Cilium template must use the endpoint var; the endpoint must derive from kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal that environment's VIP and be none of its node IPs (the [parity] contract, now machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium. Every new assertion mutation-tested and bites. One honest negative recorded: reverting only the BASE default is not caught, because both overlays patch the key — the check reads effective per-environment values, so the base default is unguarded. APPLY ORDER, and this is the operator's step — nothing was applied: PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01 incident), it will not proceed past an unready pod, existing datapath survives an agent restart, and .9 is already live before the change lands. DEV SEPARATELY, operator present. Same mechanics but live dev agents dial 192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that. ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook. NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that test's gap list. Refs R-PLAT-01, R-NET-01, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-06 20:24:51 -05:00
jakman85andClaude Opus 5	cb1916f092	feat(conformance): cover R-PLAT-01, R-NET-04, R-SEC-03 — 88.2% -> 94.1%, and fix an unrecordable-proof defect ... Coverage 45/51 -> 48/51, gaps 6 -> 3 (R-C2-01, R-C2-02, R-STATE-04), 0 safety-critical. --strict-coverage exit 0; run.sh --layer l0 11/11 PASS. R-PLAT-01 — split L0 + L2, because "3 nodes with production-grade HA control plane" is half authored and half observable. l0-static/ha-control-plane-as-code (92 assertions): environments DERIVED from the R-ENV-02 seam (never a hardcoded group name); k3s installed as `server` with no --datastore-endpoint/--disable-etcd anywhere; exactly ONE --cluster-init founder per environment; odd, quorum-capable server count with tolerance >= 1; tls-sans carry the VIP and every server IP; kube-vip leader-elected with cp_enable/svc_enable parsed from the template's OWN env pairs; each VIP outside its environment's LB pool and not a node IP. l2-behavioral/ha-control-plane: control-plane AND etcd role on every node, every node a REGISTERED etcd member, exactly 3 when the cluster says CLUSTER_ENV=prod, and an in-cluster Job curling /readyz through the VIP AND every server's own address — which proves the VIP fronts the API and that every server runs an apiserver, not one founder plus two joiners. An earlier draft probed those endpoints FROM THE RUNNER: that passes from prod's LAN and fails against dev (nodes behind the pfSense NAT), making the test a property of where it was launched. Hence the in-cluster Job. R-NET-04 — what it adds over R-NET-05/07 is the PATH SET and the totality/disjointness of the selecting rules, not another per-service check. l0-static/egress-paths-selected-by-rules (68 assertions): exactly two paths, the same closed set in schema + Kyverno policy + renderer; EXACTLY ONE CiliumEgressGatewayPolicy with a single selector matching only egress-class=vpn; nothing selects `internet` (absence of a rule IS the path); RFC1918 containment computed with `ipaddress` plus every address each overlay assigns itself; the tunnel default route and blackhole confined to table 51820 behind an ip rule that must carry a from/fwmark/iif selector; and the nft chain `policy accept` with every drop qualified — the internet-fails-OPEN direction that R-NET-06's L3 test does not assert. R-SEC-03 — refuses the tautology "a test exists, therefore it is auditable". l0-static/auditability-proofs-producible (78 assertions) drives the real machinery: each of docs/07's five proofs must have an EXECUTABLE test at the layer the spec's wording demands (fault injection and DR drill => L3), taken from status.py's own verdict; run.sh --requirements must resolve and pass the gate's --require assertion; each proof declares a sentinel record_run.py can attribute, computed with record_run.parse_sentinels itself; and TWO FORGERIES are mounted against the real tools — a green JUnit whose log holds only a SKIP token, and a ledger row claiming `proven` at the root commit — which must come back not-proven and stale-proof. The five proofs are parsed from docs/07 when the spec is mounted, not hand-listed. *** A REAL DEFECT FOUND AND FIXED, not asserted around. *** l2-behavioral's default-deny, trusted-tls and gateway-ingress-reachable declared NO pass sentinel, so record_run.py could never attribute a pass to them — the isolation and TLS audit proofs were STRUCTURALLY UNRECORDABLE (landing as unknown-no-sentinel-declared, and status.py requires all of a requirement's tests to be proven). Each now ends with an additive terminal sentinel step reached only when every prior step passed. Verified: all three carry it. TWO HOLLOW DRAFT ASSERTIONS CAUGHT BY THE AGENT ITSELF, both the failure mode this repo keeps hitting: (a) `svc_enable=false` matched a COMMENT in the kube-vip role defaults, so flipping the template to true still passed — now parses the DaemonSet template's env pairs; (b) the nft scoping check searched for the VALUE of a Jinja variable while the template carries its NAME, matching nothing and passing vacuously — templates are now rendered against role defaults first. 30 perturbations across the three checks, all caught, plus 16 synthetic L2 perturbations via a fake-kubectl shim. R-C2-01/02 judged IMPLEMENTATION gaps, so no test was written: Homepage's widgets.yaml carries only datetime and search, kubernetes.yaml is empty, automountServiceAccountToken is false, and its NetworkPolicy allows DNS egress only — no health source is even reachable. It has already drifted (crafty authors a route with no tile). A test today would go red on both; it must ship WITH the config fix. VERIFIED: full CI gate PASS. Neither cluster mutated. Refs R-PLAT-01, R-NET-04, R-SEC-03, R-TEST-01..05. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-06 19:35:18 -05:00