home-network-k3s-impl

505 Commits 140 Branches 0 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	c8c1d2935d	feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain gate / gate (push) Has been cancelled Details gate / conformance (push) Has been cancelled Details INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources, and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled to 0; verified restored against Cloudflare directly (my first check went through a pfSense override and falsely showed healthy — local resolvers are not evidence about public DNS). THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put "never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified --limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored manifests reached across a third party and destroyed dev's state. Cluster-to-cluster isolation was verified; the dependency that mattered was cluster-to-Cloudflare. Operator's architecture: dev -> .dev.home.keithjackson.dev, prod -> .prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home. PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. + apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and 12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every user, and moves gitea.home. — which is BOTH the GitOps source and the current push target. BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs. Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles" has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized: Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15 links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove --oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11 catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose occurrences and non-reconciled paths. TWO THAT CANNOT BE TOKENIZED, flagged not forced: - headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt), so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy anyway — client secrets are realm-scoped. - kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url, overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets prod.home. EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment — dev renders exactly `default` (the chart default it was already using, so its existing records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL, deliberately not `default`, so a future environment that forgets the knob owns nothing and can delete nothing rather than silently inheriting dev's identity. I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1 (DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE — ExternalDNS generates the record then skips it with "no hosted zone matching record DNS Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev. Isolation comes from the subdomain split plus txtOwnerId, which is what the incident actually needed and what a domain filter cannot provide. DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly. POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12 hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod., and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's. L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS. PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied. Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-09-04 22:43:54 -05:00
jakman85andClaude Opus 4.8	dbd73f1f88	kube-vip: v0.8.9 -> v1.2.3 (k8s 1.36 catch-up) Manifest env-var config (manager/cp_enable/svc_enable/vip_arp/leaderelection) is unchanged across 0.8->1.2, so the bump is image + re-applied auto-deploy manifest. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-30 14:58:33 -05:00
jakman85andClaude Opus 4.8	60a58f01ad	feat(infra): kube-vip control-plane HA VIP (192.168.1.49) + API tls-san Add an HA control-plane API VIP for the k3s cluster with kube-vip, and add the VIP + all node IPs to the kube-apiserver serving-cert SANs so TLS via the VIP validates. Prepares the stable failover endpoint the operator fronts with a pfSense port-forward to 192.168.1.49:6443. roles/kube_vip + playbooks/kube-vip.yml: - Templates a single manifest (RBAC + DaemonSet kube-vip-ds) into each server's k3s auto-deploy dir (/var/lib/rancher/k3s/server/manifests) so it is reproducible and independent of Flux; k3s auto-applies it. - Control-plane ONLY (cp_enable=true, svc_enable=false): Cilium keeps ownership of Service LoadBalancing (L2 pool .53-.60) — no ARP conflict. - Runs on control-plane nodes (nodeSelector node-role.kubernetes.io/control-plane), hostNetwork, NET_ADMIN/NET_RAW, image pinned ghcr.io/kube-vip/kube-vip:v0.8.9. - vip_interface auto-detected per node (the iface holding 192.168.1.5x = eth0). - Leader election (plndr-cp-lock) with 15/10/2s lease timings. - This cluster runs without kube-proxy (Cilium replacement), so a hostNetwork pod cannot reach the kubernetes ClusterIP; kube-vip is pointed at the node-local apiserver (KUBERNETES_SERVICE_HOST=127.0.0.1:6443) so leader election works. roles/k3s_server tls-san: - New k3s_tls_sans (VIP + all three node IPs + 127.0.0.1, deduped); install args now emit one --tls-san per entry. - Persist the SAN list to /etc/rancher/k3s/config.yaml (read by a running k3s) and Restart k3s handler so an existing cluster regenerates its serving cert. - k3s.yml secondaries run serial:1 so a SAN change rolls one node at a time, keeping etcd quorum (kube-vip keeps the API reachable during each restart). Verified live: DS 3/3 ready, VIP answers on 192.168.1.49:6443, all 3 certs and an s_client against the VIP carry 192.168.1.49 + .50/.51/.52 + 127.0.0.1, all nodes Ready throughout the rolling restart. Co-Authored-By: Claude Opus 4.8 <[email protected]>	2026-08-25 21:52:43 -05:00

Author

SHA1

Message

Date

jakman85andClaude Opus 5

c8c1d2935d

feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain

gate / gate (push) Has been cancelled

Details

gate / conformance (push) Has been cancelled

Details

INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).

THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.

Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.

PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.

BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.

TWO THAT CANNOT BE TOKENIZED, flagged not forced:
  - headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
    over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
    so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
    anyway — client secrets are realm-scoped.
  - kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
    reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
    overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
    prod.home.

EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.

I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.

DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.

L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.

PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.

Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

2026-09-04 22:43:54 -05:00

jakman85andClaude Opus 4.8

dbd73f1f88

kube-vip: v0.8.9 -> v1.2.3 (k8s 1.36 catch-up)

Manifest env-var config (manager/cp_enable/svc_enable/vip_arp/leaderelection) is unchanged across
0.8->1.2, so the bump is image + re-applied auto-deploy manifest.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

2026-08-30 14:58:33 -05:00

jakman85andClaude Opus 4.8

60a58f01ad

feat(infra): kube-vip control-plane HA VIP (192.168.1.49) + API tls-san

Add an HA control-plane API VIP for the k3s cluster with kube-vip, and add
the VIP + all node IPs to the kube-apiserver serving-cert SANs so TLS via the
VIP validates. Prepares the stable failover endpoint the operator fronts with
a pfSense port-forward to 192.168.1.49:6443.

roles/kube_vip + playbooks/kube-vip.yml:
- Templates a single manifest (RBAC + DaemonSet kube-vip-ds) into each server's
  k3s auto-deploy dir (/var/lib/rancher/k3s/server/manifests) so it is
  reproducible and independent of Flux; k3s auto-applies it.
- Control-plane ONLY (cp_enable=true, svc_enable=false): Cilium keeps ownership
  of Service LoadBalancing (L2 pool .53-.60) — no ARP conflict.
- Runs on control-plane nodes (nodeSelector node-role.kubernetes.io/control-plane),
  hostNetwork, NET_ADMIN/NET_RAW, image pinned ghcr.io/kube-vip/kube-vip:v0.8.9.
- vip_interface auto-detected per node (the iface holding 192.168.1.5x = eth0).
- Leader election (plndr-cp-lock) with 15/10/2s lease timings.
- This cluster runs without kube-proxy (Cilium replacement), so a hostNetwork
  pod cannot reach the kubernetes ClusterIP; kube-vip is pointed at the
  node-local apiserver (KUBERNETES_SERVICE_HOST=127.0.0.1:6443) so leader
  election works.

roles/k3s_server tls-san:
- New k3s_tls_sans (VIP + all three node IPs + 127.0.0.1, deduped); install
  args now emit one --tls-san per entry.
- Persist the SAN list to /etc/rancher/k3s/config.yaml (read by a running k3s)
  and Restart k3s handler so an existing cluster regenerates its serving cert.
- k3s.yml secondaries run serial:1 so a SAN change rolls one node at a time,
  keeping etcd quorum (kube-vip keeps the API reachable during each restart).

Verified live: DS 3/3 ready, VIP answers on 192.168.1.49:6443, all 3 certs and
an s_client against the VIP carry 192.168.1.49 + .50/.51/.52 + 127.0.0.1, all
nodes Ready throughout the rolling restart.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

2026-08-25 21:52:43 -05:00