home-network-k3s-impl/decisions/2026-10-18-coredns-ha.md

date, branch, title

date	branch	title
2026-10-18	fix/coredns-ha	coredns ran at one replica, so any node drain was a cluster-wide DNS outage

What changed

MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster DNS was a single pod on a single node, so draining or rebooting that node was a cluster-wide in-cluster DNS outage until the pod rescheduled and became ready — every Service lookup, every CNPG replica finding its primary, every operator addressing the API by name. It was invisible because 1/1 Ready looks perfectly healthy.

It also became urgent rather than latent: the scheduled reboot automation on feat/automated-upgrades drains and reboots one node at a time, unattended, overnight — turning a single point of failure into a recurring one.

New platform/coredns/hpa.yaml: a HorizontalPodAutoscaler on the coredns Deployment with minReplicas: 2, maxReplicas: 3, registered in the existing platform/coredns/kustomization.yaml alongside the split-horizon ConfigMap already there.

Why an HPA and not replicas: 2 — this is the whole decision. k3s owns the coredns Deployment and we cannot durably edit it: it is applied by k3s's own deploy controller from a bundled manifest (AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml, managedFields managers deploy@k3s-prod-1 and k3s, plus an objectset.rio.cattle.io/hash label). k3s re-stages its bundled manifests on startup, so a replicas: 2 written there — or a kubectl scale — is reverted on the next k3s restart or upgrade. Two alternatives were weighed and rejected:

• coredns.yaml.skip + our own templated copy (the pattern roles/kube_vip already uses for a file in server/manifests). Deterministic and never flaps, but it takes ownership of coredns away from k3s permanently — k3s upgrades would stop updating it and the version would need refreshing by hand forever. Too high a price for a replica count.
• A Flux-managed patch of the Deployment. Rejected outright: Flux and k3s's deploy controller would both own one object and fight continuously — the exact shape of the default-deny NetworkPolicy drift loop this cluster already suffered (~2500 events/10min).

An HPA avoids both: a separate object k3s does not own, driving the /scale subresource rather than the Deployment spec, so there is no shared-ownership fight. If k3s re-stages replicas: 1, the HPA sees it below minReplicas and scales back within one reconcile.

It is a floor-holder, not an autoscaler: the CPU metric exists only because an autoscaling/v2 HPA needs one to be valid. scaleDown.stabilizationWindowSeconds: 600 because DNS is the worst thing in the cluster to flap; scaleUp is immediate, since that is the path that restores redundancy after a k3s restart.

Immediate action taken outside this PR: kubectl -n kube-system scale deploy coredns --replicas=2 was run at the operator's explicit request, so the cluster has two replicas now rather than after merge. That hand scale is what this PR makes durable.

What I verified

• The defect: kubectl -n kube-system get deploy coredns → replicas=1 available=1.
• k3s ownership: the AddOn/coredns object exists with source /data/k3s/server/manifests/coredns.yaml; the Deployment has no ownerReferences but carries objectset.rio.cattle.io/hash and managedFields managers deploy@k3s-prod-1 / k3s. So it is k3s-managed, not Flux-managed — independently corroborated by another agent, which confirmed platform/coredns/ contains only the coredns-custom ConfigMap and does not touch the Deployment or its replica count.
• The hand scale works and holds: coredns went to 2/2 Running on two different nodes, and in-cluster DNS kept answering throughout (nslookup kubernetes.default.svc.cluster.local 10.43.0.10 → 10.43.0.1). Still spec.replicas=2 ready=2 after ~2 minutes, with the AddOn checksum unchanged — so k3s's deploy controller does not revert on a drift timer; it re-applies on manifest change and on restart.
• The HPA's preconditions hold: coredns ships requests: {cpu: 100m, memory: 70Mi} (so a Resource/Utilization metric is meaningful), and metrics-server is installed as a k3s addon and verified serving — kubectl top pod -l k8s-app=kube-dns returns 5m and 4m against that 100m request, i.e. ~5% utilisation, confirming the 80% target will never be approached and the HPA will sit at minReplicas.
• The kustomization edit is additive: git diff --stat is 1 file changed, 4 insertions(+), and - coredns was already registered in platform/kustomization.yaml, so no change was needed there.
• All three touched/added YAML files parse.

What I did NOT verify

• The HPA has never been applied. I did not reconcile it, so I have not observed it create, reach ScalingActive=True, resolve its metric against real coredns data, or hold minReplicas. Applying is an operator action (CLAUDE.md §5). The failure mode to watch for is ScalingActive=False with FailedGetResourceMetric — the HPA would then be present and inert, which looks identical to working because the hand scale already left 2 replicas in place. Check kubectl -n kube-system describe hpa coredns after merge, not just the replica count.
• The central claim that k3s reverts replicas on restart is REASONED, NOT OBSERVED. It rests on k3s re-staging bundled manifests at startup plus the measured ownership evidence above. I did not restart k3s to watch the revert happen — doing so is a rolling control-plane restart, an operator action. If k3s in fact preserves a modified replica count, this HPA is unnecessary (though harmless).
• I did not confirm the HPA wins the race against k3s's deploy controller after a restart. Both write the scale; the expectation is the HPA re-asserts within a reconcile, leaving a brief window at 1 replica. Neither the window's length nor the outcome has been measured.
• I nearly shipped a destructive change. My first write of platform/coredns/kustomization.yaml replaced the file wholesale, listing only hpa.yaml — which would have pruned the existing coredns-custom ConfigMap and broken in-cluster split-horizon DNS (R-DNS-01/03). It was caught by a guard assertion before commit and restored from HEAD, and the diff is now additive. Recorded because the near-miss is the useful part: platform/coredns/ already existed and I assumed it did not. No conformance check would obviously have caught the removal — it is valid YAML and a legitimate-looking prune.
• Whether 2 is enough. Three nodes, and coredns' anti-affinity only prefers spreading, so both replicas could in principle land on one node and a single drain would still take DNS down. I did not add a topologySpreadConstraint or a PodDisruptionBudget — both arguably belong here and neither is in this change.
• No alert fires if coredns drops to 1 replica, so a future silent regression (k3s upgrade changing the manifest, the HPA going inert) would not be reported. That gap is not closed here.
• No conformance test asserts this, so a future edit could remove the HPA with no gate stage noticing.
• No runtime conformance. vars.CLUSTER_AVAILABLE is unset, so CI proves the L0/L1 offline subset only.
• I did not re-measure branch protection or runner state (CLAUDE.md §2b).