home-network-k3s-impl/decisions/2026-10-18-coredns-ha.md

---
date: 2026-10-18
branch: fix/coredns-ha
title: "coredns ran at one replica, so any node drain was a cluster-wide DNS outage"
---

## What changed

**MEASURED on prod 2026-10-18: `kube-system/coredns` had `replicas: 1`.** In-cluster DNS was a
single pod on a single node, so draining or rebooting that node was a **cluster-wide in-cluster DNS
outage** until the pod rescheduled and became ready — every Service lookup, every CNPG replica
finding its primary, every operator addressing the API by name. It was invisible because `1/1
Ready` looks perfectly healthy.

It also became urgent rather than latent: the scheduled reboot automation on
`feat/automated-upgrades` drains and reboots one node at a time, unattended, overnight — turning a
single point of failure into a recurring one.

New `platform/coredns/hpa.yaml`: a `HorizontalPodAutoscaler` on the coredns Deployment with
`minReplicas: 2`, `maxReplicas: 3`, registered in the existing `platform/coredns/kustomization.yaml`
alongside the split-horizon ConfigMap already there.

**Why an HPA and not `replicas: 2` — this is the whole decision.** k3s owns the coredns Deployment
and we cannot durably edit it: it is applied by k3s's own deploy controller from a bundled manifest
(`AddOn/coredns`, source `/data/k3s/server/manifests/coredns.yaml`, `managedFields` managers
`deploy@k3s-prod-1` and `k3s`, plus an `objectset.rio.cattle.io/hash` label). k3s re-stages its
bundled manifests on startup, so a `replicas: 2` written there — or a `kubectl scale` — is reverted
on the next k3s restart or upgrade. Two alternatives were weighed and rejected:

* **`coredns.yaml.skip` + our own templated copy** (the pattern `roles/kube_vip` already uses for a
  file in `server/manifests`). Deterministic and never flaps, but it takes ownership of coredns away
  from k3s permanently — k3s upgrades would stop updating it and the version would need refreshing
  by hand forever. Too high a price for a replica count.
* **A Flux-managed patch of the Deployment.** Rejected outright: Flux and k3s's deploy controller
  would both own one object and fight continuously — the exact shape of the `default-deny`
  NetworkPolicy drift loop this cluster already suffered (~2500 events/10min).

An HPA avoids both: a separate object k3s does not own, driving the `/scale` subresource rather than
the Deployment spec, so there is no shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA
sees it below `minReplicas` and scales back within one reconcile.

It is a **floor-holder, not an autoscaler**: the CPU metric exists only because an `autoscaling/v2`
HPA needs one to be valid. `scaleDown.stabilizationWindowSeconds: 600` because DNS is the worst
thing in the cluster to flap; `scaleUp` is immediate, since that is the path that restores redundancy
after a k3s restart.

**Immediate action taken outside this PR:** `kubectl -n kube-system scale deploy coredns
--replicas=2` was run at the operator's explicit request, so the cluster has two replicas **now**
rather than after merge. That hand scale is what this PR makes durable.

## What I verified

* **The defect:** `kubectl -n kube-system get deploy coredns` → `replicas=1 available=1`.
* **k3s ownership:** the `AddOn/coredns` object exists with source
  `/data/k3s/server/manifests/coredns.yaml`; the Deployment has no `ownerReferences` but carries
  `objectset.rio.cattle.io/hash` and `managedFields` managers `deploy@k3s-prod-1` / `k3s`. So it is
  k3s-managed, not Flux-managed — independently corroborated by another agent, which confirmed
  `platform/coredns/` contains **only** the `coredns-custom` ConfigMap and does not touch the
  Deployment or its replica count.
* **The hand scale works and holds:** coredns went to `2/2 Running` on two different nodes, and
  in-cluster DNS kept answering throughout (`nslookup kubernetes.default.svc.cluster.local
  10.43.0.10` → `10.43.0.1`). Still `spec.replicas=2 ready=2` after ~2 minutes, with the AddOn
  checksum **unchanged** — so k3s's deploy controller does not revert on a drift timer; it
  re-applies on manifest change and on restart.
* **The HPA's preconditions hold:** coredns ships `requests: {cpu: 100m, memory: 70Mi}` (so a
  Resource/Utilization metric is meaningful), and metrics-server is installed as a k3s addon and
  verified serving — `kubectl top pod -l k8s-app=kube-dns` returns **5m and 4m** against that 100m
  request, i.e. ~5% utilisation, confirming the 80% target will never be approached and the HPA will
  sit at `minReplicas`.
* **The kustomization edit is additive:** `git diff --stat` is `1 file changed, 4 insertions(+)`,
  and `- coredns` was already registered in `platform/kustomization.yaml`, so no change was needed
  there.
* All three touched/added YAML files parse.

## What I did NOT verify

* **The HPA has never been applied.** I did not reconcile it, so I have **not** observed it create,
  reach `ScalingActive=True`, resolve its metric against real coredns data, or hold `minReplicas`.
  Applying is an operator action (CLAUDE.md §5). The failure mode to watch for is
  `ScalingActive=False` with `FailedGetResourceMetric` — the HPA would then be present and inert,
  which looks identical to working because the hand scale already left 2 replicas in place. **Check
  `kubectl -n kube-system describe hpa coredns` after merge**, not just the replica count.
* **The central claim that k3s reverts `replicas` on restart is REASONED, NOT OBSERVED.** It rests
  on k3s re-staging bundled manifests at startup plus the measured ownership evidence above. I did
  **not** restart k3s to watch the revert happen — doing so is a rolling control-plane restart, an
  operator action. If k3s in fact preserves a modified replica count, this HPA is unnecessary
  (though harmless).
* **I did not confirm the HPA wins the race against k3s's deploy controller after a restart.** Both
  write the scale; the expectation is the HPA re-asserts within a reconcile, leaving a brief window
  at 1 replica. Neither the window's length nor the outcome has been measured.
* **I nearly shipped a destructive change.** My first write of
  `platform/coredns/kustomization.yaml` replaced the file wholesale, listing only `hpa.yaml` — which
  would have pruned the existing `coredns-custom` ConfigMap and **broken in-cluster split-horizon
  DNS (R-DNS-01/03)**. It was caught by a guard assertion before commit and restored from `HEAD`, and
  the diff is now additive. Recorded because the near-miss is the useful part: `platform/coredns/`
  already existed and I assumed it did not. **No conformance check would obviously have caught the
  removal** — it is valid YAML and a legitimate-looking prune.
* **Whether 2 is enough.** Three nodes, and coredns' anti-affinity only *prefers* spreading, so both
  replicas could in principle land on one node and a single drain would still take DNS down. I did
  not add a `topologySpreadConstraint` or a PodDisruptionBudget — both arguably belong here and
  neither is in this change.
* **No alert fires if coredns drops to 1 replica**, so a future silent regression (k3s upgrade
  changing the manifest, the HPA going inert) would not be reported. That gap is not closed here.
* **No conformance test asserts this**, so a future edit could remove the HPA with no gate stage
  noticing.
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
  subset only.
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).