From 5642881c4dd851edd5156bc7c5a4f327b79f4bc3 Mon Sep 17 00:00:00 2001 From: Keith Jackson Date: Sun, 18 Oct 2026 00:44:09 -0500 Subject: [PATCH] fix(coredns): hold in-cluster DNS at 2 replicas via an HPA, because k3s owns the Deployment MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster DNS was a single pod on a single node, so draining or rebooting that node was a CLUSTER-WIDE in-cluster DNS outage until the pod rescheduled -- every Service lookup, every CNPG replica finding its primary, every operator addressing the API by name. It was invisible because 1/1 Ready looks healthy. It is urgent rather than latent now: the scheduled reboot automation on feat/automated-upgrades drains and reboots one node at a time, unattended, overnight -- turning a single point of failure into a recurring one. WHY AN HPA AND NOT replicas: 2 -- the whole decision. k3s OWNS the coredns Deployment: AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml, managedFields managers deploy@k3s-prod-1 + k3s, objectset.rio.cattle.io/hash label. k3s re-stages its bundled manifests on startup, so replicas: 2 written there -- or a kubectl scale -- is reverted on the next restart or upgrade. Rejected alternatives: * coredns.yaml.skip + our own templated copy (the roles/kube_vip pattern). Deterministic, but takes ownership of coredns from k3s permanently so k3s upgrades stop updating it. Too high a price for a replica count. * A Flux patch of the Deployment. Rejected outright: Flux and k3s's deploy controller would both own one object and fight -- the default-deny NetworkPolicy drift-loop shape (~2500 events/10min) this cluster already hit. An HPA is a separate object k3s does not own, driving /scale rather than the Deployment spec, so there is no shared-ownership fight; if k3s re-stages replicas: 1 it is re-asserted within a reconcile. Disclosed cost: a brief window at 1 replica after a k3s restart. It is a FLOOR-HOLDER, not an autoscaler -- the CPU metric exists only so the autoscaling/v2 object is valid. Measured: 5m/4m against a 100m request (~5%), so the 80% target is never approached. scaleDown stabilization 600s because DNS is the worst thing to flap; scaleUp immediate, since that restores redundancy. The kustomization edit is ADDITIVE: platform/coredns/ already existed and holds the coredns-custom split-horizon ConfigMap (R-DNS-01/03), which is untouched. UNAPPLIED. The HPA has never run, and the claim that k3s reverts replicas on restart is reasoned, not observed. After merge check `describe hpa coredns` for ScalingActive=True -- an inert HPA looks identical to a working one because the hand scale already left 2 replicas. See the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) --- decisions/2026-10-18-coredns-ha.md | 108 +++++++++++++++++++++++++++ platform/coredns/hpa.yaml | 112 ++++++++++++++++++++++++++++ platform/coredns/kustomization.yaml | 4 + 3 files changed, 224 insertions(+) create mode 100644 decisions/2026-10-18-coredns-ha.md create mode 100644 platform/coredns/hpa.yaml diff --git a/decisions/2026-10-18-coredns-ha.md b/decisions/2026-10-18-coredns-ha.md new file mode 100644 index 0000000..7206795 --- /dev/null +++ b/decisions/2026-10-18-coredns-ha.md @@ -0,0 +1,108 @@ +--- +date: 2026-10-18 +branch: fix/coredns-ha +title: "coredns ran at one replica, so any node drain was a cluster-wide DNS outage" +--- + +## What changed + +**MEASURED on prod 2026-10-18: `kube-system/coredns` had `replicas: 1`.** In-cluster DNS was a +single pod on a single node, so draining or rebooting that node was a **cluster-wide in-cluster DNS +outage** until the pod rescheduled and became ready — every Service lookup, every CNPG replica +finding its primary, every operator addressing the API by name. It was invisible because `1/1 +Ready` looks perfectly healthy. + +It also became urgent rather than latent: the scheduled reboot automation on +`feat/automated-upgrades` drains and reboots one node at a time, unattended, overnight — turning a +single point of failure into a recurring one. + +New `platform/coredns/hpa.yaml`: a `HorizontalPodAutoscaler` on the coredns Deployment with +`minReplicas: 2`, `maxReplicas: 3`, registered in the existing `platform/coredns/kustomization.yaml` +alongside the split-horizon ConfigMap already there. + +**Why an HPA and not `replicas: 2` — this is the whole decision.** k3s owns the coredns Deployment +and we cannot durably edit it: it is applied by k3s's own deploy controller from a bundled manifest +(`AddOn/coredns`, source `/data/k3s/server/manifests/coredns.yaml`, `managedFields` managers +`deploy@k3s-prod-1` and `k3s`, plus an `objectset.rio.cattle.io/hash` label). k3s re-stages its +bundled manifests on startup, so a `replicas: 2` written there — or a `kubectl scale` — is reverted +on the next k3s restart or upgrade. Two alternatives were weighed and rejected: + +* **`coredns.yaml.skip` + our own templated copy** (the pattern `roles/kube_vip` already uses for a + file in `server/manifests`). Deterministic and never flaps, but it takes ownership of coredns away + from k3s permanently — k3s upgrades would stop updating it and the version would need refreshing + by hand forever. Too high a price for a replica count. +* **A Flux-managed patch of the Deployment.** Rejected outright: Flux and k3s's deploy controller + would both own one object and fight continuously — the exact shape of the `default-deny` + NetworkPolicy drift loop this cluster already suffered (~2500 events/10min). + +An HPA avoids both: a separate object k3s does not own, driving the `/scale` subresource rather than +the Deployment spec, so there is no shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA +sees it below `minReplicas` and scales back within one reconcile. + +It is a **floor-holder, not an autoscaler**: the CPU metric exists only because an `autoscaling/v2` +HPA needs one to be valid. `scaleDown.stabilizationWindowSeconds: 600` because DNS is the worst +thing in the cluster to flap; `scaleUp` is immediate, since that is the path that restores redundancy +after a k3s restart. + +**Immediate action taken outside this PR:** `kubectl -n kube-system scale deploy coredns +--replicas=2` was run at the operator's explicit request, so the cluster has two replicas **now** +rather than after merge. That hand scale is what this PR makes durable. + +## What I verified + +* **The defect:** `kubectl -n kube-system get deploy coredns` → `replicas=1 available=1`. +* **k3s ownership:** the `AddOn/coredns` object exists with source + `/data/k3s/server/manifests/coredns.yaml`; the Deployment has no `ownerReferences` but carries + `objectset.rio.cattle.io/hash` and `managedFields` managers `deploy@k3s-prod-1` / `k3s`. So it is + k3s-managed, not Flux-managed — independently corroborated by another agent, which confirmed + `platform/coredns/` contains **only** the `coredns-custom` ConfigMap and does not touch the + Deployment or its replica count. +* **The hand scale works and holds:** coredns went to `2/2 Running` on two different nodes, and + in-cluster DNS kept answering throughout (`nslookup kubernetes.default.svc.cluster.local + 10.43.0.10` → `10.43.0.1`). Still `spec.replicas=2 ready=2` after ~2 minutes, with the AddOn + checksum **unchanged** — so k3s's deploy controller does not revert on a drift timer; it + re-applies on manifest change and on restart. +* **The HPA's preconditions hold:** coredns ships `requests: {cpu: 100m, memory: 70Mi}` (so a + Resource/Utilization metric is meaningful), and metrics-server is installed as a k3s addon and + verified serving — `kubectl top pod -l k8s-app=kube-dns` returns **5m and 4m** against that 100m + request, i.e. ~5% utilisation, confirming the 80% target will never be approached and the HPA will + sit at `minReplicas`. +* **The kustomization edit is additive:** `git diff --stat` is `1 file changed, 4 insertions(+)`, + and `- coredns` was already registered in `platform/kustomization.yaml`, so no change was needed + there. +* All three touched/added YAML files parse. + +## What I did NOT verify + +* **The HPA has never been applied.** I did not reconcile it, so I have **not** observed it create, + reach `ScalingActive=True`, resolve its metric against real coredns data, or hold `minReplicas`. + Applying is an operator action (CLAUDE.md §5). The failure mode to watch for is + `ScalingActive=False` with `FailedGetResourceMetric` — the HPA would then be present and inert, + which looks identical to working because the hand scale already left 2 replicas in place. **Check + `kubectl -n kube-system describe hpa coredns` after merge**, not just the replica count. +* **The central claim that k3s reverts `replicas` on restart is REASONED, NOT OBSERVED.** It rests + on k3s re-staging bundled manifests at startup plus the measured ownership evidence above. I did + **not** restart k3s to watch the revert happen — doing so is a rolling control-plane restart, an + operator action. If k3s in fact preserves a modified replica count, this HPA is unnecessary + (though harmless). +* **I did not confirm the HPA wins the race against k3s's deploy controller after a restart.** Both + write the scale; the expectation is the HPA re-asserts within a reconcile, leaving a brief window + at 1 replica. Neither the window's length nor the outcome has been measured. +* **I nearly shipped a destructive change.** My first write of + `platform/coredns/kustomization.yaml` replaced the file wholesale, listing only `hpa.yaml` — which + would have pruned the existing `coredns-custom` ConfigMap and **broken in-cluster split-horizon + DNS (R-DNS-01/03)**. It was caught by a guard assertion before commit and restored from `HEAD`, and + the diff is now additive. Recorded because the near-miss is the useful part: `platform/coredns/` + already existed and I assumed it did not. **No conformance check would obviously have caught the + removal** — it is valid YAML and a legitimate-looking prune. +* **Whether 2 is enough.** Three nodes, and coredns' anti-affinity only *prefers* spreading, so both + replicas could in principle land on one node and a single drain would still take DNS down. I did + not add a `topologySpreadConstraint` or a PodDisruptionBudget — both arguably belong here and + neither is in this change. +* **No alert fires if coredns drops to 1 replica**, so a future silent regression (k3s upgrade + changing the manifest, the HPA going inert) would not be reported. That gap is not closed here. +* **No conformance test asserts this**, so a future edit could remove the HPA with no gate stage + noticing. +* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline + subset only. +* I did not re-measure branch protection or runner state (CLAUDE.md §2b). diff --git a/platform/coredns/hpa.yaml b/platform/coredns/hpa.yaml new file mode 100644 index 0000000..cf0ddf1 --- /dev/null +++ b/platform/coredns/hpa.yaml @@ -0,0 +1,112 @@ +--- +# ================================================================================================= +# coredns HA floor — hold in-cluster DNS at >= 2 replicas, which k3s does not. +# ================================================================================================= +# MEASURED on prod 2026-10-18: `kube-system/coredns` ran with **replicas: 1**. In-cluster DNS is a +# single pod on a single node, so **draining or rebooting that one node is a cluster-wide +# in-cluster DNS outage** for as long as the pod takes to reschedule and become ready — every pod +# that looks up a Service, every CNPG replica finding its primary, every operator talking to the +# API by name. It is the single cheapest availability defect in the cluster, and it was invisible +# because `1/1 Ready` looks perfectly healthy. +# +# This matters more now than it did: the reboot automation (kured + unattended-upgrades) drains and +# reboots one node at a time on a schedule, which turns a latent single point of failure into a +# recurring one, unattended, in the middle of the night. +# +# ------------------------------------------------------------------------------------------------- +# WHY AN HPA, AND NOT JUST `replicas: 2` — this is the whole decision +# ------------------------------------------------------------------------------------------------- +# **k3s owns the coredns Deployment and we cannot durably edit it.** It is applied by k3s's own +# deploy controller from a bundled manifest, not by Flux: +# +# $ kubectl -n kube-system get addon coredns +# NAME SOURCE CHECKSUM +# coredns /data/k3s/server/manifests/coredns.yaml 8cd91595... +# $ kubectl -n kube-system get deploy coredns -o jsonpath='{.metadata.managedFields[*].manager}' +# deploy@k3s-prod-1 k3s +# +# and the Deployment carries an `objectset.rio.cattle.io/hash` label — a wrangler objectset. k3s +# RE-STAGES its bundled manifests on startup, so a `replicas: 2` written into that file (or a +# `kubectl scale`) is reverted the next time k3s restarts or is upgraded. MEASURED the same day: a +# hand `kubectl scale` to 2 was NOT actively reverted (it held, and the AddOn checksum did not +# change) — the deploy controller re-applies on manifest change and on restart, not on a drift +# timer. So the hand fix survives until exactly the event we are protecting against: a reboot. +# +# The two durable alternatives were weighed and rejected: +# * **`coredns.yaml.skip` + our own templated copy** (the pattern roles/kube_vip already uses for +# a manifest in server/manifests). Fully deterministic and never flaps — but it takes OWNERSHIP +# of coredns away from k3s permanently, so k3s upgrades would stop updating it and the version +# would have to be refreshed by hand forever. Too high a price for a replica count. +# * **A Flux-managed patch of the Deployment itself.** Rejected outright: Flux and k3s's deploy +# controller would both own one object and fight continuously. That is the shape of the +# `default-deny` NetworkPolicy drift loop this cluster already suffered (Kyverno generate + Flux +# both owning one resource, ~2500 events per 10 minutes). +# +# An HPA avoids both. It is a SEPARATE object that k3s does not own and will never reconcile away, +# and it drives the Deployment's `/scale` subresource rather than its spec — so there is no +# shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA observes it under `minReplicas` +# and scales back to 2 within one reconcile (~15s by default). +# +# THE COST, STATED PLAINLY: there is a brief window — seconds, after a k3s restart — where coredns +# is back at 1 replica until the HPA notices. That is strictly better than the status quo, where it +# is at 1 replica permanently, but it is not zero. Accepting that window is the trade being made +# here; the `.skip` option above is what to reach for if it ever proves unacceptable. +# +# ------------------------------------------------------------------------------------------------- +# THIS IS A FLOOR-HOLDER, NOT A REAL AUTOSCALER +# ------------------------------------------------------------------------------------------------- +# `minReplicas: 2` is the entire point. The CPU metric exists because an autoscaling/v2 HPA needs a +# metric to be a valid object, not because coredns is expected to scale on load: measured via +# metrics-server, the two pods use **5m and 4m** against a **100m** request — ~5% utilisation, so +# the 80% target will never be approached and the HPA will sit at `minReplicas` indefinitely. +# +# The CPU request is what makes this legal: coredns ships `requests: {cpu: 100m, memory: 70Mi}` +# (measured), and a Resource/Utilization metric is meaningless without a request. metrics-server is +# installed as a k3s addon and verified serving (`kubectl top pod -l k8s-app=kube-dns` returns +# figures), so the metric resolves rather than leaving the HPA with `ScalingActive=False`. +# +# `maxReplicas: 3` = the node count. coredns' pod anti-affinity prefers one pod per node, so a +# fourth replica would add no failure-domain coverage; the ceiling exists so a metrics glitch can +# never balloon DNS pods across the cluster. +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: coredns + # kube-system, because that is where k3s puts coredns. This is the only object this component + # places outside a namespace it owns, and it is additive: it creates nothing in kube-system and + # mutates only the coredns Deployment's scale subresource. + namespace: kube-system + labels: + app.kubernetes.io/name: coredns +spec: + scaleTargetRef: + apiVersion: apps/v1 + kind: Deployment + name: coredns + minReplicas: 2 + maxReplicas: 3 + metrics: + - type: Resource + resource: + name: cpu + target: + type: Utilization + # Deliberately high. See "floor-holder" above: this must not scale on normal DNS load. + averageUtilization: 80 + behavior: + scaleDown: + # 10 minutes. DNS is the worst thing in the cluster to flap: a scale-down races in-flight + # queries and every client's negative cache. The HPA should be reluctant in this direction. + stabilizationWindowSeconds: 600 + policies: + - type: Pods + value: 1 + periodSeconds: 600 + scaleUp: + # Prompt in the direction that restores redundancy — this is the path taken after a k3s + # restart has re-staged `replicas: 1`. + stabilizationWindowSeconds: 0 + policies: + - type: Pods + value: 2 + periodSeconds: 60 diff --git a/platform/coredns/kustomization.yaml b/platform/coredns/kustomization.yaml index 2de8301..d4c2f39 100644 --- a/platform/coredns/kustomization.yaml +++ b/platform/coredns/kustomization.yaml @@ -7,3 +7,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - coredns-custom.yaml + # HA floor for the coredns Deployment itself — a separate concern from the split-horizon server + # block above, and deliberately an HPA rather than a replica edit: k3s owns that Deployment via + # its own deploy controller and re-stages `replicas: 1` on restart. hpa.yaml has the reasoning. + - hpa.yaml