Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

fix(coredns): hold in-cluster DNS at 2 replicas via an HPA, because k3s owns the Deployment #74

Merged
jakman85 merged 1 commits from fix/coredns-ha into main 2026-10-18 05:50:46 +00:00
Conversation 0 Commits 1 Files Changed 3
+224
gitea-admin commented 2026-10-18 05:46:34 +00:00
Owner
Copy link
Copy source

The defect

kube-system/coredns ran with replicas: 1 (measured on prod 2026-10-18). In-cluster DNS was a single pod on a single node, so draining or rebooting that node was a cluster-wide in-cluster DNS outage until the pod rescheduled and became ready — every Service lookup, every CNPG replica finding its primary, every operator addressing the API by name. It was invisible because 1/1 Ready looks perfectly healthy.

It is urgent rather than latent now: the reboot automation on feat/automated-upgrades drains and reboots one node at a time, unattended, overnight — turning a single point of failure into a recurring one.

coredns has already been scaled to 2 by hand at the operator's request, so the cluster has redundancy now. This PR makes that durable.

Why an HPA and not replicas: 2 — the whole decision

k3s owns the coredns Deployment and we cannot durably edit it. Measured:

$ kubectl -n kube-system get addon coredns
coredns   /data/k3s/server/manifests/coredns.yaml   8cd91595...
$ kubectl -n kube-system get deploy coredns -o jsonpath='{.metadata.managedFields[*].manager}'
deploy@k3s-prod-1 k3s

plus an objectset.rio.cattle.io/hash label — a wrangler objectset. k3s re-stages its bundled manifests on startup, so replicas: 2 written there, or a kubectl scale, is reverted on the next k3s restart or upgrade. (Measured: the hand scale is not actively reverted — it held with the AddOn checksum unchanged — so the deploy controller re-applies on manifest change and on restart, not on a drift timer. Which means the hand fix survives right up until the event it protects against.)

Two alternatives weighed and rejected:

  • coredns.yaml.skip + our own templated copy — the pattern roles/kube_vip already uses for a file in server/manifests. Deterministic and never flaps, but it takes ownership of coredns away from k3s permanently: k3s upgrades would stop updating it and the version would need refreshing by hand forever. Too high a price for a replica count.
  • A Flux-managed patch of the Deployment — rejected outright. Flux and k3s's deploy controller would both own one object and fight continuously, which is the exact shape of the default-deny NetworkPolicy drift loop this cluster already suffered (~2500 events/10min).

An HPA avoids both: a separate object k3s does not own, driving the /scale subresource rather than the Deployment spec, so there is no shared-ownership fight. If k3s re-stages replicas: 1, the HPA sees it below minReplicas and scales back within a reconcile.

Disclosed cost: a brief window — seconds, after a k3s restart — at 1 replica until the HPA notices. Strictly better than permanently at 1, but not zero.

It is a floor-holder, not an autoscaler

minReplicas: 2 is the point. The CPU metric exists only because an autoscaling/v2 HPA needs one to be a valid object. Measured via metrics-server, the two pods use 5m and 4m against a 100m request (~5%), so the 80% target is never approached and the HPA sits at minReplicas indefinitely. coredns ships requests: {cpu: 100m, memory: 70Mi}, which is what makes a Resource/Utilization metric meaningful at all, and metrics-server is verified serving.

maxReplicas: 3 = node count. scaleDown.stabilizationWindowSeconds: 600 because DNS is the worst thing in the cluster to flap; scaleUp is immediate, since that is the path that restores redundancy after a restart.

Additive, and a near-miss worth recording

platform/coredns/ already existed — it holds the coredns-custom ConfigMap for split-horizon DNS (R-DNS-01/03). The first draft of this change rewrote that directory's kustomization.yaml listing only hpa.yaml, which would have pruned the ConfigMap and broken in-cluster split-horizon DNS. A guard assertion caught it pre-commit and it was restored from HEAD; the diff is now 1 file changed, 4 insertions(+). Recorded in the handoff because no conformance check would obviously have caught the removal — it is valid YAML and looks like a legitimate prune.

- coredns was already registered in platform/kustomization.yaml, so nothing was needed there.

Not verified

  • The HPA has never been applied. It has not been observed to create, reach ScalingActive=True, resolve its metric against real data, or hold minReplicas. After merge, check kubectl -n kube-system describe hpa coredns, not the replica count — an inert HPA (ScalingActive=False / FailedGetResourceMetric) looks identical to a working one here, because the hand scale already left 2 replicas in place.
  • The central claim that k3s reverts replicas on restart is reasoned, not observed. It rests on k3s re-staging bundled manifests plus the ownership evidence above; I did not restart k3s to watch it happen (a rolling control-plane restart is an operator action). If k3s in fact preserves a modified replica count, this HPA is unnecessary though harmless.
  • The HPA-vs-deploy-controller race after a restart is unmeasured — neither the window length nor the winner has been observed.
  • Whether 2 is enough. coredns' anti-affinity only prefers spreading, so both replicas could land on one node and a single drain would still take DNS down. No topologySpreadConstraint and no PodDisruptionBudget are added here — both arguably belong.
  • No alert fires if coredns drops to 1 replica, so a future silent regression (a k3s upgrade changing the manifest, or the HPA going inert) would go unreported. Not closed here.
  • No conformance test asserts this, and L2/L3 did not run.

Local gate: rc 3, zero failures — spec-immutability, decisions-log, lint, l0, render-golden, l1, approval-policy all pass; only gitea-settings (CI-only token) and L2/L3 conformance (no cluster) did not run. Not green — rc 3. No sensitive path touched.

Full detail: decisions/2026-10-18-coredns-ha.md

🤖 Generated with Claude Code

## The defect **`kube-system/coredns` ran with `replicas: 1`** (measured on prod 2026-10-18). In-cluster DNS was a single pod on a single node, so **draining or rebooting that node was a cluster-wide in-cluster DNS outage** until the pod rescheduled and became ready — every Service lookup, every CNPG replica finding its primary, every operator addressing the API by name. It was invisible because `1/1 Ready` looks perfectly healthy. It is urgent rather than latent now: the reboot automation on `feat/automated-upgrades` drains and reboots one node at a time, unattended, overnight — turning a single point of failure into a recurring one. **coredns has already been scaled to 2 by hand** at the operator's request, so the cluster has redundancy now. This PR makes that durable. ## Why an HPA and not `replicas: 2` — the whole decision **k3s owns the coredns Deployment and we cannot durably edit it.** Measured: ``` $ kubectl -n kube-system get addon coredns coredns /data/k3s/server/manifests/coredns.yaml 8cd91595... $ kubectl -n kube-system get deploy coredns -o jsonpath='{.metadata.managedFields[*].manager}' deploy@k3s-prod-1 k3s ``` plus an `objectset.rio.cattle.io/hash` label — a wrangler objectset. k3s **re-stages its bundled manifests on startup**, so `replicas: 2` written there, or a `kubectl scale`, is reverted on the next k3s restart or upgrade. (Measured: the hand scale is *not* actively reverted — it held with the AddOn checksum unchanged — so the deploy controller re-applies on manifest change and on restart, not on a drift timer. Which means the hand fix survives right up until the event it protects against.) Two alternatives weighed and rejected: - **`coredns.yaml.skip` + our own templated copy** — the pattern `roles/kube_vip` already uses for a file in `server/manifests`. Deterministic and never flaps, but it takes ownership of coredns away from k3s **permanently**: k3s upgrades would stop updating it and the version would need refreshing by hand forever. Too high a price for a replica count. - **A Flux-managed patch of the Deployment** — rejected outright. Flux and k3s's deploy controller would both own one object and fight continuously, which is the exact shape of the `default-deny` NetworkPolicy drift loop this cluster already suffered (~2500 events/10min). An HPA avoids both: a **separate object k3s does not own**, driving the `/scale` subresource rather than the Deployment spec, so there is no shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA sees it below `minReplicas` and scales back within a reconcile. **Disclosed cost:** a brief window — seconds, after a k3s restart — at 1 replica until the HPA notices. Strictly better than permanently at 1, but not zero. ## It is a floor-holder, not an autoscaler `minReplicas: 2` is the point. The CPU metric exists only because an `autoscaling/v2` HPA needs one to be a valid object. Measured via metrics-server, the two pods use **5m and 4m against a 100m request** (~5%), so the 80% target is never approached and the HPA sits at `minReplicas` indefinitely. coredns ships `requests: {cpu: 100m, memory: 70Mi}`, which is what makes a Resource/Utilization metric meaningful at all, and metrics-server is verified serving. `maxReplicas: 3` = node count. `scaleDown.stabilizationWindowSeconds: 600` because DNS is the worst thing in the cluster to flap; `scaleUp` is immediate, since that is the path that restores redundancy after a restart. ## Additive, and a near-miss worth recording `platform/coredns/` **already existed** — it holds the `coredns-custom` ConfigMap for split-horizon DNS (R-DNS-01/03). The first draft of this change rewrote that directory's `kustomization.yaml` listing only `hpa.yaml`, which would have **pruned the ConfigMap and broken in-cluster split-horizon DNS.** A guard assertion caught it pre-commit and it was restored from `HEAD`; the diff is now `1 file changed, 4 insertions(+)`. Recorded in the handoff because **no conformance check would obviously have caught the removal** — it is valid YAML and looks like a legitimate prune. `- coredns` was already registered in `platform/kustomization.yaml`, so nothing was needed there. ## Not verified - **The HPA has never been applied.** It has not been observed to create, reach `ScalingActive=True`, resolve its metric against real data, or hold `minReplicas`. **After merge, check `kubectl -n kube-system describe hpa coredns`, not the replica count** — an inert HPA (`ScalingActive=False` / `FailedGetResourceMetric`) looks *identical* to a working one here, because the hand scale already left 2 replicas in place. - **The central claim that k3s reverts `replicas` on restart is reasoned, not observed.** It rests on k3s re-staging bundled manifests plus the ownership evidence above; I did not restart k3s to watch it happen (a rolling control-plane restart is an operator action). If k3s in fact preserves a modified replica count, this HPA is unnecessary though harmless. - **The HPA-vs-deploy-controller race after a restart is unmeasured** — neither the window length nor the winner has been observed. - **Whether 2 is enough.** coredns' anti-affinity only *prefers* spreading, so both replicas could land on one node and a single drain would still take DNS down. **No `topologySpreadConstraint` and no PodDisruptionBudget are added here** — both arguably belong. - **No alert fires if coredns drops to 1 replica**, so a future silent regression (a k3s upgrade changing the manifest, or the HPA going inert) would go unreported. Not closed here. - No conformance test asserts this, and L2/L3 did not run. Local gate: **rc 3, zero failures** — `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1`, `approval-policy` all pass; only `gitea-settings` (CI-only token) and L2/L3 `conformance` (no cluster) did not run. Not green — rc 3. No sensitive path touched. Full detail: `decisions/2026-10-18-coredns-ha.md` 🤖 Generated with [Claude Code](https://claude.com/claude-code)
gitea-admin added 1 commit 2026-10-18 05:46:34 +00:00
fix(coredns): hold in-cluster DNS at 2 replicas via an HPA, because k3s owns the Deployment
gate / gate (pull_request) Successful in 1m52s
Details
gate / conformance (pull_request) Has been skipped
Details
5642881c4d 
MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster
DNS was a single pod on a single node, so draining or rebooting that node was a
CLUSTER-WIDE in-cluster DNS outage until the pod rescheduled -- every Service
lookup, every CNPG replica finding its primary, every operator addressing the
API by name. It was invisible because 1/1 Ready looks healthy.

It is urgent rather than latent now: the scheduled reboot automation on
feat/automated-upgrades drains and reboots one node at a time, unattended,
overnight -- turning a single point of failure into a recurring one.

WHY AN HPA AND NOT replicas: 2 -- the whole decision. k3s OWNS the coredns
Deployment: AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml,
managedFields managers deploy@k3s-prod-1 + k3s, objectset.rio.cattle.io/hash
label. k3s re-stages its bundled manifests on startup, so replicas: 2 written
there -- or a kubectl scale -- is reverted on the next restart or upgrade.

Rejected alternatives:
* coredns.yaml.skip + our own templated copy (the roles/kube_vip pattern).
  Deterministic, but takes ownership of coredns from k3s permanently so k3s
  upgrades stop updating it. Too high a price for a replica count.
* A Flux patch of the Deployment. Rejected outright: Flux and k3s's deploy
  controller would both own one object and fight -- the default-deny
  NetworkPolicy drift-loop shape (~2500 events/10min) this cluster already hit.

An HPA is a separate object k3s does not own, driving /scale rather than the
Deployment spec, so there is no shared-ownership fight; if k3s re-stages
replicas: 1 it is re-asserted within a reconcile. Disclosed cost: a brief
window at 1 replica after a k3s restart.

It is a FLOOR-HOLDER, not an autoscaler -- the CPU metric exists only so the
autoscaling/v2 object is valid. Measured: 5m/4m against a 100m request (~5%),
so the 80% target is never approached. scaleDown stabilization 600s because DNS
is the worst thing to flap; scaleUp immediate, since that restores redundancy.

The kustomization edit is ADDITIVE: platform/coredns/ already existed and holds
the coredns-custom split-horizon ConfigMap (R-DNS-01/03), which is untouched.

UNAPPLIED. The HPA has never run, and the claim that k3s reverts replicas on
restart is reasoned, not observed. After merge check `describe hpa coredns` for
ScalingActive=True -- an inert HPA looks identical to a working one because the
hand scale already left 2 replicas. See the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
jakman85 merged commit 24bea918ab into main 2026-10-18 05:50:46 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
agent
automated-update
blast-radius
breaking
ci-tooling
dependencies
needs-review
own-image
range-widen
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
gitea-admin jakman85 (Keith Jackson) pm-agent renovate-bot
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: Home-Network/home-network-k3s-impl#74