jakman85andClaude Opus 5
5642881c4d
MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster DNS was a single pod on a single node, so draining or rebooting that node was a CLUSTER-WIDE in-cluster DNS outage until the pod rescheduled -- every Service lookup, every CNPG replica finding its primary, every operator addressing the API by name. It was invisible because 1/1 Ready looks healthy. It is urgent rather than latent now: the scheduled reboot automation on feat/automated-upgrades drains and reboots one node at a time, unattended, overnight -- turning a single point of failure into a recurring one. WHY AN HPA AND NOT replicas: 2 -- the whole decision. k3s OWNS the coredns Deployment: AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml, managedFields managers deploy@k3s-prod-1 + k3s, objectset.rio.cattle.io/hash label. k3s re-stages its bundled manifests on startup, so replicas: 2 written there -- or a kubectl scale -- is reverted on the next restart or upgrade. Rejected alternatives: * coredns.yaml.skip + our own templated copy (the roles/kube_vip pattern). Deterministic, but takes ownership of coredns from k3s permanently so k3s upgrades stop updating it. Too high a price for a replica count. * A Flux patch of the Deployment. Rejected outright: Flux and k3s's deploy controller would both own one object and fight -- the default-deny NetworkPolicy drift-loop shape (~2500 events/10min) this cluster already hit. An HPA is a separate object k3s does not own, driving /scale rather than the Deployment spec, so there is no shared-ownership fight; if k3s re-stages replicas: 1 it is re-asserted within a reconcile. Disclosed cost: a brief window at 1 replica after a k3s restart. It is a FLOOR-HOLDER, not an autoscaler -- the CPU metric exists only so the autoscaling/v2 object is valid. Measured: 5m/4m against a 100m request (~5%), so the 80% target is never approached. scaleDown stabilization 600s because DNS is the worst thing to flap; scaleUp immediate, since that restores redundancy. The kustomization edit is ADDITIVE: platform/coredns/ already existed and holds the coredns-custom split-horizon ConfigMap (R-DNS-01/03), which is untouched. UNAPPLIED. The HPA has never run, and the claim that k3s reverts replicas on restart is reasoned, not observed. After merge check `describe hpa coredns` for ScalingActive=True -- an inert HPA looks identical to a working one because the hand scale already left 2 replicas. See the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
113 lines
6.4 KiB
YAML
113 lines
6.4 KiB
YAML
---
|
|
# =================================================================================================
|
|
# coredns HA floor — hold in-cluster DNS at >= 2 replicas, which k3s does not.
|
|
# =================================================================================================
|
|
# MEASURED on prod 2026-10-18: `kube-system/coredns` ran with **replicas: 1**. In-cluster DNS is a
|
|
# single pod on a single node, so **draining or rebooting that one node is a cluster-wide
|
|
# in-cluster DNS outage** for as long as the pod takes to reschedule and become ready — every pod
|
|
# that looks up a Service, every CNPG replica finding its primary, every operator talking to the
|
|
# API by name. It is the single cheapest availability defect in the cluster, and it was invisible
|
|
# because `1/1 Ready` looks perfectly healthy.
|
|
#
|
|
# This matters more now than it did: the reboot automation (kured + unattended-upgrades) drains and
|
|
# reboots one node at a time on a schedule, which turns a latent single point of failure into a
|
|
# recurring one, unattended, in the middle of the night.
|
|
#
|
|
# -------------------------------------------------------------------------------------------------
|
|
# WHY AN HPA, AND NOT JUST `replicas: 2` — this is the whole decision
|
|
# -------------------------------------------------------------------------------------------------
|
|
# **k3s owns the coredns Deployment and we cannot durably edit it.** It is applied by k3s's own
|
|
# deploy controller from a bundled manifest, not by Flux:
|
|
#
|
|
# $ kubectl -n kube-system get addon coredns
|
|
# NAME SOURCE CHECKSUM
|
|
# coredns /data/k3s/server/manifests/coredns.yaml 8cd91595...
|
|
# $ kubectl -n kube-system get deploy coredns -o jsonpath='{.metadata.managedFields[*].manager}'
|
|
# deploy@k3s-prod-1 k3s
|
|
#
|
|
# and the Deployment carries an `objectset.rio.cattle.io/hash` label — a wrangler objectset. k3s
|
|
# RE-STAGES its bundled manifests on startup, so a `replicas: 2` written into that file (or a
|
|
# `kubectl scale`) is reverted the next time k3s restarts or is upgraded. MEASURED the same day: a
|
|
# hand `kubectl scale` to 2 was NOT actively reverted (it held, and the AddOn checksum did not
|
|
# change) — the deploy controller re-applies on manifest change and on restart, not on a drift
|
|
# timer. So the hand fix survives until exactly the event we are protecting against: a reboot.
|
|
#
|
|
# The two durable alternatives were weighed and rejected:
|
|
# * **`coredns.yaml.skip` + our own templated copy** (the pattern roles/kube_vip already uses for
|
|
# a manifest in server/manifests). Fully deterministic and never flaps — but it takes OWNERSHIP
|
|
# of coredns away from k3s permanently, so k3s upgrades would stop updating it and the version
|
|
# would have to be refreshed by hand forever. Too high a price for a replica count.
|
|
# * **A Flux-managed patch of the Deployment itself.** Rejected outright: Flux and k3s's deploy
|
|
# controller would both own one object and fight continuously. That is the shape of the
|
|
# `default-deny` NetworkPolicy drift loop this cluster already suffered (Kyverno generate + Flux
|
|
# both owning one resource, ~2500 events per 10 minutes).
|
|
#
|
|
# An HPA avoids both. It is a SEPARATE object that k3s does not own and will never reconcile away,
|
|
# and it drives the Deployment's `/scale` subresource rather than its spec — so there is no
|
|
# shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA observes it under `minReplicas`
|
|
# and scales back to 2 within one reconcile (~15s by default).
|
|
#
|
|
# THE COST, STATED PLAINLY: there is a brief window — seconds, after a k3s restart — where coredns
|
|
# is back at 1 replica until the HPA notices. That is strictly better than the status quo, where it
|
|
# is at 1 replica permanently, but it is not zero. Accepting that window is the trade being made
|
|
# here; the `.skip` option above is what to reach for if it ever proves unacceptable.
|
|
#
|
|
# -------------------------------------------------------------------------------------------------
|
|
# THIS IS A FLOOR-HOLDER, NOT A REAL AUTOSCALER
|
|
# -------------------------------------------------------------------------------------------------
|
|
# `minReplicas: 2` is the entire point. The CPU metric exists because an autoscaling/v2 HPA needs a
|
|
# metric to be a valid object, not because coredns is expected to scale on load: measured via
|
|
# metrics-server, the two pods use **5m and 4m** against a **100m** request — ~5% utilisation, so
|
|
# the 80% target will never be approached and the HPA will sit at `minReplicas` indefinitely.
|
|
#
|
|
# The CPU request is what makes this legal: coredns ships `requests: {cpu: 100m, memory: 70Mi}`
|
|
# (measured), and a Resource/Utilization metric is meaningless without a request. metrics-server is
|
|
# installed as a k3s addon and verified serving (`kubectl top pod -l k8s-app=kube-dns` returns
|
|
# figures), so the metric resolves rather than leaving the HPA with `ScalingActive=False`.
|
|
#
|
|
# `maxReplicas: 3` = the node count. coredns' pod anti-affinity prefers one pod per node, so a
|
|
# fourth replica would add no failure-domain coverage; the ceiling exists so a metrics glitch can
|
|
# never balloon DNS pods across the cluster.
|
|
apiVersion: autoscaling/v2
|
|
kind: HorizontalPodAutoscaler
|
|
metadata:
|
|
name: coredns
|
|
# kube-system, because that is where k3s puts coredns. This is the only object this component
|
|
# places outside a namespace it owns, and it is additive: it creates nothing in kube-system and
|
|
# mutates only the coredns Deployment's scale subresource.
|
|
namespace: kube-system
|
|
labels:
|
|
app.kubernetes.io/name: coredns
|
|
spec:
|
|
scaleTargetRef:
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
name: coredns
|
|
minReplicas: 2
|
|
maxReplicas: 3
|
|
metrics:
|
|
- type: Resource
|
|
resource:
|
|
name: cpu
|
|
target:
|
|
type: Utilization
|
|
# Deliberately high. See "floor-holder" above: this must not scale on normal DNS load.
|
|
averageUtilization: 80
|
|
behavior:
|
|
scaleDown:
|
|
# 10 minutes. DNS is the worst thing in the cluster to flap: a scale-down races in-flight
|
|
# queries and every client's negative cache. The HPA should be reluctant in this direction.
|
|
stabilizationWindowSeconds: 600
|
|
policies:
|
|
- type: Pods
|
|
value: 1
|
|
periodSeconds: 600
|
|
scaleUp:
|
|
# Prompt in the direction that restores redundancy — this is the path taken after a k3s
|
|
# restart has re-staged `replicas: 1`.
|
|
stabilizationWindowSeconds: 0
|
|
policies:
|
|
- type: Pods
|
|
value: 2
|
|
periodSeconds: 60
|