jakman85andClaude Opus 5
5642881c4d
MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster DNS was a single pod on a single node, so draining or rebooting that node was a CLUSTER-WIDE in-cluster DNS outage until the pod rescheduled -- every Service lookup, every CNPG replica finding its primary, every operator addressing the API by name. It was invisible because 1/1 Ready looks healthy. It is urgent rather than latent now: the scheduled reboot automation on feat/automated-upgrades drains and reboots one node at a time, unattended, overnight -- turning a single point of failure into a recurring one. WHY AN HPA AND NOT replicas: 2 -- the whole decision. k3s OWNS the coredns Deployment: AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml, managedFields managers deploy@k3s-prod-1 + k3s, objectset.rio.cattle.io/hash label. k3s re-stages its bundled manifests on startup, so replicas: 2 written there -- or a kubectl scale -- is reverted on the next restart or upgrade. Rejected alternatives: * coredns.yaml.skip + our own templated copy (the roles/kube_vip pattern). Deterministic, but takes ownership of coredns from k3s permanently so k3s upgrades stop updating it. Too high a price for a replica count. * A Flux patch of the Deployment. Rejected outright: Flux and k3s's deploy controller would both own one object and fight -- the default-deny NetworkPolicy drift-loop shape (~2500 events/10min) this cluster already hit. An HPA is a separate object k3s does not own, driving /scale rather than the Deployment spec, so there is no shared-ownership fight; if k3s re-stages replicas: 1 it is re-asserted within a reconcile. Disclosed cost: a brief window at 1 replica after a k3s restart. It is a FLOOR-HOLDER, not an autoscaler -- the CPU metric exists only so the autoscaling/v2 object is valid. Measured: 5m/4m against a 100m request (~5%), so the 80% target is never approached. scaleDown stabilization 600s because DNS is the worst thing to flap; scaleUp immediate, since that restores redundancy. The kustomization edit is ADDITIVE: platform/coredns/ already existed and holds the coredns-custom split-horizon ConfigMap (R-DNS-01/03), which is untouched. UNAPPLIED. The HPA has never run, and the claim that k3s reverts replicas on restart is reasoned, not observed. After merge check `describe hpa coredns` for ScalingActive=True -- an inert HPA looks identical to a working one because the hand scale already left 2 replicas. See the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
14 lines
742 B
YAML
14 lines
742 B
YAML
---
|
|
# platform/coredns — in-cluster split-horizon DNS (R-DNS-01 / R-DNS-03).
|
|
# A single `coredns-custom` ConfigMap in kube-system that k3s CoreDNS imports as an extra server
|
|
# block (see coredns-custom.yaml). Rendered by the `platform` Flux Kustomization, which supplies
|
|
# ${GATEWAY_VIP} via substituteFrom cluster-vars (R-ENV-02).
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- coredns-custom.yaml
|
|
# HA floor for the coredns Deployment itself — a separate concern from the split-horizon server
|
|
# block above, and deliberately an HPA rather than a replica edit: k3s owns that Deployment via
|
|
# its own deploy controller and re-stages `replicas: 1` on restart. hpa.yaml has the reasoning.
|
|
- hpa.yaml
|