---
# =================================================================================================
# coredns HA floor — hold in-cluster DNS at >= 2 replicas, which k3s does not.
# =================================================================================================
# MEASURED on prod 2026-10-18: `kube-system/coredns` ran with **replicas: 1**. In-cluster DNS is a
# single pod on a single node, so **draining or rebooting that one node is a cluster-wide
# in-cluster DNS outage** for as long as the pod takes to reschedule and become ready — every pod
# that looks up a Service, every CNPG replica finding its primary, every operator talking to the
# API by name. It is the single cheapest availability defect in the cluster, and it was invisible
# because `1/1 Ready` looks perfectly healthy.
#
# This matters more now than it did: the reboot automation (kured + unattended-upgrades) drains and
# reboots one node at a time on a schedule, which turns a latent single point of failure into a
# recurring one, unattended, in the middle of the night.
#
# -------------------------------------------------------------------------------------------------
# WHY AN HPA, AND NOT JUST `replicas: 2` — this is the whole decision
# -------------------------------------------------------------------------------------------------
# **k3s owns the coredns Deployment and we cannot durably edit it.** It is applied by k3s's own
# deploy controller from a bundled manifest, not by Flux:
#
#   $ kubectl -n kube-system get addon coredns
#   NAME      SOURCE                                        CHECKSUM
#   coredns   /data/k3s/server/manifests/coredns.yaml        8cd91595...
#   $ kubectl -n kube-system get deploy coredns -o jsonpath='{.metadata.managedFields[*].manager}'
#   deploy@k3s-prod-1 k3s
#
# and the Deployment carries an `objectset.rio.cattle.io/hash` label — a wrangler objectset. k3s
# RE-STAGES its bundled manifests on startup, so a `replicas: 2` written into that file (or a
# `kubectl scale`) is reverted the next time k3s restarts or is upgraded. MEASURED the same day: a
# hand `kubectl scale` to 2 was NOT actively reverted (it held, and the AddOn checksum did not
# change) — the deploy controller re-applies on manifest change and on restart, not on a drift
# timer. So the hand fix survives until exactly the event we are protecting against: a reboot.
#
# The two durable alternatives were weighed and rejected:
#   * **`coredns.yaml.skip` + our own templated copy** (the pattern roles/kube_vip already uses for
#     a manifest in server/manifests). Fully deterministic and never flaps — but it takes OWNERSHIP
#     of coredns away from k3s permanently, so k3s upgrades would stop updating it and the version
#     would have to be refreshed by hand forever. Too high a price for a replica count.
#   * **A Flux-managed patch of the Deployment itself.** Rejected outright: Flux and k3s's deploy
#     controller would both own one object and fight continuously. That is the shape of the
#     `default-deny` NetworkPolicy drift loop this cluster already suffered (Kyverno generate + Flux
#     both owning one resource, ~2500 events per 10 minutes).
#
# An HPA avoids both. It is a SEPARATE object that k3s does not own and will never reconcile away,
# and it drives the Deployment's `/scale` subresource rather than its spec — so there is no
# shared-ownership fight. If k3s re-stages `replicas: 1`, the HPA observes it under `minReplicas`
# and scales back to 2 within one reconcile (~15s by default).
#
# THE COST, STATED PLAINLY: there is a brief window — seconds, after a k3s restart — where coredns
# is back at 1 replica until the HPA notices. That is strictly better than the status quo, where it
# is at 1 replica permanently, but it is not zero. Accepting that window is the trade being made
# here; the `.skip` option above is what to reach for if it ever proves unacceptable.
#
# -------------------------------------------------------------------------------------------------
# THIS IS A FLOOR-HOLDER, NOT A REAL AUTOSCALER
# -------------------------------------------------------------------------------------------------
# `minReplicas: 2` is the entire point. The CPU metric exists because an autoscaling/v2 HPA needs a
# metric to be a valid object, not because coredns is expected to scale on load: measured via
# metrics-server, the two pods use **5m and 4m** against a **100m** request — ~5% utilisation, so
# the 80% target will never be approached and the HPA will sit at `minReplicas` indefinitely.
#
# The CPU request is what makes this legal: coredns ships `requests: {cpu: 100m, memory: 70Mi}`
# (measured), and a Resource/Utilization metric is meaningless without a request. metrics-server is
# installed as a k3s addon and verified serving (`kubectl top pod -l k8s-app=kube-dns` returns
# figures), so the metric resolves rather than leaving the HPA with `ScalingActive=False`.
#
# `maxReplicas: 3` = the node count. coredns' pod anti-affinity prefers one pod per node, so a
# fourth replica would add no failure-domain coverage; the ceiling exists so a metrics glitch can
# never balloon DNS pods across the cluster.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: coredns
  # kube-system, because that is where k3s puts coredns. This is the only object this component
  # places outside a namespace it owns, and it is additive: it creates nothing in kube-system and
  # mutates only the coredns Deployment's scale subresource.
  namespace: kube-system
  labels:
    app.kubernetes.io/name: coredns
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: coredns
  minReplicas: 2
  maxReplicas: 3
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          # Deliberately high. See "floor-holder" above: this must not scale on normal DNS load.
          averageUtilization: 80
  behavior:
    scaleDown:
      # 10 minutes. DNS is the worst thing in the cluster to flap: a scale-down races in-flight
      # queries and every client's negative cache. The HPA should be reluctant in this direction.
      stabilizationWindowSeconds: 600
      policies:
        - type: Pods
          value: 1
          periodSeconds: 600
    scaleUp:
      # Prompt in the direction that restores redundancy — this is the path taken after a k3s
      # restart has re-staged `replicas: 1`.
      stabilizationWindowSeconds: 0
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60