Files
jakman85andClaude Opus 5 a2440a3125
gate / gate (pull_request) Successful in 1m57s
gate / conformance (pull_request) Has been skipped
feat(upgrades): automated OS reboots (kured) + security-only unattended-upgrades, and give Renovate a runner
Two halves of one repeatable upgrade process. Neither works alone.

(a) OS upgrades and reboots

platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own
Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml:
that tier gates platform-config and every apps wave, so a wedged reboot daemon there
could stall every application. A component added to make maintenance safer must not be
able to cause an outage.

infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other
open branches, and this is a different lifecycle), enabled for prod in group_vars.

The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT
REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's
rejection of automating away the gate between nodes: u-u is security-only with the
kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file
appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then
does only the mechanical half, including the ordering rule node_baseline has to shout.

Quorum: concurrent reboots are structurally impossible, not merely unlikely.
concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency),
and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node
holds it forever and nothing cascades.

Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait,
serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate
must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m
(unconditional) plus kured's Prometheus alert gate (semantic).

alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and
under the chart default kured would have been blocked on day one by the five alerts
measured firing with no end in sight, and never rebooted anything.

Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes
from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel
upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it.
Asserted against apt-config dump, not against the file just written. 6.12 kernels are
still installed and bootable on all three nodes, and nothing was held.

Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING
could create the sentinel. kured alone would have idled forever at 1/1 Running.

platform/kured/alerts.yaml — three rules over kured's only metric, with a companion
promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md.
Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this
cluster's 7d Prometheus retention.

(b) Image/dependency automation

Renovate was aspirational: renovate.json present and complete, with no workflow, no
in-cluster workload, and its docker+kubernetes managers disabled — so every plain
image: tag was maintained by nothing while seven files claimed otherwise.

.gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe
first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a
skipped Gitea job reports success and that is how this stayed inert.

Renovate now owns both halves; Flux image-automation is formally closed: it pushes
commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here
anyway — main is protected user_can_push:false and the flux-system GitRepository has
no secretRef. Its controllers are installed and have idled 43 days with zero CRs.

conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing;
tightens the check for this component (requires service.create+metrics.create, both
chart-default false, plus a companion NetworkPolicy requirement). Required because
assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is
shrink-only.

Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author
approving review; /approve-override deliberately NOT posted.

NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role
never executed, and whether CNPG switches its primary on a drain-induced eviction (the
most likely way the first real reboot fails). Full list in the handoff entry.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-18 00:10:29 -05:00

126 lines
6.2 KiB
YAML

---
# Default-deny baseline (R-NET-01): deny all ingress + egress, allow only DNS. `kured` is a PLATFORM
# namespace, so Kyverno's generator — scoped to `app-namespace=true` — does not produce this; we ship
# it explicitly, as crafty/headlamp/homepage/runbooks all do.
#
# THESE POLICIES DO APPLY TO kured, UNLIKE platform/vpn-ingress'. That component's identical-looking
# baseline is honest-but-inert because its DaemonSet runs `hostNetwork: true` and therefore has no
# Cilium endpoint for any policy to select. kured's chart defaults to `hostNetwork: false` and we
# keep that default (it needs the host PID namespace to reboot, not the host network stack), so it
# has a normal pod identity and every rule here is enforced.
#
# Cilium enforces the UNION of these policies (allow if ANY policy allows), so the CiliumNetworkPolicy
# below adds the kube-apiserver path on top of this deny-all floor.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: kured
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# PROMETHEUS MUST REACH :8080, OR EVERY RULE IN alerts.yaml IS INERT.
# ═══════════════════════════════════════════════════════════════════════════════════════════════
# This is not a theoretical hazard in this repo, it is a measured one. From the Longhorn entry in
# conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh: with the ServiceMonitor
# applied and no NetworkPolicy allowance, all three targets were DISCOVERED and every scrape failed
# with "context deadline exceeded" after 10.0s — "a drop, not a refusal" — and the TSDB stayed
# empty. Target discovery succeeding is therefore NOT evidence the scrape works, and the alerts
# would have been quiet rather than broken. KuredNotReporting is the standing detector for exactly
# this regression, and it can only work if this policy is correct.
#
# THE WHOLE `monitoring` NAMESPACE, NOT THE PROMETHEUS POD LABELS — following the same choice
# longhorn/networkpolicy-metrics.yaml made and for the same reason: the prometheus-operator owns
# those pod labels and rewrites them on upgrade (measured 2026-10-17, the pod carries
# `app.kubernetes.io/name: prometheus` plus five operator-managed labels including a
# controller-revision-hash), so selecting on them makes the scrape silently fail on a chart bump.
# The namespace holds only the monitoring stack.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: kured-allow-prometheus-scrape
namespace: kured
labels:
app.kubernetes.io/name: kured
spec:
podSelector:
matchLabels:
# Set by the chart's `kured.matchLabels` helper (_helpers.tpl:70-71) with release name `kured`.
app.kubernetes.io/name: kured
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
ports:
- protocol: TCP
# The container port the chart names `metrics` (daemonset.yaml:211-213), from
# `configuration.metricsPort`, which we leave at the chart default of 8080 and mirror in
# the HelmRelease's `service.port`.
port: 8080
---
# Egress allows. kured talks to exactly two things and nothing else; both are required and the
# component is useless without either.
#
# 1. THE KUBE-APISERVER — the lock annotation on its own DaemonSet (`weave.works/kured-node-lock`),
# cordon/uncordon (`nodes` patch), the node annotations, listing pods, and creating
# `pods/eviction` objects during the drain. Reached via the `kube-apiserver` Cilium entity
# rather than an IP: on this cluster the API is fronted by a kube-vip VIP that moves between
# nodes on every reboot, so any CIDR-based rule here would break during the exact operation
# this component exists to perform. Same construct as platform/headlamp/networkpolicy.yaml.
#
# 2. PROMETHEUS, on 9090 — the alert gate. The HelmRelease sets
# `prometheusUrl: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090`
# and `alertFilterMatchOnly: true`, so before taking the cluster lock kured asks Prometheus
# whether any of LonghornVolumeDegraded / LonghornVolumeFaulted / KubeNodeNotReady /
# NodeExporterDown is firing. IF THIS EGRESS IS BLOCKED, kured cannot evaluate the gate.
# It treats the query error as "do not proceed" and skips the reboot — which is the correct
# fail-safe direction, but it fails SILENTLY and indefinitely: reboots simply stop happening
# and nothing goes red except, eventually, KuredRebootPending. Do not remove this rule
# thinking it is only for metrics; metrics ingress is the policy above, this is the gate.
#
# NOTHING ELSE. No egress to the internet (kured pulls no content at runtime), no DNS beyond the
# baseline above, no Longhorn or etcd access — kured reads Longhorn's health THROUGH Prometheus
# alerts, never from Longhorn's own API, which is why that allow is absent here.
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: kured-allow-egress
namespace: kured
labels:
app.kubernetes.io/name: kured
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: kured
egress:
- toEntities:
- kube-apiserver
- toEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: monitoring
app.kubernetes.io/name: prometheus
toPorts:
- ports:
- port: "9090"
protocol: TCP