jakman85andClaude Opus 5
a2440a3125
Two halves of one repeatable upgrade process. Neither works alone. (a) OS upgrades and reboots platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml: that tier gates platform-config and every apps wave, so a wedged reboot daemon there could stall every application. A component added to make maintenance safer must not be able to cause an outage. infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other open branches, and this is a different lifecycle), enabled for prod in group_vars. The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's rejection of automating away the gate between nodes: u-u is security-only with the kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then does only the mechanical half, including the ordering rule node_baseline has to shout. Quorum: concurrent reboots are structurally impossible, not merely unlikely. concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency), and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node holds it forever and nothing cascades. Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait, serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m (unconditional) plus kured's Prometheus alert gate (semantic). alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and under the chart default kured would have been blocked on day one by the five alerts measured firing with no end in sight, and never rebooted anything. Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it. Asserted against apt-config dump, not against the file just written. 6.12 kernels are still installed and bootable on all three nodes, and nothing was held. Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING could create the sentinel. kured alone would have idled forever at 1/1 Running. platform/kured/alerts.yaml — three rules over kured's only metric, with a companion promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md. Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this cluster's 7d Prometheus retention. (b) Image/dependency automation Renovate was aspirational: renovate.json present and complete, with no workflow, no in-cluster workload, and its docker+kubernetes managers disabled — so every plain image: tag was maintained by nothing while seven files claimed otherwise. .gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a skipped Gitea job reports success and that is how this stayed inert. Renovate now owns both halves; Flux image-automation is formally closed: it pushes commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here anyway — main is protected user_can_push:false and the flux-system GitRepository has no secretRef. Its controllers are installed and have idled 43 days with zero CRs. conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing; tightens the check for this component (requires service.create+metrics.create, both chart-default false, plus a companion NetworkPolicy requirement). Required because assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is shrink-only. Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author approving review; /approve-override deliberately NOT posted. NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role never executed, and whether CNPG switches its primary on a drain-induced eviction (the most likely way the first real reboot fails). Full list in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
102 lines
9.0 KiB
YAML
102 lines
9.0 KiB
YAML
---
|
|
# Base Flux entrypoint set. Shared by BOTH environment overlays (clusters/dev, clusters/prod).
|
|
# Overlays reference this base and patch ONLY the cluster-vars ConfigMap (R-ENV-02).
|
|
apiVersion: kustomize.config.k8s.io/v1beta1
|
|
kind: Kustomization
|
|
resources:
|
|
- flux-system.yaml # GitRepository (reconciliation source)
|
|
# Cilium (the CNI) — its OWN Kustomization, split OUT of `platform` (2026-09-01). The mirror image
|
|
# of the oauth2-proxy/headlamp splits: those are leaves that must never gate a tier (wait:false);
|
|
# Cilium is the one component that legitimately SHOULD gate everything, so it keeps wait:true and
|
|
# `platform` dependsOn it. Splitting it out buys a per-CNI Ready condition, CNI-appropriate
|
|
# interval/timeout, and `flux suspend kustomization cilium` as an emergency valve that does not
|
|
# freeze the other eleven platform components. See clusters/base/cilium.yaml.
|
|
- cilium.yaml # Flux Kustomization -> ./platform/cilium (dependsOn flux-system, wait:true)
|
|
# Intel GPU device plugin — its OWN Kustomization (dependsOn cilium, wait:true), NOT a line in
|
|
# platform/kustomization.yaml. Same reasoning as the Cilium split-out, applied to a leaf: inside
|
|
# that wait:true tier a wedged 3-node DaemonSet would be indistinguishable from eleven unrelated
|
|
# components AND would stall platform-config + every apps wave on "dependency not ready" — so a
|
|
# component added to fix a silent no-op (Plex software transcoding) could cause a real outage.
|
|
# wait:true here is a HEALTH SIGNAL, not a gate: nothing in the graph dependsOn it.
|
|
# It advertises `gpu.intel.com/i915`; the renderer change that REQUESTS that resource must land
|
|
# strictly later. See clusters/base/intel-gpu-plugin.yaml.
|
|
- intel-gpu-plugin.yaml # Flux Kustomization -> ./platform/intel-gpu-plugin (dependsOn cilium)
|
|
- platform.yaml # Flux Kustomization -> ./platform (CONTROLLERS: HelmReleases / CRDs)
|
|
- platform-config.yaml # Flux Kustomization -> ./platform-config (CONFIGS: CRs; dependsOn platform)
|
|
# SECOND-ORDER CDI configs — CRs whose CRD is created by a controller AT RUNTIME, not by any
|
|
# manifest in git. cdi-operator.yaml registers only `cdis.cdi.kubevirt.io`; `storageprofiles` (and
|
|
# datavolumes, datasources, …) appear only after cdi-operator reconciles the `CDI` CR that
|
|
# platform-config applies. Because Flux dry-runs every object in a pass before applying any of them,
|
|
# keeping the StorageProfiles in platform-config was a TRUE DEADLOCK (unknown kind -> pass aborts ->
|
|
# CR never applied -> CRD never created -> tier never converges), invisible on dev and fatal on any
|
|
# cold cluster — observed on prod 2026-09-05. This tier is the durable fix; `standup.sh crds` stays
|
|
# as the belt-and-braces repair for clusters already wedged. See clusters/base/platform-config-cdi.yaml.
|
|
- platform-config-cdi.yaml # Flux Kustomization -> ./platform-config-cdi (dependsOn platform-config)
|
|
# App tier, split into dependency-ordered WAVES (was the single ./apps `apps` Kustomization).
|
|
# Each apps-wave-<N> dependsOn wave N-1 (wave 0 dependsOn platform-config) and health-gates
|
|
# (wait:true) ONLY its own wave, so a crashloop in a later wave can't stall an earlier one.
|
|
# GENERATED by `catalog/render.py --emit-waves`; add a line here only when a NEW deepest wave
|
|
# is introduced (a service deeper than any existing wave). See DESIGN-waves.md.
|
|
- apps-wave-0.yaml # Flux Kustomization -> ./apps/wave-0 (dependsOn platform-config)
|
|
- apps-wave-1.yaml # Flux Kustomization -> ./apps/wave-1 (dependsOn apps-wave-0)
|
|
- apps-wave-2.yaml # Flux Kustomization -> ./apps/wave-2 (dependsOn apps-wave-1)
|
|
# VM tier (KubeVirt) — non-gating (wait:false), separate lifecycle from the app waves.
|
|
- apps-vms.yaml # Flux Kustomization -> ./apps/vms (dependsOn platform-config)
|
|
- conformance.yaml # Flux Kustomization -> ./conformance/incluster (non-gating; R-SEC-02 trigger identity)
|
|
# oauth2-proxy forward-auth — POST-platform tier (dependsOn platform-config, wait:false). A leaf
|
|
# service that needs Keycloak + trusted TLS + split-horizon DNS must NOT sit in the gating
|
|
# `platform` tier, where it self-gated its own dependencies into a cold-start deadlock (twice).
|
|
- oauth2-proxy.yaml # Flux Kustomization -> ./platform/oauth2-proxy (dependsOn platform-config)
|
|
# headlamp dashboard — POST-platform tier (dependsOn platform-config, wait:false). A leaf UI whose
|
|
# HelmRelease can crashloop (upstream headlamp-0.40.1 `-session-ttl` chart/image desync) must NOT
|
|
# sit in the gating `platform` tier, where its unhealth stalled platform-config + oauth2-proxy +
|
|
# every apps wave/vm on "dependency not ready" (DECISIONS-LOG: "Headlamp gating stall").
|
|
- headlamp.yaml # Flux Kustomization -> ./platform/headlamp (dependsOn platform-config)
|
|
# Homepage apex landing dashboard — POST-platform tier (dependsOn platform-config, wait:false).
|
|
# A leaf UI at home.keithjackson.dev; same non-gating rationale as headlamp/oauth2-proxy (a UI must
|
|
# never gate the platform chain). Carries no SOPS secret, so clusters/dev adds no decryption patch.
|
|
- homepage.yaml # Flux Kustomization -> ./platform/homepage (dependsOn platform-config)
|
|
# Crafty Controller — Minecraft management UI, web UI Keycloak-gated by a per-pod oauth2-proxy
|
|
# reverse-proxy sidecar (Option B). POST-platform tier (dependsOn platform-config, wait:false); same
|
|
# non-gating leaf-UI rationale as headlamp/homepage/oauth2-proxy (a UI must never gate the platform).
|
|
- crafty.yaml # Flux Kustomization -> ./platform/crafty (dependsOn platform-config)
|
|
# Alert runbooks — POST-platform tier (dependsOn platform-config, wait:false). Serves the 34
|
|
# runbooks the alert rules' `runbook_url` annotations point at, at <apex>/runbooks/<name>; before
|
|
# this tier existed that path was a 404 and every alert handed the operator a dead link. Same
|
|
# non-gating leaf rationale as headlamp/homepage/crafty. Carries NO postBuild substitution on
|
|
# purpose — it renders documentation containing `${...}` shell examples that envsubst would empty;
|
|
# see clusters/base/runbooks.yaml.
|
|
- runbooks.yaml # Flux Kustomization -> ./runbooks (dependsOn platform-config)
|
|
# Velero backup controller — POST-platform tier (dependsOn platform-config, wait:false). A backup
|
|
# controller must NEVER gate the platform chain; same non-gating rationale as headlamp/oauth2-proxy.
|
|
# Carries velero-cloud.sops.yaml, so clusters/dev adds a SOPS decryption patch for it (by name).
|
|
- backup.yaml # Flux Kustomization -> ./platform/velero (dependsOn platform-config)
|
|
# WireGuard remote-access VPN (north-south INGRESS, udp/51820) — POST-platform tier
|
|
# (dependsOn platform-config, wait:false). NOT the same thing as platform/vpn-egress, which sends
|
|
# pods OUT through PIA; this lets remote CLIENTS IN. Non-gating is load-bearing rather than
|
|
# conventional here: its DaemonSet cannot start until the operator creates the out-of-band
|
|
# `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have
|
|
# stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch.
|
|
- vpn-ingress.yaml # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config)
|
|
# PM agent — reads Gitea issues and runs one worker agent per assignment that authors a
|
|
# change, gates it and opens a pull request. POST-APP tier (dependsOn apps-wave-0,
|
|
# wait:false): what it needs is GITEA, not Keycloak or the Gateway, and it must never gate
|
|
# anything — it CrashLoops by design while its credentials are still placeholders. Carries
|
|
# pm-agent-credentials.sops.yaml, so both overlays add a SOPS decryption patch for it (by
|
|
# name). See clusters/base/pm-agent.yaml and platform/pm-agent/README.md.
|
|
- pm-agent.yaml # Flux Kustomization -> ./platform/pm-agent (dependsOn apps-wave-0)
|
|
# kured (Kubernetes Reboot Daemon) — its OWN Kustomization (dependsOn platform, wait:true), NOT a
|
|
# line in platform/kustomization.yaml. Same reasoning as the Cilium and intel-gpu-plugin
|
|
# split-outs, with a sharper consequence: `platform` gates platform-config and every apps wave, so
|
|
# a wedged reboot DaemonSet placed in that wait:true tier could stall every application in the
|
|
# cluster — a component added to make maintenance SAFER must not be able to cause an outage.
|
|
# wait:true here is a HEALTH SIGNAL, not a gate: nothing dependsOn it, and nothing should.
|
|
# The `platform` edge is load-bearing rather than conventional — it supplies the
|
|
# monitoring.coreos.com CRDs this component's PrometheusRule/ServiceMonitor need (F-MON1), and
|
|
# Prometheus itself, which kured QUERIES before every reboot to gate on Longhorn volume health.
|
|
# It reboots a node only when /var/run/reboot-required exists, which on these nodes only a
|
|
# deliberate operator kernel upgrade creates — the human gate is the sentinel, not the daemon.
|
|
# See clusters/base/kured.yaml and platform/kured/helmrelease.yaml.
|
|
- kured.yaml # Flux Kustomization -> ./platform/kured (dependsOn platform)
|
|
- cluster-vars.yaml # overlay knob surface (base defaults)
|