Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
2f2e98aa19 |
No files matched your search
@@ -0,0 +1,106 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: feat/graceful-cluster-shutdown
|
||||
title: "Graceful node shutdown was never configured, and scaling a StatefulSet deleted its PVC"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
All three nodes had to be powered off to move them onto a UPS. There was **no procedure for it**:
|
||||
`infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev cluster, so
|
||||
the whole sequence was improvised live. Two defects came out of doing it.
|
||||
|
||||
**1. Kubelet graceful node shutdown was unconfigured on every prod node.** No `shutdownGracePeriod`
|
||||
anywhere in `infra/ansible/`, so `shutdown -h` stopped k3s, containerd went with it, and every pod
|
||||
was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25
|
||||
attached Longhorn volumes. `roles/k3s_server` now writes `kubelet-arg` into
|
||||
`/etc/rancher/k3s/config.yaml` with `shutdown-grace-period=90s` and
|
||||
`shutdown-grace-period-critical-pods=20s`.
|
||||
|
||||
The half that is easy to miss and silently defeats the other half: **graceful node shutdown is
|
||||
enforced by logind, not by the kubelet.** The kubelet holds an inhibitor *delay* lock and systemd
|
||||
waits at most `InhibitDelayMaxSec` — **default 5 seconds** — before proceeding regardless. A grace
|
||||
period above that cap is truncated with no error, so the feature looks configured and still
|
||||
hard-kills pods. The role now ships a `logind.conf.d` drop-in setting it to 120, and **asserts**
|
||||
`k3s_logind_inhibit_delay_max_sec >= k3s_kubelet_shutdown_grace_period` rather than documenting the
|
||||
relationship, because the failure is invisible.
|
||||
|
||||
**2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume.** `monitoring/loki` carries
|
||||
`persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}`, so the quiesce's
|
||||
`scale --replicas=0` destroyed `storage-loki-0` outright. Loki stores with `type: filesystem`, so
|
||||
that PVC was the only copy of its log data. This is the defect in this change set that actually
|
||||
destroyed something, and it was mine.
|
||||
|
||||
New `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`) encodes the sequence that
|
||||
worked, and **refuses to scale any StatefulSet whose `whenScaled` is `Delete`** — it leaves it
|
||||
running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect
|
||||
`degraded`. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed
|
||||
from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS)
|
||||
rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it.
|
||||
|
||||
`runbooks/cluster-maintenance-shutdown.md` (new, registered in `runbooks/kustomization.yaml`) is the
|
||||
operator page: shutdown order, the TrueNAS-after-nodes constraint, power-on order, the
|
||||
un-hibernate-Postgres step Flux will not do for you, and a "what looks broken but is fine" section
|
||||
separating the `4/19`-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the
|
||||
self-recovering `keycloak-operator` from the one thing that genuinely needs a human (a stuck
|
||||
`oauth2-proxy` replica).
|
||||
|
||||
## What I verified
|
||||
|
||||
* **Graceful shutdown was genuinely absent**, not merely undocumented: `grep -rn
|
||||
'shutdownGracePeriod|graceful-node-shutdown'` over `infra/ansible/` returns nothing, and no
|
||||
`logind` or `InhibitDelay` setting exists anywhere in the tree.
|
||||
* **The whole shutdown/restart cycle was executed end to end on prod 2026-10-17** and the cluster
|
||||
came back: 3/3 nodes Ready with etcd `healthz` ok, 19/19 Flux Kustomizations Ready,
|
||||
`platform-config`'s `lastAppliedRevision` equal to the GitRepository artifact revision, all 3
|
||||
CNPG clusters `healthy` with primaries elected, the UniFi VM `Running`, PiHole resolving
|
||||
(`home.keithjackson.dev` → `192.168.2.20` via `192.168.2.23`), and the Gateway serving
|
||||
`200/200/401` for apex/gitea/plex.
|
||||
* **The PVC-deletion mechanism is confirmed, not inferred:** `kubectl get sts loki -o
|
||||
jsonpath={.spec.persistentVolumeClaimRetentionPolicy}` returns
|
||||
`{"whenDeleted":"Delete","whenScaled":"Delete"}`, the PVC was `NotFound` after the cycle, and the
|
||||
Longhorn volume count had gone 25 → 24.
|
||||
* **The damage was repaired and verified:** `scale sts loki --replicas=1` (restoring the replica
|
||||
count the chart declares) recreated `storage-loki-0` as a fresh 50Gi Bound PVC, `loki-0` reached
|
||||
`2/2 Running`, and the volume count returned to 25.
|
||||
* **The guard works against the live cluster**, tested both ways: `sts_deletes_pvc_on_scale
|
||||
monitoring loki` returns true (would be skipped) and `pihole pihole` returns false (safe to
|
||||
scale).
|
||||
* **The script's read-only paths run clean on prod:** `status` reports correctly, and the discovery
|
||||
function returns the 13 workloads currently holding a PVC.
|
||||
* `bash -n` clean; the two edited Ansible YAML files parse.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **The kubelet change has NEVER BEEN APPLIED OR TESTED.** This is the significant gap. I did not
|
||||
run the role, did not restart k3s, and have **not** observed a node actually drain on
|
||||
`shutdown -h`. Everything about it is reasoning from kubelet/systemd semantics plus the measured
|
||||
absence of the setting. Applying it needs a **rolling** k3s restart (`serial: 1` — kube-vip keeps
|
||||
the API up, etcd holds at 2/3), which is an operator action (CLAUDE.md §5). **Verify by draining
|
||||
one node for real:** apply, then `shutdown -h now` on a single node and watch whether pods
|
||||
terminate gracefully instead of vanishing.
|
||||
* **Whether 90s/20s are the right numbers.** Chosen to comfortably cover a Postgres fast shutdown
|
||||
and a Longhorn flush; not measured against either. The budget is spent on every reboot, so if a
|
||||
node takes noticeably longer to go down, this is why.
|
||||
* **Whether k3s accepts `kubelet-arg` in `config.yaml` as written.** The file already carries
|
||||
`kube-apiserver-arg` as a list and `kubelet-arg` is the documented sibling, but I did not render
|
||||
or apply it, so a typo would surface only at k3s restart — and a bad `config.yaml` is exactly the
|
||||
kind of thing that makes k3s fail to come up. **Review this one before applying.**
|
||||
* **The `assert` task's `regex_replace` on the grace period** assumes a plain `90s`-style value; a
|
||||
value expressed as `1m30s` would parse to the nonsense integer `130`. Fine for the committed
|
||||
defaults, wrong if someone writes a compound duration.
|
||||
* **`resume`'s Flux-reconcile wait has never run**, and neither has `quiesce` as a script — the
|
||||
2026-10-17 cycle was done with the equivalent commands by hand, so the script encodes a sequence
|
||||
that worked but has not itself been executed end to end. Only `status` and the discovery and guard
|
||||
functions have run.
|
||||
* **Loki's lost log window is gone for good.** Up to 72h of logs (its `retention_period`), destroyed
|
||||
by my scale-down. Nothing recovers it; Velero's schedules cover the namespace but the PVC was
|
||||
deleted, not corrupted, and I did not attempt a restore because the data self-expires anyway.
|
||||
* **No other `whenScaled: Delete` StatefulSet was audited.** The guard now catches them generically,
|
||||
but I did not enumerate which other workloads carry the policy, so I cannot say loki was the only
|
||||
one at risk. The runbook gives the one-liner to list them.
|
||||
* **TrueNAS and the router are entirely outside this change.** TrueNAS is not in the Ansible
|
||||
inventory and refuses SSH, so its shutdown stays a web-UI step with no automation and no check.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1
|
||||
offline subset only.
|
||||
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
|
||||
@@ -1,99 +0,0 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: fix/oauth2-proxy-oidc-startup-race
|
||||
title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
|
||||
discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
|
||||
serving yet, so the shared Gateway answers 503 and the proxy dies:
|
||||
|
||||
```
|
||||
[provider.go:55] Performing OIDC Discovery...
|
||||
[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
|
||||
failed to discover OIDC configuration: unexpected status "503": no healthy upstream
|
||||
```
|
||||
|
||||
It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
|
||||
replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
|
||||
retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
|
||||
is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
|
||||
|
||||
`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
|
||||
polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
|
||||
accrue CrashLoopBackOff backoff and retries promptly, which is the point.
|
||||
|
||||
**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
|
||||
case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
|
||||
for a pod wedged in `Init` forever, which is not an improvement.
|
||||
|
||||
## Why this shape and not the alternatives
|
||||
|
||||
* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
|
||||
example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
|
||||
pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
|
||||
at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
|
||||
"reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
|
||||
* **No new base image.** The chart already runs an `alpine` init container for its built-in
|
||||
`initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
|
||||
`3.22` (verified to exist upstream) rather than the chart's own `latest`.
|
||||
* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
|
||||
this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
|
||||
the chart already has a hook for.
|
||||
* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
|
||||
more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
|
||||
oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
|
||||
the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
|
||||
the operator out of all of them, so the reversible, additive option wins.
|
||||
|
||||
The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
|
||||
with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
|
||||
and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
|
||||
silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
|
||||
the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **The failure is measured, not theorised.** The log above is from the live pod
|
||||
(`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
|
||||
with `--previous` after the cold boot. The surviving replica's log shows the same startup
|
||||
succeeding once Keycloak was up, which is what identifies this as a race rather than a config
|
||||
fault.
|
||||
* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
|
||||
clean and both replicas have been healthy since.
|
||||
* **The chart key is real**, at the pinned version range: `helm search repo` resolves
|
||||
`>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
|
||||
`extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
|
||||
* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
|
||||
assumed from memory.
|
||||
* The edited HelmRelease parses as YAML.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
|
||||
seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
|
||||
replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
|
||||
seven apps, so **do it when you can watch it**, not unattended.
|
||||
* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
|
||||
oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
|
||||
from the measured failure, not demonstrated against it.
|
||||
* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
|
||||
`ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
|
||||
failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
|
||||
degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
|
||||
anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
|
||||
init container's log on first apply**: it should print "Keycloak discovery reachable" within
|
||||
seconds on a warm cluster, not the give-up message.
|
||||
* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
|
||||
re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
|
||||
good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
|
||||
(`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
|
||||
Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
|
||||
this should hold — unverified.
|
||||
* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
|
||||
not a measured bound.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
|
||||
subset only.
|
||||
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
|
||||
@@ -1,137 +0,0 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: fix/vpn-ingress-l2-lease-collision
|
||||
title: "A node cannot reach a VIP it announces — so the VPN ingress node must announce nothing"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The first device enrolled on the remote-access VPN handshook, resolved names, and could not open a
|
||||
single connection to any cluster web app. The tunnel was not at fault. The fault is a property of
|
||||
Cilium L2 announcement that nothing in this repo knew about:
|
||||
|
||||
**A node cannot reach a LoadBalancer VIP whose `CiliumL2AnnouncementPolicy` lease it holds itself.**
|
||||
The announcer answers ARP for the VIP on behalf of other LAN hosts but not for its own request, so
|
||||
on the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP is dropped
|
||||
before it leaves the node.
|
||||
|
||||
`platform/vpn-ingress` is uniquely exposed because it runs WireGuard with `hostNetwork: true` and
|
||||
MASQUERADEs remote clients to the node's own address — a VPN client's packets take the node's LAN
|
||||
path and inherit its ARP failures. Ordinary pods never do. The router's `udp/51820` forward pointed
|
||||
at `k3s-prod-1`, which held `cilium-l2announce-gateway-cilium-gateway-shared`, so **all 16 `shared`
|
||||
Gateway hostnames were dead over the VPN** — while PiHole DNS kept answering, because
|
||||
`192.168.2.23` is announced by *two* Services and `k3s-prod-2` held the other lease. Names resolved
|
||||
and every app hung on connect.
|
||||
|
||||
Seven files:
|
||||
|
||||
* **`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`** — new
|
||||
`node_baseline_vpn_ingress_node`; the matching node gets
|
||||
`homelab.keithjackson.dev/vpn-ingress=true`, and a second task **removes** the label from any node
|
||||
that no longer matches, so moving the value moves the label rather than adding a second one. Same
|
||||
shape as `roles/vpn_egress_gateway`'s `egress-gateway` label. Tagged `vpn-ingress`; skipped
|
||||
entirely when the value is empty.
|
||||
* **`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`** — set to `k3s-prod-1`, with a
|
||||
`[parity]` note that it must equal the router's forward.
|
||||
* **`platform-config/cilium/l2-announcement-policy.yaml`** — `nodeSelector` becomes a
|
||||
`matchExpressions` list keeping `kubernetes.io/os: linux` and adding
|
||||
`homelab.keithjackson.dev/vpn-ingress NotIn ["true"]`. The VPN node then holds no lease, so it
|
||||
ARP-resolves every VIP. This fixes **all** VIPs for VPN clients (Gateway, crafty, plex, pihole,
|
||||
unifi), not just the one that happened to collide.
|
||||
* **`platform/monitoring/kube-prometheus-stack.yaml`** — the label added to kube-state-metrics'
|
||||
`metricLabelsAllowlist`, which is what lets the invariant be watched at all.
|
||||
* **`platform/monitoring/alerts.yaml`** — `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) joins
|
||||
`kube_lease_owner` against `kube_node_labels` via `label_replace`, plus
|
||||
`VpnIngressNodeLabelMissing` (warning, 30m) as its watchdog.
|
||||
* **`platform/monitoring/tests/vpn-ingress-l2.test.yaml`** — four promtool cases: fires on the real
|
||||
collision, quiet in the fixed state, quiet across a transient 11-minute lease move (pinning the
|
||||
`for: 15m`), and the label-absent case asserting the critical alert is inert *and* the watchdog
|
||||
fires.
|
||||
* **`runbooks/vpn-ingress-l2-collision.md`** (new, registered in `runbooks/kustomization.yaml`),
|
||||
**`runbooks/vpn-remote-access.md`**, **`platform/vpn-ingress/{daemonset.yaml,README.md}`** — step 3
|
||||
of the setup runbook said *"pick any node — all three run an identical listener"* and *"node loss =
|
||||
failover is this one line"*. Both were false and the first caused this. Failover is now documented
|
||||
as two ordered changes.
|
||||
|
||||
**Design note — why a node label and not a `cluster-vars` knob.** The first attempt added
|
||||
`WG_INGRESS_NODE` to `clusters/base/cluster-vars.yaml` and substituted it into both the policy and
|
||||
the alert. The alert half is **impossible**: `platform/monitoring/tests/extract_rules.py` rejects any
|
||||
alert whose `expr` contains a `${...}` placeholder, because the rule it would unit-test is not the
|
||||
rule deployed. Rather than duplicate the node name into the alert as a literal (two sources that can
|
||||
skew), the label is readable from both sides — the selector matches on it and `kube_node_labels`
|
||||
carries it into Prometheus. One source of truth, and as a side effect the change touches no
|
||||
sensitive path, so it merges on a green gate rather than needing a human review.
|
||||
|
||||
The DaemonSet deliberately gained **no** probe for this. A readinessProbe on VIP reachability would
|
||||
wedge a rollout behind an unrelated Gateway outage (`maxUnavailable: 1`); a liveness probe would tear
|
||||
down a tunnel carrying LAN traffic correctly.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured live against prod, `--context k3s-prod`, on 2026-10-08:
|
||||
|
||||
* **The whole tunnel is healthy**, which is what made this hard: `wg show` on `k3s-prod-1` shows the
|
||||
peer with a real endpoint and bidirectional transfer (346 KiB in / 200 KiB out);
|
||||
`net.ipv4.ip_forward=1`; `wg0` up at `10.90.0.1/24` MTU 1420; `iptables -t nat -S POSTROUTING`
|
||||
carries the `MASQUERADE` rule with 26 packets; `FORWARD -i/-o wg0` ACCEPT with 108 / 171 packets;
|
||||
DaemonSet generation 2, 3/3 ready on the content-hashed `wg-peers-m9kcckcb5t`.
|
||||
* **The ARP failure and its symmetry.** On `k3s-prod-1`: `ip neigh show 192.168.2.20` → `FAILED`,
|
||||
while `.21`/`.23` resolve to prod-2's MAC `04:0e:3c:96:b0:94`. On prod-2/prod-3, `.20` resolves to
|
||||
prod-1's `bc:e9:2f:7e:c4:a6`. TCP agrees: `.20:443` and `.20:80` FAIL from prod-1, OK from the
|
||||
other two. Repeated across 3 nodes × 5 VIPs — each node reached every VIP except the ones it alone
|
||||
announced (prod-2 ✗ `.22`/`.21`; prod-3 ✗ `.24`).
|
||||
* **Not an artefact of probing from the host netns.** `curl --interface wg0` from prod-1 fails the
|
||||
same way, and a neighbour entry is node-global state, so the forwarded path cannot differ in ARP
|
||||
resolution.
|
||||
* **The Gateway itself was never broken:** from the workstation (192.168.2.133)
|
||||
`home.keithjackson.dev` → `200`, `plex.home.keithjackson.dev` → `401`, both at `192.168.2.20`.
|
||||
From prod-1 the router (`.1`) and TrueNAS (`.30`) are `REACHABLE`.
|
||||
* **`NotIn` semantics, empirically** — the basis for the fail-back claim in the policy's comment:
|
||||
`kubectl get nodes -l 'kubernetes.io/hostname notin (k3s-prod-1)'` returns prod-2 and prod-3, and
|
||||
`NotIn` on a key no node carries returns **all three**. So an unapplied label excludes nothing and
|
||||
degrades to the old behaviour rather than taking every VIP off the LAN.
|
||||
* **The alert metric is really scraped:** `kube_lease_owner` is served by kube-state-metrics (55
|
||||
series, every `cilium-l2announce-*` lease with its holder in `lease_holder`), and `kube_node_labels`
|
||||
already carries the two `egress-*` labels through the same allowlist (`k3s-prod-3` shows
|
||||
`egress-gateway=true`), so the third entry follows a proven path.
|
||||
* **The four promtool cases pass** (`platform/monitoring/tests/run.sh` → `SUCCESS`, 55 rules
|
||||
checked), including that `VpnIngressNodeAnnouncesL2VIP` is inert when the label is absent.
|
||||
* All edited YAML parses; the rendered `nodeSelector` is the intended two-expression list.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **The fix is UNAPPLIED and therefore UNPROVEN end to end.** Applying it needs a mutating
|
||||
`kubectl`/`flux reconcile` plus an Ansible run, which are operator actions (CLAUDE.md §5). I have
|
||||
**not** observed `k3s-prod-1` resolve `192.168.2.20` after the exclusion lands, nor a VPN client
|
||||
reaching an app. The prediction rests on the measured symmetry (non-announcing nodes resolve every
|
||||
VIP, 15/15 data points), not on a test of the changed state. **Verify after merge, in this order:**
|
||||
re-run `node_baseline --tags vpn-ingress`; `kubectl get nodes -L
|
||||
homelab.keithjackson.dev/vpn-ingress` (exactly one node); `kubectl -n kube-system get lease | grep
|
||||
l2announce` (no holder is that node); `ip neigh show 192.168.2.20` from the prod-1 wireguard pod
|
||||
(expect a `lladdr`, not `FAILED`); then a client.
|
||||
* **That Cilium actually releases a lease when its holder stops matching the `nodeSelector`.** I
|
||||
assumed re-election and did not test it. If Cilium keeps renewing, the affected Leases must be
|
||||
deleted once by hand to force an election. Check this first if the fix appears not to work.
|
||||
* **The brief interruption on apply.** Moving the Gateway and `pihole-dns-udp` leases off prod-1 will
|
||||
blip `192.168.2.20` and `192.168.2.23` for as long as re-election plus gratuitous ARP takes. Not
|
||||
measured, and I did not test whether in-flight LAN sessions survive it.
|
||||
* **The Ansible tasks have never been executed.** They parse as YAML and follow
|
||||
`roles/vpn_egress_gateway`'s shape, but neither the label task nor the label-removal task has run,
|
||||
so the `changed_when`/`failed_when` conditions and the `k3s kubectl label ... -` removal path are
|
||||
unproven. The removal task's `failed_when` tolerates `'not found'` in stderr; that string was
|
||||
assumed from kubectl's behaviour, not observed.
|
||||
* **The residual off-repo coupling is NOT closed and cannot be, as built.** The router's forward is
|
||||
unreadable from the cluster. If someone re-points it without moving the label, the alert watches
|
||||
the wrong node and stays silent while the VPN breaks identically against a different set of VIPs.
|
||||
Only the runbook guards it. Closing it needs a WireGuard exporter publishing which node is
|
||||
actually terminating peer traffic (`wg show` knows — only the forwarded node has a peer
|
||||
`endpoint`); listed as "Not built" in the component README and **recommended follow-up, not done
|
||||
here**.
|
||||
* **Why Cilium behaves this way** — intended, a known upstream issue, or fixed in a later version. I
|
||||
measured the behaviour and did not read Cilium's source or issue tracker, so I cannot say whether
|
||||
an upgrade would remove the need for this exclusion.
|
||||
* **`192.168.2.21:8443` (unifi) fails from all three nodes**, including ones where ARP resolves, so
|
||||
it is a different, pre-existing fault. Not investigated, not addressed.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 offline
|
||||
subset only; none of the above is covered by a test that runs in the gate.
|
||||
* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).
|
||||
@@ -420,22 +420,3 @@ flux_bootstrap_env_path: "clusters/prod"
|
||||
# rather than leaving an inert drop-in behind, so this group-wide `true` is still correct
|
||||
# for it.
|
||||
etcd_io_priority_enabled: true
|
||||
|
||||
# --- The node terminating remote-access WireGuard — DECIDED (measured 2026-10-08) ---------------
|
||||
# k3s-prod-1 (192.168.2.10) is what the SOHO router forwards WAN udp/51820 to, confirmed from the
|
||||
# cluster rather than assumed: only prod-1's `wg show` had a peer with an `endpoint` and a recent
|
||||
# handshake, the other two listeners were idle.
|
||||
#
|
||||
# node_baseline labels it `homelab.keithjackson.dev/vpn-ingress=true`, and
|
||||
# platform-config/cilium/l2-announcement-policy.yaml excludes that label from announcing any
|
||||
# LoadBalancer VIP -- because a node CANNOT reach a VIP whose Cilium L2 announcement lease it holds
|
||||
# itself (the announcer does not answer its own ARP request; the neighbour entry goes to `FAILED`).
|
||||
# prod-1 held the `shared` Gateway lease, and since vpn-ingress MASQUERADEs clients to the node's
|
||||
# own address, all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved.
|
||||
#
|
||||
# [parity] MUST equal the node named by the router's forward (runbooks/vpn-remote-access.md step 3).
|
||||
# Nothing in CI can read that forward, so re-pointing it means changing BOTH: this value plus a
|
||||
# node_baseline re-run, and the router line. `VpnIngressNodeAnnouncesL2VIP` catches the case where
|
||||
# the label lands on a node that then takes a lease; it CANNOT catch the forward being moved
|
||||
# without this value following, which is why the runbook spells the two-step out.
|
||||
node_baseline_vpn_ingress_node: k3s-prod-1
|
||||
@@ -107,3 +107,31 @@ k3s_base_domain: "home.keithjackson.dev"
|
||||
|
||||
# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
|
||||
k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
|
||||
|
||||
# --- Graceful node shutdown (kubelet) ----------------------------------------------------------
|
||||
# MEASURED 2026-10-17, during the UPS move: graceful node shutdown was NOT configured on any prod
|
||||
# node, so `shutdown -h` did NOT drain pods. systemd stops k3s, containerd goes with it, and every
|
||||
# pod is SIGKILLed where it stands -- including 9 CloudNativePG instances mid-checkpoint, the UniFi
|
||||
# VM, and 25 attached Longhorn volumes. The whole-cluster shutdown that day had to quiesce
|
||||
# everything BY HAND first (runbooks/cluster-maintenance-shutdown.md), and the one volume that did
|
||||
# not finish detaching came back `degraded` and had to rebuild.
|
||||
#
|
||||
# With these set, the kubelet takes a systemd inhibitor lock and, on shutdown, evicts ordinary pods
|
||||
# first and critical ones last, so a plain `shutdown -h`/reboot drains the node. That makes the
|
||||
# manual scale-down in the runbook a belt-and-braces step for a PLANNED whole-cluster shutdown
|
||||
# rather than the only thing standing between a reboot and a hard kill.
|
||||
#
|
||||
# 90s/20s, not larger: the budget is spent on every reboot and node upgrade, and the kubelet waits
|
||||
# the FULL ordinary period before it starts on critical pods. Postgres does a fast shutdown in
|
||||
# well under 90s here, and Longhorn needs only enough time to flush and detach.
|
||||
k3s_kubelet_shutdown_grace_period: "90s"
|
||||
k3s_kubelet_shutdown_grace_period_critical_pods: "20s"
|
||||
|
||||
# *** THE HALF THAT IS EASY TO MISS, AND SILENTLY BREAKS THE OTHER HALF. ***
|
||||
# Graceful node shutdown works by the kubelet holding a logind INHIBITOR DELAY lock. systemd caps
|
||||
# how long any inhibitor may delay shutdown at `InhibitDelayMaxSec`, which DEFAULTS TO 5 SECONDS.
|
||||
# If that cap is below the grace period above, systemd proceeds anyway and the drain is truncated
|
||||
# to ~5s -- i.e. the feature appears configured, reports no error, and still hard-kills pods. So
|
||||
# this MUST be >= k3s_kubelet_shutdown_grace_period; the role asserts it rather than trusting it.
|
||||
# Set a little above the kubelet's total (90s + 20s) to leave headroom.
|
||||
k3s_logind_inhibit_delay_max_sec: 120
|
||||
@@ -15,3 +15,12 @@
|
||||
port: "{{ k3s_api_port }}"
|
||||
timeout: 180
|
||||
listen: Restart k3s
|
||||
|
||||
# logind re-reads InhibitDelayMaxSec only on restart. Restarting logind does NOT log users out or
|
||||
# disturb k3s -- it manages login sessions and inhibitor locks, not workloads. The kubelet picks up
|
||||
# the new cap when it next takes its lock, so no k3s restart is needed for THIS half.
|
||||
- name: Restart logind
|
||||
ansible.builtin.systemd_service:
|
||||
name: systemd-logind
|
||||
state: restarted
|
||||
listen: Restart logind
|
||||
@@ -86,6 +86,16 @@
|
||||
# k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
|
||||
# BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
|
||||
# config.yaml is unchanged and the Restart handler does not fire.
|
||||
# ── Graceful node shutdown ──────────────────────────────────────────────────────────────
|
||||
# Without these the kubelet takes no systemd inhibitor lock, so `shutdown -h` stops
|
||||
# containerd and SIGKILLs every pod where it stands -- MEASURED on all three prod nodes
|
||||
# 2026-10-17 (defaults/main.yml has the detail). The companion logind drop-in below raises
|
||||
# InhibitDelayMaxSec, WITHOUT WHICH systemd caps the drain at ~5s and these values are
|
||||
# quietly ignored.
|
||||
kubelet-arg:
|
||||
- "shutdown-grace-period={{ k3s_kubelet_shutdown_grace_period }}"
|
||||
- "shutdown-grace-period-critical-pods={{ k3s_kubelet_shutdown_grace_period_critical_pods }}"
|
||||
|
||||
kube-apiserver-arg:
|
||||
- "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
|
||||
- "oidc-client-id=kubernetes"
|
||||
@@ -183,3 +193,46 @@
|
||||
host: "127.0.0.1"
|
||||
port: "{{ k3s_api_port }}"
|
||||
timeout: 180
|
||||
|
||||
# ── Graceful node shutdown: the systemd half — tag `shutdown` ──────────────────────────────────
|
||||
# The kubelet's shutdown-grace-period is ENFORCED BY logind, not by the kubelet: the kubelet holds
|
||||
# an inhibitor DELAY lock, and systemd will wait at most `InhibitDelayMaxSec` for it (default 5s)
|
||||
# before proceeding regardless. A grace period above that cap is therefore silently truncated --
|
||||
# the feature looks configured, logs no error, and still hard-kills pods. Assert the relationship
|
||||
# instead of documenting it, because the failure is invisible.
|
||||
- name: Assert the logind inhibitor cap can actually cover the kubelet grace period
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- (k3s_logind_inhibit_delay_max_sec | int) >= (k3s_kubelet_shutdown_grace_period | regex_replace('[^0-9]', '') | int)
|
||||
fail_msg: >-
|
||||
k3s_logind_inhibit_delay_max_sec ({{ k3s_logind_inhibit_delay_max_sec }}s) is below
|
||||
k3s_kubelet_shutdown_grace_period ({{ k3s_kubelet_shutdown_grace_period }}). systemd would
|
||||
stop waiting for the kubelet's inhibitor lock before the drain finished, so pods would be
|
||||
SIGKILLed anyway and graceful node shutdown would be configured but ineffective.
|
||||
quiet: true
|
||||
tags: [shutdown]
|
||||
|
||||
- name: Ensure /etc/systemd/logind.conf.d exists
|
||||
ansible.builtin.file:
|
||||
path: /etc/systemd/logind.conf.d
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
tags: [shutdown]
|
||||
|
||||
# A drop-in, not an edit of logind.conf: the distro file stays pristine and `state: absent` here
|
||||
# would fully revert the change.
|
||||
- name: Raise the logind inhibitor delay cap so the kubelet can finish draining
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/systemd/logind.conf.d/10-k3s-graceful-shutdown.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# Managed by Ansible (roles/k3s_server). See defaults/main.yml.
|
||||
# MUST stay >= kubelet shutdown-grace-period, or the drain is truncated to this value.
|
||||
[Login]
|
||||
InhibitDelayMaxSec={{ k3s_logind_inhibit_delay_max_sec }}
|
||||
notify: Restart logind
|
||||
tags: [shutdown]
|
||||
@@ -259,30 +259,3 @@ node_baseline_dns_options:
|
||||
# Written as ONE marked block, so removing an entry here removes it from the node, and
|
||||
# an empty list removes the block entirely.
|
||||
node_baseline_hosts_pins: []
|
||||
|
||||
# --- The node that terminates remote-access WireGuard (tags: vpn-ingress) ----------------------
|
||||
# Hostname of the node the SOHO router forwards WAN udp/51820 to, i.e. the one platform/vpn-ingress
|
||||
# actually serves remote clients from. Set it in group_vars for the environment; EMPTY by default,
|
||||
# because dev has no such forward and because a wrong value is worse than none.
|
||||
#
|
||||
# WHAT IT IS FOR, AND WHY IT IS NOT COSMETIC. The matching node gets
|
||||
# `homelab.keithjackson.dev/vpn-ingress=true`, and
|
||||
# platform-config/cilium/l2-announcement-policy.yaml EXCLUDES that label from announcing any
|
||||
# LoadBalancer VIP. MEASURED on prod 2026-10-08: a node cannot reach a VIP whose Cilium L2
|
||||
# announcement lease it holds itself (the announcer does not answer its own ARP request -- the
|
||||
# neighbour entry goes to `FAILED`). Because vpn-ingress runs on `hostNetwork` and MASQUERADEs
|
||||
# clients to the node's own address, a VPN client inherits that: with the forward on the node
|
||||
# holding the `shared` Gateway lease, all 16 Gateway hostnames were dead over the VPN while PiHole
|
||||
# DNS still resolved. Names worked, every app hung, nothing alerted.
|
||||
#
|
||||
# [parity] MUST name the node the router's forward actually points at. The forward is off-repo and
|
||||
# no check can read it (runbooks/vpn-remote-access.md step 3). Changing the forward is therefore a
|
||||
# two-part change: this value (then re-run the role), and the router line.
|
||||
#
|
||||
# Only ONE node should ever carry the label: it removes that node from VIP announcement candidacy,
|
||||
# so labelling all three would leave nothing announcing any VIP at all. The task below removes the
|
||||
# label from a node that no longer matches, so moving the value moves the label.
|
||||
node_baseline_vpn_ingress_node: ""
|
||||
node_baseline_vpn_ingress_label_key: homelab.keithjackson.dev/vpn-ingress
|
||||
# k3s ships its kubeconfig here; every prod node is a k3s server, so `k3s kubectl` works locally.
|
||||
node_baseline_vpn_ingress_kubeconfig: /etc/rancher/k3s/k3s.yaml
|
||||
@@ -1109,45 +1109,3 @@
|
||||
state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
|
||||
backup: true
|
||||
tags: [dns, hosts]
|
||||
|
||||
# ── vpn-ingress node label — tag `vpn-ingress` ────────────────────────────────────────────────
|
||||
# Marks the node terminating remote-access WireGuard so Cilium's L2 announcement policy can EXCLUDE
|
||||
# it from announcing LoadBalancer VIPs (a node cannot reach a VIP it announces itself -- see
|
||||
# defaults/main.yml). Same shape as roles/vpn_egress_gateway's egress-gateway label, and for the
|
||||
# same reason: the label is the one place BOTH a nodeSelector and a PromQL alert can read it from.
|
||||
#
|
||||
# Skipped entirely when node_baseline_vpn_ingress_node is empty (dev, or before the forward exists),
|
||||
# so this adds nothing to an environment that has no WAN forward.
|
||||
- name: Label this node as the vpn-ingress node
|
||||
ansible.builtin.command:
|
||||
cmd: >-
|
||||
k3s kubectl label node {{ inventory_hostname }}
|
||||
{{ node_baseline_vpn_ingress_label_key }}=true --overwrite
|
||||
environment:
|
||||
KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}"
|
||||
register: node_baseline_vpn_ingress_label_apply
|
||||
changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_apply.stdout"
|
||||
when:
|
||||
- node_baseline_vpn_ingress_node | length > 0
|
||||
- inventory_hostname == node_baseline_vpn_ingress_node
|
||||
tags: [vpn-ingress]
|
||||
|
||||
# The other half: moving the knob must MOVE the label, not add a second one. Two labelled nodes
|
||||
# would remove two of three nodes from announcement candidacy; three would leave no announcer at
|
||||
# all and take every VIP off the LAN.
|
||||
- name: Remove the vpn-ingress label from a node that no longer holds the forward
|
||||
ansible.builtin.command:
|
||||
cmd: >-
|
||||
k3s kubectl label node {{ inventory_hostname }}
|
||||
{{ node_baseline_vpn_ingress_label_key }}-
|
||||
environment:
|
||||
KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}"
|
||||
register: node_baseline_vpn_ingress_label_remove
|
||||
changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_remove.stdout"
|
||||
failed_when:
|
||||
- node_baseline_vpn_ingress_label_remove.rc != 0
|
||||
- "'not found' not in node_baseline_vpn_ingress_label_remove.stderr"
|
||||
when:
|
||||
- node_baseline_vpn_ingress_node | length > 0
|
||||
- inventory_hostname != node_baseline_vpn_ingress_node
|
||||
tags: [vpn-ingress]
|
||||
Executable
+265
@@ -0,0 +1,265 @@
|
||||
#!/usr/bin/env bash
|
||||
# =================================================================================================
|
||||
# quiesce.sh — bring the prod cluster to a safe standstill before powering nodes off, and bring it
|
||||
# back afterwards. Companion to runbooks/cluster-maintenance-shutdown.md.
|
||||
# =================================================================================================
|
||||
# WHY THIS EXISTS. On 2026-10-17 all three nodes had to be powered off to move them onto a UPS, and
|
||||
# there was NO procedure for it: `infra/cluster.sh stop` is `virsh`-based and only ever applied to
|
||||
# the decommissioned dev cluster. The whole sequence was improvised live. This script is that
|
||||
# sequence, so the next person does not improvise it.
|
||||
#
|
||||
# WHY IT IS NOT JUST `shutdown -h`. Graceful node shutdown was unconfigured on every prod node that
|
||||
# day, so systemd stopped containerd and SIGKILLed every pod where it stood. roles/k3s_server now
|
||||
# sets shutdown-grace-period (+ the logind InhibitDelayMaxSec cap that actually enforces it), which
|
||||
# makes a bare reboot safe. This script is still the right tool for a PLANNED whole-cluster
|
||||
# shutdown, for three reasons the kubelet's drain does not cover:
|
||||
# 1. The UniFi VM needs an ACPI guest shutdown, not an eviction.
|
||||
# 2. CloudNativePG wants hibernation, not pod deletion -- and hibernation is IMPERATIVE state
|
||||
# that Flux will not set or clear for you.
|
||||
# 3. Draining three nodes one at a time makes pods RESCHEDULE onto the survivors, churning
|
||||
# Longhorn replicas for no reason when the whole cluster is going down anyway. Scaling in
|
||||
# place avoids all of it.
|
||||
#
|
||||
# SAFE TO RE-RUN. Every step is idempotent and `resume` is the exact inverse of `quiesce`.
|
||||
#
|
||||
# WHAT IT DELIBERATELY DOES NOT DO: power anything off. Node poweroff is an operator action
|
||||
# (CLAUDE.md §5) and needs the become password; the runbook gives the ansible one-liner. It also
|
||||
# does not touch TrueNAS (not in the inventory, SSH refused -- web UI only) or the router.
|
||||
set -euo pipefail
|
||||
|
||||
CONTEXT="${CONTEXT:-k3s-prod}"
|
||||
KUBECTL=(kubectl --context "$CONTEXT" --request-timeout=30s)
|
||||
TIMEOUT="${TIMEOUT:-600}"
|
||||
|
||||
# The three CloudNativePG clusters, as namespace/name. Hibernation is per-Cluster.
|
||||
CNPG=(keycloak/keycloak-db nextcloud-db/nextcloud-db semaphore-db/semaphore-db)
|
||||
|
||||
log() { printf '\033[34m[quiesce]\033[0m %s\n' "$*"; }
|
||||
ok() { printf '\033[32m[ ok ]\033[0m %s\n' "$*"; }
|
||||
warn() { printf '\033[33m[warn]\033[0m %s\n' "$*"; }
|
||||
die() { printf '\033[31m[fail]\033[0m %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
need() { command -v "$1" >/dev/null || die "$1 not on PATH — run this inside ./infra/devcontainer/dev.sh"; }
|
||||
|
||||
# ---------------------------------------------------------------- discovery
|
||||
# Every Deployment/StatefulSet whose pods mount a PVC — Longhorn OR the TrueNAS NFS bulk tier.
|
||||
# Computed from the LIVE cluster rather than hardcoded, so a newly onboarded service is covered
|
||||
# without anyone remembering to add it here. PROTECTED namespaces are excluded because scaling
|
||||
# them down would remove the very machinery that performs the detach (longhorn-system), keep the
|
||||
# API reachable (kube-system), or finish the hibernation and the VM shutdown (cnpg-system,
|
||||
# kubevirt).
|
||||
pvc_workloads() {
|
||||
"${KUBECTL[@]}" get pods -A -o json | python3 -c '
|
||||
import json,sys
|
||||
PROTECT={"longhorn-system","kube-system","cilium-secrets","cnpg-system","kubevirt","flux-system","cdi"}
|
||||
out={}
|
||||
for p in json.load(sys.stdin)["items"]:
|
||||
ns=p["metadata"]["namespace"]
|
||||
if ns in PROTECT: continue
|
||||
if not any(v.get("persistentVolumeClaim") for v in (p["spec"].get("volumes") or [])): continue
|
||||
ow=(p["metadata"].get("ownerReferences") or [])
|
||||
if not ow: continue
|
||||
k,n=ow[0]["kind"],ow[0]["name"]
|
||||
if k=="ReplicaSet": k,n="deployment",n.rsplit("-",1)[0]
|
||||
elif k=="StatefulSet": k="statefulset"
|
||||
else: continue # Jobs, and CNPG Clusters, are handled separately
|
||||
out[(ns,k,n)]=1
|
||||
for ns,k,n in sorted(out): print(ns,k,n)
|
||||
'
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- the PVC-deletion guard
|
||||
# *** DO NOT REMOVE THIS CHECK. IT EXISTS BECAUSE SCALING A StatefulSet DESTROYED DATA. ***
|
||||
# A StatefulSet may carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and then
|
||||
# scaling it to 0 DELETES ITS PVCs -- and with them the Longhorn volumes. MEASURED 2026-10-17: the
|
||||
# quiesce for the UPS move scaled `monitoring/loki` (whenScaled: Delete) to 0 and its
|
||||
# `storage-loki-0` PVC and volume were deleted outright. Loki stores with `type: filesystem`, so
|
||||
# that PVC was the only copy; 72h retention bounded the loss to a rolling window, but nothing about
|
||||
# the mechanism is bounded -- the same action on a Delete-policy StatefulSet holding real state
|
||||
# would be unrecoverable. The volume count going 25 -> 24 was the only visible symptom, and it was
|
||||
# first misread as a miscount.
|
||||
#
|
||||
# So: such a StatefulSet is NEVER scaled. It is left RUNNING and its volume hard-detaches at
|
||||
# poweroff, which costs a replica rebuild on restart (degraded -> healthy, automatic). A rebuild is
|
||||
# an enormously better outcome than a deletion, and that is the whole trade being made here.
|
||||
sts_deletes_pvc_on_scale() { # ns name -> 0 if scaling it would delete its PVCs
|
||||
local policy
|
||||
policy=$("${KUBECTL[@]}" -n "$1" get statefulset "$2" \
|
||||
-o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy.whenScaled}' 2>/dev/null || true)
|
||||
[ "$policy" = "Delete" ]
|
||||
}
|
||||
|
||||
wait_for() { # wait_for <description> <shell-condition>
|
||||
local desc="$1" cond="$2" waited=0
|
||||
log "waiting: $desc (up to ${TIMEOUT}s)"
|
||||
until eval "$cond"; do
|
||||
(( waited += 10 )); (( waited >= TIMEOUT )) && { warn "TIMED OUT waiting: $desc"; return 1; }
|
||||
sleep 10
|
||||
done
|
||||
ok "$desc"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- quiesce
|
||||
do_quiesce() {
|
||||
need kubectl; need python3
|
||||
|
||||
# Refuse to start if Longhorn is already unhealthy. Powering off on top of a degraded volume is
|
||||
# how one replica becomes zero; better to stop here and let it rebuild first.
|
||||
local unhealthy
|
||||
unhealthy=$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io \
|
||||
-o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)
|
||||
[ "$unhealthy" = "0" ] || die "$unhealthy Longhorn volume(s) are not healthy — fix or wait for rebuild BEFORE powering off"
|
||||
ok "Longhorn: all volumes healthy"
|
||||
|
||||
# 1. Flux first, or it reverts every scale-down below within its reconcile interval.
|
||||
log "suspending Flux Kustomizations"
|
||||
local k
|
||||
for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
|
||||
"${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":true}}' >/dev/null
|
||||
done
|
||||
ok "all Kustomizations suspended"
|
||||
|
||||
# 2. The VM needs an ACPI shutdown of the GUEST. KubeVirt does that on running:false; a plain
|
||||
# pod eviction would be a power cut to the VM.
|
||||
if "${KUBECTL[@]}" -n unifi get vm unifi >/dev/null 2>&1; then
|
||||
log "ACPI-stopping the UniFi VM"
|
||||
"${KUBECTL[@]}" -n unifi patch vm unifi --type=merge -p '{"spec":{"running":false}}' >/dev/null
|
||||
fi
|
||||
|
||||
# 3. Postgres: hibernation is a clean smart shutdown that KEEPS the PVCs. CNPG does one instance
|
||||
# at a time, primary last-ish, so this is not instant.
|
||||
log "hibernating CloudNativePG clusters"
|
||||
local c ns name
|
||||
for c in "${CNPG[@]}"; do
|
||||
ns="${c%%/*}"; name="${c##*/}"
|
||||
"${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
|
||||
cnpg.io/hibernation=on --overwrite >/dev/null 2>&1 || warn "could not hibernate $c"
|
||||
done
|
||||
|
||||
# 4. Everything holding a PVC, scaled in place. NFS consumers first so nothing holds a hard
|
||||
# TrueNAS mount; PiHole last because it is the LAN resolver.
|
||||
log "scaling down PVC-holding workloads"
|
||||
local line ns2 kind nm deferred=() skipped_delete_policy=()
|
||||
while read -r ns2 kind nm; do
|
||||
[ -z "${ns2:-}" ] && continue
|
||||
if [ "$kind" = "statefulset" ] && sts_deletes_pvc_on_scale "$ns2" "$nm"; then
|
||||
warn "SKIPPING $ns2/$nm — whenScaled: Delete would DELETE its PVC (see the guard above)."
|
||||
warn " Left running; its volume hard-detaches at poweroff and rebuilds on restart."
|
||||
skipped_delete_policy+=("$ns2/$nm")
|
||||
continue
|
||||
fi
|
||||
if [ "$ns2" = "pihole" ]; then deferred+=("$ns2 $kind $nm"); continue; fi
|
||||
"${KUBECTL[@]}" -n "$ns2" scale "$kind" "$nm" --replicas=0 >/dev/null 2>&1 \
|
||||
&& printf ' scaled 0 %s/%s %s\n' "$ns2" "$kind" "$nm" \
|
||||
|| warn "could not scale $ns2/$kind/$nm"
|
||||
done < <(pvc_workloads)
|
||||
for line in "${deferred[@]:-}"; do
|
||||
[ -z "${line:-}" ] && continue
|
||||
# shellcheck disable=SC2086
|
||||
set -- $line
|
||||
"${KUBECTL[@]}" -n "$1" scale "$2" "$3" --replicas=0 >/dev/null 2>&1 \
|
||||
&& printf ' scaled 0 %s/%s %s (LAN DNS — expect name resolution to stop)\n' "$1" "$2" "$3"
|
||||
done
|
||||
|
||||
# 5. The only assertion that matters. An attached volume at poweroff comes back degraded and
|
||||
# rebuilds; this is what we are buying with all of the above.
|
||||
wait_for "UniFi VMI gone" \
|
||||
'[ "$('"${KUBECTL[*]}"' -n unifi get vmi --no-headers 2>/dev/null | wc -l)" = "0" ]' || true
|
||||
wait_for "all Postgres instances stopped" \
|
||||
'[ "$('"${KUBECTL[*]}"' get pods -A --no-headers 2>/dev/null | grep -cE "(keycloak|nextcloud|semaphore)-db-[0-9]")" = "0" ]' || true
|
||||
wait_for "all Longhorn volumes detached" \
|
||||
'[ "$('"${KUBECTL[*]}"' -n longhorn-system get volumes.longhorn.io -o jsonpath="{range .items[*]}{.status.state}{\"\n\"}{end}" 2>/dev/null | grep -c attached)" = "0" ]' || \
|
||||
warn "some volumes are STILL ATTACHED — they will come back degraded and rebuild; acceptable, but note which"
|
||||
|
||||
if [ "${#skipped_delete_policy[@]}" -gt 0 ]; then
|
||||
echo
|
||||
warn "NOT scaled down (whenScaled: Delete — scaling them would delete their PVCs):"
|
||||
printf ' %s\n' "${skipped_delete_policy[@]}"
|
||||
warn "Expect these volumes to come back 'degraded' and rebuild. That is correct and automatic."
|
||||
fi
|
||||
echo
|
||||
ok "QUIESCED. Safe to power off."
|
||||
cat <<'NEXT'
|
||||
|
||||
Next, from inside the devcontainer (it holds the become password):
|
||||
cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
|
||||
-m shell -a 'shutdown -h +1 "maintenance"'
|
||||
|
||||
THEN, in this order and not before:
|
||||
1. TrueNAS (192.168.2.30) — web UI only; not in the ansible inventory and SSH is refused.
|
||||
2. Router / switch last.
|
||||
NEXT
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- resume
|
||||
do_resume() {
|
||||
need kubectl
|
||||
|
||||
# THE STEP THAT IS ALWAYS FORGOTTEN. Hibernation is imperative state: Flux neither sets nor
|
||||
# clears it, so resuming Flux alone leaves all three databases down and every app that needs
|
||||
# one looking broken for reasons that point nowhere near Postgres.
|
||||
log "un-hibernating CloudNativePG clusters"
|
||||
local c ns name
|
||||
for c in "${CNPG[@]}"; do
|
||||
ns="${c%%/*}"; name="${c##*/}"
|
||||
"${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
|
||||
cnpg.io/hibernation=off --overwrite >/dev/null 2>&1 || warn "could not un-hibernate $c"
|
||||
done
|
||||
|
||||
# Flux restores every replica count AND the VM's running:true from git, so nothing here needs to
|
||||
# undo the scale-downs by hand.
|
||||
log "resuming Flux Kustomizations"
|
||||
local k
|
||||
for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
|
||||
"${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":false}}' >/dev/null
|
||||
done
|
||||
ok "all Kustomizations resumed"
|
||||
|
||||
wait_for "Flux fully reconciled" \
|
||||
'[ "$('"${KUBECTL[*]}"' -n flux-system get kustomizations --no-headers 2>/dev/null | grep -vc True)" = "0" ]' || \
|
||||
warn "some Kustomizations are not Ready — the graph serialises behind platform-config, so give it a few minutes before worrying"
|
||||
|
||||
echo
|
||||
ok "RESUMED."
|
||||
cat <<'NEXT'
|
||||
|
||||
Known cold-start race, not a fault in this script: oauth2-proxy performs OIDC discovery at
|
||||
startup and exits 1 on the 503 it gets while Keycloak is still coming up, then sits in
|
||||
exponential CrashLoopBackOff. If a replica is stuck, delete the pod once:
|
||||
kubectl --context k3s-prod -n oauth2-proxy delete pod <name>
|
||||
NEXT
|
||||
}
|
||||
|
||||
do_status() {
|
||||
need kubectl
|
||||
printf 'nodes: %s Ready\n' "$("${KUBECTL[@]}" get nodes --no-headers | grep -c ' Ready')"
|
||||
printf 'flux: %s/%s Ready\n' \
|
||||
"$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | grep -c True)" \
|
||||
"$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | wc -l)"
|
||||
printf 'longhorn: %s attached, %s detached, %s not healthy\n' \
|
||||
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c attached)" \
|
||||
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c detached)" \
|
||||
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)"
|
||||
printf 'postgres: %s instance pod(s)\n' \
|
||||
"$("${KUBECTL[@]}" get pods -A --no-headers | grep -cE '(keycloak|nextcloud|semaphore)-db-[0-9]' || true)"
|
||||
printf 'unifi VM: %s VMI\n' "$("${KUBECTL[@]}" -n unifi get vmi --no-headers 2>/dev/null | wc -l)"
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
quiesce) do_quiesce ;;
|
||||
resume) do_resume ;;
|
||||
status) do_status ;;
|
||||
*) cat >&2 <<USAGE
|
||||
usage: $0 {quiesce|resume|status}
|
||||
|
||||
quiesce suspend Flux, ACPI-stop the UniFi VM, hibernate Postgres, scale down every
|
||||
PVC-holding workload, and wait for all Longhorn volumes to detach.
|
||||
Does NOT power anything off — see runbooks/cluster-maintenance-shutdown.md.
|
||||
resume the exact inverse: un-hibernate Postgres, resume Flux, wait for reconcile.
|
||||
status one-screen readout of nodes / Flux / Longhorn / Postgres / VM.
|
||||
|
||||
Run it inside the pinned toolbox, which supplies kubectl and the context:
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
|
||||
USAGE
|
||||
exit 2 ;;
|
||||
esac
|
||||
@@ -16,71 +16,6 @@ spec:
|
||||
externalIPs: false
|
||||
interfaces:
|
||||
- ^e(n|th)
|
||||
# ===============================================================================================
|
||||
# WHY THIS IS A matchExpressions LIST AND NOT `matchLabels: {kubernetes.io/os: linux}`
|
||||
# ===============================================================================================
|
||||
# The second expression excludes every node labelled `homelab.keithjackson.dev/vpn-ingress=true`
|
||||
# from announcing ANY LoadBalancer VIP. That is the node the SOHO router forwards WAN udp/51820 to.
|
||||
# It is not tuning; it is the fix for a measured, silent, total failure of remote access.
|
||||
#
|
||||
# MEASURED ON PROD 2026-10-08: A NODE CANNOT REACH A VIP IT ANNOUNCES ITSELF. Cilium's L2
|
||||
# announcer answers ARP for the VIP on behalf of OTHER LAN hosts, but not for the announcing node
|
||||
# itself. On the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP
|
||||
# is dropped before it leaves the node:
|
||||
#
|
||||
# k3s-prod-1, holding cilium-l2announce-gateway-cilium-gateway-shared:
|
||||
# ip neigh show 192.168.2.20 -> 192.168.2.20 dev eno1 FAILED (its own VIP)
|
||||
# nc -z 192.168.2.20 443 -> FAIL nc -z 192.168.2.20 80 -> FAIL
|
||||
# k3s-prod-2 and k3s-prod-3, holding no lease for it:
|
||||
# ip neigh show 192.168.2.20 -> lladdr bc:e9:2f:7e:c4:a6 (prod-1) nc 443/80 -> OK
|
||||
#
|
||||
# Symmetric across all 3 nodes x 5 VIPs: every node reached every VIP EXCEPT the ones it alone
|
||||
# announced (prod-2 could not reach crafty 192.168.2.22 or unifi 192.168.2.21; prod-3 could not
|
||||
# reach plex 192.168.2.24). So this is a property of L2 announcement, not of one Service.
|
||||
#
|
||||
# WHY IT ONLY HURTS THE VPN NODE. For an ordinary workload this is harmless: pods reach Services
|
||||
# through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception --
|
||||
# it runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own
|
||||
# address, so a VPN client's packets take the node's LAN path and inherit exactly this ARP
|
||||
# failure. On 2026-10-08 the forward pointed at k3s-prod-1, which held the Gateway lease, and all
|
||||
# 16 Gateway hostnames were unreachable over the VPN while PiHole DNS still answered -- so names
|
||||
# resolved and every app hung. Nothing in the cluster reported it; the alerts below now do.
|
||||
#
|
||||
# WHY A NODE LABEL AND NOT A cluster-vars KNOB. A knob would put the node's identity in git, which
|
||||
# is more reviewable -- but it CANNOT BE USED BY THE ALERT that guards this, and an unguarded
|
||||
# invariant here is worth less than a reviewable one. The alert harness
|
||||
# (platform/monitoring/tests/extract_rules.py) REFUSES any alert whose `expr` contains a
|
||||
# dollar-brace postBuild placeholder (spelled without braces here ON PURPOSE: this file is itself
|
||||
# envsubst'd, and a literal dollar-brace-dot-dot-dot is UNPARSEABLE to envsubst, which fails the
|
||||
# WHOLE Kustomization -- ci: flux-postbuild-substitution caught exactly this comment), because the
|
||||
# rule it would unit-test is not the rule that gets
|
||||
# deployed. A label is readable from BOTH sides: this selector matches on it, and
|
||||
# `kube_node_labels` carries it into Prometheus so `VpnIngressNodeAnnouncesL2VIP` can join lease
|
||||
# holders against it. One source of truth, no git-vs-cluster skew, and the policy stays
|
||||
# env-agnostic as its header above claims. The label is set by
|
||||
# infra/ansible/roles/node_baseline (node_baseline_vpn_ingress_node), the same shape
|
||||
# roles/vpn_egress_gateway already uses for `egress-gateway`.
|
||||
#
|
||||
# THE COST, STATED: VIP announcement candidacy drops from 3 nodes to 2. That is still redundant,
|
||||
# and the trade is deliberate -- the excluded node is the one whose *outbound* reach to every VIP
|
||||
# matters most, because it carries every remote client. If BOTH remaining nodes are down the VIPs
|
||||
# go dark where previously one node might have served them, but at that point the cluster has one
|
||||
# node left and far worse problems.
|
||||
#
|
||||
# `NotIn` with an ABSENT key MATCHES (k8s apimachinery: a missing label is "not in the set"), so
|
||||
# the other two nodes need no label and nothing has to be applied to them. If the label is never
|
||||
# applied anywhere, this excludes nothing and the policy degrades to its pre-2026-10-08 behaviour
|
||||
# -- the status quo, not an outage: it fails back, not down. That silent-degradation path is
|
||||
# exactly why `VpnIngressNodeLabelMissing` exists as a watchdog: without it, the alert below
|
||||
# would be structurally INERT (its join would match nothing) and would look identical to healthy.
|
||||
nodeSelector:
|
||||
matchExpressions:
|
||||
- key: kubernetes.io/os
|
||||
operator: In
|
||||
values:
|
||||
- linux
|
||||
# The node terminating remote-access WireGuard must announce nothing -- see above.
|
||||
- key: homelab.keithjackson.dev/vpn-ingress
|
||||
operator: NotIn
|
||||
values:
|
||||
- "true"
|
||||
matchLabels:
|
||||
kubernetes.io/os: linux
|
||||
@@ -1147,97 +1147,6 @@ spec:
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# REMOTE-ACCESS VPN (platform/vpn-ingress) -- NOT the egress tunnel above. Different
|
||||
# direction, different workload: this is inbound WireGuard for remote devices.
|
||||
# -----------------------------------------------------------------------------------------
|
||||
- alert: VpnIngressNodeAnnouncesL2VIP
|
||||
# THE FAILURE THIS CATCHES. MEASURED on prod 2026-10-08: a node CANNOT reach a
|
||||
# LoadBalancer VIP whose Cilium L2 announcement lease it holds itself -- the announcer
|
||||
# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and
|
||||
# packets die before leaving the node (`ip neigh show 192.168.2.20` -> FAILED on the
|
||||
# lease holder, resolved on both others). platform/vpn-ingress runs WireGuard on
|
||||
# `hostNetwork` and MASQUERADEs remote clients to the node's own address, so if the node
|
||||
# the router forwards udp/51820 to is also a lease holder, VPN clients cannot reach the
|
||||
# VIPs it announces. On 2026-10-08 that was the `shared` Gateway: all 16 app hostnames
|
||||
# dead over the VPN, while PiHole DNS kept answering (its VIP has two Services and the
|
||||
# other lease sat on prod-2) -- so the tunnel handshook, names resolved, and every app
|
||||
# hung. Nothing reported it for ~2 days.
|
||||
#
|
||||
# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
|
||||
# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so this should be
|
||||
# permanently unsatisfiable. The alert exists because the exclusion has ways to stop
|
||||
# working that no static check on that file's TEXT can see: the label being removed or
|
||||
# moved to the wrong node, a Cilium upgrade changing `NotIn` semantics or the
|
||||
# nodeSelector field, and the selector silently failing to match. This watches the
|
||||
# OUTCOME -- who actually holds the lease -- so all of them surface the same way.
|
||||
#
|
||||
# WHY THE LABEL JOIN RATHER THAN A NODE NAME. The node's identity is NOT written into
|
||||
# this expr, because the harness (tests/extract_rules.py) refuses any alert whose expr
|
||||
# contains a dollar-brace placeholder -- spelled in words because a literal one here is
|
||||
# unparseable to envsubst and would fail this whole Kustomization. Rightly so: the rule it
|
||||
# unit-tests would not be the
|
||||
# rule deployed. Joining `kube_lease_owner` against `kube_node_labels` keeps ONE source
|
||||
# of truth (the label itself, which is also what the policy selects on), so this alert
|
||||
# cannot drift out of agreement with the exclusion it guards. `label_replace` is needed
|
||||
# because kube_lease_owner names the holder `lease_holder` while kube_node_labels names
|
||||
# it `node`; the join is then on a shared `node` label. The label is in
|
||||
# kube-state-metrics' metricLabelsAllowlist (kube-prometheus-stack.yaml) -- without that
|
||||
# line this rule is inert, which `VpnIngressNodeLabelMissing` below exists to catch.
|
||||
#
|
||||
# `for: 15m` because leases legitimately move during a node reboot or a Cilium roll and
|
||||
# Cilium re-elects within seconds; a real collision persists, since the lease holder
|
||||
# keeps renewing and it never self-heals.
|
||||
expr: |
|
||||
max by (lease) (
|
||||
label_replace(
|
||||
kube_lease_owner{namespace="kube-system", lease=~"cilium-l2announce-.*"},
|
||||
"node", "$1", "lease_holder", "(.+)"
|
||||
)
|
||||
* on (node) group_left ()
|
||||
kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"}
|
||||
) == 1
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "VPN ingress node is announcing a LoadBalancer VIP ({{ $labels.lease }})"
|
||||
description: >-
|
||||
The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
|
||||
WireGuard, and it now holds the Cilium L2 announcement lease `{{ $labels.lease }}`. A
|
||||
node cannot reach a VIP it announces itself, so every remote VPN client has just lost
|
||||
all access to that VIP's Service -- silently, with DNS still resolving. Check that
|
||||
platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
|
||||
the label is still on the node the router forwards udp/51820 to.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
|
||||
- alert: VpnIngressNodeLabelMissing
|
||||
# THE WATCHDOG for the rule above, and the reason it is not optional. The exclusion in
|
||||
# l2-announcement-policy.yaml is a `NotIn` on this label, and `NotIn` with an ABSENT key
|
||||
# MATCHES -- so if the label is on NO node, the policy excludes nothing, every node is an
|
||||
# announcement candidate again, and the 2026-10-08 failure is fully re-armed. The alert
|
||||
# above would not fire either: its join would match nothing and return no series. Present,
|
||||
# evaluating, and incapable of firing -- the exact "inert critical alert" shape that
|
||||
# VPNEgressUplinkLabelMissing exists for on the egress side, and that this repo has been
|
||||
# burned by before.
|
||||
#
|
||||
# The label is applied by infra/ansible/roles/node_baseline
|
||||
# (node_baseline_vpn_ingress_node); the most likely cause of it going absent is a node
|
||||
# rebuild where that role has not been re-run.
|
||||
expr: absent(kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"})
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
|
||||
description: >-
|
||||
No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
|
||||
l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
|
||||
any node may now announce a LoadBalancer VIP, including the one terminating
|
||||
remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
|
||||
the warning matters more than its severity suggests. Re-run the node_baseline role on
|
||||
the forwarded node, and confirm the label is in kube-state-metrics'
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
|
||||
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
|
||||
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
||||
|
||||
@@ -281,20 +281,8 @@ spec:
|
||||
# without anyone having to remember to re-enable it later. Scoped to one label on one resource:
|
||||
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
|
||||
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
|
||||
# `vpn-ingress` (added 2026-10-17) marks the node the SOHO router forwards WAN udp/51820 to.
|
||||
# platform-config/cilium/l2-announcement-policy.yaml excludes that node from announcing any
|
||||
# LoadBalancer VIP, because a node cannot reach a VIP whose L2 announcement lease it holds
|
||||
# itself and vpn-ingress MASQUERADEs remote clients to the node's own address (measured
|
||||
# 2026-10-08: all 16 Gateway hostnames dead over the VPN while DNS still resolved). It is
|
||||
# allowlisted HERE because that is the only way the invariant can be WATCHED:
|
||||
# `VpnIngressNodeAnnouncesL2VIP` joins `kube_lease_owner` against this label, and the alert
|
||||
# unit-test harness refuses a dollar-brace placeholder in an alert expr (written out in words:
|
||||
# a literal one here is unparseable to envsubst and fails the whole Kustomization), so a knob
|
||||
# could not have been used. Without this line BOTH that alert and its
|
||||
# `VpnIngressNodeLabelMissing` watchdog are structurally inert. Same reasoning, and the same
|
||||
# 3-extra-series cost, as the two egress labels below.
|
||||
metricLabelsAllowlist:
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink,homelab.keithjackson.dev/vpn-ingress]"
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
|
||||
# --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
|
||||
# Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
|
||||
# `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
|
||||
|
||||
@@ -1,203 +0,0 @@
|
||||
# promtool rule-unit tests for VpnIngressNodeAnnouncesL2VIP + VpnIngressNodeLabelMissing
|
||||
# (platform/monitoring/alerts.yaml)
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh
|
||||
# (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
|
||||
# MEASURED on prod 2026-10-08: a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
|
||||
# lease it holds ITSELF -- the announcer does not answer its own ARP request, so on the lease holder
|
||||
# the neighbour entry settles at `FAILED` and packets to that VIP die before leaving the node.
|
||||
# platform/vpn-ingress runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
|
||||
# own address, so a VPN client takes that node's LAN path and inherits the failure. With the
|
||||
# router's udp/51820 forward on k3s-prod-1, which held the `shared` Gateway lease, all 16 Gateway
|
||||
# hostnames were dead over the VPN -- while PiHole DNS kept answering, because its VIP has two
|
||||
# Services and k3s-prod-2 held the other lease. The tunnel handshook, names resolved, every app
|
||||
# hung, and nothing reported it for ~2 days.
|
||||
#
|
||||
# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
|
||||
# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so VpnIngressNodeAnnouncesL2VIP
|
||||
# should be permanently unsatisfiable. THAT IS EXACTLY WHY IT NEEDS A TEST: an alert expected never
|
||||
# to fire is indistinguishable, in production, from an alert that CANNOT fire. Nothing would ever
|
||||
# reveal a mistyped label, a broken `label_replace`, or a join on the wrong label. Test 1 proves it
|
||||
# fires on the real failure; test 2 proves it is quiet in the fixed state, so it is not a rule that
|
||||
# fires on anything; test 3 pins the `for: 15m` against routine lease churn.
|
||||
#
|
||||
# THE JOIN IS THE FRAGILE PART AND IS TESTED DELIBERATELY. `kube_lease_owner` names the holder in
|
||||
# `lease_holder`; `kube_node_labels` names it `node`. The rule bridges them with `label_replace`
|
||||
# and joins `on (node) group_left ()`. Fixtures therefore carry BOTH series shapes with real label
|
||||
# sets, because a join that silently matches nothing returns no series -- which looks exactly like
|
||||
# "healthy" and is the failure mode this whole group exists to prevent.
|
||||
#
|
||||
# Fixture values are prod's real shape (measured 2026-10-08): leases
|
||||
# `cilium-l2announce-gateway-cilium-gateway-shared` (k3s-prod-1) and
|
||||
# `cilium-l2announce-pihole-pihole-dns-tcp` (k3s-prod-2), in namespace kube-system, with
|
||||
# k3s-prod-1 the vpn-ingress node.
|
||||
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# =============================================================================================
|
||||
# 1. THE BUG, EXACTLY AS IT HAPPENED -- the vpn-ingress node holds the Gateway lease: FIRES.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: vpn-ingress node holds the shared Gateway L2 lease - VpnIngressNodeAnnouncesL2VIP fires
|
||||
input_series:
|
||||
# The collision. A lease holder keeps renewing, so this series does not lapse on its own --
|
||||
# which is why the condition never self-heals and why a 15m `for:` is safe.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
|
||||
values: '1+0x40'
|
||||
# A second lease held by a DIFFERENT node, present throughout. This is the discriminator that
|
||||
# stops the rule from being "any l2announce lease exists anywhere": if the join were dropped,
|
||||
# this series alone would fire the alert, and test 2 would fail.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-pihole-pihole-dns-tcp",lease_holder="k3s-prod-2"}'
|
||||
values: '1+0x40'
|
||||
# k3s-prod-1 is the vpn-ingress node. Only this node carries the label.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
|
||||
values: '1+0x40'
|
||||
# The other two nodes publish kube_node_labels WITHOUT the vpn-ingress label. Present so the
|
||||
# join is exercised against non-matching nodes rather than against nothing at all.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
|
||||
values: '1+0x40'
|
||||
alert_rule_test:
|
||||
# Held 15m, so nothing before then.
|
||||
- eval_time: 14m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
- eval_time: 20m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts:
|
||||
# EXACTLY ONE alert: the pihole lease on prod-2 must NOT produce a second one.
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
lease: cilium-l2announce-gateway-cilium-gateway-shared
|
||||
exp_annotations:
|
||||
summary: "VPN ingress node is announcing a LoadBalancer VIP (cilium-l2announce-gateway-cilium-gateway-shared)"
|
||||
description: >-
|
||||
The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
|
||||
WireGuard, and it now holds the Cilium L2 announcement lease `cilium-l2announce-gateway-cilium-gateway-shared`. A
|
||||
node cannot reach a VIP it announces itself, so every remote VPN client has just lost
|
||||
all access to that VIP's Service -- silently, with DNS still resolving. Check that
|
||||
platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
|
||||
the label is still on the node the router forwards udp/51820 to.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
|
||||
# The label IS present, so the watchdog must stay quiet here.
|
||||
- eval_time: 40m
|
||||
alertname: VpnIngressNodeLabelMissing
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 2. THE FIXED STATE -- every lease is held by a node that is NOT the vpn-ingress node: QUIET.
|
||||
# This is what l2-announcement-policy.yaml's exclusion produces, and the assertion that keeps
|
||||
# test 1 meaningful rather than an alert that fires on anything.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: all l2 leases held by other nodes - VpnIngressNodeAnnouncesL2VIP stays quiet
|
||||
input_series:
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-2"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-plex-plex-l4-tcp",lease_holder="k3s-prod-3"}'
|
||||
values: '1+0x40'
|
||||
# A lease the vpn-ingress node DOES legitimately hold, in a different family -- proves the
|
||||
# `cilium-l2announce-.*` matcher is doing work. Every node holds operator/controller leases;
|
||||
# only the ANNOUNCEMENT ones matter here, and a matcher broad enough to catch this would page
|
||||
# on a healthy cluster forever.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-operator-resource-lock",lease_holder="k3s-prod-1"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
|
||||
values: '1+0x40'
|
||||
alert_rule_test:
|
||||
- eval_time: 40m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
- eval_time: 40m
|
||||
alertname: VpnIngressNodeLabelMissing
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 3. A LEGITIMATE, TRANSIENT LEASE MOVE -- the vpn-ingress node holds it for 11m during a node
|
||||
# reboot or a Cilium roll, then Cilium re-elects away: QUIET. This pins the `for: 15m` as
|
||||
# deliberate. Without it, every rolling restart of the cilium DaemonSet would page, and a paging
|
||||
# alert that cries wolf on routine maintenance gets muted -- which is how the real collision
|
||||
# would go unseen again.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: transient lease move shorter than for 15m - VpnIngressNodeAnnouncesL2VIP stays quiet
|
||||
input_series:
|
||||
# Held by the vpn-ingress node for 11 samples, then the series goes absent (the lease moved,
|
||||
# so kube-state-metrics retires this label set and starts a new one).
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
|
||||
values: '1+0x10 _x30'
|
||||
# Re-elected onto prod-3 from t=11m. A label VALUE change retires one series and starts
|
||||
# another, which is why this is a second series and not a changing value.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-3"}'
|
||||
values: '_x11 1+0x29'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
|
||||
values: '1+0x40'
|
||||
alert_rule_test:
|
||||
# Inside the transient window: pending at most, never firing.
|
||||
- eval_time: 10m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
# Well past it, with the lease settled on another node.
|
||||
- eval_time: 30m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 4. THE WATCHDOG -- no node carries the label at all. The policy's `NotIn` then excludes
|
||||
# NOTHING (a missing key matches), so the 2026-10-08 failure is fully re-armed, AND
|
||||
# VpnIngressNodeAnnouncesL2VIP is structurally INERT: its join matches nothing and returns no
|
||||
# series, which is indistinguishable from healthy. Both halves are asserted here, because an
|
||||
# inert critical alert is the exact shape this repo has been burned by before.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: vpn-ingress label absent everywhere - watchdog fires and the critical alert is inert
|
||||
input_series:
|
||||
# A lease held by the node that IS (in reality) the forwarded one -- but with no label
|
||||
# anywhere, nothing can tell. This is the dangerous state.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
|
||||
values: '1+0x60'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
|
||||
values: '1+0x60'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
|
||||
values: '1+0x60'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
|
||||
values: '1+0x60'
|
||||
alert_rule_test:
|
||||
# The critical alert CANNOT fire -- this is the inertness being pinned, not an aside.
|
||||
- eval_time: 45m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
# ... which is precisely why the watchdog must.
|
||||
- eval_time: 45m
|
||||
alertname: VpnIngressNodeLabelMissing
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
# absent() propagates the matcher's own labels onto the series it returns, so the
|
||||
# alert inherits this one. Asserted rather than worked around: it is harmless, and
|
||||
# pinning it means a future change to the matcher cannot alter the alert's identity
|
||||
# (and therefore its Alertmanager grouping) unnoticed.
|
||||
label_homelab_keithjackson_dev_vpn_ingress: "true"
|
||||
exp_annotations:
|
||||
summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
|
||||
description: >-
|
||||
No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
|
||||
l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
|
||||
any node may now announce a LoadBalancer VIP, including the one terminating
|
||||
remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
|
||||
the warning matters more than its severity suggests. Re-run the node_baseline role on
|
||||
the forwarded node, and confirm the label is in kube-state-metrics'
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
|
||||
@@ -63,75 +63,6 @@ spec:
|
||||
# gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
|
||||
skip-provider-button: "true"
|
||||
scope: "openid email profile"
|
||||
# ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
|
||||
# MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
|
||||
# startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
|
||||
# the Gateway answers 503 and the proxy dies:
|
||||
#
|
||||
# [provider.go:55] Performing OIDC Discovery...
|
||||
# [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
|
||||
# configuration: unexpected status "503": no healthy upstream
|
||||
#
|
||||
# It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
|
||||
# still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
|
||||
# process simply does not survive a cold start. A readinessProbe cannot help (the container is
|
||||
# already dead), and CrashLoopBackOff backoff is not tunable per-pod.
|
||||
#
|
||||
# `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
|
||||
# values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
|
||||
# container for its built-in waitForRedis gate, so this adds no new base image to the platform.
|
||||
# A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
|
||||
# point.
|
||||
#
|
||||
# BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
|
||||
# case is then exactly today's behaviour, never something new. An unbounded wait would trade a
|
||||
# CrashLoop for a pod wedged in Init forever, which is not an improvement.
|
||||
#
|
||||
# SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
|
||||
# rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
|
||||
# CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
|
||||
# silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
|
||||
# knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
|
||||
# OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
|
||||
extraInitContainers:
|
||||
- name: wait-for-idp
|
||||
image: docker.io/library/alpine:3.22
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -u
|
||||
url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
|
||||
i=0
|
||||
while [ "$i" -lt 60 ]; do
|
||||
# busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
|
||||
# failure is a non-zero exit here rather than a false pass.
|
||||
if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
|
||||
echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
|
||||
exit 0
|
||||
fi
|
||||
i=$(( i + 1 ))
|
||||
sleep 5
|
||||
done
|
||||
echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
|
||||
echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
|
||||
exit 0
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65534
|
||||
runAsGroup: 65534
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
memory: 32Mi
|
||||
|
||||
# Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
|
||||
metrics:
|
||||
enabled: true
|
||||
|
||||
@@ -117,19 +117,8 @@ boundary around this port is WireGuard's peer authentication, full stop.
|
||||
The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
|
||||
forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
|
||||
A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
|
||||
keys — so losing the forwarded node never costs a re-issued client profile. The two idle listeners
|
||||
cost ~15 MiB each.
|
||||
|
||||
> **This section used to say "losing the forwarded node is a one-line router change, with no cluster
|
||||
> work". It was wrong, and it cost two days of silently broken remote access on 2026-10-08.** The
|
||||
> nodes are warm, but they are **not interchangeable**: the node the forward points at must be the
|
||||
> one labelled `homelab.keithjackson.dev/vpn-ingress=true`, because that is the node
|
||||
> `platform-config/cilium/l2-announcement-policy.yaml` excludes from announcing LoadBalancer VIPs.
|
||||
> **A node cannot reach a VIP whose Cilium L2 announcement lease it holds itself** (see "Measured on
|
||||
> prod, 2026-10-08" below), and since this pod MASQUERADEs clients to the node's own address, a VPN
|
||||
> client inherits that. Failover is therefore **two** changes: move
|
||||
> `node_baseline_vpn_ingress_node` and re-run that role, then re-point the router —
|
||||
> `runbooks/vpn-remote-access.md` step 3.
|
||||
keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
|
||||
re-issued client profiles. The two idle listeners cost ~15 MiB each.
|
||||
|
||||
A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
|
||||
replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
|
||||
@@ -197,45 +186,6 @@ right way; one was a defect in my own manifest.
|
||||
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
|
||||
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
|
||||
|
||||
## Measured on prod, 2026-10-08 — why the forwarded node cannot be any node
|
||||
|
||||
The first device enrolled, handshook, resolved names, and could not open a single connection to any
|
||||
cluster web app. The tunnel was not at fault — every part of it measured healthy:
|
||||
`net.ipv4.ip_forward=1`, `wg0` up at `10.90.0.1/24` MTU 1420, the peer authorised with real
|
||||
bidirectional transfer (346 KiB in / 200 KiB out), `MASQUERADE` present and counting 26 flows,
|
||||
`FORWARD -i/-o wg0 ACCEPT` counting 108 / 171 packets, and PiHole DNS answering from the node.
|
||||
|
||||
**The fault was that a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
|
||||
holds itself.** Cilium's announcer answers ARP for the VIP on behalf of other LAN hosts, but not for
|
||||
the announcing node's own request, so the neighbour entry settles at `FAILED`:
|
||||
|
||||
| from | `192.168.2.20` (Gateway) | `192.168.2.22` (crafty) | `192.168.2.24` (plex) | announces |
|
||||
|---|---|---|---|---|
|
||||
| `k3s-prod-1` | **FAILED** | — | — | `.20` |
|
||||
| `k3s-prod-2` | ok | **INCOMPLETE** | ok | `.21`, `.22`, `.23-tcp` |
|
||||
| `k3s-prod-3` | ok | ok | **INCOMPLETE** | `.24` |
|
||||
|
||||
Symmetric across 3 nodes × 5 VIPs: each node reached every VIP **except** the ones it alone
|
||||
announced. It is a property of L2 announcement, not of any one Service, and it is harmless for
|
||||
ordinary pods — they reach Services through the cluster datapath and never ARP for a VIP. This
|
||||
component is the exception, because `hostNetwork` + `MASQUERADE` put remote clients on the node's
|
||||
own LAN path.
|
||||
|
||||
The forward pointed at `k3s-prod-1`, which held the `shared` Gateway lease, so **all 16 Gateway
|
||||
hostnames were dead over the VPN**. Two things hid it:
|
||||
|
||||
* **PiHole survived by accident.** `192.168.2.23` is announced by *two* Services
|
||||
(`pihole-dns-udp` on prod-1, `pihole-dns-tcp` on prod-2), so prod-2 answered the ARP prod-1 could
|
||||
not. Names resolved perfectly and every app hung — the hardest shape to diagnose from a client.
|
||||
* **Nothing reported it.** No probe in this repo compares the forwarded node against the lease
|
||||
holders, and from outside WireGuard is silent to unauthenticated traffic anyway.
|
||||
|
||||
Closed by labelling the forwarded node `homelab.keithjackson.dev/vpn-ingress=true` and excluding
|
||||
that label in `platform-config/cilium/l2-announcement-policy.yaml` (the node then ARP-resolves every
|
||||
VIP), with `VpnIngressNodeAnnouncesL2VIP` watching the lease holders so a recurrence is loud and
|
||||
`VpnIngressNodeLabelMissing` watching for the label going absent, which would silently re-arm the
|
||||
whole failure. Cost: VIP announcement candidacy drops from three nodes to two.
|
||||
|
||||
## Prerequisites outside this directory
|
||||
|
||||
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
|
||||
|
||||
@@ -62,39 +62,9 @@
|
||||
# as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
|
||||
# whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
|
||||
# DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
|
||||
# them, and if that node is lost every other node is already warm and answers the same keys, so no
|
||||
# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict,
|
||||
# and the two idle listeners cost ~15 MiB each.
|
||||
#
|
||||
# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE.
|
||||
# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the
|
||||
# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node
|
||||
# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer
|
||||
# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to
|
||||
# that VIP die before leaving the node:
|
||||
#
|
||||
# k3s-prod-1, holding the shared Gateway lease: ip neigh show 192.168.2.20 -> FAILED
|
||||
# k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr …(prod-1)
|
||||
#
|
||||
# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client
|
||||
# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway
|
||||
# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that
|
||||
# handshakes, resolves names, and then hangs on every single app.
|
||||
#
|
||||
# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled
|
||||
# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from
|
||||
# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes
|
||||
# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a
|
||||
# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test
|
||||
# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move
|
||||
# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router
|
||||
# (runbooks/vpn-remote-access.md step 3).
|
||||
# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again.
|
||||
#
|
||||
# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge
|
||||
# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness
|
||||
# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a
|
||||
# property of the CLUSTER's announcement layout, so it is watched there.
|
||||
# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no
|
||||
# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own
|
||||
# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each.
|
||||
#
|
||||
# All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
|
||||
# DaemonSet to cover the cluster; add them if a node is ever tainted.
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
# cluster-maintenance-shutdown — powering the whole cluster off, and back on
|
||||
|
||||
**Operator procedure, not an alert page.** Nothing links here from an alert.
|
||||
**Script:** `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`)
|
||||
**Related:** `/runbooks/update-rollback` is for rolling node work where the cluster stays up. This
|
||||
page is for taking **all three nodes down at once**.
|
||||
|
||||
First written after the 2026-10-17 UPS move, which was improvised live because no procedure
|
||||
existed — `infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev
|
||||
cluster, so it does nothing for these three baremetal nodes.
|
||||
|
||||
## Why you cannot just run `shutdown -h`
|
||||
|
||||
You mostly can now, and could not before.
|
||||
|
||||
Until 2026-10-17 **kubelet graceful node shutdown was unconfigured on every prod node**, so systemd
|
||||
stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood — 9 Postgres
|
||||
instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes.
|
||||
`roles/k3s_server` now sets `shutdown-grace-period` (and, critically, the
|
||||
`InhibitDelayMaxSec` logind cap that actually enforces it — without that systemd stops waiting
|
||||
after 5s and the drain is silently truncated). A plain reboot of one node is therefore safe.
|
||||
|
||||
**A planned whole-cluster shutdown still wants `quiesce`**, for three things the kubelet's drain
|
||||
does not do:
|
||||
|
||||
1. **The UniFi VM needs an ACPI guest shutdown**, not an eviction. KubeVirt does that on
|
||||
`running: false`; an eviction is a power cut to the guest.
|
||||
2. **CloudNativePG wants hibernation**, not pod deletion — and hibernation is **imperative** state
|
||||
that Flux will neither set nor clear.
|
||||
3. **Draining three nodes one at a time makes pods reschedule onto the survivors**, churning
|
||||
Longhorn replicas for no reason when everything is going down anyway. Scaling in place avoids it.
|
||||
|
||||
## The hazard that bit us — read this before scaling anything by hand
|
||||
|
||||
**A StatefulSet can carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and scaling it
|
||||
to 0 then DELETES its PVCs and their Longhorn volumes.**
|
||||
|
||||
On 2026-10-17 the hand-rolled quiesce scaled `monitoring/loki` (which has that policy) to 0, and
|
||||
`storage-loki-0` was deleted outright. Loki stores with `type: filesystem`, so that PVC was the only
|
||||
copy. Its 72h retention bounded the loss to a rolling window — nothing about the *mechanism* is
|
||||
bounded. The only visible symptom was the Longhorn volume count going 25 → 24, which was first
|
||||
misread as a miscount.
|
||||
|
||||
`quiesce.sh` now refuses to scale any such StatefulSet, leaves it running, and tells you which
|
||||
volumes to expect `degraded` on restart. **If you quiesce by hand, check the policy first:**
|
||||
|
||||
```bash
|
||||
kubectl --context k3s-prod get statefulset -A \
|
||||
-o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,WHEN_SCALED:.spec.persistentVolumeClaimRetentionPolicy.whenScaled'
|
||||
```
|
||||
|
||||
A hard-detached volume comes back `degraded` and rebuilds itself automatically. That is a vastly
|
||||
better outcome than a deleted one — take the rebuild.
|
||||
|
||||
## Shutting down
|
||||
|
||||
Run everything through the pinned toolbox; it supplies `kubectl` and the context.
|
||||
|
||||
```bash
|
||||
# 0. Look before you leap. Refuses to proceed if any Longhorn volume is unhealthy —
|
||||
# powering off on top of a degraded volume is how one replica becomes zero.
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
|
||||
|
||||
# 1. Quiesce: suspend Flux, ACPI-stop the VM, hibernate Postgres, scale down every
|
||||
# PVC-holding workload (NFS consumers first, PiHole last), wait for detach.
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh quiesce'
|
||||
```
|
||||
|
||||
**Expect DNS to stop** when PiHole scales down — it is the LAN resolver. `kubectl` is unaffected
|
||||
because the kubeconfig points at the kube-vip IP `192.168.2.9`, not a name.
|
||||
|
||||
```bash
|
||||
# 2. Power the nodes off. Inside the devcontainer, which holds the become password.
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
|
||||
-m shell -a "shutdown -h +1 \"maintenance\""'
|
||||
```
|
||||
|
||||
`+1` gives Ansible time to disconnect cleanly. The group resolves to all three nodes; confirm with
|
||||
`--list-hosts` if you want to see them first.
|
||||
|
||||
```bash
|
||||
# 3. Confirm they are actually down before touching anything else.
|
||||
for h in 192.168.2.10 192.168.2.11 192.168.2.12; do ping -c1 -W2 $h >/dev/null && echo "$h UP" || echo "$h down"; done
|
||||
```
|
||||
|
||||
**Then, in this order and not before:**
|
||||
|
||||
| # | Machine | How |
|
||||
|---|---|---|
|
||||
| 4 | **TrueNAS** `192.168.2.30` | **Web UI only.** Not in the ansible inventory, and SSH is **refused** (service disabled). `https://192.168.2.30` → power icon → Shut Down. Safe once the nodes are down, because every NFS consumer (`plex-media`, `*-media`, `qbittorrent-downloads`, `nextcloud-legacy-archive`) is already scaled to zero and nothing holds a hard mount. |
|
||||
| 5 | **Router / switch** | Last. |
|
||||
|
||||
TrueNAS must go **after** the k3s nodes: a hard NFS mount on a live node wedges kubelet in
|
||||
D-state (`/runbooks/nfs-export-latency`, and the load-average symptom in the 2026-10-?? note).
|
||||
|
||||
## Bringing it back
|
||||
|
||||
**Power on in the reverse order: router/switch → TrueNAS → all three nodes together.** etcd needs
|
||||
2 of 3 for quorum, so start the nodes as a group rather than one at a time.
|
||||
|
||||
```bash
|
||||
# 1. Nodes Ready, etcd quorum, storage reachable.
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get --raw=/healthz/etcd'
|
||||
|
||||
# 2. Resume: un-hibernate Postgres, then resume Flux.
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh resume'
|
||||
```
|
||||
|
||||
**Un-hibernating Postgres is the step everyone forgets.** The annotation is imperative and survives
|
||||
a Flux resume, so resuming Flux alone leaves all three databases down and Keycloak, Nextcloud and
|
||||
Semaphore looking broken for reasons that point nowhere near Postgres. `resume` does it first, for
|
||||
exactly that reason.
|
||||
|
||||
Flux restores every scaled-down replica count **and** the VM's `running: true` from git, so nothing
|
||||
needs undoing by hand.
|
||||
|
||||
### What "still broken" looks like when it is actually fine
|
||||
|
||||
* **`4/19` Kustomizations Ready for several minutes.** The dependency graph serialises behind
|
||||
`platform-config`; 15 of them report `dependency ... is not ready` until it finishes. Check
|
||||
`platform-config` itself is `Ready=True` and its `lastAppliedRevision` matches the GitRepository
|
||||
artifact — if it is, wait.
|
||||
* **One or more Longhorn volumes `degraded`.** Any volume that was still attached at poweroff
|
||||
rebuilds its lagging replica automatically. On 2026-10-17 it was Prometheus's 50Gi TSDB; it went
|
||||
healthy on its own. Worry only if it is still degraded after ~30 minutes.
|
||||
* **`keycloak-operator` crashlooping, then recovering.** It is the CRD reconciler, not in the
|
||||
request path; Keycloak's StatefulSet stays 2/2 and auth never goes down. See
|
||||
`/runbooks/keycloak-*` and the known OOM-at-450Mi behaviour.
|
||||
|
||||
### What is actually broken and needs you
|
||||
|
||||
* **An `oauth2-proxy` replica stuck in CrashLoopBackOff.** It performs OIDC discovery at startup and
|
||||
**exits 1** on the `503 no healthy upstream` it gets while Keycloak is still coming up, then sits
|
||||
in exponential backoff long after Keycloak is fine. It has no startup dependency on Keycloak.
|
||||
Delete the pod once and it comes up clean:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n oauth2-proxy delete pod <name>'
|
||||
```
|
||||
|
||||
* **Orphaned `Unknown`/`Error` Job pods** from CronJobs that were mid-run at poweroff. They never
|
||||
recover; delete them.
|
||||
|
||||
## Verifying you are actually back
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
|
||||
# nodes 3 Ready | flux 19/19 | longhorn 0 not healthy | postgres 9 pods | unifi 1 VMI
|
||||
|
||||
# DNS, through PiHole's VIP
|
||||
./infra/devcontainer/dev.sh run bash -c 'nslookup home.keithjackson.dev 192.168.2.23'
|
||||
|
||||
# The Gateway actually serving (401 on an auth-gated host is CORRECT)
|
||||
for h in home.keithjackson.dev gitea.home.keithjackson.dev plex.home.keithjackson.dev; do
|
||||
curl -s -o /dev/null -w "$h %{http_code}\n" -k --resolve $h:443:192.168.2.20 https://$h/
|
||||
done
|
||||
```
|
||||
|
||||
Expect `200`, `200`, `401`.
|
||||
@@ -52,6 +52,13 @@ configMapGenerator:
|
||||
- backup-failure=backup-failure.md
|
||||
- backup-missed=backup-missed.md
|
||||
- backup-partial=backup-partial.md
|
||||
# Operator procedure, not an alert page — nothing links here from an alert. Served because
|
||||
# the whole-cluster shutdown had NO written procedure until 2026-10-17 (infra/cluster.sh is
|
||||
# virsh-based and only ever applied to the decommissioned dev cluster), so the UPS move was
|
||||
# improvised live, and the one hazard it exposed -- a StatefulSet with
|
||||
# persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 --
|
||||
# destroyed monitoring/loki's volume before anyone noticed.
|
||||
- cluster-maintenance-shutdown=cluster-maintenance-shutdown.md
|
||||
- cdi-import-stuck=cdi-import-stuck.md
|
||||
- cert-expiry=cert-expiry.md
|
||||
- cert-issuance=cert-issuance.md
|
||||
@@ -119,11 +126,6 @@ configMapGenerator:
|
||||
- update-rollback=update-rollback.md
|
||||
- vpn-down=vpn-down.md
|
||||
- vpn-egress-stale=vpn-egress-stale.md
|
||||
# `VpnIngressNodeAnnouncesL2VIP` — the INBOUND remote-access VPN, not the egress tunnel the two
|
||||
# pages above cover. A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease
|
||||
# it holds itself, and vpn-ingress MASQUERADEs clients to the node's own address, so the
|
||||
# forwarded node must announce nothing (measured 2026-10-08).
|
||||
- vpn-ingress-l2-collision=vpn-ingress-l2-collision.md
|
||||
# Operator procedure, not an alert page. Served because the component it completes is INERT
|
||||
# until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
|
||||
# and because the one check that actually proves it cannot be run from inside the house.
|
||||
|
||||
@@ -1,165 +0,0 @@
|
||||
# vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP
|
||||
|
||||
**Alerts:** `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) and `VpnIngressNodeLabelMissing`
|
||||
(warning, 30m)
|
||||
**File:** `platform/monitoring/alerts.yaml`
|
||||
**Related:** `/runbooks/vpn-remote-access` is the setup and enrolment runbook for this component.
|
||||
`/runbooks/vpn-egress-stale` and `/runbooks/vpn-down` are the **other** VPN — the outbound PIA
|
||||
tunnel — and have nothing to do with this.
|
||||
|
||||
## What this means here
|
||||
|
||||
**A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
|
||||
announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's
|
||||
own request. On the lease holder the neighbour entry settles at `FAILED`, and every packet to that
|
||||
VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three
|
||||
nodes and five VIPs — each node reached every VIP **except** the ones it alone announced.
|
||||
|
||||
For an ordinary workload this is harmless: pods reach Services through the cluster datapath and
|
||||
never ARP for a VIP. `platform/vpn-ingress` is the exception. It runs WireGuard with
|
||||
`hostNetwork: true` and MASQUERADEs remote clients to the node's own address, so a remote client's
|
||||
packets take that node's LAN path and inherit its ARP failures. **If the node the router forwards
|
||||
`udp/51820` to is also an L2 lease holder, every remote VPN client silently loses all access to the
|
||||
Services behind the VIPs it announces.**
|
||||
|
||||
The node terminating WireGuard is marked with `homelab.keithjackson.dev/vpn-ingress=true`, and
|
||||
`platform-config/cilium/l2-announcement-policy.yaml` excludes that label from announcing anything,
|
||||
so this should be unreachable.
|
||||
|
||||
**The symptom your users report will not sound like a network fault.** On 2026-10-08 the forward sat
|
||||
on `k3s-prod-1`, which held the `shared` Gateway lease, so all 16 Gateway hostnames were dead over
|
||||
the VPN — while PiHole DNS kept answering, because `192.168.2.23` is announced by *two* Services and
|
||||
`k3s-prod-2` held the other lease. So the tunnel handshakes, `wg show` looks perfect, names resolve,
|
||||
and **every app hangs on connect**. Expect "the VPN connects but nothing loads", not "the VPN is
|
||||
down".
|
||||
|
||||
Which Services are affected depends on which lease moved — read `{{ $labels.lease }}` from the
|
||||
alert. `cilium-l2announce-gateway-cilium-gateway-shared` is the worst case: it is every web app at
|
||||
once.
|
||||
|
||||
### If it is `VpnIngressNodeLabelMissing` that fired
|
||||
|
||||
That is the watchdog, and it means the protection is **switched off**, not that something is merely
|
||||
untidy. The exclusion is a `NotIn` on the label, and `NotIn` with an absent key **matches** — so
|
||||
with the label on no node, the policy excludes nothing and every node is an announcement candidate
|
||||
again. `VpnIngressNodeAnnouncesL2VIP` is simultaneously **inert**: its join matches nothing and
|
||||
returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as
|
||||
meaningless until this warning is cleared. Usual cause: a node was rebuilt and `node_baseline` has
|
||||
not been re-run on it.
|
||||
|
||||
## First commands
|
||||
|
||||
Run cluster commands through the pinned toolbox, which supplies the kube context:
|
||||
|
||||
```bash
|
||||
# 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines.
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n kube-system get lease | grep l2announce'
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress'
|
||||
# Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it.
|
||||
|
||||
# 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a
|
||||
# homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels.
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo'
|
||||
|
||||
# 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when
|
||||
# the label is on the node (kube-state-metrics metricLabelsAllowlist).
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \
|
||||
wget -qO- localhost:8080/metrics | grep vpn_ingress | head'
|
||||
```
|
||||
|
||||
**Confirm the actual damage**, which is the one measurement that is unambiguous. From the WireGuard
|
||||
pod on the affected node (it shares the host netns, so this is the path a VPN client takes):
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress get pods -o wide' # pick the pod on that node
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- \
|
||||
sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"'
|
||||
# BROKEN: 192.168.2.20 dev eno1 FAILED
|
||||
# HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY
|
||||
```
|
||||
|
||||
The `ping` is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so
|
||||
**ignore its 100% packet loss**; the neighbour state is the answer. Substitute the VIP that belongs
|
||||
to `{{ $labels.lease }}` if the Gateway is not the one that moved.
|
||||
|
||||
**Which node actually holds the forward** (the router cannot be queried, but the cluster knows):
|
||||
only the forwarded node's `wg show` has a peer with an `endpoint` and a recent `latest handshake`.
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do
|
||||
echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done'
|
||||
```
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
1. **The label is on the wrong node** — someone re-pointed the router's forward during a node
|
||||
failover and did not move `node_baseline_vpn_ingress_node`. This is the most likely cause by a
|
||||
wide margin, because the router is off-repo and nothing can check it. Confirm with the `wg show`
|
||||
loop above: the node with the live peer endpoint is the real forward target.
|
||||
*Fix:* merge a PR setting `node_baseline_vpn_ingress_node`
|
||||
(`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`) to that node, then re-run the role —
|
||||
`ansible-playbook -i inventory/hosts.ini <playbook> -e k3s_hosts=k3s_prod --tags vpn-ingress`.
|
||||
The role labels the new node and **removes** the label from the old one. Then confirm Cilium
|
||||
re-elected the lease off the newly-labelled node.
|
||||
|
||||
2. **The exclusion was edited out of `platform-config/cilium/l2-announcement-policy.yaml`**, or the
|
||||
file was reverted to a bare `matchLabels: {kubernetes.io/os: linux}`. Command 2 above shows this
|
||||
directly. *Fix:* restore the `matchExpressions` form; the file's own header explains why it is
|
||||
shaped that way.
|
||||
|
||||
3. **The label is missing entirely** (`VpnIngressNodeLabelMissing` firing) — a node rebuild without
|
||||
a `node_baseline` re-run. *Fix:* re-run the role as above.
|
||||
|
||||
4. **The label is on the node but not in Prometheus** — `metricLabelsAllowlist` in
|
||||
`platform/monitoring/kube-prometheus-stack.yaml` lost the `vpn-ingress` entry. Then the critical
|
||||
alert is inert *and* the watchdog fires. Command 3 above distinguishes this from cause 3.
|
||||
|
||||
5. **Cilium kept renewing a lease its holder is no longer eligible for.** The exclusion is correct
|
||||
and in place, but the lease never re-elected. This was never verified when the fix was authored
|
||||
(see `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md`). *Fix:* delete that one Lease
|
||||
object to force an election — `kubectl -n kube-system delete lease <name>` — and watch it come
|
||||
back on a different node.
|
||||
|
||||
6. **A Cilium upgrade changed `NotIn` semantics or the `nodeSelector` field.** Least likely, but it
|
||||
is the reason this is a runtime alert rather than a static check on the policy text.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
**Act now, don't wait for a window.** Remote access is fully broken for the affected VIPs while this
|
||||
is firing, and it does **not** self-heal: a lease holder keeps renewing, so this condition persists
|
||||
indefinitely once entered. That is also why the 15m `for:` is safe — a transient move during a node
|
||||
reboot or a Cilium roll resolves on its own well inside it.
|
||||
|
||||
**The fastest honest mitigation is to move the forward, not the leases.** Re-pointing the router's
|
||||
`udp/51820` at a node that holds no lease for the affected VIP restores service in seconds, with no
|
||||
cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the
|
||||
`node_baseline_vpn_ingress_node` PR and role re-run so the two agree again; until you do, this alert
|
||||
is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08,
|
||||
prod-2 announced crafty `192.168.2.22` and unifi `192.168.2.21`, prod-3 announced plex
|
||||
`192.168.2.24`, so neither is a cost-free target.
|
||||
|
||||
**Do not** `kubectl edit` the `CiliumL2AnnouncementPolicy` to fix this — Flux owns it and will revert
|
||||
you, and the drift is the kind that outlives the memory of making it. The two acceptable hand
|
||||
actions are deleting a single Lease to force re-election (cause 5), and `kubectl label node` as a
|
||||
*temporary* stopgap while the PR lands — note that a node rebuild or a `node_baseline` run will undo
|
||||
a hand-applied label, so it is not a fix.
|
||||
|
||||
**Stop and escalate to the operator if** the leases will not leave the excluded node after a forced
|
||||
re-election, or if the intended forward target is unclear. Both need a decision about which node
|
||||
carries remote access, which is an operator call, not a runbook step.
|
||||
|
||||
**Known gap, so you are not misled:** if someone re-points the router without moving the label,
|
||||
`VpnIngressNodeAnnouncesL2VIP` watches the wrong node and **stays silent** while the VPN breaks in
|
||||
exactly the way described above. There is no in-cluster signal for that today; closing it needs a
|
||||
WireGuard exporter publishing which node is actually terminating peer traffic
|
||||
(`platform/vpn-ingress/README.md`, "Not built") — the `wg show` loop above is the manual substitute.
|
||||
Until then, if remote access is broken and both alerts are quiet, **check the forward against the
|
||||
label by hand before believing the alerts.**
|
||||
@@ -171,57 +171,14 @@ server without a `wg0` "ready".
|
||||
On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
|
||||
`192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.
|
||||
|
||||
> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node.
|
||||
>
|
||||
> This step used to say "pick any node — all three run an identical listener". **That sentence
|
||||
> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a
|
||||
> cluster-vars knob instead of a free choice here.
|
||||
>
|
||||
> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
|
||||
> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease
|
||||
> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they
|
||||
> leave the node. Measured on prod 2026-10-08:
|
||||
>
|
||||
> ```text
|
||||
> on k3s-prod-1 (held the shared Gateway lease): ip neigh show 192.168.2.20 -> FAILED
|
||||
> on k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr … (prod-1)
|
||||
> ```
|
||||
>
|
||||
> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN
|
||||
> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway
|
||||
> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so
|
||||
> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported
|
||||
> it.
|
||||
>
|
||||
`infra/ansible/roles/node_baseline` now labels the forwarded node
|
||||
> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in
|
||||
> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes
|
||||
> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI
|
||||
> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a
|
||||
> labelled node taking a lease, but it CANNOT catch the forward being moved without the label.
|
||||
|
||||
Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`).
|
||||
Note it here:
|
||||
Pick any node — all three run an identical listener. Note which one you chose here:
|
||||
|
||||
```
|
||||
# WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set <date>
|
||||
# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml
|
||||
```
|
||||
|
||||
**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that
|
||||
was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and
|
||||
reintroduces the failure above against a different set of addresses (prod-2 announces crafty and
|
||||
unifi; prod-3 announces plex). Do both, in this order:
|
||||
|
||||
1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role
|
||||
(`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium
|
||||
then moves the leases off the newly-labelled node and onto its peers.
|
||||
2. **Re-point the router** forward at the new node's address.
|
||||
|
||||
Every device keeps working with the same profile either way; all three nodes answer the same keys,
|
||||
and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that
|
||||
handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because
|
||||
that alert watches the node the knob names, not the node your router actually points at.
|
||||
**Node loss = failover is this one line.** Re-point the forward at another node address; every
|
||||
device keeps working with the same profile, because all three nodes answer the same keys.
|
||||
|
||||
Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
|
||||
already silent to unauthenticated probes (README, "Why this is safe to expose").
|
||||
|
||||
Reference in new issue
Block a user