Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
2f2e98aa19 |
No files matched your search
@@ -0,0 +1,106 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: feat/graceful-cluster-shutdown
|
||||
title: "Graceful node shutdown was never configured, and scaling a StatefulSet deleted its PVC"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
All three nodes had to be powered off to move them onto a UPS. There was **no procedure for it**:
|
||||
`infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev cluster, so
|
||||
the whole sequence was improvised live. Two defects came out of doing it.
|
||||
|
||||
**1. Kubelet graceful node shutdown was unconfigured on every prod node.** No `shutdownGracePeriod`
|
||||
anywhere in `infra/ansible/`, so `shutdown -h` stopped k3s, containerd went with it, and every pod
|
||||
was SIGKILLed where it stood — 9 CloudNativePG instances mid-checkpoint, the UniFi VM, and 25
|
||||
attached Longhorn volumes. `roles/k3s_server` now writes `kubelet-arg` into
|
||||
`/etc/rancher/k3s/config.yaml` with `shutdown-grace-period=90s` and
|
||||
`shutdown-grace-period-critical-pods=20s`.
|
||||
|
||||
The half that is easy to miss and silently defeats the other half: **graceful node shutdown is
|
||||
enforced by logind, not by the kubelet.** The kubelet holds an inhibitor *delay* lock and systemd
|
||||
waits at most `InhibitDelayMaxSec` — **default 5 seconds** — before proceeding regardless. A grace
|
||||
period above that cap is truncated with no error, so the feature looks configured and still
|
||||
hard-kills pods. The role now ships a `logind.conf.d` drop-in setting it to 120, and **asserts**
|
||||
`k3s_logind_inhibit_delay_max_sec >= k3s_kubelet_shutdown_grace_period` rather than documenting the
|
||||
relationship, because the failure is invisible.
|
||||
|
||||
**2. Scaling a StatefulSet to 0 DELETED its PVC and Longhorn volume.** `monitoring/loki` carries
|
||||
`persistentVolumeClaimRetentionPolicy: {whenDeleted: Delete, whenScaled: Delete}`, so the quiesce's
|
||||
`scale --replicas=0` destroyed `storage-loki-0` outright. Loki stores with `type: filesystem`, so
|
||||
that PVC was the only copy of its log data. This is the defect in this change set that actually
|
||||
destroyed something, and it was mine.
|
||||
|
||||
New `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`) encodes the sequence that
|
||||
worked, and **refuses to scale any StatefulSet whose `whenScaled` is `Delete`** — it leaves it
|
||||
running, accepts that its volume hard-detaches at poweroff, and prints which volumes to expect
|
||||
`degraded`. A rebuild is a vastly better outcome than a deletion. Workload discovery is computed
|
||||
from the live cluster (every Deployment/StatefulSet whose pods mount a PVC, Longhorn or TrueNAS NFS)
|
||||
rather than hardcoded, so a newly onboarded service is covered without anyone remembering to add it.
|
||||
|
||||
`runbooks/cluster-maintenance-shutdown.md` (new, registered in `runbooks/kustomization.yaml`) is the
|
||||
operator page: shutdown order, the TrueNAS-after-nodes constraint, power-on order, the
|
||||
un-hibernate-Postgres step Flux will not do for you, and a "what looks broken but is fine" section
|
||||
separating the `4/19`-Kustomizations-Ready cascade, the degraded-and-rebuilding volume and the
|
||||
self-recovering `keycloak-operator` from the one thing that genuinely needs a human (a stuck
|
||||
`oauth2-proxy` replica).
|
||||
|
||||
## What I verified
|
||||
|
||||
* **Graceful shutdown was genuinely absent**, not merely undocumented: `grep -rn
|
||||
'shutdownGracePeriod|graceful-node-shutdown'` over `infra/ansible/` returns nothing, and no
|
||||
`logind` or `InhibitDelay` setting exists anywhere in the tree.
|
||||
* **The whole shutdown/restart cycle was executed end to end on prod 2026-10-17** and the cluster
|
||||
came back: 3/3 nodes Ready with etcd `healthz` ok, 19/19 Flux Kustomizations Ready,
|
||||
`platform-config`'s `lastAppliedRevision` equal to the GitRepository artifact revision, all 3
|
||||
CNPG clusters `healthy` with primaries elected, the UniFi VM `Running`, PiHole resolving
|
||||
(`home.keithjackson.dev` → `192.168.2.20` via `192.168.2.23`), and the Gateway serving
|
||||
`200/200/401` for apex/gitea/plex.
|
||||
* **The PVC-deletion mechanism is confirmed, not inferred:** `kubectl get sts loki -o
|
||||
jsonpath={.spec.persistentVolumeClaimRetentionPolicy}` returns
|
||||
`{"whenDeleted":"Delete","whenScaled":"Delete"}`, the PVC was `NotFound` after the cycle, and the
|
||||
Longhorn volume count had gone 25 → 24.
|
||||
* **The damage was repaired and verified:** `scale sts loki --replicas=1` (restoring the replica
|
||||
count the chart declares) recreated `storage-loki-0` as a fresh 50Gi Bound PVC, `loki-0` reached
|
||||
`2/2 Running`, and the volume count returned to 25.
|
||||
* **The guard works against the live cluster**, tested both ways: `sts_deletes_pvc_on_scale
|
||||
monitoring loki` returns true (would be skipped) and `pihole pihole` returns false (safe to
|
||||
scale).
|
||||
* **The script's read-only paths run clean on prod:** `status` reports correctly, and the discovery
|
||||
function returns the 13 workloads currently holding a PVC.
|
||||
* `bash -n` clean; the two edited Ansible YAML files parse.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **The kubelet change has NEVER BEEN APPLIED OR TESTED.** This is the significant gap. I did not
|
||||
run the role, did not restart k3s, and have **not** observed a node actually drain on
|
||||
`shutdown -h`. Everything about it is reasoning from kubelet/systemd semantics plus the measured
|
||||
absence of the setting. Applying it needs a **rolling** k3s restart (`serial: 1` — kube-vip keeps
|
||||
the API up, etcd holds at 2/3), which is an operator action (CLAUDE.md §5). **Verify by draining
|
||||
one node for real:** apply, then `shutdown -h now` on a single node and watch whether pods
|
||||
terminate gracefully instead of vanishing.
|
||||
* **Whether 90s/20s are the right numbers.** Chosen to comfortably cover a Postgres fast shutdown
|
||||
and a Longhorn flush; not measured against either. The budget is spent on every reboot, so if a
|
||||
node takes noticeably longer to go down, this is why.
|
||||
* **Whether k3s accepts `kubelet-arg` in `config.yaml` as written.** The file already carries
|
||||
`kube-apiserver-arg` as a list and `kubelet-arg` is the documented sibling, but I did not render
|
||||
or apply it, so a typo would surface only at k3s restart — and a bad `config.yaml` is exactly the
|
||||
kind of thing that makes k3s fail to come up. **Review this one before applying.**
|
||||
* **The `assert` task's `regex_replace` on the grace period** assumes a plain `90s`-style value; a
|
||||
value expressed as `1m30s` would parse to the nonsense integer `130`. Fine for the committed
|
||||
defaults, wrong if someone writes a compound duration.
|
||||
* **`resume`'s Flux-reconcile wait has never run**, and neither has `quiesce` as a script — the
|
||||
2026-10-17 cycle was done with the equivalent commands by hand, so the script encodes a sequence
|
||||
that worked but has not itself been executed end to end. Only `status` and the discovery and guard
|
||||
functions have run.
|
||||
* **Loki's lost log window is gone for good.** Up to 72h of logs (its `retention_period`), destroyed
|
||||
by my scale-down. Nothing recovers it; Velero's schedules cover the namespace but the PVC was
|
||||
deleted, not corrupted, and I did not attempt a restore because the data self-expires anyway.
|
||||
* **No other `whenScaled: Delete` StatefulSet was audited.** The guard now catches them generically,
|
||||
but I did not enumerate which other workloads carry the policy, so I cannot say loki was the only
|
||||
one at risk. The runbook gives the one-liner to list them.
|
||||
* **TrueNAS and the router are entirely outside this change.** TrueNAS is not in the Ansible
|
||||
inventory and refuses SSH, so its shutdown stays a web-UI step with no automation and no check.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1
|
||||
offline subset only.
|
||||
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
|
||||
@@ -1,99 +0,0 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: fix/oauth2-proxy-oidc-startup-race
|
||||
title: "oauth2-proxy dies on a cold start because OIDC discovery runs once, before Keycloak serves"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Exposed by the 2026-10-17 whole-cluster shutdown for the UPS move. oauth2-proxy performs OIDC
|
||||
discovery **once, at startup**, and `os.Exit(1)`s if it fails. On a cold boot Keycloak is not
|
||||
serving yet, so the shared Gateway answers 503 and the proxy dies:
|
||||
|
||||
```
|
||||
[provider.go:55] Performing OIDC Discovery...
|
||||
[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: error initialising provider: ...
|
||||
failed to discover OIDC configuration: unexpected status "503": no healthy upstream
|
||||
```
|
||||
|
||||
It then sits in **exponential CrashLoopBackOff long after Keycloak is healthy** — one of the two
|
||||
replicas was still down minutes later and only a manual `kubectl delete pod` cleared it. There is no
|
||||
retry flag; the process does not survive a cold start. A readinessProbe cannot help (the container
|
||||
is already dead), and CrashLoopBackOff backoff is not tunable per-pod.
|
||||
|
||||
`platform/oauth2-proxy/helmrelease.yaml` gains an `extraInitContainers` gate (`wait-for-idp`) that
|
||||
polls Keycloak's discovery doc until it answers, then lets the proxy start. A pod in `Init` does not
|
||||
accrue CrashLoopBackOff backoff and retries promptly, which is the point.
|
||||
|
||||
**Bounded, and fails open.** After ~5 minutes it gives up and starts the proxy anyway, so the worst
|
||||
case is exactly today's behaviour and never something new. An unbounded wait would trade a CrashLoop
|
||||
for a pod wedged in `Init` forever, which is not an improvement.
|
||||
|
||||
## Why this shape and not the alternatives
|
||||
|
||||
* **`extraInitContainers` is the chart's own documented hook for precisely this.** Its commented
|
||||
example in `values.yaml` is literally `- name: wait-for-idp`. The key was **confirmed against the
|
||||
pinned chart** (`helm show values oauth2-proxy/oauth2-proxy --version 10.7.1` → `extraInitContainers: []`
|
||||
at line 428), not assumed — a wrong Helm values key is silently ignored, which is the
|
||||
"reviewed, merged, apparently done, still broken" shape this repo keeps getting bitten by.
|
||||
* **No new base image.** The chart already runs an `alpine` init container for its built-in
|
||||
`initContainers.waitForRedis` gate, so alpine is a base this chart uses regardless. Pinned to
|
||||
`3.22` (verified to exist upstream) rather than the chart's own `latest`.
|
||||
* **Flux `postRenderers` rejected:** it would work independently of the chart schema, but nothing in
|
||||
this tree uses postRenderers, so it would introduce a mechanism with no precedent for a problem
|
||||
the chart already has a hook for.
|
||||
* **`--skip-oidc-discovery` rejected:** it removes the startup fetch entirely, which is arguably
|
||||
more elegant, but it means hardcoding four Keycloak endpoints and I could not verify offline that
|
||||
oauth2-proxy then fetches JWKS lazily rather than at startup. oauth2-proxy gates SSO for PiHole,
|
||||
the \*arr apps, qBittorrent, Homepage, Prometheus, Alertmanager and Hubble — a wrong guess locks
|
||||
the operator out of all of them, so the reversible, additive option wins.
|
||||
|
||||
The `$$`-free convention matters here: shell variables are written bare (`$i`, `$url`, `$(( ))`)
|
||||
with **no braced form anywhere**, because `platform` declares `postBuild.substituteFrom cluster-vars`
|
||||
and envsubst consumes braced expansions — a braced shell variable renders to the empty string and
|
||||
silently breaks the loop. `${BASE_DOMAIN}` is braced on purpose. Same hazard and same convention as
|
||||
the gitea and nextcloud OIDC bootstrap hooks, and `ci: flux-postbuild-substitution` enforces it.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **The failure is measured, not theorised.** The log above is from the live pod
|
||||
(`oauth2-proxy-5d6c74876b-bzb2p`, `lastState.terminated: {reason: Error, exitCode: 1}`), captured
|
||||
with `--previous` after the cold boot. The surviving replica's log shows the same startup
|
||||
succeeding once Keycloak was up, which is what identifies this as a race rather than a config
|
||||
fault.
|
||||
* **Deleting the pod is a sufficient workaround** — done on prod 2026-10-17, the replacement came up
|
||||
clean and both replicas have been healthy since.
|
||||
* **The chart key is real**, at the pinned version range: `helm search repo` resolves
|
||||
`>=10.0.0 <11.0.0` to chart `10.7.1` (app `7.15.5`), whose values carry `extraInitContainers: []`,
|
||||
`extraContainers: []` and the built-in `initContainers.waitForRedis` (which uses `alpine`).
|
||||
* **`docker.io/library/alpine:3.22` exists** — confirmed against the registry tag list rather than
|
||||
assumed from memory.
|
||||
* The edited HelmRelease parses as YAML.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **UNAPPLIED. The init container has never run.** I did not reconcile the HelmRelease, so I have not
|
||||
seen `wait-for-idp` execute, exit 0, or gate the proxy. Applying it rolls both oauth2-proxy
|
||||
replicas, which is a mutating action (CLAUDE.md §5) — and it rolls the component that gates SSO for
|
||||
seven apps, so **do it when you can watch it**, not unattended.
|
||||
* **I did not re-create the cold-start race to prove the fix.** That needs Keycloak down while
|
||||
oauth2-proxy starts; the honest test is the next whole-cluster boot. Until then this is reasoned
|
||||
from the measured failure, not demonstrated against it.
|
||||
* **busybox `wget`'s TLS behaviour here is assumed, not tested.** The claim is that alpine's
|
||||
`ca-certificates-bundle` lets it validate the Gateway's LE-prod certificate, so a genuine TLS
|
||||
failure exits non-zero rather than passing falsely. If that is wrong in either direction the gate
|
||||
degrades safely — a never-succeeding probe just burns the 5-minute budget and then starts the proxy
|
||||
anyway — but the fast path would be silently dead and this would be doing nothing. **Check the
|
||||
init container's log on first apply**: it should print "Keycloak discovery reachable" within
|
||||
seconds on a warm cluster, not the give-up message.
|
||||
* **Whether a pod can reach the Gateway hostname at all from this namespace is inherited, not
|
||||
re-proven.** The main container does exactly this fetch today and succeeds, so the path is known
|
||||
good — but note the standing trap that an arbitrary pod gets 403 from every Gateway hostname
|
||||
(`decisions/2026-10-04-plex-advertise-plexdirect.md`); oauth2-proxy is exempt only because
|
||||
Keycloak's ingress policy admits it. An init container in the same pod inherits that identity, so
|
||||
this should hold — unverified.
|
||||
* **60 attempts x 5s is not tuned.** It is "comfortably longer than Keycloak's observed cold start",
|
||||
not a measured bound.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is unset, so CI proves the L0/L1 offline
|
||||
subset only.
|
||||
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
|
||||
@@ -1,198 +0,0 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: fix/vpn-ingress-l2-lease-collision
|
||||
title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it
|
||||
was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is
|
||||
not re-derived — it is very easy to re-derive from `ip neigh` alone.
|
||||
|
||||
### The claim, and how it died
|
||||
|
||||
The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2
|
||||
announcement lease it holds itself* — the announcer does not answer its own ARP request, the
|
||||
neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because
|
||||
`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
|
||||
own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded
|
||||
that label from announcing, with two alerts and four promtool tests behind it.
|
||||
|
||||
The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*`
|
||||
leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) —
|
||||
the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five
|
||||
VIPs and the other two nodes should reach **all**.
|
||||
|
||||
**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node:
|
||||
|
||||
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|
||||
|---|---|---|
|
||||
| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
|
||||
| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms |
|
||||
| `nc -z 192.168.2.20 443` | **OK** | OK |
|
||||
| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` |
|
||||
| `192.168.2.21:8443` | FAIL | **FAIL** (both) |
|
||||
|
||||
The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the
|
||||
asserted **effect is absent**. That is sufficient to falsify the causal claim without having to
|
||||
explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**.
|
||||
|
||||
### Why it is inert
|
||||
|
||||
No traffic path resolves a LoadBalancer VIP by ARP:
|
||||
|
||||
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
|
||||
`Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath
|
||||
is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP
|
||||
is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**.
|
||||
* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's
|
||||
backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet.
|
||||
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
|
||||
which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a
|
||||
bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT,
|
||||
not wire traffic.
|
||||
|
||||
### The two probes that misled me
|
||||
|
||||
Both were cited as evidence in the previous version of this entry.
|
||||
|
||||
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing
|
||||
artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong.
|
||||
* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a
|
||||
host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including
|
||||
the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it
|
||||
discriminates nothing.
|
||||
|
||||
**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then
|
||||
routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns
|
||||
probe cannot measure the forwarded path in either direction — it reports success where the forwarded
|
||||
path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not
|
||||
sharing a hook, and that is the assumption the original diagnosis rested on.
|
||||
|
||||
### Files
|
||||
|
||||
Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`,
|
||||
`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`,
|
||||
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`,
|
||||
`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted:
|
||||
`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`.
|
||||
|
||||
Kept, rewritten from scratch:
|
||||
|
||||
* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is
|
||||
inert and which probes lie, and the internet-egress facts below.
|
||||
* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster
|
||||
work, internet does not") that works outwards from the client profile, and a short section saying
|
||||
the L2 lease is a dead end, with the unifi `.21` note.
|
||||
|
||||
**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is
|
||||
proposed by this branch any more.**
|
||||
|
||||
### The internet-routing report: no server-side fix was needed, and none was made
|
||||
|
||||
The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full
|
||||
one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has
|
||||
emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`,
|
||||
2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and
|
||||
`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel
|
||||
feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that
|
||||
already exists. **I deliberately shipped no code change here** — writing one would have meant
|
||||
inventing a fix for a symptom I could not reproduce (see below).
|
||||
|
||||
The server side was measured end to end and is capable and correct, so the cause is in the client
|
||||
profile or the client. The triage section names the two candidates that fit the symptom exactly —
|
||||
a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and
|
||||
leak public traffic outside the tunnel — each with a one-line test that settles it.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on
|
||||
2026-10-17:
|
||||
|
||||
* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1`
|
||||
(`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again.
|
||||
* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a
|
||||
real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere.
|
||||
* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three
|
||||
nodes — so the successes are service-aware DNAT, not wire delivery.
|
||||
* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with
|
||||
`REACHABLE` neighbour entries. This is the measurement that destroys the original control.
|
||||
* **The datapath facts** quoted above from `cilium-dbg status --verbose` and
|
||||
`cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`:
|
||||
`kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`,
|
||||
`routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three.
|
||||
* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` →
|
||||
`dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP.
|
||||
So the 200 is not loopback delivery by the host stack — it is BPF.
|
||||
* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1`
|
||||
(read, never written); `iptables -t nat -S POSTROUTING` carries
|
||||
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0
|
||||
packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both
|
||||
present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it;
|
||||
`curl http://1.1.1.1/` returns `301` from each node.
|
||||
* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by
|
||||
`podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod
|
||||
`egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on
|
||||
`-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the
|
||||
home WAN IP, not PIA's.
|
||||
* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned
|
||||
`142.250.113.138`.
|
||||
* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one
|
||||
enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both
|
||||
`FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The
|
||||
`wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters.
|
||||
* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f`
|
||||
(2026-10-05) and never changed.
|
||||
* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could
|
||||
not test what a real remote client does, and — as established above — **no probe runnable from the
|
||||
pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a
|
||||
VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a
|
||||
LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one
|
||||
remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in
|
||||
the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this
|
||||
unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show`
|
||||
shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the
|
||||
client loads a cluster app — and repeat it on a node that holds leases and one that does not.*
|
||||
* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same
|
||||
reason. **I need a client connected** to learn it; the router is unreadable from the cluster.
|
||||
* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP
|
||||
failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes.
|
||||
My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**,
|
||||
rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A
|
||||
competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and
|
||||
`wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is
|
||||
needed for the falsification** — the asserted cause is present today without its effect. I did not
|
||||
retro-measure anything, and I did not read Cilium's source or issue tracker.
|
||||
* **The operator's report cannot be a description of "right now."** No client has connected since
|
||||
the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster
|
||||
services and the LAN but not the internet" must describe a session before the power-cycle or one
|
||||
that has since dropped. **I did not establish when it was observed, or with which profile.**
|
||||
* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not**
|
||||
find the fault, and I shipped no fix for it. The two candidates in the triage section (a
|
||||
`--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not
|
||||
measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line
|
||||
decides split vs full and halves the search — and a connected client for step 2.
|
||||
* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and
|
||||
lives only on the operator's machine and the device; it is not in the repo, and I did not look for
|
||||
it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing.
|
||||
* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without
|
||||
a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU
|
||||
would see large transfers hang while small requests succeed. The runbook's existing MTU note
|
||||
(drop to 1280) already covers the remedy.
|
||||
* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole
|
||||
IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a
|
||||
client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an
|
||||
interface with no IPv6 address.
|
||||
* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate,
|
||||
pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1
|
||||
offline subset only. Nothing above is covered by a test that runs in CI.
|
||||
* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there
|
||||
is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating
|
||||
`kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material).
|
||||
* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).
|
||||
@@ -107,3 +107,31 @@ k3s_base_domain: "home.keithjackson.dev"
|
||||
|
||||
# Derived, so the realm path is stated once. Override k3s_base_domain, not this.
|
||||
k3s_oidc_issuer_url: "https://keycloak.{{ k3s_base_domain }}/realms/homelab"
|
||||
|
||||
# --- Graceful node shutdown (kubelet) ----------------------------------------------------------
|
||||
# MEASURED 2026-10-17, during the UPS move: graceful node shutdown was NOT configured on any prod
|
||||
# node, so `shutdown -h` did NOT drain pods. systemd stops k3s, containerd goes with it, and every
|
||||
# pod is SIGKILLed where it stands -- including 9 CloudNativePG instances mid-checkpoint, the UniFi
|
||||
# VM, and 25 attached Longhorn volumes. The whole-cluster shutdown that day had to quiesce
|
||||
# everything BY HAND first (runbooks/cluster-maintenance-shutdown.md), and the one volume that did
|
||||
# not finish detaching came back `degraded` and had to rebuild.
|
||||
#
|
||||
# With these set, the kubelet takes a systemd inhibitor lock and, on shutdown, evicts ordinary pods
|
||||
# first and critical ones last, so a plain `shutdown -h`/reboot drains the node. That makes the
|
||||
# manual scale-down in the runbook a belt-and-braces step for a PLANNED whole-cluster shutdown
|
||||
# rather than the only thing standing between a reboot and a hard kill.
|
||||
#
|
||||
# 90s/20s, not larger: the budget is spent on every reboot and node upgrade, and the kubelet waits
|
||||
# the FULL ordinary period before it starts on critical pods. Postgres does a fast shutdown in
|
||||
# well under 90s here, and Longhorn needs only enough time to flush and detach.
|
||||
k3s_kubelet_shutdown_grace_period: "90s"
|
||||
k3s_kubelet_shutdown_grace_period_critical_pods: "20s"
|
||||
|
||||
# *** THE HALF THAT IS EASY TO MISS, AND SILENTLY BREAKS THE OTHER HALF. ***
|
||||
# Graceful node shutdown works by the kubelet holding a logind INHIBITOR DELAY lock. systemd caps
|
||||
# how long any inhibitor may delay shutdown at `InhibitDelayMaxSec`, which DEFAULTS TO 5 SECONDS.
|
||||
# If that cap is below the grace period above, systemd proceeds anyway and the drain is truncated
|
||||
# to ~5s -- i.e. the feature appears configured, reports no error, and still hard-kills pods. So
|
||||
# this MUST be >= k3s_kubelet_shutdown_grace_period; the role asserts it rather than trusting it.
|
||||
# Set a little above the kubelet's total (90s + 20s) to leave headroom.
|
||||
k3s_logind_inhibit_delay_max_sec: 120
|
||||
@@ -15,3 +15,12 @@
|
||||
port: "{{ k3s_api_port }}"
|
||||
timeout: 180
|
||||
listen: Restart k3s
|
||||
|
||||
# logind re-reads InhibitDelayMaxSec only on restart. Restarting logind does NOT log users out or
|
||||
# disturb k3s -- it manages login sessions and inhibitor locks, not workloads. The kubelet picks up
|
||||
# the new cap when it next takes its lock, so no k3s restart is needed for THIS half.
|
||||
- name: Restart logind
|
||||
ansible.builtin.systemd_service:
|
||||
name: systemd-logind
|
||||
state: restarted
|
||||
listen: Restart logind
|
||||
@@ -86,6 +86,16 @@
|
||||
# k3s_base_domain (defaults/main.yml), whose [parity] contract with the cluster's
|
||||
# BASE_DOMAIN is documented there. Dev's default renders the value it already runs, so
|
||||
# config.yaml is unchanged and the Restart handler does not fire.
|
||||
# ── Graceful node shutdown ──────────────────────────────────────────────────────────────
|
||||
# Without these the kubelet takes no systemd inhibitor lock, so `shutdown -h` stops
|
||||
# containerd and SIGKILLs every pod where it stands -- MEASURED on all three prod nodes
|
||||
# 2026-10-17 (defaults/main.yml has the detail). The companion logind drop-in below raises
|
||||
# InhibitDelayMaxSec, WITHOUT WHICH systemd caps the drain at ~5s and these values are
|
||||
# quietly ignored.
|
||||
kubelet-arg:
|
||||
- "shutdown-grace-period={{ k3s_kubelet_shutdown_grace_period }}"
|
||||
- "shutdown-grace-period-critical-pods={{ k3s_kubelet_shutdown_grace_period_critical_pods }}"
|
||||
|
||||
kube-apiserver-arg:
|
||||
- "oidc-issuer-url={{ k3s_oidc_issuer_url }}"
|
||||
- "oidc-client-id=kubernetes"
|
||||
@@ -183,3 +193,46 @@
|
||||
host: "127.0.0.1"
|
||||
port: "{{ k3s_api_port }}"
|
||||
timeout: 180
|
||||
|
||||
# ── Graceful node shutdown: the systemd half — tag `shutdown` ──────────────────────────────────
|
||||
# The kubelet's shutdown-grace-period is ENFORCED BY logind, not by the kubelet: the kubelet holds
|
||||
# an inhibitor DELAY lock, and systemd will wait at most `InhibitDelayMaxSec` for it (default 5s)
|
||||
# before proceeding regardless. A grace period above that cap is therefore silently truncated --
|
||||
# the feature looks configured, logs no error, and still hard-kills pods. Assert the relationship
|
||||
# instead of documenting it, because the failure is invisible.
|
||||
- name: Assert the logind inhibitor cap can actually cover the kubelet grace period
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- (k3s_logind_inhibit_delay_max_sec | int) >= (k3s_kubelet_shutdown_grace_period | regex_replace('[^0-9]', '') | int)
|
||||
fail_msg: >-
|
||||
k3s_logind_inhibit_delay_max_sec ({{ k3s_logind_inhibit_delay_max_sec }}s) is below
|
||||
k3s_kubelet_shutdown_grace_period ({{ k3s_kubelet_shutdown_grace_period }}). systemd would
|
||||
stop waiting for the kubelet's inhibitor lock before the drain finished, so pods would be
|
||||
SIGKILLed anyway and graceful node shutdown would be configured but ineffective.
|
||||
quiet: true
|
||||
tags: [shutdown]
|
||||
|
||||
- name: Ensure /etc/systemd/logind.conf.d exists
|
||||
ansible.builtin.file:
|
||||
path: /etc/systemd/logind.conf.d
|
||||
state: directory
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0755"
|
||||
tags: [shutdown]
|
||||
|
||||
# A drop-in, not an edit of logind.conf: the distro file stays pristine and `state: absent` here
|
||||
# would fully revert the change.
|
||||
- name: Raise the logind inhibitor delay cap so the kubelet can finish draining
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/systemd/logind.conf.d/10-k3s-graceful-shutdown.conf
|
||||
owner: root
|
||||
group: root
|
||||
mode: "0644"
|
||||
content: |
|
||||
# Managed by Ansible (roles/k3s_server). See defaults/main.yml.
|
||||
# MUST stay >= kubelet shutdown-grace-period, or the drain is truncated to this value.
|
||||
[Login]
|
||||
InhibitDelayMaxSec={{ k3s_logind_inhibit_delay_max_sec }}
|
||||
notify: Restart logind
|
||||
tags: [shutdown]
|
||||
Executable
+265
@@ -0,0 +1,265 @@
|
||||
#!/usr/bin/env bash
|
||||
# =================================================================================================
|
||||
# quiesce.sh — bring the prod cluster to a safe standstill before powering nodes off, and bring it
|
||||
# back afterwards. Companion to runbooks/cluster-maintenance-shutdown.md.
|
||||
# =================================================================================================
|
||||
# WHY THIS EXISTS. On 2026-10-17 all three nodes had to be powered off to move them onto a UPS, and
|
||||
# there was NO procedure for it: `infra/cluster.sh stop` is `virsh`-based and only ever applied to
|
||||
# the decommissioned dev cluster. The whole sequence was improvised live. This script is that
|
||||
# sequence, so the next person does not improvise it.
|
||||
#
|
||||
# WHY IT IS NOT JUST `shutdown -h`. Graceful node shutdown was unconfigured on every prod node that
|
||||
# day, so systemd stopped containerd and SIGKILLed every pod where it stood. roles/k3s_server now
|
||||
# sets shutdown-grace-period (+ the logind InhibitDelayMaxSec cap that actually enforces it), which
|
||||
# makes a bare reboot safe. This script is still the right tool for a PLANNED whole-cluster
|
||||
# shutdown, for three reasons the kubelet's drain does not cover:
|
||||
# 1. The UniFi VM needs an ACPI guest shutdown, not an eviction.
|
||||
# 2. CloudNativePG wants hibernation, not pod deletion -- and hibernation is IMPERATIVE state
|
||||
# that Flux will not set or clear for you.
|
||||
# 3. Draining three nodes one at a time makes pods RESCHEDULE onto the survivors, churning
|
||||
# Longhorn replicas for no reason when the whole cluster is going down anyway. Scaling in
|
||||
# place avoids all of it.
|
||||
#
|
||||
# SAFE TO RE-RUN. Every step is idempotent and `resume` is the exact inverse of `quiesce`.
|
||||
#
|
||||
# WHAT IT DELIBERATELY DOES NOT DO: power anything off. Node poweroff is an operator action
|
||||
# (CLAUDE.md §5) and needs the become password; the runbook gives the ansible one-liner. It also
|
||||
# does not touch TrueNAS (not in the inventory, SSH refused -- web UI only) or the router.
|
||||
set -euo pipefail
|
||||
|
||||
CONTEXT="${CONTEXT:-k3s-prod}"
|
||||
KUBECTL=(kubectl --context "$CONTEXT" --request-timeout=30s)
|
||||
TIMEOUT="${TIMEOUT:-600}"
|
||||
|
||||
# The three CloudNativePG clusters, as namespace/name. Hibernation is per-Cluster.
|
||||
CNPG=(keycloak/keycloak-db nextcloud-db/nextcloud-db semaphore-db/semaphore-db)
|
||||
|
||||
log() { printf '\033[34m[quiesce]\033[0m %s\n' "$*"; }
|
||||
ok() { printf '\033[32m[ ok ]\033[0m %s\n' "$*"; }
|
||||
warn() { printf '\033[33m[warn]\033[0m %s\n' "$*"; }
|
||||
die() { printf '\033[31m[fail]\033[0m %s\n' "$*" >&2; exit 1; }
|
||||
|
||||
need() { command -v "$1" >/dev/null || die "$1 not on PATH — run this inside ./infra/devcontainer/dev.sh"; }
|
||||
|
||||
# ---------------------------------------------------------------- discovery
|
||||
# Every Deployment/StatefulSet whose pods mount a PVC — Longhorn OR the TrueNAS NFS bulk tier.
|
||||
# Computed from the LIVE cluster rather than hardcoded, so a newly onboarded service is covered
|
||||
# without anyone remembering to add it here. PROTECTED namespaces are excluded because scaling
|
||||
# them down would remove the very machinery that performs the detach (longhorn-system), keep the
|
||||
# API reachable (kube-system), or finish the hibernation and the VM shutdown (cnpg-system,
|
||||
# kubevirt).
|
||||
pvc_workloads() {
|
||||
"${KUBECTL[@]}" get pods -A -o json | python3 -c '
|
||||
import json,sys
|
||||
PROTECT={"longhorn-system","kube-system","cilium-secrets","cnpg-system","kubevirt","flux-system","cdi"}
|
||||
out={}
|
||||
for p in json.load(sys.stdin)["items"]:
|
||||
ns=p["metadata"]["namespace"]
|
||||
if ns in PROTECT: continue
|
||||
if not any(v.get("persistentVolumeClaim") for v in (p["spec"].get("volumes") or [])): continue
|
||||
ow=(p["metadata"].get("ownerReferences") or [])
|
||||
if not ow: continue
|
||||
k,n=ow[0]["kind"],ow[0]["name"]
|
||||
if k=="ReplicaSet": k,n="deployment",n.rsplit("-",1)[0]
|
||||
elif k=="StatefulSet": k="statefulset"
|
||||
else: continue # Jobs, and CNPG Clusters, are handled separately
|
||||
out[(ns,k,n)]=1
|
||||
for ns,k,n in sorted(out): print(ns,k,n)
|
||||
'
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- the PVC-deletion guard
|
||||
# *** DO NOT REMOVE THIS CHECK. IT EXISTS BECAUSE SCALING A StatefulSet DESTROYED DATA. ***
|
||||
# A StatefulSet may carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and then
|
||||
# scaling it to 0 DELETES ITS PVCs -- and with them the Longhorn volumes. MEASURED 2026-10-17: the
|
||||
# quiesce for the UPS move scaled `monitoring/loki` (whenScaled: Delete) to 0 and its
|
||||
# `storage-loki-0` PVC and volume were deleted outright. Loki stores with `type: filesystem`, so
|
||||
# that PVC was the only copy; 72h retention bounded the loss to a rolling window, but nothing about
|
||||
# the mechanism is bounded -- the same action on a Delete-policy StatefulSet holding real state
|
||||
# would be unrecoverable. The volume count going 25 -> 24 was the only visible symptom, and it was
|
||||
# first misread as a miscount.
|
||||
#
|
||||
# So: such a StatefulSet is NEVER scaled. It is left RUNNING and its volume hard-detaches at
|
||||
# poweroff, which costs a replica rebuild on restart (degraded -> healthy, automatic). A rebuild is
|
||||
# an enormously better outcome than a deletion, and that is the whole trade being made here.
|
||||
sts_deletes_pvc_on_scale() { # ns name -> 0 if scaling it would delete its PVCs
|
||||
local policy
|
||||
policy=$("${KUBECTL[@]}" -n "$1" get statefulset "$2" \
|
||||
-o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy.whenScaled}' 2>/dev/null || true)
|
||||
[ "$policy" = "Delete" ]
|
||||
}
|
||||
|
||||
wait_for() { # wait_for <description> <shell-condition>
|
||||
local desc="$1" cond="$2" waited=0
|
||||
log "waiting: $desc (up to ${TIMEOUT}s)"
|
||||
until eval "$cond"; do
|
||||
(( waited += 10 )); (( waited >= TIMEOUT )) && { warn "TIMED OUT waiting: $desc"; return 1; }
|
||||
sleep 10
|
||||
done
|
||||
ok "$desc"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- quiesce
|
||||
do_quiesce() {
|
||||
need kubectl; need python3
|
||||
|
||||
# Refuse to start if Longhorn is already unhealthy. Powering off on top of a degraded volume is
|
||||
# how one replica becomes zero; better to stop here and let it rebuild first.
|
||||
local unhealthy
|
||||
unhealthy=$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io \
|
||||
-o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)
|
||||
[ "$unhealthy" = "0" ] || die "$unhealthy Longhorn volume(s) are not healthy — fix or wait for rebuild BEFORE powering off"
|
||||
ok "Longhorn: all volumes healthy"
|
||||
|
||||
# 1. Flux first, or it reverts every scale-down below within its reconcile interval.
|
||||
log "suspending Flux Kustomizations"
|
||||
local k
|
||||
for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
|
||||
"${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":true}}' >/dev/null
|
||||
done
|
||||
ok "all Kustomizations suspended"
|
||||
|
||||
# 2. The VM needs an ACPI shutdown of the GUEST. KubeVirt does that on running:false; a plain
|
||||
# pod eviction would be a power cut to the VM.
|
||||
if "${KUBECTL[@]}" -n unifi get vm unifi >/dev/null 2>&1; then
|
||||
log "ACPI-stopping the UniFi VM"
|
||||
"${KUBECTL[@]}" -n unifi patch vm unifi --type=merge -p '{"spec":{"running":false}}' >/dev/null
|
||||
fi
|
||||
|
||||
# 3. Postgres: hibernation is a clean smart shutdown that KEEPS the PVCs. CNPG does one instance
|
||||
# at a time, primary last-ish, so this is not instant.
|
||||
log "hibernating CloudNativePG clusters"
|
||||
local c ns name
|
||||
for c in "${CNPG[@]}"; do
|
||||
ns="${c%%/*}"; name="${c##*/}"
|
||||
"${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
|
||||
cnpg.io/hibernation=on --overwrite >/dev/null 2>&1 || warn "could not hibernate $c"
|
||||
done
|
||||
|
||||
# 4. Everything holding a PVC, scaled in place. NFS consumers first so nothing holds a hard
|
||||
# TrueNAS mount; PiHole last because it is the LAN resolver.
|
||||
log "scaling down PVC-holding workloads"
|
||||
local line ns2 kind nm deferred=() skipped_delete_policy=()
|
||||
while read -r ns2 kind nm; do
|
||||
[ -z "${ns2:-}" ] && continue
|
||||
if [ "$kind" = "statefulset" ] && sts_deletes_pvc_on_scale "$ns2" "$nm"; then
|
||||
warn "SKIPPING $ns2/$nm — whenScaled: Delete would DELETE its PVC (see the guard above)."
|
||||
warn " Left running; its volume hard-detaches at poweroff and rebuilds on restart."
|
||||
skipped_delete_policy+=("$ns2/$nm")
|
||||
continue
|
||||
fi
|
||||
if [ "$ns2" = "pihole" ]; then deferred+=("$ns2 $kind $nm"); continue; fi
|
||||
"${KUBECTL[@]}" -n "$ns2" scale "$kind" "$nm" --replicas=0 >/dev/null 2>&1 \
|
||||
&& printf ' scaled 0 %s/%s %s\n' "$ns2" "$kind" "$nm" \
|
||||
|| warn "could not scale $ns2/$kind/$nm"
|
||||
done < <(pvc_workloads)
|
||||
for line in "${deferred[@]:-}"; do
|
||||
[ -z "${line:-}" ] && continue
|
||||
# shellcheck disable=SC2086
|
||||
set -- $line
|
||||
"${KUBECTL[@]}" -n "$1" scale "$2" "$3" --replicas=0 >/dev/null 2>&1 \
|
||||
&& printf ' scaled 0 %s/%s %s (LAN DNS — expect name resolution to stop)\n' "$1" "$2" "$3"
|
||||
done
|
||||
|
||||
# 5. The only assertion that matters. An attached volume at poweroff comes back degraded and
|
||||
# rebuilds; this is what we are buying with all of the above.
|
||||
wait_for "UniFi VMI gone" \
|
||||
'[ "$('"${KUBECTL[*]}"' -n unifi get vmi --no-headers 2>/dev/null | wc -l)" = "0" ]' || true
|
||||
wait_for "all Postgres instances stopped" \
|
||||
'[ "$('"${KUBECTL[*]}"' get pods -A --no-headers 2>/dev/null | grep -cE "(keycloak|nextcloud|semaphore)-db-[0-9]")" = "0" ]' || true
|
||||
wait_for "all Longhorn volumes detached" \
|
||||
'[ "$('"${KUBECTL[*]}"' -n longhorn-system get volumes.longhorn.io -o jsonpath="{range .items[*]}{.status.state}{\"\n\"}{end}" 2>/dev/null | grep -c attached)" = "0" ]' || \
|
||||
warn "some volumes are STILL ATTACHED — they will come back degraded and rebuild; acceptable, but note which"
|
||||
|
||||
if [ "${#skipped_delete_policy[@]}" -gt 0 ]; then
|
||||
echo
|
||||
warn "NOT scaled down (whenScaled: Delete — scaling them would delete their PVCs):"
|
||||
printf ' %s\n' "${skipped_delete_policy[@]}"
|
||||
warn "Expect these volumes to come back 'degraded' and rebuild. That is correct and automatic."
|
||||
fi
|
||||
echo
|
||||
ok "QUIESCED. Safe to power off."
|
||||
cat <<'NEXT'
|
||||
|
||||
Next, from inside the devcontainer (it holds the become password):
|
||||
cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
|
||||
-m shell -a 'shutdown -h +1 "maintenance"'
|
||||
|
||||
THEN, in this order and not before:
|
||||
1. TrueNAS (192.168.2.30) — web UI only; not in the ansible inventory and SSH is refused.
|
||||
2. Router / switch last.
|
||||
NEXT
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------- resume
|
||||
do_resume() {
|
||||
need kubectl
|
||||
|
||||
# THE STEP THAT IS ALWAYS FORGOTTEN. Hibernation is imperative state: Flux neither sets nor
|
||||
# clears it, so resuming Flux alone leaves all three databases down and every app that needs
|
||||
# one looking broken for reasons that point nowhere near Postgres.
|
||||
log "un-hibernating CloudNativePG clusters"
|
||||
local c ns name
|
||||
for c in "${CNPG[@]}"; do
|
||||
ns="${c%%/*}"; name="${c##*/}"
|
||||
"${KUBECTL[@]}" -n "$ns" annotate cluster.postgresql.cnpg.io "$name" \
|
||||
cnpg.io/hibernation=off --overwrite >/dev/null 2>&1 || warn "could not un-hibernate $c"
|
||||
done
|
||||
|
||||
# Flux restores every replica count AND the VM's running:true from git, so nothing here needs to
|
||||
# undo the scale-downs by hand.
|
||||
log "resuming Flux Kustomizations"
|
||||
local k
|
||||
for k in $("${KUBECTL[@]}" -n flux-system get kustomizations -o name); do
|
||||
"${KUBECTL[@]}" -n flux-system patch "${k}" --type=merge -p '{"spec":{"suspend":false}}' >/dev/null
|
||||
done
|
||||
ok "all Kustomizations resumed"
|
||||
|
||||
wait_for "Flux fully reconciled" \
|
||||
'[ "$('"${KUBECTL[*]}"' -n flux-system get kustomizations --no-headers 2>/dev/null | grep -vc True)" = "0" ]' || \
|
||||
warn "some Kustomizations are not Ready — the graph serialises behind platform-config, so give it a few minutes before worrying"
|
||||
|
||||
echo
|
||||
ok "RESUMED."
|
||||
cat <<'NEXT'
|
||||
|
||||
Known cold-start race, not a fault in this script: oauth2-proxy performs OIDC discovery at
|
||||
startup and exits 1 on the 503 it gets while Keycloak is still coming up, then sits in
|
||||
exponential CrashLoopBackOff. If a replica is stuck, delete the pod once:
|
||||
kubectl --context k3s-prod -n oauth2-proxy delete pod <name>
|
||||
NEXT
|
||||
}
|
||||
|
||||
do_status() {
|
||||
need kubectl
|
||||
printf 'nodes: %s Ready\n' "$("${KUBECTL[@]}" get nodes --no-headers | grep -c ' Ready')"
|
||||
printf 'flux: %s/%s Ready\n' \
|
||||
"$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | grep -c True)" \
|
||||
"$("${KUBECTL[@]}" -n flux-system get kustomizations --no-headers | wc -l)"
|
||||
printf 'longhorn: %s attached, %s detached, %s not healthy\n' \
|
||||
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c attached)" \
|
||||
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.state}{"\n"}{end}' | grep -c detached)" \
|
||||
"$("${KUBECTL[@]}" -n longhorn-system get volumes.longhorn.io -o jsonpath='{range .items[*]}{.status.robustness}{"\n"}{end}' | grep -cvE '^(healthy|unknown)$' || true)"
|
||||
printf 'postgres: %s instance pod(s)\n' \
|
||||
"$("${KUBECTL[@]}" get pods -A --no-headers | grep -cE '(keycloak|nextcloud|semaphore)-db-[0-9]' || true)"
|
||||
printf 'unifi VM: %s VMI\n' "$("${KUBECTL[@]}" -n unifi get vmi --no-headers 2>/dev/null | wc -l)"
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
quiesce) do_quiesce ;;
|
||||
resume) do_resume ;;
|
||||
status) do_status ;;
|
||||
*) cat >&2 <<USAGE
|
||||
usage: $0 {quiesce|resume|status}
|
||||
|
||||
quiesce suspend Flux, ACPI-stop the UniFi VM, hibernate Postgres, scale down every
|
||||
PVC-holding workload, and wait for all Longhorn volumes to detach.
|
||||
Does NOT power anything off — see runbooks/cluster-maintenance-shutdown.md.
|
||||
resume the exact inverse: un-hibernate Postgres, resume Flux, wait for reconcile.
|
||||
status one-screen readout of nodes / Flux / Longhorn / Postgres / VM.
|
||||
|
||||
Run it inside the pinned toolbox, which supplies kubectl and the context:
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
|
||||
USAGE
|
||||
exit 2 ;;
|
||||
esac
|
||||
@@ -63,75 +63,6 @@ spec:
|
||||
# gets oauth2-proxy's HTML sign-in page (HTTP 403) instead of an immediate redirect (docs/15 UX).
|
||||
skip-provider-button: "true"
|
||||
scope: "openid email profile"
|
||||
# ── Cold-start gate: do not start the proxy until Keycloak's discovery doc answers ──────────
|
||||
# MEASURED on prod 2026-10-17, during the UPS move: oauth2-proxy performs OIDC discovery ONCE at
|
||||
# startup and EXITS 1 if it fails. On a whole-cluster cold boot Keycloak is not serving yet, so
|
||||
# the Gateway answers 503 and the proxy dies:
|
||||
#
|
||||
# [provider.go:55] Performing OIDC Discovery...
|
||||
# [main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover OIDC
|
||||
# configuration: unexpected status "503": no healthy upstream
|
||||
#
|
||||
# It then sits in exponential CrashLoopBackOff long after Keycloak is healthy -- one replica was
|
||||
# still down minutes later and only a manual pod delete cleared it. There is no retry flag; the
|
||||
# process simply does not survive a cold start. A readinessProbe cannot help (the container is
|
||||
# already dead), and CrashLoopBackOff backoff is not tunable per-pod.
|
||||
#
|
||||
# `extraInitContainers` is the chart's OWN documented hook for this -- its commented example in
|
||||
# values.yaml is literally `- name: wait-for-idp` -- and the chart already uses an alpine init
|
||||
# container for its built-in waitForRedis gate, so this adds no new base image to the platform.
|
||||
# A pod in Init does not accrue CrashLoopBackOff backoff and retries promptly, which is the whole
|
||||
# point.
|
||||
#
|
||||
# BOUNDED, AND FAILS OPEN. After ~5 minutes it gives up and starts the proxy anyway: the worst
|
||||
# case is then exactly today's behaviour, never something new. An unbounded wait would trade a
|
||||
# CrashLoop for a pod wedged in Init forever, which is not an improvement.
|
||||
#
|
||||
# SHELL VARIABLES ARE WRITTEN BARE ($i, $url, $(( ))), WITH NO BRACED FORM ANYWHERE. This file is
|
||||
# rendered through Flux envsubst (platform declares postBuild.substituteFrom cluster-vars), which
|
||||
# CONSUMES braced expansions -- so a braced shell variable renders to the empty string and
|
||||
# silently breaks the loop. BASE_DOMAIN below is braced ON PURPOSE: it is a real cluster-vars
|
||||
# knob and must be substituted. Same hazard, and the same convention, as the gitea and nextcloud
|
||||
# OIDC bootstrap hooks. (ci: flux-postbuild-substitution enforces it.)
|
||||
extraInitContainers:
|
||||
- name: wait-for-idp
|
||||
image: docker.io/library/alpine:3.22
|
||||
imagePullPolicy: IfNotPresent
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- |
|
||||
set -u
|
||||
url=https://keycloak.${BASE_DOMAIN}/realms/homelab/.well-known/openid-configuration
|
||||
i=0
|
||||
while [ "$i" -lt 60 ]; do
|
||||
# busybox wget validates against alpine's ca-certificates-bundle, so a real TLS
|
||||
# failure is a non-zero exit here rather than a false pass.
|
||||
if wget -q -O /dev/null --timeout=5 "$url" 2>/dev/null; then
|
||||
echo "[wait-for-idp] Keycloak discovery reachable; starting oauth2-proxy"
|
||||
exit 0
|
||||
fi
|
||||
i=$(( i + 1 ))
|
||||
sleep 5
|
||||
done
|
||||
echo "[wait-for-idp] discovery still unreachable after ~5m; starting anyway so this" >&2
|
||||
echo "[wait-for-idp] gate can never be worse than the un-gated behaviour it replaces" >&2
|
||||
exit 0
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65534
|
||||
runAsGroup: 65534
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop:
|
||||
- ALL
|
||||
resources:
|
||||
requests:
|
||||
cpu: 10m
|
||||
memory: 16Mi
|
||||
limits:
|
||||
memory: 32Mi
|
||||
|
||||
# Metrics for the ServiceMonitor (../oauth2-proxy servicemonitor in platform-config).
|
||||
metrics:
|
||||
enabled: true
|
||||
|
||||
@@ -186,86 +186,6 @@ right way; one was a defect in my own manifest.
|
||||
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
|
||||
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
|
||||
|
||||
## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach
|
||||
|
||||
A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose
|
||||
Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from
|
||||
announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is
|
||||
real but inert, and the distinction matters enough to record, because the wrong version is very
|
||||
easy to re-derive from `ip neigh` alone.
|
||||
|
||||
Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case,
|
||||
after the UPS power-cycle — from the host netns of the `wireguard` pod on each node:
|
||||
|
||||
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|
||||
|---|---|---|
|
||||
| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
|
||||
| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms |
|
||||
| `nc -z 192.168.2.20 443` | **OK** | OK |
|
||||
| plex `192.168.2.24:32400` | **`200`** | `200` |
|
||||
|
||||
So the announcer really does **not** answer its own ARP request — that part of the observation is
|
||||
reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**:
|
||||
|
||||
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
|
||||
`Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**,
|
||||
so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before
|
||||
any route or neighbour lookup for the VIP happens**.
|
||||
* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the
|
||||
shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without
|
||||
a packet crossing the LAN at all.
|
||||
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
|
||||
which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real**
|
||||
service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes —
|
||||
that is service-aware DNAT, not wire traffic.
|
||||
|
||||
**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis:
|
||||
|
||||
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an
|
||||
artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not
|
||||
anything is wrong.
|
||||
* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the
|
||||
two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it
|
||||
discriminates nothing — and it is not the client path either (see below).
|
||||
|
||||
**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are
|
||||
**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is
|
||||
**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the
|
||||
forwarded path in either direction: it reports success where the forwarded path might fail, and
|
||||
`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with
|
||||
a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE`
|
||||
counters move.
|
||||
|
||||
## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses
|
||||
|
||||
`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the
|
||||
component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature
|
||||
to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel
|
||||
client cannot reach the internet, nothing in the list below is the reason:
|
||||
|
||||
| checked | result on all three nodes |
|
||||
|---|---|
|
||||
| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` |
|
||||
| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change |
|
||||
| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE |
|
||||
| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present |
|
||||
| default route | `default via 192.168.2.1 dev eno1 onlink`, intact |
|
||||
| node's own internet reachability | `http://1.1.1.1/` returns `301` |
|
||||
|
||||
**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is
|
||||
closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by
|
||||
`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled
|
||||
`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity
|
||||
traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway
|
||||
node. A remote client's apparent public address is therefore the **home WAN IP**, which is the
|
||||
intended behaviour and the thing to state when enrolling a device.
|
||||
|
||||
**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one
|
||||
enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and
|
||||
the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the
|
||||
power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the
|
||||
forwarded node shows a peer with an `endpoint` and a recent `latest handshake`.
|
||||
|
||||
## Prerequisites outside this directory
|
||||
|
||||
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
# cluster-maintenance-shutdown — powering the whole cluster off, and back on
|
||||
|
||||
**Operator procedure, not an alert page.** Nothing links here from an alert.
|
||||
**Script:** `infra/maintenance/quiesce.sh` (`quiesce` | `resume` | `status`)
|
||||
**Related:** `/runbooks/update-rollback` is for rolling node work where the cluster stays up. This
|
||||
page is for taking **all three nodes down at once**.
|
||||
|
||||
First written after the 2026-10-17 UPS move, which was improvised live because no procedure
|
||||
existed — `infra/cluster.sh stop` is `virsh`-based and only ever applied to the decommissioned dev
|
||||
cluster, so it does nothing for these three baremetal nodes.
|
||||
|
||||
## Why you cannot just run `shutdown -h`
|
||||
|
||||
You mostly can now, and could not before.
|
||||
|
||||
Until 2026-10-17 **kubelet graceful node shutdown was unconfigured on every prod node**, so systemd
|
||||
stopped k3s, containerd went with it, and every pod was SIGKILLed where it stood — 9 Postgres
|
||||
instances mid-checkpoint, the UniFi VM, and 25 attached Longhorn volumes.
|
||||
`roles/k3s_server` now sets `shutdown-grace-period` (and, critically, the
|
||||
`InhibitDelayMaxSec` logind cap that actually enforces it — without that systemd stops waiting
|
||||
after 5s and the drain is silently truncated). A plain reboot of one node is therefore safe.
|
||||
|
||||
**A planned whole-cluster shutdown still wants `quiesce`**, for three things the kubelet's drain
|
||||
does not do:
|
||||
|
||||
1. **The UniFi VM needs an ACPI guest shutdown**, not an eviction. KubeVirt does that on
|
||||
`running: false`; an eviction is a power cut to the guest.
|
||||
2. **CloudNativePG wants hibernation**, not pod deletion — and hibernation is **imperative** state
|
||||
that Flux will neither set nor clear.
|
||||
3. **Draining three nodes one at a time makes pods reschedule onto the survivors**, churning
|
||||
Longhorn replicas for no reason when everything is going down anyway. Scaling in place avoids it.
|
||||
|
||||
## The hazard that bit us — read this before scaling anything by hand
|
||||
|
||||
**A StatefulSet can carry `persistentVolumeClaimRetentionPolicy.whenScaled: Delete`, and scaling it
|
||||
to 0 then DELETES its PVCs and their Longhorn volumes.**
|
||||
|
||||
On 2026-10-17 the hand-rolled quiesce scaled `monitoring/loki` (which has that policy) to 0, and
|
||||
`storage-loki-0` was deleted outright. Loki stores with `type: filesystem`, so that PVC was the only
|
||||
copy. Its 72h retention bounded the loss to a rolling window — nothing about the *mechanism* is
|
||||
bounded. The only visible symptom was the Longhorn volume count going 25 → 24, which was first
|
||||
misread as a miscount.
|
||||
|
||||
`quiesce.sh` now refuses to scale any such StatefulSet, leaves it running, and tells you which
|
||||
volumes to expect `degraded` on restart. **If you quiesce by hand, check the policy first:**
|
||||
|
||||
```bash
|
||||
kubectl --context k3s-prod get statefulset -A \
|
||||
-o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,WHEN_SCALED:.spec.persistentVolumeClaimRetentionPolicy.whenScaled'
|
||||
```
|
||||
|
||||
A hard-detached volume comes back `degraded` and rebuilds itself automatically. That is a vastly
|
||||
better outcome than a deleted one — take the rebuild.
|
||||
|
||||
## Shutting down
|
||||
|
||||
Run everything through the pinned toolbox; it supplies `kubectl` and the context.
|
||||
|
||||
```bash
|
||||
# 0. Look before you leap. Refuses to proceed if any Longhorn volume is unhealthy —
|
||||
# powering off on top of a degraded volume is how one replica becomes zero.
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
|
||||
|
||||
# 1. Quiesce: suspend Flux, ACPI-stop the VM, hibernate Postgres, scale down every
|
||||
# PVC-holding workload (NFS consumers first, PiHole last), wait for detach.
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh quiesce'
|
||||
```
|
||||
|
||||
**Expect DNS to stop** when PiHole scales down — it is the LAN resolver. `kubectl` is unaffected
|
||||
because the kubeconfig points at the kube-vip IP `192.168.2.9`, not a name.
|
||||
|
||||
```bash
|
||||
# 2. Power the nodes off. Inside the devcontainer, which holds the become password.
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'cd /work/infra/ansible && ansible -i inventory/hosts.ini k3s_prod -b \
|
||||
-m shell -a "shutdown -h +1 \"maintenance\""'
|
||||
```
|
||||
|
||||
`+1` gives Ansible time to disconnect cleanly. The group resolves to all three nodes; confirm with
|
||||
`--list-hosts` if you want to see them first.
|
||||
|
||||
```bash
|
||||
# 3. Confirm they are actually down before touching anything else.
|
||||
for h in 192.168.2.10 192.168.2.11 192.168.2.12; do ping -c1 -W2 $h >/dev/null && echo "$h UP" || echo "$h down"; done
|
||||
```
|
||||
|
||||
**Then, in this order and not before:**
|
||||
|
||||
| # | Machine | How |
|
||||
|---|---|---|
|
||||
| 4 | **TrueNAS** `192.168.2.30` | **Web UI only.** Not in the ansible inventory, and SSH is **refused** (service disabled). `https://192.168.2.30` → power icon → Shut Down. Safe once the nodes are down, because every NFS consumer (`plex-media`, `*-media`, `qbittorrent-downloads`, `nextcloud-legacy-archive`) is already scaled to zero and nothing holds a hard mount. |
|
||||
| 5 | **Router / switch** | Last. |
|
||||
|
||||
TrueNAS must go **after** the k3s nodes: a hard NFS mount on a live node wedges kubelet in
|
||||
D-state (`/runbooks/nfs-export-latency`, and the load-average symptom in the 2026-10-?? note).
|
||||
|
||||
## Bringing it back
|
||||
|
||||
**Power on in the reverse order: router/switch → TrueNAS → all three nodes together.** etcd needs
|
||||
2 of 3 for quorum, so start the nodes as a group rather than one at a time.
|
||||
|
||||
```bash
|
||||
# 1. Nodes Ready, etcd quorum, storage reachable.
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
||||
./infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get --raw=/healthz/etcd'
|
||||
|
||||
# 2. Resume: un-hibernate Postgres, then resume Flux.
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh resume'
|
||||
```
|
||||
|
||||
**Un-hibernating Postgres is the step everyone forgets.** The annotation is imperative and survives
|
||||
a Flux resume, so resuming Flux alone leaves all three databases down and Keycloak, Nextcloud and
|
||||
Semaphore looking broken for reasons that point nowhere near Postgres. `resume` does it first, for
|
||||
exactly that reason.
|
||||
|
||||
Flux restores every scaled-down replica count **and** the VM's `running: true` from git, so nothing
|
||||
needs undoing by hand.
|
||||
|
||||
### What "still broken" looks like when it is actually fine
|
||||
|
||||
* **`4/19` Kustomizations Ready for several minutes.** The dependency graph serialises behind
|
||||
`platform-config`; 15 of them report `dependency ... is not ready` until it finishes. Check
|
||||
`platform-config` itself is `Ready=True` and its `lastAppliedRevision` matches the GitRepository
|
||||
artifact — if it is, wait.
|
||||
* **One or more Longhorn volumes `degraded`.** Any volume that was still attached at poweroff
|
||||
rebuilds its lagging replica automatically. On 2026-10-17 it was Prometheus's 50Gi TSDB; it went
|
||||
healthy on its own. Worry only if it is still degraded after ~30 minutes.
|
||||
* **`keycloak-operator` crashlooping, then recovering.** It is the CRD reconciler, not in the
|
||||
request path; Keycloak's StatefulSet stays 2/2 and auth never goes down. See
|
||||
`/runbooks/keycloak-*` and the known OOM-at-450Mi behaviour.
|
||||
|
||||
### What is actually broken and needs you
|
||||
|
||||
* **An `oauth2-proxy` replica stuck in CrashLoopBackOff.** It performs OIDC discovery at startup and
|
||||
**exits 1** on the `503 no healthy upstream` it gets while Keycloak is still coming up, then sits
|
||||
in exponential backoff long after Keycloak is fine. It has no startup dependency on Keycloak.
|
||||
Delete the pod once and it comes up clean:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n oauth2-proxy delete pod <name>'
|
||||
```
|
||||
|
||||
* **Orphaned `Unknown`/`Error` Job pods** from CronJobs that were mid-run at poweroff. They never
|
||||
recover; delete them.
|
||||
|
||||
## Verifying you are actually back
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/maintenance/quiesce.sh status'
|
||||
# nodes 3 Ready | flux 19/19 | longhorn 0 not healthy | postgres 9 pods | unifi 1 VMI
|
||||
|
||||
# DNS, through PiHole's VIP
|
||||
./infra/devcontainer/dev.sh run bash -c 'nslookup home.keithjackson.dev 192.168.2.23'
|
||||
|
||||
# The Gateway actually serving (401 on an auth-gated host is CORRECT)
|
||||
for h in home.keithjackson.dev gitea.home.keithjackson.dev plex.home.keithjackson.dev; do
|
||||
curl -s -o /dev/null -w "$h %{http_code}\n" -k --resolve $h:443:192.168.2.20 https://$h/
|
||||
done
|
||||
```
|
||||
|
||||
Expect `200`, `200`, `401`.
|
||||
@@ -52,6 +52,13 @@ configMapGenerator:
|
||||
- backup-failure=backup-failure.md
|
||||
- backup-missed=backup-missed.md
|
||||
- backup-partial=backup-partial.md
|
||||
# Operator procedure, not an alert page — nothing links here from an alert. Served because
|
||||
# the whole-cluster shutdown had NO written procedure until 2026-10-17 (infra/cluster.sh is
|
||||
# virsh-based and only ever applied to the decommissioned dev cluster), so the UPS move was
|
||||
# improvised live, and the one hazard it exposed -- a StatefulSet with
|
||||
# persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 --
|
||||
# destroyed monitoring/loki's volume before anyone noticed.
|
||||
- cluster-maintenance-shutdown=cluster-maintenance-shutdown.md
|
||||
- cdi-import-stuck=cdi-import-stuck.md
|
||||
- cert-expiry=cert-expiry.md
|
||||
- cert-issuance=cert-issuance.md
|
||||
|
||||
@@ -273,84 +273,6 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
|
||||
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
|
||||
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
|
||||
|
||||
## Triage — "the client reaches the LAN and cluster apps, but not the internet"
|
||||
|
||||
Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom
|
||||
almost always lives, and because the server side has been measured and is not the cause.
|
||||
|
||||
**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.**
|
||||
|
||||
| the profile says | meaning |
|
||||
|---|---|
|
||||
| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. |
|
||||
| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. |
|
||||
|
||||
`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit —
|
||||
`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client
|
||||
profile only**, so an already-issued profile keeps whatever it was rendered with until it is
|
||||
re-rendered and re-imported on the device. The two forms differ by one flag:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet>' # full tunnel
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet> --split' # LAN only
|
||||
```
|
||||
|
||||
Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into
|
||||
`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is
|
||||
editing `AllowedIPs` in the profile **on the device** — the server does not care.
|
||||
|
||||
**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the
|
||||
client connected, then watch the two counters that only forwarded traffic moves:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c "
|
||||
wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5;
|
||||
iptables -t nat -L POSTROUTING -n -v | head -5"'
|
||||
```
|
||||
|
||||
* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which
|
||||
node the router's `udp/51820` forward points at (the router is unreadable from the cluster).
|
||||
* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is
|
||||
upstream of the cluster (router forward, client, or carrier), not here.
|
||||
* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side
|
||||
failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there,
|
||||
which breaks internet while node-local cluster VIPs still work. It has not been observed.
|
||||
|
||||
**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly:
|
||||
|
||||
* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack
|
||||
client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel —
|
||||
so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4:
|
||||
`curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it,
|
||||
and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is
|
||||
blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it).
|
||||
* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified
|
||||
2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a
|
||||
blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on
|
||||
a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all.
|
||||
|
||||
**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the
|
||||
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain
|
||||
**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route
|
||||
`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the
|
||||
PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled
|
||||
`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the
|
||||
**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6).
|
||||
|
||||
## Triage — which L2 announcement lease a node holds does NOT matter
|
||||
|
||||
Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node
|
||||
could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`:
|
||||
that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh`
|
||||
for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer
|
||||
VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend
|
||||
is a node-local socket. Full measurements and the two misleading probes
|
||||
(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`.
|
||||
|
||||
`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a
|
||||
separate, pre-existing fault in that service and has nothing to do with the VPN.
|
||||
|
||||
## Revoking a device
|
||||
|
||||
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
|
||||
|
||||
Reference in new issue
Block a user