a92fa117012b8e7d5ed4ae8eecd7b83dbf72a4e3
609
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
38e4bf1459 |
docs(vpn): record the PIA server collision, and the accidental proof it handed us
The stand-up broke dev, so the "one gateway per PIA server" rule and the measurement behind it are now written down where the next person will look: platform/vpn-egress/README.md and the session's DECISIONS-LOG entry. Both carry the verbatim journal lines — same server, same pushed `ifconfig 10.72.192.7` on both gateways — and the flap timestamps, because "same account, inside the connection limit" is true and is exactly what makes this non-obvious. Also corrected: the dev/prod table now shows the two environments on different servers with different exit IPs (dev 158.173.241.36, prod 158.173.241.11), and the open item about them sharing one public IP is replaced with the real remaining decision — whether prod gets its own PIA account, which would remove the hand-maintained per-environment server value entirely. The unplanned bonus, recorded in both places: moving prod to .135 gave it a genuinely NEW tunnel address on a real reconnect (10.72.192.7 ifindex 128 -> 10.150.192.86 ifindex 129) — the precise case Cilium does not follow and the one defect 1 was about. The egress-uplink label moved on its own, Cilium re-resolved, and all four vpn-class workloads kept egressing with no manual step. Better evidence than the synthetic address swap used to characterise it, and it cost nothing to capture. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
37a6c94168 |
fix(vpn): two gateways on one pinned PIA server evict each other every 2 minutes
Standing up prod broke dev, and the mechanism is worth spelling out because
"same PIA account, well inside its connection limit" is true and still not enough.
The vaulted .ovpn pins ONE server. With both gateways using it, dev and prod
landed on the SAME server AND were handed the SAME tunnel address — both journals
read `[Server-10779-2a] Peer Connection Initiated with [AF_INET]158.173.241.112:1198`
and `ifconfig 10.72.192.7 255.255.192.0`. Each reconnect steals that address on
that server; the loser then receives nothing, trips `--ping-restart` after 60s,
reconnects and steals it back. Measured cycle: dev restarting at 22:57:05,
22:59:03, 23:01:05 UTC with prod interleaved about a minute off, each ~2 minutes
apart, indefinitely. Neither environment had reliable vpn-class egress — prod's
probes looked green only because they happened to land inside an up-window.
Fix: `vpn_egress_gateway_pia_remote` ("<host> <port>", empty = honour the .ovpn)
rewrites just the `remote` line of the deployed .ovpn, and prod's group_vars sets
it to 158.173.241.135. That is another member of the same PIA Stockholm LB pool —
`sweden.privacy.network` resolved from k3s-prod-3 to
158.173.241.{34,55,112,135,239} — and every server in the pool takes the same
credentials and the same inline CA, so only the address differs. Dev keeps the
.ovpn's .112 untouched, which is why the override is per-group rather than in the
role or a second vaulted .ovpn.
Two details that matter for whoever touches this next:
* The regex is anchored to a line STARTING with `remote` plus whitespace, so
`remote-cert-tls` and `remote-random` are not collateral damage. The task keeps
`no_log: true` — the transformation never widens what is logged.
* A PIA server address is NOT a credential, so it lives in
group_vars/vpn_egress_gateway_prod/vars.yml in the clear, next to the vault
rather than inside it. Keeping it out of the vault is what makes it reviewable.
If prod ever gets its own PIA account, this override can be deleted again.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
04606d48b2 |
docs(vpn): stop crediting the nftables kill-switch for a drop it never makes
Four places in the tree told the reader that the host nftables table is the authoritative fail-closed backstop for egress-class=vpn pods: platform/vpn-egress README, egress-steering.yaml, the role README, and the vpn-killswitch test's own GIVEN/THEN prose. It is not, and cannot be — Cilium's eBPF egress-gateway datapath bpf_redirect()s those skbs to the egress device without invoking netfilter POSTROUTING. Corrected with the measurement in each place rather than softened, because a guard the repo trusts and that does not fire is worse than no guard. What replaces the claim, everywhere: vpn-class fails closed because CILIUM drops the traffic when it cannot use its cached egress interface, verified by tcpdump on both the WAN and tun0 during a tunnel stop and a tunnel restart (0 packets out either). The nft table keeps a narrower, now-proven job: node-originated traffic sourced from a tunnel address (killswitch_drop moved 0 -> 3 packets on k3s-prod-3 when such a packet was routed at the WAN). Also recorded: * egress-steering.yaml — `interface: tun0` is resolved ONCE and cached, and the gateway node is what tells Cilium it changed (the egress-uplink node label). A future reader's instinct will be to have the node write to that object; the comment says not to, because it is Flux-managed, and points at the label and `cilium-dbg bpf egress list` as the debugging entry points instead. * egress-steering.yaml — why the new NFS media mounts are not affected by pod-scoped steering: an NFS PV is mounted by the node kernel in the host netns, so podSelector never matches it. The 192.168.0.0/16 exclusion covers the server too, but that is the second reason, not the first. * vpn-killswitch test — its recovery step depends on the re-steer path, and could not have passed before it existed. Its no-leak assertion was always right; only the explanation was wrong. Also states the prerequisite plainly: the test cannot run at all without a labelled gateway node. * platform README — prod is live on k3s-prod-3 with the before/after egress IPs, and the note that prod's vpn-class pods had NO internet beforehand (CEGP with no matching node -> `Gateway IP: Not Found` -> dropped), so arming the gateway restored egress rather than moving it off the WAN. * role README — the standing "add the shared fwmark to the interface contract" escalation is closed as not implementable on Cilium OSS, rather than left open for someone to attempt. DECISIONS-LOG carries the full session entry, including the drafted ADR-0015 / docs-03 amendment for the architect (specs are immutable to agents) and the five things left for the operator — chief among them that a silent gateway outage still has no alert, which is what let the dev gateway sit dead for 80 minutes. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
60e3f09c67 |
feat(vpn): prod egress-gateway group — k3s-prod-3, singleton, opt-in only
`[vpn_egress_gateway_prod]` is a SEPARATE inventory group with its own vaulted credentials, not a second member of `[vpn_egress_gateway]`. That group is the default target of playbooks/vpn.egress.gateway.yml, so adding a prod host to it would mean a flagless `ansible-playbook playbooks/vpn.egress.gateway.yml` silently reconfigures a prod node's kill-switch datapath. Prod is reached only by asking for it, with `--list-hosts` first: -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3 Node 3 because node 1's SSD was replaced yesterday (least operational history) and node 2 carries the §0b memattr history. Exactly one node, forever: Cilium OSS picks "the first node in lexical ordering" among nodeSelector matches, so a labelled k3s-prod-1 would silently steal the gateway role — and the public egress IP — from k3s-prod-3, with no event and no alert. vault.yml is a verbatim copy of the dev group's CIPHERTEXT: same PIA account per the operator, and copying the encrypted blob means the plaintext was never decrypted to disk, through a shell, or into a transcript. The two files are now independent — rotating means re-encrypting both, which is the right behaviour for two environments. Also corrects a stale inventory note that claimed k3s-prod-3 has wlp1s0 up on 192.168.2.195. Re-verified before choosing it as the gateway, because a second live uplink in a kill-switch datapath is not a footnote: there is no wl* netdev on the node at all, no phy80211 device, the /etc/network/interfaces wlp1s0 stanza is commented out, 192.168.2.195 is assigned nowhere, and the only IPv4 addresses are lo and eno1 192.168.2.12. Nothing needed bringing down. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6f698f59af |
fix(vpn): Cilium never re-resolves tun0 — the gateway dies on every reconnect
Measured on k3s-dev-3, not reasoned about. Three findings, one of which had the dev gateway silently dead for ~80 minutes before this change: 1. Cilium caches (egress IP, egress ifindex) when it first resolves the CEGP's `interface: tun0` and does NOT re-resolve when tun0 changes. Every tunnel restart re-creates tun0 with a NEW ifindex, so vpn-class egress is blackholed from then on. Observed: node had tun0 at ifindex 1987 while `cilium-dbg bpf egress list` still said 1986; pod curl timed out. Forcing a re-apply moved the map to 1987 and egress recovered instantly. `egress-gateway-reconciliation- trigger-interval: 1s` does NOT cover netlink link events — dev has that same 1s setting and still did not self-heal, 20s+ after recovery. Fix: vpn-egress-resteer.sh publishes "<tun0 addr>-<ifindex>" as a node label. A node update makes the gateway re-process the policy (verified: cached ifindex 1986 -> 1987 within 10s). A node label, not an annotation on the policy, because the policy is Flux-managed GitOps and the node's labels are already this role's territory. Fired by the OpenVPN up-script on every connect, plus a 2-minute convergence timer for the boot case where the kube-API is not up yet. Proof: `systemctl stop` then `start` of the tunnel now recovers on its own (map 1989 -> 1990, PIA IP returns); before this commit the identical sequence stayed broken indefinitely. 2. The nft kill-switch CANNOT see the traffic it claims to guard. A counters-only diagnostic table on the gateway node recorded, while a vpn-class pod egressed: 0 packets in forward and 0 in postrouting for both the pod CIDR and the PIA range, against 45166 packets on the same postrouting hook for WAN traffic and 16 packets for a node-originated curl out tun0. Cilium's eBPF datapath bpf_redirect()s these skbs to the egress device without invoking netfilter. So `ip saddr @set oifname != tun0 drop` can never fire for vpn-class pods. What actually fails closed is Cilium's own drop when the cached egress interface is unusable — tcpdump during a tunnel stop AND during a restart showed 0 packets leaving either the WAN or tun0. Every rule now carries a `counter` so the claim stays falsifiable, and every comment that called this table the authoritative vpn-class backstop is corrected rather than left to mislead the next reader. Also removed the `meta mark 0x1c8 oifname != tun0 drop` rule and its variable: nothing stamps that mark (the only Cilium mark on the node is 0x200/0xf00, the L7-proxy rule to table 2004), the traffic never reaches netfilter anyway, and a live rule matching an arbitrary mark is a hazard to whatever picks it next. 3. The up-script flushed the nft source set on every connect, so a packet still sourced from the previous tunnel address matched neither the ip-rule nor the drop rule. The set is now additive (`add element` is idempotent; the down-script already retained it): an element can only ever tighten the guard. Finding 2 means that window could not have leaked POD traffic, but node-originated tunnel traffic does traverse netfilter, and additive is free. Kill-switch reloads no longer disarm the guard either: the .nft file uses `table` + `flush table`, which clears rules but leaves the named set's elements intact, so an Ansible run cannot empty the set mid-flight. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
839f7fed4a | merge: velero works against R2 — plugin pin, NFS volume policy, metrics | ||
|
jakman85andClaude Opus 5
|
2d5171e064 |
feat(velero): scrape the backup controller — three critical alerts could never fire
platform/monitoring/alerts.yaml has shipped VeleroBackupFailure (critical), VeleroBackupPartialFailure and VeleroNoRecentBackup since the observability wave. None of them could ever have fired. The velero chart defaults `metrics.serviceMonitor.enabled: false`, and prod confirms it live: `kubectl -n velero get servicemonitor` returns no resources, so velero_backup_failure_total and velero_backup_last_successful_timestamp were never scraped. A nightly backup could fail — or silently stop running — for weeks with nothing to say so. That is exactly what R-STATE-03 is for, which makes the scrape part of the backup path rather than an observability extra. Both flags verified against chart 12.1.0: `helm template` emits neither object without the monitoring.coreos.com/v1 capability, and emits both PodMonitor and ServiceMonitor with it — so the chart gates on capability detection AND `enabled`, and helm-controller supplies real capabilities from the API server. No additionalLabels are needed: platform/monitoring already sets serviceMonitorSelectorNilUsesHelmValues and podMonitorSelectorNilUsesHelmValues to false, so Prometheus selects every monitor regardless of labels. Independent of the R2/bulk-tier fix in the previous commit; droppable on its own if the PM wants to sequence it separately. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
2e259a808f |
fix(velero): prod backups could not write metadata to R2, and opt-out FSB would have shipped 8.6 TiB
Velero was resumed on prod (live `flux resume`; suspension was never in git) and its BackupStorageLocation validated Available against Cloudflare R2 on the first try — bucket k3s-prod-backup, prefix `velero`, region auto, checksumAlgorithm "". Backups still failed, and a second, unrelated hazard was armed. Both are fixed here; both were measured on prod, not reasoned about. 1. velero-plugin-for-aws v1.14.2 cannot write to R2 at all. It sets PutObject's `Tagging` field unconditionally, so the SDK emits an empty `x-amz-tagging` header on every PUT. R2 does not implement object tagging in any form and answers HTTP 501 NotImplemented. Two separate Backups reproduced it on the write of velero/backups/<name>/velero-backup.json: kopia uploaded the volume data fine (it uses its own S3 client), then the backup went phase=Failed with no metadata object — i.e. unrestorable. No config avoids it; `config.tagging` with a real value sends a tag set R2 also rejects. Upstream PR #309 makes the field conditional and landed on release-1.14 after v1.14.2 was cut, so v1.14.3-rc.2 is the earliest build with the fix. Pinned to it, with the exit condition (v1.14.3 GA) stated at the pin. 2. The opt-out FSB schedule would have uploaded the NFS bulk tier. Schedule (1) runs defaultVolumesToFsBackup:true and excludes only monitoring and unifi. Since 2026-09-11 prod also carries four static NFS PVs — plex-media, sonarr-media and radarr-media all bound to the SAME export /mnt/Storage/plex (8.32 TiB measured via zfs list), plus nextcloud-legacy-archive at 270 GiB. The 03:00 run would have read that library three times over NFS and pushed ~8.6 TiB to R2 nightly, retained 720h. Namespace exclusion cannot express the fix: those four namespaces also hold real Longhorn PVCs (plex/sonarr/radarr config, nextcloud-data) whose loss is unrecoverable. So the opt-out is now per VOLUME, by SOURCE TYPE: a velero volume policy (ConfigMap velero-volume-policies) skips any volume whose source is NFS, and schedule (1) references it. That tests the property that makes a volume bulk-tier (R-STATE-04: the bulk tier IS the NFS export) instead of a name list the next PV would slip past. The per-pod backup-volumes-excludes annotation was rejected for failing OPEN — a new NFS mount would be backed up in full until someone remembered. PROVEN on prod (velero 1.18.1 / chart 12.1.0): a Backup of ns nextcloud with defaultVolumesToFsBackup:true plus this policy produced exactly ONE PodVolumeBackup — `data` (nextcloud-data, Longhorn, 875,417,102 bytes, Completed). The NFS `legacy-archive` volume produced none. Residual gap, stated rather than hidden: the rule matches NFS only. A future bulk volume on another out-of-cluster source needs its own condition. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
3ec67bafdd | merge: platform-admins may reach the media app UIs | ||
|
jakman85andClaude Opus 5
|
5da116b34d |
fix(oauth2-proxy): platform-admins may reach the media UIs — the immediate 403
The four forward-auth media UIs (jackett, qbittorrent, radarr, sonarr) gate on `--allowed-group=media-users` alone. On prod the `homelab` realm has exactly one user — the operator's `jakman85` — and it is a member of `platform-admins` ONLY. `media-users` has zero members, so those four hosts were reachable by nobody: oauth2-proxy completed OIDC discovery, redirected to Keycloak, got a token whose `groups` claim is `["platform-admins"]`, found no allowed group in it, and answered 403. No login form appeared because the browser already held a Keycloak SSO session from a platform-admins login — hence "403 immediately". pihole and crafty were unaffected: they already gate on platform-admins. `--allowed-group` is repeatable and OR-ed, so listing both groups means "media consumers OR platform operators". This grants no real privilege — platform-admins already maps to cluster-admin via k8s-oidc-rbac.yaml (oidc:platform-admins), so a member can already exec into these pods — and it keeps media-users meaningful for non-admin household accounts. Ruled out while diagnosing: client-secret drift (all 7 sidecar secrets hash-match keycloak-realm-clients), stale apex URLs (every live sidecar carries keycloak.prod.home.keithjackson.dev), missing group mappers (all 7 clients emit claim=groups, full.path=false), and the central oauth2-proxy pod restarts — those were Keycloak returning 503 "no healthy upstream" to OIDC discovery at startup, a consequence of the Keycloak roll, not a cause. Goldens re-rendered with the canonical command: python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation R-IDP-06 Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6b953cc56b |
fix(arrs): the config PVC was mounted where the app never looks
sonarr, radarr, jackett and qbittorrent all mounted their Longhorn config PVC at the renderer's default /data/config, but every one of these images runs with `-data=/config`. So the PVC held nothing but lost+found while the app wrote its entire configuration to the container's EPHEMERAL writable layer — destroyed on every pod recreation: restart, reschedule, image bump. Measured in-cluster before fixing, per app, rather than inferred: /data/config dev=2080 [lost+found] <- the Longhorn PVC /config dev=2053 [Sentry asp config.xml logs ...] <- the node's own disk and a marker file written to /config did not appear in /data/config, so they are demonstrably different storage. qbittorrent's /config sat on dev=66309 (/dev/nvme0n1p5), i.e. whichever node it happened to land on. This is the SAME failure catalog/plex.yaml:54 already records having fixed for Plex; plex was the only one of the five that was right. The live /config contents were copied onto each PVC BEFORE this landed, so jackett's API key and qbittorrent's credentials — which other apps reference — survive the move rather than being regenerated. catalog/sonarr.yaml:61 documented this as a latent bug and deliberately left it because it changes live on-disk state. It is being fixed now because all four pods had just been recreated, so their configs were freshly generated and nothing had accumulated to lose. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
f5e62d9169 | merge: bulk NFS volumes for plex, sonarr, radarr and nextcloud | ||
|
jakman85andClaude Opus 5
|
daac05e7e8 |
feat(bulk): prod mounts the NFS export through pfSense's WAN address
The operator added the pfSense port-forward, so BULK_NFS_SERVER is no longer a placeholder. Prod gets 192.168.2.102 (pfSense WAN, DNAT tcp/2049 -> 192.168.1.9); dev stays on the base 192.168.1.9 because it is already on pfSense's LAN. This value could not be left as a placeholder safely: spec.nfs.server is unvalidated by Kubernetes, so the PV would be created, the PVC would BIND, and the failure would only surface as four pods stuck in ContainerCreating. Verified from k3s-prod-1 before committing, deliberately as uid 1000 rather than root (the export carries no_root_squash, so a root write proves nothing, and LSIO images default to PUID 911 which would go Ready and then fail every import EACCES): mount negotiated nfs4 vers=4.1 proto=tcp, write succeeded, and ln() across downloads/ -> tv/ gave link count 2 — real hardlinks, so no silent copy-then-delete. The Nextcloud export was additionally tightened to server-side ro= for prod's three nodes; a write as root is refused. Its content is config/, config_old/ and config_backup/ with no data/ directory, so it is an archive, not a live data store. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
da8a0eac04 |
test(l0): R-STATE-04 gets a real check — bulk storage provably is not the replicated tier
The gate BLOCKED this branch, correctly, and the block was the interesting part.
Attaching the media library selects R-STATE-04 as a touched requirement ("bulk media storage has
a capacity strategy distinct from the replicated tier"), and R-STATE-04 had NO scaffolded test, so
the coverage REQUIRE stage failed both l0 and l1. There were two ways out. One was to keep
avoiding the token — the existing discipline in catalog/plex.yaml and catalog/qbittorrent.yaml,
which say in as many words that naming the requirement would block the merge. That made sense while
the tier was a stub. It does not make sense in the change that DELIVERS the tier: it would mean
shipping the requirement and claiming nothing. So: the check.
WHAT IT ASSERTS, and why each one is a real failure mode rather than a shape
The `bulk` StorageClass sat as `server: REPLACE-ME` for six roadmap phases, and the only thing
keeping a multi-terabyte library off the 3-replica `longhorn` class was four hand-written comments.
That is the class of failure being closed: a bulk volume on the replicated tier does not fail
loudly — Longhorn accepts the PVC and starts replicating 3x of data three mini-PCs cannot hold.
(1) DISTINCTNESS — no bulk PV/PVC on any class the repo declares with the Longhorn provisioner
(derived from platform/longhorn/, so adding a Longhorn class cannot create a blind spot),
and the class is explicitly EMPTY.
(2) CAPACITY STRATEGY as arithmetic — a bulk volume at least an order of magnitude above the
same service's replicated volume. The requirement is about two magnitudes of data; a "bulk"
volume sized like a config PVC means the tier was named, not applied. The 10x floor is
CALIBRATED against the real corpus, not guessed: the media library is ~460x its config
volume and the Nextcloud archive ~13x its 20Gi, so a higher floor would fire on a legitimate
volume and teach the next reader to weaken the check. My first draft used 100x and did
exactly that — the corpus caught it.
(3) PRE-BOUND FROM BOTH ENDS (claimRef + volumeName, agreeing). Guards the most deceptive
failure available in this tier: a dynamic bulk class hands out a FRESH EMPTY directory, the
pod mounts, the PVC Binds, everything is green, and the 8 TiB was never attached.
(4) Retain, always. (5) mountOptions carry `hard` and exactly one nfsvers=4.x, and never `soft`
— a soft mount hands a server hiccup to the app as a short read, which for a media library
is silent corruption; and v4 is what keeps the firewall surface at one port.
(6) ONE EXPORT, AT MOST ONE MOUNT PER WORKLOAD. This is the assertion I most wanted. Splitting
one export into two subPath mounts looks TIGHTER and breaks link() with EXDEV; Sonarr and
Radarr then degrade to copy-then-delete, rewriting every import over NFS and destroying the
seeding copy, surfacing weeks later as "imports are slow and my ratio is gone". No reviewer
sees that in a diff.
(7) READ-ONLY COUPLING — nfs.readOnly, the volumeMount flag and ReadOnlyMany must agree, so
dropping one still leaves the others; and every bulk mode is a `...Many`.
(8) The server is a BARE cluster-vars token, that knob has a base default, and EVERY overlay
supplies it — the seam that lets one authored definition serve two clusters that reach this
host by different addresses.
(9) LOAD-BEARING and NON-VACUOUS. Wired unconditionally into ci/l0.sh as Stage 3e for the same
reason as 3c/3d: Stage 3b scopes to touched requirements, and the path map keys NEITHER
apps/* NOR platform/longhorn/* to this requirement — so the change most likely to break
this could attribute nothing and be gated by nothing. An EMPTY bulk corpus FAILS rather
than passing trivially, and a floor on the assertion count keeps a pass from degenerating.
It reads the COMMITTED apps/ goldens rather than calling the renderer. Asking the renderer would
only prove it agrees with itself; the goldens are what Flux reconciles, and render==golden is
already its own gate stage.
PERTURBED, because a check that has never failed is not evidence. Each of these was applied to the
real tree and each fired with the intended message: bulk PVC moved onto `longhorn`; Retain -> Delete;
`hard` -> `soft`; the token replaced by the literal 192.168.1.9; sonarr's single mount split into
two subPath mounts (assertion 6); and the prod overlay's knob deleted.
Also hardens render.py: `size` is structurally required of a bulk entry (a PV with no capacity is
rejected by the API server) but the JSON Schema only enforces `export`, so omitting it raised a bare
KeyError three frames down. It now raises a sentence naming the service, the volume and the remedy.
Found by the same perturbation pass.
Coverage: 48/51 -> 49/51 (96.1%), 2 gaps, 0 safety-critical.
BASE_REF=main make -C ci gate -> MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
cac413e87a |
feat(cnpg): synchronous replication, quorum of 1, dataDurability preferred
Three instances replicating ASYNCHRONOUSLY still lose data on failover: COMMIT returns as soon as the primary has flushed locally, so a primary that dies before its standbys catch up takes the last few transactions with it and the promoted standby comes up quietly missing them. On keycloak-db that is a just-created user, a rotated credential or a consent vanishing at exactly the moment of a failover. `method: any` + `number: 1` makes COMMIT wait for ONE of the two standbys to confirm — the point of having them. dataDurability: preferred, NEVER required. `required` is a strict quorum: fewer than `number` healthy standbys and writes BLOCK. Postgres here shares three nodes with etcd and the CNI, so one degraded standby — a node reboot, a Longhorn replica rebuild, a Cilium restart — would turn a partial degradation into a total write outage, and on the SSO database that takes kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth with it. `preferred` keeps synchronous commit while a healthy standby is there and relaxes the required count dynamically when one is not. dataDurability is only honoured while standbyNamesPre/standbyNamesPost are unset — both are. NON-DISRUPTIVE: this compiles to `synchronous_standby_names`, a reload-only PostgreSQL parameter. No pod restart, no switchover, no client disconnect. It is the last of the three commits because it wants standbys that already exist and are already on their final nodes. Field shape verified against the live clusters.postgresql.cnpg.io CRD before relying on it (CNPG 1.30.0, byte-identical schema on k3s-prod and k3s-dev): spec.postgresql.synchronous requires `method` (enum any|first) and `number` (x-kubernetes-validation self > 0); `dataDurability` is enum required| preferred. None of the three Clusters previously had a `postgresql:` block, so this adds one rather than merging into existing parameters. DEV IMPACT: identical and non-disruptive, same reload-only path. Verified: yamllint clean; both tiers render the block. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit f1ce19461415e77eb2f1ca5109ebab49edf0860b) |
||
|
jakman85andClaude Opus 5
|
82aff4d053 |
feat(apps): attach the media library and the legacy Nextcloud dataset over NFS
plex, sonarr, radarr and nextcloud each gain one bulk volume. Authored in the catalog and re-rendered, never hand-edited into apps/ (render-golden: 10/10). Layout, the export change the operator must apply, the uid analysis and the validation probes are in docs/BULK-NFS-STORAGE.md; everything there was measured on the host over read-only SSH, not assumed. WHAT THE LAYOUT TURNED OUT TO BE, AND WHY IT DECIDED THE DESIGN /mnt/Storage/plex holds movies/ tv/ photos/ AND downloads/ as SIBLINGS in one dataset — so the arrs' import source and their libraries are one filesystem on the server, and the download client's completed folders are already named radarr/ and tv-sonarr/. The k3s arrs are joining a tree an arr stack has been driving for a long time. That sibling fact is the whole argument for the arrs mounting the export ROOT rather than a subPath per directory. Sonarr and Radarr import by hardlinking or atomically renaming, both same-filesystem operations, and to the kernel "same filesystem" means SAME MOUNT. Two subPath mounts of one export are two mounts: link() across them returns EXDEV regardless of what the server knows, and the arrs do not fail loudly — they degrade to copy-then-delete, rewriting every file over NFS and destroying the seeding copy. The smallest subtree containing both tv/ and downloads/ IS the export root, so the root mount is the MINIMAL mount that preserves hardlinks. Least privilege bounded below by a hard requirement. The cost is stated, not hidden: sonarr can also write /media/movies. plex /mnt/Storage/plex -> /media ro ReadOnlyMany sonarr /mnt/Storage/plex -> /media rw ReadWriteMany radarr /mnt/Storage/plex -> /media rw ReadWriteMany nextcloud /mnt/Storage/nextcloud -> /mnt/legacy-nextcloud ro ReadOnlyMany (subPath: nextcloud) Plex is read-only because it indexes and streams; every write on that tree belongs to the arrs, and saying so removes Plex from the set of things that can damage 8.33 T. All three media consumers use the SAME mountPath deliberately: when qbittorrent is eventually given this export, the arrs and the download client will already agree on absolute paths, which is the other precondition for hardlinked imports. *** /media AND NOT /data — A COLLISION CAUGHT WHILE RENDERING. *** The arrs' Longhorn config PVC renders at the renderer default /data/config. A media mount at /data is mounted OVER it and MASKS the config volume: the pod comes up with its app database shadowed by an 8 T media tree. That discovery also exposed a SEPARATE PRE-EXISTING BUG, reported and deliberately NOT fixed here: LinuxServer arr images run with -data=/config, so /data/config is very likely the wrong home for that PVC and the app has been writing its EPHEMERAL layer — the exact failure catalog/plex.yaml's mountPath comment records already having been bitten by. Plex was fixed; sonarr, radarr, qbittorrent and jackett were not. Fixing it starts those apps from an empty config volume, so it needs its own change with the operator present. This change does not make it worse. UID/GID — THE FAILURE THIS TIER ALWAYS HAS The export is sec=sys + no_root_squash, so the client's numeric uid passes through verbatim and the server's ownership is the entire access story. The tree is 0755 owned 1000:3000, so ONLY uid 1000 can write it. LinuxServer images default to 911. Without intervention sonarr and radarr would have mounted cleanly, gone Ready, and failed EVERY import with EACCES. And fsGroup cannot fix that — the in-tree NFS plugin does NO ownership management, so podSecurityContext.fsGroup is a no-op on an NFS volume. Reaching for it is how this gets misdiagnosed. So PUID=1000/PGID=3000 are added to both arrs (PGID 3000 keeps what they write group-readable to the live non-k3s consumers of the same export), and fsGroup: 3000 is added for the LONGHORN config PVC only, so changing PUID does not depend on the image re-chowning /config. plex already runs as 1000 and reads a 0755 tree; nextcloud reads as www-data. Neither needs work. NEXTCLOUD IS MOUNTED READ-ONLY, WHICH IS A DEVIATION FROM THE BRIEF — ON EVIDENCE The task said nextcloud writes. The dataset says otherwise: of 270 G, 267 G is config_backup/, the rest is config/ (103 M) and config_old/ (3.4 G). There is NO data/ directory. This is the OLD non-k3s instance's configuration plus a backup blob — an archive — and that instance is STILL LIVE on it (exported rw to 192.168.1.6 and .5, which the operator has said must stay active). Two Nextclouds writing one instance's config directory, each with its own database and therefore its own oc_filecache, is unrecoverable corruption. "Do not grant more than needed" lands on ro, and ro is the trivially reversible choice. If the intent is in fact a MIGRATION, that is a larger and more deliberate change than a mount; the one-line flip is documented. Read-only for nextcloud can additionally be enforced SERVER-side (only nextcloud consumes that dataset from k3s), and the sharenfs value in the doc uses ro= for the k3s clients. The media tree cannot: plex and the arrs may land on any node, so the server sees the same addresses for reader and writers. Plex's ro is therefore a client-side guarantee — stated as a limitation, not left implied. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
e8d8601ca6 |
feat(cnpg): required pod anti-affinity + switchover primary rolls
`instances: 3` alone does not keep the three instances apart. All three Clusters used CNPG's default `podAntiAffinityType: preferred`, which is a scheduler HINT — two instances may share a node whenever that scores better, and they WILL on the path that matters: when CNPG recreates an instance after a node loss, the surviving nodes are the only candidates and one of them already runs an instance. `preferred` is how a 3-instance cluster decays back into the single-node failure domain the previous commit exists to remove. `required` makes it a scheduling constraint. On a 3-node cluster with no spare, the cost is that an instance whose node is gone stays Pending until a node returns. That is correct: a replacement squeezed onto an already-occupied node dies with that node while making the cluster LOOK three-way replicated. 2 of 3 instances still serve and CNPG still promotes. ALSO SETS primaryUpdateMethod: switchover. This is not cosmetic. Changing the affinity changes the instance pod spec, so CNPG performs a rolling update, and the CRD default for primaryUpdateMethod is `restart` (verified against the live CNPG 1.30.0 CRD on both clusters) — a delete-and-recreate of the PRIMARY pod, i.e. tens of seconds of no Postgres. On keycloak-db that is an SSO outage, which also means kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. With standbys now present, `switchover` promotes an already-updated streaming replica instead, reducing the roll to a connection reset. Known constraint documented in the manifest: under `switchover` the operator rejects an update that changes imageName and a postgresql parameter in the same reconcile. Verified satisfiable before merge (read-only): k3s-prod and k3s-dev each have 3 Ready, schedulable, untainted nodes, and every pgdata PVC already has a Longhorn replica on all three (replica-soft-anti-affinity=false), so any instance can attach from any node. DEV IMPACT: identical, and dev can absorb it. Longhorn dev nodes are 124.9 GiB each with 152.0 GiB already scheduled under a 200% over-provisioning ceiling (249.8 GiB); the six new 5Gi volumes add 30 GiB scheduled per node -> 182.0, still inside the ceiling, and only ~600 MiB per volume is really consumed. Prod is far looser: 239 -> 269 GiB scheduled against a 829 GiB ceiling. THIS IS THE DISRUPTIVE COMMIT of the three. Land it on its own reconcile, after the scale-out has produced two Ready standbys per Cluster, so the blast radius is one clean switchover per database rather than a scale-out and a roll at once. Expect a brief SSO interruption for keycloak-db. Verified: yamllint clean; both tiers render the new fields; field names, types and enum values checked against the live clusters.postgresql.cnpg.io CRD (CNPG 1.30.0, identical on prod and dev). Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit e1d364238d73e6ee99c9a50108fe8b4e09481a7e) |
||
|
jakman85andClaude Opus 5
|
2590bc4317 |
feat(env): BULK_NFS_SERVER — the one knob prod cannot answer yet, and why it is a placeholder
The bulk tier's PersistentVolumes need a server address, and there is no single correct one: the
two clusters sit on OPPOSITE SIDES of pfSense from the storage host, which has exactly one
interface (vmbr0 192.168.1.9/24, on pfSense's LAN).
dev 192.168.1.40/.41/.42 — the SAME segment. Reaches it directly, and already holds one NFS
export from it (/mnt/Storage/k8s-volumes, in the host's /etc/exports). Proven path, so
base's default is the dev value per this file's existing convention, and dev restates it.
prod 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does not traverse that today; 22 and 8006
answer only because pfSense port-forwards them via 192.168.2.102. Opening the storage path
is a PENDING operator decision whose two candidate outcomes give DIFFERENT values here:
192.168.2.102 (another port-forward) or 192.168.1.9 (a route). So prod holds
TBD-PROD-BULK-NFS-SERVER, this repo's established loud-placeholder convention.
Two findings recorded in the prod overlay because they change what the operator has to do:
* THE EXPORT'S CLIENT LIST IS THE SAME UNDER BOTH OUTCOMES. Measured today from this
workstation (192.168.2.132) through the existing port-forward: the storage host logged the
connection as coming from 192.168.2.132 — a WAN-side address — not from a pfSense LAN address
(`ss -tn` peer 192.168.2.132; the host's only route off-LAN is its default via 192.168.1.1).
pfSense DNATs WITHOUT source NAT, so the NFS server sees prod's REAL node addresses either
way. The `zfs set sharenfs=` work therefore does not have to wait for the routing decision.
* ONE PORT, NOT FOUR. The PVs mount nfsvers=4.1, and NFSv4 carries mount and lock in-protocol —
no rpcbind/111, mountd or statd. Whichever outcome is chosen, it opens tcp/2049 and nothing
else. NFSv3 would have needed four ports, three of them on ephemeral numbers.
*** AND ONE HONEST WEAKNESS, stated in the overlay where an operator will hit it. ***
Unlike the LAN knobs above it, this placeholder does NOT fail at apply time. A PV's
spec.nfs.server is an unvalidated string: the API server accepts it, the PV is created, and the
PVC BINDS (binding is pure control-plane). The failure appears only when a pod mounts —
MountVolume.SetUp "Failed to resolve server TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr
and nextcloud would sit in ContainerCreating on prod. There is no CRD validation to lean on here,
so the guard has to be procedural: set the value before prod reconciles a revision carrying the
bulk volumes. Dev renders a working value and is where these manifests are meant to be proven.
Related finding, NOT fixed here because the file does not exist on main: the static guard this
convention is supposed to lean on, `bootstrap/standup.sh`'s `s3_is_placeholder` family, is absent
from main (it lives on unmerged prod branches), so
conformance/tests/l0-static/placeholder-guard-coupling exits 3 SKIPPED and nothing in the gate
currently refuses a placeholder-shaped overlay value. When standup.sh lands, its guard needs an
arm for this knob and a consumer that calls it — assertions (2) and (3) of that check then cover
this value automatically, since the detector's SEED vocabulary already matches "TBD".
Evidence: `kubectl kustomize clusters/dev` -> BULK_NFS_SERVER: 192.168.1.9;
`kubectl kustomize clusters/prod` -> BULK_NFS_SERVER: TBD-PROD-BULK-NFS-SERVER. Both exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
cec550045b |
feat(storage): the bulk tier becomes real — static NFS PVs, authored once per environment
The `bulk` class has existed since Phase 0 as a deliberate stub: the StorageClass named an NFS
CSI driver that is not installed and a server/share of REPLACE-ME, so plex, sonarr, radarr and
qbittorrent all carry a comment saying "attach the library once OQ-2 is resolved". This is that
resolution — for the case that actually exists here, which is NOT the case the stub anticipated.
The stub assumed DYNAMIC provisioning: ask a CSI driver for a bulk volume and get a fresh empty
subdirectory. That is the wrong shape for this platform's bulk data. The media library is 8.33 TiB
that already exists, on ZFS datasets on the Proxmox host, with live non-k3s consumers. Nothing
needs provisioning; existing datasets need ATTACHING. So a bulk storage entry now renders a
STATIC, pre-bound NFS PersistentVolume plus its PVC, and needs no CSI driver at all (the in-tree
NFS volume source is not deprecated and has no CSI migration).
Schema 1.4.0 adds, on a storage entry:
export absolute NFS export path; REQUIRED when class is bulk, FORBIDDEN otherwise (an `if`
in the schema, so a bulk entry cannot be authored without saying what it attaches).
subPath mount a subtree instead of the volume root — any class.
readOnly any class; on bulk it also sets the PV's nfs.readOnly, so the mount really is `ro`.
The SERVER is deliberately NOT a schema field. It is the `${BULK_NFS_SERVER}` cluster-vars knob,
because dev and prod reach the SAME storage host by different addresses and R-ENV-02 says the
service definition is authored once. A ServiceDefinition says WHICH dataset; the overlay says
where the host is.
Three decisions worth the reader's time, argued in full in the r_bulk_pvs docstring:
* PV/PVC, not an inline `nfs:` pod volume. Inline is fewer objects and would work. It cannot
express `persistentVolumeReclaimPolicy: Retain`, which is the one property most worth having
in git for 8.33 TiB of data with consumers outside this cluster; it puts read-only enforcement
one Deployment edit away from silent removal; and it is invisible to `kubectl get pv,pvc`,
which is where an operator looks for attached external storage.
* storageClassName: "" on both ends, NOT the `bulk` StorageClass. Explicitly empty is the
canonical marker for a statically provisioned volume. It also sidesteps a trap: a
StorageClass's `provisioner` is IMMUTABLE, so re-pointing the deployed `bulk` stub would make
Flux fail on that object forever.
* Access mode is always `…Many` and follows the read/write posture — ReadOnlyMany for a reader,
ReadWriteMany for a writer. One export is consumed by several namespaces AND several nodes at
once, which is exactly what ReadWriteOnce cannot express; which `Many` it is then makes the
intent legible in `kubectl get pv`.
mountOptions are a platform decision held in the renderer rather than a per-service field:
nfsvers=4.1,hard,noatime,timeo=600,retrans=2. NFSv4 carries mount and lock in-protocol, so the
only port that must be reachable is tcp/2049 — no rpcbind/111, mountd or statd. That is a
firewall decision as much as a storage one. `hard` over `soft` because a soft mount converts a
server hiccup into a short read handed to the application.
The contract doc records the trap that will otherwise be discovered in production: fsGroup is a
NO-OP on an NFS volume (the in-tree plugin does no ownership management), so a writable bulk mount
requires the container's own uid/gid to suit the export's ownership. A mount that succeeds and
then cannot write is this tier's characteristic failure.
Every pre-existing golden renders byte-identical: the new keys are emitted only when declared, and
`readOnly` defaults to False. render-golden: 10/10. selftest: PASS.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
f338e91770 |
feat(cnpg): instances 3 on all three Clusters — a standby to fail over to
All three CNPG Clusters ran `instances: 1`. Longhorn's 3-way replication saves the DATA when a node dies; it does not save the SERVICE — a single instance has to be rescheduled, re-attach its volume and replay WAL before Postgres accepts a connection again. With nothing in the spec keeping them apart, all three primaries had drifted onto ONE node (k3s-prod-2 on prod, k3s-dev-3 on dev). On 2026-09-07 that node took SSO, Nextcloud and Semaphore down together, and with SSO went kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Three instances give each Cluster a streaming standby CNPG can promote in seconds, repointing the -rw Service with no volume re-attach and no restore. UNBLOCKED BY WORKING BACKUPS, NOT A SUBSTITUTE FOR THEM. Standbys replicate corruption and deletions faithfully; only the Barman archive is a backup, and adding standbys while archive_mode=on with a failing archive_command would have multiplied the disk-full clock, not hedged it. All three Clusters on both k3s-prod and k3s-dev now report ContinuousArchiving=True with a real firstRecoverabilityPoint (verified read-only, 2026-09-07). Scaling out is non-disruptive on its own: CNPG creates -2 then -3 sequentially, each seeded by pg_basebackup from the RUNNING primary. The primary is not restarted and clients are not disconnected. The anti-affinity change that DOES cause a switchover is deliberately the next commit. DEV IMPACT: identical — dev is on the same shape (three single-instance DBs, all on k3s-dev-3) and gets the same scale-out. Dev has three untainted nodes and Longhorn headroom for it (see the next commit's message for the numbers). Cost: 2 extra 5Gi PVCs per Cluster; ~30 GiB provisioned cluster-wide against live pgdata of ~600 MiB each, on Longhorn with 200% over-provisioning. Verified: yamllint clean; `kubectl kustomize --load-restrictor LoadRestrictionsNone platform-config` and `kubectl kustomize apps/wave-0` both render `instances: 3`. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit ed6fe6cb019b96f677a92448a0347f979d1a6d02) |
||
|
jakman85
|
a624c638ef | merge: docs/node-1-drive-replaced | ||
|
jakman85
|
90b39de527 | merge: fix/swap-off-persists-across-reboot | ||
|
jakman85
|
0247d796a3 | merge: fix/founder-moved-to-node-2 | ||
|
jakman85
|
c9e9dcf960 | merge: fix/r-env-03-overlay-selected-credentials | ||
|
jakman85
|
89cf59ffb7 | docs(conformance): the GIVEN/WHEN/THEN header states the exception it now allows | ||
|
jakman85
|
e319777849 |
docs(pm): draft the R-ENV-03 amendment, and name the gate hole that hid it
The requirement's literal reading ("no prod-specific artifact in a shared path")
forbids the environment-selected PAIR that R-ENV-02 prescribes for a value that
must differ per environment while the manifests stay identical — and a Secret's
difference cannot live in cluster-vars, because it is ciphertext in a SOPS file.
Drafted wording that keeps the guard's teeth (an artifact only prod APPLIES
still fails) and removes the false positive, for the System Architect: the spec
repo is read-only here and CI-gated by R-TEST-06.
Also recorded, checked rather than assumed: the merge that turned R-ENV-03 red
was gated with a touched-requirement subset that never selected R-ENV-03, so
ci/l0.sh Stage 3b filtered the already-red check out of the run that printed
PASS. Replaying that delivery's twelve paths through
fix/touched-requirements-fail-closed's requirement-paths.map still yields no
R-ENV-03, so landing that branch would NOT have caught this. The two changes
that would (unconditional whole-repo L0 stages; a changed conformance test
attributing its own requirement labels) are ci/ work and are left to that
branch, not smuggled in here.
|
||
|
jakman85
|
730d12ddc4 |
fix(conformance): R-ENV-03 shared-path rule tests the harm, not the filename
The blanket regex around RENDERED_DIRS failed any tracked file whose name
carries a delimited `prod` token. That caught the four per-environment backup
credential halves (BACKUP_CREDENTIALS_SUFFIX) even though BOTH halves are
applied to BOTH clusters as base resources and the overlay selects only which
one the shared consumer REFERENCES — i.e. exactly the R-ENV-02 contract the
regex exists to protect. Renaming the files to dodge the regex was rejected as
gaming the guard, so the rule now encodes the property instead.
The fix is not an allowlist of those four paths. A prod-token file in a shared
path is excused only when the tree proves all of:
(A) it declares nothing but Secrets whose names end in the environment suffix
(1) its own directory's kustomization applies it, and EVERY environment value
of the selector resolves to a Secret that some resource of that same
kustomization declares, in the same namespace
(2) nothing names the suffixed Secret — or the suffixed file — literally;
every reference is `<base-name>${VAR}`
(3) that VAR is defined for the base and for every overlay, and prod's value
is the one that selects this file
References are read from parsed YAML values, never raw text: the mechanism is
spelled out by name in comments beside the resources, and a comment references
nothing.
No other assertion is weakened; the check still runs 96 assertions, and the
pass output now notes each excused file and why.
|
||
|
jakman85andClaude Opus 5
|
df1d2ab44d |
docs(prod): node 1's drive was replaced — and §2's exit condition still is not met
Records the 2026-09-07 I/O-starvation incident and the drive swap: the C400 was replaced with a Samsung 850 EVO, and under the same load class (an active 18-replica Longhorn rebuild, with node 1 as the write target) I/O pressure fell from full avg300 11.36 to 3.79 — from ~4x worse than its peers into node 3's band. Updates the three §0 cells that had become false (disk model, drive health, kernel — all three nodes now run 7.1.8, not 6.12) and adds §0c rather than rewriting the 'NOT hardware-symmetric' analysis, which is still the correct reasoning about the state it described. Two things the rebuild taught, recorded so the next one is cheaper: Longhorn refuses a replaced disk with DiskFilesystemChanged and needs disable -> remove -> re-add in that order, and the inventory's [k3s_prod_primary] decides BOTH --cluster-init and k3s_api_host, so rebuilding the founder without moving that designation would have founded a second cluster. The replacement does NOT close §2: etcd and Longhorn still share one SATA spindle. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
69a25cff82 |
fix(node_health): say 'active via <unit>', not a bare parenthetical
The unit name in the swap-active detail read as an unlabelled aside. Naming it as the unit that activated the swap is what makes the line actionable. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c1e36d9353 |
docs(node_baseline): say why the Longhorn claim guard's swap list is now always empty
The claim guard builds node_baseline_lh_protected_devices from ansible_mounts plus `swapon --show`. With swap.target masked, a baselined node has no active swap, so that half of the list contributes nothing — which reads like a guard that quietly stopped guarding. It is not: the claim candidate comes from whatever is mounted at node_baseline_longhorn_claim_mountpoint, and a swap partition is neither mounted nor ext4, so it could never be the candidate. Say so where the next reader will ask. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
9d00a3fcd3 |
fix(node_health): memory.swap blamed the wrong thing, and could not see the defect
The WARN read "roles/node_baseline runs swapoff and neutralises the fstab entry; this node has not been baselined yet". That was a MIS-DIAGNOSIS. All three prod nodes HAD been baselined, had their fstab swap line commented out by the role, and were still swapping 977 MiB each after a reboot. Anyone reading that WARN would have re-run the role that had already run, and watched swap come back next boot. The detail now names the actual mechanism — systemd-gpt-auto-generator re-creating a .swap unit from the partition TYPE GUID 0657fd6d-a4ab-43c4-84e5-0933c84b4f4f at every boot, without reading /etc/fstab — says in as many words that this is not "the node has not been baselined", and points at `node-baseline.yml --tags system`. The check was also blind in the other direction: SwapTotal == 0 is not evidence that swap is disabled, only that nothing has been swapped on yet this boot, so a node freshly swapoff-ed PASSed while still set to swap again on reboot — exactly the state the old role left behind. The collector now reports memory.swap_persistence (swap.target masked?, .swap units the generators emitted this boot, which of them are masked, leftover active fstab swap lines — all world-readable, no root, no systemctl parsing), and the check WARNs on "clean now, swapping after the next reboot". Rendered against every fact shape (swap active + unmasked, swap off + unmasked, after the fix, a dev VM with no swap partition at all, per-unit mask only, persistence unreadable, an older collector's output with the key absent, and /proc/meminfo unreadable) and verified live on k3s-prod-1 post-reboot: [PASS] memory.swap — no swap active and none can be activated: swap.target is masked Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0e92d258dd |
fix(node_baseline): swap-off now survives a reboot — mask swap.target
The role's `swapoff -a` + commented fstab line held only until the next boot,
on every prod node. Measured 2026-09-07: all three came back from a reboot with
977 MiB of swap ACTIVE (k3s-prod-1 /dev/sda4, k3s-prod-3 /dev/sda4, k3s-prod-2
/dev/nvme0n1p4) while their fstab swap entry was commented out by this role. The
role reported success and the setting silently expired.
CAUSE. systemd-gpt-auto-generator claims any GPT partition typed
0657fd6d-a4ab-43c4-84e5-0933c84b4f4f ("Linux swap") from the PARTITION TABLE,
synthesises a .swap unit into /run/systemd/generator.late/ and links it into
swap.target.wants/. It never reads /etc/fstab, so neutralising fstab cannot stop
it. Re-running the generator by hand on k3s-prod-1 says so verbatim:
Probed fstype 'swap' on partition /dev/disk/by-diskseq/1-part4.
Adding swap: /dev/disk/by-diskseq/1-part4
MECHANISM. Mask swap.target. `systemctl show` on the generated unit says its only
inbound dependency is `WantedBy=swap.target` — Requires/RequiredBy/BoundBy empty,
and the matching .device unit has an empty Wants=, so a device appearing does not
pull its swap unit in by itself. swap.target is the single choke point for every
activation path: gpt-auto, systemd-fstab-generator, systemd-cryptsetup-generator.
It is device-shape independent (no device name appears in the change, so one task
covers sda4, nvme0n1p4 and dev's vda VMs), boot-safe (swap.target is Wants=/After=
from sysinit.target and After= from tmp.mount; RequiredBy= is EMPTY on all three
prod and all three dev nodes, and a masked unit reached only via Wants= is skipped,
not failed), and inert if the device is absent or renamed. The generated units are
masked too, as a second independent barrier.
Rejected: `systemd.gpt_auto=0` (disables GPT auto-discovery wholesale, needs a GRUB
edit + reboot to become true, and on DEV would switch off root auto-discovery — dev's
vda1 carries the discoverable GUID 4f68bce3-e8cd-4db1-96e7-fbcaf984b709); clearing
the partition's GPT type GUID (a partition-table write to the live boot disk of a node
serving etcd and Longhorn replicas, and an irreversible decision about that 977 MiB
that belongs to the operator); `systemd.swap=0` (systemd-fstab-generator only — this
swap does not come from fstab).
`swapoff -a` is now conditional on `swapon --show` returning something, so the task
reports honestly instead of hiding a command that ran anyway behind changed_when: false.
PROVEN on k3s-prod-1 (all 18 Longhorn volumes healthy before the reboot):
run 1 changed=2, run 2 changed=0
after `reboot`: /proc/swaps empty, `free` Swap 0B, swap.target masked, the
generated unit still emitted by the generator and still masked, `systemctl
--failed` empty, `systemctl is-system-running` = running.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
08fbe85638 |
fix(ansible): move the founder designation to node 2
Node 1's SSD failed and was replaced; Debian was reinstalled, so node 1 is now a
JOINING node with no cluster state. It was still listed in [k3s_prod_primary],
and that group decides two things at once:
1. which play gets k3s_server_mode: primary -> --cluster-init, which FOUNDS A
NEW CLUSTER, and
2. what k3s_api_host resolves to -- the --server join target.
So re-provisioning node 1 unchanged would have pointed a rebuilt, empty node at
ITSELF and founded a second cluster beside the live one. The role's
`creates: /etc/rancher/k3s/k3s.yaml` guard does not help: on a fresh install that
file is absent, so cluster-init WOULD have run.
Node 2 is the senior surviving server (up since 2026-09-01) and takes the founder
slot. Its own cluster-init task is inert precisely because k3s.yaml already exists
there. Verified after the change: k3s_api_host renders 192.168.2.11, and
groups[k3s_primary_group][0] is k3s-prod-2.
This is a latent gap the role documented but did not handle -- its comment assumes
the founder is always alive ('On a WARM single-node rejoin the founder is up and
answering too'). Rebuilding the founder itself was not covered.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
863af01062 |
docs: the §2 shared-device measurement came back, and it does not hold
infra/PROD-PROVISIONING.md §2 recorded the etcd/Longhorn shared-partition risk as "accepted to be MEASURED, not assumed", with an exit condition. The measurement arrived on 2026-09-07 in production, 2 d 7 h after bring-up, as an outage. §2 now carries "THE MEASUREMENT CAME BACK": the journal evidence, the three-node IO PSI table (11.36 / 0.75 / 1.29), and the honest verdict that the section's own premise — "every prod node is SSD or NVMe, where rebuild I/O costs latency rather than collapsing it" — was right about node 2 and wrong about node 1. A 2011 C400 with 61,440 reallocated sectors and no power-loss protection is not the "SSD" that reasoning assumed, and §0 had already named the three-way drive asymmetry as the fault line. The exit condition is OPEN and the second bullet (dedicated device) applies; that remains the operator's. Also records what the three software mitigations in this branch do and do not buy, including the point that no cgroup or scheduler control arbitrates the ext4 journal both tenants share — and the gap that etcd's own fsync histograms, which the exit condition literally asks for, are not collected because kubeEtcd is disabled for k3s. PSI is the signal that exists. DECISIONS-LOG gains the full entry: the four I/O mechanisms found INERT on these nodes (IOSchedulingClass, IOSchedulingPriority, io.latency, bare IOWeight), the finding that concurrentReplicaRebuildPerNodeLimit was already set and already held during the outage, the third instance of config that looked applied and was not (orphanAutoDeletion), and the three things still requiring the operator. infra/ansible/README.md: CORRECTED a stale claim that k3s-prod-2/-3 "stay commented out" of the inventory. They have been live in [k3s_prod_secondary] since 2026-09-03. That sentence is exactly the kind used to reason about blast radius — `-e k3s_hosts=k3s_prod` targets THREE hosts, not one — so it is replaced with the --list-hosts command that settles it. The new etcd-io-priority play is added to the run order, the layer table and the prod command list. Co-Authored-By: Claude Opus 5 <[email protected]> (cherry picked from commit a00a27a2567ea6a805c8c196b75df2c92a7c63a4) |
||
|
jakman85
|
f7a9229d81 |
fix(ansible): the NVMe node would have died on an undefined variable
Self-review of the role I had just written. On k3s-prod-2 the scheduler path is
deliberately skipped, so the `stat` on /sys/fs/cgroup/system.slice/k3s.service is
skipped too — and a SKIPPED registration has no `.stat` key at all. Ansible
templates `loop:` BEFORE it evaluates `when:`, so
loop: "{{ ['system.slice'] + (['system.slice/k3s.service']
if _eio_k3s_cgroup.stat.exists else []) }}"
raises on that host even though the task would never have run. The role would have
failed on the one node it is designed to leave alone — and it would have failed at
the very end, after doing nothing, which reads like a broken mitigation rather than
an intentional skip.
`| default(false)` on both uses. Exercised offline against a simulated skipped
registration; `--syntax-check` re-run.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit 5ccf5111a54857abb934a64c41332b7aa5046323)
|
||
|
jakman85
|
a819c50ffe |
feat(ansible): arbitrate the shared etcd/Longhorn device — and prove it landed
roles/etcd_io_priority + playbooks/etcd-io-priority.yml, enabled for [k3s_prod]
only. Mitigation for the 2026-09-07 k3s-prod-1 outage: etcd's WAL fdatasync took
3.4-6.7 s (expects 1 s), the node went NotReady while the host stayed up, and
/proc/pressure/io full avg300 read 11.36 there against 0.75 (node 2, NVMe) and
1.29 (node 3).
WHAT IT DOES, on the SATA nodes only:
1. k3s-etcd-io-priority.service (oneshot, Before=k3s.service) switches the
shared device to BFQ and drops nr_requests 64 -> 32.
2. IOWeight=10000 drop-ins on system.slice AND k3s.service, which BFQ turns
into io.bfq.weight=1000 — a 10:1 share against the default-weighted
kubepods.slice where Longhorn's instance-managers live.
3. Reads all of it back out of the kernel and ASSERTS. If the scheduler did
not switch or the weight did not land, the play fails.
THE THREE OBVIOUS ANSWERS ARE INERT ON THESE NODES, ALL CHECKED LIVE, ALL
DELIBERATELY ABSENT:
* IOSchedulingClass=realtime on k3s.service. mq-deadline DOES honour the
priority class, so this looks right — but ioprio is inherited across fork
and kept across exec (block/blk-ioc.c __copy_io copies ioc->ioprio) and
CONTAINERD IS A CHILD OF THE K3S SERVER PROCESS. Verified on k3s-prod-1:
containerd pid 84553 has ppid 84333 = k3s-server, and
ionice -c2 -n0 -- sh -c 'ionice -p $$; sh -c "ionice -p \$\$"'
prints best-effort prio 0 at both levels. Realtime for k3s is realtime for
every container on the node, Longhorn included. No relative gain.
* IOSchedulingPriority (the level inside a class): mq-deadline buckets by
CLASS only, so BE-0 vs BE-4 is a no-op.
* io.latency, which would be the ideal scheduler-independent protection:
CONFIG_BLK_CGROUP_IOLATENCY is not built in Debian trixie's 7.1.8 kernel
(only IOCOST is) and no io.latency file exists under /sys/fs/cgroup.
AND ONE NON-OBVIOUS TRAP: bare IOWeight= is DECORATION here. systemd writes it
to io.weight (blk-iocost — compiled in but disabled, io.cost.qos is empty) and
io.bfq.weight (BFQ — not loaded). Neither reader exists on the stock nodes, so
the scheduler change and the weights have to be one role with one verification.
The value is 10000 rather than a "reasonable" 1000 for the same reason: systemd
rescales 1..10000 onto io.bfq.weight's 1..1000, so 1000 maps to ~101 against
BFQ's unweighted default of 100. The role asserts the result is >= 500.
k3s-prod-2 IS SKIPPED ON PURPOSE. The role discovers the device under
k3s_data_dir per node (verified: prod-1/3 -> /dev/sda5 -> sda, prod-2 ->
/dev/nvme0n1p5 -> nvme0n1) and writes NOTHING on NVMe rather than leaving inert
files behind — BFQ is a poor fit for NVMe and node 2 measured 0.75 IO PSI.
DEV IS UNTOUCHED. Master switch defaults false and is set true only in
group_vars/k3s_prod; dev has a dedicated etcd device (docs/plans/ssd-etcd-
isolation.md) and nothing to arbitrate. Verified with --list-hosts: no extra
vars -> the three dev hosts (where the role is a no-op); -e k3s_hosts=k3s_prod
-> the three prod hosts; --limit k3s-prod-3 -> one.
NOTHING IS RESTARTED. The oneshot writes two sysfs files; the weights go in
through daemon-reload. If daemon-reload turns out not to re-realize a running
k3s.service's cgroup, the verification FAILS and names the rolling restart as
the fix rather than quietly leaving the weight unapplied.
WHAT IT CANNOT DO, stated in the role README and worth repeating: it does not
separate the two tenants. Both still write one ext4 filesystem, and no cgroup
control arbitrates the shared jbd2 journal that etcd's fdatasync waits on. It is
also UNPROVEN on this hardware — it fails loudly if the settings do not land, it
cannot tell you they helped. Apply to k3s-prod-3 first and watch
NodeIOPressureHigh. Replacing the C400 / giving etcd its own device remains the
real fix and the operator's (PROD-PROVISIONING.md §2 exit condition).
Syntax-checked, and the fact-derivation and assert expressions were exercised
offline against all three node shapes plus negative controls (a node still on
mq-deadline fails the gate; IOWeight=1000 -> 101 fails the floor).
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit de8c1f832724c011b03466842e18296de07a6776)
|
||
|
jakman85
|
efc6db6c1f |
feat(obs): alert on I/O pressure — the metric that missed the prod-1 outage
On 2026-09-07 06:05:21Z k3s-prod-1 went NotReady ("Kubelet stopped posting node
status") with the host up, pingable and SSH-able. The cause was disk I/O
starvation of the embedded etcd, from its own journal:
slow fdatasync took=3.56s / 6.71s / 4.95s / 3.40s (expects 1s)
apply request took too long took=8.30s (expects 100ms)
etcdserver: too many requests
/proc/pressure/io FULL avg300 read 11.36 on k3s-prod-1 against 0.75 on
k3s-prod-2 (NVMe) and 1.29 on k3s-prod-3 — all three nodes put /data/k3s
(etcd's WAL) and /data/longhorn (every replica) on ONE partition, and node 1's
is a 2011 consumer SATA SSD with 61,440 reallocated sectors.
The stack alerted on CPU-PSI and memory and had NO I/O alert at all, so the
first signal the operator got was a node already out of the cluster. Adds:
NodeIOPressureHigh full IO PSI > 5% for 10m warning
NodeIOPressureCritical full IO PSI > 10% for 5m critical
Thresholds are measured, not guessed: rate() is a 0-1 fraction, so the outage's
11.36% is 0.1136 and the healthy steady state on the same node is ~0.01.
Verified live that node-exporter already exports
node_pressure_io_{stalled,waiting}_seconds_total for all three prod nodes, so
these rules fire on arrival — no new exporter, no scrape change. (etcd's own
fsync histograms are NOT available: kube-prometheus-stack's kubeEtcd scrape job
is deliberately disabled for k3s, so PSI is the signal we actually have.)
PSI *full* (every runnable task blocked on disk), not *some*: `some` is noisy on
a storage node and did not separate the failing node from its peers.
Dashboard gains the matching I/O row and is retitled accordingly. Not scoped to
prod — dev shares this file and its k3s VMs are HDD-backed, where the same alert
is if anything more likely to earn its keep.
Co-Authored-By: Claude Opus 5 <[email protected]>
(cherry picked from commit cb28fc80e7e2085d2539530d55f90fc92e3b1bba)
|
||
|
jakman85
|
09265e0381 | merge: OIDC issuer/callback URLs become per-environment | ||
|
jakman85andClaude Opus 5
|
e65a76dc51 |
docs(log): record the prod OIDC-URL outage and the rule it establishes
Non-secret, environment-specific values must not live inside an encrypted
Secret: Flux's ${BASE_DOMAIN} substitution rewrites rendered manifest text and
cannot reach inside ciphertext, so such a value silently freezes at whatever
one environment happened to need. Dev was correct only by coincidence (dev IS
the apex); prod was wrong from its first reconcile, for both Headlamp and
Semaphore. StrictPostBuildSubstitutions cannot catch this — there is no
unresolved ${...} to complain about.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
bc9c93d2ab |
fix(keycloak): the reconciler could not read its own Secret
Shipped in the previous delivery and failing on every run since: FATAL: cannot read /etc/keycloak-realm-clients (PermissionError) BackoffLimitExceeded The client-secrets volume used defaultMode 0400 — owner-only — but the owner of a projected Secret volume is root, while the pod runs runAsUser/runAsGroup 65532. The sibling script volume worked only because it is 0444. So the healer that exists to correct client-secret drift was itself inert. It failed CLOSED (no writes attempted, no drift introduced), and the two alerts shipped alongside it are exactly what surfaces this: KeycloakClientSecretReconciler Failing fires on the failed Jobs. The observability caught its own subject. Fixed with fsGroup 65532 + defaultMode 0440 rather than relaxing to world-readable 0444: fsGroup makes the projected volume group-owned by the running user, so the secret stays unreadable to any other uid in the container. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6548f9a89c |
fix(semaphore): per-environment OIDC URLs — same latent break as Headlamp
Semaphore's whole SEMAPHORE_OIDC_PROVIDERS JSON blob lived inside the SOPS-encrypted
semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
client secret. That buried two environment-specific, NON-SECRET URLs in ciphertext —
provider_url (the realm issuer) and redirect_url — where Flux's ${BASE_DOMAIN} postBuild
substitution cannot reach. Both were frozen at `home.keithjackson.dev`, so prod's Semaphore
was pointed at dev's Keycloak and dev's own redirect URI: identical root cause to the
Headlamp login timeout, not yet reported only because nobody had tried prod SSO.
FIX: split the blob. The URLs are public and now live as a plaintext literal in
catalog/semaphore.yaml (rendered to the apps/wave-1 golden), ${BASE_DOMAIN}-substituted like
every other hostname. Only the client secret stays encrypted, in its own key
`oidc-client-secret`, and the KUBELET splices it back via dependent-env-var expansion:
$(SEMAPHORE_OIDC_CLIENT_SECRET) resolves against the secretKeyRef env var declared
immediately above it (order is load-bearing — the kubelet resolves env in list order and
leaves a forward reference as literal text).
Verified: the new key holds the same value as keycloak-realm-clients/semaphore-client-secret
(the mirror invariant), Semaphore v2.19.10's util.OidcProvider consumes client_secret as
before, and `flux envsubst --strict` leaves $(...) untouched while resolving ${BASE_DOMAIN} —
dev renders keycloak.home.keithjackson.dev, prod keycloak.prod.home.keithjackson.dev.
apps/wave-1/semaphore.yaml is a render golden; regenerated with the canonical
`catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
bfa39bc5d0 |
fix(headlamp): per-environment OIDC URLs — prod was dialing DEV's Keycloak
The operator could not log in to Headlamp on prod: Get "https://keycloak.home.keithjackson.dev/realms/homelab/.well-known/openid-configuration": dial tcp 192.168.1.53:443: i/o timeout ROOT CAUSE: OIDC_ISSUER_URL and OIDC_CALLBACK_URL were keys INSIDE the SOPS-encrypted headlamp-oidc-credentials.sops.yaml. Flux's ${BASE_DOMAIN} postBuild substitution rewrites the RENDERED manifest text and can never reach inside ciphertext, so both values were frozen at the apex `home.keithjackson.dev` in BOTH environments. Dev IS the apex, so dev looked correct; prod (BASE_DOMAIN = prod.home.keithjackson.dev) therefore pointed at dev's Keycloak at 192.168.1.53 — on the dev LAN behind pfSense, unroutable from prod's 192.168.2.0/24. FIX: an issuer URL and a callback URL are PUBLIC. Move both out of the Secret into the HelmRelease's spec.values.env, written with ${BASE_DOMAIN} so substitution resolves them per environment. The Secret keeps only what is genuinely secret or environment-invariant (OIDC_CLIENT_SECRET, OIDC_CLIENT_ID, OIDC_SCOPES). Exactly ONE source per variable: the chart renders `env:` after `envFrom:`, and an explicit env entry silently WINS over the same name from envFrom — so the two sets must not overlap, and they no longer do. The five load-bearing names the chart's $(...) args require (OIDC_CLIENT_ID/CLIENT_SECRET/ISSUER_URL/SCOPES/CALLBACK_URL) are all still delivered; verified by rendering platform/headlamp per overlay, `flux envsubst --strict`, and `helm template` of chart 0.41.0 against the substituted values. Side effect, deliberate: naming OIDC_CALLBACK_URL in `env:` makes the chart finally EMIT `-oidc-callback-url=$(OIDC_CALLBACK_URL)` — it gates that arg on the value being set via `env:`/`config.oidc`, which a Secret key alone never tripped, so the Secret's callback URL had been dead weight. The value is byte-identical to the single redirectUri registered on the Keycloak `headlamp` client (also ${BASE_DOMAIN}-templated), the only URI Keycloak accepts. [parity] holds: prod's kube-apiserver oidc-issuer-url is an install-time k3s flag from k3s_base_domain=prod.home.keithjackson.dev, which is exactly what Headlamp now uses. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
3ca4ad6011 | merge: R2 off-cluster backups + per-environment backup credentials | ||
|
jakman85andClaude Opus 5
|
708a05dfc1 |
fix(devcontainer): restore multi-cluster kubeconfig selection
infra/devcontainer/kubeconfig.sh was never staged into the delivery-A commit, so the branch built on main carried main's single-cluster tooling and `--context k3s-prod` failed with 'context does not exist' mid-incident. Not Flux-reconciled, so it never affected the cluster — but it breaks prod diagnosis from a checkout of this branch, which is when you need it most. Carries the concurrency fix: the merged kubeconfig holds both contexts and NO current-context, with each invocation getting a per-invocation selector file prepended to KUBECONFIG. Two concurrent dev.sh runs with different CLUSTER values previously overwrote each other's current-context mid-command — reproduced live, a CLUSTER=dev reader returned k3s-prod nodes on 3 of 7 iterations. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
161430d915 |
feat(backup): the overlay picks the credential — BACKUP_CREDENTIALS_SUFFIX
The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).
cluster-vars BACKUP_CREDENTIALS_SUFFIX base "" | dev "" | prod "-prod"
velero HelmRelease credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
3x CNPG Cluster s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.
THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.
Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).
VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:
strict envsubst PASSES for all 8 (env x tier) combinations
dev velero existingSecret velero-cloud-credentials
dev keycloak/nextcloud/semaphore-db <db>-backup-credentials (both key refs each)
prod velero existingSecret velero-cloud-credentials-prod
prod keycloak/nextcloud/semaphore-db <db>-backup-credentials-prod (both key refs each)
both all EIGHT Secret objects present in the rendered output (dev and prod alike)
no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
escapes in apps/wave-0, which are pre-existing and unrelated.
Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
dev velero s3://velero (root) + cnpg s3://cnpg @ http://192.168.1.9:9000 region us-east-1
prod velero k3s-prod-backup/velero + barman prefix @ https://<acct>.r2.cloudflarestorage.com region auto
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e7c4a95c2d |
feat(backup): the four backup credentials become TWO objects each, one per environment
The four backup-credential SOPS Secrets are SHARED BASE manifests: platform/ and apps/ are
rendered by the same Flux tiers on both clusters, and the ONLY per-environment difference in
this repo is cluster-vars substitution (R-ENV-02). So writing the R2 token into all four
(0a6a862) did not "give prod credentials" — it REPLACED DEV'S. Merging that as-is would have
handed dev's Velero and all three dev CNPG Clusters an R2 key id to sign MinIO requests with:
403 on every WAL segment, and archive_mode=on turns that into a disk-full clock on the pgdata
PVC, not a degraded backup. Dev's PROVEN DR restore (R-STATE-03) is the thing that would have
broken.
This commit splits the material. Each credential is now TWO Secret objects, both applied to
BOTH clusters, differing only in name suffix:
dev -> velero-cloud-credentials keycloak/nextcloud/semaphore-db-backup-credentials
prod -> velero-cloud-credentials-prod ...-db-backup-credentials-prod
The consumer picks one by name; that wiring is the NEXT commit, so this one changes no
rendered reference and is a no-op for both live clusters.
WHY DEV KEEPS THE UNSUFFIXED NAME. Dev is live. Renaming its Secret would change
`s3Credentials.accessKeyId.name` on three running CNPG Clusters — which projects into the
instance pods' environment and therefore ROLLS keycloak-db, nextcloud-db and semaphore-db —
and would roll the velero Deployment, all to rename an object whose contents do not change.
This repo already makes exactly this call for dev's live identifiers (EXTERNALDNS_TXT_OWNER_ID
"default", VELERO_BACKUP_PREFIX ""): the live cluster keeps the name it is running, and the
NEW environment takes the new one. The asymmetry is deliberate and documented at each
kustomization.
WHERE THE MATERIAL CAME FROM. Neither side is invented:
- the four unsuffixed files are restored byte-for-byte from `main` (the MinIO material as
it stood before 0a6a862), so dev's Secrets are IDENTICAL to what dev is running;
- the four `-prod` files carry the R2 token from 0a6a862 (already on this branch as
|
||
|
jakman85andClaude Opus 5
|
0a0527e7db |
feat(backup): prod's off-cluster DR lands on R2 — and stops the WAL clock
Prod has been archiving WAL to `TBD-PROD-S3-BACKUP-ENDPOINT` since bootstrap. Verified live 2026-09-07 on all three CNPG Clusters: ContinuousArchiving=False, "unexpected failure invoking barman-cloud-wal-archive: exit status 4". Because archive_mode is ON, that is not a degraded backup — it is a DISK-FULL CLOCK. Postgres will not recycle a segment it has not archived, so keycloak-db's pgdata PVC is at 579 MiB used / 4360 MiB free and climbing monotonically. When it fills, the primary PANICs and takes SSO down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Wiring the endpoint is what lets Postgres drain the whole accumulated backlog in one pass. THE KNOB LAYOUT (three new, one repurposed) and why it is shaped this way. The operator created exactly ONE R2 bucket, `k3s-prod-backup`, but the two backup tools want differently-SHAPED values: the Velero chart takes `bucket` and `prefix` as separate fields, while CNPG's barmanObjectStore.destinationPath is one full s3:// URL whose first path segment IS the bucket. So: S3_BACKUP_ENDPOINT dev http://192.168.1.9:9000 prod https://<acct>.r2... S3_BACKUP_REGION dev us-east-1 prod auto VELERO_BACKUP_BUCKET dev velero prod k3s-prod-backup VELERO_BACKUP_PREFIX dev "" prod velero CNPG_BACKUP_PATH dev s3://cnpg prod s3://k3s-prod-backup/barman There is deliberately NO separate `S3_BACKUP_BUCKET`: CNPG's bucket already lives inside CNPG_BACKUP_PATH, and a second knob holding the same bucket name twice is a rotation waiting to drift. Three knobs, each shaped for its own consumer, and no two of them can contradict each other. TENANCY. Inside k3s-prod-backup, velero and barman get disjoint TOP-LEVEL prefixes — s3://k3s-prod-backup/velero/ and s3://k3s-prod-backup/barman/<cluster>/ — with the bucket ROOT left empty so a misrendered prefix lands somewhere obviously wrong instead of silently colliding. This is not tidiness: Velero ADOPTS every backup under its location and TTL-DELETES it on retention, so an overlapping prefix would let it reap WAL segments; and interleaving two Postgres servers' WAL into one prefix corrupts the chain outright. Dev↔prod isolation is doubly independent — different endpoint AND different bucket names — so neither cluster can reach the other's objects even with a mis-substituted prefix. DEV IS UNCHANGED, BYTE-FOR-BYTE, and that is asserted rather than assumed. Base defaults are dev's LIVE values (verified against the running cluster: BSL bucket=velero with NO prefix, region us-east-1; all three CNPG Clusters ContinuousArchiving=True at s3://cnpg on the MinIO), and the dev overlay now restates all five explicitly for the same reason LONGHORN_DATA_PATH is restated — a silent drift in base would ORPHAN the backups dev's PROVEN restore depends on. The empty-prefix case was the one real risk, and it is settled empirically: with prefix "" the chart omits objectStorage.prefix from the rendered BackupStorageLocation ENTIRELY (helm template, chart 12.1.0), so dev's live BSL is unchanged, not merely equivalent. R2's TWO GOTCHAS, both wired up front: - region must be exactly `auto`. head-bucket answers {"BucketRegion":"WNAM"}, which is a PHYSICAL LOCATION HINT, not a signing region. This was hardcoded us-east-1 inside the Velero HelmRelease — correct-by-accident for MinIO, wrong for anything else — so it is a knob now. - checksumAlgorithm: "". velero-plugin-for-aws v1.14.2 bundles an AWS SDK with flexible checksums on by default and sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT; R2 rejects those with HTTP 400 InvalidArgument, failing every backup and every kopia blob upload. It must be PRESENT-but-empty (the plugin branches on the key existing), and the chart does emit an empty value — verified by rendering it. Harmless against MinIO, so it is unconditional rather than a fourth knob. WHAT THIS DOES NOT DO. Velero stays SUSPENDED on prod. `spec.suspend: true` is owned by field manager `flux` (verified: kustomize-controller's managedFields do not include f:suspend), so server-side apply will not strip it, and nothing here declares suspension — which would also trip R-ENV-03, since a prod overlay may patch only cluster-vars and a Flux Kustomization's `decryption`. Unsuspending is the operator's, after `standup.sh --s3-endpoint <r2> s3check` proves a round trip. NO DNS KNOB IS TOUCHED. BASE_DOMAIN stays prod.home.keithjackson.dev and every EXTERNALDNS_* value is untouched. The apex inversion on work/prod-standup-campaign was deliberately NOT carried: this branch ports that branch's R2 reasoning knob by knob, and merges nothing. VERIFIED: ci/l0.sh green (flux-postbuild-substitution 272 assertions, 0 hazards, 33 knobs defined / 28 referenced, every new token resolved in BOTH overlays); render-golden, lint, schema, spec-immutability green; touched requirement R-STATE-03 covered. Refs R-STATE-03, R-IDP-05, R-ENV-02, R-ENV-03. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
dc2d1ce17e |
fix(conformance): the R-NET-04 scan descended into nested agent worktrees
TRACKED_YAML rglob'd from the repo root excluding only .git and _artifacts, so it also walked .claude/worktrees/agent-*/ — the isolated worktrees created for agent runs. The check counted 5 CiliumEgressGatewayPolicy objects where exactly 1 is authored, failing R-NET-04 for a reason unrelated to egress steering. Now skips dot-DIRECTORIES (covering .git, .claude, .cache) while still matching dot-FILES such as .sops.yaml. .claude/ is gitignored, so nothing beneath it is authored surface. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6a8953e288 |
feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision
|
||
|
jakman85andClaude Opus 5
|
9739e9be91 |
chore(backup): R2 credentials for the off-cluster DR target
Written by platform/backup/set-r2-credentials.sh (operator, 2026-09-07T05:41:34Z) into all four consumers, in the two shapes they need: velero wants an AWS INI file, CNPG/barman wants discrete ACCESS_KEY_ID / ACCESS_SECRET_KEY. All four verified to carry the SAME 32-character key id, so no consumer was missed. Proven from a pod INSIDE prod against https://7ce452a1b3d6c86daf2c020edcb82d37.r2.cloudflarestorage.com: head-bucket on k3s-prod-backup returns BucketRegion WNAM, and PUT and DELETE both succeed. Byte-integrity of GET is NOT yet proven (the aws-cli image has no cmp); s3check is what will establish it. Credentials only — no endpoint or bucket wiring. That is feat/r2-backup-prod. Co-Authored-By: Claude Opus 5 <[email protected]> |