Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).
New cluster-vars knobs (base default = dev literal, per the file's existing convention):
K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml
CILIUM_LB_POOL_STOP "
UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml
CRAFTY_L4_VIP platform/crafty/service-l4.yaml
S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
semaphore}-db.yaml, and platform/velero/helmrelease.yaml
Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.
clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.
PROD ADDRESS PLAN (operator, 2026-08-31):
nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
Cilium LB pool .20-.30
CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.
Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.
DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.
Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).
Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
failed ... write to WAL: no space left on device". Scrapes still return 200
(targets show healthy, lastScrape=now) but samples are never stored, so the
TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
present) — the rules are correct; the monitor was simply blind.
`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.
- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale
Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Parameterize the in-cluster split-horizon target as GATEWAY_VIP (R-ENV-02): base
default = dev L2-announced Gateway VIP (10.73.0.240, first IP of the Cilium LB pool).
coredns-custom answers *.home.keithjackson.dev with ${GATEWAY_VIP} so the unified graph
holds across dev/prod via cluster-vars only. Prod overlay carries a TODO to set the prod
LAN VIP alongside the prod L2 pool range when hardware is built.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The in-cluster Gluetun gateway is retired; VPN egress is now a node-level host
OpenVPN split-tunnel (Ansible role vpn_egress_gateway) steered by a Cilium
Egress Gateway policy with a host nftables kill-switch (ADR-0015). Credentials
moved host-side to the Ansible vault. Update docs/comments only:
- platform/vpn-egress/README.md: rewrite for ADR-0015 (live + conformance-green);
drop the resolved "option 1 not viable with Gluetun" finding to a brief history note.
- platform/kustomization.yaml: Phase 3 comment -> node tunnel + Cilium steering.
- platform/kyverno/.../require-egress-class-label.yaml: ADR-0013 -> ADR-0015 steering.
- schemas/servicedefinition.schema.json: egress=vpn description -> node tunnel + Cilium.
- clusters/base/cluster-vars.yaml: reframe PIA_REGION / CLUSTER_POD_CIDR WHY comments
to ADR-0015 (Cilium excludedCIDRs); both keys retained (still referenced/documented).
- platform/vpn-egress/namespace.yaml: no privileged gateway pod under ADR-0015.
- pia-credentials.sops.yaml / pia-custom-ovpn.sops.yaml: comment headers only note
they are now UNUSED (creds host-side), retained pending cleanup; payloads untouched.
No runtime manifest behavior, alert exprs, kustomization resources, or SOPS payloads
changed. No spec-repo edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
PIA's PF gateway 10.31.128.1 sits inside the /8 cluster-pool, so
FIREWALL_OUTBOUND_SUBNETS=10.0.0.0/8 made gluetun route it around the tunnel and
port forwarding timed out (getSignature to 10.31.128.1:19999). /16 covers actual
pod allocations and excludes PIA's 10.31.x.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CORRECTION: the connection failures were gluetun dialing RETIRED PIA IPs (46.246.x
= dead → timeouts, which I mis-read as a network block). A fresh PIA .ovpn connects
from the same host, and PIA's current SE Stockholm servers (158.173.x) are reachable.
Cilium was independently cleared (default-allow, host-fw off, Hubble FORWARDED).
Fix: UPDATER_PERIOD=24h so gluetun fetches PIA's current servers; keep
OPENVPN_MTU/MSSFIX=1350 for the real Cilium-VXLAN 1450 pod MTU (data channel);
PIA default UDP port. Re-enable vpn-egress. Full correction in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Decisive diagnosis: OpenVPN's outbound init packets are 14 bytes with zero
replies (so MTU was never the blocker), pod UDP egress is healthy (NTP replies),
but TCP 501/443 to PIA IPs time out while Google/Cloudflare are OPEN — across many
PIA servers. This 192.168.2.x WiFi segment blackholes PIA's IP ranges (and
intercepts DNS). Not code-fixable. Reverted the port/MTU red-herring env to PIA
defaults; disabled vpn-egress pending an operator network fix (move to the
unfiltered 192.168.1.x LAN, or unblock PIA on the router). Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Port 53 also got no PIA replies: this network transparently intercepts outbound
:53, so dig works but OpenVPN-over-53 is swallowed. An NTP/123 probe from the host
returned fine, proving arbitrary non-53 UDP works bidirectionally (so it's
port-specific: 1197 conflict + 53 DNS-intercept, not a UDP block). Move to 8080
(community-verified for PIA, non-DNS, non-1197).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Wiring the port via "${PIA_OPENVPN_PORT}" made Flux/kustomize emit an unquoted
53 (int) after substitution, which k8s rejects for an env value ("expected
string, got 53"). Same gotcha as the Longhorn numberOfReplicas fix. Hardcode the
quoted string; cluster-var removed with a NOTE explaining why.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator identified the block: the home network NATs its own OpenVPN server on
WAN:1197, colliding with the cluster's outbound PIA-on-1197 flow (port-specific,
not a general UDP block — UDP/53 worked in tcpdump). PIA accepts OpenVPN on
53/1194/1197/1198/8080/9201; chose 53 (empirically proven to traverse this net).
- cluster-vars: add PIA_OPENVPN_PORT="53"
- gluetun: OPENVPN_ENDPOINT_PORT=${PIA_OPENVPN_PORT}
- re-enable vpn-egress; DECISIONS-LOG resolution noted
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gluetun's PIA OpenVPN provider rejects "Sweden"; the valid PF-capable Sweden
region identifier is "SE Stockholm" ("SE Streaming Optimized" lacks port
forwarding). Preserves the operator's Sweden choice.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
gluetun-gateway.yaml referenced ${PIA_REGION}/${CLUSTER_POD_CIDR} but they were never defined
in cluster-vars, and the README didn't say where to set them. Add both to
clusters/base/cluster-vars.yaml (ConfigMap data:, string values, Flux postBuild substitution).
CLUSTER_POD_CIDR is 10.0.0.0/8 (Cilium cluster-pool — pods get 10.x, confirmed live), NOT the
k3s 10.42.0.0/16 I'd assumed. README now names the file + format + per-env override + PF-region note.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Phase-2 exit tests (were Phase-0 scaffolds), authored against the live platform:
- trusted-tls (R-TLS-01/03): in-cluster probe dials the gateway Service (env-agnostic
via --connect-to) with wildcard SNI and verifies the served cert CHAINS TO THE LE
STAGING ROOT (operator decision A; dev uses LE-staging per ADR-0006). Fixture embeds
the '(STAGING) Pretend Pear X1' root. Probes public + internal-only hostnames (R-TLS-03).
PASSES.
- dns-exposure (R-DNS-02): a LoadBalancer Service with an external-dns hostname; polls
public DNS until ExternalDNS auto-publishes the record; teardown + policy=sync removes it.
ExternalDNS fix (root cause of the DNS test): domain-filter was the platform SUBDOMAIN
home.keithjackson.dev, which filters OUT the actual Cloudflare zone keithjackson.dev
('no hosted zone matching record DNS Name' — record generated but skipped). Add a
DNS_ZONE cluster-var (keithjackson.dev) and set domainFilters to it. policy=sync +
txt-registry still only manage ExternalDNS-owned records (the pre-existing wildcard is
untouched). Resolves the file's WS-NET R-DNS-02 TODO.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Kyverno HelmRelease install failed at Helm templating: config.webhooks was a
LIST [{failurePolicy, namespaceSelector}], but the kyverno 3.3.9 chart expects
config.webhooks to be a MAP (webhooks.namespaceSelector) — the chart helper reads
.namespaceSelector off the value directly ('can't evaluate field namespaceSelector
in type []interface {}'). Also, failurePolicy is not a per-webhook field here; the
chart's global switch is config.forceFailurePolicyIgnore.enabled (bool).
Fix: config.webhooks -> map with namespaceSelector (system+platform excludes);
add config.forceFailurePolicyIgnore.enabled = ${KYVERNO_FORCE_FAILURE_POLICY_IGNORE}.
Renamed the cluster-var KYVERNO_WEBHOOK_FAILURE_POLICY (Fail/Ignore) ->
KYVERNO_FORCE_FAILURE_POLICY_IGNORE (false base / true dev). Verified with
helm template (renders 73 objects, no error); gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Prep only — nothing deployed; running cluster untouched.
Reconciliation graph (operator-approved Option A): author clusters/dev/rollout/
= 4 per-component Flux Kustomizations (kyverno->cilium->longhorn->monitoring,
monitoring dependsOn longhorn). Dev overlay $patch:delete's the monolithic base
platform/apps and adds ./rollout, so only the no-secret tier reconciles; the
secret-blocked components are simply not created until hand-offs arrive. The
no-secret tier carries no SOPS content, so it needs no sops-age key. prod is
untouched (still builds the monolith). Convergence path coded in (delete rollout/
+ the two deletes) — R-ENV-02 divergence flagged for architect review.
Component hardening via cluster-vars (prod keeps strict posture):
- KYVERNO_VALIDATION_ACTION Enforce->Audit(dev), _WEBHOOK_FAILURE_POLICY
Fail->Ignore(dev), _ADMISSION_REPLICAS 3->1(dev); webhook namespaceSelector
excludes system+platform namespaces.
- MONITORING_STORAGE_CLASS var; GRAFANA_ADMIN_SECRET var (dev ''=chart default);
Alertmanager SMTP/secret deferred to Phase-4 with verbatim re-enable markers.
Cilium parity verified against the live CNI (no-op adopt). Validated:
kubectl kustomize dev/prod/all-components, envsubst->YAML parse, make -C ci gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Flux postBuild.substituteFrom is case-sensitive; platform/ components reference
${UPPERCASE} tokens but cluster-vars keys were lowercase -> none would resolve
(literal ${...} would reach the HelmReleases). Uppercased the ConfigMap data
keys + the dev/prod overlay patch paths. Every component ref now resolves;
clusters/dev & clusters/prod build; gate green. Clears the K1 landmine before
any platform un-suspend.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 5