c8c1d2935d2c2c20758f9695cd8265901ac8efe2
21
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
9b8979a427 |
feat(prod): temp Gitea source + wireless disabled + flux bootstrap path
Autonomous prod bring-up, operator away. Three changes that together clear the last blockers before `flux bootstrap`. === 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared === clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable (Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT. clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done looks like'. Repo verified PUBLIC with anonymous read two independent ways (API "private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef is needed — documented so a future auth failure is not misread as a network fault. bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED libvirt NAT, and forbade agent use; the operator has now directed this for prod. === 2. WIRELESS DISABLED ON PROD (operator-reported) === k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting /etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed nothing' the contended branch exists to prevent. Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a wireless interface and nothing else, take links down and unload modules with NO reboot, then report the resulting state. dhcpcd is now in the contended set without weakening fail-closed. Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down. === 3. flux_bootstrap PATH PARAMETERISED === bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide. DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level canary and a deep-path content canary) confirming the harness detects change — and both canaries proven restored. Honest scope note recorded: kustomize strips comments, so the harness proves the built stream, not source comments. L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0. === STILL BLOCKING A FULL PROD RECONCILE (operator decisions) === - S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at 192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a workstation container would create false confidence. Velero/CNPG backups will fail loudly. - Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl SSO will not work on prod until split-horizon is solved for it (§4). - The temp Gitea is a single container on the operator's workstation — a single point of failure for prod reconciliation, acceptable for bring-up only. Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
05baae90ec |
feat(prod): kernel fix + disk layout — prod nodes ready for k3s
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
b19c5db9ae |
feat(prod): node 3 onboarded — and it is NOT immune; BIOS-setting hypothesis disproven
k3s-prod-3 online at the planned 192.168.2.12 (serial 8CC015307N, same chassis/BIOS/CPU),
enabled in inventory, vaulted become confirmed, package floor installed. Health: 0 FAIL /
4 WARN / 14 PASS / 1 SKIP — the only node with zero FAILs. WD Blue WDS500G2B0A SATA SSD at
100% life, 3213 h, 0 grown bad blocks, 0 erase fails. Matched Crucial 2x16 GB pair — the only
node with symmetric memory.
*** NODE 3 IS NOT IMMUNE TO THE §0b BUG. *** Its firmware DOES publish a MAT
(MEMATTR=0xbeece018); the ABSENCE of a memattr diagnostic line means the kernel read a VALID
table, which is the opposite of immunity. There are THREE outcomes, not two: bail-out on a
garbage version >2, a valid parse with a sane size, and the catastrophic path. Seven boots across
three machines show the MAT address MOVES BOOT-TO-BOOT on every node with settings constant, and
each node has been seen on more than one path. Blast radius widens from two nodes to ALL THREE;
all three need the >= 6.13 kernel.
THE CORRUPTING BYTES ARE THE OS BOOT CHAIN, not firmware and not DRAM: garbage version
1463964749 = 0x5742544D appeared on node 2 AND node 3 — two different machines, identical bytes.
The overwriter is the shared shim/GRUB/EFI-stub/initrd placement writing into the page the
firmware pointed at. So initrd size, module set and boot device are part of the causal chain.
BIOS-SETTING HYPOTHESIS: DISPROVEN, and the three-way diff cannot support it even in principle.
All 240 attributes read on all three; firmware identical fleet-wide. The discriminating set (both
working nodes agree, broken differs) is seven attributes, of which:
- Fast Boot is the ONLY live candidate, and only as a probability-shifter — HP documents it
'skips some preboot initialization steps', which changes the heap top and hence where the MAT
lands. Different mechanism from the previously-killed MRC-caching theory; that stays dead.
- Legacy Boot Options is AFFIRMATIVELY INERT: 'Configure Legacy Support and Secure Boot' =
'Legacy Support Disable and Secure Boot Enable' on ALL THREE, Option ROM Launch Policy =
'All UEFI' on all three. No CSM is built on node 2 either. A naive diff gets this wrong.
- the rest are coincidence. After Power Loss is ELIMINATED by node 3, which works and shares
node 2's Power Off — though it is now a §1 checklist violation on TWO nodes.
FATAL CONFOUND: the 1=3!=2 partition is perfectly collinear with node 2 being the only
single-DIMM AND the only NVMe-booting node. Adding node 3 did not break the confound, it DOUBLED
it. No setting can be the cause anyway: node 2 booted good once and bad twice with settings held
constant.
EXPERIMENT DESIGN RECORDED because the obvious approach is statistically worthless: p(bad) is
known only to n=3, 95% CI ~0.09-0.99, so even 8 clean boots prove nothing. Elimination needs ~20
boots/arm; an odds shift needs ~50. Zero-touch first step: 20 reboots of node 2 unchanged, logging
MEMATTR address vs outcome — both failures were at exactly 0xbeeb0018 and all five successes
elsewhere, so if outcome maps onto a narrow address range the mechanism is settled without
touching anything.
ROLE BUG FIXED: node 3's WD drive reports a SMART threshold for only 1 of 25 attributes, and
smartctl OMITS the key rather than emitting 0 — summary.txt.j2 dereferenced a.thresh bare in five
places and the report task died, so no artifact was written. Added a thr() macro rendering 'n/a'
(deliberately not 0, which would read as a real ATA cannot-trip threshold). Would have hit any
drive of this class.
Also: three nodes, three different disk models, only one NVMe, and every node has a single disk
with no unpartitioned spare — so §2's identical-layout requirement is unmet on all three.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
cb69a53825 |
feat(prod): enable node 2 + HARDWARE FAULT — it addresses 2.6 GiB of its 16 GB
k3s-prod-2 (192.168.2.11) came online 2026-09-01. Enabled in the inventory, baseline package
floor installed, first health run completed. Vaulted become works on it unchanged.
*** OPEN FAULT: the node can only address ~2.6 GiB of a healthy, correctly-installed 16 GB
DIMM. *** Recorded in PROD-PROVISIONING §0b.
dmidecode 16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s (DIMM3 empty)
BIOS e820 reports ~16 GiB usable, incl. 0x100000000-0x42f7fffff
kernel Memory: 2568544K/16546088K available (13958980K RESERVED)
last_pfn revised DOWN 0x42f800 (~16.7 GiB) -> 0xc3c10 (~3.06 GiB) in early boot
zones Normal present 3340288 pages (~12.7 GiB) but MANAGED 0
MemTotal 2,753,284 kB = 2.6 GiB
The DIMM is NOT the problem: it is present, correctly identified, and at rated speed. The
kernel sees all 16 GB and then reserves 13.3 GiB. Everything below ~3 GiB usable, everything
above 4 GiB present-but-unmanaged — the signature of memory remapping above 4 GB not being
applied, not of failing memory. Node 1 is the control: identical chassis, identical BIOS R21
Ver. 02.26.00, identical CPU, and it addresses its full 31 GiB. Per-machine state, not a
firmware-version defect.
WARNING RECORDED: do NOT assume the incoming 32 GB fixes this. If the fault is the slot, the
board or a BIOS setting rather than the DIMM, 32 GB presents identically and the money is
wasted. Verify MemTotal after any RAM change.
ALSO FOUND — the prod nodes are NOT hardware-symmetric:
node 1 Micron C400 512GB SATA SSD (~2011) — 69% life, 15 grown bad blocks, 187 erase
failures, 32 unexpected power losses
node 2 Samsung PM9A1 512GB NVMe — 96% life, 2142 power-on hours, 34 unexpected
power losses, overall-health PASSED
Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires
an identical disk layout across all three nodes because Longhorn replica placement assumes
symmetry, so this asymmetry is now a live input to the disk decision rather than a footnote.
Node 2's NVMe is the drive node 1 should have had.
Node 2 otherwise: 0 FAIL / 3 WARN / 12 PASS / 2 SKIP. Same three WARNs as node 1 (swap still
active pending full baseline, single IPv4 resolver, and the NVMe's 34 unexpected power losses),
plus the same unknowable-ECC SKIP.
Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
ff0533106b |
feat(prod): operator decisions — node DNS, service VIPs, become vault, collections path
1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public
upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS
forwarding on dev cannot recur. Prod deliberately differs from dev, which is
PiHole-first because dev needs the local jackson.lan zone.
SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and
only one is configured here:
pods -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin
answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster;
PiHole is NOT in this path. Prod is already correct (.20).
LAN -> PiHole. Not a cluster concern.
nodes -> THIS FILE.
The one thing public node DNS breaks: the k3s API SERVER is a node process carrying
oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the
issuer through the NODE resolver, so prod would get the PUBLIC record instead of the
prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node
resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are
recorded in PROD-PROVISIONING.md §4 for the operator to pick.
2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the
first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering
within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target
exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is
not DR for prod.
3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and
proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un'
--become` returns root. The L0 no-plaintext-secrets check asserts every ansible
vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate.
4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so
EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now
sets collections_path = collections, and the collections are installed there from the
existing collections/requirements.yml (gitignored — pinned by requirements, fetched at
setup, vendored bytes are not source).
5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers
are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of
rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power
losses on a consumer drive with NO power-loss protection, in a single failure domain.
etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without
PLP is exactly how a member corrupts.
6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open
item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on
one node is outweighed by the other two and causes node ejection. That holds for the
dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity +
rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates
faithfully to all three replicas, so quorum sees no disagreement — low probability, and
the only mitigation is different hardware. Follow-up noted: Longhorn's
snapshotDataIntegrity is left at the chart default rather than set explicitly, and it
is the mechanism that operationalises the operator's own argument.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
c1f24e4a6d |
fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03)
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (
|
||
|
jakman85andClaude Opus 5
|
e623d27e2d |
feat(ansible): parameterise target groups so prod reuses the roles unchanged (R-ENV-02)
roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.
TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
* which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
(k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
* which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
Follows the k3s_api_host convention already in this repo. No per-run flag.
DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.
BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).
FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.
KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.
Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.
Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
666cc0d809 |
feat(ansible): prod node inventory + vaulted become; smartmontools into node_baseline
Prod inventory scaffolding for the three bare-metal nodes, and the become posture the operator ruled for production. BECOME: prod does NOT get dev's NOPASSWD sudo. Prod keeps a real sudo boundary and supplies the password to Ansible as an ansible-vault-encrypted `ansible_become_password` in inventory/group_vars/k3s_prod/vault.yml, mirroring the existing hypervisor and proxmox_hosts groups. The decrypting passphrase stays outside the repo (~/.config/home-k3s/ansible-vault-pass), so plays still run unattended with no plaintext secret in git. Committed here: README.md + vault.yml.example only — the operator creates the encrypted vault (`ansible-vault create`), as with every other vaulted group. PACKAGES BELONG IN THE ROLE (operator ruling): smartmontools is now declared in node_baseline_packages, not hand-installed on a node. Drive health tooling is a permanent part of a bare-metal node's baseline — dev nodes are VMs and never needed it. Same click-ops-never principle as the rest of the repo. ADDRESSING: nodes .10/.11/.12, reached DIRECTLY on the LAN (no pfSense NAT hop, unlike dev's 192.168.2.102:225x mapping), so ansible_host is the real node IP. kube-vip control-plane VIP is 192.168.2.9 — deliberately BELOW the nodes and OUTSIDE the Cilium LB pool (.20-.30) so kube-vip and LB-IPAM can never contend for an address. The ingress Gateway VIP (.20) and the LB pool are Flux/cluster-vars concerns and deliberately do not appear in Ansible. BLOCKED-BY-DESIGN: node_baseline_longhorn_device / node_baseline_etcd_device are present but set to an obviously invalid /dev/BLOCKED-... path pending the operator's disk-layout decision (node 1 is a single SATA SSD; the role's /dev/vdb,/dev/vdc defaults are virtio names that do not exist on bare metal). Leaving them unset would silently fall back to those equally-nonexistent defaults — a confusing failure that looks like a wiring bug. This fails just as hard and says exactly why. KNOWN FOLLOW-UP: roles/k3s_server and every node-host playbook hardcode the literal group names k3s_primary/k3s_secondary/k3s_servers, so the k3s_prod_* groups are not selected by them as-is. Harmless now (nothing can accidentally run against prod) but the "same roles, unchanged" reuse R-ENV-02 promises needs the target group parameterized before §6 of PROD-PROVISIONING.md can execute. Documented in hosts.ini and both READMEs. Serves: R-ENV-01, R-ENV-03, R-SEC-01. Task: T-7.1. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
6739339f53 |
Replace MinIO backup target with versitygw (s3_backup_target role)
MinIO's open-source edition is de-prioritised (console removed, commercial
pivot), so swap the off-cluster S3 backup target on the Proxmox host to
versitygw v1.7.0 (github.com/versity/versitygw) — a maintained Go S3 gateway
serving a POSIX filesystem as S3.
Rename role minio_backup_target -> s3_backup_target and playbook backup-minio
-> backup-s3. The role now:
- decommissions any pre-existing MinIO (service/binary/env/mc-client/user/
object-layout), KEEPING the ZFS dataset Storage/k3s-backups and its mount
- installs the pinned, checksummed versitygw binary (fetched+extracted on the
controller, host has no internet), a dedicated versitygw system user, and a
Type=simple hardened systemd unit running the posix backend rooted at
/mnt/Storage/k3s-backups/data, bound to 192.168.1.9:9000 (same endpoint),
plain HTTP / path-style
- ensures the dataset has xattr=sa (versitygw stores object metadata in xattrs)
- pre-creates the velero + cnpg buckets (bucket == top-level subdir)
Credentials are REUSED: versitygw's ROOT key is set to the same scoped
access_key/secret the cluster already uses, so endpoint + key + secret + bucket
names are byte-identical to the old MinIO target. The GitOps SOPS secret
(platform/backup/minio-credentials.sops.yaml) and all cluster-side Velero/CNPG
config need ZERO change. The vaulted group_vars now expose
s3_backup_target_access_key/_secret_key (same values, MinIO root creds retired).
Verified live on the Proxmox host: versitygw active+enabled on 192.168.1.9:9000,
MinIO fully removed, dataset intact; SigV4 ListBuckets with the reused key
returns velero+cnpg; a throwaway in-cluster pod reaches http://192.168.1.9:9000
(versitygw S3 response). Re-run is idempotent (no host changes).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
ac32f01f7b |
feat(backup): MinIO S3 backup target on Proxmox host for Velero + CNPG
Stand up a single-node/single-drive MinIO S3 endpoint on the Proxmox host as the off-cluster backup target for Velero (R-STATE-03) and CloudNativePG (R-IDP-05). Ansible (author + executed): - roles/minio_backup_target + playbooks/backup-minio.yml — idempotent, additive only: ZFS dataset Storage/k3s-backups (zstd, quota=100G, /mnt/Storage/k3s-backups, no sharenfs); minio system user; pinned+checksummed MinIO/mc binaries (fetched on the controller since the host has no WAN, copied to host); hardened systemd unit bound to the host's cluster-reachable LAN IP (auto-discovered) on :9000; velero + cnpg buckets; a non-root access key scoped by IAM policy to only those two buckets. - inventory: new [proxmox_hosts] group ([email protected].2.102); overrides the cluster-wide ansible_user. Does not disturb existing k3s node entries. - group_vars/proxmox_hosts/vault.yml (ansible-vault): root + scoped MinIO creds. GitOps hand-off: - platform/backup/minio-credentials.sops.yaml (+ .example) — SOPS/age-encrypted endpoint + scoped key/secret for the Velero/CNPG wiring (WS-STORAGE/WS-SEC). Not yet wired into a kustomization (left for the consuming agents). Verified live: dataset props correct; minio active+enabled bound to 192.168.1.9:9000; S3 endpoint http://192.168.1.9:9000/minio/health/ready returns 200 from an in-cluster pod; buckets velero+cnpg present; scoped key created+policy attached; re-run is idempotent. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
81ef78295a |
infra: direct kube-API access via kube-vip VIP + NAT (retire SSH tunnel)
Controller now reaches the kube-API directly through the pfSense NAT (192.168.2.50:6443 -> kube-vip VIP 192.168.1.49:6443), validating TLS against the VIP via tls-server-name. Removes the dev.sh SSH-tunnel scaffolding and updates the ansible source-of-truth so a rebuild regenerates a direct-access kubeconfig. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
60a58f01ad |
feat(infra): kube-vip control-plane HA VIP (192.168.1.49) + API tls-san
Add an HA control-plane API VIP for the k3s cluster with kube-vip, and add the VIP + all node IPs to the kube-apiserver serving-cert SANs so TLS via the VIP validates. Prepares the stable failover endpoint the operator fronts with a pfSense port-forward to 192.168.1.49:6443. roles/kube_vip + playbooks/kube-vip.yml: - Templates a single manifest (RBAC + DaemonSet kube-vip-ds) into each server's k3s auto-deploy dir (/var/lib/rancher/k3s/server/manifests) so it is reproducible and independent of Flux; k3s auto-applies it. - Control-plane ONLY (cp_enable=true, svc_enable=false): Cilium keeps ownership of Service LoadBalancing (L2 pool .53-.60) — no ARP conflict. - Runs on control-plane nodes (nodeSelector node-role.kubernetes.io/control-plane), hostNetwork, NET_ADMIN/NET_RAW, image pinned ghcr.io/kube-vip/kube-vip:v0.8.9. - vip_interface auto-detected per node (the iface holding 192.168.1.5x = eth0). - Leader election (plndr-cp-lock) with 15/10/2s lease timings. - This cluster runs without kube-proxy (Cilium replacement), so a hostNetwork pod cannot reach the kubernetes ClusterIP; kube-vip is pointed at the node-local apiserver (KUBERNETES_SERVICE_HOST=127.0.0.1:6443) so leader election works. roles/k3s_server tls-san: - New k3s_tls_sans (VIP + all three node IPs + 127.0.0.1, deduped); install args now emit one --tls-san per entry. - Persist the SAN list to /etc/rancher/k3s/config.yaml (read by a running k3s) and Restart k3s handler so an existing cluster regenerates its serving cert. - k3s.yml secondaries run serial:1 so a SAN change rolls one node at a time, keeping etcd quorum (kube-vip keeps the API reachable during each restart). Verified live: DS 3/3 ready, VIP answers on 192.168.1.49:6443, all 3 certs and an s_client against the VIP carry 192.168.1.49 + .50/.51/.52 + 127.0.0.1, all nodes Ready throughout the rolling restart. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
0037fb40fa |
ansible(cilium): rewrite controller kubeconfig to SSH-tunnel endpoint (kube-API not NAT-forwarded)
Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
1ff25318c9 |
ansible(proxmox): route Ansible via pfSense NAT; decouple k3s node-ip from ansible_host; mount vault-pass into dev container
- inventory routes SSH via 192.168.2.102:2250/2251/2252; new k3s_node_ip carries each node's real 192.168.1.x for --node-ip / k3s_api_host / Cilium parity - dev.sh bind-mounts ~/.config/home-k3s/ansible-vault-pass (ro) so vaulted playbooks can run in the pinned container (ansible.cfg vault_password_file) Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d57c9cbf12 |
infra(net): repoint ansible inventory + Cilium VIPs to the Proxmox LAN
- ansible hosts.ini: node ansible_host -> 192.168.1.50/.51/.52, with a clear note that the PM may need to swap these for the pfSense NAT-forwarded address:port once that NAT rule exists (not guessed here). k3s_api_host derives from the inventory, so it follows automatically. - Cilium CiliumLoadBalancerIPPool -> 192.168.1.53-.60 (matches the tofu cilium_l2_vip reservation on the new LAN). - CiliumL2AnnouncementPolicy: broaden interface selector to ^e(n|th) so the Proxmox NIC name (ens18/eth0) need not be pinned; flagged verify-at-bringup. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
e9d738b705 |
vpn_egress_gateway: commit the ansible-vault host VPN creds vault (ADR-0015)
The node-tunnel PIA credentials (username/password/.ovpn) live host-side in Ansible vault per docs/07. The encrypted vault.yml was provisioned but never committed, so the vpn.egress.gateway playbook was not reproducible from a fresh checkout. Commit it (ansible-vault encrypted; the vault password stays out of the repo via ansible.cfg vault_password_file). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
2c109e2a31 |
feat(vpn): node-level split-tunnel egress gateway Ansible role (ADR-0015)
Author the vpn_egress_gateway role + playbook that stands up the NODE-level PIA OpenVPN tunnel on egress-gateway node(s) — the host half of ADR-0015. The CiliumEgressGatewayPolicy steering vpn-class pods is separate GitOps against the same contract (group vpn_egress_gateway, node label homelab.keithjackson.dev/egress-gateway=true, host iface tun0). Honors all four ADR-0015 invariants: - Split-tunnel (inv 1&2): OpenVPN route-noexec + pull-filter ignore redirect-gateway => installs no routes; policy routing in a dedicated table (51820) with `ip rule from <tun0-addr>` steers ONLY Cilium-SNAT'd vpn-class traffic. Node default route stays on the WAN, tunnel up or down. - Scoped nftables kill-switch (inv 3&4): own inet table, policy accept, drops only tunnel-sourced (@vpn_egress_gateway_src set) packets not leaving tun0 => vpn-class fails closed (blackhole), node/internet-class fail open (WAN), never matched. PIA creds + the proven sweden-aes-128-cbc-udp-ip .ovpn come from Ansible vault (host-side per ADR-0015/docs-07); vault keys documented, no secret values committed. ansible-lint (production) + yamllint clean. Not deployed: green conformance is gated on the live tunnel (operator host creds) — see role README. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
ab0a7977bf |
Phase 1 T-1.4 DONE: Flux reconciling clusters/dev from host Gitea
host-prereqs.yml opened the firewall (via the operator's ansible-vault become password); Flux GitRepository READY=True (main@sha1:a2f3bdc). Commit the encrypted hypervisor vault (ansible_become_password) so rebuild is fully hands-off; passphrase stays out-of-repo. platform/apps still not reconciled. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
4e2ca88bd4 |
Wire ansible-vault for host become password (unattended host-prereqs)
Where the sudo password goes, codified: - inventory: new [hypervisor] group (localhost, connection=local). - host-prereqs.yml targets 'hypervisor'; become password from vaulted group_vars/hypervisor/vault.yml (ansible_become_password) — user creates it encrypted (safe to commit); README documents the exact commands. - ansible.cfg vault_password_file = ~/.config/home-k3s/ansible-vault-pass (passphrase kept OUTSIDE the repo, never committed). - ci/yamllint.yaml ignores **/vault.yml (vault blob isn't parseable YAML). Nodes keep NOPASSWD sudo (group_vars/all) — vault scoped to the host only. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
f2a7e42068 |
Phase 1 (author-only): node provisioning IaC — tofu + cloud-init + Ansible (T-1.1/1.2/1.3)
OpenTofu libvirt module (k3s-dev NAT 10.73.0.0/24, 3x 6GiB/4vCPU VMs, 20G OS +
40G thin data, host-passthrough), cloud-init baseline (k3s/Cilium/Longhorn
prereqs, static net), Ansible k3s-HA + Cilium (kube-proxy replacement, Gateway
API, Hubble; flannel/kube-proxy/Traefik disabled). PM fix: escape ${...} in a
cloud-init template comment (tofu templatefile parse). tofu fmt/init/validate =
Success; ansible --syntax-check + yamllint pass. NOTHING executed (no apply).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|