Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
1,078 Commits 140 Branches 0 Tags
d8b25a67feeea46f3e1311d4e8fb15aedaaad8e8
Commit Graph
14 Commits
Author SHA1 Message Date
jakman85andClaude Opus 5 60e3f09c67 feat(vpn): prod egress-gateway group — k3s-prod-3, singleton, opt-in only 
`[vpn_egress_gateway_prod]` is a SEPARATE inventory group with its own vaulted
credentials, not a second member of `[vpn_egress_gateway]`. That group is the
default target of playbooks/vpn.egress.gateway.yml, so adding a prod host to it
would mean a flagless `ansible-playbook playbooks/vpn.egress.gateway.yml`
silently reconfigures a prod node's kill-switch datapath. Prod is reached only by
asking for it, with `--list-hosts` first:

  -e vpn_egress_gateway_hosts=vpn_egress_gateway_prod --limit k3s-prod-3

Node 3 because node 1's SSD was replaced yesterday (least operational history)
and node 2 carries the §0b memattr history. Exactly one node, forever: Cilium OSS
picks "the first node in lexical ordering" among nodeSelector matches, so a
labelled k3s-prod-1 would silently steal the gateway role — and the public egress
IP — from k3s-prod-3, with no event and no alert.

vault.yml is a verbatim copy of the dev group's CIPHERTEXT: same PIA account per
the operator, and copying the encrypted blob means the plaintext was never
decrypted to disk, through a shell, or into a transcript. The two files are now
independent — rotating means re-encrypting both, which is the right behaviour for
two environments.

Also corrects a stale inventory note that claimed k3s-prod-3 has wlp1s0 up on
192.168.2.195. Re-verified before choosing it as the gateway, because a second
live uplink in a kill-switch datapath is not a footnote: there is no wl* netdev on
the node at all, no phy80211 device, the /etc/network/interfaces wlp1s0 stanza is
commented out, 192.168.2.195 is assigned nowhere, and the only IPv4 addresses are
lo and eno1 192.168.2.12. Nothing needed bringing down.

Co-Authored-By: Claude Opus 5 <[email protected]>
 2026-09-12 17:38:46 -05:00
jakman85andClaude Opus 5 08fbe85638 fix(ansible): move the founder designation to node 2 
Node 1's SSD failed and was replaced; Debian was reinstalled, so node 1 is now a
JOINING node with no cluster state. It was still listed in [k3s_prod_primary],
and that group decides two things at once:

  1. which play gets k3s_server_mode: primary -> --cluster-init, which FOUNDS A
     NEW CLUSTER, and
  2. what k3s_api_host resolves to -- the --server join target.

So re-provisioning node 1 unchanged would have pointed a rebuilt, empty node at
ITSELF and founded a second cluster beside the live one. The role's
`creates: /etc/rancher/k3s/k3s.yaml` guard does not help: on a fresh install that
file is absent, so cluster-init WOULD have run.

Node 2 is the senior surviving server (up since 2026-09-01) and takes the founder
slot. Its own cluster-init task is inert precisely because k3s.yaml already exists
there. Verified after the change: k3s_api_host renders 192.168.2.11, and
groups[k3s_primary_group][0] is k3s-prod-2.

This is a latent gap the role documented but did not handle -- its comment assumes
the founder is always alive ('On a WARM single-node rejoin the founder is up and
answering too'). Rebuilding the founder itself was not covered.

Co-Authored-By: Claude Opus 5 <[email protected]>
 2026-09-07 21:41:54 -05:00
jakman85andClaude Opus 5 6a8953e288 feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing 
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.

Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.

WHAT THIS FIXES

1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
   ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
   anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
   to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
   volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
   database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
   the hour, and evaluates to 3 firing series on prod today.
   Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
   nextcloud-db's exporter cannot log in to its own database (permission denied for
   database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
   for it — the obvious metric would have been blind to one database in three.

2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
   by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
   job="kubelet" (the apiserver copy of the same series reports namespace=default),
   and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
   available=0 forever and would otherwise have fired critical permanently.

3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
   firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
   a node carries the egress-gateway label, with dev's behaviour unchanged today.

4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
   LITERALLY and the import still reports success, then sits at Done=True and never
   re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
   $(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
   green. The CronJob is the level-triggered correction; two alerts make its silence
   observable. Repaired live already — this stops it recurring.

Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.

Co-Authored-By: Claude Opus 5 <[email protected]>
 2026-09-07 00:59:22 -05:00
jakman85andClaude Opus 5 b19c5db9ae feat(prod): node 3 onboarded — and it is NOT immune; BIOS-setting hypothesis disproven 
k3s-prod-3 online at the planned 192.168.2.12 (serial 8CC015307N, same chassis/BIOS/CPU),
enabled in inventory, vaulted become confirmed, package floor installed. Health: 0 FAIL /
4 WARN / 14 PASS / 1 SKIP — the only node with zero FAILs. WD Blue WDS500G2B0A SATA SSD at
100% life, 3213 h, 0 grown bad blocks, 0 erase fails. Matched Crucial 2x16 GB pair — the only
node with symmetric memory.

*** NODE 3 IS NOT IMMUNE TO THE §0b BUG. *** Its firmware DOES publish a MAT
(MEMATTR=0xbeece018); the ABSENCE of a memattr diagnostic line means the kernel read a VALID
table, which is the opposite of immunity. There are THREE outcomes, not two: bail-out on a
garbage version >2, a valid parse with a sane size, and the catastrophic path. Seven boots across
three machines show the MAT address MOVES BOOT-TO-BOOT on every node with settings constant, and
each node has been seen on more than one path. Blast radius widens from two nodes to ALL THREE;
all three need the >= 6.13 kernel.

THE CORRUPTING BYTES ARE THE OS BOOT CHAIN, not firmware and not DRAM: garbage version
1463964749 = 0x5742544D appeared on node 2 AND node 3 — two different machines, identical bytes.
The overwriter is the shared shim/GRUB/EFI-stub/initrd placement writing into the page the
firmware pointed at. So initrd size, module set and boot device are part of the causal chain.

BIOS-SETTING HYPOTHESIS: DISPROVEN, and the three-way diff cannot support it even in principle.
All 240 attributes read on all three; firmware identical fleet-wide. The discriminating set (both
working nodes agree, broken differs) is seven attributes, of which:
  - Fast Boot is the ONLY live candidate, and only as a probability-shifter — HP documents it
    'skips some preboot initialization steps', which changes the heap top and hence where the MAT
    lands. Different mechanism from the previously-killed MRC-caching theory; that stays dead.
  - Legacy Boot Options is AFFIRMATIVELY INERT: 'Configure Legacy Support and Secure Boot' =
    'Legacy Support Disable and Secure Boot Enable' on ALL THREE, Option ROM Launch Policy =
    'All UEFI' on all three. No CSM is built on node 2 either. A naive diff gets this wrong.
  - the rest are coincidence. After Power Loss is ELIMINATED by node 3, which works and shares
    node 2's Power Off — though it is now a §1 checklist violation on TWO nodes.
FATAL CONFOUND: the 1=3!=2 partition is perfectly collinear with node 2 being the only
single-DIMM AND the only NVMe-booting node. Adding node 3 did not break the confound, it DOUBLED
it. No setting can be the cause anyway: node 2 booted good once and bad twice with settings held
constant.

EXPERIMENT DESIGN RECORDED because the obvious approach is statistically worthless: p(bad) is
known only to n=3, 95% CI ~0.09-0.99, so even 8 clean boots prove nothing. Elimination needs ~20
boots/arm; an odds shift needs ~50. Zero-touch first step: 20 reboots of node 2 unchanged, logging
MEMATTR address vs outcome — both failures were at exactly 0xbeeb0018 and all five successes
elsewhere, so if outcome maps onto a narrow address range the mechanism is settled without
touching anything.

ROLE BUG FIXED: node 3's WD drive reports a SMART threshold for only 1 of 25 attributes, and
smartctl OMITS the key rather than emitting 0 — summary.txt.j2 dereferenced a.thresh bare in five
places and the report task died, so no artifact was written. Added a thr() macro rendering 'n/a'
(deliberately not 0, which would read as a real ATA cannot-trip threshold). Would have hit any
drive of this class.

Also: three nodes, three different disk models, only one NVMe, and every node has a single disk
with no unpartitioned spare — so §2's identical-layout requirement is unmet on all three.

Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-03 22:32:25 -05:00
jakman85andClaude Opus 5 cb69a53825 feat(prod): enable node 2 + HARDWARE FAULT — it addresses 2.6 GiB of its 16 GB 
k3s-prod-2 (192.168.2.11) came online 2026-09-01. Enabled in the inventory, baseline package
floor installed, first health run completed. Vaulted become works on it unchanged.

*** OPEN FAULT: the node can only address ~2.6 GiB of a healthy, correctly-installed 16 GB
DIMM. *** Recorded in PROD-PROVISIONING §0b.

  dmidecode        16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s (DIMM3 empty)
  BIOS e820        reports ~16 GiB usable, incl. 0x100000000-0x42f7fffff
  kernel           Memory: 2568544K/16546088K available (13958980K RESERVED)
  last_pfn         revised DOWN 0x42f800 (~16.7 GiB) -> 0xc3c10 (~3.06 GiB) in early boot
  zones            Normal present 3340288 pages (~12.7 GiB) but MANAGED 0
  MemTotal         2,753,284 kB = 2.6 GiB

The DIMM is NOT the problem: it is present, correctly identified, and at rated speed. The
kernel sees all 16 GB and then reserves 13.3 GiB. Everything below ~3 GiB usable, everything
above 4 GiB present-but-unmanaged — the signature of memory remapping above 4 GB not being
applied, not of failing memory. Node 1 is the control: identical chassis, identical BIOS R21
Ver. 02.26.00, identical CPU, and it addresses its full 31 GiB. Per-machine state, not a
firmware-version defect.

WARNING RECORDED: do NOT assume the incoming 32 GB fixes this. If the fault is the slot, the
board or a BIOS setting rather than the DIMM, 32 GB presents identically and the money is
wasted. Verify MemTotal after any RAM change.

ALSO FOUND — the prod nodes are NOT hardware-symmetric:
  node 1  Micron C400 512GB SATA SSD (~2011) — 69% life, 15 grown bad blocks, 187 erase
          failures, 32 unexpected power losses
  node 2  Samsung PM9A1 512GB NVMe        — 96% life, 2142 power-on hours, 34 unexpected
          power losses, overall-health PASSED
Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires
an identical disk layout across all three nodes because Longhorn replica placement assumes
symmetry, so this asymmetry is now a live input to the disk decision rather than a footnote.
Node 2's NVMe is the drive node 1 should have had.

Node 2 otherwise: 0 FAIL / 3 WARN / 12 PASS / 2 SKIP. Same three WARNs as node 1 (swap still
active pending full baseline, single IPv4 resolver, and the NVMe's 34 unexpected power losses),
plus the same unknowable-ECC SKIP.

Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-01 20:56:58 -05:00
jakman85andClaude Opus 5 e623d27e2d feat(ansible): parameterise target groups so prod reuses the roles unchanged (R-ENV-02) 
roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.

TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
  * which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
    (k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
    only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
  * which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
    k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
    Follows the k3s_api_host convention already in this repo. No per-run flag.

DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.

BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
  ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).

FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.

KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.

Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.

Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-08-31 23:53:36 -05:00
jakman85andClaude Opus 5 666cc0d809 feat(ansible): prod node inventory + vaulted become; smartmontools into node_baseline 
Prod inventory scaffolding for the three bare-metal nodes, and the become posture the
operator ruled for production.

BECOME: prod does NOT get dev's NOPASSWD sudo. Prod keeps a real sudo boundary and
supplies the password to Ansible as an ansible-vault-encrypted `ansible_become_password`
in inventory/group_vars/k3s_prod/vault.yml, mirroring the existing hypervisor and
proxmox_hosts groups. The decrypting passphrase stays outside the repo
(~/.config/home-k3s/ansible-vault-pass), so plays still run unattended with no plaintext
secret in git. Committed here: README.md + vault.yml.example only — the operator creates
the encrypted vault (`ansible-vault create`), as with every other vaulted group.

PACKAGES BELONG IN THE ROLE (operator ruling): smartmontools is now declared in
node_baseline_packages, not hand-installed on a node. Drive health tooling is a
permanent part of a bare-metal node's baseline — dev nodes are VMs and never needed it.
Same click-ops-never principle as the rest of the repo.

ADDRESSING: nodes .10/.11/.12, reached DIRECTLY on the LAN (no pfSense NAT hop, unlike
dev's 192.168.2.102:225x mapping), so ansible_host is the real node IP. kube-vip
control-plane VIP is 192.168.2.9 — deliberately BELOW the nodes and OUTSIDE the Cilium
LB pool (.20-.30) so kube-vip and LB-IPAM can never contend for an address. The ingress
Gateway VIP (.20) and the LB pool are Flux/cluster-vars concerns and deliberately do
not appear in Ansible.

BLOCKED-BY-DESIGN: node_baseline_longhorn_device / node_baseline_etcd_device are present
but set to an obviously invalid /dev/BLOCKED-... path pending the operator's disk-layout
decision (node 1 is a single SATA SSD; the role's /dev/vdb,/dev/vdc defaults are virtio
names that do not exist on bare metal). Leaving them unset would silently fall back to
those equally-nonexistent defaults — a confusing failure that looks like a wiring bug.
This fails just as hard and says exactly why.

KNOWN FOLLOW-UP: roles/k3s_server and every node-host playbook hardcode the literal
group names k3s_primary/k3s_secondary/k3s_servers, so the k3s_prod_* groups are not
selected by them as-is. Harmless now (nothing can accidentally run against prod) but the
"same roles, unchanged" reuse R-ENV-02 promises needs the target group parameterized
before §6 of PROD-PROVISIONING.md can execute. Documented in hosts.ini and both READMEs.

Serves: R-ENV-01, R-ENV-03, R-SEC-01. Task: T-7.1. Gate: MERGE GATE: PASS.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-08-31 22:44:10 -05:00
jakman85andClaude Opus 4.8 6739339f53 Replace MinIO backup target with versitygw (s3_backup_target role) 
MinIO's open-source edition is de-prioritised (console removed, commercial
pivot), so swap the off-cluster S3 backup target on the Proxmox host to
versitygw v1.7.0 (github.com/versity/versitygw) — a maintained Go S3 gateway
serving a POSIX filesystem as S3.

Rename role minio_backup_target -> s3_backup_target and playbook backup-minio
-> backup-s3. The role now:
  - decommissions any pre-existing MinIO (service/binary/env/mc-client/user/
    object-layout), KEEPING the ZFS dataset Storage/k3s-backups and its mount
  - installs the pinned, checksummed versitygw binary (fetched+extracted on the
    controller, host has no internet), a dedicated versitygw system user, and a
    Type=simple hardened systemd unit running the posix backend rooted at
    /mnt/Storage/k3s-backups/data, bound to 192.168.1.9:9000 (same endpoint),
    plain HTTP / path-style
  - ensures the dataset has xattr=sa (versitygw stores object metadata in xattrs)
  - pre-creates the velero + cnpg buckets (bucket == top-level subdir)

Credentials are REUSED: versitygw's ROOT key is set to the same scoped
access_key/secret the cluster already uses, so endpoint + key + secret + bucket
names are byte-identical to the old MinIO target. The GitOps SOPS secret
(platform/backup/minio-credentials.sops.yaml) and all cluster-side Velero/CNPG
config need ZERO change. The vaulted group_vars now expose
s3_backup_target_access_key/_secret_key (same values, MinIO root creds retired).

Verified live on the Proxmox host: versitygw active+enabled on 192.168.1.9:9000,
MinIO fully removed, dataset intact; SigV4 ListBuckets with the reused key
returns velero+cnpg; a throwaway in-cluster pod reaches http://192.168.1.9:9000
(versitygw S3 response). Re-run is idempotent (no host changes).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-28 22:48:38 -05:00
jakman85andClaude Opus 4.8 ac32f01f7b feat(backup): MinIO S3 backup target on Proxmox host for Velero + CNPG 
Stand up a single-node/single-drive MinIO S3 endpoint on the Proxmox host as the
off-cluster backup target for Velero (R-STATE-03) and CloudNativePG (R-IDP-05).

Ansible (author + executed):
- roles/minio_backup_target + playbooks/backup-minio.yml — idempotent, additive only:
  ZFS dataset Storage/k3s-backups (zstd, quota=100G, /mnt/Storage/k3s-backups, no
  sharenfs); minio system user; pinned+checksummed MinIO/mc binaries (fetched on the
  controller since the host has no WAN, copied to host); hardened systemd unit bound to
  the host's cluster-reachable LAN IP (auto-discovered) on :9000; velero + cnpg buckets;
  a non-root access key scoped by IAM policy to only those two buckets.
- inventory: new [proxmox_hosts] group ([email protected].2.102); overrides the cluster-wide
  ansible_user. Does not disturb existing k3s node entries.
- group_vars/proxmox_hosts/vault.yml (ansible-vault): root + scoped MinIO creds.

GitOps hand-off:
- platform/backup/minio-credentials.sops.yaml (+ .example) — SOPS/age-encrypted endpoint +
  scoped key/secret for the Velero/CNPG wiring (WS-STORAGE/WS-SEC). Not yet wired into a
  kustomization (left for the consuming agents).

Verified live: dataset props correct; minio active+enabled bound to 192.168.1.9:9000;
S3 endpoint http://192.168.1.9:9000/minio/health/ready returns 200 from an in-cluster pod;
buckets velero+cnpg present; scoped key created+policy attached; re-run is idempotent.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-28 22:15:05 -05:00
jakman85andClaude Opus 4.8 1ff25318c9 ansible(proxmox): route Ansible via pfSense NAT; decouple k3s node-ip from ansible_host; mount vault-pass into dev container 
- inventory routes SSH via 192.168.2.102:2250/2251/2252; new k3s_node_ip carries
  each node's real 192.168.1.x for --node-ip / k3s_api_host / Cilium parity
- dev.sh bind-mounts ~/.config/home-k3s/ansible-vault-pass (ro) so vaulted
  playbooks can run in the pinned container (ansible.cfg vault_password_file)

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-25 11:55:37 -05:00
jakman85andClaude Opus 4.8 d57c9cbf12 infra(net): repoint ansible inventory + Cilium VIPs to the Proxmox LAN 
- ansible hosts.ini: node ansible_host -> 192.168.1.50/.51/.52, with a
  clear note that the PM may need to swap these for the pfSense
  NAT-forwarded address:port once that NAT rule exists (not guessed here).
  k3s_api_host derives from the inventory, so it follows automatically.
- Cilium CiliumLoadBalancerIPPool -> 192.168.1.53-.60 (matches the tofu
  cilium_l2_vip reservation on the new LAN).
- CiliumL2AnnouncementPolicy: broaden interface selector to ^e(n|th) so
  the Proxmox NIC name (ens18/eth0) need not be pinned; flagged
  verify-at-bringup.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-25 10:23:37 -05:00
jakman85andClaude Opus 4.8 2c109e2a31 feat(vpn): node-level split-tunnel egress gateway Ansible role (ADR-0015) 
Author the vpn_egress_gateway role + playbook that stands up the NODE-level
PIA OpenVPN tunnel on egress-gateway node(s) — the host half of ADR-0015. The
CiliumEgressGatewayPolicy steering vpn-class pods is separate GitOps against the
same contract (group vpn_egress_gateway, node label
homelab.keithjackson.dev/egress-gateway=true, host iface tun0).

Honors all four ADR-0015 invariants:
- Split-tunnel (inv 1&2): OpenVPN route-noexec + pull-filter ignore
  redirect-gateway => installs no routes; policy routing in a dedicated table
  (51820) with `ip rule from <tun0-addr>` steers ONLY Cilium-SNAT'd vpn-class
  traffic. Node default route stays on the WAN, tunnel up or down.
- Scoped nftables kill-switch (inv 3&4): own inet table, policy accept, drops
  only tunnel-sourced (@vpn_egress_gateway_src set) packets not leaving tun0 =>
  vpn-class fails closed (blackhole), node/internet-class fail open (WAN),
  never matched.

PIA creds + the proven sweden-aes-128-cbc-udp-ip .ovpn come from Ansible vault
(host-side per ADR-0015/docs-07); vault keys documented, no secret values
committed. ansible-lint (production) + yamllint clean. Not deployed: green
conformance is gated on the live tunnel (operator host creds) — see role README.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-24 01:09:19 -05:00
jakman85andClaude Opus 4.8 4e2ca88bd4 Wire ansible-vault for host become password (unattended host-prereqs) 
Where the sudo password goes, codified:
- inventory: new [hypervisor] group (localhost, connection=local).
- host-prereqs.yml targets 'hypervisor'; become password from vaulted
  group_vars/hypervisor/vault.yml (ansible_become_password) — user creates it
  encrypted (safe to commit); README documents the exact commands.
- ansible.cfg vault_password_file = ~/.config/home-k3s/ansible-vault-pass
  (passphrase kept OUTSIDE the repo, never committed).
- ci/yamllint.yaml ignores **/vault.yml (vault blob isn't parseable YAML).
Nodes keep NOPASSWD sudo (group_vars/all) — vault scoped to the host only.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-08-16 15:04:44 -05:00
jakman85andClaude Opus 4.8 f2a7e42068 Phase 1 (author-only): node provisioning IaC — tofu + cloud-init + Ansible (T-1.1/1.2/1.3) 
OpenTofu libvirt module (k3s-dev NAT 10.73.0.0/24, 3x 6GiB/4vCPU VMs, 20G OS +
40G thin data, host-passthrough), cloud-init baseline (k3s/Cilium/Longhorn
prereqs, static net), Ansible k3s-HA + Cilium (kube-proxy replacement, Gateway
API, Hubble; flannel/kube-proxy/Traefik disabled). PM fix: escape ${...} in a
cloud-init template comment (tofu templatefile parse). tofu fmt/init/validate =
Success; ansible --syntax-check + yamllint pass. NOTHING executed (no apply).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-08-16 01:21:00 -05:00