Commit Graph
17 Commits
Author SHA1 Message Date
jakman85andClaude Opus 5 cb69a53825 feat(prod): enable node 2 + HARDWARE FAULT — it addresses 2.6 GiB of its 16 GB
k3s-prod-2 (192.168.2.11) came online 2026-09-01. Enabled in the inventory, baseline package
floor installed, first health run completed. Vaulted become works on it unchanged.

*** OPEN FAULT: the node can only address ~2.6 GiB of a healthy, correctly-installed 16 GB
DIMM. *** Recorded in PROD-PROVISIONING §0b.

  dmidecode        16 GB Samsung M471A2K43DB1-CTD in DIMM1 @ 2667 MT/s (DIMM3 empty)
  BIOS e820        reports ~16 GiB usable, incl. 0x100000000-0x42f7fffff
  kernel           Memory: 2568544K/16546088K available (13958980K RESERVED)
  last_pfn         revised DOWN 0x42f800 (~16.7 GiB) -> 0xc3c10 (~3.06 GiB) in early boot
  zones            Normal present 3340288 pages (~12.7 GiB) but MANAGED 0
  MemTotal         2,753,284 kB = 2.6 GiB

The DIMM is NOT the problem: it is present, correctly identified, and at rated speed. The
kernel sees all 16 GB and then reserves 13.3 GiB. Everything below ~3 GiB usable, everything
above 4 GiB present-but-unmanaged — the signature of memory remapping above 4 GB not being
applied, not of failing memory. Node 1 is the control: identical chassis, identical BIOS R21
Ver. 02.26.00, identical CPU, and it addresses its full 31 GiB. Per-machine state, not a
firmware-version defect.

WARNING RECORDED: do NOT assume the incoming 32 GB fixes this. If the fault is the slot, the
board or a BIOS setting rather than the DIMM, 32 GB presents identically and the money is
wasted. Verify MemTotal after any RAM change.

ALSO FOUND — the prod nodes are NOT hardware-symmetric:
  node 1  Micron C400 512GB SATA SSD (~2011) — 69% life, 15 grown bad blocks, 187 erase
          failures, 32 unexpected power losses
  node 2  Samsung PM9A1 512GB NVMe        — 96% life, 2142 power-on hours, 34 unexpected
          power losses, overall-health PASSED
Same chassis, same BIOS, same CPU, same partition layout — very different storage. §2 requires
an identical disk layout across all three nodes because Longhorn replica placement assumes
symmetry, so this asymmetry is now a live input to the disk decision rather than a footnote.
Node 2's NVMe is the drive node 1 should have had.

Node 2 otherwise: 0 FAIL / 3 WARN / 12 PASS / 2 SKIP. Same three WARNs as node 1 (swap still
active pending full baseline, single IPv4 resolver, and the NVMe's 34 unexpected power losses),
plus the same unknowable-ECC SKIP.

Serves: R-ENV-01, R-ENV-03, R-OBS-04. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 20:56:58 -05:00
jakman85andClaude Opus 5 ff0533106b feat(prod): operator decisions — node DNS, service VIPs, become vault, collections path
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public
   upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS
   forwarding on dev cannot recur. Prod deliberately differs from dev, which is
   PiHole-first because dev needs the local jackson.lan zone.

   SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and
   only one is configured here:
     pods  -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin
              answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster;
              PiHole is NOT in this path. Prod is already correct (.20).
     LAN   -> PiHole. Not a cluster concern.
     nodes -> THIS FILE.
   The one thing public node DNS breaks: the k3s API SERVER is a node process carrying
   oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the
   issuer through the NODE resolver, so prod would get the PUBLIC record instead of the
   prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node
   resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are
   recorded in PROD-PROVISIONING.md §4 for the operator to pick.

2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the
   first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering
   within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target
   exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is
   not DR for prod.

3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and
   proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un'
   --become` returns root. The L0 no-plaintext-secrets check asserts every ansible
   vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate.

4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so
   EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now
   sets collections_path = collections, and the collections are installed there from the
   existing collections/requirements.yml (gitignored — pinned by requirements, fetched at
   setup, vendored bytes are not source).

5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers
   are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of
   rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power
   losses on a consumer drive with NO power-loss protection, in a single failure domain.
   etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without
   PLP is exactly how a member corrupts.

6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open
   item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on
   one node is outweighed by the other two and causes node ejection. That holds for the
   dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity +
   rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates
   faithfully to all three replicas, so quorum sees no disagreement — low probability, and
   the only mitigation is different hardware. Follow-up noted: Longhorn's
   snapshotDataIntegrity is left at the chart default rather than set explicitly, and it
   is the mechanism that operationalises the operator's own argument.

Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 00:40:29 -05:00
jakman85andClaude Opus 5 c1f24e4a6d fix(dns): resilient node resolvers — mechanism-detecting, per-environment (R-DNS-01/03)
Re-authors the fix stranded on origin/fix/node-dns-resilient-upstream (7a34590). That
commit was NOT landed: it wrote a netplan drop-in and ran `netplan apply`, but netplan
does not exist on these nodes any more (Debian 13 trixie generic cloud image; not in
node_baseline_packages). Landing it would have failed the handler with command-not-found
and BROKEN node-baseline.yml on every node — worse than not landing it. Its dev-LAN
literals also sat in role defaults, which prod inherits, pointing prod nodes at a PiHole
on a network they cannot reach.

The underlying problem is real and verified: a SINGLE upstream resolver is a hard SPOF.
PiHole answers jackson.lan instantly but its external recursion flaps — 0/25 for public
names (registry-1.docker.io, quay.io, ghcr.io) while gitea.jackson.lan resolved 5/5 —
breaking containerd image pulls and CoreDNS `forward . /etc/resolv.conf`.

MECHANISM DETECTION (new `dns` tag in node_baseline). Five read-only probes feed one
set_fact yielding three outcomes, because the two environments no longer share a resolver
stack (dev-1 is behind resolved's 127.0.0.53 stub; prod-1 has a direct resolv.conf and no
cloud-init at all):
  * systemd-resolved — resolved running AND owning /etc/resolv.conf (proven by the
    symlink pointing into /run/systemd/resolve/, or by every nameserver being a resolved
    loopback). Writes resolved.conf.d/99-k3s-dns.conf with an ordered DNS=, Domains=, and
    FallbackDNS= EXPLICITLY EMPTIED so systemd's compiled-in public resolvers can never
    silently bypass the local zone and ad-blocking.
  * resolv-conf-file — plain directly-managed file (prod's shape). Ordered nameserver
    lines + options timeout:2 attempts:2, backup: true. Deliberately NO `rotate`, which
    would destroy PiHole-first ordering.
  * contended — resolvconf/openresolv present or NetworkManager active. Writes NOTHING
    and fails loudly: a change that silently gets reverted is worse than no change.
Resolved-running-but-not-owning-resolv.conf correctly falls through to the file branch.
Ordering is the policy that keeps the local zone working — the second server is only
consulted after the first fails, so a gateway NXDOMAIN can never beat PiHole to a
jackson.lan answer.

SEAMS. Dev values in role defaults ([192.168.1.18, 192.168.1.1], search jackson.lan);
prod values in inventory/group_vars/k3s_prod/vars.yml. NOT cluster-vars — that ConfigMap
is consumed by Flux postBuild for platform/ and apps/, and no Ansible code reads it.

PROD'S SECOND UPSTREAM IS AN OPEN OPERATOR DECISION and is left as a loud
BLOCKED-pending-second-upstream placeholder, the same precedent as the node_baseline disk
devices, with a dns-tagged assert that stops the play before any file is written. A
public resolver was NOT invented as a default: it would bypass the local zone, which is a
decision, not a default. PROD-PROVISIONING.md §4 was rewritten (the old bullet pointed at
the now-dropped branch) to state the choice, including that prod routing to the dev
PiHole is unverified across the pfSense boundary.

tofu: servers = [var.dns_server, var.network_gateway] — the one hunk of the old branch
worth taking as-is. Dev-only by nature.

ALSO FIXES A FALSE WARN THIS WOULD OTHERWISE CREATE: node_health_collect.py parsed
/etc/resolv.conf only, so on dev it read the 127.0.0.53 stub and would report a permanent
single-upstream WARN even after a correct fix. It now asks resolvectl for the real
upstreams when it sees the stub — and if that fails, nameservers is NULL with the reason
recorded, never the stub and never a guess. assess.yml turns null into SKIP, not WARN and
not PASS. The honesty discipline is preserved exactly.

Verified with NO play run against any node: syntax-check + yamllint + tofu fmt clean;
--tags packages still selects ONLY the apt task (the prod package-floor escape hatch
bypasses DNS entirely); --list-hosts unchanged; -c local simulation driving the REAL
detection expression and REAL file bodies extracted from the role by script (so the test
cannot drift) passes all 5 scenarios — dev stub -> systemd-resolved, prod shape ->
resolv-conf-file, NetworkManager and resolvconf -> contended; dev defaults pass the
assert byte-identically while prod overrides fail it by name; collector unit-tested
against 8 resolver shapes.

Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-09-01 00:11:49 -05:00
jakman85andClaude Opus 5 e623d27e2d feat(ansible): parameterise target groups so prod reuses the roles unchanged (R-ENV-02)
roles/k3s_server, roles/cilium and every node-host playbook hardcoded the dev group
names k3s_primary/k3s_secondary/k3s_servers, so the prod inventory groups could not be
selected at all. That broke the R-ENV-02 promise that prod reuses the SAME roles, and
blocked PROD-PROVISIONING.md §6 outright.

TWO KNOBS, deliberately separate — Ansible templates a play's `hosts:` BEFORE any host
context exists, so group_vars is unreadable there, while roles run WITH host context:
  * which hosts a play RUNS ON — k3s_hosts, k3s_primary_hosts/k3s_secondary_hosts
    (k3s.yml's split), vpn_egress_gateway_hosts; each `| default(<dev group>)`, set by -e
    only. Follows node-health.yml's established `<name>_hosts | default(dev)` pattern.
  * which group a ROLE means by "my primary"/"my servers" — k3s_primary_group,
    k3s_servers_group in group_vars/all.yml (dev), overridden in group_vars/k3s_prod.
    Follows the k3s_api_host convention already in this repo. No per-run flag.

DEV IS UNCHANGED — the key evidence. --list-hosts/--list-tasks/--syntax-check captured
for all 9 playbooks at HEAD and after: sorted host lists IDENTICAL for all 9, same
primary/secondary split, same 6-play site.yml. (Raw --list-hosts PRINT ORDER varies run
to run; proven to be Ansible's own set ordering by running UNMODIFIED HEAD cilium.yml
four times -> 3,1,2 / 1,3,2 / 1,2,3 / 1,2,3. Hence sorted comparison.) --list-tasks
differs in exactly two files, only by added TAGS annotations and one new assert; no task
reordered, renamed or removed. Var resolution verified per environment with -c local:
dev -> k3s_primary/192.168.1.50/sans .50,.49,127.0.0.1,.51,.52; prod ->
k3s_prod_primary/192.168.2.10/vip .9/sans .10,.9,127.0.0.1,.11,.12.

BREAKS THE SMARTMONTOOLS CIRCULAR DEPENDENCY. node_baseline tasks are now tagged
packages/system/disks, so the package install can run alone:
  ansible-playbook playbooks/node-baseline.yml -e k3s_hosts=k3s_prod --tags packages
Skipping is the guard — no disks-tagged task is selected, so the
BLOCKED-pending-disk-decision paths are never templated, let alone passed to mkfs.
Belt-and-braces, a new disks-tagged assert fires first in a FULL run and names §2
explicitly instead of failing obscurely at mkfs. `tags: never` was deliberately NOT used
(it would change the default run).

FOOTGUN FOUND AND FIXED: roles/cilium, cluster_verify and flux_bootstrap all hardcoded
.cache/dev.kubeconfig — the exact file infra/devcontainer/dev.sh exports as $KUBECONFIG.
Once prod became targetable, a prod cilium/verify run would have SILENTLY OVERWRITTEN it,
leaving the operator's kubectl quietly answering from PROD. Now cluster_kubeconfig_name
(dev.kubeconfig -> prod.kubeconfig). Dev path byte-identical.

KNOWN, DOCUMENTED, NOT FIXED HERE: flux-bootstrap.yml retargets but roles/flux_bootstrap
shells out to bootstrap/flux/bootstrap.sh with a hardcoded --path=clusters/dev — a
bootstrap/ subtree change, not a target-group change (§6, cross-referenced to §7).
vpn.egress.gateway.yml is parameterised but prod has no egress group yet and would need
its own vaulted PIA credentials. --check against prod is still impossible until the prod
vault exists — even the packages-only run needs it.

Nothing was run against the dev cluster: --list-hosts/--list-tasks/--syntax-check, plus
debug/assert with -c local, only.

Serves: R-ENV-01, R-ENV-02, R-ENV-03. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-31 23:53:36 -05:00
jakman85andClaude Opus 5 666cc0d809 feat(ansible): prod node inventory + vaulted become; smartmontools into node_baseline
Prod inventory scaffolding for the three bare-metal nodes, and the become posture the
operator ruled for production.

BECOME: prod does NOT get dev's NOPASSWD sudo. Prod keeps a real sudo boundary and
supplies the password to Ansible as an ansible-vault-encrypted `ansible_become_password`
in inventory/group_vars/k3s_prod/vault.yml, mirroring the existing hypervisor and
proxmox_hosts groups. The decrypting passphrase stays outside the repo
(~/.config/home-k3s/ansible-vault-pass), so plays still run unattended with no plaintext
secret in git. Committed here: README.md + vault.yml.example only — the operator creates
the encrypted vault (`ansible-vault create`), as with every other vaulted group.

PACKAGES BELONG IN THE ROLE (operator ruling): smartmontools is now declared in
node_baseline_packages, not hand-installed on a node. Drive health tooling is a
permanent part of a bare-metal node's baseline — dev nodes are VMs and never needed it.
Same click-ops-never principle as the rest of the repo.

ADDRESSING: nodes .10/.11/.12, reached DIRECTLY on the LAN (no pfSense NAT hop, unlike
dev's 192.168.2.102:225x mapping), so ansible_host is the real node IP. kube-vip
control-plane VIP is 192.168.2.9 — deliberately BELOW the nodes and OUTSIDE the Cilium
LB pool (.20-.30) so kube-vip and LB-IPAM can never contend for an address. The ingress
Gateway VIP (.20) and the LB pool are Flux/cluster-vars concerns and deliberately do
not appear in Ansible.

BLOCKED-BY-DESIGN: node_baseline_longhorn_device / node_baseline_etcd_device are present
but set to an obviously invalid /dev/BLOCKED-... path pending the operator's disk-layout
decision (node 1 is a single SATA SSD; the role's /dev/vdb,/dev/vdc defaults are virtio
names that do not exist on bare metal). Leaving them unset would silently fall back to
those equally-nonexistent defaults — a confusing failure that looks like a wiring bug.
This fails just as hard and says exactly why.

KNOWN FOLLOW-UP: roles/k3s_server and every node-host playbook hardcode the literal
group names k3s_primary/k3s_secondary/k3s_servers, so the k3s_prod_* groups are not
selected by them as-is. Harmless now (nothing can accidentally run against prod) but the
"same roles, unchanged" reuse R-ENV-02 promises needs the target group parameterized
before §6 of PROD-PROVISIONING.md can execute. Documented in hosts.ini and both READMEs.

Serves: R-ENV-01, R-ENV-03, R-SEC-01. Task: T-7.1. Gate: MERGE GATE: PASS.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-31 22:44:10 -05:00
jakman85andClaude Opus 4.8 6739339f53 Replace MinIO backup target with versitygw (s3_backup_target role)
MinIO's open-source edition is de-prioritised (console removed, commercial
pivot), so swap the off-cluster S3 backup target on the Proxmox host to
versitygw v1.7.0 (github.com/versity/versitygw) — a maintained Go S3 gateway
serving a POSIX filesystem as S3.

Rename role minio_backup_target -> s3_backup_target and playbook backup-minio
-> backup-s3. The role now:
  - decommissions any pre-existing MinIO (service/binary/env/mc-client/user/
    object-layout), KEEPING the ZFS dataset Storage/k3s-backups and its mount
  - installs the pinned, checksummed versitygw binary (fetched+extracted on the
    controller, host has no internet), a dedicated versitygw system user, and a
    Type=simple hardened systemd unit running the posix backend rooted at
    /mnt/Storage/k3s-backups/data, bound to 192.168.1.9:9000 (same endpoint),
    plain HTTP / path-style
  - ensures the dataset has xattr=sa (versitygw stores object metadata in xattrs)
  - pre-creates the velero + cnpg buckets (bucket == top-level subdir)

Credentials are REUSED: versitygw's ROOT key is set to the same scoped
access_key/secret the cluster already uses, so endpoint + key + secret + bucket
names are byte-identical to the old MinIO target. The GitOps SOPS secret
(platform/backup/minio-credentials.sops.yaml) and all cluster-side Velero/CNPG
config need ZERO change. The vaulted group_vars now expose
s3_backup_target_access_key/_secret_key (same values, MinIO root creds retired).

Verified live on the Proxmox host: versitygw active+enabled on 192.168.1.9:9000,
MinIO fully removed, dataset intact; SigV4 ListBuckets with the reused key
returns velero+cnpg; a throwaway in-cluster pod reaches http://192.168.1.9:9000
(versitygw S3 response). Re-run is idempotent (no host changes).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 22:48:38 -05:00
jakman85andClaude Opus 4.8 ac32f01f7b feat(backup): MinIO S3 backup target on Proxmox host for Velero + CNPG
Stand up a single-node/single-drive MinIO S3 endpoint on the Proxmox host as the
off-cluster backup target for Velero (R-STATE-03) and CloudNativePG (R-IDP-05).

Ansible (author + executed):
- roles/minio_backup_target + playbooks/backup-minio.yml — idempotent, additive only:
  ZFS dataset Storage/k3s-backups (zstd, quota=100G, /mnt/Storage/k3s-backups, no
  sharenfs); minio system user; pinned+checksummed MinIO/mc binaries (fetched on the
  controller since the host has no WAN, copied to host); hardened systemd unit bound to
  the host's cluster-reachable LAN IP (auto-discovered) on :9000; velero + cnpg buckets;
  a non-root access key scoped by IAM policy to only those two buckets.
- inventory: new [proxmox_hosts] group ([email protected].2.102); overrides the cluster-wide
  ansible_user. Does not disturb existing k3s node entries.
- group_vars/proxmox_hosts/vault.yml (ansible-vault): root + scoped MinIO creds.

GitOps hand-off:
- platform/backup/minio-credentials.sops.yaml (+ .example) — SOPS/age-encrypted endpoint +
  scoped key/secret for the Velero/CNPG wiring (WS-STORAGE/WS-SEC). Not yet wired into a
  kustomization (left for the consuming agents).

Verified live: dataset props correct; minio active+enabled bound to 192.168.1.9:9000;
S3 endpoint http://192.168.1.9:9000/minio/health/ready returns 200 from an in-cluster pod;
buckets velero+cnpg present; scoped key created+policy attached; re-run is idempotent.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 22:15:05 -05:00
jakman85andClaude Opus 4.8 81ef78295a infra: direct kube-API access via kube-vip VIP + NAT (retire SSH tunnel)
Controller now reaches the kube-API directly through the pfSense NAT
(192.168.2.50:6443 -> kube-vip VIP 192.168.1.49:6443), validating TLS
against the VIP via tls-server-name. Removes the dev.sh SSH-tunnel
scaffolding and updates the ansible source-of-truth so a rebuild
regenerates a direct-access kubeconfig.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 22:10:27 -05:00
jakman85andClaude Opus 4.8 60a58f01ad feat(infra): kube-vip control-plane HA VIP (192.168.1.49) + API tls-san
Add an HA control-plane API VIP for the k3s cluster with kube-vip, and add
the VIP + all node IPs to the kube-apiserver serving-cert SANs so TLS via the
VIP validates. Prepares the stable failover endpoint the operator fronts with
a pfSense port-forward to 192.168.1.49:6443.

roles/kube_vip + playbooks/kube-vip.yml:
- Templates a single manifest (RBAC + DaemonSet kube-vip-ds) into each server's
  k3s auto-deploy dir (/var/lib/rancher/k3s/server/manifests) so it is
  reproducible and independent of Flux; k3s auto-applies it.
- Control-plane ONLY (cp_enable=true, svc_enable=false): Cilium keeps ownership
  of Service LoadBalancing (L2 pool .53-.60) — no ARP conflict.
- Runs on control-plane nodes (nodeSelector node-role.kubernetes.io/control-plane),
  hostNetwork, NET_ADMIN/NET_RAW, image pinned ghcr.io/kube-vip/kube-vip:v0.8.9.
- vip_interface auto-detected per node (the iface holding 192.168.1.5x = eth0).
- Leader election (plndr-cp-lock) with 15/10/2s lease timings.
- This cluster runs without kube-proxy (Cilium replacement), so a hostNetwork
  pod cannot reach the kubernetes ClusterIP; kube-vip is pointed at the
  node-local apiserver (KUBERNETES_SERVICE_HOST=127.0.0.1:6443) so leader
  election works.

roles/k3s_server tls-san:
- New k3s_tls_sans (VIP + all three node IPs + 127.0.0.1, deduped); install
  args now emit one --tls-san per entry.
- Persist the SAN list to /etc/rancher/k3s/config.yaml (read by a running k3s)
  and Restart k3s handler so an existing cluster regenerates its serving cert.
- k3s.yml secondaries run serial:1 so a SAN change rolls one node at a time,
  keeping etcd quorum (kube-vip keeps the API reachable during each restart).

Verified live: DS 3/3 ready, VIP answers on 192.168.1.49:6443, all 3 certs and
an s_client against the VIP carry 192.168.1.49 + .50/.51/.52 + 127.0.0.1, all
nodes Ready throughout the rolling restart.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 21:52:43 -05:00
jakman85andClaude Opus 4.8 0037fb40fa ansible(cilium): rewrite controller kubeconfig to SSH-tunnel endpoint (kube-API not NAT-forwarded)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 12:15:12 -05:00
jakman85andClaude Opus 4.8 1ff25318c9 ansible(proxmox): route Ansible via pfSense NAT; decouple k3s node-ip from ansible_host; mount vault-pass into dev container
- inventory routes SSH via 192.168.2.102:2250/2251/2252; new k3s_node_ip carries
  each node's real 192.168.1.x for --node-ip / k3s_api_host / Cilium parity
- dev.sh bind-mounts ~/.config/home-k3s/ansible-vault-pass (ro) so vaulted
  playbooks can run in the pinned container (ansible.cfg vault_password_file)

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 11:55:37 -05:00
jakman85andClaude Opus 4.8 d57c9cbf12 infra(net): repoint ansible inventory + Cilium VIPs to the Proxmox LAN
- ansible hosts.ini: node ansible_host -> 192.168.1.50/.51/.52, with a
  clear note that the PM may need to swap these for the pfSense
  NAT-forwarded address:port once that NAT rule exists (not guessed here).
  k3s_api_host derives from the inventory, so it follows automatically.
- Cilium CiliumLoadBalancerIPPool -> 192.168.1.53-.60 (matches the tofu
  cilium_l2_vip reservation on the new LAN).
- CiliumL2AnnouncementPolicy: broaden interface selector to ^e(n|th) so
  the Proxmox NIC name (ens18/eth0) need not be pinned; flagged
  verify-at-bringup.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 10:23:37 -05:00
jakman85andClaude Opus 4.8 e9d738b705 vpn_egress_gateway: commit the ansible-vault host VPN creds vault (ADR-0015)
The node-tunnel PIA credentials (username/password/.ovpn) live host-side in Ansible vault per
docs/07. The encrypted vault.yml was provisioned but never committed, so the vpn.egress.gateway
playbook was not reproducible from a fresh checkout. Commit it (ansible-vault encrypted; the
vault password stays out of the repo via ansible.cfg vault_password_file).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 15:00:58 -05:00
jakman85andClaude Opus 4.8 2c109e2a31 feat(vpn): node-level split-tunnel egress gateway Ansible role (ADR-0015)
Author the vpn_egress_gateway role + playbook that stands up the NODE-level
PIA OpenVPN tunnel on egress-gateway node(s) — the host half of ADR-0015. The
CiliumEgressGatewayPolicy steering vpn-class pods is separate GitOps against the
same contract (group vpn_egress_gateway, node label
homelab.keithjackson.dev/egress-gateway=true, host iface tun0).

Honors all four ADR-0015 invariants:
- Split-tunnel (inv 1&2): OpenVPN route-noexec + pull-filter ignore
  redirect-gateway => installs no routes; policy routing in a dedicated table
  (51820) with `ip rule from <tun0-addr>` steers ONLY Cilium-SNAT'd vpn-class
  traffic. Node default route stays on the WAN, tunnel up or down.
- Scoped nftables kill-switch (inv 3&4): own inet table, policy accept, drops
  only tunnel-sourced (@vpn_egress_gateway_src set) packets not leaving tun0 =>
  vpn-class fails closed (blackhole), node/internet-class fail open (WAN),
  never matched.

PIA creds + the proven sweden-aes-128-cbc-udp-ip .ovpn come from Ansible vault
(host-side per ADR-0015/docs-07); vault keys documented, no secret values
committed. ansible-lint (production) + yamllint clean. Not deployed: green
conformance is gated on the live tunnel (operator host creds) — see role README.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 01:09:19 -05:00
jakman85andClaude Opus 4.8 ab0a7977bf Phase 1 T-1.4 DONE: Flux reconciling clusters/dev from host Gitea
host-prereqs.yml opened the firewall (via the operator's ansible-vault become
password); Flux GitRepository READY=True (main@sha1:a2f3bdc). Commit the
encrypted hypervisor vault (ansible_become_password) so rebuild is fully
hands-off; passphrase stays out-of-repo. platform/apps still not reconciled.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 15:14:09 -05:00
jakman85andClaude Opus 4.8 4e2ca88bd4 Wire ansible-vault for host become password (unattended host-prereqs)
Where the sudo password goes, codified:
- inventory: new [hypervisor] group (localhost, connection=local).
- host-prereqs.yml targets 'hypervisor'; become password from vaulted
  group_vars/hypervisor/vault.yml (ansible_become_password) — user creates it
  encrypted (safe to commit); README documents the exact commands.
- ansible.cfg vault_password_file = ~/.config/home-k3s/ansible-vault-pass
  (passphrase kept OUTSIDE the repo, never committed).
- ci/yamllint.yaml ignores **/vault.yml (vault blob isn't parseable YAML).
Nodes keep NOPASSWD sudo (group_vars/all) — vault scoped to the host only.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 15:04:44 -05:00
jakman85andClaude Opus 4.8 f2a7e42068 Phase 1 (author-only): node provisioning IaC — tofu + cloud-init + Ansible (T-1.1/1.2/1.3)
OpenTofu libvirt module (k3s-dev NAT 10.73.0.0/24, 3x 6GiB/4vCPU VMs, 20G OS +
40G thin data, host-passthrough), cloud-init baseline (k3s/Cilium/Longhorn
prereqs, static net), Ansible k3s-HA + Cilium (kube-proxy replacement, Gateway
API, Hubble; flannel/kube-proxy/Traefik disabled). PM fix: escape ${...} in a
cloud-init template comment (tofu templatefile parse). tofu fmt/init/validate =
Success; ansible --syntax-check + yamllint pass. NOTHING executed (no apply).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 01:21:00 -05:00