Home-Network/home-network-k3s-impl
2
0 Fork 0
Code Issues 2 Pull Requests 5 Actions Packages Projects Releases Wiki Activity
501 Commits 139 Branches 0 Tags
b19c5db9ae43cf007eb45a1d2c0fd04befa108fc
Commit Graph
8 Commits
Author SHA1 Message Date
jakman85andClaude Opus 5 6b4da27f84 feat(prod): SOPS decryption (same age key as dev) + GitOps source on main
gate / gate (push) Has been cancelled
Details
gate / conformance (push) Has been cancelled
Details 
Operator decisions, 2026-09-01:
  1. Prod uses the SAME SOPS age key as dev.
  2. Prod tracks the SAME BRANCH as dev (`main`). Dev is a temporary instance and will be
     decommissioned once prod is deemed stable.

SOPS: eight `decryption: {provider: sops, secretRef: sops-age}` patches mirroring dev
exactly — platform, apps-wave-0/1/2, oauth2-proxy, headlamp, crafty, velero. The sops-age
Secret is still created OUT OF BAND on prod (bootstrap/sops/install-age-key.sh); the
overlay references it by name only, never inlines key material.

RECORDED CONSEQUENCE of the shared branch: once prod is live, EVERY MERGE TO main
RECONCILES BOTH CLUSTERS. There is no promotion step and no soak window — dev and prod
move together. That is a deliberate consequence of dev being temporary, not an oversight,
but it removes the "prove it on dev first" safety net that the last few days have leaned
on heavily. Worth revisiting if dev outlives its expected life.

*** BOOTSTRAP BLOCKER recorded, NOT papered over *** — the source URL cannot work on a
cold prod cluster, for two independent reasons found while wiring this:

1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER — the one slated for
   decommission. Prod syncing from it means prod's source of truth lives inside the
   cluster being retired. Gitea must move to prod (or off-cluster) BEFORE dev is torn
   down. This makes Gitea migration the FIRST task of the decommission plan, not the last.

2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
   answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP} via a template
   plugin. On prod that is 192.168.2.20 — PROD's own gateway, where no Gitea listens. Prod
   Flux would resolve gitea.home.keithjackson.dev to itself and never reach a repository.
   This is the same split-horizon mechanism that makes dev work, inverted against us.

Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
mirror) with a cutover to the in-cluster URL once Gitea actually runs on prod — precisely
the sequence docs/05 describes for dev's own Phase-6 cutover. The branch is settled and
correct; only the host is blocked. Documented at the patch site and in
PROD-PROVISIONING.md §7.

Gate: MERGE GATE: PASS.

Serves: R-SEC-01, R-GITOPS-01, R-ENV-02, R-ENV-03. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-01 16:27:32 -05:00
jakman85andClaude Opus 5 666a60776a fix(flux): close the substitution race — platform dependsOn root + strict substitution (P0/P1) 
P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix
for the incident, not the feature gate. Verified against kustomize-controller v1.4.0
`kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION-
gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent
cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision
N is exactly what writes a newly-added cluster-vars knob.

Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked
the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all
11 substituting Kustomizations chain transitively through it (platform-config -> platform;
the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config).
The "not chaining through platform" set is EMPTY, so this one edge protects the whole
graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/
sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready
on apply success without health-gating its children.

Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest
anywhere under clusters/<env> — including clusters/base — now stalls platform and
everything downstream, where today it would not. Fail-closed, per doctrine.

P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it
matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README).
There is no gotk-components.yaml in git, and the live kustomize-controller Deployment
carries NO Flux ownership labels — it was installed imperatively by `flux install` from
bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT
the controllers. Nothing under clusters/** can reach a controller flag.

So it uses Flux's documented mechanism for hand-installed clusters: a new
bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a
JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to
kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a
container's args is atomic and a merge patch would replace all five install-generated
args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the
patch, ASSERT the gate string is present so a future Flux bump that breaks the patch
target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is
vendored, so components can never drift from the pinned FLUX_VERSION.

Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true
would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial
Deployment as a Flux-managed resource (a future prune would delete kustomize-controller).

ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive
on a git push — the controller Deployment is outside Flux's inventory, so an operator must
re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is
level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled
BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race
turns a transient empty render into a transient hard Kustomization failure.

ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had
no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own
sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left
platform blocked forever on a flux-system Kustomization that does not exist, and nothing
downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev.

Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7):
prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs
the prod age-key decision, a security choice not a copy), and no GitRepository source URL
patch (needs a decision on whether prod tracks the same branch as dev, which would make a
dev-facing merge reconcile prod).

Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS.

Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-01 01:45:04 -05:00
jakman85andClaude Opus 5 ff0533106b feat(prod): operator decisions — node DNS, service VIPs, become vault, collections path
gate / gate (push) Has been cancelled
Details
gate / conformance (push) Has been cancelled
Details 
1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public
   upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS
   forwarding on dev cannot recur. Prod deliberately differs from dev, which is
   PiHole-first because dev needs the local jackson.lan zone.

   SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and
   only one is configured here:
     pods  -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin
              answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster;
              PiHole is NOT in this path. Prod is already correct (.20).
     LAN   -> PiHole. Not a cluster concern.
     nodes -> THIS FILE.
   The one thing public node DNS breaks: the k3s API SERVER is a node process carrying
   oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the
   issuer through the NODE resolver, so prod would get the PUBLIC record instead of the
   prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node
   resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are
   recorded in PROD-PROVISIONING.md §4 for the operator to pick.

2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the
   first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering
   within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target
   exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is
   not DR for prod.

3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and
   proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un'
   --become` returns root. The L0 no-plaintext-secrets check asserts every ansible
   vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate.

4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so
   EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now
   sets collections_path = collections, and the collections are installed there from the
   existing collections/requirements.yml (gitignored — pinned by requirements, fetched at
   setup, vendored bytes are not source).

5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers
   are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of
   rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power
   losses on a consumer drive with NO power-loss protection, in a single failure domain.
   etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without
   PLP is exactly how a member corrupts.

6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open
   item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on
   one node is outweighed by the other two and causes node ejection. That holds for the
   dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity +
   rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates
   faithfully to all three replicas, so quorum sees no disagreement — low probability, and
   the only mitigation is different hardware. Follow-up noted: Longhorn's
   snapshotDataIntegrity is left at the chart default rather than set explicitly, and it
   is the mechanism that operationalises the operator's own argument.

Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-09-01 00:40:29 -05:00
jakman85andClaude Opus 5 3f980d2a17 feat(env): tokenize hardcoded dev LAN addresses into cluster-vars (R-ENV-02) 
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).

New cluster-vars knobs (base default = dev literal, per the file's existing convention):
  K3S_API_HOST          platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
  CILIUM_LB_POOL_START  platform-config/cilium/l2-ip-pool.yaml
  CILIUM_LB_POOL_STOP   "
  UNIFI_L4_VIP          apps/vms/unifi/50-service-l4.yaml
  CRAFTY_L4_VIP         platform/crafty/service-l4.yaml
  S3_BACKUP_ENDPOINT    platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
                        semaphore}-db.yaml, and platform/velero/helmrelease.yaml

Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.

clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.

PROD ADDRESS PLAN (operator, 2026-08-31):
  nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
  Cilium LB pool .20-.30

CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.

Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.

DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.

Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
 2026-08-31 22:43:36 -05:00
jakman85andClaude Opus 4.8 eb31d2ef58 fix(monitoring): cap Prometheus TSDB by size to stop false absent()/==0 alerts 
The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).

Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
  failed ... write to WAL: no space left on device". Scrapes still return 200
  (targets show healthy, lastScrape=now) but samples are never stored, so the
  TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
  absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
  expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
  up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
  present) — the rules are correct; the monitor was simply blind.

`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.

- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale

Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-28 14:38:12 -05:00
jakman85andClaude Opus 4.8 bba67d03ce feat(dns): add GATEWAY_VIP cluster-var for split-horizon target (R-DNS-03) 
Parameterize the in-cluster split-horizon target as GATEWAY_VIP (R-ENV-02): base
default = dev L2-announced Gateway VIP (10.73.0.240, first IP of the Cilium LB pool).
coredns-custom answers *.home.keithjackson.dev with ${GATEWAY_VIP} so the unified graph
holds across dev/prod via cluster-vars only. Prod overlay carries a TODO to set the prod
LAN VIP alongside the prod L2 pool range when hardware is built.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
 2026-08-24 22:28:37 -05:00
jakman85andClaude Opus 4.8 9743b50c3b fix(K1): uppercase cluster-vars keys to match component substitution refs 
Flux postBuild.substituteFrom is case-sensitive; platform/ components reference
${UPPERCASE} tokens but cluster-vars keys were lowercase -> none would resolve
(literal ${...} would reach the HelmReleases). Uppercased the ConfigMap data
keys + the dev/prod overlay patch paths. Every component ref now resolves;
clusters/dev & clusters/prod build; gate green. Clears the K1 landmine before
any platform un-suspend.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-08-16 15:16:30 -05:00
jakman85andClaude Opus 4.8 af4f6c07ae T-0.1 Flux GitOps layout + dev/prod overlays + platform component skeletons (R-GITOPS-01/03, R-ENV-02, R-PLAT-01) 
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
 2026-08-16 00:08:31 -05:00