home-network-k3s-impl

512 Commits 140 Branches 0 Tags

3 Commits

Include renames

Author	SHA1	Message	Date
jakman85andClaude Opus 5	4f37fc375e	feat(prod): STANDUP COMPLETE — 12/13 tiers Ready, LE wildcard issued, dev untouched Prod is up and serving. 113 pods, 0 not Running/Completed. wildcard-home Certificate Ready (Let's Encrypt PROD, .prod.home.keithjackson.dev). Gateway VIP 192.168.2.20, crafty L4 .22, unifi L4 .21. 36 DNS records published under prod.home.keithjackson.dev, every TXT registry entry owner=prod. End-to-end HTTPS with certificate validation (no -k), pinned to prod's gateway: apex 200, keycloak 302 (login), gitea 200, grafana 302 (SSO). DEV UNTOUCHED: all 57 pre-existing records byte-identical to the pre-standup baseline. velero remains suspended — S3_BACKUP_ENDPOINT is still the TBD placeholder, and resuming it would create a BackupStorageLocation that can never reach a bucket: a tier reporting healthy while backups silently do not exist. TWO BUGS IN MY OWN VERIFICATION, both of which would have made the safety check worthless. Recording them because a check you cannot trust is worse than none — it trains you to skip the one alarm that matters. 1. SNAP_DIR defaulted to $TMPDIR. Every `dev.sh run` is a FRESH container, so the DNS baseline was discarded between steps and `verify`'s before/after comparison — the check that detects damage to the other cluster's records — could never run across invocations. Now repo-relative (bind-mounted, so it persists on the host) and gitignored as a run artifact. 2. The first real comparison cried wolf: "RECORDS OUTSIDE prod.home CHANGED", listing pairs of IDENTICAL lines. Cause was locale-dependent sort collation — the host ordered `.keithjackson.dev` differently from the container, so a diff of two identical sets reported changes. Proved it was noise by re-comparing under LC_ALL=C: all 57 records identical. dns_snapshot now strips CRs and sorts under LC_ALL=C so ordering is reproducible everywhere. ACTION REQUIRED (operator, pfSense): the local override is a wildcard on .home.keithjackson.dev and now also swallows the new prod.home. subtree, resolving EVERY prod hostname to 192.168.2.50 (dev). Cloudflare correctly returns 192.168.2.20. Prod is reachable from anywhere using public DNS, but NOT from behind pfSense. Narrow the override, or add one pointing prod.home.* at 192.168.2.20. That override is also the trap that produced a false healthy reading during the 2026-09-04 incident, and it is subtle because it PARTIALLY works: dev's *.home.keithjackson.dev wildcard matches prod.home.keithjackson.dev (one label) so the apex returns 200 over valid TLS, while keycloak.prod.home... (two labels) fails with a certificate-name error. I hit this myself during verification and mistook it for a broken prod cert. Never verify DNS or TLS here through a local resolver. Refs R-ENV-02, R-GITOPS-01, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-05 10:02:20 -05:00
jakman85andClaude Opus 5	1b056dc9d9	fix(prod): break the CDI cold-start deadlock, and make preflight catch its class `platform-config` stopped dead on prod: StorageProfile/longhorn dry-run failed: no matches for kind "StorageProfile" in version "cdi.kubevirt.io/v1beta1" A TRUE DEADLOCK, not a transient. platform-config applies the CDI CR and platform/kubevirt/cdi-storageprofiles.yaml in the SAME pass. StorageProfile's CRD does not exist until cdi-operator has reconciled that CR — and Flux server-side dry-runs EVERY object before applying ANY of them, so the unknown kind aborts the whole tier, the CR is never applied, the CRD is never created, and platform-config can never converge. Every tier below it stays blocked. Same root cause as the Kyverno CRD bundle, different operator: Flux cannot order a resource against a CRD that one of its own siblings is responsible for creating. It is INVISIBLE on an established cluster — dev's CDI CRDs have been there for months — and fatal on a cold one. Observed on prod: cdi-operator Available for 12h, no CDI CR, no storageprofiles CRD. `crds` now breaks it by applying ONLY the CDI CR — the minimum — and leaving every other object to Flux. The CR carries no ${} tokens, so applying it here cannot diverge from what Flux would render. It refuses if the `cdi` namespace is absent (platform has not run yet) rather than applying into nothing, waits for the CRD, and requires it to reach Established. On prod the CRD appeared in 10s and platform-config was unblocked. `preflight` now reports the CRDs that later tiers apply resources against (storageprofiles/cdis/kubevirts/volumesnapshotclasses) and dumps the cdi namespace when one is missing, so this is caught in seconds instead of after `resume` burns its 900s wait discovering it. It also names the stale-discovery- cache variant, which presents identically but clears on a forced reconcile. THE DURABLE FIX IS A REPO CHANGE, deliberately NOT made here: move cdi-storageprofiles.yaml out of platform-config into its own Kustomization with dependsOn: platform-config, so the CR lands one reconcile before the resources needing its CRDs. That edits the reconciliation graph both clusters share, and prod currently reads a pinned revision from the bootstrap Gitea while dev is mid-migration on a separate branch — so it wants doing deliberately, not at 02:00 to unblock a bring-up. Until it lands, C2 is a prerequisite for C5 on any cold cluster, and PROD-STANDUP.md says so. Refs R-GITOPS-01, R-ENV-02. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-05 08:05:46 -05:00
jakman85andClaude Opus 5	4160fee6ce	feat(prod): standup.sh + PROD-STANDUP.md — the complete standup sequence, in git The last steps of a prod bring-up were living in shell history: resume the tiers in the right order, clear a wedged HelmRelease, fix external-dns without deleting the other cluster's DNS. That is the same failure the Kyverno CRD pre-apply had, so it gets the same treatment. bootstrap/standup.sh — environment-agnostic (R-ENV-02), no dev/prod fork. Six idempotent steps: preflight, crds, unwedge, externaldns, resume, verify. THE GUARD IS THE POINT. --expect-server is MANDATORY and is compared against the live API server before any step runs; there is no override flag. Dev is live and shares the keithjackson.dev Cloudflare zone with prod, and on 2026-09-04 prod deleted every one of dev's public records. Negative control run: prod's kubeconfig with dev's server URL refuses and changes nothing. preflight fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base sentinel, or `default` (dev's owner) — the exact condition that caused the incident. It also surfaces SUSPENDED HelmReleases separately, because a suspended release keeps its last Ready condition and a stale `Ready=True` otherwise hides that Flux is not managing it at all (external-dns today). externaldns is the dangerous step, and the ordering IS the mitigation. A Helm upgrade resets replicas to 1 and applies new args in ONE operation, so there is no window to check the result first; if --txt-owner-id were missing the pod starts as owner `default` and syncs away dev's records. So the step PROVES THE OUTCOME BEFORE CAUSING IT: render the chart with the LIVE HelmRelease values and assert --txt-owner-id=<owner> appears, aborting if not. Rendering is deterministic and touches nothing, which turns a race into a precondition. Then snapshot Cloudflare, upgrade, re-verify live args (scale to 0 and fail if wrong), and diff the zone. resume walks the dependency order and STOPS at the first tier that does not go Ready — a partial understood bring-up beats a forced green. velero is skipped while S3_BACKUP_ENDPOINT is the TBD placeholder. TWO BUGS CAUGHT WHILE VALIDATING THIS, both by measuring instead of assuming: 1. preflight counted Kyverno CRDs with grep 'kyverno.io' and reported 20 of 22 — the policy-report types are in the wgpolicyk8s.io group. A preflight that under-reports reads as "the pre-apply failed" when it succeeded. Fixed and re-run: 22. 2. The documented node-health invocation used -e k3s_hosts=k3s_prod and SILENTLY RAN AGAINST DEV'S THREE NODES. Diagnostic plays deliberately keep their own target var (node_health_hosts, memattr_survey_hosts, vpn_egress_gateway_hosts) so they stay pointable while a bring-up is scoped elsewhere — a good design I did NOT change. The hazard is that the wrong flag does not error, it falls through to the dev default. Caught only by --list-hosts; the doc now leads with that warning and a dry-run command. Verified after fixing: node-health and all six site.yml plays target prod. .claude/settings.local.json — permission rules for the above. Every entry hardcodes prod's kubeconfig AND prod's --expect-server (or prod's ansible groups), so the permission strings are themselves prod-scoped and cannot be turned against dev. Deliberately NO rule for `dev.sh run bash -c *`, which would have been blanket cluster access wearing a narrow-looking mask. Refs R-ENV-02, R-GITOPS-01, R-SEC-02. Co-Authored-By: Claude Opus 5 <[email protected]>	2026-09-05 00:37:48 -05:00

Author

SHA1

Message

Date

jakman85andClaude Opus 5

4f37fc375e

feat(prod): STANDUP COMPLETE — 12/13 tiers Ready, LE wildcard issued, dev untouched

Prod is up and serving. 113 pods, 0 not Running/Completed. wildcard-home
Certificate Ready (Let's Encrypt PROD, *.prod.home.keithjackson.dev). Gateway
VIP 192.168.2.20, crafty L4 .22, unifi L4 .21. 36 DNS records published under
prod.home.keithjackson.dev, every TXT registry entry owner=prod.

End-to-end HTTPS with certificate validation (no -k), pinned to prod's gateway:
apex 200, keycloak 302 (login), gitea 200, grafana 302 (SSO).

DEV UNTOUCHED: all 57 pre-existing records byte-identical to the pre-standup
baseline. velero remains suspended — S3_BACKUP_ENDPOINT is still the TBD
placeholder, and resuming it would create a BackupStorageLocation that can never
reach a bucket: a tier reporting healthy while backups silently do not exist.

TWO BUGS IN MY OWN VERIFICATION, both of which would have made the safety check
worthless. Recording them because a check you cannot trust is worse than none —
it trains you to skip the one alarm that matters.

1. SNAP_DIR defaulted to $TMPDIR. Every `dev.sh run` is a FRESH container, so the
   DNS baseline was discarded between steps and `verify`'s before/after
   comparison — the check that detects damage to the other cluster's records —
   could never run across invocations. Now repo-relative (bind-mounted, so it
   persists on the host) and gitignored as a run artifact.

2. The first real comparison cried wolf: "RECORDS OUTSIDE prod.home CHANGED",
   listing pairs of IDENTICAL lines. Cause was locale-dependent sort collation —
   the host ordered `*.keithjackson.dev` differently from the container, so a
   diff of two identical sets reported changes. Proved it was noise by
   re-comparing under LC_ALL=C: all 57 records identical. dns_snapshot now strips
   CRs and sorts under LC_ALL=C so ordering is reproducible everywhere.

ACTION REQUIRED (operator, pfSense): the local override is a wildcard on
*.home.keithjackson.dev and now also swallows the new prod.home.* subtree,
resolving EVERY prod hostname to 192.168.2.50 (dev). Cloudflare correctly
returns 192.168.2.20. Prod is reachable from anywhere using public DNS, but NOT
from behind pfSense. Narrow the override, or add one pointing prod.home.* at
192.168.2.20.

That override is also the trap that produced a false healthy reading during the
2026-09-04 incident, and it is subtle because it PARTIALLY works: dev's
*.home.keithjackson.dev wildcard matches prod.home.keithjackson.dev (one label)
so the apex returns 200 over valid TLS, while keycloak.prod.home... (two labels)
fails with a certificate-name error. I hit this myself during verification and
mistook it for a broken prod cert. Never verify DNS or TLS here through a local
resolver.

Refs R-ENV-02, R-GITOPS-01, R-PLAT-01.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-05 10:02:20 -05:00

jakman85andClaude Opus 5

1b056dc9d9

fix(prod): break the CDI cold-start deadlock, and make preflight catch its class

`platform-config` stopped dead on prod:

    StorageProfile/longhorn dry-run failed: no matches for kind
    "StorageProfile" in version "cdi.kubevirt.io/v1beta1"

A TRUE DEADLOCK, not a transient. platform-config applies the CDI CR *and*
platform/kubevirt/cdi-storageprofiles.yaml in the SAME pass. StorageProfile's
CRD does not exist until cdi-operator has reconciled that CR — and Flux
server-side dry-runs EVERY object before applying ANY of them, so the unknown
kind aborts the whole tier, the CR is never applied, the CRD is never created,
and platform-config can never converge. Every tier below it stays blocked.

Same root cause as the Kyverno CRD bundle, different operator: Flux cannot order
a resource against a CRD that one of its own siblings is responsible for
creating. It is INVISIBLE on an established cluster — dev's CDI CRDs have been
there for months — and fatal on a cold one. Observed on prod: cdi-operator
Available for 12h, no CDI CR, no storageprofiles CRD.

`crds` now breaks it by applying ONLY the CDI CR — the minimum — and leaving
every other object to Flux. The CR carries no ${} tokens, so applying it here
cannot diverge from what Flux would render. It refuses if the `cdi` namespace is
absent (platform has not run yet) rather than applying into nothing, waits for
the CRD, and requires it to reach Established. On prod the CRD appeared in 10s
and platform-config was unblocked.

`preflight` now reports the CRDs that later tiers apply resources against
(storageprofiles/cdis/kubevirts/volumesnapshotclasses) and dumps the cdi
namespace when one is missing, so this is caught in seconds instead of after
`resume` burns its 900s wait discovering it. It also names the stale-discovery-
cache variant, which presents identically but clears on a forced reconcile.

THE DURABLE FIX IS A REPO CHANGE, deliberately NOT made here: move
cdi-storageprofiles.yaml out of platform-config into its own Kustomization with
dependsOn: platform-config, so the CR lands one reconcile before the resources
needing its CRDs. That edits the reconciliation graph both clusters share, and
prod currently reads a pinned revision from the bootstrap Gitea while dev is
mid-migration on a separate branch — so it wants doing deliberately, not at
02:00 to unblock a bring-up. Until it lands, C2 is a prerequisite for C5 on any
cold cluster, and PROD-STANDUP.md says so.

Refs R-GITOPS-01, R-ENV-02.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-05 08:05:46 -05:00

jakman85andClaude Opus 5

4160fee6ce

feat(prod): standup.sh + PROD-STANDUP.md — the complete standup sequence, in git

The last steps of a prod bring-up were living in shell history: resume the
tiers in the right order, clear a wedged HelmRelease, fix external-dns without
deleting the other cluster's DNS. That is the same failure the Kyverno CRD
pre-apply had, so it gets the same treatment.

bootstrap/standup.sh — environment-agnostic (R-ENV-02), no dev/prod fork. Six
idempotent steps: preflight, crds, unwedge, externaldns, resume, verify.

THE GUARD IS THE POINT. --expect-server is MANDATORY and is compared against
the live API server before any step runs; there is no override flag. Dev is
live and shares the keithjackson.dev Cloudflare zone with prod, and on
2026-09-04 prod deleted every one of dev's public records. Negative control
run: prod's kubeconfig with dev's server URL refuses and changes nothing.

preflight fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base
sentinel, or `default` (dev's owner) — the exact condition that caused the
incident. It also surfaces SUSPENDED HelmReleases separately, because a
suspended release keeps its last Ready condition and a stale `Ready=True`
otherwise hides that Flux is not managing it at all (external-dns today).

externaldns is the dangerous step, and the ordering IS the mitigation. A Helm
upgrade resets replicas to 1 and applies new args in ONE operation, so there is
no window to check the result first; if --txt-owner-id were missing the pod
starts as owner `default` and syncs away dev's records. So the step PROVES THE
OUTCOME BEFORE CAUSING IT: render the chart with the LIVE HelmRelease values and
assert --txt-owner-id=<owner> appears, aborting if not. Rendering is
deterministic and touches nothing, which turns a race into a precondition. Then
snapshot Cloudflare, upgrade, re-verify live args (scale to 0 and fail if
wrong), and diff the zone.

resume walks the dependency order and STOPS at the first tier that does not go
Ready — a partial understood bring-up beats a forced green. velero is skipped
while S3_BACKUP_ENDPOINT is the TBD placeholder.

TWO BUGS CAUGHT WHILE VALIDATING THIS, both by measuring instead of assuming:

1. preflight counted Kyverno CRDs with grep 'kyverno.io' and reported 20 of 22
   — the policy-report types are in the wgpolicyk8s.io group. A preflight that
   under-reports reads as "the pre-apply failed" when it succeeded. Fixed and
   re-run: 22.

2. The documented node-health invocation used -e k3s_hosts=k3s_prod and
   SILENTLY RAN AGAINST DEV'S THREE NODES. Diagnostic plays deliberately keep
   their own target var (node_health_hosts, memattr_survey_hosts,
   vpn_egress_gateway_hosts) so they stay pointable while a bring-up is scoped
   elsewhere — a good design I did NOT change. The hazard is that the wrong flag
   does not error, it falls through to the dev default. Caught only by
   --list-hosts; the doc now leads with that warning and a dry-run command.
   Verified after fixing: node-health and all six site.yml plays target prod.

.claude/settings.local.json — permission rules for the above. Every entry
hardcodes prod's kubeconfig AND prod's --expect-server (or prod's ansible
groups), so the permission strings are themselves prod-scoped and cannot be
turned against dev. Deliberately NO rule for `dev.sh run bash -c *`, which
would have been blanket cluster access wearing a narrow-looking mask.

Refs R-ENV-02, R-GITOPS-01, R-SEC-02.

Co-Authored-By: Claude Opus 5 <[email protected]>

2026-09-05 00:37:48 -05:00