6f61c9622daf94076caa656b3cf74aff0d28c34d
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
04569abd78 |
docs(prod): node-maintenance runbook — a failed drain leaves a cordon that stops self-healing
Two unplanned node events (prod-3 ungraceful power-down; prod-2 memory upgrade) produced one finding worth more than either incident. A FAILED `kubectl drain` STILL CORDONS, and Longhorn's `disable-scheduling-on-cordoned-node` is `true` — so a cordoned node gets no replica scheduling. Observed after prod-2's reboot: ALL 17 VOLUMES sat attached/degraded with prod-2 holding 2 of 18 replicas and ZERO rebuilds in flight. Nothing was driving recovery, every pod was Running, and no alert fired. `kubectl uncordon` started rebuilds immediately; converged 17/17 healthy in ~6 minutes (measured: 9→10→11→13→14→16→17 over 5 polls). So uncordon is not tidying up — it is what re-enables storage redundancy. A green pod list proves nothing about it. DRAIN CANNOT COMPLETE ON THIS CLUSTER, and the two usual flags do not fix it. `--ignore-daemonsets --delete-emptydir-data` clears the reported objections and then hangs: all three instance-manager PDBs report ALLOWED DISRUPTIONS = 0 and node-drain-policy is block-if-contains-last-replica. Requesting replica eviction does not rescue it either — with 3 nodes and 3 replicas there is no fourth node to move a replica to, so eviction can never complete. That is structural. Documented options in preference order: just reboot (proven twice here; the 600s replenishment interval means a quick reboot triggers no full rebuild); or scale down the workloads holding engines on that node so the PDBs free up. Changing node-drain-policy to always-allow is advised against at 3/3. Also recorded: check the kube-vip leader before rebooting a server — it holds the API VIP via a lease and is a DaemonSet, so a DRAIN does not move the VIP but a REBOOT does, and kubectl blips for seconds. Plus a post-maintenance checklist whose first item is verifying no cordon remains, and whose memory check exists because this hardware silently strands RAM on the wrong kernel (§0b). KNOWN, NOT FIXED: pod distribution does not rebalance — 52/10/50 after prod-2's cordon+reboot. For the many single-replica apps here a spread constraint would change nothing (one replica, one node). It self-corrects as pods restart. Recorded because it concentrates failure: all three CNPG databases currently run on prod-1, so losing that node takes Keycloak, Nextcloud and Semaphore together. The fix is database HA (instances: 2), a resource decision the operator owns. Header refreshed: phases A/B/C are done, 12/13 tiers, 112 pods, all three nodes at full 32 GB after node 2's DIMM upgrade. Refs R-STATE-01, R-STATE-02, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
765ad032db |
fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
9063e53ce2 |
docs(prod): prod verified end-to-end through the real resolver — dnsmasq profile installed
Operator installed infra/workstation/dnsmasq/install.sh. Verified afterwards: RESOLUTION (live NetworkManager dnsmasq, `getent hosts`): prod.home / gitea.prod / keycloak.prod / grafana.prod -> 192.168.2.20 unifi.prod -> 192.168.2.21 crafty-mc.prod -> 192.168.2.22 gitea.home / home.keithjackson.dev -> 192.168.2.50 DEV, UNCHANGED HTTPS with NO --resolve override and NO -k, all landing on 192.168.2.20: apex 200 · gitea 200 · headlamp 200 · semaphore 200 keycloak 302 (login) · grafana 302 (SSO) · nextcloud 302 · pihole 302 CERTIFICATE, confirmed by SNI — this is the check that actually settles the earlier confusion, because the resolver bug made dev's cert look like a broken prod cert: subject CN=*.prod.home.keithjackson.dev issuer C=US, O=Let's Encrypt, CN=YR1 (LE production) SAN *.prod.home.keithjackson.dev, prod.home.keithjackson.dev valid 2026-09-05 .. 2026-12-04 Dev checked in the same pass still serves CN=*.home.keithjackson.dev. Also noted so it is not misread as a fault: `curl --cert-status` reports "(91) No OCSP response received" because Let's Encrypt retired OCSP responders in 2025 — there is nothing to staple. The plain requests validated normally. Prod is now reachable and trusted from the workstation by name. Outstanding items are unchanged: MinIO routing (prod is on pfSense's WAN), the apex flip being the same event as the legacy-prod 443 cutover, and delivering the backup-tenancy commits to prod via a c8c1d29-based branch. Refs R-DNS-02, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6d62aca0d2 |
feat(workstation): prod dnsmasq profile — and CORRECT the topology I had wrong
OPERATOR CORRECTION (2026-09-05). I had been calling the resolver that shadows
prod "a pfSense override" throughout this repo. Both halves of that were wrong:
- It is the WORKSTATION'S OWN dnsmasq, not pfSense. The live resolver is the
one NetworkManager spawns (dns=dnsmasq, --conf-dir=/etc/NetworkManager/
dnsmasq.d); the standalone dnsmasq.service is INACTIVE, so /etc/dnsmasq.d is
dormant. The dev profile's `address=/.home.keithjackson.dev/192.168.2.50`
matches the domain AND EVERY SUBDOMAIN, so it also swallowed the whole new
prod.home.* tree and answered it with dev's address.
- The k3s prod cluster is NOT behind pfSense. It sits on the SOHO router
network 192.168.2.0/24, which is pfSense's WAN; pfSense CONTAINS the dev LAN
192.168.1.0/24. This workstation is on 192.168.2.0/24 too, which is why the
rules below use prod's real VIPs and need no NAT hop, unlike dev's .50.
New infra/workstation/dnsmasq/ — authored, not click-ops:
20-prod.home.keithjackson.dev.conf prod.home.* -> 192.168.2.20 (Gateway),
unifi -> .21, crafty-mc -> .22 (L4 VIPs)
install.sh installs + reloads; refuses non-root
dnsmasq resolves `address=` by LONGEST MATCH, so the prod rules win WITHOUT
editing or reordering the dev profile. PROVEN, not assumed: ran a throwaway
dnsmasq on 127.0.0.1:5354 loaded with both profiles —
prod.home / gitea.prod / keycloak.prod -> 192.168.2.20
unifi.prod -> 192.168.2.21 crafty-mc.prod -> 192.168.2.22
gitea.home / home.keithjackson.dev -> 192.168.2.50 (DEV, UNCHANGED)
`dnsmasq --test` on the new file: syntax OK.
install.sh pins the destination because getting it wrong is SILENT: a file in
/etc/dnsmasq.d exists, looks installed, and changes nothing. It also uses
`systemctl reload NetworkManager` rather than SIGHUP — dnsmasq does not re-read
its config on SIGHUP, so a HUP would look like it worked and not apply anything.
WHY THE MISLABEL MATTERED: the trap PARTIALLY works. Dev serves a
*.home.keithjackson.dev wildcard cert, which matches prod.home.keithjackson.dev
(one label) so the prod apex returns 200 over VALID TLS, while
keycloak.prod.home.keithjackson.dev (two labels) fails with a certificate-name
error. It reads as a broken prod cert; it is the wrong server answering. I made
exactly that misreading during verification.
Also recorded, because it changes what the apex flip IS: the SOHO router NATs
inbound 443 to the pfSense WAN (192.168.2.102) to serve the EXISTING NON-K3S
production environment, which must stay live until the switch is deliberately
made. So §10a sequence P is not just a k3s-side DNS/TLS change — it is the
legacy-prod cutover, and P as written does NOT cover repointing that NAT. The
two must be planned together or the apex resolves to k3s while inbound 443 still
lands on the legacy box. Phase C touches none of it: prod's Gateway is
192.168.2.20:443 (a different address) and its cert is DNS-01, needing no
inbound 443. And the MinIO timeout is now correctly described as expected
topology — crossing INTO pfSense from its WAN — not a misconfiguration.
VERIFIED: full CI gate PASS (spec-immutability, lint, l0, render-golden, l1).
Refs R-DNS-02, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
b80e8eef82 |
feat(backup): prod shares dev's MinIO — plus the path tenancy that makes it survivable
Operator ruling: "the S3 backup location is the same as Dev." Prod's
S3_BACKUP_ENDPOINT is now http://192.168.1.9:9000 and the TBD-PROD-* placeholder
is retired. Set EXPLICITLY rather than by omitting the op — an omitted op would
inherit the same value silently, and a shared backup target must be visible in
the overlay that owns it.
SHARING THE ENDPOINT WAS NOT SAFE AS AUTHORED. Nothing about the backup PATHS
was per-environment, and enabling prod against dev's MinIO would have been the
2026-09-04 external-dns incident again, in a worse place:
VELERO — bucket `velero`, NO prefix, and both clusters run schedules named
`daily` / `daily-resources-only`. Velero SYNCS backups it finds in its BSL
into itself as Backup CRs and GARBAGE-COLLECTS them at TTL, so prod would
have adopted dev's backups and then deleted them on schedule.
CNPG — the worse one. destinationPath `s3://cnpg` with CNPG appending
serverName, which defaults to the CLUSTER NAME. Both environments have
keycloak-db, nextcloud-db and semaphore-db, so prod's Postgres would archive
WAL into the EXACT prefix dev's does. Two servers interleaving WAL into one
Barman stream CORRUPTS THE CHAIN and destroys PITR — and dev's restore is
PROVEN (R-STATE-03). Sharing an endpoint is fine; sharing a path is not.
Two knobs, authored once, environments differing only by scoped overrides
(R-ENV-02):
VELERO_BACKUP_PREFIX dev "" | prod "prod"
CNPG_BACKUP_PATH dev "s3://cnpg" | prod "s3://cnpg/prod"
THE BASE VALUES ARE DEV'S CURRENT LAYOUT ON PURPOSE. Dev's existing backups live
at the bucket root and at s3://cnpg; giving dev a prefix would ORPHAN the very
backups its proven restore depends on — the same reasoning that keeps dev's
EXTERNALDNS_TXT_OWNER_ID at `default`. Prod has no history to preserve.
*** PROD STILL HAS NO BACKUP COVERAGE — the endpoint is not reachable. ***
Measured from a throwaway pod INSIDE the prod cluster (the workstation proves
nothing about pod egress, and a workstation-side reading has already misled this
project once): curl http://192.168.1.9:9000/minio/health/live → connection timed
out after 15s. pfSense does not route prod's pod network to the dev LAN's MinIO.
OPERATOR ACTION REQUIRED. velero stays suspended; note apps-wave-0 applies GREEN
regardless — the token IS defined so strict substitution succeeds — while every
write fails at RUNTIME. A green apps-wave-0 does not mean backups work.
New `standup.sh s3check` step probes the endpoint from a throwaway in-cluster
pod and fails closed, with `--s3-endpoint <url>` to test a CANDIDATE before
authoring it into cluster-vars. That is how the unreachability above was
established rather than assumed.
VERIFIED: full CI gate PASS. flux-postbuild-substitution 254 -> 260 assertions,
and neither new knob appears in the dead-knob advisory list, so both are really
consumed. Rendered both overlays: all three CNPG destinationPaths and Velero's
prefix carry the tokens; dev resolves to its existing layout, prod to the nested
paths.
DELIVERY: prod reconciles the bootstrap Gitea's main at
|
||
|
jakman85andClaude Opus 5
|
4f37fc375e |
feat(prod): STANDUP COMPLETE — 12/13 tiers Ready, LE wildcard issued, dev untouched
Prod is up and serving. 113 pods, 0 not Running/Completed. wildcard-home Certificate Ready (Let's Encrypt PROD, *.prod.home.keithjackson.dev). Gateway VIP 192.168.2.20, crafty L4 .22, unifi L4 .21. 36 DNS records published under prod.home.keithjackson.dev, every TXT registry entry owner=prod. End-to-end HTTPS with certificate validation (no -k), pinned to prod's gateway: apex 200, keycloak 302 (login), gitea 200, grafana 302 (SSO). DEV UNTOUCHED: all 57 pre-existing records byte-identical to the pre-standup baseline. velero remains suspended — S3_BACKUP_ENDPOINT is still the TBD placeholder, and resuming it would create a BackupStorageLocation that can never reach a bucket: a tier reporting healthy while backups silently do not exist. TWO BUGS IN MY OWN VERIFICATION, both of which would have made the safety check worthless. Recording them because a check you cannot trust is worse than none — it trains you to skip the one alarm that matters. 1. SNAP_DIR defaulted to $TMPDIR. Every `dev.sh run` is a FRESH container, so the DNS baseline was discarded between steps and `verify`'s before/after comparison — the check that detects damage to the other cluster's records — could never run across invocations. Now repo-relative (bind-mounted, so it persists on the host) and gitignored as a run artifact. 2. The first real comparison cried wolf: "RECORDS OUTSIDE prod.home CHANGED", listing pairs of IDENTICAL lines. Cause was locale-dependent sort collation — the host ordered `*.keithjackson.dev` differently from the container, so a diff of two identical sets reported changes. Proved it was noise by re-comparing under LC_ALL=C: all 57 records identical. dns_snapshot now strips CRs and sorts under LC_ALL=C so ordering is reproducible everywhere. ACTION REQUIRED (operator, pfSense): the local override is a wildcard on *.home.keithjackson.dev and now also swallows the new prod.home.* subtree, resolving EVERY prod hostname to 192.168.2.50 (dev). Cloudflare correctly returns 192.168.2.20. Prod is reachable from anywhere using public DNS, but NOT from behind pfSense. Narrow the override, or add one pointing prod.home.* at 192.168.2.20. That override is also the trap that produced a false healthy reading during the 2026-09-04 incident, and it is subtle because it PARTIALLY works: dev's *.home.keithjackson.dev wildcard matches prod.home.keithjackson.dev (one label) so the apex returns 200 over valid TLS, while keycloak.prod.home... (two labels) fails with a certificate-name error. I hit this myself during verification and mistook it for a broken prod cert. Never verify DNS or TLS here through a local resolver. Refs R-ENV-02, R-GITOPS-01, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
1b056dc9d9 |
fix(prod): break the CDI cold-start deadlock, and make preflight catch its class
`platform-config` stopped dead on prod:
StorageProfile/longhorn dry-run failed: no matches for kind
"StorageProfile" in version "cdi.kubevirt.io/v1beta1"
A TRUE DEADLOCK, not a transient. platform-config applies the CDI CR *and*
platform/kubevirt/cdi-storageprofiles.yaml in the SAME pass. StorageProfile's
CRD does not exist until cdi-operator has reconciled that CR — and Flux
server-side dry-runs EVERY object before applying ANY of them, so the unknown
kind aborts the whole tier, the CR is never applied, the CRD is never created,
and platform-config can never converge. Every tier below it stays blocked.
Same root cause as the Kyverno CRD bundle, different operator: Flux cannot order
a resource against a CRD that one of its own siblings is responsible for
creating. It is INVISIBLE on an established cluster — dev's CDI CRDs have been
there for months — and fatal on a cold one. Observed on prod: cdi-operator
Available for 12h, no CDI CR, no storageprofiles CRD.
`crds` now breaks it by applying ONLY the CDI CR — the minimum — and leaving
every other object to Flux. The CR carries no ${} tokens, so applying it here
cannot diverge from what Flux would render. It refuses if the `cdi` namespace is
absent (platform has not run yet) rather than applying into nothing, waits for
the CRD, and requires it to reach Established. On prod the CRD appeared in 10s
and platform-config was unblocked.
`preflight` now reports the CRDs that later tiers apply resources against
(storageprofiles/cdis/kubevirts/volumesnapshotclasses) and dumps the cdi
namespace when one is missing, so this is caught in seconds instead of after
`resume` burns its 900s wait discovering it. It also names the stale-discovery-
cache variant, which presents identically but clears on a forced reconcile.
THE DURABLE FIX IS A REPO CHANGE, deliberately NOT made here: move
cdi-storageprofiles.yaml out of platform-config into its own Kustomization with
dependsOn: platform-config, so the CR lands one reconcile before the resources
needing its CRDs. That edits the reconciliation graph both clusters share, and
prod currently reads a pinned revision from the bootstrap Gitea while dev is
mid-migration on a separate branch — so it wants doing deliberately, not at
02:00 to unblock a bring-up. Until it lands, C2 is a prerequisite for C5 on any
cold cluster, and PROD-STANDUP.md says so.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
4160fee6ce |
feat(prod): standup.sh + PROD-STANDUP.md — the complete standup sequence, in git
The last steps of a prod bring-up were living in shell history: resume the tiers in the right order, clear a wedged HelmRelease, fix external-dns without deleting the other cluster's DNS. That is the same failure the Kyverno CRD pre-apply had, so it gets the same treatment. bootstrap/standup.sh — environment-agnostic (R-ENV-02), no dev/prod fork. Six idempotent steps: preflight, crds, unwedge, externaldns, resume, verify. THE GUARD IS THE POINT. --expect-server is MANDATORY and is compared against the live API server before any step runs; there is no override flag. Dev is live and shares the keithjackson.dev Cloudflare zone with prod, and on 2026-09-04 prod deleted every one of dev's public records. Negative control run: prod's kubeconfig with dev's server URL refuses and changes nothing. preflight fails closed if EXTERNALDNS_TXT_OWNER_ID is empty, still the base sentinel, or `default` (dev's owner) — the exact condition that caused the incident. It also surfaces SUSPENDED HelmReleases separately, because a suspended release keeps its last Ready condition and a stale `Ready=True` otherwise hides that Flux is not managing it at all (external-dns today). externaldns is the dangerous step, and the ordering IS the mitigation. A Helm upgrade resets replicas to 1 and applies new args in ONE operation, so there is no window to check the result first; if --txt-owner-id were missing the pod starts as owner `default` and syncs away dev's records. So the step PROVES THE OUTCOME BEFORE CAUSING IT: render the chart with the LIVE HelmRelease values and assert --txt-owner-id=<owner> appears, aborting if not. Rendering is deterministic and touches nothing, which turns a race into a precondition. Then snapshot Cloudflare, upgrade, re-verify live args (scale to 0 and fail if wrong), and diff the zone. resume walks the dependency order and STOPS at the first tier that does not go Ready — a partial understood bring-up beats a forced green. velero is skipped while S3_BACKUP_ENDPOINT is the TBD placeholder. TWO BUGS CAUGHT WHILE VALIDATING THIS, both by measuring instead of assuming: 1. preflight counted Kyverno CRDs with grep 'kyverno.io' and reported 20 of 22 — the policy-report types are in the wgpolicyk8s.io group. A preflight that under-reports reads as "the pre-apply failed" when it succeeded. Fixed and re-run: 22. 2. The documented node-health invocation used -e k3s_hosts=k3s_prod and SILENTLY RAN AGAINST DEV'S THREE NODES. Diagnostic plays deliberately keep their own target var (node_health_hosts, memattr_survey_hosts, vpn_egress_gateway_hosts) so they stay pointable while a bring-up is scoped elsewhere — a good design I did NOT change. The hazard is that the wrong flag does not error, it falls through to the dev default. Caught only by --list-hosts; the doc now leads with that warning and a dry-run command. Verified after fixing: node-health and all six site.yml plays target prod. .claude/settings.local.json — permission rules for the above. Every entry hardcodes prod's kubeconfig AND prod's --expect-server (or prod's ansible groups), so the permission strings are themselves prod-scoped and cannot be turned against dev. Deliberately NO rule for `dev.sh run bash -c *`, which would have been blanket cluster access wearing a narrow-looking mask. Refs R-ENV-02, R-GITOPS-01, R-SEC-02. Co-Authored-By: Claude Opus 5 <[email protected]> |