04606d48b2bdab7996ecc0e605f75a0ec1704c34
54
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
daac05e7e8 |
feat(bulk): prod mounts the NFS export through pfSense's WAN address
The operator added the pfSense port-forward, so BULK_NFS_SERVER is no longer a placeholder. Prod gets 192.168.2.102 (pfSense WAN, DNAT tcp/2049 -> 192.168.1.9); dev stays on the base 192.168.1.9 because it is already on pfSense's LAN. This value could not be left as a placeholder safely: spec.nfs.server is unvalidated by Kubernetes, so the PV would be created, the PVC would BIND, and the failure would only surface as four pods stuck in ContainerCreating. Verified from k3s-prod-1 before committing, deliberately as uid 1000 rather than root (the export carries no_root_squash, so a root write proves nothing, and LSIO images default to PUID 911 which would go Ready and then fail every import EACCES): mount negotiated nfs4 vers=4.1 proto=tcp, write succeeded, and ln() across downloads/ -> tv/ gave link count 2 — real hardlinks, so no silent copy-then-delete. The Nextcloud export was additionally tightened to server-side ro= for prod's three nodes; a write as root is refused. Its content is config/, config_old/ and config_backup/ with no data/ directory, so it is an archive, not a live data store. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
2590bc4317 |
feat(env): BULK_NFS_SERVER — the one knob prod cannot answer yet, and why it is a placeholder
The bulk tier's PersistentVolumes need a server address, and there is no single correct one: the
two clusters sit on OPPOSITE SIDES of pfSense from the storage host, which has exactly one
interface (vmbr0 192.168.1.9/24, on pfSense's LAN).
dev 192.168.1.40/.41/.42 — the SAME segment. Reaches it directly, and already holds one NFS
export from it (/mnt/Storage/k8s-volumes, in the host's /etc/exports). Proven path, so
base's default is the dev value per this file's existing convention, and dev restates it.
prod 192.168.2.10/.11/.12 — pfSense's WAN. tcp/2049 does not traverse that today; 22 and 8006
answer only because pfSense port-forwards them via 192.168.2.102. Opening the storage path
is a PENDING operator decision whose two candidate outcomes give DIFFERENT values here:
192.168.2.102 (another port-forward) or 192.168.1.9 (a route). So prod holds
TBD-PROD-BULK-NFS-SERVER, this repo's established loud-placeholder convention.
Two findings recorded in the prod overlay because they change what the operator has to do:
* THE EXPORT'S CLIENT LIST IS THE SAME UNDER BOTH OUTCOMES. Measured today from this
workstation (192.168.2.132) through the existing port-forward: the storage host logged the
connection as coming from 192.168.2.132 — a WAN-side address — not from a pfSense LAN address
(`ss -tn` peer 192.168.2.132; the host's only route off-LAN is its default via 192.168.1.1).
pfSense DNATs WITHOUT source NAT, so the NFS server sees prod's REAL node addresses either
way. The `zfs set sharenfs=` work therefore does not have to wait for the routing decision.
* ONE PORT, NOT FOUR. The PVs mount nfsvers=4.1, and NFSv4 carries mount and lock in-protocol —
no rpcbind/111, mountd or statd. Whichever outcome is chosen, it opens tcp/2049 and nothing
else. NFSv3 would have needed four ports, three of them on ephemeral numbers.
*** AND ONE HONEST WEAKNESS, stated in the overlay where an operator will hit it. ***
Unlike the LAN knobs above it, this placeholder does NOT fail at apply time. A PV's
spec.nfs.server is an unvalidated string: the API server accepts it, the PV is created, and the
PVC BINDS (binding is pure control-plane). The failure appears only when a pod mounts —
MountVolume.SetUp "Failed to resolve server TBD-PROD-BULK-NFS-SERVER" — so plex, sonarr, radarr
and nextcloud would sit in ContainerCreating on prod. There is no CRD validation to lean on here,
so the guard has to be procedural: set the value before prod reconciles a revision carrying the
bulk volumes. Dev renders a working value and is where these manifests are meant to be proven.
Related finding, NOT fixed here because the file does not exist on main: the static guard this
convention is supposed to lean on, `bootstrap/standup.sh`'s `s3_is_placeholder` family, is absent
from main (it lives on unmerged prod branches), so
conformance/tests/l0-static/placeholder-guard-coupling exits 3 SKIPPED and nothing in the gate
currently refuses a placeholder-shaped overlay value. When standup.sh lands, its guard needs an
arm for this knob and a consumer that calls it — assertions (2) and (3) of that check then cover
this value automatically, since the detector's SEED vocabulary already matches "TBD".
Evidence: `kubectl kustomize clusters/dev` -> BULK_NFS_SERVER: 192.168.1.9;
`kubectl kustomize clusters/prod` -> BULK_NFS_SERVER: TBD-PROD-BULK-NFS-SERVER. Both exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
161430d915 |
feat(backup): the overlay picks the credential — BACKUP_CREDENTIALS_SUFFIX
The previous commit created the two halves. This one wires the choice, so dev renders MinIO
credentials and prod renders R2 credentials from the SAME base manifests (R-ENV-02: the
overlay patch is the entire difference between environments; no base manifest is forked).
cluster-vars BACKUP_CREDENTIALS_SUFFIX base "" | dev "" | prod "-prod"
velero HelmRelease credentials.existingSecret: velero-cloud-credentials${BACKUP_CREDENTIALS_SUFFIX}
3x CNPG Cluster s3Credentials.*.name: <db>-backup-credentials${BACKUP_CREDENTIALS_SUFFIX}
ONE SUFFIX, NOT FOUR SECRET-NAME KNOBS. Four independent knobs can be updated three-quarters
of the way, leaving one consumer reading the OTHER environment's key with nothing rendering
wrong — and StrictPostBuildSubstitutions cannot catch that: strict mode fails on an UNDEFINED
variable, never on a defined-but-stale one. With one suffix the four move together or not at
all. The cost is that the rendered Secret name is not greppable from the consumer, so both
kustomizations spell the two names out in full beside the resources.
THE BASE DEFAULT IS "" AND THAT IS LOAD-BEARING. It renders the UNSUFFIXED names — objects
that exist in every render, and that dev is running today. The alternative failure mode is
strictly worse than the shared-Secret state this replaces: velero's pod hangs in
ContainerCreating forever when `existingSecret` names a missing Secret (the volume never
mounts), and a CNPG Cluster whose s3Credentials Secret is absent stops archiving WAL — which
under archive_mode=on is a filling pgdata PVC, not a missing backup. So an unresolvable
default breaks the referencing cluster HARDER than a wrong key does. Both overlays state the
value explicitly anyway, so neither inherits it silently.
Prod's overlay change is a cluster-vars op only. The velero Flux Kustomization patch is
untouched (it stays decryption-only, and velero stays suspended on prod).
VERIFIED — `kubectl kustomize --load-restrictor LoadRestrictionsNone` on clusters/dev and
clusters/prod for the cluster-vars values, then on each of the four affected tiers (platform,
platform-config, platform/velero, apps/wave-0) piped through `flux envsubst --strict` with
that environment's variables:
strict envsubst PASSES for all 8 (env x tier) combinations
dev velero existingSecret velero-cloud-credentials
dev keycloak/nextcloud/semaphore-db <db>-backup-credentials (both key refs each)
prod velero existingSecret velero-cloud-credentials-prod
prod keycloak/nextcloud/semaphore-db <db>-backup-credentials-prod (both key refs each)
both all EIGHT Secret objects present in the rendered output (dev and prod alike)
no `${...}` survives in any rendered tier except gitea's three DELIBERATE `$${...}` shell
escapes in apps/wave-0, which are pre-existing and unrelated.
Read alongside the endpoints the same overlays already set, the two environments are now
isolated on credential AND target:
dev velero s3://velero (root) + cnpg s3://cnpg @ http://192.168.1.9:9000 region us-east-1
prod velero k3s-prod-backup/velero + barman prefix @ https://<acct>.r2.cloudflarestorage.com region auto
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
0a0527e7db |
feat(backup): prod's off-cluster DR lands on R2 — and stops the WAL clock
Prod has been archiving WAL to `TBD-PROD-S3-BACKUP-ENDPOINT` since bootstrap. Verified live 2026-09-07 on all three CNPG Clusters: ContinuousArchiving=False, "unexpected failure invoking barman-cloud-wal-archive: exit status 4". Because archive_mode is ON, that is not a degraded backup — it is a DISK-FULL CLOCK. Postgres will not recycle a segment it has not archived, so keycloak-db's pgdata PVC is at 579 MiB used / 4360 MiB free and climbing monotonically. When it fills, the primary PANICs and takes SSO down, and with it kubectl OIDC, Grafana, Headlamp, Gitea and forward-auth. Wiring the endpoint is what lets Postgres drain the whole accumulated backlog in one pass. THE KNOB LAYOUT (three new, one repurposed) and why it is shaped this way. The operator created exactly ONE R2 bucket, `k3s-prod-backup`, but the two backup tools want differently-SHAPED values: the Velero chart takes `bucket` and `prefix` as separate fields, while CNPG's barmanObjectStore.destinationPath is one full s3:// URL whose first path segment IS the bucket. So: S3_BACKUP_ENDPOINT dev http://192.168.1.9:9000 prod https://<acct>.r2... S3_BACKUP_REGION dev us-east-1 prod auto VELERO_BACKUP_BUCKET dev velero prod k3s-prod-backup VELERO_BACKUP_PREFIX dev "" prod velero CNPG_BACKUP_PATH dev s3://cnpg prod s3://k3s-prod-backup/barman There is deliberately NO separate `S3_BACKUP_BUCKET`: CNPG's bucket already lives inside CNPG_BACKUP_PATH, and a second knob holding the same bucket name twice is a rotation waiting to drift. Three knobs, each shaped for its own consumer, and no two of them can contradict each other. TENANCY. Inside k3s-prod-backup, velero and barman get disjoint TOP-LEVEL prefixes — s3://k3s-prod-backup/velero/ and s3://k3s-prod-backup/barman/<cluster>/ — with the bucket ROOT left empty so a misrendered prefix lands somewhere obviously wrong instead of silently colliding. This is not tidiness: Velero ADOPTS every backup under its location and TTL-DELETES it on retention, so an overlapping prefix would let it reap WAL segments; and interleaving two Postgres servers' WAL into one prefix corrupts the chain outright. Dev↔prod isolation is doubly independent — different endpoint AND different bucket names — so neither cluster can reach the other's objects even with a mis-substituted prefix. DEV IS UNCHANGED, BYTE-FOR-BYTE, and that is asserted rather than assumed. Base defaults are dev's LIVE values (verified against the running cluster: BSL bucket=velero with NO prefix, region us-east-1; all three CNPG Clusters ContinuousArchiving=True at s3://cnpg on the MinIO), and the dev overlay now restates all five explicitly for the same reason LONGHORN_DATA_PATH is restated — a silent drift in base would ORPHAN the backups dev's PROVEN restore depends on. The empty-prefix case was the one real risk, and it is settled empirically: with prefix "" the chart omits objectStorage.prefix from the rendered BackupStorageLocation ENTIRELY (helm template, chart 12.1.0), so dev's live BSL is unchanged, not merely equivalent. R2's TWO GOTCHAS, both wired up front: - region must be exactly `auto`. head-bucket answers {"BucketRegion":"WNAM"}, which is a PHYSICAL LOCATION HINT, not a signing region. This was hardcoded us-east-1 inside the Velero HelmRelease — correct-by-accident for MinIO, wrong for anything else — so it is a knob now. - checksumAlgorithm: "". velero-plugin-for-aws v1.14.2 bundles an AWS SDK with flexible checksums on by default and sends x-amz-sdk-checksum-algorithm / x-amz-checksum-crc32 on every PUT; R2 rejects those with HTTP 400 InvalidArgument, failing every backup and every kopia blob upload. It must be PRESENT-but-empty (the plugin branches on the key existing), and the chart does emit an empty value — verified by rendering it. Harmless against MinIO, so it is unconditional rather than a fourth knob. WHAT THIS DOES NOT DO. Velero stays SUSPENDED on prod. `spec.suspend: true` is owned by field manager `flux` (verified: kustomize-controller's managedFields do not include f:suspend), so server-side apply will not strip it, and nothing here declares suspension — which would also trip R-ENV-03, since a prod overlay may patch only cluster-vars and a Flux Kustomization's `decryption`. Unsuspending is the operator's, after `standup.sh --s3-endpoint <r2> s3check` proves a round trip. NO DNS KNOB IS TOUCHED. BASE_DOMAIN stays prod.home.keithjackson.dev and every EXTERNALDNS_* value is untouched. The apex inversion on work/prod-standup-campaign was deliberately NOT carried: this branch ports that branch's R2 reasoning knob by knob, and merges nothing. VERIFIED: ci/l0.sh green (flux-postbuild-substitution 272 assertions, 0 hazards, 33 knobs defined / 28 referenced, every new token resolved in BOTH overlays); render-golden, lint, schema, spec-immutability green; touched requirement R-STATE-03 covered. Refs R-STATE-03, R-IDP-05, R-ENV-02, R-ENV-03. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
d1a6628a97 |
feat(ha): the CNI dials the control-plane VIP — kube-vip becomes a CNI prerequisite
OPERATOR DECISION 2026-09-06: "use the VIP as intended — a virtual address to reach
the cluster and provide fault tolerant HA. Without a change it will not work in HA."
THE DEFECT. Cilium's kube-proxy replacement dialled a SINGLE NODE. Re-confirmed live
before changing anything: all three prod cilium-agent pods carried
KUBERNETES_SERVICE_HOST=192.168.2.10 — k3s-prod-1's own address — while the kube-vip
VIP 192.168.2.9 was Running on all three nodes with a held lease. Lose prod-1 and the
CNI's control-plane path is dead on an "HA" cluster. clusters/prod's own comment said
the value must be the VIP and NOT .10, then called it "Not yet decided" — I treated
that stale comment as an open question when the evidence already settled it.
SPLIT, NOT REPOINT, because k3s_api_host had two consumers with opposite needs:
k3s_api_host unchanged — the FOUNDER's IP: the cold-bootstrap join
target and a tls-san. infra/tofu/outputs.tf GENERATES this
name, so renaming would ripple into tofu and the runbooks.
k3s_apiserver_endpoint NEW = {{ kube_vip_address }} — the HA address; what Cilium
consumes. Derived, not restated, so an environment overrides
ONE line and the VIP, the SAN and the CNI endpoint move
together. Prod needed no new override.
THE SECOND HAZARD, WHICH MY BRIEF DID NOT NAME and the agent found: pointing Cilium at
the VIP breaks a COLD bootstrap for the same reason the join would — site.yml ran
node-baseline -> k3s -> cilium, so Cilium would install against an address nothing
announced yet. That is the 2026-09-01 deadlock shape. Fixed by SEQUENCE, not value:
site.yml now imports kube-vip.yml BETWEEN k3s and cilium. It cannot go earlier
(kube-vip is a k3s auto-deploy manifest, so it needs a cluster) and it works there
because the DaemonSet is hostNetwork with KUBERNETES_SERVICE_HOST=127.0.0.1 — needing
neither CNI nor kube-proxy. roles/kube_vip gained a read-only `wait_for` on the VIP so
a cold run stops THERE with a named cause instead of surfacing as unreachable agents.
Also fixed a pre-existing bug that only mattered once this play joined site.yml: the
role's read-only interface probe ran without `check_mode: false`, so under --check it
was skipped and the assert fired with a misleading "set kube_vip_interface explicitly".
VERIFIED BY ME, not just reported:
renders dev K3S_API_HOST=192.168.1.49 prod=192.168.2.9 (each environment's VIP)
cilium template consumes k3s_apiserver_endpoint, not k3s_api_host
cold join still `--server https://{{ k3s_api_host }}:6443` — the founder
order site.yml: node-baseline -> k3s -> kube-vip -> cilium -> verify -> flux
gate full CI gate PASS
Agent-side, additionally: in-memory render of cilium-values.yaml.j2 per host showing
k8sServiceHost=the VIP with the join still on the founder; k3s_tls_sans renders
['.10','.9','127.0.0.1','.11','.12'] so NO cert reissue and NO k3s restart is involved
— .9 already answers under full CA verification; and --check --diff of kube-vip.yml
against prod reported 6 ok / 0 changed, i.e. the role edits change nothing about the
kube-vip prod already runs.
ha-control-plane-as-code 92 -> 112 assertions, turning two escalations into machine
checks: the Cilium template must use the endpoint var; the endpoint must derive from
kube_vip_address and k3s_api_host must not; per environment K3S_API_HOST must equal
that environment's VIP and be none of its node IPs (the [parity] contract, now
machine-checked rather than a comment); site.yml must order k3s -> kube-vip -> cilium.
Every new assertion mutation-tested and bites. One honest negative recorded: reverting
only the BASE default is not caught, because both overlays patch the key — the check
reads effective per-environment values, so the base default is unguarded.
APPLY ORDER, and this is the operator's step — nothing was applied:
PROD FIRST, and it is the low-risk direction (prod is the broken one). Push and let
Flux reconcile platform/cilium: the value change rolls the cilium-agent DaemonSet
ONE NODE AT A TIME (maxUnavailable: 1 — the knob that bounded the 2026-09-01
incident), it will not proceed past an unready pod, existing datapath survives an
agent restart, and .9 is already live before the change lands.
DEV SEPARATELY, operator present. Same mechanics but live dev agents dial
192.168.1.50, and dev is mid-DNS-migration on dev-live. Do not fold it into that.
ANSIBLE: nothing to run for either live cluster; the Ansible half matters only for a
cold build. Remove the now-redundant standalone kube-vip.yml step from any runbook.
NEW NAMED GAP, unwritten: the L2 test cannot assert the running agents' live
KUBERNETES_SERVICE_HOST, which is the drift the L0 check cannot see. Recorded in that
test's gap list.
Refs R-PLAT-01, R-NET-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
f21e89d9d2 |
fix(gitops): CDI cold-start deadlock repaired IN THE GRAPH, not by a pre-apply
`platform-config` applied the CDI CR *and* cdi-storageprofiles.yaml in ONE pass.
StorageProfile's CRD does not exist until cdi-operator reconciles that CR, and
Flux server-side dry-runs every object before applying any — so the unknown kind
aborted the tier, the CR was never applied, the CRD was never created, and the
tier could NEVER converge. Observed on prod: cdi-operator Available 12h, no CDI
CR, no storageprofiles CRD, every downstream tier blocked.
The profiles now live in their own tier ./platform-config-cdi with
dependsOn: platform-config and wait: true. Because dependsOn is REVISION-gated
and platform-config runs wait: true, the CDI CR must be applied AND healthy —
exactly the event that creates the CRD — before the profiles are applied. Cold
clusters order themselves. `standup.sh crds` is retained as a fallback for
clusters already stuck, and because it is idempotent.
THE GENERALISABLE RULE (recorded in clusters/README.md): a CR whose CRD is
created by a CONTROLLER AT RUNTIME is second-order and needs its own pass.
cdi-operator.yaml registers exactly ONE CRD (cdis.cdi.kubevirt.io); storageprofiles,
datavolumes, datasources and dataimportcrons appear only after it reconciles the
CDI CR. An audit found StorageProfile is the only second-order kind in
platform-config — KubeVirt, CDI, KeycloakRealmImport, CNPG Cluster,
VolumeSnapshotClass, ClusterIssuer, PrometheusRule, ServiceMonitor and the Cilium
L2 CRs all have their CRD in git or in a chart.
Moving the CDI CR up into `platform` was rejected: that would put CDI's health on
the tier the CNI gates and everything waits on — the mistake the oauth2-proxy and
headlamp splits exist to undo.
apps-vms gains dependsOn: platform-config-cdi. LOAD-BEARING, not cosmetic:
apps/vms/unifi/20-datavolume.yaml provisions on storageClassName: longhorn-vm and
its comment claimed platform-config guaranteed the profiles — which this move
would have silently falsified.
THE PRUNE TRANSIENT, MEASURED RATHER THAN ASSUMED. platform-config has
prune: true, so it deletes both profiles once, CDI recreates them with empty spec,
and the new tier re-applies the overrides ~30-60s later. Compared our override
against CDI's unaided inference for the same Longhorn provisioner on dev:
longhorn-vm (we override) spec RWX/Block -> status ReadWriteMany/Block
longhorn-static (no override) spec unset -> status ReadWriteOnce/Block
Both infer cloneStrategy: snapshot. So cloneStrategy is NOT at risk; accessModes
IS — unaided CDI gives RWO. A DataVolume provisioned on longhorn-vm inside the
window would be stamped RWO/Block and be PERMANENTLY non-live-migratable. Flux can
no longer cause it (apps-vms gates on the new tier); a human must not do it by hand.
Corroborating evidence that the recreate happens at all: dev has 5 StorageClasses
and 5 StorageProfiles, three with fluxOwner=NONE, spec unset and status populated
— CDI creating and inferring them with no Flux involvement.
MERGE PRECONDITION: `flux get kustomization platform-config` must be Ready on the
target first. If it is not, the profiles are pruned and sit at CDI defaults until
it recovers — and prod was in that state hours earlier. Fallback is the two-step
the Cilium move used (merge with prune: false, verify the ownership label moved,
restore prune: true).
substituteFrom: cluster-vars is declared even though the manifest has no ${TOKEN},
deliberately: conformance/tests/l0-static/flux-postbuild-substitution only audits
paths whose Kustomization declares it, so omitting it would move the file OUT of
the audited set — a blind spot for a future literal-token ship. Gate confirms it
stayed in (path=./platform-config-cdi files=2 defined=0 hazards=0). No SOPS
decryption stanza: the moved file is two plain CRs, so no overlay change was needed.
VERIFIED INDEPENDENTLY OF THE AGENT: platform-config renders 0 StorageProfile
while keeping the CDI and KubeVirt CRs; platform-config-cdi renders exactly the
two profiles; the graph is byte-identical across dev and prod with apps-vms
carrying both edges and no cycle. Full CI gate PASS.
KNOWN FOLLOW-UP: bootstrap/standup.sh TIER_ORDER does not yet include
platform-config-cdi, so `resume` will not wait on it. Another agent holds that
file; the PM will add it.
Refs R-GITOPS-01, R-ENV-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
2d7d302c7e |
feat(prod): cut prod's GitOps source over to prod's OWN in-cluster Gitea
Prod reconciled from http://192.168.2.132:3000 — a podman container on the operator's WORKSTATION, holding a DHCP lease on WIFI. Production's source of truth was a laptop: a lease change or a closed lid stalls all reconciliation. Dev was pointed there too (deliberately, to break a strand risk during its DNS migration), so BOTH clusters depended on it simultaneously. Prod now reconciles from https://gitea.prod.home.keithjackson.dev/... — the Gitea prod itself runs, which apps-wave-0 has been serving for 28h. THE SELF-REFERENCE IS THE WHOLE DIFFICULTY, and it is why this is a commit rather than a kubectl patch. The root Kustomization renders ./clusters/prod, which CONTAINS the patch that sets GitRepository.spec.url — so a live patch is reverted on the next reconcile unless the AUTHORED value moves with it. Dev hit the identical trap on 2026-09-05. The ordered cutover: 1. push the repo to prod's Gitea (18 branches, refs verified identical, 499 commits of history — a real move, not a shallow copy); 2. fast-forward prod Gitea's `main` to |
||
|
jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
9b8979a427 |
feat(prod): temp Gitea source + wireless disabled + flux bootstrap path
Autonomous prod bring-up, operator away. Three changes that together clear the last blockers before `flux bootstrap`. === 1. TEMPORARY BOOTSTRAP GITEA — the last hard blocker, cleared === clusters/prod's source URL could not work: Gitea runs ON the dev cluster (the one slated for decommission) and split-horizon DNS resolves gitea.home.keithjackson.dev to PROD's OWN gateway. Stood up the existing bootstrap/gitea/bootstrap-gitea.sh on the operator's workstation with HOST_NAT_IP=192.168.2.132; all three prod nodes verified reachable (Gitea 1.22.6 via the API), no firewall change needed — prod is on the same /24, no NAT. clusters/prod now points at http://192.168.2.132:3000/... under a boxed TEMPORARY banner with a 5-step cutover checklist ending 'delete this whole block — dev's stanza is what done looks like'. Repo verified PUBLIC with anonymous read two independent ways (API "private":false; git ls-remote unauthenticated with GIT_TERMINAL_PROMPT=0), so NO secretRef is needed — documented so a future auth failure is not misread as a network fault. bootstrap-gitea.sh's header corrected: it documented HOST_NAT_IP=10.73.0.1, the RETIRED libvirt NAT, and forbade agent use; the operator has now directed this for prod. === 2. WIRELESS DISABLED ON PROD (operator-reported) === k3s-prod-3 was on ethernet AND wireless, and the wifi DHCP client was overwriting /etc/resolv.conf. Root cause of the silence: node_baseline's DNS `contended` detection checked resolvconf and NetworkManager but NOT dhcpcd, so the role wrote resolvers, reported success, and dhcpcd reverted them on lease renewal — the exact 'green run that fixed nothing' the contended branch exists to prevent. Now IaC, not a one-off: survey wireless hardware, **REFUSE to disable it if the path to the node runs over the wireless interface** (the guard that meant an agent dying mid-apply cost nothing), blacklist the driver so it never reloads, stop dhcpcd only where it holds a wireless interface and nothing else, take links down and unload modules with NO reboot, then report the resulting state. dhcpcd is now in the contended set without weakening fail-closed. Applied and verified: node 3's wlp1s0 is GONE from /sys/class/net, dhcpcd inactive and its unit not-found, ONE default route per node, all three on 8.8.8.8/1.1.1.1 resolving public names. Safe and idempotent on nodes 1 and 2, whose wifi hardware was already down. === 3. flux_bootstrap PATH PARAMETERISED === bootstrap/flux/bootstrap.sh already read FLUX_ENV_PATH but the role never exported it, so it ALWAYS bootstrapped clusters/dev — prod was impossible. Two behaviour-bearing lines: role default clusters/dev + `environment: FLUX_ENV_PATH`, overridden to clusters/prod in group_vars/k3s_prod. Proven per host with a read-only Ansible probe: prod hosts resolve clusters/prod, localhost/hypervisor resolves clusters/dev. Only two definitions repo-wide. DEV PROVEN BYTE-IDENTICAL for the source change: all 12 substituting Flux paths rendered for both trees, 27/27 SHA-256 artefacts identical, with TWO POSITIVE CONTROLS (an overlay-level canary and a deep-path content canary) confirming the harness detects change — and both canaries proven restored. Honest scope note recorded: kustomize strips comments, so the harness proves the built stream, not source comments. L0 substitution check: 106 assertions, 0 hazards. ci/l0.sh and ci/lint.sh exit 0. === STILL BLOCKING A FULL PROD RECONCILE (operator decisions) === - S3_BACKUP_ENDPOINT remains TBD-PROD-*: prod CANNOT reach dev's versitygw at 192.168.1.9:9000 (verified UNREACHABLE from all three — the pfSense boundary), and a prod backup target is a deferred operator decision. Deliberately NOT invented: a DR target on a workstation container would create false confidence. Velero/CNPG backups will fail loudly. - Prod's public resolvers mean the API server's oidc-issuer-url resolves PUBLICLY, so kubectl SSO will not work on prod until split-horizon is solved for it (§4). - The temp Gitea is a single container on the operator's workstation — a single point of failure for prod reconciliation, acceptable for bring-up only. Serves: R-GITOPS-01, R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03. Task: T-7.1. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
05baae90ec |
feat(prod): kernel fix + disk layout — prod nodes ready for k3s
Prod is being stood up now, without waiting for node 2's RAM upgrade. Three coupled
changes, all applied and verified on all three nodes.
=== 1. KERNEL >= 6.13 — now a PREREQUISITE, not a nice-to-have ===
A k3s SERVER that intermittently boots with 2.6 GiB is not a degraded node; it fails under
etcd plus its workload share, unpredictably, on any reboot. Fixed before k3s goes on.
roles/node_baseline gains a `kernel`-tagged block: asserts the suite matches the node's
Debian release, writes a deb822 trixie-backports source, installs, reads back the exact
version, writes a provenance/rollback note to the node. Scoped by
node_baseline_backports_kernel (default FALSE, true only in group_vars/k3s_prod) — chosen
over a hostname test so a fourth prod node inherits it from group membership. Verified:
--check against dev skips all 8 tasks on all 3 hosts. Two deliberate calls: `state: latest`
not `present` (linux-image-amd64 was already installed from main, so `present` would have
silently fixed nothing), and NO preferences.d pin (backports' NotAutomatic +
ButAutomaticUpgrades already gives 100/500; an explicit Pin-Priority: 100 would switch OFF
automatic security upgrades for the very kernel being installed).
Installed 7.1.8+deb13-amd64 on all three; 6.12.107 kept bootable.
THE DECISIVE PROOF: node 2 booted at MEMATTR=0xbeeb0018 — the EXACT address that gave it
2.6 GiB twice — and now reserves 634,904K instead of 13,958,980K. MemTotal 15.4 GiB. Same
address, handled correctly; not a lucky draw.
§0b's claim that a fixed kernel prints 'FW_BUG Corrupted...' is CONDITIONAL and was
corrected: >=6.13 has three rejection gates and only the third prints that line; all three
nodes hit gate 1 (version > 2). Proof instead is boot-independent — decompressing
/boot/vmlinuz on all three yields the FW_BUG and descriptor-size strings verbatim; neither
exists in 6.12.
TWO MORE BUGS IN MY OWN HEALTH CHECK, found because the fix couldn't otherwise be shown:
- MEMATTR_VERSION_RE matched (\d+) but the kernel prints that field with %d — SIGNED. Node
2's real value was -1569980416, which did not match, so state fell through to `clean` and
the node reported PASS "the kernel accepted it without complaint". A FALSE PASS on the
exact bug the check exists to find. Now (-?\d+).
- bad_version was WARN unconditionally, so a FIXED node would have stayed WARN forever.
Now keys on memattr_kernel_fixed: same line, WARN below 6.13, PASS at/above.
=== 2. DISK LAYOUT — Longhorn claimed, per-node paths DISCOVERED not hardcoded ===
Operator decision: reuse the ~424 GB partition; accept the shared-device I/O risk (dev's
etcd/Longhorn contention was on a spinning HDD, prod is SSD/NVMe).
Device paths differ per node (sda5 / nvme0n1p5 / sda5), so the role DISCOVERS the partition
via `findmnt --mountpoint` on a declared claim mountpoint rather than trusting inventory.
Per-host host_vars would have frozen today's hardware into inventory, where it goes stale on
the first re-image and the stale value's failure mode is mkfs on the WRONG partition.
Eleven fail-closed asserts before any write: real block device, not backing / /boot /usr
/var /home /tmp or active swap (built from the node's OWN gathered facts, never a hardcoded
list), already ext4, no foreign label, >= 100 GiB, and empty. Relabel, never reformat.
/srv verified empty first on every node (ls -A -> lost+found only; du 20K; the "5.1% used"
was ext4's default 5% root reserve).
=== 3. k3s DATA-DIR MOVED — /var was 11.8 GiB and would have killed etcd ===
Measured on the fully-deployed dev cluster: containerd 21-25 GB/node, /var total 58-70 GB.
Prod's /var is 11.8 GiB and held BOTH etcd and the image store, so prod would have filled
/var during its first image pulls — and a full /var means etcd cannot write its WAL. Cluster
death, not degradation, arriving during bring-up. Invisible when the disk decision was made,
because "etcd on root" sounded like the 30 GiB / rather than a separate 12 GiB /var.
Now: p5 -> /data (LABEL=k3s-data), with /data/longhorn and /data/k3s. Neutral names chosen
deliberately — the partition now has two tenants, and reusing /srv would make "claimed" and
"unclaimed" indistinguishable to the idempotency pivot, while a bare `data` label would make
a stranger's `data` volume look already-claimed and weaken an assert.
The transition off the existing LABEL=longhorn mount is DECLARED, not inferred
(node_baseline_data_retired_fs_label/_retired_mount, defaults ""), because changing the label
makes a claimed node answer "no" to the pivot and walk into the greenfield path. Self-retiring
once blkid -L k3s-data succeeds. THREE independent gates keep live dev out: retired vars
default ""; gated on longhorn_source == claim-mountpoint (dev is `device`); and the emptiness
assert fails closed on dev's 18 volumes even if both were defeated.
k3s_data_dir is one variable read by two roles (k3s_server --data-dir; node_baseline derives
the etcd mount). Role default is k3s's own default, so dev is unchanged and keeps its
dedicated etcd device. node_baseline_etcd_source becomes `shared-data-partition` and is
LOAD-BEARING: the role asserts per run that the filesystem backing the etcd dir really IS the
data partition — had the mount silently failed, etcd plus 21-25 GB of images would land back
on the OS filesystem invisibly until the disk filled.
LONGHORN_DATA_PATH is a new cluster-vars knob (R-ENV-02): base and DEV /var/lib/longhorn,
prod /data/longhorn.
DEV PROVEN UNCHANGED, which is the property that mattered — the live cluster has 18 Longhorn
volumes on that path. All 12 substituting Flux paths rendered through
`kubectl kustomize | flux envsubst --strict` for both trees: 12/12 per-file SHA-256 identical
and an identical aggregate SHA over every rendered dev byte. With a POSITIVE CONTROL showing
the same harness detects prod changing on exactly one line — an all-identical result is
worthless if the harness cannot see change. Live check after: dev's default-data-path setting
still /var/lib/longhorn, 18 volumes attached, 0 Kustomizations unready.
Idempotent: run 2 is changed=0 on all three nodes. Also fixed a real tag bug — include_tasks
does NOT propagate its own tags to included tasks (unlike import_tasks), so `--tags disks`
selected the include and ran nothing inside it while reporting success.
OPEN DEFECTS, tracked not forgotten: node 2's warm-reboot hang (one clean reboot since, NOT a
fix); and node 3 runs dhcpcd for its wifi interface, which rewrites /etc/resolv.conf on lease
renewal — the role's `contended` detection checks resolvconf and NetworkManager only, so node
3's decided resolvers will silently revert.
Gate: MERGE GATE: PASS. L0 substitution check 106 assertions.
Serves: R-ENV-01, R-ENV-02, R-ENV-03, R-STATE-01, R-STATE-02, R-PLAT-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
22174186ff |
chore(flux): restore platform prune:true — MERGE 2 of 2 (Cilium split cleanup)
Closes the transition opened in merge 1 (
|
||
|
jakman85andClaude Opus 5
|
63f98aab26 |
feat(flux): give Cilium its own Kustomization — MERGE 1 of 2 (transition)
Cilium was one of ~12 components inside a single `platform` Kustomization with wait:true,
so the CNI's health was indistinguishable from eleven unrelated components in the tier's
Ready condition, and the only emergency valve was freezing the whole platform tier. This
repo already set the precedent in reverse (oauth2-proxy and headlamp were split OUT so
unhealthy leaves would not gate a tier); Cilium is the mirror image — the one component
that legitimately SHOULD gate everything.
New clusters/base/cilium.yaml: path ./platform/cilium, wait:true, prune:true, interval 1h
(a CNI does not need a 10m re-render cadence; kustomize-controller is source-triggered so
a push still reconciles immediately — the interval is only the drift-correction floor),
timeout 10m (a 3-node DaemonSet rollout with wait exceeds the 5m default),
dependsOn:[flux-system], and postBuild.substituteFrom: cluster-vars — the last is
LOAD-BEARING, not decorative: helmrelease.yaml references ${K3S_API_HOST} and
kustomize-controller now runs --feature-gates=StrictPostBuildSubstitutions=true, so
without it this Kustomization would fail immediately.
*** THE HAZARD WAS FAR WORSE THAN "the CNI gets pruned" ***
platform/cilium/namespace.yaml is the ONLY declaration of Namespace/kube-system anywhere
under ./platform, and the live object carries kustomize.toolkit.fluxcd.io/name=platform.
With prune:true, the moment ./platform/cilium leaves platform's path the stale set is not
"the Cilium HelmRelease" — it is `kubectl delete namespace kube-system`, taking coredns,
metrics-server, local-path AND Cilium. A full cluster kill, not a CNI kill.
TWO INDEPENDENT PROTECTIONS, deliberately not trusting either alone:
1. prune:false on `platform` for this transition merge — Flux's documented Procedure 1 for
moving resources between Kustomizations. Chosen over the per-resource
`kustomize.toolkit.fluxcd.io/prune: disabled` annotation because that needs THREE merges
and leaves the new Kustomization owning resources it can never prune.
2. The pruner's own owner-label filter. In v1.4.0, prune() passes Inclusions:
GetOwnerLabels(obj.Name...), so platform only deletes objects whose LIVE labels still
say name=platform. The new cilium Kustomization relabels all three to name=cilium on
apply, and platform.dependsOn:[cilium] is REVISION-GATED, so platform provably cannot
reconcile revision N until cilium has APPLIED revision N. The relabel deterministically
precedes the prune.
Protection 2 alone would probably have sufficed. Not betting kube-system on "probably".
NOTED FOR THE FUTURE, in-file: anyone moving a platform/* component must first check
whether it is the sole declarer of a shared Namespace — that turns a component-scoped
prune into a cluster-scoped one.
Verified: `kubectl kustomize platform` goes 88 -> 85 documents, 3 removed / 0 added /
0 CHANGED, and `kubectl kustomize platform/cilium` reproduces those 3 byte-identical
(sha256-matched). A pure ownership move; no Cilium value touched. L0 substitution gate
103 assertions (was 102), new cilium row, 0 hazards. Server-side dry-run against the live
cluster accepted both CRs. clusters/prod renders clean off the same base.
MERGE 2 (restore prune:true) follows only after verifying: 13/13 Ready; kube-system, the
HelmRelease and the HelmRepository all relabelled to name=cilium; platform's inventory no
longer lists them; and Cilium pod ages/restarts UNCHANGED from
cilium-2f8h8 r=0 2026-09-01T05:55:27Z / cilium-425m8 r=1 2026-08-30T13:25:10Z /
cilium-54cww r=0 2026-09-01T05:55:26Z, HelmRelease v11.
Serves: R-PLAT-01, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6b4da27f84 |
feat(prod): SOPS decryption (same age key as dev) + GitOps source on main
Operator decisions, 2026-09-01:
1. Prod uses the SAME SOPS age key as dev.
2. Prod tracks the SAME BRANCH as dev (`main`). Dev is a temporary instance and will be
decommissioned once prod is deemed stable.
SOPS: eight `decryption: {provider: sops, secretRef: sops-age}` patches mirroring dev
exactly — platform, apps-wave-0/1/2, oauth2-proxy, headlamp, crafty, velero. The sops-age
Secret is still created OUT OF BAND on prod (bootstrap/sops/install-age-key.sh); the
overlay references it by name only, never inlines key material.
RECORDED CONSEQUENCE of the shared branch: once prod is live, EVERY MERGE TO main
RECONCILES BOTH CLUSTERS. There is no promotion step and no soak window — dev and prod
move together. That is a deliberate consequence of dev being temporary, not an oversight,
but it removes the "prove it on dev first" safety net that the last few days have leaned
on heavily. Worth revisiting if dev outlives its expected life.
*** BOOTSTRAP BLOCKER recorded, NOT papered over *** — the source URL cannot work on a
cold prod cluster, for two independent reasons found while wiring this:
1. SELF-HOSTING CIRCULARITY. Gitea runs ON THE DEV CLUSTER — the one slated for
decommission. Prod syncing from it means prod's source of truth lives inside the
cluster being retired. Gitea must move to prod (or off-cluster) BEFORE dev is torn
down. This makes Gitea migration the FIRST task of the decommission plan, not the last.
2. SPLIT-HORIZON DNS SENDS IT TO THE WRONG PLACE. platform/coredns/coredns-custom.yaml
answers the ENTIRE home.keithjackson.dev zone with ${GATEWAY_VIP} via a template
plugin. On prod that is 192.168.2.20 — PROD's own gateway, where no Gitea listens. Prod
Flux would resolve gitea.home.keithjackson.dev to itself and never reach a repository.
This is the same split-horizon mechanism that makes dev work, inverted against us.
Bring-up therefore needs a TEMPORARY source (the bootstrap host-Gitea, or an off-cluster
mirror) with a cutover to the in-cluster URL once Gitea actually runs on prod — precisely
the sequence docs/05 describes for dev's own Phase-6 cutover. The branch is settled and
correct; only the host is blocked. Documented at the patch site and in
PROD-PROVISIONING.md §7.
Gate: MERGE GATE: PASS.
Serves: R-SEC-01, R-GITOPS-01, R-ENV-02, R-ENV-03. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
666a60776a |
fix(flux): close the substitution race — platform dependsOn root + strict substitution (P0/P1)
P0 — clusters/base/platform.yaml gains `dependsOn: [flux-system]`. This is the actual fix for the incident, not the feature gate. Verified against kustomize-controller v1.4.0 `kustomization_controller.go :: checkDependencies` (our version): dependsOn is REVISION- gated, not merely Ready-gated — when the dependency shares a GitRepository, the dependent cannot reconcile revision N until the dependency has APPLIED revision N. Applying revision N is exactly what writes a newly-added cluster-vars knob. Graph verified mechanically, not by eye: built clusters/dev and clusters/prod and walked the resulting objects. `platform` was the ONLY Kustomization without a dependsOn, and all 11 substituting Kustomizations chain transitively through it (platform-config -> platform; the waves, apps-vms, oauth2-proxy, headlamp, homepage, crafty, velero -> platform-config). The "not chaining through platform" set is EMPTY, so this one edge protects the whole graph. DFS over the full edge set after the edit: no cycles. The root (clusters/dev/ sync.yaml) runs wait: false, which is what makes the edge deadlock-free — it reaches Ready on apply success without health-gating its children. Accepted trade-off, documented in-file: the root becomes a gate. A malformed manifest anywhere under clusters/<env> — including clusters/base — now stalls platform and everything downstream, where today it would not. Fail-closed, per doctrine. P1 — StrictPostBuildSubstitutions. MY PREMISE WAS WRONG about where this belongs and it matters: this repo deliberately does NOT use `flux bootstrap` (see bootstrap/flux/README). There is no gotk-components.yaml in git, and the live kustomize-controller Deployment carries NO Flux ownership labels — it was installed imperatively by `flux install` from bootstrap.sh and is not in Flux's inventory. `flux-system` self-manages the CR graph, NOT the controllers. Nothing under clusters/** can reach a controller flag. So it uses Flux's documented mechanism for hand-installed clusters: a new bootstrap/flux/components/kustomization.yaml wrapping `gotk-components.yaml` with a JSON-6902 patch appending --feature-gates=StrictPostBuildSubstitutions=true to kustomize-controller. JSON-6902 `add /args/-` deliberately, NOT strategic merge — a container's args is atomic and a merge patch would replace all five install-generated args. bootstrap.sh step 1 now renders (`flux install --export` into a temp dir, apply the patch, ASSERT the gate string is present so a future Flux bump that breaks the patch target fails loudly) then applies. FLUX_INSTALL_PLAIN=1 is a loud escape hatch. Nothing is vendored, so components can never drift from the pinned FLUX_VERSION. Rejected: vendoring gotk-components.yaml under the root's path (the root's prune: true would put Flux's own controllers and CRDs in a prune inventory), and shipping a partial Deployment as a Flux-managed resource (a future prune would delete kustomize-controller). ROLLOUT ORDER MATTERS. P0 arrives automatically on the next reconcile. P1 does NOT arrive on a git push — the controller Deployment is outside Flux's inventory, so an operator must re-run bootstrap.sh step 1, which restarts kustomize-controller (~10-20s; Flux is level-triggered so an interrupted reconcile is simply retried). P0 must be reconciled BEFORE P1 is rolled out: with the gate on but the ordering edge absent, the same race turns a transient empty render into a transient hard Kustomization failure. ALSO FIXES A PHASE-7 BLOCKER THIS CHANGE WOULD OTHERWISE HAVE CREATED. clusters/prod had no root Kustomization — dev/sync.yaml even carried the note "a prod overlay needs its own sync.yaml -> ./clusters/prod". With the new dependsOn, a prod bootstrap would have left platform blocked forever on a flux-system Kustomization that does not exist, and nothing downstream would ever reconcile. clusters/prod/sync.yaml is now authored, mirroring dev. Two further prod-overlay gaps found and DOCUMENTED, not guessed (PROD-PROVISIONING §7): prod has no SOPS `decryption` patch (every encrypted Secret would fail to decrypt — needs the prod age-key decision, a security choice not a copy), and no GitRepository source URL patch (needs a decision on whether prod tracks the same branch as dev, which would make a dev-facing merge reconcile prod). Substitution gate re-run: check OK, 102 assertions, 0 hazards. Gate: MERGE GATE: PASS. Serves: R-ENV-02, R-GITOPS-01, R-PLAT-01, R-ENV-03. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
ff0533106b |
feat(prod): operator decisions — node DNS, service VIPs, become vault, collections path
1. PROD NODE DNS = 8.8.8.8 + 1.1.1.1 (operator, 2026-09-01). Two INDEPENDENT public
upstreams, so the single-upstream SPOF that broke containerd pulls and CoreDNS
forwarding on dev cannot recur. Prod deliberately differs from dev, which is
PiHole-first because dev needs the local jackson.lan zone.
SCOPE CORRECTED IN THE COMMENTS. Three resolvers answer *.home.keithjackson.dev and
only one is configured here:
pods -> CoreDNS -> platform/coredns/coredns-custom.yaml, whose `template` plugin
answers the WHOLE zone with ${GATEWAY_VIP}. Self-contained in-cluster;
PiHole is NOT in this path. Prod is already correct (.20).
LAN -> PiHole. Not a cluster concern.
nodes -> THIS FILE.
The one thing public node DNS breaks: the k3s API SERVER is a node process carrying
oidc-issuer-url=https://keycloak.home.keithjackson.dev/realms/homelab and reaches the
issuer through the NODE resolver, so prod would get the PUBLIC record instead of the
prod Gateway VIP — breaking kubectl OIDC SSO. Dev avoids this only because its node
resolver IS PiHole. Not blocking (prod has no cluster or Keycloak yet); options are
recorded in PROD-PROVISIONING.md §4 for the operator to pick.
2. SERVICE VIPs DECIDED: UNIFI_L4_VIP 192.168.2.21, CRAFTY_L4_VIP 192.168.2.22 — the
first two addresses after GATEWAY_VIP (.20), mirroring dev's .53/.54/.55 ordering
within the .20-.30 pool. S3_BACKUP_ENDPOINT stays TBD-PROD-* : no prod backup target
exists, and dev's versitygw is across the pfSense boundary on the dev host, so it is
not DR for prod.
3. PROD BECOME VAULT committed, ansible-vault encrypted, matching the hypervisor and
proxmox_hosts pattern. Verified working: `ansible k3s_prod -m command -a 'id -un'
--become` returns root. The L0 no-plaintext-secrets check asserts every ansible
vault.yml starts with $ANSIBLE_VAULT, so this stays honest by gate.
4. TOOLCHAIN GAP FIXED: the devcontainer image does NOT ship the Galaxy collections, so
EVERY ansible play died at parse time on community.general.modprobe. ansible.cfg now
sets collections_path = collections, and the collections are installed there from the
existing collections/requirements.yml (gitignored — pinned by requirements, fetched at
setup, vendored bytes are not source).
5. DRIVE HEALTH MEASURED and recorded in §2 with the raw attribute table so the numbers
are auditable. Verdict: usable, but not for production etcd/Longhorn state — 31% of
rated life consumed, 15 grown bad blocks, 187 erase failures, and 32 unexpected power
losses on a consumer drive with NO power-loss protection, in a single failure domain.
etcd is the worst possible tenant: it fsyncs constantly and unclean power loss without
PLP is exactly how a member corrupts.
6. NON-ECC RAM: RISK ACCEPTED by the operator, recorded as a CLOSED decision, not an open
item — do not re-litigate. Rationale: 3-node k3s homelab, not enterprise; corruption on
one node is outweighed by the other two and causes node ejection. That holds for the
dominant mode (etcd raft CRC + quorum; Longhorn 3 replicas + hard anti-affinity +
rebuild). Recorded residual: corruption introduced UPSTREAM of replication replicates
faithfully to all three replicas, so quorum sees no disagreement — low probability, and
the only mitigation is different hardware. Follow-up noted: Longhorn's
snapshotDataIntegrity is left at the chart default rather than set explicitly, and it
is the mechanism that operationalises the operator's own argument.
Serves: R-DNS-01, R-DNS-03, R-ENV-02, R-ENV-03, R-SEC-01. Task: T-7.1.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
3f980d2a17 |
feat(env): tokenize hardcoded dev LAN addresses into cluster-vars (R-ENV-02)
R-ENV-02 promises dev and prod differ ONLY by scoped cluster-vars overrides. In fact
dev LAN addresses were hardcoded into base manifests, so clusters/prod would have come
up ANNOUNCING DEV ADDRESSES (192.168.1.x) on the prod LAN (192.168.2.x).
New cluster-vars knobs (base default = dev literal, per the file's existing convention):
K3S_API_HOST platform/cilium/helmrelease.yaml (+ the values.yaml mirror)
CILIUM_LB_POOL_START platform-config/cilium/l2-ip-pool.yaml
CILIUM_LB_POOL_STOP "
UNIFI_L4_VIP apps/vms/unifi/50-service-l4.yaml
CRAFTY_L4_VIP platform/crafty/service-l4.yaml
S3_BACKUP_ENDPOINT platform/keycloak/postgres-cluster.yaml, apps/wave-0/{nextcloud,
semaphore}-db.yaml, and platform/velero/helmrelease.yaml
Also fixes GATEWAY_VIP's base default, which was still the RETIRED libvirt address
10.73.0.240, and adds the GATEWAY_VIP patch that clusters/prod was missing entirely.
clusters/base/apps-vms.yaml gains postBuild.substituteFrom (it was the one consuming
Kustomization without it, so a ${TOKEN} there would have shipped as a literal string).
The envsubst contract is documented there: ${VAR} substitutes and escapes as $${VAR};
$VAR and $(cmd) are left alone; ${ARR[0]} hard-errors. The three UOS_URL expansions in
the unifi cloud-init are escaped accordingly.
PROD ADDRESS PLAN (operator, 2026-08-31):
nodes .10/.11/.12 · control-plane VIP .9 (Ansible-side) · GATEWAY_VIP .20
Cilium LB pool .20-.30
CONTAINMENT INVARIANT, now documented at the pool and in the prod overlay: GATEWAY_VIP
MUST equal CILIUM_LB_POOL_START. The shared Gateway pins no address, so Cilium LB-IPAM
gives it the first free address of a matching pool block. An initial prod proposal put
.20 outside a .21-.30 pool; the Gateway would have come up on .21 while split-horizon
coredns-custom kept answering *.home.keithjackson.dev with .20 — an address nothing
announces — blackholing in-cluster OIDC hairpin traffic while reporting healthy. The
pool was widened to .20-.30. The control-plane VIP (.9) stays outside the pool so
kube-vip and LB-IPAM can never contend.
Values prod still cannot know stay INVALID `TBD-PROD-*` placeholders on purpose: base
defaults are dev values, so an omitted patch means prod silently inherits a dev address
— far worse than a loud failed apply.
DEV IS BYTE-IDENTICAL. Verified by kustomize build over 13 paths before/after, then
flux envsubst with the dev cluster-vars values applied to both sides: zero drift. The
only deltas are the intended additive keys in the overlay ConfigMap itself. No
unresolved ${TOKEN} survives. render-golden green (10 goldens); yamllint clean.
Serves: R-ENV-02. Task: T-7.1 (prerequisite). Gate: MERGE GATE: PASS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
cbc2301b30 |
docs: reconcile docs/comments with post-upgrade cluster state
Text-only cleanup after the upgrade campaign. Updates stale version strings, prose, and comments to the verified current facts; no Kubernetes behavior or version pins changed except the kubectl toolchain bump (below). Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie, Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB, dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18, Grafana 13.2.0. - Version strings across infra/ and platform/ READMEs + comments - kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew) - Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh) - COMPLETED/historical banners on the three executed plan docs - Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges - Appended dated DECISIONS-LOG entry (2026-08-30) superseding the 2026-08-26 LIVE STATE line Left as revalidation items (not blind-edited): external-snapshotter v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1, conformance ImageVolume feature-gate assertions under 1.36. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d1248e6778 |
feat(backup): wire Velero + CNPG Barman to the live MinIO S3 target
Author the GitOps side of off-cluster backups against the on-Proxmox MinIO (http://192.168.1.9:9000, path-style; buckets velero + cnpg). Velero (R-STATE-03): - HelmRelease (chart 8.x / Velero 1.15) + velero-plugin-for-aws init container + node-agent kopia File System Backup (deployNodeAgent, uploaderType=kopia, snapshotsEnabled=false). BSL default -> bucket velero, s3ForcePathStyle. - Two daily Schedules: `daily` backs up the whole cluster's resources+PVC data (defaultVolumesToFsBackup opt-out) EXCEPT monitoring+unifi; `daily-resources-only` captures monitoring+unifi RESOURCES only (Prometheus TSDB, Loki 72h logs, and the ~7.4GiB UniFi VM disk are reconstructable — excluded from volume backup). - velero-cloud-credentials SOPS Secret (cloud INI key, ns velero). - Own non-gating Flux tier clusters/base/backup.yaml (dependsOn platform-config, wait:false) + clusters/dev SOPS decryption patch — mirrors headlamp. CNPG native Barman PITR (R-IDP-05 / R-STATE-03): - keycloak-db, nextcloud-db, semaphore-db gain spec.backup.barmanObjectStore (s3://cnpg/<cluster>, endpointURL MinIO, gzip WAL+data, 30d retention) + a daily ScheduledBackup (6-field cron, immediate) each. - Per-namespace scoped-key SOPS Secrets (<db>-backup-credentials), same-ns only. - render.py: emit-waves now recognizes <svc>-db-backup-credentials.sops.yaml as a DB-wave sidecar (+ self-test); wave-0 kustomization regenerated. Creds mirror platform/backup/minio-credentials.sops.yaml (scoped key k3s-backup), SOPS round-trip verified with the cluster age key. gate: green. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
0437871807 |
Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI
Add Crafty Controller 4 (registry.gitlab.com/crafty-controller/crafty-4:4.10.8) as a platform component: a web panel that runs Java + Bedrock Minecraft servers as subprocesses in one pod + a 20Gi Longhorn PVC (mounted 5x by subPath). Auth = Option B (per-app oauth2-proxy reverse-proxy sidecar), since Cilium 1.16.5 cannot do ext_authz on the shared Gateway: - oauth2-proxy v7.8.1 sidecar (:4180) reverse-proxies to Crafty's self-signed HTTPS UI (127.0.0.1:8443), gated to Keycloak homelab realm group platform-admins. - Confidential `crafty` client provisioned live via kcadm + captured in the realm import; client secret + cookie secret via SOPS/age (mirrored in realm-clients). - Web UI: HTTPRoute crafty.home.keithjackson.dev -> Service :80 -> sidecar :4180. - Game ports: Cilium L2 LoadBalancer on VIP 192.168.1.55 (next free after .53/.54), starter set TCP 25565-25567 (Java) + UDP 19132-19134 (Bedrock), NOT auth-gated. - fsGroup:0 / runAsGroup:0 / runAsUser:1000 for the image's crafty:root identity on the root:root Longhorn PVC (the seerr fsGroup lesson). - Own non-gating Flux tier clusters/base/crafty.yaml (dependsOn platform-config, wait:false) + sops-decryption patch in clusters/dev — a leaf UI must never gate the platform. - Explicit default-deny NetworkPolicy + CiliumNetworkPolicies (platform namespace, not app-namespace, so Kyverno does not auto-generate the baseline). make -C ci gate green; kustomize build of platform/crafty, clusters/base, clusters/dev clean. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
eb31d2ef58 |
fix(monitoring): cap Prometheus TSDB by size to stop false absent()/==0 alerts
The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).
Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
failed ... write to WAL: no space left on device". Scrapes still return 200
(targets show healthy, lastScrape=now) but samples are never stored, so the
TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
present) — the rules are correct; the monitor was simply blind.
`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.
- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale
Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
1749603f87 |
feat(homepage): apex landing dashboard at home.keithjackson.dev
Deploy Homepage (gethomepage) as the apex landing dashboard aggregating links to cluster services, in its OWN non-gating post-platform Flux Kustomization (clusters/base/homepage.yaml, dependsOn platform-config, wait:false) — mirrors headlamp/oauth2-proxy so a leaf UI can never gate the platform chain. - platform/homepage/: plain manifests (ns, config ConfigMap, Deployment, Service, default-deny+Cilium ingress NP, apex HTTPRoute). Non-root pod, HOMEPAGE_ALLOWED_HOSTS, bundled mdi-* icons (DNS-only egress). - gateway.yaml: add https-apex + http-apex-redirect listeners for the bare apex (the *.home wildcard listener does NOT match it). Apex is already a SAN on the wildcard cert — no new Certificate. - Forward-auth annotated (auth-mode=forward-auth) but NOT enforced: the Gateway ext-authz mechanism is an unbuilt Phase-6 WS-NET TODO (same status as all other forward-auth apps). Flagged as PM follow-up. Gate: lint/l0/render-golden(10/10)/l1 green; conformance skipped (no cluster). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
ebe0371b1b |
platform: move Headlamp out of the gating platform tier + fix -session-ttl crashloop
Headlamp's HelmRelease sat in the wait:true `platform` Flux tier. Its pod crashlooped (`flag provided but not defined: -session-ttl` — chart 0.40.1 renders a container arg the appVersion v0.40.1 backend lacks; upstream kubernetes-sigs/headlamp#4883), and because it gated `platform`, the whole chain stalled: platform Ready=Unknown -> platform-config, oauth2-proxy, apps-wave-0/1/2 and apps-vms all Ready=False "dependency not ready". Part A (unblock + durable lesson): a leaf UI must never gate the chain. Move all Headlamp resources into a new non-gating Flux Kustomization clusters/base/headlamp.yaml (dependsOn platform-config, wait:false, SOPS decryption patched per-env in clusters/dev) — mirroring oauth2-proxy exactly. The new Kustomization owns the whole platform/headlamp component; its HTTPRoute moves from platform-config into platform/headlamp/kustomization.yaml. Removed headlamp from platform/kustomization.yaml and its httproute from platform-config/kustomization.yaml. Now a crashlooping Headlamp can never stall platform/platform-config/apps. Part B (make it start): pin the chart to 0.41.0 (`>=0.41.0 <0.42.0`). The backend added `f.Int("session-ttl", …)` in v0.41.0 and chart 0.41.0 pins that same image (appVersion == chart version), so chart+image realign; every arg the chart renders for our values is supported. OIDC config + per-user RBAC (clusterRoleBinding.create:false) unchanged. Gate: `make -C ci gate` PASS. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
960cab937b |
fix(flux): move oauth2-proxy to a post-platform tier (dependsOn platform-config)
oauth2-proxy is a leaf forward-auth service that depends on Keycloak + trusted wildcard TLS + split-horizon DNS. It lived in the gating `platform` Flux Kustomization (wait:true), so on a cold start it Failed until those dependencies existed, keeping the whole `platform` tier not-Ready. That in turn blocked `platform-config` — which brings up the very Keycloak instance, wildcard Certificate and L2 VIP oauth2-proxy needs (and previously carried oauth2-proxy's own httproute/servicemonitor) — from ever reconciling. A self-gating cold-start deadlock, twice-proven (DECISIONS-LOG sessions 7 & 9). Break the cycle by moving the whole oauth2-proxy component out of the `platform` tier into its own non-gating post-platform Flux Kustomization: - clusters/base/oauth2-proxy.yaml: new Kustomization -> ./platform/oauth2-proxy, dependsOn platform-config, wait:false, substituteFrom cluster-vars. - clusters/base/kustomization.yaml: reference the new Kustomization. - platform/kustomization.yaml: drop the oauth2-proxy component (no longer in the gating tier). - platform/oauth2-proxy/kustomization.yaml: pull in httproute + servicemonitor so the full component travels together. - platform-config/kustomization.yaml: drop the in-place oauth2-proxy httproute/servicemonitor refs (moved with the controller). - clusters/dev/kustomization.yaml: add the SOPS decryption patch for the new oauth2-proxy Kustomization (it now owns the encrypted credentials Secret that the `platform` Kustomization used to decrypt). Now `platform` reaches Ready without oauth2-proxy; platform-config brings up Keycloak/TLS/DNS; then oauth2-proxy reconciles against dependencies that already exist. wait:false guarantees its health can never gate anything above it. No config of oauth2-proxy itself changed — only its reconciliation tier. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
da1f833de7 |
fix(vms): drop postBuild substituteFrom on apps-vms — it corrupted VM cloud-init
Flux's postBuild.substituteFrom runs envsubst over the WHOLE manifest, including
the VM cloud-init userData. It was substituting every shell `${...}` in the UOS
install script (${UOS_URL}, the API-resolver ${...}) to empty — which is why the
first UOS install silently did nothing — and ${PIPESTATUS[0]} then broke the
build entirely ("missing closing brace"), blocking apps-vms reconcile.
VM manifests use no cluster-vars, so substitution is unneeded. Removed it; the
cloud-init shell now reaches the guest intact. (A future VM needing a cluster-var
must escape its shell `${...}` as `$${...}`.)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
317b3f02f1 |
feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
First KubeVirt VM app + first real CDI http import. UniFi OS Server (UOS) is Ubiquiti's current self-host product (the .deb is deprecated); it bundles UniFi OS + Network app + its own datastore, so no MongoDB companion. - apps/vms/unifi: CDI DataVolume (Ubuntu 24.04 cloud image -> longhorn-vm RWX/Block, live-migratable) + VirtualMachine (evictionStrategy: LiveMigrate) with cloud-init that installs podman/slirp4netns/jq and runs the UOS installer NON-INTERACTIVELY. The installer URL is resolved at boot from Ubiquiti's public firmware API (fw-update.ui.com/api/firmware-latest) — always-latest, no pinned URL to maintain (currently UOS v5.1.37). - L4 exposure: two LoadBalancer Services (TCP+UDP) sharing a pinned, L2-announced Cilium VIP 192.168.1.54 (device inform/adoption + STUN can't ride the L7 Gateway); external-dns publishes unifi.home.keithjackson.dev. UI on :11443. - default-deny + additive CNPs (controller ingress, egress, and CDI-importer egress for the image pull). - clusters/base/apps-vms.yaml: NON-GATING Flux Kustomization (wait:false, dependsOn platform-config) — VMs are a separate lifecycle from the app waves, so a slow VM import/install can't stall the app tier. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
36a1a4e1f9 |
refactor(apps): split monolithic apps Kustomization into dependency waves
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.
- catalog/render.py: compute waves by longest dependency-path depth over the
ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
consumer lands strictly higher — the "DB below consumer" rule falls out of
longest-path, no special case). `--emit-waves` generates the per-wave
kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
wave 1: sonarr radarr nextcloud semaphore gitea-actions
wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.
Cutover: preceded by disabling prune on `apps` (
|
||
|
jakman85andClaude Opus 4.8
|
1bfb0db6f1 |
chore(flux): cutover step 1 — disable prune on apps before wave split
Temporary: with prune:false, removing the `apps` Kustomization object in the follow-up wave-split commit will NOT garbage-collect its owned resources (CNPG Clusters -> PVCs). The apps-wave-* Kustomizations adopt them in place. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d99d171468 |
Phase-6: repoint Flux source to in-cluster Gitea
Cut the GitOps source over from the bootstrap host-Gitea (http://192.168.2.103:3000) to the now-healthy in-cluster Gitea at https://gitea.home.keithjackson.dev. Public repo, LE-prod wildcard cert publicly trusted, resolved in-cluster via coredns->Gateway, so no secretRef/caCert needed. Bootstrap host-Gitea retained (stopped) as a recovery path. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
28a34c4477 |
fix(kyverno): raise admission-controller resources + dev replicas 2 to stop fail-closed crashloop/502s
Admission controller was crashlooping (9 restarts/4h, exit 255, NOT OOM: live usage ~65Mi/384Mi limit, 7m cpu). Root cause: chart-default 100m CPU request starved the process under node contention, so client-go calls hit context-deadline-exceeded, leader-election lease renewal timed out, leadership was lost and the webhook server closed. Raise the CPU request to 250m (no CPU limit, to avoid CFS throttling) with modest memory headroom (256Mi/512Mi), and raise dev admission replicas 1->2 so a single fail-closed pod restart no longer 502s cluster-wide admission-gated writes. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
604552e51c |
flux(dev): set GATEWAY_VIP to Proxmox Gateway VIP 192.168.1.53 (split-horizon DNS)
Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
fff44992e7 |
migrate(proxmox): bootstrap source URL -> 192.168.2.103; Cilium k8sServiceHost -> 192.168.1.50
Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
8a7700e49a |
fix(tls): switch dev to LE production issuer so in-cluster clients trust the wildcard cert
oauth2-proxy + Grafana OIDC discovery to keycloak.home.keithjackson.dev failed x509 'unknown authority' — the wildcard was LE-staging (untrusted). Prod issuer is registered + Ready; switching ACME_CLUSTER_ISSUER dev override staging->prod re-issues a publicly trusted cert. Needed for functional SSO (R-IDP) + R-TLS-01/03 'trusted'. NOTE: the trusted-tls conformance test embeds the LE-staging root and must be updated to the LE production root (ISRG Root X1) / system trust — follow-up. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
bba67d03ce |
feat(dns): add GATEWAY_VIP cluster-var for split-horizon target (R-DNS-03)
Parameterize the in-cluster split-horizon target as GATEWAY_VIP (R-ENV-02): base
default = dev L2-announced Gateway VIP (10.73.0.240, first IP of the Cilium LB pool).
coredns-custom answers *.home.keithjackson.dev with ${GATEWAY_VIP} so the unified graph
holds across dev/prod via cluster-vars only. Prod overlay carries a TODO to set the prod
LAN VIP alongside the prod L2 pool range when hardware is built.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
aafb4ca378 |
docs(vpn-egress): reflect ADR-0015 node-tunnel egress, retire gluetun narrative
The in-cluster Gluetun gateway is retired; VPN egress is now a node-level host OpenVPN split-tunnel (Ansible role vpn_egress_gateway) steered by a Cilium Egress Gateway policy with a host nftables kill-switch (ADR-0015). Credentials moved host-side to the Ansible vault. Update docs/comments only: - platform/vpn-egress/README.md: rewrite for ADR-0015 (live + conformance-green); drop the resolved "option 1 not viable with Gluetun" finding to a brief history note. - platform/kustomization.yaml: Phase 3 comment -> node tunnel + Cilium steering. - platform/kyverno/.../require-egress-class-label.yaml: ADR-0013 -> ADR-0015 steering. - schemas/servicedefinition.schema.json: egress=vpn description -> node tunnel + Cilium. - clusters/base/cluster-vars.yaml: reframe PIA_REGION / CLUSTER_POD_CIDR WHY comments to ADR-0015 (Cilium excludedCIDRs); both keys retained (still referenced/documented). - platform/vpn-egress/namespace.yaml: no privileged gateway pod under ADR-0015. - pia-credentials.sops.yaml / pia-custom-ovpn.sops.yaml: comment headers only note they are now UNUSED (creds host-side), retained pending cleanup; payloads untouched. No runtime manifest behavior, alert exprs, kustomization resources, or SOPS payloads changed. No spec-repo edits. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
7d2cea8402 |
fix(vpn): narrow FIREWALL_OUTBOUND_SUBNETS to 10.0.0.0/16 (PF gateway overlap)
PIA's PF gateway 10.31.128.1 sits inside the /8 cluster-pool, so FIREWALL_OUTBOUND_SUBNETS=10.0.0.0/8 made gluetun route it around the tunnel and port forwarding timed out (getSignature to 10.31.128.1:19999). /16 covers actual pod allocations and excludes PIA's 10.31.x. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
457edb7a38 |
fix(vpn): enable gluetun server-list updater (bundled list was stale) + re-enable
CORRECTION: the connection failures were gluetun dialing RETIRED PIA IPs (46.246.x = dead → timeouts, which I mis-read as a network block). A fresh PIA .ovpn connects from the same host, and PIA's current SE Stockholm servers (158.173.x) are reachable. Cilium was independently cleared (default-allow, host-fw off, Hubble FORWARDED). Fix: UPDATER_PERIOD=24h so gluetun fetches PIA's current servers; keep OPENVPN_MTU/MSSFIX=1350 for the real Cilium-VXLAN 1450 pod MTU (data channel); PIA default UDP port. Re-enable vpn-egress. Full correction in DECISIONS-LOG. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
95a55952cc |
fix(vpn): disable vpn-egress — network blackholes PIA IPs (not ports/MTU/cluster)
Decisive diagnosis: OpenVPN's outbound init packets are 14 bytes with zero replies (so MTU was never the blocker), pod UDP egress is healthy (NTP replies), but TCP 501/443 to PIA IPs time out while Google/Cloudflare are OPEN — across many PIA servers. This 192.168.2.x WiFi segment blackholes PIA's IP ranges (and intercepts DNS). Not code-fixable. Reverted the port/MTU red-herring env to PIA defaults; disabled vpn-egress pending an operator network fix (move to the unfiltered 192.168.1.x LAN, or unblock PIA on the router). Full trace in DECISIONS-LOG. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
4cde690008 |
fix(vpn): PIA OpenVPN port 53 -> 8080 (53 is DNS-intercepted on this network)
Port 53 also got no PIA replies: this network transparently intercepts outbound :53, so dig works but OpenVPN-over-53 is swallowed. An NTP/123 probe from the host returned fine, proving arbitrary non-53 UDP works bidirectionally (so it's port-specific: 1197 conflict + 53 DNS-intercept, not a UDP block). Move to 8080 (community-verified for PIA, non-DNS, non-1197). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
734093cf92 |
fix(vpn): hardcode OPENVPN_ENDPOINT_PORT "53" as string (Flux var int-coercion)
Wiring the port via "${PIA_OPENVPN_PORT}" made Flux/kustomize emit an unquoted
53 (int) after substitution, which k8s rejects for an env value ("expected
string, got 53"). Same gotcha as the Longhorn numberOfReplicas fix. Hardcode the
quoted string; cluster-var removed with a NOTE explaining why.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
ec5ceb328f |
fix(vpn): PIA OpenVPN on UDP/53 (avoid home WAN:1197 conflict); re-enable vpn-egress
Operator identified the block: the home network NATs its own OpenVPN server on
WAN:1197, colliding with the cluster's outbound PIA-on-1197 flow (port-specific,
not a general UDP block — UDP/53 worked in tcpdump). PIA accepts OpenVPN on
53/1194/1197/1198/8080/9201; chose 53 (empirically proven to traverse this net).
- cluster-vars: add PIA_OPENVPN_PORT="53"
- gluetun: OPENVPN_ENDPOINT_PORT=${PIA_OPENVPN_PORT}
- re-enable vpn-egress; DECISIONS-LOG resolution noted
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
9a108443e8 |
fix(vpn): PIA_REGION Sweden -> "SE Stockholm" (Gluetun OpenVPN region name)
Gluetun's PIA OpenVPN provider rejects "Sweden"; the valid PF-capable Sweden
region identifier is "SE Stockholm" ("SE Streaming Optimized" lacks port
forwarding). Preserves the operator's Sweden choice.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
0d55c793f3 |
fix(vpn): default PIA_REGION to Sweden (operator choice, PF-capable)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
6591e11ad1 |
fix(vpn): define PIA_REGION + CLUSTER_POD_CIDR cluster-vars; correct pod CIDR + doc where
gluetun-gateway.yaml referenced ${PIA_REGION}/${CLUSTER_POD_CIDR} but they were never defined
in cluster-vars, and the README didn't say where to set them. Add both to
clusters/base/cluster-vars.yaml (ConfigMap data:, string values, Flux postBuild substitution).
CLUSTER_POD_CIDR is 10.0.0.0/8 (Cilium cluster-pool — pods get 10.x, confirmed live), NOT the
k3s 10.42.0.0/16 I'd assumed. README now names the file + format + per-env override + PF-region note.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
d06947cbfd |
feat(gitops): self-managing root Kustomization (fixes F-OPS1)
The on-cluster cluster-vars ConfigMap wasn't reconciled by Flux — this bootstrap
model had no Kustomization watching clusters/dev, so cluster-vars/overlay changes
needed a manual 'kubectl apply -k clusters/dev' (e.g. DNS_ZONE substituted empty
until re-applied). Add clusters/dev/sync.yaml: a self-managing root Kustomization
(flux2 gotk-sync pattern, name flux-system) that reconciles ./clusters/dev from git
— cluster-vars, the GitRepository, platform/platform-config/apps, and itself. A git
push now suffices. Bootstrap unchanged ('kubectl apply -k clusters/dev' creates it
once). Verified: dev build contains all 6 objects (root + existing), prod unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
20de78f5eb |
feat(conformance): author R-TLS + R-DNS L2 tests; fix ExternalDNS zone filter
Phase-2 exit tests (were Phase-0 scaffolds), authored against the live platform:
- trusted-tls (R-TLS-01/03): in-cluster probe dials the gateway Service (env-agnostic
via --connect-to) with wildcard SNI and verifies the served cert CHAINS TO THE LE
STAGING ROOT (operator decision A; dev uses LE-staging per ADR-0006). Fixture embeds
the '(STAGING) Pretend Pear X1' root. Probes public + internal-only hostnames (R-TLS-03).
PASSES.
- dns-exposure (R-DNS-02): a LoadBalancer Service with an external-dns hostname; polls
public DNS until ExternalDNS auto-publishes the record; teardown + policy=sync removes it.
ExternalDNS fix (root cause of the DNS test): domain-filter was the platform SUBDOMAIN
home.keithjackson.dev, which filters OUT the actual Cloudflare zone keithjackson.dev
('no hosted zone matching record DNS Name' — record generated but skipped). Add a
DNS_ZONE cluster-var (keithjackson.dev) and set domainFilters to it. policy=sync +
txt-registry still only manage ExternalDNS-owned records (the pre-existing wildcard is
untouched). Resolves the file's WS-NET R-DNS-02 TODO.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
c6331c84bf |
fix(kyverno): correct chart webhook schema (map + forceFailurePolicyIgnore)
Kyverno HelmRelease install failed at Helm templating: config.webhooks was a
LIST [{failurePolicy, namespaceSelector}], but the kyverno 3.3.9 chart expects
config.webhooks to be a MAP (webhooks.namespaceSelector) — the chart helper reads
.namespaceSelector off the value directly ('can't evaluate field namespaceSelector
in type []interface {}'). Also, failurePolicy is not a per-webhook field here; the
chart's global switch is config.forceFailurePolicyIgnore.enabled (bool).
Fix: config.webhooks -> map with namespaceSelector (system+platform excludes);
add config.forceFailurePolicyIgnore.enabled = ${KYVERNO_FORCE_FAILURE_POLICY_IGNORE}.
Renamed the cluster-var KYVERNO_WEBHOOK_FAILURE_POLICY (Fail/Ignore) ->
KYVERNO_FORCE_FAILURE_POLICY_IGNORE (false base / true dev). Verified with
helm template (renders 73 objects, no error); gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
ca982864fc |
fix(platform): controllers/configs split so CRDs precede their CRs
Un-suspend deadlocked: the single platform Kustomization applies HelmReleases
(which register CRDs) AND custom resources of those CRDs (ClusterIssuer,
Certificate, Kyverno ClusterPolicies) in ONE pass. Flux's server-side apply
fails atomically ('no matches for kind ClusterIssuer') → nothing installs, so
the CRD-registering HelmReleases never run either. Classic ordering deadlock.
Fix (flux2 controllers/configs pattern, applies to dev+prod equally — no
R-ENV-02 divergence): add a platform-config Flux Kustomization (dependsOn
platform, wait) carrying the CRs; platform keeps only the controllers.
- New clusters/base/platform-config.yaml (path ./platform-config, substituteFrom
cluster-vars, no decryption — no SOPS content); wired into base + apps now
dependsOn platform-config (transitively waits for platform).
- platform-config/ references the CR files IN PLACE via ../platform/... (Flux
builds LoadRestrictionsNone; Kyverno policies must stay under platform/kyverno/
for the CI L0 gate). CR resources dropped from the platform
component kustomizations (cert-manager, gateway, kyverno).
- Authored the missing Cilium L2 resources (were only a TODO): CiliumLoadBalancerIPPool
(dev NAT 10.73.0.240-.250) + CiliumL2AnnouncementPolicy (ens*), so Gateway
LoadBalancer listeners get a VIP. Prod range must be parameterized later.
Verified: platform build has zero CRs; platform-config build has 2 ClusterIssuers,
6 ClusterPolicies, Certificate, Gateway, both Cilium L2 CRs; dev/prod build; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
a108d5ab98 |
correction: credential-forward + unified graph; activate SOPS (author-first)
Operator directive: don't diverge architecture to avoid asking for credentials; provision them upfront via SOPS and keep the existing (unified) graph. Roadmap reconciliation: Phase 2 = Network/Storage/TLS/DNS/Ingress, so cert-manager/externaldns/gateway (the credential-gated pieces) ARE Phase 2; monitoring is Phase 4. Earlier A2/rollout scope was inverted by credential-avoidance. Reverts (credential-driven): remove clusters/dev/rollout/ + the $patch:delete's; restore unified base graph (dev = ../base + cluster-vars + SOPS decryption, R-ENV-02 intact); restore monitoring files (Phase 4). Kept non-credential improvements: tofu for_each/rejoin, Kyverno parametrization + webhook excludes. platform/kustomization.yaml now phased (single shared list): Phase-2 set = kyverno, cilium, longhorn, cert-manager, externaldns, gateway; later components commented with phase markers. SOPS activated: .sops.yaml at repo root; bootstrap/sops/secret-templates/*.yaml.example for all secrets (cloudflare x2, grafana, alertmanager-smtp, keycloak, oauth2-proxy, pia); bootstrap/sops/CREDENTIALS.md checklist; pre-wired commented cloudflare-api-token lines in cert-manager + externaldns; filled non-secret ACME email. Agents never touch real secret values (operator encrypts). Validated: kubectl kustomize dev(unified)/prod/platform(Phase-2 only), templates inert, make -C ci gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d9a80e5350 |
phase2-prep: per-component no-secret rollout + component hardening (author-first)
Prep only — nothing deployed; running cluster untouched. Reconciliation graph (operator-approved Option A): author clusters/dev/rollout/ = 4 per-component Flux Kustomizations (kyverno->cilium->longhorn->monitoring, monitoring dependsOn longhorn). Dev overlay $patch:delete's the monolithic base platform/apps and adds ./rollout, so only the no-secret tier reconciles; the secret-blocked components are simply not created until hand-offs arrive. The no-secret tier carries no SOPS content, so it needs no sops-age key. prod is untouched (still builds the monolith). Convergence path coded in (delete rollout/ + the two deletes) — R-ENV-02 divergence flagged for architect review. Component hardening via cluster-vars (prod keeps strict posture): - KYVERNO_VALIDATION_ACTION Enforce->Audit(dev), _WEBHOOK_FAILURE_POLICY Fail->Ignore(dev), _ADMISSION_REPLICAS 3->1(dev); webhook namespaceSelector excludes system+platform namespaces. - MONITORING_STORAGE_CLASS var; GRAFANA_ADMIN_SECRET var (dev ''=chart default); Alertmanager SMTP/secret deferred to Phase-4 with verbatim re-enable markers. Cilium parity verified against the live CNI (no-op adopt). Validated: kubectl kustomize dev/prod/all-components, envsubst->YAML parse, make -C ci gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
9743b50c3b |
fix(K1): uppercase cluster-vars keys to match component substitution refs
Flux postBuild.substituteFrom is case-sensitive; platform/ components reference
${UPPERCASE} tokens but cluster-vars keys were lowercase -> none would resolve
(literal ${...} would reach the HelmReleases). Uppercased the ConfigMap data
keys + the dev/prod overlay patch paths. Every component ref now resolves;
clusters/dev & clusters/prod build; gate green. Clears the K1 landmine before
any platform un-suspend.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|