Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).
Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.
- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
2026-08-26 LIVE STATE line
Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add Crafty Controller 4 (registry.gitlab.com/crafty-controller/crafty-4:4.10.8)
as a platform component: a web panel that runs Java + Bedrock Minecraft servers as
subprocesses in one pod + a 20Gi Longhorn PVC (mounted 5x by subPath).
Auth = Option B (per-app oauth2-proxy reverse-proxy sidecar), since Cilium 1.16.5
cannot do ext_authz on the shared Gateway:
- oauth2-proxy v7.8.1 sidecar (:4180) reverse-proxies to Crafty's self-signed HTTPS
UI (127.0.0.1:8443), gated to Keycloak homelab realm group platform-admins.
- Confidential `crafty` client provisioned live via kcadm + captured in the realm
import; client secret + cookie secret via SOPS/age (mirrored in realm-clients).
- Web UI: HTTPRoute crafty.home.keithjackson.dev -> Service :80 -> sidecar :4180.
- Game ports: Cilium L2 LoadBalancer on VIP 192.168.1.55 (next free after .53/.54),
starter set TCP 25565-25567 (Java) + UDP 19132-19134 (Bedrock), NOT auth-gated.
- fsGroup:0 / runAsGroup:0 / runAsUser:1000 for the image's crafty:root identity on
the root:root Longhorn PVC (the seerr fsGroup lesson).
- Own non-gating Flux tier clusters/base/crafty.yaml (dependsOn platform-config,
wait:false) + sops-decryption patch in clusters/dev — a leaf UI must never gate
the platform.
- Explicit default-deny NetworkPolicy + CiliumNetworkPolicies (platform namespace,
not app-namespace, so Kyverno does not auto-generate the baseline).
make -C ci gate green; kustomize build of platform/crafty, clusters/base,
clusters/dev clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).
Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
failed ... write to WAL: no space left on device". Scrapes still return 200
(targets show healthy, lastScrape=now) but samples are never stored, so the
TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
present) — the rules are correct; the monitor was simply blind.
`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.
- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale
Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Deploy Homepage (gethomepage) as the apex landing dashboard aggregating
links to cluster services, in its OWN non-gating post-platform Flux
Kustomization (clusters/base/homepage.yaml, dependsOn platform-config,
wait:false) — mirrors headlamp/oauth2-proxy so a leaf UI can never gate
the platform chain.
- platform/homepage/: plain manifests (ns, config ConfigMap, Deployment,
Service, default-deny+Cilium ingress NP, apex HTTPRoute). Non-root pod,
HOMEPAGE_ALLOWED_HOSTS, bundled mdi-* icons (DNS-only egress).
- gateway.yaml: add https-apex + http-apex-redirect listeners for the bare
apex (the *.home wildcard listener does NOT match it). Apex is already a
SAN on the wildcard cert — no new Certificate.
- Forward-auth annotated (auth-mode=forward-auth) but NOT enforced: the
Gateway ext-authz mechanism is an unbuilt Phase-6 WS-NET TODO (same status
as all other forward-auth apps). Flagged as PM follow-up.
Gate: lint/l0/render-golden(10/10)/l1 green; conformance skipped (no cluster).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Headlamp's HelmRelease sat in the wait:true `platform` Flux tier. Its pod
crashlooped (`flag provided but not defined: -session-ttl` — chart 0.40.1
renders a container arg the appVersion v0.40.1 backend lacks; upstream
kubernetes-sigs/headlamp#4883), and because it gated `platform`, the whole
chain stalled: platform Ready=Unknown -> platform-config, oauth2-proxy,
apps-wave-0/1/2 and apps-vms all Ready=False "dependency not ready".
Part A (unblock + durable lesson): a leaf UI must never gate the chain. Move
all Headlamp resources into a new non-gating Flux Kustomization
clusters/base/headlamp.yaml (dependsOn platform-config, wait:false, SOPS
decryption patched per-env in clusters/dev) — mirroring oauth2-proxy exactly.
The new Kustomization owns the whole platform/headlamp component; its HTTPRoute
moves from platform-config into platform/headlamp/kustomization.yaml. Removed
headlamp from platform/kustomization.yaml and its httproute from
platform-config/kustomization.yaml. Now a crashlooping Headlamp can never stall
platform/platform-config/apps.
Part B (make it start): pin the chart to 0.41.0 (`>=0.41.0 <0.42.0`). The
backend added `f.Int("session-ttl", …)` in v0.41.0 and chart 0.41.0 pins that
same image (appVersion == chart version), so chart+image realign; every arg the
chart renders for our values is supported. OIDC config + per-user RBAC
(clusterRoleBinding.create:false) unchanged.
Gate: `make -C ci gate` PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy is a leaf forward-auth service that depends on Keycloak +
trusted wildcard TLS + split-horizon DNS. It lived in the gating `platform`
Flux Kustomization (wait:true), so on a cold start it Failed until those
dependencies existed, keeping the whole `platform` tier not-Ready. That in
turn blocked `platform-config` — which brings up the very Keycloak instance,
wildcard Certificate and L2 VIP oauth2-proxy needs (and previously carried
oauth2-proxy's own httproute/servicemonitor) — from ever reconciling. A
self-gating cold-start deadlock, twice-proven (DECISIONS-LOG sessions 7 & 9).
Break the cycle by moving the whole oauth2-proxy component out of the
`platform` tier into its own non-gating post-platform Flux Kustomization:
- clusters/base/oauth2-proxy.yaml: new Kustomization -> ./platform/oauth2-proxy,
dependsOn platform-config, wait:false, substituteFrom cluster-vars.
- clusters/base/kustomization.yaml: reference the new Kustomization.
- platform/kustomization.yaml: drop the oauth2-proxy component (no longer in
the gating tier).
- platform/oauth2-proxy/kustomization.yaml: pull in httproute + servicemonitor
so the full component travels together.
- platform-config/kustomization.yaml: drop the in-place oauth2-proxy
httproute/servicemonitor refs (moved with the controller).
- clusters/dev/kustomization.yaml: add the SOPS decryption patch for the new
oauth2-proxy Kustomization (it now owns the encrypted credentials Secret that
the `platform` Kustomization used to decrypt).
Now `platform` reaches Ready without oauth2-proxy; platform-config brings up
Keycloak/TLS/DNS; then oauth2-proxy reconciles against dependencies that
already exist. wait:false guarantees its health can never gate anything above
it. No config of oauth2-proxy itself changed — only its reconciliation tier.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Flux's postBuild.substituteFrom runs envsubst over the WHOLE manifest, including
the VM cloud-init userData. It was substituting every shell `${...}` in the UOS
install script (${UOS_URL}, the API-resolver ${...}) to empty — which is why the
first UOS install silently did nothing — and ${PIPESTATUS[0]} then broke the
build entirely ("missing closing brace"), blocking apps-vms reconcile.
VM manifests use no cluster-vars, so substitution is unneeded. Removed it; the
cloud-init shell now reaches the guest intact. (A future VM needing a cluster-var
must escape its shell `${...}` as `$${...}`.)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
First KubeVirt VM app + first real CDI http import. UniFi OS Server (UOS) is
Ubiquiti's current self-host product (the .deb is deprecated); it bundles UniFi
OS + Network app + its own datastore, so no MongoDB companion.
- apps/vms/unifi: CDI DataVolume (Ubuntu 24.04 cloud image -> longhorn-vm
RWX/Block, live-migratable) + VirtualMachine (evictionStrategy: LiveMigrate)
with cloud-init that installs podman/slirp4netns/jq and runs the UOS installer
NON-INTERACTIVELY. The installer URL is resolved at boot from Ubiquiti's public
firmware API (fw-update.ui.com/api/firmware-latest) — always-latest, no pinned
URL to maintain (currently UOS v5.1.37).
- L4 exposure: two LoadBalancer Services (TCP+UDP) sharing a pinned, L2-announced
Cilium VIP 192.168.1.54 (device inform/adoption + STUN can't ride the L7
Gateway); external-dns publishes unifi.home.keithjackson.dev. UI on :11443.
- default-deny + additive CNPs (controller ingress, egress, and CDI-importer
egress for the image pull).
- clusters/base/apps-vms.yaml: NON-GATING Flux Kustomization (wait:false,
dependsOn platform-config) — VMs are a separate lifecycle from the app waves,
so a slow VM import/install can't stall the app tier.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.
- catalog/render.py: compute waves by longest dependency-path depth over the
ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
consumer lands strictly higher — the "DB below consumer" rule falls out of
longest-path, no special case). `--emit-waves` generates the per-wave
kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
wave 1: sonarr radarr nextcloud semaphore gitea-actions
wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.
Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that
object here does not GC its resources; the wave Kustomizations SSA-adopt the
byte-identical manifests in place (no restart, no PVC churn).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Temporary: with prune:false, removing the `apps` Kustomization object in the
follow-up wave-split commit will NOT garbage-collect its owned resources
(CNPG Clusters -> PVCs). The apps-wave-* Kustomizations adopt them in place.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Cut the GitOps source over from the bootstrap host-Gitea
(http://192.168.2.103:3000) to the now-healthy in-cluster Gitea at
https://gitea.home.keithjackson.dev. Public repo, LE-prod wildcard
cert publicly trusted, resolved in-cluster via coredns->Gateway, so
no secretRef/caCert needed. Bootstrap host-Gitea retained (stopped)
as a recovery path.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Admission controller was crashlooping (9 restarts/4h, exit 255, NOT OOM:
live usage ~65Mi/384Mi limit, 7m cpu). Root cause: chart-default 100m CPU
request starved the process under node contention, so client-go calls hit
context-deadline-exceeded, leader-election lease renewal timed out, leadership
was lost and the webhook server closed. Raise the CPU request to 250m (no CPU
limit, to avoid CFS throttling) with modest memory headroom (256Mi/512Mi), and
raise dev admission replicas 1->2 so a single fail-closed pod restart no longer
502s cluster-wide admission-gated writes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy + Grafana OIDC discovery to keycloak.home.keithjackson.dev failed x509
'unknown authority' — the wildcard was LE-staging (untrusted). Prod issuer is registered
+ Ready; switching ACME_CLUSTER_ISSUER dev override staging->prod re-issues a publicly
trusted cert. Needed for functional SSO (R-IDP) + R-TLS-01/03 'trusted'.
NOTE: the trusted-tls conformance test embeds the LE-staging root and must be updated to
the LE production root (ISRG Root X1) / system trust — follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Parameterize the in-cluster split-horizon target as GATEWAY_VIP (R-ENV-02): base
default = dev L2-announced Gateway VIP (10.73.0.240, first IP of the Cilium LB pool).
coredns-custom answers *.home.keithjackson.dev with ${GATEWAY_VIP} so the unified graph
holds across dev/prod via cluster-vars only. Prod overlay carries a TODO to set the prod
LAN VIP alongside the prod L2 pool range when hardware is built.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The in-cluster Gluetun gateway is retired; VPN egress is now a node-level host
OpenVPN split-tunnel (Ansible role vpn_egress_gateway) steered by a Cilium
Egress Gateway policy with a host nftables kill-switch (ADR-0015). Credentials
moved host-side to the Ansible vault. Update docs/comments only:
- platform/vpn-egress/README.md: rewrite for ADR-0015 (live + conformance-green);
drop the resolved "option 1 not viable with Gluetun" finding to a brief history note.
- platform/kustomization.yaml: Phase 3 comment -> node tunnel + Cilium steering.
- platform/kyverno/.../require-egress-class-label.yaml: ADR-0013 -> ADR-0015 steering.
- schemas/servicedefinition.schema.json: egress=vpn description -> node tunnel + Cilium.
- clusters/base/cluster-vars.yaml: reframe PIA_REGION / CLUSTER_POD_CIDR WHY comments
to ADR-0015 (Cilium excludedCIDRs); both keys retained (still referenced/documented).
- platform/vpn-egress/namespace.yaml: no privileged gateway pod under ADR-0015.
- pia-credentials.sops.yaml / pia-custom-ovpn.sops.yaml: comment headers only note
they are now UNUSED (creds host-side), retained pending cleanup; payloads untouched.
No runtime manifest behavior, alert exprs, kustomization resources, or SOPS payloads
changed. No spec-repo edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
PIA's PF gateway 10.31.128.1 sits inside the /8 cluster-pool, so
FIREWALL_OUTBOUND_SUBNETS=10.0.0.0/8 made gluetun route it around the tunnel and
port forwarding timed out (getSignature to 10.31.128.1:19999). /16 covers actual
pod allocations and excludes PIA's 10.31.x.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CORRECTION: the connection failures were gluetun dialing RETIRED PIA IPs (46.246.x
= dead → timeouts, which I mis-read as a network block). A fresh PIA .ovpn connects
from the same host, and PIA's current SE Stockholm servers (158.173.x) are reachable.
Cilium was independently cleared (default-allow, host-fw off, Hubble FORWARDED).
Fix: UPDATER_PERIOD=24h so gluetun fetches PIA's current servers; keep
OPENVPN_MTU/MSSFIX=1350 for the real Cilium-VXLAN 1450 pod MTU (data channel);
PIA default UDP port. Re-enable vpn-egress. Full correction in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Decisive diagnosis: OpenVPN's outbound init packets are 14 bytes with zero
replies (so MTU was never the blocker), pod UDP egress is healthy (NTP replies),
but TCP 501/443 to PIA IPs time out while Google/Cloudflare are OPEN — across many
PIA servers. This 192.168.2.x WiFi segment blackholes PIA's IP ranges (and
intercepts DNS). Not code-fixable. Reverted the port/MTU red-herring env to PIA
defaults; disabled vpn-egress pending an operator network fix (move to the
unfiltered 192.168.1.x LAN, or unblock PIA on the router). Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Port 53 also got no PIA replies: this network transparently intercepts outbound
:53, so dig works but OpenVPN-over-53 is swallowed. An NTP/123 probe from the host
returned fine, proving arbitrary non-53 UDP works bidirectionally (so it's
port-specific: 1197 conflict + 53 DNS-intercept, not a UDP block). Move to 8080
(community-verified for PIA, non-DNS, non-1197).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Wiring the port via "${PIA_OPENVPN_PORT}" made Flux/kustomize emit an unquoted
53 (int) after substitution, which k8s rejects for an env value ("expected
string, got 53"). Same gotcha as the Longhorn numberOfReplicas fix. Hardcode the
quoted string; cluster-var removed with a NOTE explaining why.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator identified the block: the home network NATs its own OpenVPN server on
WAN:1197, colliding with the cluster's outbound PIA-on-1197 flow (port-specific,
not a general UDP block — UDP/53 worked in tcpdump). PIA accepts OpenVPN on
53/1194/1197/1198/8080/9201; chose 53 (empirically proven to traverse this net).
- cluster-vars: add PIA_OPENVPN_PORT="53"
- gluetun: OPENVPN_ENDPOINT_PORT=${PIA_OPENVPN_PORT}
- re-enable vpn-egress; DECISIONS-LOG resolution noted
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gluetun's PIA OpenVPN provider rejects "Sweden"; the valid PF-capable Sweden
region identifier is "SE Stockholm" ("SE Streaming Optimized" lacks port
forwarding). Preserves the operator's Sweden choice.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
gluetun-gateway.yaml referenced ${PIA_REGION}/${CLUSTER_POD_CIDR} but they were never defined
in cluster-vars, and the README didn't say where to set them. Add both to
clusters/base/cluster-vars.yaml (ConfigMap data:, string values, Flux postBuild substitution).
CLUSTER_POD_CIDR is 10.0.0.0/8 (Cilium cluster-pool — pods get 10.x, confirmed live), NOT the
k3s 10.42.0.0/16 I'd assumed. README now names the file + format + per-env override + PF-region note.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The on-cluster cluster-vars ConfigMap wasn't reconciled by Flux — this bootstrap
model had no Kustomization watching clusters/dev, so cluster-vars/overlay changes
needed a manual 'kubectl apply -k clusters/dev' (e.g. DNS_ZONE substituted empty
until re-applied). Add clusters/dev/sync.yaml: a self-managing root Kustomization
(flux2 gotk-sync pattern, name flux-system) that reconciles ./clusters/dev from git
— cluster-vars, the GitRepository, platform/platform-config/apps, and itself. A git
push now suffices. Bootstrap unchanged ('kubectl apply -k clusters/dev' creates it
once). Verified: dev build contains all 6 objects (root + existing), prod unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Phase-2 exit tests (were Phase-0 scaffolds), authored against the live platform:
- trusted-tls (R-TLS-01/03): in-cluster probe dials the gateway Service (env-agnostic
via --connect-to) with wildcard SNI and verifies the served cert CHAINS TO THE LE
STAGING ROOT (operator decision A; dev uses LE-staging per ADR-0006). Fixture embeds
the '(STAGING) Pretend Pear X1' root. Probes public + internal-only hostnames (R-TLS-03).
PASSES.
- dns-exposure (R-DNS-02): a LoadBalancer Service with an external-dns hostname; polls
public DNS until ExternalDNS auto-publishes the record; teardown + policy=sync removes it.
ExternalDNS fix (root cause of the DNS test): domain-filter was the platform SUBDOMAIN
home.keithjackson.dev, which filters OUT the actual Cloudflare zone keithjackson.dev
('no hosted zone matching record DNS Name' — record generated but skipped). Add a
DNS_ZONE cluster-var (keithjackson.dev) and set domainFilters to it. policy=sync +
txt-registry still only manage ExternalDNS-owned records (the pre-existing wildcard is
untouched). Resolves the file's WS-NET R-DNS-02 TODO.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Kyverno HelmRelease install failed at Helm templating: config.webhooks was a
LIST [{failurePolicy, namespaceSelector}], but the kyverno 3.3.9 chart expects
config.webhooks to be a MAP (webhooks.namespaceSelector) — the chart helper reads
.namespaceSelector off the value directly ('can't evaluate field namespaceSelector
in type []interface {}'). Also, failurePolicy is not a per-webhook field here; the
chart's global switch is config.forceFailurePolicyIgnore.enabled (bool).
Fix: config.webhooks -> map with namespaceSelector (system+platform excludes);
add config.forceFailurePolicyIgnore.enabled = ${KYVERNO_FORCE_FAILURE_POLICY_IGNORE}.
Renamed the cluster-var KYVERNO_WEBHOOK_FAILURE_POLICY (Fail/Ignore) ->
KYVERNO_FORCE_FAILURE_POLICY_IGNORE (false base / true dev). Verified with
helm template (renders 73 objects, no error); gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Un-suspend deadlocked: the single platform Kustomization applies HelmReleases
(which register CRDs) AND custom resources of those CRDs (ClusterIssuer,
Certificate, Kyverno ClusterPolicies) in ONE pass. Flux's server-side apply
fails atomically ('no matches for kind ClusterIssuer') → nothing installs, so
the CRD-registering HelmReleases never run either. Classic ordering deadlock.
Fix (flux2 controllers/configs pattern, applies to dev+prod equally — no
R-ENV-02 divergence): add a platform-config Flux Kustomization (dependsOn
platform, wait) carrying the CRs; platform keeps only the controllers.
- New clusters/base/platform-config.yaml (path ./platform-config, substituteFrom
cluster-vars, no decryption — no SOPS content); wired into base + apps now
dependsOn platform-config (transitively waits for platform).
- platform-config/ references the CR files IN PLACE via ../platform/... (Flux
builds LoadRestrictionsNone; Kyverno policies must stay under platform/kyverno/
for the CI L0 gate). CR resources dropped from the platform
component kustomizations (cert-manager, gateway, kyverno).
- Authored the missing Cilium L2 resources (were only a TODO): CiliumLoadBalancerIPPool
(dev NAT 10.73.0.240-.250) + CiliumL2AnnouncementPolicy (ens*), so Gateway
LoadBalancer listeners get a VIP. Prod range must be parameterized later.
Verified: platform build has zero CRs; platform-config build has 2 ClusterIssuers,
6 ClusterPolicies, Certificate, Gateway, both Cilium L2 CRs; dev/prod build; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Prep only — nothing deployed; running cluster untouched.
Reconciliation graph (operator-approved Option A): author clusters/dev/rollout/
= 4 per-component Flux Kustomizations (kyverno->cilium->longhorn->monitoring,
monitoring dependsOn longhorn). Dev overlay $patch:delete's the monolithic base
platform/apps and adds ./rollout, so only the no-secret tier reconciles; the
secret-blocked components are simply not created until hand-offs arrive. The
no-secret tier carries no SOPS content, so it needs no sops-age key. prod is
untouched (still builds the monolith). Convergence path coded in (delete rollout/
+ the two deletes) — R-ENV-02 divergence flagged for architect review.
Component hardening via cluster-vars (prod keeps strict posture):
- KYVERNO_VALIDATION_ACTION Enforce->Audit(dev), _WEBHOOK_FAILURE_POLICY
Fail->Ignore(dev), _ADMISSION_REPLICAS 3->1(dev); webhook namespaceSelector
excludes system+platform namespaces.
- MONITORING_STORAGE_CLASS var; GRAFANA_ADMIN_SECRET var (dev ''=chart default);
Alertmanager SMTP/secret deferred to Phase-4 with verbatim re-enable markers.
Cilium parity verified against the live CNI (no-op adopt). Validated:
kubectl kustomize dev/prod/all-components, envsubst->YAML parse, make -C ci gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Flux postBuild.substituteFrom is case-sensitive; platform/ components reference
${UPPERCASE} tokens but cluster-vars keys were lowercase -> none would resolve
(literal ${...} would reach the HelmReleases). Uppercased the ConfigMap data
keys + the dev/prod overlay patch paths. Every component ref now resolves;
clusters/dev & clusters/prod build; gate green. Clears the K1 landmine before
any platform un-suspend.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Run a single Gitea container on the host (bootstrap/gitea/), outside the cluster,
while the cluster is built; Flux pulls this repo from it over the k3s-dev NAT
gateway (http://10.73.0.1:3000/Home-Network/home-network-k3s-impl.git, public repo
=> no creds, NAT-only). Wired the flux-system GitRepository url in clusters/dev.
Migrates to in-cluster Gitea in Phase 6. Pins confirmed. Author-only; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Phase 0 foundation shell (read-only spec at ../home-network-k3s). Top-level
tree + rules; workstream agents fill their subtrees on feature branches.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
