Stand up a single-node/single-drive MinIO S3 endpoint on the Proxmox host as the
off-cluster backup target for Velero (R-STATE-03) and CloudNativePG (R-IDP-05).
Ansible (author + executed):
- roles/minio_backup_target + playbooks/backup-minio.yml — idempotent, additive only:
ZFS dataset Storage/k3s-backups (zstd, quota=100G, /mnt/Storage/k3s-backups, no
sharenfs); minio system user; pinned+checksummed MinIO/mc binaries (fetched on the
controller since the host has no WAN, copied to host); hardened systemd unit bound to
the host's cluster-reachable LAN IP (auto-discovered) on :9000; velero + cnpg buckets;
a non-root access key scoped by IAM policy to only those two buckets.
- inventory: new [proxmox_hosts] group ([email protected].2.102); overrides the cluster-wide
ansible_user. Does not disturb existing k3s node entries.
- group_vars/proxmox_hosts/vault.yml (ansible-vault): root + scoped MinIO creds.
GitOps hand-off:
- platform/backup/minio-credentials.sops.yaml (+ .example) — SOPS/age-encrypted endpoint +
scoped key/secret for the Velero/CNPG wiring (WS-STORAGE/WS-SEC). Not yet wired into a
kustomization (left for the consuming agents).
Verified live: dataset props correct; minio active+enabled bound to 192.168.1.9:9000;
S3 endpoint http://192.168.1.9:9000/minio/health/ready returns 200 from an in-cluster pod;
buckets velero+cnpg present; scoped key created+policy attached; re-run is idempotent.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The plural --allowed-groups does not exist in oauth2-proxy v7.8.1; the shared
oauth2-proxy new pod and the crafty auth sidecar both crashlooped on 'unknown flag'.
The flag is the singular, repeatable --allowed-group.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add Crafty Controller 4 (registry.gitlab.com/crafty-controller/crafty-4:4.10.8)
as a platform component: a web panel that runs Java + Bedrock Minecraft servers as
subprocesses in one pod + a 20Gi Longhorn PVC (mounted 5x by subPath).
Auth = Option B (per-app oauth2-proxy reverse-proxy sidecar), since Cilium 1.16.5
cannot do ext_authz on the shared Gateway:
- oauth2-proxy v7.8.1 sidecar (:4180) reverse-proxies to Crafty's self-signed HTTPS
UI (127.0.0.1:8443), gated to Keycloak homelab realm group platform-admins.
- Confidential `crafty` client provisioned live via kcadm + captured in the realm
import; client secret + cookie secret via SOPS/age (mirrored in realm-clients).
- Web UI: HTTPRoute crafty.home.keithjackson.dev -> Service :80 -> sidecar :4180.
- Game ports: Cilium L2 LoadBalancer on VIP 192.168.1.55 (next free after .53/.54),
starter set TCP 25565-25567 (Java) + UDP 19132-19134 (Bedrock), NOT auth-gated.
- fsGroup:0 / runAsGroup:0 / runAsUser:1000 for the image's crafty:root identity on
the root:root Longhorn PVC (the seerr fsGroup lesson).
- Own non-gating Flux tier clusters/base/crafty.yaml (dependsOn platform-config,
wait:false) + sops-decryption patch in clusters/dev — a leaf UI must never gate
the platform.
- Explicit default-deny NetworkPolicy + CiliumNetworkPolicies (platform namespace,
not app-namespace, so Kyverno does not auto-generate the baseline).
make -C ci gate green; kustomize build of platform/crafty, clusters/base,
clusters/dev clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Build the deferred forward-auth ENFORCEMENT for non-OIDC UIs (pihole/*arr/
qbittorrent/homepage; Crafty next), gated by Keycloak via oauth2-proxy.
- helmrelease: add `--allowed-groups=platform-admins` (R-IDP-06 floor:
authenticated AND in group) + `--skip-provider-button` (302 straight to
Keycloak; verified the interstitial 403 sign-in page otherwise).
- forward-auth-cec.yaml: interim self-authored CiliumClusterwideEnvoyConfig
ext_authz listener -> oauth2-proxy /oauth2/auth, failure_mode_allow:false,
per-route opt-in via virtualHost. TEMPLATE — not Flux-wired (PM deploys with
an L2 VIP + DNS repoint). The ext_authz stanza is verified in-cluster.
- README: document the mechanism, why per-route ext_authz on the shared Gateway
is NOT clean on Cilium 1.16.5 (machine-owned cilium-gateway-shared CEC; native
ExternalAuth/GEP-1494 in no released Cilium), the three options + recommendation,
the opt-in steps, the in-cluster proof, and the human browser test.
Keycloak `oauth2-proxy` client already carries the callback redirect URI + a
bare `groups` claim (full.path=false) — no realm change needed.
Proof (cilium-envoy v1.30.8): a standalone envoy running this exact ext_authz
stanza against the live oauth2-proxy denied unauthenticated requests (403,
oauth2-proxy sign-in body; upstream never served), while a no-filter control
route on the same backend returned 200. R-IDP-02 property holds.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).
Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
failed ... write to WAL: no space left on device". Scrapes still return 200
(targets show healthy, lastScrape=now) but samples are never stored, so the
TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
present) — the rules are correct; the monitor was simply blind.
`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.
- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale
Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Plex Deployment had no Plex env vars. Add via the ServiceDefinition env list
(rendered verbatim by catalog/render.py):
- PLEX_UID/PLEX_GID=1000 (image chowns /config to this and drops; matches the
Longhorn ownership convention seerr/semaphore use)
- TZ=Etc/UTC (no other service sets TZ repo-wide)
- ALLOWED_NETWORKS=10.0.0.0/8,192.168.0.0/16 so in-cluster clients (Seerr ->
plex:32400) are trusted as LAN (pod CIDR 10.0.0.0/16 + svc 10.43.0.0/16 + LAN)
PLEX_CLAIM intentionally not set (4-min TTL is unusable via git; one-time live
claim persists on /config). ADVERTISE_IP deferred: the only candidate host
resolves via split-horizon DNS to the internal Gateway VIP and its remote
reachability is unverified — a wrong value breaks remote access, so omitted
per prefer-safety, documented as a follow-up in catalog/plex.yaml.
Homepage still labeled seerr as "Overseerr"; renamed the tile to "Seerr"
(icon mdi-play-box-multiple keeps — not overseerr-specific).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add the offline_access optional client scope to the `kubernetes`
kubelogin client so the operator's kubectl can request long-lived
OFFLINE refresh tokens (--oidc-extra-scope=offline_access) instead of
being forced to re-auth on the realm's ONLINE session timeouts
(ssoSessionIdleTimeout 30m / ssoSessionMaxLifespan 10h). Offline tokens
honour offlineSessionIdleTimeout 30d with no hard max lifespan.
- keycloak-realm-import.yaml: add optionalClientScopes: [offline_access]
to the kubernetes client so a fresh realm import reproduces it.
- k8s-oidc-rbac.yaml: document the kubeconfig exec-arg
(--oidc-extra-scope=offline_access) + clear-cache/re-login steps.
Realm session timeouts unchanged; scope is optional (opt-in) and
client-only. Applied live via kcadm (reversible).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Homepage v2.1.2 treats proxmox.yaml as a required config file (present in
the image skeleton dir /app/src/skeleton alongside services/settings/
bookmarks/widgets/docker/kubernetes/custom.*). On startup it copies a
default skeleton into /app/config for any required file that is missing.
The ConfigMap provided every skeleton file except proxmox.yaml, so the pod
tried to copyfile the proxmox skeleton into the read-only subPath-mounted
/app/config and crashed with EACCES -> CrashLoopBackOff -> Gateway "no
healthy upstream".
Provide an empty proxmox.yaml in the ConfigMap and mount it via subPath
like the other config files, so Homepage finds all required files and
never writes at runtime. Verified the full required set against the running
image's skeleton dir (9 files); proxmox.yaml was the only missing one.
Bumped checksum/config annotation to re-roll the pod.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Deploy Homepage (gethomepage) as the apex landing dashboard aggregating
links to cluster services, in its OWN non-gating post-platform Flux
Kustomization (clusters/base/homepage.yaml, dependsOn platform-config,
wait:false) — mirrors headlamp/oauth2-proxy so a leaf UI can never gate
the platform chain.
- platform/homepage/: plain manifests (ns, config ConfigMap, Deployment,
Service, default-deny+Cilium ingress NP, apex HTTPRoute). Non-root pod,
HOMEPAGE_ALLOWED_HOSTS, bundled mdi-* icons (DNS-only egress).
- gateway.yaml: add https-apex + http-apex-redirect listeners for the bare
apex (the *.home wildcard listener does NOT match it). Apex is already a
SAN on the wildcard cert — no new Certificate.
- Forward-auth annotated (auth-mode=forward-auth) but NOT enforced: the
Gateway ext-authz mechanism is an unbuilt Phase-6 WS-NET TODO (same status
as all other forward-auth apps). Flagged as PM follow-up.
Gate: lint/l0/render-golden(10/10)/l1 green; conformance skipped (no cluster).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Headlamp's HelmRelease sat in the wait:true `platform` Flux tier. Its pod
crashlooped (`flag provided but not defined: -session-ttl` — chart 0.40.1
renders a container arg the appVersion v0.40.1 backend lacks; upstream
kubernetes-sigs/headlamp#4883), and because it gated `platform`, the whole
chain stalled: platform Ready=Unknown -> platform-config, oauth2-proxy,
apps-wave-0/1/2 and apps-vms all Ready=False "dependency not ready".
Part A (unblock + durable lesson): a leaf UI must never gate the chain. Move
all Headlamp resources into a new non-gating Flux Kustomization
clusters/base/headlamp.yaml (dependsOn platform-config, wait:false, SOPS
decryption patched per-env in clusters/dev) — mirroring oauth2-proxy exactly.
The new Kustomization owns the whole platform/headlamp component; its HTTPRoute
moves from platform-config into platform/headlamp/kustomization.yaml. Removed
headlamp from platform/kustomization.yaml and its httproute from
platform-config/kustomization.yaml. Now a crashlooping Headlamp can never stall
platform/platform-config/apps.
Part B (make it start): pin the chart to 0.41.0 (`>=0.41.0 <0.42.0`). The
backend added `f.Int("session-ttl", …)` in v0.41.0 and chart 0.41.0 pins that
same image (appVersion == chart version), so chart+image realign; every arg the
chart renders for our values is supported. OIDC config + per-user RBAC
(clusterRoleBinding.create:false) unchanged.
Gate: `make -C ci gate` PASS.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add Headlamp (kubernetes-sigs/headlamp Helm chart) at
headlamp.home.keithjackson.dev with native Keycloak OIDC login and PER-USER
RBAC: Headlamp forwards each user's id_token to the k3s apiserver, so the
existing oidc:* group bindings (platform-admins->cluster-admin, viewers->view)
apply per user. The ServiceAccount is deliberately NOT bound to cluster-admin
(clusterRoleBinding.create=false), since user requests ride the forwarded token.
Key integration nuance: the apiserver is configured oidc-client-id=kubernetes,
so it only accepts tokens whose aud contains `kubernetes`. The new `headlamp`
Keycloak client carries an oidc-audience-mapper (included.client.audience=
kubernetes) so tokens carry aud=[headlamp, kubernetes] -- accepted by both
Headlamp's own verifier and the apiserver, leaving the apiserver config
untouched. A groups mapper emits the `groups` claim the apiserver maps to RBAC.
- platform/headlamp: namespace, HelmRepository/HelmRelease, SOPS OIDC secret
(externalSecret envFrom), default-deny NetworkPolicy + Cilium ingress
(gateway)/egress (apiserver+Keycloak) policies, README. Wired into
platform/kustomization.yaml (platform tier).
- platform/headlamp/httproute.yaml wired into platform-config (Gateway API CR,
dependsOn platform), modeled on grafana-httproute.
- keycloak-realm-import: add `headlamp` confidential client + groups/audience
mappers + HEADLAMP_CLIENT_SECRET placeholder. Client also created LIVE via
kcadm (realm import no-ops on the existing realm).
- keycloak-realm-clients.sops: add headlamp-client-secret (mirrored into
headlamp-oidc-credentials clientSecret). Both encrypted to the repo age key.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two SSO follow-ups the operator hit after Nextcloud OIDC login worked.
1. Sign-out "redirect failed". On logout user_oidc redirects the browser to
Keycloak's end_session_endpoint with post_logout_redirect_uri=<Nextcloud
base '/'> and client_id=nextcloud but NO id_token_hint (provider
sendIdTokenHint=0). Keycloak then validates that redirect against the
client's post.logout.redirect.uris attribute, which was empty -> "Invalid
redirect uri" -> the operator's "redirect failed". Register
https://nextcloud.home.keithjackson.dev/* on the nextcloud client
(attributes.post.logout.redirect.uris). Applied live via kcadm and captured
here so git stays authoritative (a RealmImport no-ops on the existing realm).
2. postStart step (d) mirrors platform-admins -> built-in admin group, but the
previous single best-effort pass ran occ group:adduser once with output
swallowed and no retry. At early boot (container Started ~9s after create)
that call can transiently fail (DB / maintenance race), silently dropping the
promotion until the next restart. Harden step (d) with a verify-and-retry
loop (like step (c)): re-read group:list and re-add any platform-admins
member still missing from admin until verified (bounded ~30s), no-op for
existing members.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy is a leaf forward-auth service that depends on Keycloak +
trusted wildcard TLS + split-horizon DNS. It lived in the gating `platform`
Flux Kustomization (wait:true), so on a cold start it Failed until those
dependencies existed, keeping the whole `platform` tier not-Ready. That in
turn blocked `platform-config` — which brings up the very Keycloak instance,
wildcard Certificate and L2 VIP oauth2-proxy needs (and previously carried
oauth2-proxy's own httproute/servicemonitor) — from ever reconciling. A
self-gating cold-start deadlock, twice-proven (DECISIONS-LOG sessions 7 & 9).
Break the cycle by moving the whole oauth2-proxy component out of the
`platform` tier into its own non-gating post-platform Flux Kustomization:
- clusters/base/oauth2-proxy.yaml: new Kustomization -> ./platform/oauth2-proxy,
dependsOn platform-config, wait:false, substituteFrom cluster-vars.
- clusters/base/kustomization.yaml: reference the new Kustomization.
- platform/kustomization.yaml: drop the oauth2-proxy component (no longer in
the gating tier).
- platform/oauth2-proxy/kustomization.yaml: pull in httproute + servicemonitor
so the full component travels together.
- platform-config/kustomization.yaml: drop the in-place oauth2-proxy
httproute/servicemonitor refs (moved with the controller).
- clusters/dev/kustomization.yaml: add the SOPS decryption patch for the new
oauth2-proxy Kustomization (it now owns the encrypted credentials Secret that
the `platform` Kustomization used to decrypt).
Now `platform` reaches Ready without oauth2-proxy; platform-config brings up
Keycloak/TLS/DNS; then oauth2-proxy reconciles against dependencies that
already exist. wait:false guarantees its health can never gate anything above
it. No config of oauth2-proxy itself changed — only its reconciliation tier.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Extends the SSO work to the two remaining `auth: oidc` apps still on local-admin
logins, each as a native confidential client of the Keycloak `homelab` realm with
its local admin kept as break-glass (docs/15 availability caveat; R-IDP-02/06).
Realm (source of truth): add the `nextcloud` and `semaphore` confidential clients
to platform/keycloak/keycloak-realm-import.yaml (modeled on grafana/gitea) + their
NEXTCLOUD_CLIENT_SECRET / SEMAPHORE_CLIENT_SECRET placeholders. As before, a
KeycloakRealmImport no-ops on an already-existing realm, so both live clients were
ALSO created via kcadm (nextcloud e95049ed-ca10-4085-97d9-4bbef495fe61, semaphore
1835bfe9-b125-4026-8f69-33557519cf4a); the YAML keeps git authoritative.
Secrets (SOPS/age): add nextcloud-client-secret + semaphore-client-secret to
keycloak-realm-clients.sops.yaml (keycloak ns). Mirror the nextcloud secret as
oidc-client-secret into nextcloud-bootstrap (nextcloud ns). For semaphore, the
whole SEMAPHORE_OIDC_PROVIDERS JSON (which embeds the client secret) is held
encrypted as oidc-providers-json in semaphore-bootstrap (semaphore ns). All values
verified == the live clients; existing keys untouched; no plaintext in git.
App config (catalog source + re-render):
- nextcloud: add NEXTCLOUD_OIDC_CLIENT_SECRET env + an idempotent postStart hook
that installs/enables user_oidc and runs `occ user_oidc:provider keycloak`
(create-or-update; secret via --clientsecret-env so it never hits argv; occ as
www-data; group provisioning on the `groups` claim). Syntax verified against the
running user_oidc 8.11.0 on Nextcloud 31.0.14; su-to-www-data env passthrough
verified. Missing-secret/not-installed guards prevent crashloops.
- semaphore: add SEMAPHORE_OIDC_PROVIDERS env sourced from the encrypted
oidc-providers-json (provider_url=realm issuer/auto-discovery, redirect
/api/auth/oidc/keycloak/redirect). Mechanism verified against semaphore v2.19.10
source (util ConfigType oidc_providers env tag + OidcProvider struct + router
redirect path); no kubelet service-link env collision.
apps/wave-1/{nextcloud,semaphore}.yaml re-rendered via
`render.py --no-image-automation` (diffs are env/lifecycle-only).
Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run; live
homelab clients now include gitea/nextcloud/semaphore; static gate (lint/l0/l1)
green. Browser SSO login for each still needs a human check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Wires the bootstrapping-critical Gitea git host to the Keycloak `homelab` realm
as a native confidential OIDC client, moving it off the local-admin-only login
while keeping the local admin as the break-glass path (docs/15 availability
caveat; R-IDP-02/06).
Realm (source of truth): add the `gitea` confidential client to
platform/keycloak/keycloak-realm-import.yaml (modeled on grafana) + its
GITEA_CLIENT_SECRET placeholder sourced from the keycloak-realm-clients Secret.
NOTE: a KeycloakRealmImport no-ops on an already-existing realm, so the live
client was ALSO created via kcadm (id 70a6ca60-317a-42ce-8331-a60e380838ee);
the YAML keeps git authoritative for a rebuild.
Secrets (SOPS/age, never plaintext): add gitea-client-secret to
platform/keycloak/keycloak-realm-clients.sops.yaml (keycloak ns) and mirror the
same value as oidc-client-secret into apps/wave-0/gitea-bootstrap-secret.sops.yaml
(gitea ns) so the app can read it in-namespace. Both == the live client secret.
App config (catalog source + re-render): catalog/gitea.yaml gains the
GITEA_OIDC_CLIENT_SECRET env (from the mirrored Secret), [oauth2_client]
auto-registration/account-linking app.ini env, and an idempotent postStart step
that registers the "keycloak" OAuth2 login source (auto-discovery; groups claim;
platform-admins -> Gitea admin). apps/wave-0/gitea.yaml re-rendered via
`render.py --no-image-automation`.
Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run;
static gate (lint/l0/l1) green. Browser SSO login still needs a human check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Lets users kubectl into the cluster via Keycloak SSO, mapping realm groups to
cluster RBAC. The cert-based admin kubeconfig is untouched and remains a fully
independent break-glass path (works even if Keycloak/OIDC is down).
- keycloak-realm-import.yaml: new `kubernetes` PUBLIC client + PKCE (S256) — no
client secret in any kubeconfig; kubelogin runs auth-code+PKCE to localhost.
groups mapper + email/profile/roles scopes.
- k8s-oidc-rbac.yaml (NEW, platform-config): ClusterRoleBindings
oidc:platform-admins -> cluster-admin, oidc:viewers -> view.
- k3s_server role: kube-apiserver-arg OIDC flags (issuer=homelab realm,
client-id=kubernetes, username-claim=email + oidc: prefix, groups-claim=groups
+ oidc: prefix). No oidc-ca-file (nodes reach the issuer via the gateway VIP
and trust the public LE cert). Applied via the rolling Restart k3s handler.
Keycloak/RBAC land via Flux now; the API-server flags take effect on the next
`ansible-playbook playbooks/k3s.yml` rolling restart (deferred until cluster load
from the VM-import incident settles).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The UniFi Block import got past the device-permission fix but then failed
ReplicaSchedulingFailure / "insufficient storage": CDI sizes scratch ~= the disk
virtual size and, on the default 3-replica `longhorn` class, the ~60Gi scratch x3
(189G) on top of the disk's x3 (180G) overflowed the Longhorn tier (~125G/node).
- platform/longhorn/storageclasses.yaml: NEW `longhorn-scratch` StorageClass
(numberOfReplicas: 1) — CDI scratch is ephemeral/throwaway, replicating it 3x is
waste; if a node dies mid-import CDI just retries.
- platform/kubevirt/cdi-cr.yaml: spec.config.scratchSpaceStorageClass ->
longhorn-scratch.
- apps/vms/unifi/20-datavolume.yaml: boot disk 60Gi -> 40Gi (UOS min is 25Gi;
40Gi is comfortable for a home site and lighter on the replicated tier).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds the VM/CDI storage tier approved by the PM (RWX for live-migration):
- platform/external-snapshotter (NEW, controllers pass): the cluster-singleton
snapshot.storage.k8s.io control plane (CRDs + snapshot-controller in
kube-system), pinned v8.2.0 (Kubernetes deps aligned to 1.31 since v8.1.0;
this cluster is k3s v1.31.5). These CRDs were absent — only snapshots.longhorn.io
and kubevirt.io snapshot CRDs existed.
- platform/longhorn/volumesnapshotclass.yaml (NEW, platform-config): the
driver.longhorn.io VolumeSnapshotClass (type: snap, in-cluster).
- platform/kubevirt/cdi-storageprofiles.yaml (NEW, platform-config): StorageProfile
longhorn -> cloneStrategy: snapshot; StorageProfile longhorn-vm -> snapshot clone
+ claimPropertySets RWX/Block. Non-degraded default virt class clears
CDIDefaultStorageClassDegraded.
- platform/longhorn/storageclasses.yaml: NEW longhorn-vm StorageClass —
migratable: "true" (Longhorn's KubeVirt-live-migration knob; RWX/Block-only on
1.7.3), is-default-virt-class (NOT the k8s default; longhorn stays RWO default).
- platform/kubevirt/kubevirt-cr.yaml: comment only — LiveMigration is GA on v1.9
(no feature gate); no cluster-wide evictionStrategy (would wedge RWO-disk drains);
live-migration stays per-VM opt-in on a longhorn-vm disk.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The alert fired continuously for ~21h while the control plane was fully
healthy (both virt-controller and virt-api pods 1/1 Ready, KubeVirt CR
Deployed/Available=True). Two bugs, each making it fire regardless of state:
1. `kubevirt_virt_controller_ready` does not exist — the real metric is
`kubevirt_virt_controller_ready_status` (KubeVirt suffixes `_status`),
so that clause was always ABSENT (never matched).
2. `absent(up{job="kubevirt"} == 1)` — the KubeVirt scrape job label is
`kubevirt-prometheus-metrics` (the Service name), not `kubevirt`, so
`up{job="kubevirt"}` matched nothing and absent() was permanently 1.
Fix: use `sum(kubevirt_virt_{controller,api}_ready_status) == 0` (fires only
when every replica is unready) and the correct job label. virt-api added to
match the alert's stated scope (virt-controller/virt-api).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The cdi-operator manages its own `service-monitor-cdi` (scheme https +
cdi-metrics-reader-token) which backs CDI's shipped recording rules
(kubevirt_cdi_operator_up / cluster:kubevirt_cdi_operator_up:sum) and the
CDIOperatorDown alert. Our extra `cdi` ServiceMonitor scraped the same
cdi-prometheus-metrics service (HTTPS :8443) over http, so every scrape
failed with 400 Bad Request, adding down (up=0) targets for the CDI pods.
That poisoned the up-aggregation the recording rule sums, driving it to 0
and firing a false CDIOperatorDown even though the operator was Running,
Ready, and Available=True. Remove the duplicate; keep the kubevirt SM
(virt-operator ships none of its own).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Admission controller was crashlooping (9 restarts/4h, exit 255, NOT OOM:
live usage ~65Mi/384Mi limit, 7m cpu). Root cause: chart-default 100m CPU
request starved the process under node contention, so client-go calls hit
context-deadline-exceeded, leader-election lease renewal timed out, leadership
was lost and the webhook server closed. Raise the CPU request to 250m (no CPU
limit, to avoid CFS throttling) with modest memory headroom (256Mi/512Mi), and
raise dev admission replicas 1->2 so a single fail-closed pod restart no longer
502s cluster-wide admission-gated writes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The offline `kyverno test` CLI cannot resolve the Flux postBuild token
${KYVERNO_VALIDATION_ACTION} in validationFailureAction, so the 4 policies
carrying it failed to load and their result rows reported "Not found" (14/17
cases). Render a throwaway copy of platform/kyverno with the base/prod value
(Enforce) before `kyverno test` in both ci/l0.sh and scripts/run-l0.sh; this
does not change in-cluster enforcement.
Add tests/values.yaml supplying Namespace labels so the offline CLI resolves
namespaceSelector matches -- without it the R-SEC-02/R-NET-05/R-IDP-02 Pod
cases were silently EXCLUDED (vacuous), not evaluated.
Pin kyverno-json 0.0.3 in ci/versions.env (0.0.5 has no upstream release;
0.0.3 is what run-l0.sh and the devcontainer already use).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The 3x 6GB nodes are OOM (k3s-dev-2's kubelet stopped; dev-1/dev-3 <600MB free). KubeVirt runs
a full control plane (~1.5-2GB) for ZERO running VMs — its IaaS conformance (R-PLAT-03/04,
R-NET-01, R-STATE-01) is already certified, so disabling the running pods loses no delivered
capability. Re-enable (uncomment kubevirt in platform + platform-config) once the VMs' RAM is
grown. Manifests retained; reversible.
The 39Gi/node Longhorn data disks at 1:1 over-provisioning can't fit the media stack
(plex 20Gi replica failed: 'insufficient storage'). Volumes are thin-provisioned config-only,
so 200% lets the full catalog fit without a disk resize; physical fill risk is low. Also
right-sized plex config (20Gi was excessive for metadata). Prod-grade path: grow the data
disks (WS-STORAGE follow-up).
VMs did not boot on the dev cluster. Three distinct live-verified causes,
all fixed GitOps-side:
1. containerDisk root failed at OCI init:
exec "/container-disk-binary/usr/bin/container-disk": no such file.
KubeVirt v1.9 supplies that binary to the volumerootdisk init container
via a k8s-native `image` volume (VolumeSource.Image), gated by the ALPHA
`ImageVolume` feature gate. k3s v1.31 has it DISABLED, so the apiserver
silently prunes the image volume + its mount -> binary missing -> crash.
(Proved: a server-side dry-run of a pod with an `image` volume drops it.)
Fix: convert the VM roots in all three tests to CDI DataVolumes imported
onto Longhorn PVCs (mounted disks bypass the container-disk/image-volume
path). This is also the spec-required disk model (docs/06, R-STATE-01).
vm-lifecycle now PASSES green live (VMI Running on start + restart).
2. Guest dropped to emergency mode: the containerdisks/fedora:41 image is
UEFI-native (ESP + /boot mounted via systemd GPT auto-discovery); under
KubeVirt's SeaBIOS default the ESP+/boot by-uuid mounts time out ->
local-fs.target fails -> cloud-init runcmd/poweroff never runs.
Fix: boot the VMs with EFI (OVMF, secureBoot:false) in all three tests
and fixtures/vm.yaml. Confirmed live: emergency mode gone with EFI.
3. VMs ran under TCG software emulation (guests ~10x slow, saturating nodes
until one went NotReady and VMs were evicted). Cause: useEmulation:true
in the KubeVirt CR makes virt-controller NOT request devices.kubevirt.io/
kvm, so /dev/kvm is never mounted and qemu falls back to TCG even though
all nodes advertise kvm=1k / hypervisor=true. (Proved: `ls /dev/kvm` in a
virt-launcher compute container = "No such file"; qemu accel = tcg.)
Fix: useEmulation:false so the kvm device is requested and VMs run on real
(nested) KVM. Requires Flux reconcile to take effect live (PM action).
The virt-handler clientcertificates tls.crt error is UNRELATED: node-labeller
(294 labels, hypervisor=true), device plugins, and VMI scheduling all work.
Also raised test exec timeouts to 30m (CDI import + boot) and, in
vm-default-deny, import the root BEFORE applying the default-deny NetworkPolicy
so the CDI importer's egress to quay.io is not severed by the policy.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Grafana's pod predated the grafana-oidc-credentials secret, so generic_oauth started with
an empty client_secret (disabled) → /login/generic_oauth didn't redirect to Keycloak.
Add a podAnnotation to force one roll; bump on future OIDC secret changes.
In-cluster clients resolving a platform hostname over its PUBLIC name got the
off-cluster answer, so pod->keycloak.home.keithjackson.dev hit a path that does not
serve the *.home.keithjackson.dev SNI -> `tls: unrecognized name` (oauth2-proxy OIDC
discovery crash; Grafana generic_oauth would hit the same wall).
Add a GitOps-managed coredns-custom ConfigMap (kube-system) that k3s CoreDNS imports as
a `*.server` block. The template plugin answers the whole home.keithjackson.dev zone with
${GATEWAY_VIP} (the shared Gateway VIP), so pod->public-hostname traffic hairpins through
the Gateway with the correct SNI -> wildcard cert validates. CoreDNS reloads the ConfigMap
live (no pod restart). Wired into platform/kustomization.yaml (Phase 2 network core).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy read the base64 cookie value as raw bytes (44) → 'must be 16/24/32 bytes'
crashloop. Regenerated cookie-secret as openssl rand -hex 16 (32 bytes). client-id +
client-secret unchanged (still match the realm oauth2-proxy client). Value tracked in
~/.config/home-k3s/keycloak-sso-secrets.txt.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add platform-health.automated-updates PrometheusRule group carried by platform-config:
ImageUpdateAutomationFailing (write-back broken -> advances silently stop),
ImagePolicyStalled (registry/range can't resolve a stable tag), and UpdateHealthGateFailed
(a bad update health-gated and auto-rolling-back via HelmRelease remediation). Routes to
Alertmanager email so a bad update is visible.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Map a ServiceDefinition `update` block to ImageRepository/ImagePolicy + the
$imagepolicy setter marker: stable-only (semver range + bare-X.Y.Z filterTags,
pre-release excluded, R-UPD-01), pin -> fixed image with no marker / suspended
policy (R-UPD-03). Worked pihole example under per-service/ (reference only, not
deployed; WS-GITOPS emits the equivalent into apps/<svc>/). Add renovate.json for
Helm chart / dependency / operator-manifest bumps: auto-merge minor/patch/digest on
green CI, major needs review, image managers disabled so Renovate never fights Flux
over a tag.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Install image-reflector-controller + image-automation-controller as part of the
flux-system install (bootstrap.sh --components-extra) so the image.toolkit CRDs
pre-exist before reconcile — no F-MON1 split needed for the ImageUpdateAutomation CR.
Refine the cluster-wide ImageUpdateAutomation for machine-authored git write-back
(setter strategy, path ./apps) reusing the flux-system GitRepository secretRef, and
document the operator git-write-credential hand-off + SOPS secret template
(gitea-image-automation-credentials). image-automation stays deferred in
platform/kustomization.yaml until that credential is provisioned.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Generate the 3 OIDC client secrets (SOPS-encrypted, minimal manifests) so the realm
import can inject the grafana + oauth2-proxy client secrets and both consumers match:
- platform/keycloak/keycloak-realm-clients.sops.yaml (grafana + oauth2-proxy client secrets)
- platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml (client-id/secret/cookie)
- platform/monitoring/grafana-oidc-credentials.sops.yaml (grafana client secret)
Wire each into its component kustomization and re-enable oauth2-proxy in platform +
platform-config (reverses the temp deferral now that its secret exists).
Values generated by PM per operator instruction (random; recorded out-of-repo at
~/.config/home-k3s/keycloak-sso-secrets.txt for later rotation). Encrypted via sops in the
dev container to the age recipient in .sops.yaml; plaintext never committed.
Serves: R-IDP-01, R-IDP-02, R-IDP-04, R-SEC-01.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy's HelmRelease mounts existingSecret oauth2-proxy-credentials, an operator
hand-off not yet filled. Under the unified graph (platform wait:true), the not-ready HR
blocks platform -> platform-config -> the whole downstream graph including KubeVirt.
Deferring it (comment its line in platform/kustomization.yaml + its two entries in
platform-config) lets the no-secret components (KubeVirt/CDI, CNPG, Keycloak operator +
instance) reconcile and be certified now. Re-enable both together once the operator fills
platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml.
Serves: R-IDP-02 (deferred). PM sequencing correction — enable per phase AND readiness.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
pia-credentials and pia-custom-ovpn were gluetun's PIA creds; gluetun is retired and the node
tunnel uses host-side Ansible vault creds (docs/07), so these cluster Secrets are unused. Remove
them from the kustomization + delete the encrypted files (recoverable from git history). Flux
will prune the Secrets from the vpn-egress namespace.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The policy is wired into the kustomization, reconciled by Flux, and conformance-green; gluetun
is retired. Update the header banner + bottom checklist to reflect that. Comment-only — the
CiliumEgressGatewayPolicy spec is unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The in-cluster Gluetun gateway is retired; VPN egress is now a node-level host
OpenVPN split-tunnel (Ansible role vpn_egress_gateway) steered by a Cilium
Egress Gateway policy with a host nftables kill-switch (ADR-0015). Credentials
moved host-side to the Ansible vault. Update docs/comments only:
- platform/vpn-egress/README.md: rewrite for ADR-0015 (live + conformance-green);
drop the resolved "option 1 not viable with Gluetun" finding to a brief history note.
- platform/kustomization.yaml: Phase 3 comment -> node tunnel + Cilium steering.
- platform/kyverno/.../require-egress-class-label.yaml: ADR-0013 -> ADR-0015 steering.
- schemas/servicedefinition.schema.json: egress=vpn description -> node tunnel + Cilium.
- clusters/base/cluster-vars.yaml: reframe PIA_REGION / CLUSTER_POD_CIDR WHY comments
to ADR-0015 (Cilium excludedCIDRs); both keys retained (still referenced/documented).
- platform/vpn-egress/namespace.yaml: no privileged gateway pod under ADR-0015.
- pia-credentials.sops.yaml / pia-custom-ovpn.sops.yaml: comment headers only note
they are now UNUSED (creds host-side), retained pending cleanup; payloads untouched.
No runtime manifest behavior, alert exprs, kustomization resources, or SOPS payloads
changed. No spec-repo edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 4.8