Commit Graph
100 Commits
Author SHA1 Message Date
jakman85andClaude Opus 4.8 af5fc5b25f feat(tofu,ansible): SSD-backed etcd disk per k3s node (Phase 1: code + plan)
Interim bridge to stop Longhorn replica-rebuild I/O storms on the shared HDD
Storage pool from starving etcd WAL fsync (crash-looping k3s / flapping the
kube-vip VIP). Adds a small dedicated SSD disk per node for etcd only; Longhorn
stays on HDD. See docs/plans/ssd-etcd-isolation.md.

tofu (infra/tofu):
- New vars etcd_ssd_datastore (default "local-zfs" = rpool/data, Micron 5100
  SATA SSD mirror w/ PLP) and etcd_disk_gib (default 16).
- New virtio2 disk block on the SSD datastore per VM: discard=on, cache=none
  (honor guest fsync straight to the ZFS ZIL — safe on PLP SSD), iothread=false.
  Kept last (ascending interface) so bpg does not reorder virtio0/virtio1.
- Revert the debian_cloud_image resource to the long type name
  proxmox_virtual_environment_download_file to match the R2 state: bpg >= 0.111
  dropped the cross-type moved{} linkage to the short proxmox_download_file, so
  the stale moved block failed EVERY plan with "Resource type mismatch" (broken
  on main too). Aligning code to state yields a clean plan with no image
  re-download. Short-name migration is deferred to a deliberate `tofu state mv`.

ansible (node_baseline):
- Format /dev/vdc (label k3s-etcd) and mount it at
  /var/lib/rancher/k3s/server/db (defaults,noatime; no nofail so a missing etcd
  disk fails loudly rather than silently landing etcd on the root HDD) before
  k3s starts, so a rebuilt node comes up with etcd isolated on SSD. Data-safe:
  filesystem module never reformats an existing fs.

Phase 1 only: non-disruptive. No tofu apply, no node/VM/etcd/Longhorn changes.
tofu plan verified: 0 to add, 3 to change, 0 to destroy — every VM an in-place
disk ADD (virtio2), dev-3 also an in-place memory 16384->24576; no VM replace.
Merge gate green (render==golden included).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-29 17:21:06 -05:00
jakman85 7910128675 Merge: tofu memory_mib 16384→24576 (k3s node RAM bump target) 2026-08-29 14:58:15 -05:00
jakman85andClaude Opus 4.8 a60d34a4c3 tofu: bump k3s node memory_mib default 16384 -> 24576 (24 GiB)
Raise per-node RAM from 16 GiB to 24 GiB for proper failover headroom.
Matches the live rolling resize of the three k3s-dev VMs on Proxmox
(qm set --memory 24576 per node, one at a time, KSM enabled first).

Single memory_mib var applies to all three nodes via for_each, so a
future `tofu apply` would resize all three at once and break etcd
quorum — the resize MUST be done live/rolling, not via apply.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-29 12:55:45 -05:00
jakman85andClaude Opus 4.8 178d130dc3 fix(velero): disable CRD-upgrade hook instead of pinning bitnami/kubectl
Supersedes the previous kubectl-image repoint. The velero chart's pre-install
`velero-upgrade-crds` HOOK job runs docker.io/bitnami/kubectl, whose public
Docker Hub catalog Bitnami removed (Aug 2025); `bitnami/kubectl:1.31` 404s,
leaving the hook in Init:ImagePullBackOff so Velero never installs.

Repointing the hook to the exact-version rancher/kubectl:v1.31.5 (prior commit)
does not work: that image is distroless, and the hook's init runs
`/bin/sh -c "cp $(which sh)/... $(which kubectl) /tmp"` — with no /bin/sh it
crashes StartError (verified on-cluster).

The hook exists ONLY to UPDATE CRDs on chart upgrades — it is NOT how CRDs are
first installed. The chart ships all 13 Velero CRDs in its native `crds/`
directory, which Helm applies unconditionally on install (before hooks),
independent of this flag; the CRDs are already present on-cluster from Helm's
crds/ handling. Setting `upgradeCRDs: false` drops the fragile external
kubectl-image dependency entirely while CRDs still install. Trade-off: Helm
won't manage crds/ after install, so CRD schema upgrades become a
manual/Renovate step; acceptable given Velero's CRDs are stable.

All other Velero values (BSL -> versitygw s3://velero, kopia FSB node-agent, the
two Schedules, the velero-cloud-credentials secret) are unchanged.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-29 11:09:45 -05:00
jakman85andClaude Opus 4.8 8fdd2a3f60 fix(velero): repoint CRD-hook kubectl image off the removed bitnami mirror
The velero chart's pre-install `velero-upgrade-crds` hook defaults its kubectl
image to docker.io/bitnami/kubectl, whose public Docker Hub catalog Bitnami
removed (Aug 2025). `docker.io/bitnami/kubectl:1.31` now 404s, leaving the hook
job in Init:ImagePullBackOff so Velero never installs (no Deployment,
node-agent, or BackupStorageLocation).

Set kubectl.image to the Rancher-maintained mirror docker.io/rancher/kubectl,
pinned to the cluster's exact k8s minor v1.31.5 (multi-arch amd64/arm64,
verified pullable via the Docker Hub registry API). All other Velero values
(BSL → versitygw s3://velero, kopia FSB node-agent, the two Schedules, the
velero-cloud-credentials secret) are unchanged.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-29 11:01:17 -05:00
jakman85 b5d65e9bc8 Merge: Velero (kopia FSB) + CNPG Barman backups → S3 target (resolves OQ-BKP) 2026-08-28 22:50:23 -05:00
jakman85andClaude Opus 4.8 6739339f53 Replace MinIO backup target with versitygw (s3_backup_target role)
MinIO's open-source edition is de-prioritised (console removed, commercial
pivot), so swap the off-cluster S3 backup target on the Proxmox host to
versitygw v1.7.0 (github.com/versity/versitygw) — a maintained Go S3 gateway
serving a POSIX filesystem as S3.

Rename role minio_backup_target -> s3_backup_target and playbook backup-minio
-> backup-s3. The role now:
  - decommissions any pre-existing MinIO (service/binary/env/mc-client/user/
    object-layout), KEEPING the ZFS dataset Storage/k3s-backups and its mount
  - installs the pinned, checksummed versitygw binary (fetched+extracted on the
    controller, host has no internet), a dedicated versitygw system user, and a
    Type=simple hardened systemd unit running the posix backend rooted at
    /mnt/Storage/k3s-backups/data, bound to 192.168.1.9:9000 (same endpoint),
    plain HTTP / path-style
  - ensures the dataset has xattr=sa (versitygw stores object metadata in xattrs)
  - pre-creates the velero + cnpg buckets (bucket == top-level subdir)

Credentials are REUSED: versitygw's ROOT key is set to the same scoped
access_key/secret the cluster already uses, so endpoint + key + secret + bucket
names are byte-identical to the old MinIO target. The GitOps SOPS secret
(platform/backup/minio-credentials.sops.yaml) and all cluster-side Velero/CNPG
config need ZERO change. The vaulted group_vars now expose
s3_backup_target_access_key/_secret_key (same values, MinIO root creds retired).

Verified live on the Proxmox host: versitygw active+enabled on 192.168.1.9:9000,
MinIO fully removed, dataset intact; SigV4 ListBuckets with the reused key
returns velero+cnpg; a throwaway in-cluster pod reaches http://192.168.1.9:9000
(versitygw S3 response). Re-run is idempotent (no host changes).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 22:48:38 -05:00
jakman85andClaude Opus 4.8 d1248e6778 feat(backup): wire Velero + CNPG Barman to the live MinIO S3 target
Author the GitOps side of off-cluster backups against the on-Proxmox MinIO
(http://192.168.1.9:9000, path-style; buckets velero + cnpg).

Velero (R-STATE-03):
- HelmRelease (chart 8.x / Velero 1.15) + velero-plugin-for-aws init container +
  node-agent kopia File System Backup (deployNodeAgent, uploaderType=kopia,
  snapshotsEnabled=false). BSL default -> bucket velero, s3ForcePathStyle.
- Two daily Schedules: `daily` backs up the whole cluster's resources+PVC data
  (defaultVolumesToFsBackup opt-out) EXCEPT monitoring+unifi; `daily-resources-only`
  captures monitoring+unifi RESOURCES only (Prometheus TSDB, Loki 72h logs, and the
  ~7.4GiB UniFi VM disk are reconstructable — excluded from volume backup).
- velero-cloud-credentials SOPS Secret (cloud INI key, ns velero).
- Own non-gating Flux tier clusters/base/backup.yaml (dependsOn platform-config,
  wait:false) + clusters/dev SOPS decryption patch — mirrors headlamp.

CNPG native Barman PITR (R-IDP-05 / R-STATE-03):
- keycloak-db, nextcloud-db, semaphore-db gain spec.backup.barmanObjectStore
  (s3://cnpg/<cluster>, endpointURL MinIO, gzip WAL+data, 30d retention) + a daily
  ScheduledBackup (6-field cron, immediate) each.
- Per-namespace scoped-key SOPS Secrets (<db>-backup-credentials), same-ns only.
- render.py: emit-waves now recognizes <svc>-db-backup-credentials.sops.yaml as a
  DB-wave sidecar (+ self-test); wave-0 kustomization regenerated.

Creds mirror platform/backup/minio-credentials.sops.yaml (scoped key k3s-backup),
SOPS round-trip verified with the cluster age key. gate: green.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 22:37:17 -05:00
jakman85 1bee2554f8 Merge: MinIO S3 backup target on Proxmox (ansible role + live) + SOPS creds 2026-08-28 22:16:25 -05:00
jakman85andClaude Opus 4.8 ac32f01f7b feat(backup): MinIO S3 backup target on Proxmox host for Velero + CNPG
Stand up a single-node/single-drive MinIO S3 endpoint on the Proxmox host as the
off-cluster backup target for Velero (R-STATE-03) and CloudNativePG (R-IDP-05).

Ansible (author + executed):
- roles/minio_backup_target + playbooks/backup-minio.yml — idempotent, additive only:
  ZFS dataset Storage/k3s-backups (zstd, quota=100G, /mnt/Storage/k3s-backups, no
  sharenfs); minio system user; pinned+checksummed MinIO/mc binaries (fetched on the
  controller since the host has no WAN, copied to host); hardened systemd unit bound to
  the host's cluster-reachable LAN IP (auto-discovered) on :9000; velero + cnpg buckets;
  a non-root access key scoped by IAM policy to only those two buckets.
- inventory: new [proxmox_hosts] group ([email protected].2.102); overrides the cluster-wide
  ansible_user. Does not disturb existing k3s node entries.
- group_vars/proxmox_hosts/vault.yml (ansible-vault): root + scoped MinIO creds.

GitOps hand-off:
- platform/backup/minio-credentials.sops.yaml (+ .example) — SOPS/age-encrypted endpoint +
  scoped key/secret for the Velero/CNPG wiring (WS-STORAGE/WS-SEC). Not yet wired into a
  kustomization (left for the consuming agents).

Verified live: dataset props correct; minio active+enabled bound to 192.168.1.9:9000;
S3 endpoint http://192.168.1.9:9000/minio/health/ready returns 200 from an in-cluster pod;
buckets velero+cnpg present; scoped key created+policy attached; re-run is idempotent.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 22:15:05 -05:00
jakman85andClaude Opus 4.8 748e9ca55d fix(oauth2-proxy): --allowed-groups → --allowed-group (correct oauth2-proxy v7.8.1 flag)
The plural --allowed-groups does not exist in oauth2-proxy v7.8.1; the shared
oauth2-proxy new pod and the crafty auth sidecar both crashlooped on 'unknown flag'.
The flag is the singular, repeatable --allowed-group.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 15:57:55 -05:00
jakman85 d5773c780d Merge: Crafty Controller (Minecraft panel) behind Keycloak oauth2-proxy reverse-proxy + L4 game-port VIP 2026-08-28 15:40:55 -05:00
jakman85andClaude Opus 4.8 0437871807 Onboard Crafty Controller (Minecraft panel) with Keycloak-gated web UI
Add Crafty Controller 4 (registry.gitlab.com/crafty-controller/crafty-4:4.10.8)
as a platform component: a web panel that runs Java + Bedrock Minecraft servers as
subprocesses in one pod + a 20Gi Longhorn PVC (mounted 5x by subPath).

Auth = Option B (per-app oauth2-proxy reverse-proxy sidecar), since Cilium 1.16.5
cannot do ext_authz on the shared Gateway:
- oauth2-proxy v7.8.1 sidecar (:4180) reverse-proxies to Crafty's self-signed HTTPS
  UI (127.0.0.1:8443), gated to Keycloak homelab realm group platform-admins.
- Confidential `crafty` client provisioned live via kcadm + captured in the realm
  import; client secret + cookie secret via SOPS/age (mirrored in realm-clients).

- Web UI: HTTPRoute crafty.home.keithjackson.dev -> Service :80 -> sidecar :4180.
- Game ports: Cilium L2 LoadBalancer on VIP 192.168.1.55 (next free after .53/.54),
  starter set TCP 25565-25567 (Java) + UDP 19132-19134 (Bedrock), NOT auth-gated.
- fsGroup:0 / runAsGroup:0 / runAsUser:1000 for the image's crafty:root identity on
  the root:root Longhorn PVC (the seerr fsGroup lesson).
- Own non-gating Flux tier clusters/base/crafty.yaml (dependsOn platform-config,
  wait:false) + sops-decryption patch in clusters/dev — a leaf UI must never gate
  the platform.
- Explicit default-deny NetworkPolicy + CiliumNetworkPolicies (platform namespace,
  not app-namespace, so Kyverno does not auto-generate the baseline).

make -C ci gate green; kustomize build of platform/crafty, clusters/base,
clusters/dev clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 15:39:51 -05:00
jakman85 a207103c7b Merge: oauth2-proxy forward-auth hardening (platform-admins floor, skip-button) + ext_authz mechanism research/template 2026-08-28 15:20:38 -05:00
jakman85andClaude Opus 4.8 3e7e9386a0 feat(oauth2-proxy): forward-auth ext_authz enforcement mechanism + group floor
Build the deferred forward-auth ENFORCEMENT for non-OIDC UIs (pihole/*arr/
qbittorrent/homepage; Crafty next), gated by Keycloak via oauth2-proxy.

- helmrelease: add `--allowed-groups=platform-admins` (R-IDP-06 floor:
  authenticated AND in group) + `--skip-provider-button` (302 straight to
  Keycloak; verified the interstitial 403 sign-in page otherwise).
- forward-auth-cec.yaml: interim self-authored CiliumClusterwideEnvoyConfig
  ext_authz listener -> oauth2-proxy /oauth2/auth, failure_mode_allow:false,
  per-route opt-in via virtualHost. TEMPLATE — not Flux-wired (PM deploys with
  an L2 VIP + DNS repoint). The ext_authz stanza is verified in-cluster.
- README: document the mechanism, why per-route ext_authz on the shared Gateway
  is NOT clean on Cilium 1.16.5 (machine-owned cilium-gateway-shared CEC; native
  ExternalAuth/GEP-1494 in no released Cilium), the three options + recommendation,
  the opt-in steps, the in-cluster proof, and the human browser test.

Keycloak `oauth2-proxy` client already carries the callback redirect URI + a
bare `groups` claim (full.path=false) — no realm change needed.

Proof (cilium-envoy v1.30.8): a standalone envoy running this exact ext_authz
stanza against the live oauth2-proxy denied unauthenticated requests (403,
oauth2-proxy sign-in body; upstream never served), while a no-filter control
route on the same backend returned 200. R-IDP-02 property holds.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 15:18:47 -05:00
jakman85 3c7d7e0a1b Merge: Prometheus retentionSize cap + dev PVC 5Gi→8Gi (fixes full-disk false alerts) 2026-08-28 14:39:51 -05:00
jakman85andClaude Opus 4.8 eb31d2ef58 fix(monitoring): cap Prometheus TSDB by size to stop false absent()/==0 alerts
The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).

Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
  failed ... write to WAL: no space left on device". Scrapes still return 200
  (targets show healthy, lastScrape=now) but samples are never stored, so the
  TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
  absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
  expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
  up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
  present) — the rules are correct; the monitor was simply blind.

`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.

- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale

Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 14:38:12 -05:00
jakman85 96a5db9ed7 Merge: plex env vars (UID/GID/TZ/ALLOWED_NETWORKS) + Homepage Overseerr→Seerr rename 2026-08-28 13:21:19 -05:00
jakman85andClaude Opus 4.8 cde2cd138f plex: add env vars (PLEX_UID/GID, TZ, ALLOWED_NETWORKS); homepage: rename Overseerr->Seerr
Plex Deployment had no Plex env vars. Add via the ServiceDefinition env list
(rendered verbatim by catalog/render.py):
- PLEX_UID/PLEX_GID=1000 (image chowns /config to this and drops; matches the
  Longhorn ownership convention seerr/semaphore use)
- TZ=Etc/UTC (no other service sets TZ repo-wide)
- ALLOWED_NETWORKS=10.0.0.0/8,192.168.0.0/16 so in-cluster clients (Seerr ->
  plex:32400) are trusted as LAN (pod CIDR 10.0.0.0/16 + svc 10.43.0.0/16 + LAN)

PLEX_CLAIM intentionally not set (4-min TTL is unusable via git; one-time live
claim persists on /config). ADVERTISE_IP deferred: the only candidate host
resolves via split-horizon DNS to the internal Gateway VIP and its remote
reachability is unverified — a wrong value breaks remote access, so omitted
per prefer-safety, documented as a follow-up in catalog/plex.yaml.

Homepage still labeled seerr as "Overseerr"; renamed the tile to "Seerr"
(icon mdi-play-box-multiple keeps — not overseerr-specific).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 13:20:34 -05:00
jakman85 6fddca726b Merge: offline_access scope on kubernetes client (long-lived kubectl OIDC sessions) 2026-08-28 13:04:33 -05:00
jakman85andClaude Opus 4.8 330c99eb8b feat(keycloak): enable offline_access for kubernetes OIDC client
Add the offline_access optional client scope to the `kubernetes`
kubelogin client so the operator's kubectl can request long-lived
OFFLINE refresh tokens (--oidc-extra-scope=offline_access) instead of
being forced to re-auth on the realm's ONLINE session timeouts
(ssoSessionIdleTimeout 30m / ssoSessionMaxLifespan 10h). Offline tokens
honour offlineSessionIdleTimeout 30d with no hard max lifespan.

- keycloak-realm-import.yaml: add optionalClientScopes: [offline_access]
  to the kubernetes client so a fresh realm import reproduces it.
- k8s-oidc-rbac.yaml: document the kubeconfig exec-arg
  (--oidc-extra-scope=offline_access) + clear-cache/re-login steps.

Realm session timeouts unchanged; scope is optional (opt-in) and
client-only. Applied live via kcadm (reversible).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 13:01:37 -05:00
jakman85andClaude Opus 4.8 b6a48203bc fix(homepage): add required proxmox.yaml config to stop CrashLoopBackOff
Homepage v2.1.2 treats proxmox.yaml as a required config file (present in
the image skeleton dir /app/src/skeleton alongside services/settings/
bookmarks/widgets/docker/kubernetes/custom.*). On startup it copies a
default skeleton into /app/config for any required file that is missing.
The ConfigMap provided every skeleton file except proxmox.yaml, so the pod
tried to copyfile the proxmox skeleton into the read-only subPath-mounted
/app/config and crashed with EACCES -> CrashLoopBackOff -> Gateway "no
healthy upstream".

Provide an empty proxmox.yaml in the ConfigMap and mount it via subPath
like the other config files, so Homepage finds all required files and
never writes at runtime. Verified the full required set against the running
image's skeleton dir (9 files); proxmox.yaml was the only missing one.
Bumped checksum/config annotation to re-roll the pod.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 12:59:09 -05:00
jakman85 099c1408b8 Merge: Homepage apex landing dashboard (non-gating tier; forward-auth deferred) 2026-08-28 12:18:40 -05:00
jakman85andClaude Opus 4.8 1749603f87 feat(homepage): apex landing dashboard at home.keithjackson.dev
Deploy Homepage (gethomepage) as the apex landing dashboard aggregating
links to cluster services, in its OWN non-gating post-platform Flux
Kustomization (clusters/base/homepage.yaml, dependsOn platform-config,
wait:false) — mirrors headlamp/oauth2-proxy so a leaf UI can never gate
the platform chain.

- platform/homepage/: plain manifests (ns, config ConfigMap, Deployment,
  Service, default-deny+Cilium ingress NP, apex HTTPRoute). Non-root pod,
  HOMEPAGE_ALLOWED_HOSTS, bundled mdi-* icons (DNS-only egress).
- gateway.yaml: add https-apex + http-apex-redirect listeners for the bare
  apex (the *.home wildcard listener does NOT match it). Apex is already a
  SAN on the wildcard cert — no new Certificate.
- Forward-auth annotated (auth-mode=forward-auth) but NOT enforced: the
  Gateway ext-authz mechanism is an unbuilt Phase-6 WS-NET TODO (same status
  as all other forward-auth apps). Flagged as PM follow-up.

Gate: lint/l0/render-golden(10/10)/l1 green; conformance skipped (no cluster).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 12:17:25 -05:00
jakman85andClaude Opus 4.8 ebe0371b1b platform: move Headlamp out of the gating platform tier + fix -session-ttl crashloop
Headlamp's HelmRelease sat in the wait:true `platform` Flux tier. Its pod
crashlooped (`flag provided but not defined: -session-ttl` — chart 0.40.1
renders a container arg the appVersion v0.40.1 backend lacks; upstream
kubernetes-sigs/headlamp#4883), and because it gated `platform`, the whole
chain stalled: platform Ready=Unknown -> platform-config, oauth2-proxy,
apps-wave-0/1/2 and apps-vms all Ready=False "dependency not ready".

Part A (unblock + durable lesson): a leaf UI must never gate the chain. Move
all Headlamp resources into a new non-gating Flux Kustomization
clusters/base/headlamp.yaml (dependsOn platform-config, wait:false, SOPS
decryption patched per-env in clusters/dev) — mirroring oauth2-proxy exactly.
The new Kustomization owns the whole platform/headlamp component; its HTTPRoute
moves from platform-config into platform/headlamp/kustomization.yaml. Removed
headlamp from platform/kustomization.yaml and its httproute from
platform-config/kustomization.yaml. Now a crashlooping Headlamp can never stall
platform/platform-config/apps.

Part B (make it start): pin the chart to 0.41.0 (`>=0.41.0 <0.42.0`). The
backend added `f.Int("session-ttl", …)` in v0.41.0 and chart 0.41.0 pins that
same image (appVersion == chart version), so chart+image realign; every arg the
chart renders for our values is supported. OIDC config + per-user RBAC
(clusterRoleBinding.create:false) unchanged.

Gate: `make -C ci gate` PASS.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 12:01:01 -05:00
jakman85 d8883e4481 Merge: Headlamp k8s dashboard with Keycloak OIDC + per-user RBAC (audience-mapper) 2026-08-28 11:46:15 -05:00
jakman85andClaude Opus 4.8 5fb097d019 platform: deploy Headlamp dashboard with Keycloak OIDC + per-user RBAC
Add Headlamp (kubernetes-sigs/headlamp Helm chart) at
headlamp.home.keithjackson.dev with native Keycloak OIDC login and PER-USER
RBAC: Headlamp forwards each user's id_token to the k3s apiserver, so the
existing oidc:* group bindings (platform-admins->cluster-admin, viewers->view)
apply per user. The ServiceAccount is deliberately NOT bound to cluster-admin
(clusterRoleBinding.create=false), since user requests ride the forwarded token.

Key integration nuance: the apiserver is configured oidc-client-id=kubernetes,
so it only accepts tokens whose aud contains `kubernetes`. The new `headlamp`
Keycloak client carries an oidc-audience-mapper (included.client.audience=
kubernetes) so tokens carry aud=[headlamp, kubernetes] -- accepted by both
Headlamp's own verifier and the apiserver, leaving the apiserver config
untouched. A groups mapper emits the `groups` claim the apiserver maps to RBAC.

- platform/headlamp: namespace, HelmRepository/HelmRelease, SOPS OIDC secret
  (externalSecret envFrom), default-deny NetworkPolicy + Cilium ingress
  (gateway)/egress (apiserver+Keycloak) policies, README. Wired into
  platform/kustomization.yaml (platform tier).
- platform/headlamp/httproute.yaml wired into platform-config (Gateway API CR,
  dependsOn platform), modeled on grafana-httproute.
- keycloak-realm-import: add `headlamp` confidential client + groups/audience
  mappers + HEADLAMP_CLIENT_SECRET placeholder. Client also created LIVE via
  kcadm (realm import no-ops on the existing realm).
- keycloak-realm-clients.sops: add headlamp-client-secret (mirrored into
  headlamp-oidc-credentials clientSecret). Both encrypted to the repo age key.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 11:44:33 -05:00
jakman85 b243c753b1 Merge: migrate proxmox_virtual_environment_download_file -> proxmox_download_file (ADR-007, moved block) 2026-08-28 11:23:06 -05:00
jakman85andClaude Opus 4.8 0cf0c669f4 tofu: migrate to non-deprecated proxmox_download_file (ADR-007)
bpg/proxmox deprecates proxmox_virtual_environment_download_file ("Use
proxmox_download_file instead. This resource will be removed in v1.0."),
the short-name rename from ADR-007. Both are the same dual-registered
Framework resource, so rename the resource type + import_from reference
and add a `moved` block. OpenTofu 1.9 drives the provider's MoveState
(migration.PrefixMoveState) to transfer state in place: same id, no
re-download of the Debian image, no change to the VMs.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 11:22:06 -05:00
jakman85 ffb439d59e Merge fix/reconcile-apps-goldens: reconcile goldens to --no-image-automation + enforcing render-golden gate stage 2026-08-28 11:13:07 -05:00
jakman85andClaude Opus 4.8 fe0ee698c4 fix(apps): reconcile goldens to one canonical render + enforce render==golden gate
The committed apps/ goldens were a MIX of two render flag sets: 8 services
were rendered with `--no-image-automation` (no image CRs) while gitea and
nextcloud carried stale ImageRepository/ImagePolicy CRs from a render WITHOUT
the flag. Neither single command reproduced all goldens, and the gate did not
catch it.

Part A — canonical render:
  python3 catalog/render.py --all catalog -o apps --wave-layout --no-image-automation
`--no-image-automation` is canonical: image automation is authored-but-DISABLED
(DECISIONS-LOG; image-reflector/-automation controllers not installed), and
every app was onboarded via this flag. Re-render drops the ImageRepository +
ImagePolicy objects from gitea + nextcloud only (52 line deletions, purely the
trailing image-automation blocks). No functional content changed — the
nextcloud postStart hardening and gitea OIDC hook are untouched.

Part B — enforcement:
  ci/render-golden.sh renders every onboarded catalog service with the canonical
  command into a temp dir and FAILS on ANY diff vs committed apps/ (missing
  golden or content drift), across all 10 services (not just the 3 L1 spot-checks).
  Not-onboarded catalog entries (keycloak, vm-demo) are excluded per DESIGN-waves.
  Wired as a required no-cluster stage in ci/gate.sh, ci/Makefile, and both the
  live (.gitea/workflows/gate.yml) and dormant (merge-gate.yaml) pipelines.

Proof: tampering a golden makes the stage exit 1 (blocks merge); reverting
restores exit 0. `make -C ci gate` green in the devcontainer.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-28 11:11:52 -05:00
jakman85 85338a8144 Merge fix/nextcloud-logout-admin: post-logout redirect URI + hardened admin-promotion hook 2026-08-27 20:19:51 -05:00
jakman85andClaude Opus 4.8 14521d37d3 fix(nextcloud-sso): apply step-(d) retry hardening at SOURCE (catalog)
apps/wave-1/nextcloud.yaml is GENERATED from catalog/nextcloud.yaml by
catalog/render.py, so the previous commit hardened only the rendered golden —
the next render would have reverted step (d) to the old single-pass version.

Apply the same verify-and-retry hardening to the postStart hook in
catalog/nextcloud.yaml (the source of truth). Re-rendered with
`render.py --all catalog -o apps --wave-layout`; apps/wave-1/nextcloud.yaml is
byte-identical to the golden committed in the previous commit (render round-trip
clean), so no apps/ change is needed here.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 20:18:51 -05:00
jakman85andClaude Opus 4.8 7f86e1099b fix(nextcloud-sso): register post-logout redirect URI + harden admin promotion
Two SSO follow-ups the operator hit after Nextcloud OIDC login worked.

1. Sign-out "redirect failed". On logout user_oidc redirects the browser to
   Keycloak's end_session_endpoint with post_logout_redirect_uri=<Nextcloud
   base '/'> and client_id=nextcloud but NO id_token_hint (provider
   sendIdTokenHint=0). Keycloak then validates that redirect against the
   client's post.logout.redirect.uris attribute, which was empty -> "Invalid
   redirect uri" -> the operator's "redirect failed". Register
   https://nextcloud.home.keithjackson.dev/* on the nextcloud client
   (attributes.post.logout.redirect.uris). Applied live via kcadm and captured
   here so git stays authoritative (a RealmImport no-ops on the existing realm).

2. postStart step (d) mirrors platform-admins -> built-in admin group, but the
   previous single best-effort pass ran occ group:adduser once with output
   swallowed and no retry. At early boot (container Started ~9s after create)
   that call can transiently fail (DB / maintenance race), silently dropping the
   promotion until the next restart. Harden step (d) with a verify-and-retry
   loop (like step (c)): re-read group:list and re-add any platform-admins
   member still missing from admin until verified (bounded ~30s), no-op for
   existing members.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 20:13:45 -05:00
jakman85 7d93decd61 tofu: fill encrypted secrets 2026-08-27 19:53:46 -05:00
jakman85 0996c1da0d Merge feature/tofu-sops-secrets: SOPS-encrypted tofu secrets + tf.sh wrapper 2026-08-27 19:09:57 -05:00
jakman85andClaude Opus 4.8 216bea2a16 tofu: move secrets into committed SOPS/age file + tf.sh wrapper
Replace the gitignored plaintext infra/tofu/.r2.env and backend.hcl with a
single committed, SOPS/age-encrypted dotenv (secrets.sops.env) plus a thin
wrapper (tf.sh), so tofu runs with no manual secret entry and secrets live
encrypted in git like the repo's other secrets.

- .sops.yaml: creation rule encrypting infra/tofu/secrets.sops.env (all values)
  to the existing age recipient.
- secrets.sops.env: encrypted, PLACEHOLDER values only. Holds R2 S3 keys, the
  state-encryption passphrase (TF_STATE_PASSPHRASE scalar), and the Proxmox API
  token/endpoint. Operator fills real values via `sops`.
- tf.sh: sops exec-env decrypts the dotenv, exports PROXMOX_VE_*/AWS_*, builds
  the multi-line TF_ENCRYPTION HCL from TF_STATE_PASSPHRASE, and runs tofu
  (injecting -backend-config=backend.r2.hcl on init). Proxmox token stays env-only,
  never a tofu variable/state.
- backend.r2.hcl: committed non-secret R2 bucket/key/endpoint (account id is not
  a credential), passed to `tofu init`.
- Retire .r2.env/backend.hcl(.example): drop the committed templates; keep the
  gitignore lines as a safety net for leftover local plaintext.
- Update backend.tf/README/REMOTE-STATE for the new day-2 flow and DR story
  (repo + age key is sufficient).

Verified: placeholder file round-trips sops -e/-d with the repo age key; tf.sh
exports the expected VAR NAMES and builds valid TF_ENCRYPTION HCL (stub tofu, no
real plan/secrets).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 19:07:52 -05:00
jakman85 01f3dd0044 Merge fix/nextcloud-oidc-reach: allow_local_remote_servers for private-IP IdP + platform-admins→admin (#2,#5) 2026-08-27 19:03:30 -05:00
jakman85andClaude Opus 4.8 7f92adfac6 fix(nextcloud): allow SSRF to private IdP + promote platform-admins to admin
Task 1 — browser SSO "Could not reach OpenID Connect provider" (confirmed root
cause, not guessed). keycloak.home.keithjackson.dev resolves split-horizon to the
PRIVATE Cilium gateway VIP (192.168.1.53). Nextcloud's SSRF guard refuses its own
server-side HTTP requests to RFC1918 addresses, so user_oidc's discovery/JWKS/token
fetch AT LOGIN throws OCP\Http\Client\LocalServerException "violates local access
rules" -> surfaced to the user as "Could not reach the provider". Proven from the
live pod: nextcloud.log carries the exact LocalServerException from
DiscoveryService::obtainDiscovery with Guzzle option allow_local_address:false; a
Nextcloud-context IClientService->get() of the discovery URL returns
LocalServerException with the flag false and HTTP 200 with it true (A/B toggled).
The postStart (b) curl check passed regardless because it uses system libcurl, not
Nextcloud's guard — which is why the provider registered yet login still failed.

Fix (reproducible, survives pod rolls): the postStart hook now runs
`occ config:system:set allow_local_remote_servers --value=true --type=boolean`
right after the install/maintenance wait, on every start.

Task 2 — promote Keycloak `platform-admins` members to Nextcloud admins (#5).
user_oidc 8.11 has NO native group->admin mapping (occ user_oidc:provider offers
group PROVISIONING and a login WHITELIST only), so the hook mirrors the provisioned
`platform-admins` membership into the built-in `admin` group idempotently (occ
group:list --output=json parsed via php argv; occ group:adduser admin per member).
A user is promotable after their first OIDC login provisions them + the next pod
start (or one manual `occ group:adduser admin <uid>`). The LOCAL admin stays as
break-glass (docs/15).

apps/wave-1/nextcloud.yaml re-rendered from catalog (single-file render; byte-diff
confined to the embedded hook). Rendered hook passes `sh -n`; parse step validated
in-pod. `make -C ci gate` green (lint/l0/l1; conformance + spec-immutability skip:
no cluster / spec repo unmounted).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 19:00:09 -05:00
jakman85 318f85de11 Merge fix/plex-config-mountpath: mount config PVC at /config (Plex was writing ephemeral /config, unclaimed) 2026-08-27 18:54:44 -05:00
jakman85andClaude Opus 4.8 d835a27875 fix(plex): mount config PVC at /config (image's real config dir)
The plexinc/pms-docker image writes its identity/claim token and the
Library metadata DB under /config, but the config PVC was mounted at the
renderer default /data/config. Plex therefore wrote all state to the
container's EPHEMERAL layer: the server stayed UNCLAIMED (servers.plex.tv
returns 401 with an empty X-Plex-Token) and lost everything on restart,
which is why seerr's "add Plex" failed even with SSL off.

Set the config volume mountPath to /config in catalog/plex.yaml and
re-render apps/wave-0/plex.yaml (only the mountPath line changes; all
other services byte-identical). No fsGroup/securityContext needed:
verified in-cluster the image inits as root (s6-svscan PID 1), chowns
/config to the plex user, then drops privileges (the "Plex Media Server"
process runs as plex) — root can chown the mounted PVC regardless of
fsGroup.

PLEX_CLAIM auto-claim is intentionally NOT wired: the env secretKeyRef
schema has no `optional`, so a placeholder/absent claim Secret would
crashloop the pod. The claim is an operator step (UI sign-in after the
roll, or add a PLEX_CLAIM env from a fresh https://plex.tv/claim token).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 18:53:46 -05:00
jakman85andClaude Opus 4.8 13b1f1d9a4 docs(decisions): record 2026-08-27 session (R2 state, seerr, app-SSO, oauth2-proxy tier, R-STATE-02)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 18:24:14 -05:00
jakman85 0fcbf32742 Merge fix/oauth2-proxy-post-platform-tier: move oauth2-proxy to post-platform tier (breaks cold-start deadlock) 2026-08-27 18:15:25 -05:00
jakman85 1f67a8ae72 Merge feature/conformance-r-state-02-node-loss: L3 node-loss test (R-STATE-02, closes last safety-critical gap) 2026-08-27 18:13:33 -05:00
jakman85andClaude Opus 4.8 960cab937b fix(flux): move oauth2-proxy to a post-platform tier (dependsOn platform-config)
oauth2-proxy is a leaf forward-auth service that depends on Keycloak +
trusted wildcard TLS + split-horizon DNS. It lived in the gating `platform`
Flux Kustomization (wait:true), so on a cold start it Failed until those
dependencies existed, keeping the whole `platform` tier not-Ready. That in
turn blocked `platform-config` — which brings up the very Keycloak instance,
wildcard Certificate and L2 VIP oauth2-proxy needs (and previously carried
oauth2-proxy's own httproute/servicemonitor) — from ever reconciling. A
self-gating cold-start deadlock, twice-proven (DECISIONS-LOG sessions 7 & 9).

Break the cycle by moving the whole oauth2-proxy component out of the
`platform` tier into its own non-gating post-platform Flux Kustomization:

- clusters/base/oauth2-proxy.yaml: new Kustomization -> ./platform/oauth2-proxy,
  dependsOn platform-config, wait:false, substituteFrom cluster-vars.
- clusters/base/kustomization.yaml: reference the new Kustomization.
- platform/kustomization.yaml: drop the oauth2-proxy component (no longer in
  the gating tier).
- platform/oauth2-proxy/kustomization.yaml: pull in httproute + servicemonitor
  so the full component travels together.
- platform-config/kustomization.yaml: drop the in-place oauth2-proxy
  httproute/servicemonitor refs (moved with the controller).
- clusters/dev/kustomization.yaml: add the SOPS decryption patch for the new
  oauth2-proxy Kustomization (it now owns the encrypted credentials Secret that
  the `platform` Kustomization used to decrypt).

Now `platform` reaches Ready without oauth2-proxy; platform-config brings up
Keycloak/TLS/DNS; then oauth2-proxy reconciles against dependencies that
already exist. wait:false guarantees its health can never gate anything above
it. No config of oauth2-proxy itself changed — only its reconciliation tier.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 18:12:43 -05:00
jakman85andClaude Opus 4.8 fdf5c69adc test(conformance): add L3 node-loss test for R-STATE-02 (Longhorn node-loss resilience)
R-STATE-02 was the last safety-critical requirement with no verifying test,
exempted via DOCUMENTED_GAPS. Author the L3 fault-injection test that proves the
guarantee and remove the exemption.

tests/l3-fault-injection/node-loss/chainsaw-test.yaml:
  GIVEN a 3-replica Longhorn volume with written data attached on node N,
  WHEN  N is removed from service (cordon + evict the attached workload, forcing
        the volume to detach from N),
  THEN  a fresh workload on a SURVIVING node re-attaches the same volume and reads
        the canary intact — volume stays available, cluster reconverges.

Safe-by-construction for the 3-node embedded-etcd dev cluster:
  - Disruption scoped to EXACTLY ONE node, at the scheduling layer only (cordon +
    evict our own pod). kubelet/etcd stay up — etcd quorum is never touched.
  - Never `kubectl drain`; only our own writer pod is deleted, so no unrelated or
    system pod is disturbed.
  - Victim node labelled conformance.home.k3s/cordoned-by=<test-ns>; `finally`
    ALWAYS uncordons every node with THAT label and removes it — restored even on
    failure/timeout, and never touches an operator-cordoned node.
  - Self-skips green until a longhorn StorageClass AND >= 3 Ready nodes exist.

Bookkeeping:
  - run.sh: remove R-STATE-02 from DOCUMENTED_GAPS; add node-loss to CONF_HEAVY_TESTS
    (serial pass, matches the other Longhorn/KubeVirt L3 tests).
  - incluster/rbac.yaml: enable the scoped nodes get/list/patch rule the test needs
    (cordon/uncordon/label); no eviction rights (no drain).
  - README.md / coverage.py docstring: refresh the documented-gap example.

Coverage: 36/51 -> 37/51 (70.6% -> 72.5%); safety-critical gaps 1 -> 0. Only
R-STATE-02 changed ([] -> node-loss); no other requirement or golden touched.

NOTE: LIVE execution is DEFERRED to an operator-chosen maintenance window. This
change AUTHORS + statically validates the test (renders, chainsaw-lints, harness
selects it, coverage updates). The static gate runs L2/L3 as SKIP without a
cluster (expected). The node-disruption test was NOT run live.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 18:12:33 -05:00
jakman85andClaude Opus 4.8 d34c639269 fix(oidc): make gitea + nextcloud OIDC login-source registration persist
The Keycloak OIDC postStart registration for gitea and nextcloud ran but
never persisted, leaving `gitea admin auth list` empty and nextcloud's
user_oidc with "No providers configured" despite the apps installing/
enabling and the client secret being present.

Two distinct root causes, confirmed by running the registration manually
against the live warm pods:

- nextcloud: a QUOTING BUG, not a race. The OCC() wrapper runs
  `su -c "php occ $*"`, which word-split `--scope="openid email profile"`
  back into three tokens, so occ saw `email profile` as extra positional
  args -> "Too many arguments to user_oidc:provider" -> exit 1, swallowed
  by `|| echo failed`. Failed on EVERY boot. Fixed by running the provider
  call as ONE `su -c` with the scope single-quoted so www-data's shell
  keeps it a single argument; keep the OCC() wrapper only for single-token
  subcommands.
- gitea: a first-boot RACE. The direct `admin auth add-oauth` persists once
  the app/network is warm (verified live), but fired too early at postStart
  and its failure was swallowed (`|| echo ... >&2`, exit 0).

Hardening applied to both catalog sources (rendered into apps/wave-*):
- wait (bounded ~90s) for the Keycloak discovery doc to return HTTP 200;
- nextcloud also waits for `installed: true` AND `maintenance: false`;
- register idempotently (create-or-update) then VERIFY the source/provider
  actually persisted, retrying the whole cycle (guards exit-0-but-nothing-
  stuck);
- on give-up, log LOUDLY to stderr AND a marker file on the persistent
  volume (/data, /var/www/html) instead of swallowing — next failure is
  visible;
- never crashloop: missing-secret guard retained, break-glass local admin
  stays usable.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 17:41:54 -05:00
jakman85andClaude Opus 4.8 2919a1d72e chore: gitignore .claude/ (agent worktrees pollute lint + git add -A)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 17:19:42 -05:00
jakman85 76ba88a51e Merge feature/oidc-sso-apps: native Keycloak OIDC SSO for gitea, nextcloud, semaphore 2026-08-27 17:19:01 -05:00
jakman85andClaude Opus 4.8 f163026954 feat(nextcloud,semaphore): wire native Keycloak OIDC SSO (break-glass admins retained)
Extends the SSO work to the two remaining `auth: oidc` apps still on local-admin
logins, each as a native confidential client of the Keycloak `homelab` realm with
its local admin kept as break-glass (docs/15 availability caveat; R-IDP-02/06).

Realm (source of truth): add the `nextcloud` and `semaphore` confidential clients
to platform/keycloak/keycloak-realm-import.yaml (modeled on grafana/gitea) + their
NEXTCLOUD_CLIENT_SECRET / SEMAPHORE_CLIENT_SECRET placeholders. As before, a
KeycloakRealmImport no-ops on an already-existing realm, so both live clients were
ALSO created via kcadm (nextcloud e95049ed-ca10-4085-97d9-4bbef495fe61, semaphore
1835bfe9-b125-4026-8f69-33557519cf4a); the YAML keeps git authoritative.

Secrets (SOPS/age): add nextcloud-client-secret + semaphore-client-secret to
keycloak-realm-clients.sops.yaml (keycloak ns). Mirror the nextcloud secret as
oidc-client-secret into nextcloud-bootstrap (nextcloud ns). For semaphore, the
whole SEMAPHORE_OIDC_PROVIDERS JSON (which embeds the client secret) is held
encrypted as oidc-providers-json in semaphore-bootstrap (semaphore ns). All values
verified == the live clients; existing keys untouched; no plaintext in git.

App config (catalog source + re-render):
- nextcloud: add NEXTCLOUD_OIDC_CLIENT_SECRET env + an idempotent postStart hook
  that installs/enables user_oidc and runs `occ user_oidc:provider keycloak`
  (create-or-update; secret via --clientsecret-env so it never hits argv; occ as
  www-data; group provisioning on the `groups` claim). Syntax verified against the
  running user_oidc 8.11.0 on Nextcloud 31.0.14; su-to-www-data env passthrough
  verified. Missing-secret/not-installed guards prevent crashloops.
- semaphore: add SEMAPHORE_OIDC_PROVIDERS env sourced from the encrypted
  oidc-providers-json (provider_url=realm issuer/auto-discovery, redirect
  /api/auth/oidc/keycloak/redirect). Mechanism verified against semaphore v2.19.10
  source (util ConfigType oidc_providers env tag + OidcProvider struct + router
  redirect path); no kubelet service-link env collision.
apps/wave-1/{nextcloud,semaphore}.yaml re-rendered via
`render.py --no-image-automation` (diffs are env/lifecycle-only).

Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run; live
homelab clients now include gitea/nextcloud/semaphore; static gate (lint/l0/l1)
green. Browser SSO login for each still needs a human check.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 17:17:13 -05:00
jakman85andClaude Opus 4.8 87388712f0 feat(gitea): wire native Keycloak OIDC SSO (break-glass admin retained)
Wires the bootstrapping-critical Gitea git host to the Keycloak `homelab` realm
as a native confidential OIDC client, moving it off the local-admin-only login
while keeping the local admin as the break-glass path (docs/15 availability
caveat; R-IDP-02/06).

Realm (source of truth): add the `gitea` confidential client to
platform/keycloak/keycloak-realm-import.yaml (modeled on grafana) + its
GITEA_CLIENT_SECRET placeholder sourced from the keycloak-realm-clients Secret.
NOTE: a KeycloakRealmImport no-ops on an already-existing realm, so the live
client was ALSO created via kcadm (id 70a6ca60-317a-42ce-8331-a60e380838ee);
the YAML keeps git authoritative for a rebuild.

Secrets (SOPS/age, never plaintext): add gitea-client-secret to
platform/keycloak/keycloak-realm-clients.sops.yaml (keycloak ns) and mirror the
same value as oidc-client-secret into apps/wave-0/gitea-bootstrap-secret.sops.yaml
(gitea ns) so the app can read it in-namespace. Both == the live client secret.

App config (catalog source + re-render): catalog/gitea.yaml gains the
GITEA_OIDC_CLIENT_SECRET env (from the mirrored Secret), [oauth2_client]
auto-registration/account-linking app.ini env, and an idempotent postStart step
that registers the "keycloak" OAuth2 login source (auto-discovery; groups claim;
platform-admins -> Gitea admin). apps/wave-0/gitea.yaml re-rendered via
`render.py --no-image-automation`.

Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run;
static gate (lint/l0/l1) green. Browser SSO login still needs a human check.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 17:01:46 -05:00
jakman85andClaude Opus 4.8 7b8777c1e1 fix(seerr): add fsGroup podSecurityContext to resolve non-root PVC CrashLoop
Seerr v3.4.1 (ghcr.io/seerr-team/seerr) runs as the non-root user node:node
(UID/GID 1000), unlike the old root-running sctx/overseerr image. Its root-owned
Longhorn RWO config PVC at /app/config was not writable by UID 1000, so the pod
CrashLoopBackOff'd with:

  Error: EACCES: permission denied, mkdir '/app/config/logs/'

Add a podSecurityContext with fsGroup=1000 (plus runAsUser/runAsGroup/runAsNonRoot)
to the seerr ServiceDefinition — the same schema 1.3.0 mechanism semaphore uses.
Longhorn's CSIDriver fsGroupPolicy: ReadWriteOnceWithFSType applies fsGroup to this
RWO filesystem volume, group-owning the PVC to GID 1000 so the process can write.
Re-rendered apps/wave-2/seerr.yaml from the catalog source.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 16:58:17 -05:00
jakman85andClaude Opus 4.8 df947b33d0 infra(tofu): fix backend.tf HCL header + document encryption-adoption migration
backend.tf began with a stray YAML `---` doc separator (invalid HCL) that broke
`tofu` parsing. Remove it. Also document the one-time `unencrypted` fallback method
needed to adopt encryption on pre-existing plaintext state (add for the migrate,
remove after), so the record is complete. State migration to R2 verified: 4/4
resources read back through the encrypt-only method (proves ciphertext at rest).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 16:31:23 -05:00
jakman85 7b16621585 Merge feature/tofu-remote-state-r2: off-cluster R2 encrypted tofu state scaffolding 2026-08-27 16:20:26 -05:00
jakman85andClaude Opus 4.8 8b0dcbf33b fix(seerr): move to maintained Seerr image and correct config mountPath
Two operator-flagged defects in the seerr ServiceDefinition, fixed at source
(catalog/seerr.yaml) and re-rendered into apps/wave-2/seerr.yaml:

- Image: sctx/overseerr:latest (original Overseerr, unmaintained) ->
  ghcr.io/seerr-team/seerr:v3.4.1. Overseerr and Jellyseerr have merged into
  the actively maintained Seerr project (seerr-team/seerr, docs.seerr.dev);
  ghcr.io/seerr-team/seerr is the officially recommended image. Pinned to the
  current stable tag v3.4.1 per the arr-stack convention (sonarr/radarr/
  qbittorrent/gitea pin explicit tags; R-UPD-01). update.range bumped
  ">=1.33.0" -> ">=3.4.0" to track the Seerr 3.x line. Plex-native (native
  auth) support is retained.

- Config mountPath: /data/config -> /app/config. Overseerr/Jellyseerr/Seerr
  store settings + request/cache DB at /app/config; the previous /data/config
  left the config PVC effectively unused (config not persisted across
  restarts). Set via the schema-supported storage[].mountPath field.

Dependencies (sonarr:8989, radarr:7878, plex:32400), NetworkPolicies,
public exposure, and native auth are unchanged. Render == golden verified;
lint, schema, and L1 render checks green.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 16:15:00 -05:00
jakman85andClaude Opus 4.8 485b9e3888 infra(tofu): scaffold off-cluster R2 remote state (encrypted)
Local-only tfstate meant losing the workstation would strand a rebuild (apply
would try to recreate the 3 live nodes). Move state to Cloudflare R2, which sits
OUTSIDE the Proxmox->VM->cluster->Gitea chain, so a full cluster rebuild can still
read it; encrypted at rest via OpenTofu native AES-GCM.

Adds a partial s3 backend (R2 skip-flags), gitignored backend.hcl/.r2.env for the
account-specific + secret bits, their .example templates, and a migration+DR runbook.
Locking (use_lockfile) deferred to an OpenTofu >=1.10 bump (1.9.0 lacks it).
Migration (tofu init -migrate-state) runs once the operator provisions the R2 bucket.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-27 15:57:12 -05:00
jakman85 91c82d2756 Merge docs/reconcile-with-reality: docs match reality; merge gate green again
Reconciles the documentation with the deployed platform (the root README still
claimed Phase 0 / no cluster; the infra docs described a retired libvirt cluster;
DECISIONS-LOG.md had gone ~85 commits stale), and fixes the two independent
defects that had left the merge gate BLOCKED on main:

  - yamllint commas errors in the hand-authored apps/vms/unifi manifests
    (whitespace only; kustomize output byte-identical);
  - ci/touched-requirements.sh scanning whole file contents instead of the diff,
    which mis-attributed requirements to changes that never touched them and made
    appending to the decision log fail CI.

Gate verified in the devcontainer: spec-immutability / lint / l0 / l1 green,
MERGE GATE: PASS.
2026-08-27 00:12:02 -05:00
jakman85andClaude Opus 5 9e8b694ba9 docs(conformance): record that the matcher debt is half paid
gate / gate (push) Has been cancelled
gate / conformance (push) Has been cancelled
The content-scan half of the 'stop keying on doc-header tokens' follow-up is done
(ci/touched-requirements.sh). The path-map half remains: clusters/* and ci/* are
still keyed wholesale to R-ENV-02 + R-GITOPS-01, which is why those two stay in
DOCUMENTED_GAPS. Notes the accepted consequence for header-declared manifests.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-27 00:08:16 -05:00
jakman85andClaude Opus 5 f7d3884ad2 fix(ci): scope the requirement matcher to the diff and to non-doc files
`ci/touched-requirements.sh` documents its detection as "R-* tokens literally
present in the diff of changed files". It did not do that — it grepped the ENTIRE
CONTENT of every changed file. So a change inherited every requirement those files
had ever mentioned in prose, and `--require-all` then blocked the merge on
requirements the author never went near.

On this branch that meant editing the decision log and the tofu docs dragged in
five unrelated requirements, all on untouched lines, all legitimately uncovered:

  [coverage] REQUIRE: FAILED — required requirement(s) uncovered: [5 IDs]

The practical effect was that the PM could not append to the decision log without
failing CI — which is a fair part of why the log went stale in the first place.
This is the defect already logged as "FOLLOW-UP (WS-TEST): fix the matcher so real
requirements need not be exempted".

Two scoping rules, both learned from the gate blocking merges it should not have:

  1. Scan the ADDED/REMOVED lines of the diff, as the header always claimed.
     Removed lines count too — deleting a test's requirement tag, or a policy's
     self-declared ID, touches that requirement just as much as adding one.

  2. Skip `*.md`. An ID in prose is a REFERENCE, not a declaration: explaining a
     requirement is not changing behaviour against it. Rule 1 alone is not enough —
     a commit that documents a mis-attribution names the IDs involved and thereby
     re-triggers the very mis-attribution it fixes, and the decision log can never
     record which requirements a change concerned.

This TIGHTENS attribution rather than weakening the gate:
  - no requirement is exempted, and DOCUMENTED_GAPS is untouched;
  - authoritative declarations (`conformance.home.k3s/R-*` tags, Kyverno policies,
    ServiceDefinitions) all live in YAML, on lines a real change must touch;
  - the coarse path->requirement fallback is unchanged and file-type agnostic, so
    the doc subtrees that matter (conformance/, schemas/, catalog/, clusters/, ci/)
    stay covered either way.

Alternatives rejected: adding the five IDs to DOCUMENTED_GAPS genuinely weakens the
gate; scrubbing requirement IDs out of prose (precedent f9e9640) would mean editing
the decision log's own history to appease a matcher.

Gate semantics change — flagged in DECISIONS-LOG.md for review.

Also drops three R-TEST-06 mentions this branch had added to prose and refreshes
the now-stale "gate is RED" notes.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-27 00:06:28 -05:00
jakman85andClaude Opus 5 12c40d8c9c docs: reconcile the documentation with the deployed reality
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.

Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.

README.md
  - Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
  - Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
  - Real conformance number (36/51, 70.6%) and how to regenerate it.
  - Devcontainer workflow; note that the gate was red and why.

infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
  cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
  paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
  the toolchain, off-repo state that must not be lost, and the fact that Flux's
  git source is an app inside the cluster (`git push origin main` IS the deploy).

infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
  rationale are still worth keeping; the libvirt execution steps are not.

infra/README.md — Proxmox, plus a map of the subtree and where the cluster
  actually lives (it is not on this workstation).

infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
  is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
  talks to an HTTP API. Everything runs in the container now. Documents the full
  CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
  and the read-only spec repo especially — without the latter, spec-immutability
  silently skips instead of verifying).

infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
  is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
  against the running nodes), pfSense NAT marked done, and a warning that
  terraform.tfstate is absent on a fresh workstation — an apply with empty state
  would try to recreate the three live nodes.

infra/tofu/variables.tf — section comment disagreed with its own defaults.

infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
  domains that do not exist; behaviour unchanged, but it no longer lies.

apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
  since been dropped. Now: live status, 40Gi, resolved open questions marked
  resolved, and the two constraints the next person will otherwise rediscover —
  KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
  postBuild.substituteFrom (envsubst corrupts the cloud-init).

DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
  migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
  the Flux source cutover, CI going live in-cluster, ten app onboardings and the
  renderer bugs they exposed, the wave split (verified: DBs adopted, not
  recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
  VM, and kubectl SSO. Plus verified live state and the carried-forward open list.

Two corrections to the record while verifying:
  - kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
    [oidc:platform-admins]), so the rolling restart 3472f1d deferred has
    happened.
  - UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
    24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
    and the CNP explicitly permits cluster -> :11443, so it is genuinely not
    listening rather than a policy artifact.

Docs and comments only; no manifest semantics change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-27 00:00:05 -05:00
jakman85andClaude Opus 5 4cf9572c58 fix(vms): normalize comma spacing in unifi manifests — unblock the lint gate
The hand-authored UniFi VM manifests used column-aligned inline flow mappings
(`{ name: inform,   port: 8080,  protocol: TCP }`), which yamllint's `commas`
rule rejects as "too many spaces after comma". That made `make -C ci lint` FAIL,
so the merge gate has been BLOCKED on `main` since the UniFi commits landed —
every other static stage was green.

Collapse each run of spaces after a comma to one, and re-align the trailing
comments so the port tables stay readable. Whitespace only:
`kubectl kustomize apps/vms` renders byte-for-byte identically before and after.

The lint config is left untouched — relaxing the rule to make failing files pass
is moving the goalposts, not fixing the code.

Also refresh these files' stale `STAGING / DRAFT` headers: the tree is live,
reconciled by the non-gating `apps-vms` Flux Kustomization.

Gate after this change: spec-immutability / lint / l0 / l1 all green,
MERGE GATE: PASS.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-26 23:59:02 -05:00
jakman85andClaude Opus 4.8 3472f1d161 feat(auth): Keycloak OIDC login + RBAC for kubectl (cert kubeconfig stays break-glass)
Lets users kubectl into the cluster via Keycloak SSO, mapping realm groups to
cluster RBAC. The cert-based admin kubeconfig is untouched and remains a fully
independent break-glass path (works even if Keycloak/OIDC is down).

- keycloak-realm-import.yaml: new `kubernetes` PUBLIC client + PKCE (S256) — no
  client secret in any kubeconfig; kubelogin runs auth-code+PKCE to localhost.
  groups mapper + email/profile/roles scopes.
- k8s-oidc-rbac.yaml (NEW, platform-config): ClusterRoleBindings
  oidc:platform-admins -> cluster-admin, oidc:viewers -> view.
- k3s_server role: kube-apiserver-arg OIDC flags (issuer=homelab realm,
  client-id=kubernetes, username-claim=email + oidc: prefix, groups-claim=groups
  + oidc: prefix). No oidc-ca-file (nodes reach the issuer via the gateway VIP
  and trust the public LE cert). Applied via the rolling Restart k3s handler.

Keycloak/RBAC land via Flux now; the API-server flags take effect on the next
`ansible-playbook playbooks/k3s.yml` rolling restart (deferred until cluster load
from the VM-import incident settles).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 17:37:05 -05:00
jakman85andClaude Opus 4.8 ab592028f6 fix(vms): actually apply inline userData + drop secretGenerator (amend 760084e)
760084e only committed the unifi-cloudinit.yaml deletion — the `git add` errored
on that already-staged pathspec and skipped the other two files, leaving the
kustomization still referencing the deleted file (broken build). Commit the
intended changes: 30-virtualmachine.yaml inline userData + kustomization.yaml
without the secretGenerator.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 16:59:03 -05:00
jakman85andClaude Opus 4.8 760084ea8c fix(vms): revert UniFi cloud-init to minimal inline userData (1112B)
userDataSecretRef tripped a Flux server-side-apply schema quirk on the KubeVirt
VirtualMachine CRD ("field not declared in schema"). Go back to inline userData,
trimmed to 1112 bytes (< the 2048 limit) by dropping the break-glass SSH key —
no longer needed for debugging now that the postBuild cloud-init corruption is
fixed. Keeps qemu-guest-agent (AgentConnected proves cloud-init ran) and the
public-API UOS resolver + non-interactive install. Removes the secretGenerator +
unifi-cloudinit.yaml. (Add SSH back via userDataSecretRef only if the SSA quirk is
worked around and guest access is needed.)

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 16:42:04 -05:00
jakman85andClaude Opus 4.8 e99e3d1666 fix(vms): deliver UniFi cloud-init via Secret (userDataSecretRef) — 2048B limit
Adding the SSH key pushed the inline cloudInitNoCloud.userData over KubeVirt's
2048-byte limit ("userdata exceeds 2048 byte limit. Should use UserDataSecretRef").
Move the cloud-init into apps/vms/unifi/unifi-cloudinit.yaml and wrap it as the
`unifi-cloudinit` Secret via a Kustomize secretGenerator (disableNameSuffixHash
so the VM can reference it by fixed name); VM now uses userDataSecretRef. Content
is non-sensitive (public key + public installer script), so not SOPS-encrypted.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 16:34:55 -05:00
jakman85andClaude Opus 4.8 da1f833de7 fix(vms): drop postBuild substituteFrom on apps-vms — it corrupted VM cloud-init
Flux's postBuild.substituteFrom runs envsubst over the WHOLE manifest, including
the VM cloud-init userData. It was substituting every shell `${...}` in the UOS
install script (${UOS_URL}, the API-resolver ${...}) to empty — which is why the
first UOS install silently did nothing — and ${PIPESTATUS[0]} then broke the
build entirely ("missing closing brace"), blocking apps-vms reconcile.

VM manifests use no cluster-vars, so substitution is unneeded. Removed it; the
cloud-init shell now reaches the guest intact. (A future VM needing a cluster-var
must escape its shell `${...}` as `$${...}`.)

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 16:26:27 -05:00
jakman85andClaude Opus 4.8 d5f8212415 feat(vms): add SSH key + qemu-guest-agent to UniFi VM for ops/debugging
The first UOS bring-up left no way into the guest (cloud-init set no
credentials), so the in-guest UOS install couldn't be observed. Add:
- ssh_authorized_keys (public key, safe to commit) on the default ubuntu user
- qemu-guest-agent (KubeVirt AgentConnected + virtctl guest access)
- tee the install-uos output to the serial console + /var/log/uos-install.log,
  and record the installer exit code, so the install is observable.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 16:17:11 -05:00
jakman85andClaude Opus 4.8 7a3da96d9f fix(cdi): single-replica scratch class + 40Gi UniFi disk (fix insufficient-storage import)
The UniFi Block import got past the device-permission fix but then failed
ReplicaSchedulingFailure / "insufficient storage": CDI sizes scratch ~= the disk
virtual size and, on the default 3-replica `longhorn` class, the ~60Gi scratch x3
(189G) on top of the disk's x3 (180G) overflowed the Longhorn tier (~125G/node).

- platform/longhorn/storageclasses.yaml: NEW `longhorn-scratch` StorageClass
  (numberOfReplicas: 1) — CDI scratch is ephemeral/throwaway, replicating it 3x is
  waste; if a node dies mid-import CDI just retries.
- platform/kubevirt/cdi-cr.yaml: spec.config.scratchSpaceStorageClass ->
  longhorn-scratch.
- apps/vms/unifi/20-datavolume.yaml: boot disk 60Gi -> 40Gi (UOS min is 25Gi;
  40Gi is comfortable for a home site and lighter on the replicated tier).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 15:47:02 -05:00
jakman85andClaude Opus 4.8 d8101bca38 chore(ansible): set private_key_file for k3s node connectivity
The k3s nodes authorize ~/.ssh/home_network for the `k3s` user, but ansible.cfg
set no private_key_file and there's no ssh_config entry for the SSH-NAT host, so
node-config plays (e.g. the rolling k3s config.yaml + Restart handler) failed
with Permission denied (publickey). Pin the key so those plays connect without a
per-run --private-key.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 15:27:29 -05:00
jakman85andClaude Opus 4.8 df5979fceb fix(k3s): enable nonroot-devices so CDI can import to Block-mode PVCs
The UniFi VM boot-disk DataVolume (RWX + Block on longhorn-vm) failed to import:
CDI's non-root importer (UID 107, cap-drop ALL) hit
`blockdev: cannot open /dev/cdi-block-volume: Permission denied`.

Kubernetes only applies fsGroup ownership to *filesystem* volumes — it never
chowns a raw block device — and Longhorn's CSIDriver (fsGroupPolicy:
ReadWriteOnceWithFSType) skips fsGroup on RWX/Block volumes regardless, so the
device stays root:root. The CDI-documented fix is the container-runtime setting
`device_ownership_from_security_context`, exposed on k3s (>= v1.31.3+k3s1, this
cluster is v1.31.5) as `nonroot-devices: true`. It makes containerd set the
in-container device node's owner from the pod's securityContext.

Added to the Ansible-managed /etc/rancher/k3s/config.yaml alongside tls-san;
applied via the existing rolling `Restart k3s` handler (serial:1, kube-vip keeps
the API up, etcd holds 2/3). Node-scoped, security-neutral (pods stay non-root),
and preserves the longhorn-vm RWX/Block/migratable contract unchanged.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 15:24:07 -05:00
jakman85andClaude Opus 4.8 317b3f02f1 feat(vms): onboard UniFi OS Server as a KubeVirt VM on a non-gating track
First KubeVirt VM app + first real CDI http import. UniFi OS Server (UOS) is
Ubiquiti's current self-host product (the .deb is deprecated); it bundles UniFi
OS + Network app + its own datastore, so no MongoDB companion.

- apps/vms/unifi: CDI DataVolume (Ubuntu 24.04 cloud image -> longhorn-vm
  RWX/Block, live-migratable) + VirtualMachine (evictionStrategy: LiveMigrate)
  with cloud-init that installs podman/slirp4netns/jq and runs the UOS installer
  NON-INTERACTIVELY. The installer URL is resolved at boot from Ubiquiti's public
  firmware API (fw-update.ui.com/api/firmware-latest) — always-latest, no pinned
  URL to maintain (currently UOS v5.1.37).
- L4 exposure: two LoadBalancer Services (TCP+UDP) sharing a pinned, L2-announced
  Cilium VIP 192.168.1.54 (device inform/adoption + STUN can't ride the L7
  Gateway); external-dns publishes unifi.home.keithjackson.dev. UI on :11443.
- default-deny + additive CNPs (controller ingress, egress, and CDI-importer
  egress for the image pull).
- clusters/base/apps-vms.yaml: NON-GATING Flux Kustomization (wait:false,
  dependsOn platform-config) — VMs are a separate lifecycle from the app waves,
  so a slow VM import/install can't stall the app tier.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 15:05:05 -05:00
jakman85andClaude Opus 4.8 36a1a4e1f9 refactor(apps): split monolithic apps Kustomization into dependency waves
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.

- catalog/render.py: compute waves by longest dependency-path depth over the
  ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
  consumer lands strictly higher — the "DB below consumer" rule falls out of
  longest-path, no special case). `--emit-waves` generates the per-wave
  kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
  wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
  wave 1: sonarr radarr nextcloud semaphore gitea-actions
  wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
  chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.

Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that
object here does not GC its resources; the wave Kustomizations SSA-adopt the
byte-identical manifests in place (no restart, no PVC churn).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 12:19:00 -05:00
jakman85andClaude Opus 4.8 1bfb0db6f1 chore(flux): cutover step 1 — disable prune on apps before wave split
Temporary: with prune:false, removing the `apps` Kustomization object in the
follow-up wave-split commit will NOT garbage-collect its owned resources
(CNPG Clusters -> PVCs). The apps-wave-* Kustomizations adopt them in place.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 12:14:59 -05:00
jakman85andClaude Opus 4.8 3586769460 feat(storage): CSI snapshots + smart-clone + migratable RWX for KubeVirt live-migration
Adds the VM/CDI storage tier approved by the PM (RWX for live-migration):

- platform/external-snapshotter (NEW, controllers pass): the cluster-singleton
  snapshot.storage.k8s.io control plane (CRDs + snapshot-controller in
  kube-system), pinned v8.2.0 (Kubernetes deps aligned to 1.31 since v8.1.0;
  this cluster is k3s v1.31.5). These CRDs were absent — only snapshots.longhorn.io
  and kubevirt.io snapshot CRDs existed.
- platform/longhorn/volumesnapshotclass.yaml (NEW, platform-config): the
  driver.longhorn.io VolumeSnapshotClass (type: snap, in-cluster).
- platform/kubevirt/cdi-storageprofiles.yaml (NEW, platform-config): StorageProfile
  longhorn -> cloneStrategy: snapshot; StorageProfile longhorn-vm -> snapshot clone
  + claimPropertySets RWX/Block. Non-degraded default virt class clears
  CDIDefaultStorageClassDegraded.
- platform/longhorn/storageclasses.yaml: NEW longhorn-vm StorageClass —
  migratable: "true" (Longhorn's KubeVirt-live-migration knob; RWX/Block-only on
  1.7.3), is-default-virt-class (NOT the k8s default; longhorn stays RWO default).
- platform/kubevirt/kubevirt-cr.yaml: comment only — LiveMigration is GA on v1.9
  (no feature gate); no cluster-wide evictionStrategy (would wedge RWO-disk drains);
  live-migration stays per-VM opt-in on a longhorn-vm disk.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 12:06:15 -05:00
jakman85andClaude Opus 4.8 b57b3e9d71 fix(kubevirt): correct KubeVirtControlPlaneDown expr — wrong metric name + job label
The alert fired continuously for ~21h while the control plane was fully
healthy (both virt-controller and virt-api pods 1/1 Ready, KubeVirt CR
Deployed/Available=True). Two bugs, each making it fire regardless of state:

  1. `kubevirt_virt_controller_ready` does not exist — the real metric is
     `kubevirt_virt_controller_ready_status` (KubeVirt suffixes `_status`),
     so that clause was always ABSENT (never matched).
  2. `absent(up{job="kubevirt"} == 1)` — the KubeVirt scrape job label is
     `kubevirt-prometheus-metrics` (the Service name), not `kubevirt`, so
     `up{job="kubevirt"}` matched nothing and absent() was permanently 1.

Fix: use `sum(kubevirt_virt_{controller,api}_ready_status) == 0` (fires only
when every replica is unready) and the correct job label. virt-api added to
match the alert's stated scope (virt-controller/virt-api).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 10:45:23 -05:00
jakman85andClaude Opus 4.8 6a7e9d15e7 fix(kubevirt): drop broken http CDI ServiceMonitor causing false CDIOperatorDown
The cdi-operator manages its own `service-monitor-cdi` (scheme https +
cdi-metrics-reader-token) which backs CDI's shipped recording rules
(kubevirt_cdi_operator_up / cluster:kubevirt_cdi_operator_up:sum) and the
CDIOperatorDown alert. Our extra `cdi` ServiceMonitor scraped the same
cdi-prometheus-metrics service (HTTPS :8443) over http, so every scrape
failed with 400 Bad Request, adding down (up=0) targets for the CDI pods.
That poisoned the up-aggregation the recording rule sums, driving it to 0
and firing a false CDIOperatorDown even though the operator was Running,
Ready, and Available=True. Remove the duplicate; keep the kubevirt SM
(virt-operator ships none of its own).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 10:20:48 -05:00
jakman85andClaude Opus 4.8 948c390272 conformance: serialize heavy L2/L3 tests + author alerting SMTP sink
TASK 1 — reliably single-pass green L2/L3:
Split run_chainsaw into a LIGHT pass (concurrent, config parallelism) and a
HEAVY pass (serial, parallel=1). The KubeVirt-CDI/Longhorn/DR tests
(vm-lifecycle, vm-data-survival, vm-default-deny, data-survival, dr-restore)
spike embedded-etcd on the small dev VMs; running them serially in their own
pass keeps their etcd spikes from ever coinciding with another test's setup —
the coincidence that let one heavy test's apiserver blip fail-close Kyverno
admission and abort every in-flight test at 0.01s. Heavy set is keyed by test
dir basename and overridable via CONF_HEAVY_TESTS. Each pass writes its own
JUnit; lib/merge_junit.py stitches them into the one stable report.junit.xml
CI collects, so the split is invisible downstream. Overall rc is the union of
both passes (no masking — real failures still surface).

Proof: full --layer l2,l3 run now completes single-pass green: 24/24 passed,
0 failures/errors/skipped. vpn-killswitch, which previously failed with an
apiserver EOF (etcd blip), passed; no mass environmental setup-aborts.

TASK 2 (T-4.1) — author conformance/tests/l3-fault-injection/alerting/
fixtures/smtp-sink.yaml: a hermetic Mailpit SMTP capture sink (Service
smtp-sink, SMTP 1025, HTTP API 8025) the alerting test applies as its mail
sink. restricted-PodSecurity compliant, in-memory (no PVC). Unblocks the
previously-aborting alerting scaffold; it now passes.

Touches conformance/ + ci-read content only — no apps/ or platform/ workload.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 02:56:56 -05:00
jakman85andClaude Opus 4.8 a02c0e2d1a apps: re-enable semaphore — both crashloop root causes fixed
Restores the four semaphore resources a peer temp-disabled in 0abadd3 to unblock
the apps Kustomization while the crashloop was diagnosed. Both root causes are now
fixed and verified against the live image:
  1. permission-denied writing /etc/semaphore/config.json (non-root UID 1001 vs
     root-owned Longhorn PVC) -> podSecurityContext.fsGroup=1001 (schema 1.3.0);
     verified the pod got past config write and successfully pinged its PostgreSQL.
  2. config panic on SEMAPHORE_PORT=tcp://<svcIP>:3000 (kubelet Docker-link
     injection for the same-named Service colliding with Semaphore's listen port)
     -> explicit SEMAPHORE_PORT=":3000" env shadowing the service-link var.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 02:12:13 -05:00
jakman85andClaude Opus 4.8 41d696e9fe fix(apps): semaphore SEMAPHORE_PORT — shadow kubelet service-link injection
With the config PVC now writable (fsGroup), semaphore got past config write and
pinged its PostgreSQL successfully, then crashed on:
  panic: value of field 'Port' is not valid: tcp://<svcIP>:3000

Because the Service is named `semaphore`, the kubelet injects the Docker-link-style
SEMAPHORE_PORT=tcp://<clusterIP>:3000, which collides with Semaphore's OWN
SEMAPHORE_PORT (its HTTP listen port). Add an explicit SEMAPHORE_PORT=":3000" env
entry — user-defined env takes precedence over service-link vars — pinning the
listen port back to :3000 (matches containerPort/Service). Uses the existing env
field; only apps/semaphore.yaml changes, all others byte-identical.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 02:09:22 -05:00
jakman85andClaude Opus 4.8 0abadd3131 apps: temp-disable semaphore (interactive-setup crashloop blocked apps reconcile)
The semaphoreui/semaphore image falls through to its interactive setup wizard
and panics on /etc/semaphore/config.json permission-denied; the env-only pattern
doesn't yield a non-interactive config. Backed out to unblock the apps
Kustomization; files retained for a proper config-model follow-up.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 02:04:26 -05:00
jakman85andClaude Opus 4.8 28498f1099 fix(apps): semaphore fsGroup — schema 1.3.0 podSecurityContext for non-root PVC write
The semaphoreui/semaphore image runs as non-root UID 1001 / GID 0 and writes its
generated config.json into the config dir on boot. A mounted Longhorn PVC is
root:root, so UID 1001 got "permission denied" and the pod crash-looped.

First-class the fix into the render contract (same mechanism the retired
gitea-bootstrap-patch was replaced by): schema 1.3.0 adds an optional pod-level
`podSecurityContext`, emitted verbatim onto spec.template.spec.securityContext by
render.py. Absent => every other app renders byte-identical (verified). semaphore
declares fsGroup=1001 (kubelet group-owns the PVC + adds the supplemental group so
UID 1001 can write) plus runAsUser/runAsGroup/runAsNonRoot pinning the image identity.

render.py selftest extended (verbatim render + absence). L0 (schema+kyverno 17/17),
L1 3/3, lint all green.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 02:00:45 -05:00
jakman85andClaude Opus 4.8 77496f9a8e feat(apps): onboard semaphore (Phase 6) — second CNPG-DB + OIDC-target app
Semaphore (Ansible UI, docs/13 row 10): internal exposure, internet egress,
config on Longhorn + external PostgreSQL, auth oidc. Reuses the CNPG-DB pattern
nextcloud established (apps/nextcloud-db.yaml, commit 4be6752).

- catalog/semaphore.yaml: ServiceDefinition (schema >=1.2.0 env) wiring the
  SEMAPHORE_DB_* connection to semaphore-db-rw.semaphore-db:5432, break-glass
  SEMAPHORE_ADMIN* local admin, and the FIXED base64 SEMAPHORE_ACCESS_KEY_ENCRYPTION.
- apps/semaphore.yaml: rendered (render.py --no-image-automation) — Deployment,
  config PVC, Service, default-deny + egress/gateway/internet CNPs, HTTPRoute
  semaphore.home.keithjackson.dev:3000, ServiceMonitor, SLI alert.
- apps/semaphore-db.yaml: CNPG Cluster in dedicated ns semaphore-db, initdb from
  the owner secret, inheritedMetadata app.kubernetes.io/name=semaphore-db so the
  dep-edge egress CNP matches.
- apps/semaphore-db-app.sops.yaml + apps/semaphore-bootstrap-secret.sops.yaml:
  SOPS(age) owner creds (basic-auth) + mirrored db-password, break-glass admin,
  and the persistent access-key encryption key.

OIDC (Keycloak realm client) is DEFERRED like gitea/nextcloud — comes up on the
break-glass local admin; realm-client wiring is the batched follow-up.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 01:47:22 -05:00
jakman85andClaude Opus 4.8 4be6752c71 feat(apps): onboard nextcloud (Phase 6) — first CNPG-DB + OIDC-target patterns
Onboard nextcloud (docs/13 row 7): public, DR-critical file sync backed by an
external PostgreSQL. Establishes the reusable CNPG-DB pattern for later DB-backed
services.

- catalog/nextcloud.yaml: ServiceDefinition (image nextcloud:31-apache :80, public,
  internet, data 20Gi longhorn @/var/www/html, backup daily-offcluster, auth oidc
  +platform-admins, ingress nextcloud.home.keithjackson.dev, dependency
  nextcloud-db:5432). Uses schema 1.2.0 env for the Postgres connection, break-glass
  admin, trusted-domain + OVERWRITE* proxy settings.
- apps/nextcloud-db.yaml: CloudNativePG Cluster in a dedicated nextcloud-db namespace
  (modeled on keycloak-db; no default-deny, mirroring the verified keycloak posture),
  inheritedMetadata stamps app.kubernetes.io/name=nextcloud-db so the rendered egress
  CNP matches. initdb consumes a pre-provisioned owner secret.
- apps/nextcloud-db-app.sops.yaml + apps/nextcloud-bootstrap-secret.sops.yaml: SOPS
  (age) — shared DB password mirrored across the two namespaces (Pods cannot read a
  Secret cross-namespace), plus the break-glass local admin.
- apps/nextcloud.yaml: rendered manifests (catalog/render.py --no-image-automation).
- apps/kustomization.yaml: wire in the four new resources.

OIDC realm-client wiring DEFERRED (WS-APPS step-3 escape hatch): editing the reconciled
realm import in platform-config risks the running Keycloak realm / whole platform-config
apply, and nextcloud OIDC needs post-install occ steps with no first-class env — nextcloud
comes up INSTALLED on the break-glass local admin (as Gitea did); the Keycloak client is a
follow-up. Off-cluster backup awaits Velero/OQ-BKP (intent declared).

Gates: render --selftest PASS; other apps byte-identical; ci l0 17/17; l1 3/3; lint clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 01:32:00 -05:00
jakman85andClaude Opus 4.8 c8dd4f3f9d feat(catalog): add SD env/envFrom/lifecycle; retire gitea bootstrap patch
Extend the ServiceDefinition contract with optional per-container `env`,
`envFrom`, and `lifecycle` fields (k8s EnvVar / EnvFromSource / Lifecycle
shapes) so headless-install / DB-backed apps configure themselves through
the SD instead of a supplementary kustomize patch.

- schema: bump 1.1.0 -> 1.2.0; add optional env/envFrom/lifecycle to the
  container kind (literal + secretKeyRef/configMapKeyRef; secretRef/
  configMapRef; postStart/preStop exec via $defs/execHandler). Existing
  fields unchanged; all fields optional -> absent renders byte-identical.
- render.py: emit env/envFrom/lifecycle verbatim onto the Deployment app
  container when present; --selftest coverage for the new fields.
- catalog/gitea.yaml: fold the headless-install env (INSTALL_LOCK, fixed
  SECRET_KEY via secretKeyRef, DOMAIN/ROOT_URL, actions, break-glass admin
  creds) + the idempotent postStart break-glass admin bootstrap inline.
- apps/gitea.yaml: re-rendered with env/lifecycle inline (effective pod
  spec provably equivalent to base+patch).
- DELETE apps/gitea-bootstrap-patch.yaml + drop it from apps/kustomization
  patches. gitea-bootstrap-secret.sops.yaml kept (still sourced by the env).

Proves the renderer feature by retiring the known TODO; unblocks env + DB
onboarding for nextcloud/semaphore/unifi/minecraft. All other rendered apps
byte-identical. Gates: selftest PASS, l0 17/17 + schema, l1 goldens, lint.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 01:13:52 -05:00
jakman85andClaude Opus 4.8 b871877c1b fix(render): namespace-pin east-west selectors so declared deps actually connect
Follow-up to the reciprocal-ingress fix: in-cluster verification (Hubble) revealed
the east-west path was STILL fully blocked — and that the pre-existing EGRESS half
was also broken, contrary to the original diagnosis. Root cause: a CiliumNetworkPolicy
`toEndpoints`/`fromEndpoints` selector is scoped to the POLICY's OWN namespace unless
it carries the reserved `k8s:io.kubernetes.pod.namespace` label. Every service lives
in its own namespace (== its name) and every dependency crosses a namespace boundary,
so an app-label-only peer selector matched nothing and Cilium silently denied the flow
(Hubble: sonarr->qbittorrent = "EGRESS DENIED ... Policy denied DROPPED").

Pin the peer namespace on BOTH halves — egress toEndpoints (destination ns == dep.to)
and the reciprocal ingress fromEndpoints (source ns == source name). This is a strict
TIGHTENING (more precise, still L3/L4), not a weakening. Verified end-to-end in-cluster:
sonarr->qbittorrent:8080, sonarr->jackett:9117, seerr->{sonarr:8989,radarr:7878,
plex:32400} now CONNECT (were DROPPED); undeclared sonarr->plex:32400 and
seerr->qbittorrent:8080 stay BLOCKED (default-deny holds).

Selftest + L0 (17/17) + L1 (3/3, goldens byte-identical) + lint all pass. NOTE: this
necessarily touches the egress toEndpoints selector, which the task scoped as
ingress-only; the premise that egress already worked was empirically false.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 01:01:56 -05:00
jakman85andClaude Opus 4.8 14beb85b65 fix(render): emit reciprocal ingress-allow for declared dependencies (R-NET-02)
A `dependencies` edge is declared ONCE on the SOURCE, but with the default-deny-
BOTH-directions baseline (docs/03 R-NET-01/02) it needs TWO matching halves: the
source egress-allow AND the destination ingress-allow. render.py emitted only the
egress half, so every declared east-west dependency was silently dropped by the
destination's default-deny ingress — verified in-cluster: sonarr->qbittorrent:8080,
sonarr->jackett:9117 and seerr->sonarr:8989 were all BLOCKED.

Add a cross-service reverse-dependency pass (build_reverse_map) and render the
reciprocal ingress half (r_reverse_dependency_policies): for each source that
declares a dependency on X, X gains an additive ingress-allow CiliumNetworkPolicy
`X-allow-<source>-<port>` whose fromEndpoints matches the source identity, mirroring
the egress half's L3/L4 granularity. Deterministic (sorted), additive (default-deny
ingress stays []), egress side unchanged. A lone render with no cross-catalog context
renders no reciprocal ingress, so the L1 render goldens are byte-identical.

New ingress CNPs: qbittorrent<-{sonarr,radarr}:8080, jackett<-{sonarr,radarr}:9117,
sonarr<-seerr:8989, radarr<-seerr:7878, plex<-seerr:32400. Selftest + L0 (17/17) +
L1 (3/3) + lint all pass.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 00:54:12 -05:00
jakman85andClaude Opus 4.8 b6f72eab0c feat(apps): onboard seerr (Overseerr request UI) — completes arr↔seerr↔plex graph
Phase 6: add the seerr ServiceDefinition and its rendered manifests. seerr is the
final missing node in the media dependency graph (docs/13 row 5; docs/03 allow rules).

- catalog/seerr.yaml: public | egress internet | config on longhorn (5Gi) |
  auth native (Plex accounts, documented SSO exception, docs/15) | 3 explicit egress
  deps: sonarr:8989, radarr:7878, plex:32400. No bulk volume (OQ-2 unresolved).
- apps/seerr.yaml: rendered via catalog/render.py --no-image-automation (default-deny
  DNS-only baseline + 3 dep CiliumNetworkPolicies + gateway-ingress + internet world
  egress + HTTPRoute seerr.home.keithjackson.dev:5055 + ServiceMonitor + alerts).
- apps/kustomization.yaml: add seerr.yaml (with the arr/media stack).

Gates: render --selftest PASS; ci lint PASS; L0 kyverno 17/17 (schema + default-deny +
egress-class + auth-mode + host-ns); L1 render 3/3. Other apps byte-identical.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 00:27:47 -05:00
jakman85andClaude Opus 4.8 67bc3175f5 fix(ci): drop upload-artifact@v4 (unsupported by Gitea Actions/GHES)
The static gate passes (lint + Kyverno L0 + L1 render) but the trailing
actions/upload-artifact@v4 step fails with "not supported on GHES" — Gitea Actions
does not implement the artifact-v2 backend — and its always() failure sank the whole
job. Remove the step; JUnit/coverage remain in the job workspace. Re-add via the
@v3 action or Gitea's artifact backend later if retention is needed.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 00:09:04 -05:00
jakman85andClaude Opus 4.8 0f75fa29ca fix(ci): call spec-immutability.sh directly in gate (make masks exit 3 as 2)
`make -C ci spec-immutability || [ $? -eq 3 ]` fails on the honest skip: GNU make
exits 2 when the recipe exits 3, so the guard never matches. Invoke the script
directly so the exit-3 skip (spec repo not checked out in CI) is tolerated while a
real spec drift (exit 1) still blocks.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 00:06:47 -05:00
jakman85andClaude Opus 4.8 79e7f92315 fix(ci): kyverno CLI arch x86_64 in install-tools; wait for dind certs
- install-tools.sh: the kyverno CLI release tarball is named _linux_x86_64, not
  _linux_amd64 (mirrors infra/devcontainer/Containerfile). The gate job's kyverno
  install 404'd on amd64; map amd64 -> x86_64 for kyverno only.
- act-runner: wait for /certs/client/ca.pem (published by the DinD sidecar) before
  exec-ing the entrypoint, eliminating the startup crash-loop race.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 00:04:04 -05:00
jakman85andClaude Opus 4.8 bc4519bc4c Phase-6: enable Gitea Actions, in-cluster act_runner, static CI gate
STEP 1 — enable Gitea Actions: pin GITEA__actions__ENABLED=true and
DEFAULT_ACTIONS_URL=github in the gitea bootstrap patch (was implicit upstream
default; make it a declared property).

STEP 2 — deploy + register an act_runner in a dedicated non-app namespace
(gitea-actions) so the privileged DinD executor sidecar is admitted without a
Kyverno policy exception (restrict-host-namespaces scopes to app namespaces only).
Docker executor + DinD sidecar over a shared TLS-cert emptyDir; runner state on a
Longhorn PVC (registers once). Registration token delivered as a SOPS-encrypted
Secret decrypted by Flux. Runner mounts no ServiceAccount token (R-SEC-02).

STEP 3 — author .gitea/workflows/gate.yml: the live enforced merge gate running
the STATIC layers (spec-immutability best-effort, lint, Kyverno L0, L1 render) via
`make -C ci lint l0 l1` with the pinned toolchain installed at job start. L2/L3
conformance is an if:false placeholder (needs a toolchain image + scoped cluster
creds — a design decision, not wired). merge-gate.yaml reduced to workflow_dispatch
(dormant full-pipeline blueprint). install-tools.sh gains check-jsonschema so the
schema stage validates in CI instead of skipping.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 23:51:52 -05:00
jakman85andClaude Opus 4.8 81ef78295a infra: direct kube-API access via kube-vip VIP + NAT (retire SSH tunnel)
Controller now reaches the kube-API directly through the pfSense NAT
(192.168.2.50:6443 -> kube-vip VIP 192.168.1.49:6443), validating TLS
against the VIP via tls-server-name. Removes the dev.sh SSH-tunnel
scaffolding and updates the ansible source-of-truth so a rebuild
regenerates a direct-access kubeconfig.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 22:10:27 -05:00
jakman85andClaude Opus 4.8 60a58f01ad feat(infra): kube-vip control-plane HA VIP (192.168.1.49) + API tls-san
Add an HA control-plane API VIP for the k3s cluster with kube-vip, and add
the VIP + all node IPs to the kube-apiserver serving-cert SANs so TLS via the
VIP validates. Prepares the stable failover endpoint the operator fronts with
a pfSense port-forward to 192.168.1.49:6443.

roles/kube_vip + playbooks/kube-vip.yml:
- Templates a single manifest (RBAC + DaemonSet kube-vip-ds) into each server's
  k3s auto-deploy dir (/var/lib/rancher/k3s/server/manifests) so it is
  reproducible and independent of Flux; k3s auto-applies it.
- Control-plane ONLY (cp_enable=true, svc_enable=false): Cilium keeps ownership
  of Service LoadBalancing (L2 pool .53-.60) — no ARP conflict.
- Runs on control-plane nodes (nodeSelector node-role.kubernetes.io/control-plane),
  hostNetwork, NET_ADMIN/NET_RAW, image pinned ghcr.io/kube-vip/kube-vip:v0.8.9.
- vip_interface auto-detected per node (the iface holding 192.168.1.5x = eth0).
- Leader election (plndr-cp-lock) with 15/10/2s lease timings.
- This cluster runs without kube-proxy (Cilium replacement), so a hostNetwork
  pod cannot reach the kubernetes ClusterIP; kube-vip is pointed at the
  node-local apiserver (KUBERNETES_SERVICE_HOST=127.0.0.1:6443) so leader
  election works.

roles/k3s_server tls-san:
- New k3s_tls_sans (VIP + all three node IPs + 127.0.0.1, deduped); install
  args now emit one --tls-san per entry.
- Persist the SAN list to /etc/rancher/k3s/config.yaml (read by a running k3s)
  and Restart k3s handler so an existing cluster regenerates its serving cert.
- k3s.yml secondaries run serial:1 so a SAN change rolls one node at a time,
  keeping etcd quorum (kube-vip keeps the API reachable during each restart).

Verified live: DS 3/3 ready, VIP answers on 192.168.1.49:6443, all 3 certs and
an s_client against the VIP carry 192.168.1.49 + .50/.51/.52 + 127.0.0.1, all
nodes Ready throughout the rolling restart.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 21:52:43 -05:00
jakman85andClaude Opus 4.8 d99d171468 Phase-6: repoint Flux source to in-cluster Gitea
Cut the GitOps source over from the bootstrap host-Gitea
(http://192.168.2.103:3000) to the now-healthy in-cluster Gitea at
https://gitea.home.keithjackson.dev. Public repo, LE-prod wildcard
cert publicly trusted, resolved in-cluster via coredns->Gateway, so
no secretRef/caCert needed. Bootstrap host-Gitea retained (stopped)
as a recovery path.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 18:50:19 -05:00
jakman85andClaude Opus 4.8 d32b5c4ec0 fix(renderer): Recreate strategy for Deployments mounting RWO PVCs
Any single-replica app with a `storage:` entry mounts a ReadWriteOnce Longhorn
PVC. The default RollingUpdate strategy starts the new pod before terminating the
old one, but the old pod still holds the RWO volume -> the new pod hangs in
ContainerCreating and the rollout deadlocks (observed on gitea's update). Emit
spec.strategy.type=Recreate whenever storage is present so the old pod releases
the PVC first. Non-storage apps (and VMs) keep the default (no strategy emitted).

Re-rendered apps: gitea, pihole, sonarr, radarr, plex, qbittorrent, jackett now
carry strategy: Recreate. Updated the egress-class L1 render goldens (storage
fixture) accordingly; all L1 render checks pass.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 18:14:45 -05:00
jakman85andClaude Opus 4.8 73c4475d34 fix(gitea): mount PVC at /data + headless-install bootstrap (Phase 6)
Two defects blocked in-cluster Gitea from becoming the Flux source:

1. Storage mountPath: the renderer hard-coded /data/<name>, so Gitea's PVC
   mounted at /data/data while Gitea writes all state to /data — real repos/DB
   landed on the ephemeral rootfs. Add an OPTIONAL `mountPath` to the storage
   schema + render.py (default stays /data/<name>, back-compat). catalog/gitea.yaml
   sets data.mountPath: /data. Only gitea.yaml's rendered mount changes; every
   other app renders byte-identical.

2. Headless install: the instance was un-installed (INSTALL_LOCK=false, ephemeral
   SECRET_KEY, no admin, API 404). A Gitea-only supplementary manifest
   (apps/gitea-bootstrap-patch.yaml) patches the rendered Deployment with
   GITEA__security__INSTALL_LOCK=true, a FIXED SECRET_KEY from a persistent SOPS
   secret (apps/gitea-bootstrap-secret.sops.yaml), ROOT_URL/DOMAIN, and a postStart
   hook that idempotently creates a break-glass local admin. State persists on the
   now-correct /data Longhorn PVC.

The env/lifecycle wiring is a supplementary kustomize patch because the
ServiceDefinition schema/renderer does not yet express env/envFrom/lifecycle —
noted as a renderer follow-up (TODO in the patch).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 18:07:56 -05:00
jakman85andClaude Opus 4.8 9f52f195a7 test(conformance): lower chainsaw parallelism 4->2 to avoid dev etcd overload cascades
On the small dev VMs, 4 concurrent heavy tests (KubeVirt CDI + Longhorn) spiked
embedded-etcd into leader-changes/apiserver-ServiceUnavailable; with fail-closed
Kyverno admission this cascaded into mass test aborts. 2 lets the full L2/L3 suite
complete in a single clean pass.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 17:21:48 -05:00
jakman85andClaude Opus 4.8 63b87ffe46 test(conformance): update DNS tests to Proxmox Gateway VIP 192.168.1.53 (was 10.73.0.24x)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 16:52:09 -05:00
jakman85andClaude Opus 4.8 28a34c4477 fix(kyverno): raise admission-controller resources + dev replicas 2 to stop fail-closed crashloop/502s
Admission controller was crashlooping (9 restarts/4h, exit 255, NOT OOM:
live usage ~65Mi/384Mi limit, 7m cpu). Root cause: chart-default 100m CPU
request starved the process under node contention, so client-go calls hit
context-deadline-exceeded, leader-election lease renewal timed out, leadership
was lost and the webhook server closed. Raise the CPU request to 250m (no CPU
limit, to avoid CFS throttling) with modest memory headroom (256Mi/512Mi), and
raise dev admission replicas 1->2 so a single fail-closed pod restart no longer
502s cluster-wide admission-gated writes.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 16:43:43 -05:00
jakman85andClaude Opus 4.8 1b0848c224 fix(kyverno): correct default-deny-shape to require both Ingress+Egress via deny.conditions (scalar-array pattern was inverted)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 16:00:42 -05:00