Adding the SSH key pushed the inline cloudInitNoCloud.userData over KubeVirt's
2048-byte limit ("userdata exceeds 2048 byte limit. Should use UserDataSecretRef").
Move the cloud-init into apps/vms/unifi/unifi-cloudinit.yaml and wrap it as the
`unifi-cloudinit` Secret via a Kustomize secretGenerator (disableNameSuffixHash
so the VM can reference it by fixed name); VM now uses userDataSecretRef. Content
is non-sensitive (public key + public installer script), so not SOPS-encrypted.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Flux's postBuild.substituteFrom runs envsubst over the WHOLE manifest, including
the VM cloud-init userData. It was substituting every shell `${...}` in the UOS
install script (${UOS_URL}, the API-resolver ${...}) to empty — which is why the
first UOS install silently did nothing — and ${PIPESTATUS[0]} then broke the
build entirely ("missing closing brace"), blocking apps-vms reconcile.
VM manifests use no cluster-vars, so substitution is unneeded. Removed it; the
cloud-init shell now reaches the guest intact. (A future VM needing a cluster-var
must escape its shell `${...}` as `$${...}`.)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The first UOS bring-up left no way into the guest (cloud-init set no
credentials), so the in-guest UOS install couldn't be observed. Add:
- ssh_authorized_keys (public key, safe to commit) on the default ubuntu user
- qemu-guest-agent (KubeVirt AgentConnected + virtctl guest access)
- tee the install-uos output to the serial console + /var/log/uos-install.log,
and record the installer exit code, so the install is observable.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The UniFi Block import got past the device-permission fix but then failed
ReplicaSchedulingFailure / "insufficient storage": CDI sizes scratch ~= the disk
virtual size and, on the default 3-replica `longhorn` class, the ~60Gi scratch x3
(189G) on top of the disk's x3 (180G) overflowed the Longhorn tier (~125G/node).
- platform/longhorn/storageclasses.yaml: NEW `longhorn-scratch` StorageClass
(numberOfReplicas: 1) — CDI scratch is ephemeral/throwaway, replicating it 3x is
waste; if a node dies mid-import CDI just retries.
- platform/kubevirt/cdi-cr.yaml: spec.config.scratchSpaceStorageClass ->
longhorn-scratch.
- apps/vms/unifi/20-datavolume.yaml: boot disk 60Gi -> 40Gi (UOS min is 25Gi;
40Gi is comfortable for a home site and lighter on the replicated tier).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The k3s nodes authorize ~/.ssh/home_network for the `k3s` user, but ansible.cfg
set no private_key_file and there's no ssh_config entry for the SSH-NAT host, so
node-config plays (e.g. the rolling k3s config.yaml + Restart handler) failed
with Permission denied (publickey). Pin the key so those plays connect without a
per-run --private-key.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The UniFi VM boot-disk DataVolume (RWX + Block on longhorn-vm) failed to import:
CDI's non-root importer (UID 107, cap-drop ALL) hit
`blockdev: cannot open /dev/cdi-block-volume: Permission denied`.
Kubernetes only applies fsGroup ownership to *filesystem* volumes — it never
chowns a raw block device — and Longhorn's CSIDriver (fsGroupPolicy:
ReadWriteOnceWithFSType) skips fsGroup on RWX/Block volumes regardless, so the
device stays root:root. The CDI-documented fix is the container-runtime setting
`device_ownership_from_security_context`, exposed on k3s (>= v1.31.3+k3s1, this
cluster is v1.31.5) as `nonroot-devices: true`. It makes containerd set the
in-container device node's owner from the pod's securityContext.
Added to the Ansible-managed /etc/rancher/k3s/config.yaml alongside tls-san;
applied via the existing rolling `Restart k3s` handler (serial:1, kube-vip keeps
the API up, etcd holds 2/3). Node-scoped, security-neutral (pods stay non-root),
and preserves the longhorn-vm RWX/Block/migratable contract unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
First KubeVirt VM app + first real CDI http import. UniFi OS Server (UOS) is
Ubiquiti's current self-host product (the .deb is deprecated); it bundles UniFi
OS + Network app + its own datastore, so no MongoDB companion.
- apps/vms/unifi: CDI DataVolume (Ubuntu 24.04 cloud image -> longhorn-vm
RWX/Block, live-migratable) + VirtualMachine (evictionStrategy: LiveMigrate)
with cloud-init that installs podman/slirp4netns/jq and runs the UOS installer
NON-INTERACTIVELY. The installer URL is resolved at boot from Ubiquiti's public
firmware API (fw-update.ui.com/api/firmware-latest) — always-latest, no pinned
URL to maintain (currently UOS v5.1.37).
- L4 exposure: two LoadBalancer Services (TCP+UDP) sharing a pinned, L2-announced
Cilium VIP 192.168.1.54 (device inform/adoption + STUN can't ride the L7
Gateway); external-dns publishes unifi.home.keithjackson.dev. UI on :11443.
- default-deny + additive CNPs (controller ingress, egress, and CDI-importer
egress for the image pull).
- clusters/base/apps-vms.yaml: NON-GATING Flux Kustomization (wait:false,
dependsOn platform-config) — VMs are a separate lifecycle from the app waves,
so a slow VM import/install can't stall the app tier.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.
- catalog/render.py: compute waves by longest dependency-path depth over the
ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
consumer lands strictly higher — the "DB below consumer" rule falls out of
longest-path, no special case). `--emit-waves` generates the per-wave
kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
wave 1: sonarr radarr nextcloud semaphore gitea-actions
wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.
Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that
object here does not GC its resources; the wave Kustomizations SSA-adopt the
byte-identical manifests in place (no restart, no PVC churn).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Temporary: with prune:false, removing the `apps` Kustomization object in the
follow-up wave-split commit will NOT garbage-collect its owned resources
(CNPG Clusters -> PVCs). The apps-wave-* Kustomizations adopt them in place.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adds the VM/CDI storage tier approved by the PM (RWX for live-migration):
- platform/external-snapshotter (NEW, controllers pass): the cluster-singleton
snapshot.storage.k8s.io control plane (CRDs + snapshot-controller in
kube-system), pinned v8.2.0 (Kubernetes deps aligned to 1.31 since v8.1.0;
this cluster is k3s v1.31.5). These CRDs were absent — only snapshots.longhorn.io
and kubevirt.io snapshot CRDs existed.
- platform/longhorn/volumesnapshotclass.yaml (NEW, platform-config): the
driver.longhorn.io VolumeSnapshotClass (type: snap, in-cluster).
- platform/kubevirt/cdi-storageprofiles.yaml (NEW, platform-config): StorageProfile
longhorn -> cloneStrategy: snapshot; StorageProfile longhorn-vm -> snapshot clone
+ claimPropertySets RWX/Block. Non-degraded default virt class clears
CDIDefaultStorageClassDegraded.
- platform/longhorn/storageclasses.yaml: NEW longhorn-vm StorageClass —
migratable: "true" (Longhorn's KubeVirt-live-migration knob; RWX/Block-only on
1.7.3), is-default-virt-class (NOT the k8s default; longhorn stays RWO default).
- platform/kubevirt/kubevirt-cr.yaml: comment only — LiveMigration is GA on v1.9
(no feature gate); no cluster-wide evictionStrategy (would wedge RWO-disk drains);
live-migration stays per-VM opt-in on a longhorn-vm disk.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The alert fired continuously for ~21h while the control plane was fully
healthy (both virt-controller and virt-api pods 1/1 Ready, KubeVirt CR
Deployed/Available=True). Two bugs, each making it fire regardless of state:
1. `kubevirt_virt_controller_ready` does not exist — the real metric is
`kubevirt_virt_controller_ready_status` (KubeVirt suffixes `_status`),
so that clause was always ABSENT (never matched).
2. `absent(up{job="kubevirt"} == 1)` — the KubeVirt scrape job label is
`kubevirt-prometheus-metrics` (the Service name), not `kubevirt`, so
`up{job="kubevirt"}` matched nothing and absent() was permanently 1.
Fix: use `sum(kubevirt_virt_{controller,api}_ready_status) == 0` (fires only
when every replica is unready) and the correct job label. virt-api added to
match the alert's stated scope (virt-controller/virt-api).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The cdi-operator manages its own `service-monitor-cdi` (scheme https +
cdi-metrics-reader-token) which backs CDI's shipped recording rules
(kubevirt_cdi_operator_up / cluster:kubevirt_cdi_operator_up:sum) and the
CDIOperatorDown alert. Our extra `cdi` ServiceMonitor scraped the same
cdi-prometheus-metrics service (HTTPS :8443) over http, so every scrape
failed with 400 Bad Request, adding down (up=0) targets for the CDI pods.
That poisoned the up-aggregation the recording rule sums, driving it to 0
and firing a false CDIOperatorDown even though the operator was Running,
Ready, and Available=True. Remove the duplicate; keep the kubevirt SM
(virt-operator ships none of its own).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
TASK 1 — reliably single-pass green L2/L3:
Split run_chainsaw into a LIGHT pass (concurrent, config parallelism) and a
HEAVY pass (serial, parallel=1). The KubeVirt-CDI/Longhorn/DR tests
(vm-lifecycle, vm-data-survival, vm-default-deny, data-survival, dr-restore)
spike embedded-etcd on the small dev VMs; running them serially in their own
pass keeps their etcd spikes from ever coinciding with another test's setup —
the coincidence that let one heavy test's apiserver blip fail-close Kyverno
admission and abort every in-flight test at 0.01s. Heavy set is keyed by test
dir basename and overridable via CONF_HEAVY_TESTS. Each pass writes its own
JUnit; lib/merge_junit.py stitches them into the one stable report.junit.xml
CI collects, so the split is invisible downstream. Overall rc is the union of
both passes (no masking — real failures still surface).
Proof: full --layer l2,l3 run now completes single-pass green: 24/24 passed,
0 failures/errors/skipped. vpn-killswitch, which previously failed with an
apiserver EOF (etcd blip), passed; no mass environmental setup-aborts.
TASK 2 (T-4.1) — author conformance/tests/l3-fault-injection/alerting/
fixtures/smtp-sink.yaml: a hermetic Mailpit SMTP capture sink (Service
smtp-sink, SMTP 1025, HTTP API 8025) the alerting test applies as its mail
sink. restricted-PodSecurity compliant, in-memory (no PVC). Unblocks the
previously-aborting alerting scaffold; it now passes.
Touches conformance/ + ci-read content only — no apps/ or platform/ workload.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Restores the four semaphore resources a peer temp-disabled in 0abadd3 to unblock
the apps Kustomization while the crashloop was diagnosed. Both root causes are now
fixed and verified against the live image:
1. permission-denied writing /etc/semaphore/config.json (non-root UID 1001 vs
root-owned Longhorn PVC) -> podSecurityContext.fsGroup=1001 (schema 1.3.0);
verified the pod got past config write and successfully pinged its PostgreSQL.
2. config panic on SEMAPHORE_PORT=tcp://<svcIP>:3000 (kubelet Docker-link
injection for the same-named Service colliding with Semaphore's listen port)
-> explicit SEMAPHORE_PORT=":3000" env shadowing the service-link var.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
With the config PVC now writable (fsGroup), semaphore got past config write and
pinged its PostgreSQL successfully, then crashed on:
panic: value of field 'Port' is not valid: tcp://<svcIP>:3000
Because the Service is named `semaphore`, the kubelet injects the Docker-link-style
SEMAPHORE_PORT=tcp://<clusterIP>:3000, which collides with Semaphore's OWN
SEMAPHORE_PORT (its HTTP listen port). Add an explicit SEMAPHORE_PORT=":3000" env
entry — user-defined env takes precedence over service-link vars — pinning the
listen port back to :3000 (matches containerPort/Service). Uses the existing env
field; only apps/semaphore.yaml changes, all others byte-identical.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The semaphoreui/semaphore image falls through to its interactive setup wizard
and panics on /etc/semaphore/config.json permission-denied; the env-only pattern
doesn't yield a non-interactive config. Backed out to unblock the apps
Kustomization; files retained for a proper config-model follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The semaphoreui/semaphore image runs as non-root UID 1001 / GID 0 and writes its
generated config.json into the config dir on boot. A mounted Longhorn PVC is
root:root, so UID 1001 got "permission denied" and the pod crash-looped.
First-class the fix into the render contract (same mechanism the retired
gitea-bootstrap-patch was replaced by): schema 1.3.0 adds an optional pod-level
`podSecurityContext`, emitted verbatim onto spec.template.spec.securityContext by
render.py. Absent => every other app renders byte-identical (verified). semaphore
declares fsGroup=1001 (kubelet group-owns the PVC + adds the supplemental group so
UID 1001 can write) plus runAsUser/runAsGroup/runAsNonRoot pinning the image identity.
render.py selftest extended (verbatim render + absence). L0 (schema+kyverno 17/17),
L1 3/3, lint all green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Semaphore (Ansible UI, docs/13 row 10): internal exposure, internet egress,
config on Longhorn + external PostgreSQL, auth oidc. Reuses the CNPG-DB pattern
nextcloud established (apps/nextcloud-db.yaml, commit 4be6752).
- catalog/semaphore.yaml: ServiceDefinition (schema >=1.2.0 env) wiring the
SEMAPHORE_DB_* connection to semaphore-db-rw.semaphore-db:5432, break-glass
SEMAPHORE_ADMIN* local admin, and the FIXED base64 SEMAPHORE_ACCESS_KEY_ENCRYPTION.
- apps/semaphore.yaml: rendered (render.py --no-image-automation) — Deployment,
config PVC, Service, default-deny + egress/gateway/internet CNPs, HTTPRoute
semaphore.home.keithjackson.dev:3000, ServiceMonitor, SLI alert.
- apps/semaphore-db.yaml: CNPG Cluster in dedicated ns semaphore-db, initdb from
the owner secret, inheritedMetadata app.kubernetes.io/name=semaphore-db so the
dep-edge egress CNP matches.
- apps/semaphore-db-app.sops.yaml + apps/semaphore-bootstrap-secret.sops.yaml:
SOPS(age) owner creds (basic-auth) + mirrored db-password, break-glass admin,
and the persistent access-key encryption key.
OIDC (Keycloak realm client) is DEFERRED like gitea/nextcloud — comes up on the
break-glass local admin; realm-client wiring is the batched follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Onboard nextcloud (docs/13 row 7): public, DR-critical file sync backed by an
external PostgreSQL. Establishes the reusable CNPG-DB pattern for later DB-backed
services.
- catalog/nextcloud.yaml: ServiceDefinition (image nextcloud:31-apache :80, public,
internet, data 20Gi longhorn @/var/www/html, backup daily-offcluster, auth oidc
+platform-admins, ingress nextcloud.home.keithjackson.dev, dependency
nextcloud-db:5432). Uses schema 1.2.0 env for the Postgres connection, break-glass
admin, trusted-domain + OVERWRITE* proxy settings.
- apps/nextcloud-db.yaml: CloudNativePG Cluster in a dedicated nextcloud-db namespace
(modeled on keycloak-db; no default-deny, mirroring the verified keycloak posture),
inheritedMetadata stamps app.kubernetes.io/name=nextcloud-db so the rendered egress
CNP matches. initdb consumes a pre-provisioned owner secret.
- apps/nextcloud-db-app.sops.yaml + apps/nextcloud-bootstrap-secret.sops.yaml: SOPS
(age) — shared DB password mirrored across the two namespaces (Pods cannot read a
Secret cross-namespace), plus the break-glass local admin.
- apps/nextcloud.yaml: rendered manifests (catalog/render.py --no-image-automation).
- apps/kustomization.yaml: wire in the four new resources.
OIDC realm-client wiring DEFERRED (WS-APPS step-3 escape hatch): editing the reconciled
realm import in platform-config risks the running Keycloak realm / whole platform-config
apply, and nextcloud OIDC needs post-install occ steps with no first-class env — nextcloud
comes up INSTALLED on the break-glass local admin (as Gitea did); the Keycloak client is a
follow-up. Off-cluster backup awaits Velero/OQ-BKP (intent declared).
Gates: render --selftest PASS; other apps byte-identical; ci l0 17/17; l1 3/3; lint clean.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Extend the ServiceDefinition contract with optional per-container `env`,
`envFrom`, and `lifecycle` fields (k8s EnvVar / EnvFromSource / Lifecycle
shapes) so headless-install / DB-backed apps configure themselves through
the SD instead of a supplementary kustomize patch.
- schema: bump 1.1.0 -> 1.2.0; add optional env/envFrom/lifecycle to the
container kind (literal + secretKeyRef/configMapKeyRef; secretRef/
configMapRef; postStart/preStop exec via $defs/execHandler). Existing
fields unchanged; all fields optional -> absent renders byte-identical.
- render.py: emit env/envFrom/lifecycle verbatim onto the Deployment app
container when present; --selftest coverage for the new fields.
- catalog/gitea.yaml: fold the headless-install env (INSTALL_LOCK, fixed
SECRET_KEY via secretKeyRef, DOMAIN/ROOT_URL, actions, break-glass admin
creds) + the idempotent postStart break-glass admin bootstrap inline.
- apps/gitea.yaml: re-rendered with env/lifecycle inline (effective pod
spec provably equivalent to base+patch).
- DELETE apps/gitea-bootstrap-patch.yaml + drop it from apps/kustomization
patches. gitea-bootstrap-secret.sops.yaml kept (still sourced by the env).
Proves the renderer feature by retiring the known TODO; unblocks env + DB
onboarding for nextcloud/semaphore/unifi/minecraft. All other rendered apps
byte-identical. Gates: selftest PASS, l0 17/17 + schema, l1 goldens, lint.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Follow-up to the reciprocal-ingress fix: in-cluster verification (Hubble) revealed
the east-west path was STILL fully blocked — and that the pre-existing EGRESS half
was also broken, contrary to the original diagnosis. Root cause: a CiliumNetworkPolicy
`toEndpoints`/`fromEndpoints` selector is scoped to the POLICY's OWN namespace unless
it carries the reserved `k8s:io.kubernetes.pod.namespace` label. Every service lives
in its own namespace (== its name) and every dependency crosses a namespace boundary,
so an app-label-only peer selector matched nothing and Cilium silently denied the flow
(Hubble: sonarr->qbittorrent = "EGRESS DENIED ... Policy denied DROPPED").
Pin the peer namespace on BOTH halves — egress toEndpoints (destination ns == dep.to)
and the reciprocal ingress fromEndpoints (source ns == source name). This is a strict
TIGHTENING (more precise, still L3/L4), not a weakening. Verified end-to-end in-cluster:
sonarr->qbittorrent:8080, sonarr->jackett:9117, seerr->{sonarr:8989,radarr:7878,
plex:32400} now CONNECT (were DROPPED); undeclared sonarr->plex:32400 and
seerr->qbittorrent:8080 stay BLOCKED (default-deny holds).
Selftest + L0 (17/17) + L1 (3/3, goldens byte-identical) + lint all pass. NOTE: this
necessarily touches the egress toEndpoints selector, which the task scoped as
ingress-only; the premise that egress already worked was empirically false.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
A `dependencies` edge is declared ONCE on the SOURCE, but with the default-deny-
BOTH-directions baseline (docs/03 R-NET-01/02) it needs TWO matching halves: the
source egress-allow AND the destination ingress-allow. render.py emitted only the
egress half, so every declared east-west dependency was silently dropped by the
destination's default-deny ingress — verified in-cluster: sonarr->qbittorrent:8080,
sonarr->jackett:9117 and seerr->sonarr:8989 were all BLOCKED.
Add a cross-service reverse-dependency pass (build_reverse_map) and render the
reciprocal ingress half (r_reverse_dependency_policies): for each source that
declares a dependency on X, X gains an additive ingress-allow CiliumNetworkPolicy
`X-allow-<source>-<port>` whose fromEndpoints matches the source identity, mirroring
the egress half's L3/L4 granularity. Deterministic (sorted), additive (default-deny
ingress stays []), egress side unchanged. A lone render with no cross-catalog context
renders no reciprocal ingress, so the L1 render goldens are byte-identical.
New ingress CNPs: qbittorrent<-{sonarr,radarr}:8080, jackett<-{sonarr,radarr}:9117,
sonarr<-seerr:8989, radarr<-seerr:7878, plex<-seerr:32400. Selftest + L0 (17/17) +
L1 (3/3) + lint all pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The static gate passes (lint + Kyverno L0 + L1 render) but the trailing
actions/upload-artifact@v4 step fails with "not supported on GHES" — Gitea Actions
does not implement the artifact-v2 backend — and its always() failure sank the whole
job. Remove the step; JUnit/coverage remain in the job workspace. Re-add via the
@v3 action or Gitea's artifact backend later if retention is needed.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
`make -C ci spec-immutability || [ $? -eq 3 ]` fails on the honest skip: GNU make
exits 2 when the recipe exits 3, so the guard never matches. Invoke the script
directly so the exit-3 skip (spec repo not checked out in CI) is tolerated while a
real spec drift (exit 1) still blocks.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- install-tools.sh: the kyverno CLI release tarball is named _linux_x86_64, not
_linux_amd64 (mirrors infra/devcontainer/Containerfile). The gate job's kyverno
install 404'd on amd64; map amd64 -> x86_64 for kyverno only.
- act-runner: wait for /certs/client/ca.pem (published by the DinD sidecar) before
exec-ing the entrypoint, eliminating the startup crash-loop race.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
STEP 1 — enable Gitea Actions: pin GITEA__actions__ENABLED=true and
DEFAULT_ACTIONS_URL=github in the gitea bootstrap patch (was implicit upstream
default; make it a declared property).
STEP 2 — deploy + register an act_runner in a dedicated non-app namespace
(gitea-actions) so the privileged DinD executor sidecar is admitted without a
Kyverno policy exception (restrict-host-namespaces scopes to app namespaces only).
Docker executor + DinD sidecar over a shared TLS-cert emptyDir; runner state on a
Longhorn PVC (registers once). Registration token delivered as a SOPS-encrypted
Secret decrypted by Flux. Runner mounts no ServiceAccount token (R-SEC-02).
STEP 3 — author .gitea/workflows/gate.yml: the live enforced merge gate running
the STATIC layers (spec-immutability best-effort, lint, Kyverno L0, L1 render) via
`make -C ci lint l0 l1` with the pinned toolchain installed at job start. L2/L3
conformance is an if:false placeholder (needs a toolchain image + scoped cluster
creds — a design decision, not wired). merge-gate.yaml reduced to workflow_dispatch
(dormant full-pipeline blueprint). install-tools.sh gains check-jsonschema so the
schema stage validates in CI instead of skipping.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Controller now reaches the kube-API directly through the pfSense NAT
(192.168.2.50:6443 -> kube-vip VIP 192.168.1.49:6443), validating TLS
against the VIP via tls-server-name. Removes the dev.sh SSH-tunnel
scaffolding and updates the ansible source-of-truth so a rebuild
regenerates a direct-access kubeconfig.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add an HA control-plane API VIP for the k3s cluster with kube-vip, and add
the VIP + all node IPs to the kube-apiserver serving-cert SANs so TLS via the
VIP validates. Prepares the stable failover endpoint the operator fronts with
a pfSense port-forward to 192.168.1.49:6443.
roles/kube_vip + playbooks/kube-vip.yml:
- Templates a single manifest (RBAC + DaemonSet kube-vip-ds) into each server's
k3s auto-deploy dir (/var/lib/rancher/k3s/server/manifests) so it is
reproducible and independent of Flux; k3s auto-applies it.
- Control-plane ONLY (cp_enable=true, svc_enable=false): Cilium keeps ownership
of Service LoadBalancing (L2 pool .53-.60) — no ARP conflict.
- Runs on control-plane nodes (nodeSelector node-role.kubernetes.io/control-plane),
hostNetwork, NET_ADMIN/NET_RAW, image pinned ghcr.io/kube-vip/kube-vip:v0.8.9.
- vip_interface auto-detected per node (the iface holding 192.168.1.5x = eth0).
- Leader election (plndr-cp-lock) with 15/10/2s lease timings.
- This cluster runs without kube-proxy (Cilium replacement), so a hostNetwork
pod cannot reach the kubernetes ClusterIP; kube-vip is pointed at the
node-local apiserver (KUBERNETES_SERVICE_HOST=127.0.0.1:6443) so leader
election works.
roles/k3s_server tls-san:
- New k3s_tls_sans (VIP + all three node IPs + 127.0.0.1, deduped); install
args now emit one --tls-san per entry.
- Persist the SAN list to /etc/rancher/k3s/config.yaml (read by a running k3s)
and Restart k3s handler so an existing cluster regenerates its serving cert.
- k3s.yml secondaries run serial:1 so a SAN change rolls one node at a time,
keeping etcd quorum (kube-vip keeps the API reachable during each restart).
Verified live: DS 3/3 ready, VIP answers on 192.168.1.49:6443, all 3 certs and
an s_client against the VIP carry 192.168.1.49 + .50/.51/.52 + 127.0.0.1, all
nodes Ready throughout the rolling restart.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Cut the GitOps source over from the bootstrap host-Gitea
(http://192.168.2.103:3000) to the now-healthy in-cluster Gitea at
https://gitea.home.keithjackson.dev. Public repo, LE-prod wildcard
cert publicly trusted, resolved in-cluster via coredns->Gateway, so
no secretRef/caCert needed. Bootstrap host-Gitea retained (stopped)
as a recovery path.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Any single-replica app with a `storage:` entry mounts a ReadWriteOnce Longhorn
PVC. The default RollingUpdate strategy starts the new pod before terminating the
old one, but the old pod still holds the RWO volume -> the new pod hangs in
ContainerCreating and the rollout deadlocks (observed on gitea's update). Emit
spec.strategy.type=Recreate whenever storage is present so the old pod releases
the PVC first. Non-storage apps (and VMs) keep the default (no strategy emitted).
Re-rendered apps: gitea, pihole, sonarr, radarr, plex, qbittorrent, jackett now
carry strategy: Recreate. Updated the egress-class L1 render goldens (storage
fixture) accordingly; all L1 render checks pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Two defects blocked in-cluster Gitea from becoming the Flux source:
1. Storage mountPath: the renderer hard-coded /data/<name>, so Gitea's PVC
mounted at /data/data while Gitea writes all state to /data — real repos/DB
landed on the ephemeral rootfs. Add an OPTIONAL `mountPath` to the storage
schema + render.py (default stays /data/<name>, back-compat). catalog/gitea.yaml
sets data.mountPath: /data. Only gitea.yaml's rendered mount changes; every
other app renders byte-identical.
2. Headless install: the instance was un-installed (INSTALL_LOCK=false, ephemeral
SECRET_KEY, no admin, API 404). A Gitea-only supplementary manifest
(apps/gitea-bootstrap-patch.yaml) patches the rendered Deployment with
GITEA__security__INSTALL_LOCK=true, a FIXED SECRET_KEY from a persistent SOPS
secret (apps/gitea-bootstrap-secret.sops.yaml), ROOT_URL/DOMAIN, and a postStart
hook that idempotently creates a break-glass local admin. State persists on the
now-correct /data Longhorn PVC.
The env/lifecycle wiring is a supplementary kustomize patch because the
ServiceDefinition schema/renderer does not yet express env/envFrom/lifecycle —
noted as a renderer follow-up (TODO in the patch).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
On the small dev VMs, 4 concurrent heavy tests (KubeVirt CDI + Longhorn) spiked
embedded-etcd into leader-changes/apiserver-ServiceUnavailable; with fail-closed
Kyverno admission this cascaded into mass test aborts. 2 lets the full L2/L3 suite
complete in a single clean pass.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Admission controller was crashlooping (9 restarts/4h, exit 255, NOT OOM:
live usage ~65Mi/384Mi limit, 7m cpu). Root cause: chart-default 100m CPU
request starved the process under node contention, so client-go calls hit
context-deadline-exceeded, leader-election lease renewal timed out, leadership
was lost and the webhook server closed. Raise the CPU request to 250m (no CPU
limit, to avoid CFS throttling) with modest memory headroom (256Mi/512Mi), and
raise dev admission replicas 1->2 so a single fail-closed pod restart no longer
502s cluster-wide admission-gated writes.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The offline `kyverno test` CLI cannot resolve the Flux postBuild token
${KYVERNO_VALIDATION_ACTION} in validationFailureAction, so the 4 policies
carrying it failed to load and their result rows reported "Not found" (14/17
cases). Render a throwaway copy of platform/kyverno with the base/prod value
(Enforce) before `kyverno test` in both ci/l0.sh and scripts/run-l0.sh; this
does not change in-cluster enforcement.
Add tests/values.yaml supplying Namespace labels so the offline CLI resolves
namespaceSelector matches -- without it the R-SEC-02/R-NET-05/R-IDP-02 Pod
cases were silently EXCLUDED (vacuous), not evaluated.
Pin kyverno-json 0.0.3 in ci/versions.env (0.0.5 has no upstream release;
0.0.3 is what run-l0.sh and the devcontainer already use).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Bake the conformance harness toolchain into infra/devcontainer so `make -C ci gate`
runs identically here and in CI:
- Containerfile: install kyverno CLI (1.13.4), kyverno-json, chainsaw (0.2.12),
yamllint (1.35.1), check-jsonschema (0.29.4), yq (4.44.6) via new pinned RUN block;
add diffutils + python3-pip; drop stale LIBVIRT_DEFAULT_URI; refresh header for the
Proxmox(bpg)+conformance toolbox; extend /etc/dev-toolchain-versions provenance.
- versions.env: add KYVERNO/KYVERNO_JSON/CHAINSAW/YAMLLINT/YQ/CHECK_JSONSCHEMA;
align KUBECTL 1.32.0 -> 1.31.4. dev.sh build(): pass the new --build-args.
- ci/versions.env: pin CHECK_JSONSCHEMA_VERSION=0.29.4 (single source of truth).
Fix harness seams so the gate can run:
- dev.sh run(): mount the sibling spec repo (home-network-k3s) read-only at
/home-network-k3s for ci/spec-immutability.sh (R-TEST-06); conditional so the
container still runs without it.
- conformance/lib/common.sh conf_dirs_for_layer(): accept a comma/space list of
layers and union the dirs, so ci/conformance.sh `run.sh --layer l2,l3` selects
both layers in one pass (backward compatible for single layers).
NOTE: ci/versions.env pins KYVERNO_JSON_VERSION=0.0.5, which has no upstream release
(max tag is v0.0.3; platform/kyverno/scripts/run-l0.sh already defaults to v0.0.3).
Baked v0.0.3 and flagged for PM to correct ci/versions.env. kyverno-json is not used
by any static gate stage.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- drop stale libvirt socket mount + LIBVIRT_DEFAULT_URI (old KVM cluster)
- auto-open an SSH tunnel (local :16443 -> primary 127.0.0.1:6443 over the
pfSense SSH NAT) on each run so kubectl/flux/helm reach the cluster with no
manual setup; DEV_NO_TUNNEL=1 opts out, K3S_SSH_* env overrides the NAT front
- refresh header/usage comments (bpg/proxmox, not dmacvicar/libvirt)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
- inventory routes SSH via 192.168.2.102:2250/2251/2252; new k3s_node_ip carries
each node's real 192.168.1.x for --node-ip / k3s_api_host / Cilium parity
- dev.sh bind-mounts ~/.config/home-k3s/ansible-vault-pass (ro) so vaulted
playbooks can run in the pinned container (ansible.cfg vault_password_file)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
