Commit Graph
318 Commits
Author SHA1 Message Date
jakman85andClaude Opus 4.8 28498f1099 fix(apps): semaphore fsGroup — schema 1.3.0 podSecurityContext for non-root PVC write
The semaphoreui/semaphore image runs as non-root UID 1001 / GID 0 and writes its
generated config.json into the config dir on boot. A mounted Longhorn PVC is
root:root, so UID 1001 got "permission denied" and the pod crash-looped.

First-class the fix into the render contract (same mechanism the retired
gitea-bootstrap-patch was replaced by): schema 1.3.0 adds an optional pod-level
`podSecurityContext`, emitted verbatim onto spec.template.spec.securityContext by
render.py. Absent => every other app renders byte-identical (verified). semaphore
declares fsGroup=1001 (kubelet group-owns the PVC + adds the supplemental group so
UID 1001 can write) plus runAsUser/runAsGroup/runAsNonRoot pinning the image identity.

render.py selftest extended (verbatim render + absence). L0 (schema+kyverno 17/17),
L1 3/3, lint all green.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 02:00:45 -05:00
jakman85andClaude Opus 4.8 77496f9a8e feat(apps): onboard semaphore (Phase 6) — second CNPG-DB + OIDC-target app
Semaphore (Ansible UI, docs/13 row 10): internal exposure, internet egress,
config on Longhorn + external PostgreSQL, auth oidc. Reuses the CNPG-DB pattern
nextcloud established (apps/nextcloud-db.yaml, commit 4be6752).

- catalog/semaphore.yaml: ServiceDefinition (schema >=1.2.0 env) wiring the
  SEMAPHORE_DB_* connection to semaphore-db-rw.semaphore-db:5432, break-glass
  SEMAPHORE_ADMIN* local admin, and the FIXED base64 SEMAPHORE_ACCESS_KEY_ENCRYPTION.
- apps/semaphore.yaml: rendered (render.py --no-image-automation) — Deployment,
  config PVC, Service, default-deny + egress/gateway/internet CNPs, HTTPRoute
  semaphore.home.keithjackson.dev:3000, ServiceMonitor, SLI alert.
- apps/semaphore-db.yaml: CNPG Cluster in dedicated ns semaphore-db, initdb from
  the owner secret, inheritedMetadata app.kubernetes.io/name=semaphore-db so the
  dep-edge egress CNP matches.
- apps/semaphore-db-app.sops.yaml + apps/semaphore-bootstrap-secret.sops.yaml:
  SOPS(age) owner creds (basic-auth) + mirrored db-password, break-glass admin,
  and the persistent access-key encryption key.

OIDC (Keycloak realm client) is DEFERRED like gitea/nextcloud — comes up on the
break-glass local admin; realm-client wiring is the batched follow-up.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 01:47:22 -05:00
jakman85andClaude Opus 4.8 4be6752c71 feat(apps): onboard nextcloud (Phase 6) — first CNPG-DB + OIDC-target patterns
Onboard nextcloud (docs/13 row 7): public, DR-critical file sync backed by an
external PostgreSQL. Establishes the reusable CNPG-DB pattern for later DB-backed
services.

- catalog/nextcloud.yaml: ServiceDefinition (image nextcloud:31-apache :80, public,
  internet, data 20Gi longhorn @/var/www/html, backup daily-offcluster, auth oidc
  +platform-admins, ingress nextcloud.home.keithjackson.dev, dependency
  nextcloud-db:5432). Uses schema 1.2.0 env for the Postgres connection, break-glass
  admin, trusted-domain + OVERWRITE* proxy settings.
- apps/nextcloud-db.yaml: CloudNativePG Cluster in a dedicated nextcloud-db namespace
  (modeled on keycloak-db; no default-deny, mirroring the verified keycloak posture),
  inheritedMetadata stamps app.kubernetes.io/name=nextcloud-db so the rendered egress
  CNP matches. initdb consumes a pre-provisioned owner secret.
- apps/nextcloud-db-app.sops.yaml + apps/nextcloud-bootstrap-secret.sops.yaml: SOPS
  (age) — shared DB password mirrored across the two namespaces (Pods cannot read a
  Secret cross-namespace), plus the break-glass local admin.
- apps/nextcloud.yaml: rendered manifests (catalog/render.py --no-image-automation).
- apps/kustomization.yaml: wire in the four new resources.

OIDC realm-client wiring DEFERRED (WS-APPS step-3 escape hatch): editing the reconciled
realm import in platform-config risks the running Keycloak realm / whole platform-config
apply, and nextcloud OIDC needs post-install occ steps with no first-class env — nextcloud
comes up INSTALLED on the break-glass local admin (as Gitea did); the Keycloak client is a
follow-up. Off-cluster backup awaits Velero/OQ-BKP (intent declared).

Gates: render --selftest PASS; other apps byte-identical; ci l0 17/17; l1 3/3; lint clean.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 01:32:00 -05:00
jakman85andClaude Opus 4.8 c8dd4f3f9d feat(catalog): add SD env/envFrom/lifecycle; retire gitea bootstrap patch
Extend the ServiceDefinition contract with optional per-container `env`,
`envFrom`, and `lifecycle` fields (k8s EnvVar / EnvFromSource / Lifecycle
shapes) so headless-install / DB-backed apps configure themselves through
the SD instead of a supplementary kustomize patch.

- schema: bump 1.1.0 -> 1.2.0; add optional env/envFrom/lifecycle to the
  container kind (literal + secretKeyRef/configMapKeyRef; secretRef/
  configMapRef; postStart/preStop exec via $defs/execHandler). Existing
  fields unchanged; all fields optional -> absent renders byte-identical.
- render.py: emit env/envFrom/lifecycle verbatim onto the Deployment app
  container when present; --selftest coverage for the new fields.
- catalog/gitea.yaml: fold the headless-install env (INSTALL_LOCK, fixed
  SECRET_KEY via secretKeyRef, DOMAIN/ROOT_URL, actions, break-glass admin
  creds) + the idempotent postStart break-glass admin bootstrap inline.
- apps/gitea.yaml: re-rendered with env/lifecycle inline (effective pod
  spec provably equivalent to base+patch).
- DELETE apps/gitea-bootstrap-patch.yaml + drop it from apps/kustomization
  patches. gitea-bootstrap-secret.sops.yaml kept (still sourced by the env).

Proves the renderer feature by retiring the known TODO; unblocks env + DB
onboarding for nextcloud/semaphore/unifi/minecraft. All other rendered apps
byte-identical. Gates: selftest PASS, l0 17/17 + schema, l1 goldens, lint.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 01:13:52 -05:00
jakman85andClaude Opus 4.8 b871877c1b fix(render): namespace-pin east-west selectors so declared deps actually connect
Follow-up to the reciprocal-ingress fix: in-cluster verification (Hubble) revealed
the east-west path was STILL fully blocked — and that the pre-existing EGRESS half
was also broken, contrary to the original diagnosis. Root cause: a CiliumNetworkPolicy
`toEndpoints`/`fromEndpoints` selector is scoped to the POLICY's OWN namespace unless
it carries the reserved `k8s:io.kubernetes.pod.namespace` label. Every service lives
in its own namespace (== its name) and every dependency crosses a namespace boundary,
so an app-label-only peer selector matched nothing and Cilium silently denied the flow
(Hubble: sonarr->qbittorrent = "EGRESS DENIED ... Policy denied DROPPED").

Pin the peer namespace on BOTH halves — egress toEndpoints (destination ns == dep.to)
and the reciprocal ingress fromEndpoints (source ns == source name). This is a strict
TIGHTENING (more precise, still L3/L4), not a weakening. Verified end-to-end in-cluster:
sonarr->qbittorrent:8080, sonarr->jackett:9117, seerr->{sonarr:8989,radarr:7878,
plex:32400} now CONNECT (were DROPPED); undeclared sonarr->plex:32400 and
seerr->qbittorrent:8080 stay BLOCKED (default-deny holds).

Selftest + L0 (17/17) + L1 (3/3, goldens byte-identical) + lint all pass. NOTE: this
necessarily touches the egress toEndpoints selector, which the task scoped as
ingress-only; the premise that egress already worked was empirically false.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 01:01:56 -05:00
jakman85andClaude Opus 4.8 14beb85b65 fix(render): emit reciprocal ingress-allow for declared dependencies (R-NET-02)
A `dependencies` edge is declared ONCE on the SOURCE, but with the default-deny-
BOTH-directions baseline (docs/03 R-NET-01/02) it needs TWO matching halves: the
source egress-allow AND the destination ingress-allow. render.py emitted only the
egress half, so every declared east-west dependency was silently dropped by the
destination's default-deny ingress — verified in-cluster: sonarr->qbittorrent:8080,
sonarr->jackett:9117 and seerr->sonarr:8989 were all BLOCKED.

Add a cross-service reverse-dependency pass (build_reverse_map) and render the
reciprocal ingress half (r_reverse_dependency_policies): for each source that
declares a dependency on X, X gains an additive ingress-allow CiliumNetworkPolicy
`X-allow-<source>-<port>` whose fromEndpoints matches the source identity, mirroring
the egress half's L3/L4 granularity. Deterministic (sorted), additive (default-deny
ingress stays []), egress side unchanged. A lone render with no cross-catalog context
renders no reciprocal ingress, so the L1 render goldens are byte-identical.

New ingress CNPs: qbittorrent<-{sonarr,radarr}:8080, jackett<-{sonarr,radarr}:9117,
sonarr<-seerr:8989, radarr<-seerr:7878, plex<-seerr:32400. Selftest + L0 (17/17) +
L1 (3/3) + lint all pass.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 00:54:12 -05:00
jakman85andClaude Opus 4.8 b6f72eab0c feat(apps): onboard seerr (Overseerr request UI) — completes arr↔seerr↔plex graph
Phase 6: add the seerr ServiceDefinition and its rendered manifests. seerr is the
final missing node in the media dependency graph (docs/13 row 5; docs/03 allow rules).

- catalog/seerr.yaml: public | egress internet | config on longhorn (5Gi) |
  auth native (Plex accounts, documented SSO exception, docs/15) | 3 explicit egress
  deps: sonarr:8989, radarr:7878, plex:32400. No bulk volume (OQ-2 unresolved).
- apps/seerr.yaml: rendered via catalog/render.py --no-image-automation (default-deny
  DNS-only baseline + 3 dep CiliumNetworkPolicies + gateway-ingress + internet world
  egress + HTTPRoute seerr.home.keithjackson.dev:5055 + ServiceMonitor + alerts).
- apps/kustomization.yaml: add seerr.yaml (with the arr/media stack).

Gates: render --selftest PASS; ci lint PASS; L0 kyverno 17/17 (schema + default-deny +
egress-class + auth-mode + host-ns); L1 render 3/3. Other apps byte-identical.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 00:27:47 -05:00
jakman85andClaude Opus 4.8 67bc3175f5 fix(ci): drop upload-artifact@v4 (unsupported by Gitea Actions/GHES)
The static gate passes (lint + Kyverno L0 + L1 render) but the trailing
actions/upload-artifact@v4 step fails with "not supported on GHES" — Gitea Actions
does not implement the artifact-v2 backend — and its always() failure sank the whole
job. Remove the step; JUnit/coverage remain in the job workspace. Re-add via the
@v3 action or Gitea's artifact backend later if retention is needed.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 00:09:04 -05:00
jakman85andClaude Opus 4.8 0f75fa29ca fix(ci): call spec-immutability.sh directly in gate (make masks exit 3 as 2)
`make -C ci spec-immutability || [ $? -eq 3 ]` fails on the honest skip: GNU make
exits 2 when the recipe exits 3, so the guard never matches. Invoke the script
directly so the exit-3 skip (spec repo not checked out in CI) is tolerated while a
real spec drift (exit 1) still blocks.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 00:06:47 -05:00
jakman85andClaude Opus 4.8 79e7f92315 fix(ci): kyverno CLI arch x86_64 in install-tools; wait for dind certs
- install-tools.sh: the kyverno CLI release tarball is named _linux_x86_64, not
  _linux_amd64 (mirrors infra/devcontainer/Containerfile). The gate job's kyverno
  install 404'd on amd64; map amd64 -> x86_64 for kyverno only.
- act-runner: wait for /certs/client/ca.pem (published by the DinD sidecar) before
  exec-ing the entrypoint, eliminating the startup crash-loop race.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 00:04:04 -05:00
jakman85andClaude Opus 4.8 bc4519bc4c Phase-6: enable Gitea Actions, in-cluster act_runner, static CI gate
STEP 1 — enable Gitea Actions: pin GITEA__actions__ENABLED=true and
DEFAULT_ACTIONS_URL=github in the gitea bootstrap patch (was implicit upstream
default; make it a declared property).

STEP 2 — deploy + register an act_runner in a dedicated non-app namespace
(gitea-actions) so the privileged DinD executor sidecar is admitted without a
Kyverno policy exception (restrict-host-namespaces scopes to app namespaces only).
Docker executor + DinD sidecar over a shared TLS-cert emptyDir; runner state on a
Longhorn PVC (registers once). Registration token delivered as a SOPS-encrypted
Secret decrypted by Flux. Runner mounts no ServiceAccount token (R-SEC-02).

STEP 3 — author .gitea/workflows/gate.yml: the live enforced merge gate running
the STATIC layers (spec-immutability best-effort, lint, Kyverno L0, L1 render) via
`make -C ci lint l0 l1` with the pinned toolchain installed at job start. L2/L3
conformance is an if:false placeholder (needs a toolchain image + scoped cluster
creds — a design decision, not wired). merge-gate.yaml reduced to workflow_dispatch
(dormant full-pipeline blueprint). install-tools.sh gains check-jsonschema so the
schema stage validates in CI instead of skipping.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 23:51:52 -05:00
jakman85andClaude Opus 4.8 81ef78295a infra: direct kube-API access via kube-vip VIP + NAT (retire SSH tunnel)
Controller now reaches the kube-API directly through the pfSense NAT
(192.168.2.50:6443 -> kube-vip VIP 192.168.1.49:6443), validating TLS
against the VIP via tls-server-name. Removes the dev.sh SSH-tunnel
scaffolding and updates the ansible source-of-truth so a rebuild
regenerates a direct-access kubeconfig.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 22:10:27 -05:00
jakman85andClaude Opus 4.8 60a58f01ad feat(infra): kube-vip control-plane HA VIP (192.168.1.49) + API tls-san
Add an HA control-plane API VIP for the k3s cluster with kube-vip, and add
the VIP + all node IPs to the kube-apiserver serving-cert SANs so TLS via the
VIP validates. Prepares the stable failover endpoint the operator fronts with
a pfSense port-forward to 192.168.1.49:6443.

roles/kube_vip + playbooks/kube-vip.yml:
- Templates a single manifest (RBAC + DaemonSet kube-vip-ds) into each server's
  k3s auto-deploy dir (/var/lib/rancher/k3s/server/manifests) so it is
  reproducible and independent of Flux; k3s auto-applies it.
- Control-plane ONLY (cp_enable=true, svc_enable=false): Cilium keeps ownership
  of Service LoadBalancing (L2 pool .53-.60) — no ARP conflict.
- Runs on control-plane nodes (nodeSelector node-role.kubernetes.io/control-plane),
  hostNetwork, NET_ADMIN/NET_RAW, image pinned ghcr.io/kube-vip/kube-vip:v0.8.9.
- vip_interface auto-detected per node (the iface holding 192.168.1.5x = eth0).
- Leader election (plndr-cp-lock) with 15/10/2s lease timings.
- This cluster runs without kube-proxy (Cilium replacement), so a hostNetwork
  pod cannot reach the kubernetes ClusterIP; kube-vip is pointed at the
  node-local apiserver (KUBERNETES_SERVICE_HOST=127.0.0.1:6443) so leader
  election works.

roles/k3s_server tls-san:
- New k3s_tls_sans (VIP + all three node IPs + 127.0.0.1, deduped); install
  args now emit one --tls-san per entry.
- Persist the SAN list to /etc/rancher/k3s/config.yaml (read by a running k3s)
  and Restart k3s handler so an existing cluster regenerates its serving cert.
- k3s.yml secondaries run serial:1 so a SAN change rolls one node at a time,
  keeping etcd quorum (kube-vip keeps the API reachable during each restart).

Verified live: DS 3/3 ready, VIP answers on 192.168.1.49:6443, all 3 certs and
an s_client against the VIP carry 192.168.1.49 + .50/.51/.52 + 127.0.0.1, all
nodes Ready throughout the rolling restart.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 21:52:43 -05:00
jakman85andClaude Opus 4.8 d99d171468 Phase-6: repoint Flux source to in-cluster Gitea
Cut the GitOps source over from the bootstrap host-Gitea
(http://192.168.2.103:3000) to the now-healthy in-cluster Gitea at
https://gitea.home.keithjackson.dev. Public repo, LE-prod wildcard
cert publicly trusted, resolved in-cluster via coredns->Gateway, so
no secretRef/caCert needed. Bootstrap host-Gitea retained (stopped)
as a recovery path.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 18:50:19 -05:00
jakman85andClaude Opus 4.8 d32b5c4ec0 fix(renderer): Recreate strategy for Deployments mounting RWO PVCs
Any single-replica app with a `storage:` entry mounts a ReadWriteOnce Longhorn
PVC. The default RollingUpdate strategy starts the new pod before terminating the
old one, but the old pod still holds the RWO volume -> the new pod hangs in
ContainerCreating and the rollout deadlocks (observed on gitea's update). Emit
spec.strategy.type=Recreate whenever storage is present so the old pod releases
the PVC first. Non-storage apps (and VMs) keep the default (no strategy emitted).

Re-rendered apps: gitea, pihole, sonarr, radarr, plex, qbittorrent, jackett now
carry strategy: Recreate. Updated the egress-class L1 render goldens (storage
fixture) accordingly; all L1 render checks pass.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 18:14:45 -05:00
jakman85andClaude Opus 4.8 73c4475d34 fix(gitea): mount PVC at /data + headless-install bootstrap (Phase 6)
Two defects blocked in-cluster Gitea from becoming the Flux source:

1. Storage mountPath: the renderer hard-coded /data/<name>, so Gitea's PVC
   mounted at /data/data while Gitea writes all state to /data — real repos/DB
   landed on the ephemeral rootfs. Add an OPTIONAL `mountPath` to the storage
   schema + render.py (default stays /data/<name>, back-compat). catalog/gitea.yaml
   sets data.mountPath: /data. Only gitea.yaml's rendered mount changes; every
   other app renders byte-identical.

2. Headless install: the instance was un-installed (INSTALL_LOCK=false, ephemeral
   SECRET_KEY, no admin, API 404). A Gitea-only supplementary manifest
   (apps/gitea-bootstrap-patch.yaml) patches the rendered Deployment with
   GITEA__security__INSTALL_LOCK=true, a FIXED SECRET_KEY from a persistent SOPS
   secret (apps/gitea-bootstrap-secret.sops.yaml), ROOT_URL/DOMAIN, and a postStart
   hook that idempotently creates a break-glass local admin. State persists on the
   now-correct /data Longhorn PVC.

The env/lifecycle wiring is a supplementary kustomize patch because the
ServiceDefinition schema/renderer does not yet express env/envFrom/lifecycle —
noted as a renderer follow-up (TODO in the patch).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 18:07:56 -05:00
jakman85andClaude Opus 4.8 9f52f195a7 test(conformance): lower chainsaw parallelism 4->2 to avoid dev etcd overload cascades
On the small dev VMs, 4 concurrent heavy tests (KubeVirt CDI + Longhorn) spiked
embedded-etcd into leader-changes/apiserver-ServiceUnavailable; with fail-closed
Kyverno admission this cascaded into mass test aborts. 2 lets the full L2/L3 suite
complete in a single clean pass.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 17:21:48 -05:00
jakman85andClaude Opus 4.8 63b87ffe46 test(conformance): update DNS tests to Proxmox Gateway VIP 192.168.1.53 (was 10.73.0.24x)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 16:52:09 -05:00
jakman85andClaude Opus 4.8 28a34c4477 fix(kyverno): raise admission-controller resources + dev replicas 2 to stop fail-closed crashloop/502s
Admission controller was crashlooping (9 restarts/4h, exit 255, NOT OOM:
live usage ~65Mi/384Mi limit, 7m cpu). Root cause: chart-default 100m CPU
request starved the process under node contention, so client-go calls hit
context-deadline-exceeded, leader-election lease renewal timed out, leadership
was lost and the webhook server closed. Raise the CPU request to 250m (no CPU
limit, to avoid CFS throttling) with modest memory headroom (256Mi/512Mi), and
raise dev admission replicas 1->2 so a single fail-closed pod restart no longer
502s cluster-wide admission-gated writes.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 16:43:43 -05:00
jakman85andClaude Opus 4.8 1b0848c224 fix(kyverno): correct default-deny-shape to require both Ingress+Egress via deny.conditions (scalar-array pattern was inverted)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 16:00:42 -05:00
jakman85andClaude Opus 4.8 08113ccfbb ci(l0): fix kyverno test expectations for CLI 1.13.4; pin kyverno-json 0.0.3
The offline `kyverno test` CLI cannot resolve the Flux postBuild token
${KYVERNO_VALIDATION_ACTION} in validationFailureAction, so the 4 policies
carrying it failed to load and their result rows reported "Not found" (14/17
cases). Render a throwaway copy of platform/kyverno with the base/prod value
(Enforce) before `kyverno test` in both ci/l0.sh and scripts/run-l0.sh; this
does not change in-cluster enforcement.

Add tests/values.yaml supplying Namespace labels so the offline CLI resolves
namespaceSelector matches -- without it the R-SEC-02/R-NET-05/R-IDP-02 Pod
cases were silently EXCLUDED (vacuous), not evaluated.

Pin kyverno-json 0.0.3 in ci/versions.env (0.0.5 has no upstream release;
0.0.3 is what run-l0.sh and the devcontainer already use).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 15:57:29 -05:00
jakman85andClaude Opus 4.8 bb48f675ee ci: bake pinned test toolchain into devcontainer + fix conformance layer/spec-immutability seams
Bake the conformance harness toolchain into infra/devcontainer so `make -C ci gate`
runs identically here and in CI:
- Containerfile: install kyverno CLI (1.13.4), kyverno-json, chainsaw (0.2.12),
  yamllint (1.35.1), check-jsonschema (0.29.4), yq (4.44.6) via new pinned RUN block;
  add diffutils + python3-pip; drop stale LIBVIRT_DEFAULT_URI; refresh header for the
  Proxmox(bpg)+conformance toolbox; extend /etc/dev-toolchain-versions provenance.
- versions.env: add KYVERNO/KYVERNO_JSON/CHAINSAW/YAMLLINT/YQ/CHECK_JSONSCHEMA;
  align KUBECTL 1.32.0 -> 1.31.4. dev.sh build(): pass the new --build-args.
- ci/versions.env: pin CHECK_JSONSCHEMA_VERSION=0.29.4 (single source of truth).

Fix harness seams so the gate can run:
- dev.sh run(): mount the sibling spec repo (home-network-k3s) read-only at
  /home-network-k3s for ci/spec-immutability.sh (R-TEST-06); conditional so the
  container still runs without it.
- conformance/lib/common.sh conf_dirs_for_layer(): accept a comma/space list of
  layers and union the dirs, so ci/conformance.sh `run.sh --layer l2,l3` selects
  both layers in one pass (backward compatible for single layers).

NOTE: ci/versions.env pins KYVERNO_JSON_VERSION=0.0.5, which has no upstream release
(max tag is v0.0.3; platform/kyverno/scripts/run-l0.sh already defaults to v0.0.3).
Baked v0.0.3 and flagged for PM to correct ci/versions.env. kyverno-json is not used
by any static gate stage.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 15:43:19 -05:00
jakman85andClaude Opus 4.8 0d7495486e devcontainer: wire dev.sh to the Proxmox k3s cluster
- drop stale libvirt socket mount + LIBVIRT_DEFAULT_URI (old KVM cluster)
- auto-open an SSH tunnel (local :16443 -> primary 127.0.0.1:6443 over the
  pfSense SSH NAT) on each run so kubectl/flux/helm reach the cluster with no
  manual setup; DEV_NO_TUNNEL=1 opts out, K3S_SSH_* env overrides the NAT front
- refresh header/usage comments (bpg/proxmox, not dmacvicar/libvirt)

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 14:59:19 -05:00
jakman85andClaude Opus 4.8 21669da075 flux: re-enable KubeVirt + CDI (new 16GB Proxmox nodes with nested virt)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 13:21:00 -05:00
jakman85andClaude Opus 4.8 e6b3a03ff4 flux: re-enable oauth2-proxy now Keycloak is live and split-horizon DNS points at 192.168.1.53
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 13:17:04 -05:00
jakman85andClaude Opus 4.8 604552e51c flux(dev): set GATEWAY_VIP to Proxmox Gateway VIP 192.168.1.53 (split-horizon DNS)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 13:15:16 -05:00
jakman85andClaude Opus 4.8 efd6dbeb3b flux(platform-config): temp-disable orphaned oauth2-proxy refs (missing ns) to unblock reconcile
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 13:02:08 -05:00
jakman85andClaude Opus 4.8 2fea1e261c flux(platform): temp-disable oauth2-proxy to break cold-start OIDC deadlock (re-enable after platform-config green)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 12:48:05 -05:00
jakman85andClaude Opus 4.8 c6301a79c5 fix(bootstrap): subshell the repo_root fallback so trailing pwd doesn't corrupt the path
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 12:27:34 -05:00
jakman85andClaude Opus 4.8 fff44992e7 migrate(proxmox): bootstrap source URL -> 192.168.2.103; Cilium k8sServiceHost -> 192.168.1.50
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 12:22:08 -05:00
jakman85andClaude Opus 4.8 0037fb40fa ansible(cilium): rewrite controller kubeconfig to SSH-tunnel endpoint (kube-API not NAT-forwarded)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 12:15:12 -05:00
jakman85andClaude Opus 4.8 1004c8eae6 ansible(node_baseline): don't fail on qemu-guest-agent when no virtio channel (agent disabled in tofu); enable-only
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 12:04:19 -05:00
jakman85andClaude Opus 4.8 1ff25318c9 ansible(proxmox): route Ansible via pfSense NAT; decouple k3s node-ip from ansible_host; mount vault-pass into dev container
- inventory routes SSH via 192.168.2.102:2250/2251/2252; new k3s_node_ip carries
  each node's real 192.168.1.x for --node-ip / k3s_api_host / Cilium parity
- dev.sh bind-mounts ~/.config/home-k3s/ansible-vault-pass (ro) so vaulted
  playbooks can run in the pinned container (ansible.cfg vault_password_file)

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 11:55:37 -05:00
jakman85andClaude Opus 4.8 28d200caf2 tofu(proxmox): use debian generic image + serial console to fix first-boot kernel panic (bpg#1639)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 11:45:29 -05:00
jakman85andClaude Opus 4.8 fca8d8f800 tofu(proxmox): download base image as import content-type for disk import_from
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 10:59:01 -05:00
jakman85andClaude Opus 4.8 a2392a4a43 tofu(proxmox): API-only native cloud-init; move node baseline to Ansible node_baseline role
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 10:55:48 -05:00
jakman85 8a8f28bf80 infra(proxmox): bump node disks to 64GB OS + 128GB Longhorn data (per operator) 2026-08-25 10:31:39 -05:00
jakman85andClaude Opus 4.8 d57c9cbf12 infra(net): repoint ansible inventory + Cilium VIPs to the Proxmox LAN
- ansible hosts.ini: node ansible_host -> 192.168.1.50/.51/.52, with a
  clear note that the PM may need to swap these for the pfSense
  NAT-forwarded address:port once that NAT rule exists (not guessed here).
  k3s_api_host derives from the inventory, so it follows automatically.
- Cilium CiliumLoadBalancerIPPool -> 192.168.1.53-.60 (matches the tofu
  cilium_l2_vip reservation on the new LAN).
- CiliumL2AnnouncementPolicy: broaden interface selector to ^e(n|th) so
  the Proxmox NIC name (ens18/eth0) need not be pinned; flagged
  verify-at-bringup.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 10:23:37 -05:00
jakman85andClaude Opus 4.8 4bb9c8fcbd infra(tofu): docs + example for the Proxmox migration
Update tofu README (Proxmox targets table, operator prerequisites:
enable Snippets content on `local`, endpoint scheme fix, pfSense NAT),
terraform.tfvars.example (Proxmox vars + env-only auth, token never a
tofu var), and the devcontainer versions.env provider note.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 10:23:28 -05:00
jakman85andClaude Opus 4.8 39123d7806 infra(cloud-init): deliver baseline via bpg user-data snippet; drop netplan/meta-data templates
bpg/proxmox generates cloud-init network-config (static IP/gw/DNS) and
meta-data natively from the VM initialization block, so remove
network-config.yaml.tftpl and meta-data.yaml.tftpl. The rich first-boot
baseline (users, SSH key, packages, kernel modules, Longhorn /dev/vdb
mount, swap-off) is preserved and shipped as a user-data snippet
referenced by initialization.user_data_file_id. Comments updated
(libvirt -> Proxmox); /dev/vdb still valid (data disk is virtio1).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 10:23:21 -05:00
jakman85andClaude Opus 4.8 53ca698edc infra(tofu): migrate node provisioning from libvirt to bpg/proxmox
Replace the dmacvicar/libvirt provider + resources with the modern
bpg/proxmox provider. The dev cluster moves off the memory-constrained
local libvirt host onto the Proxmox VE box.

- versions.tf: pin bpg/proxmox ~> 0.66; provider reads auth from env
  (PROXMOX_VE_*, token never a tofu var) and normalizes a schemeless
  PROXMOX_VE_ENDPOINT (prepend https://) to avoid the :8006 HTTP 301.
- variables.tf: Proxmox placement vars (node=proxmox, datastore=Storage,
  image_datastore=local, bridge=vmbr0); LAN 192.168.1.0/24 gw .1 DNS .18;
  node_ips .50/.51/.52; sizing 6 vCPU / 16384 MiB / 40+100 GiB; cpu_type
  host; Cilium VIP range .53-.60.
- main.tf: proxmox_virtual_environment_download_file (cloud image -> local),
  per-node user-data snippet (proxmox_virtual_environment_file), and
  proxmox_virtual_environment_vm per node via name-keyed for_each (imported
  OS disk virtio0, blank data disk virtio1, native initialization ip_config
  + dns + user_data_file_id).
- remove moved.tf (libvirt-specific count->for_each state migration).

Authoring only; no apply. Traces R-ENV-01, R-PLAT-01.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 10:23:13 -05:00
jakman85 4f28ba6945 Merge chore/free-memory-disable-kubevirt (relieve OOM) 2026-08-25 09:29:30 -05:00
jakman85 971c730400 chore(memory): temp-disable KubeVirt to relieve node memory pressure
The 3x 6GB nodes are OOM (k3s-dev-2's kubelet stopped; dev-1/dev-3 <600MB free). KubeVirt runs
a full control plane (~1.5-2GB) for ZERO running VMs — its IaaS conformance (R-PLAT-03/04,
R-NET-01, R-STATE-01) is already certified, so disabling the running pods loses no delivered
capability. Re-enable (uncomment kubevirt in platform + platform-config) once the VMs' RAM is
grown. Manifests retained; reversible.
2026-08-25 09:29:29 -05:00
jakman85 c16e0889fd Merge fix/internet-egress-and-tags (world egress per class + image tags) 2026-08-25 09:02:06 -05:00
jakman85 7e8ae472c9 fix(net): renderer emits world/internet egress allow per egress class + fix jackett/plex tags
Root cause the media stack surfaced: the default-deny baseline is DNS-only egress (correct for
R-NET-01 east-west), but the egress CLASS (internet/vpn) had no policy ALLOWING north-south
egress — so qbittorrent/sonarr/radarr couldn't reach the internet at all (VPN downloaders
broken). Add an additive per-app CiliumNetworkPolicy '<app>-allow-internet-egress' with
toEntities:[world] (non-cluster); the egress-class steering + kill-switch (ADR-0015) govern the
PATH (internet=WAN, vpn=tun0). East-west stays denied except declared dependencies; R-NET-01
baseline unchanged. Re-rendered all apps + regenerated L1 goldens. Also fixed jackett + plex
image tags (bad pins -> :latest; Renovate/update policy pins later).
2026-08-25 09:02:05 -05:00
jakman85 f9e96402c0 chore(arr): scrub unscaffolded R-STATE-04 token from bulk-storage comments (gate hygiene) 2026-08-25 08:46:08 -05:00
jakman85 c8c862908b Merge feature/onboard-arr: Sonarr + Radarr (VPN + arr->downloader allows) 2026-08-25 08:45:56 -05:00
jakman85 ca1239c2c1 Merge feature/onboard-vpn-downloaders: qBittorrent + Jackett (VPN egress) 2026-08-25 08:45:42 -05:00
jakman85 aea8fca966 Merge chore/longhorn-capacity-plex-size 2026-08-25 08:45:12 -05:00
jakman85 4f74c8a2f0 chore(storage): Longhorn over-provisioning 100->200% + right-size plex config 20->8Gi
The 39Gi/node Longhorn data disks at 1:1 over-provisioning can't fit the media stack
(plex 20Gi replica failed: 'insufficient storage'). Volumes are thin-provisioned config-only,
so 200% lets the full catalog fit without a disk resize; physical fill risk is low. Also
right-sized plex config (20Gi was excessive for metadata). Prod-grade path: grow the data
disks (WS-STORAGE follow-up).
2026-08-25 08:45:12 -05:00