Commit Graph
14 Commits
Author SHA1 Message Date
jakman85andClaude Opus 4.8 cbc2301b30 docs: reconcile docs/comments with post-upgrade cluster state
Text-only cleanup after the upgrade campaign. Updates stale version
strings, prose, and comments to the verified current facts; no
Kubernetes behavior or version pins changed except the kubectl
toolchain bump (below).

Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie,
Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB,
dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18,
Grafana 13.2.0.

- Version strings across infra/ and platform/ READMEs + comments
- kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance
  versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew)
- Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms
  with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh)
- COMPLETED/historical banners on the three executed plan docs
- Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges
- Appended dated DECISIONS-LOG entry (2026-08-30) superseding the
  2026-08-26 LIVE STATE line

Left as revalidation items (not blind-edited): external-snapshotter
v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1,
conformance ImageVolume feature-gate assertions under 1.36.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-30 15:13:12 -05:00
jakman85andClaude Opus 4.8 36a1a4e1f9 refactor(apps): split monolithic apps Kustomization into dependency waves
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.

- catalog/render.py: compute waves by longest dependency-path depth over the
  ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
  consumer lands strictly higher — the "DB below consumer" rule falls out of
  longest-path, no special case). `--emit-waves` generates the per-wave
  kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
  wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
  wave 1: sonarr radarr nextcloud semaphore gitea-actions
  wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
  chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.

Cutover: preceded by disabling prune on `apps` (1bfb0db) so removing that
object here does not GC its resources; the wave Kustomizations SSA-adopt the
byte-identical manifests in place (no restart, no PVC churn).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-26 12:19:00 -05:00
jakman85andClaude Opus 4.8 c6301a79c5 fix(bootstrap): subshell the repo_root fallback so trailing pwd doesn't corrupt the path
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-25 12:27:34 -05:00
jakman85andClaude Opus 4.8 37f4fa6767 feat(updates): install Flux image controllers + cluster-wide write-back engine (R-UPD-02)
Install image-reflector-controller + image-automation-controller as part of the
flux-system install (bootstrap.sh --components-extra) so the image.toolkit CRDs
pre-exist before reconcile — no F-MON1 split needed for the ImageUpdateAutomation CR.
Refine the cluster-wide ImageUpdateAutomation for machine-authored git write-back
(setter strategy, path ./apps) reusing the flux-system GitRepository secretRef, and
document the operator git-write-credential hand-off + SOPS secret template
(gitea-image-automation-credentials). image-automation stays deferred in
platform/kustomization.yaml until that credential is provisioned.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:14:39 -05:00
jakman85andClaude Opus 4.8 91aa7acab8 wip(vpn): gluetun custom-provider mode with operator's PIA .ovpn
PIA-provider server selection is flaky here (random dead IPs); the operator's .ovpn
uses sweden.privacy.network (load-balanced, reliable). Switch to custom mode
(VPN_SERVICE_PROVIDER=custom, OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn), mount the
pia-custom-ovpn Secret, keep creds + kill-switch + tun-mtu 1300. Drops PIA-provider
server list/updater and native PF (Phase-6 concern). Pending the operator-provisioned
pia-custom-ovpn Secret (not merged until it exists).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-23 00:40:08 -05:00
jakman85andClaude Opus 4.8 1a98429434 feat(phase3): author shared VPN egress gateway (Gluetun/PIA/WireGuard) — author-first
Per ADR-0013: platform/vpn-egress/ = a single shared Gluetun gateway (PIA over WireGuard,
built-in kill-switch FIREWALL=on, native port forwarding), namespace (non-app, NET_ADMIN +
/dev/net/tun), control-API Service, and a pre-wired (commented) pia-credentials SOPS line.
Defined the PIA secret location (platform/vpn-egress, keys username/password → OPENVPN_USER/
OPENVPN_PASSWORD) and updated the SOPS template accordingly.

NOT wired into platform/kustomization.yaml (Phase 3, author-first) — deploy needs PIA creds +
validation of the egress-steering mechanism (Cilium Egress Gateway → the Gluetun node's wg0;
documented in README with options + a fail-closed rationale, deliberately NOT committed as an
untested CiliumEgressGatewayPolicy). WireGuard chosen (UDP, protocol != TCP); PIA supports it on
all regions and the secret stays username/password.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-21 22:19:22 -05:00
jakman85 06fa317063 Added fixes before token usage cutoff. Added required credentials for dns api for cert-manager and external-dns 2026-08-16 23:12:25 -05:00
jakman85andClaude Opus 4.8 4f3be7ef79 docs(sops): exact Cloudflare token creation steps (Zone scope, not Account)
The terse 'Zone->DNS->Edit + Zone->Read' sent the operator to the Account
permission scope. Spell out switching the scope dropdown to Zone + the exact
two Zone-scoped rows + Specific-zone resource.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 22:41:02 -05:00
jakman85andClaude Opus 4.8 a108d5ab98 correction: credential-forward + unified graph; activate SOPS (author-first)
Operator directive: don't diverge architecture to avoid asking for credentials;
provision them upfront via SOPS and keep the existing (unified) graph.

Roadmap reconciliation: Phase 2 = Network/Storage/TLS/DNS/Ingress, so
cert-manager/externaldns/gateway (the credential-gated pieces) ARE Phase 2;
monitoring is Phase 4. Earlier A2/rollout scope was inverted by credential-avoidance.

Reverts (credential-driven): remove clusters/dev/rollout/ + the $patch:delete's;
restore unified base graph (dev = ../base + cluster-vars + SOPS decryption,
R-ENV-02 intact); restore monitoring files (Phase 4). Kept non-credential
improvements: tofu for_each/rejoin, Kyverno parametrization + webhook excludes.

platform/kustomization.yaml now phased (single shared list): Phase-2 set =
kyverno, cilium, longhorn, cert-manager, externaldns, gateway; later components
commented with phase markers.

SOPS activated: .sops.yaml at repo root; bootstrap/sops/secret-templates/*.yaml.example
for all secrets (cloudflare x2, grafana, alertmanager-smtp, keycloak, oauth2-proxy, pia);
bootstrap/sops/CREDENTIALS.md checklist; pre-wired commented cloudflare-api-token lines
in cert-manager + externaldns; filled non-secret ACME email. Agents never touch real
secret values (operator encrypts).

Validated: kubectl kustomize dev(unified)/prod/platform(Phase-2 only), templates inert,
make -C ci gate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 22:09:20 -05:00
jakman85andClaude Opus 4.8 3df5071bd1 Codify host firewall + Gitea bootstrap as IaC (hands-off rebuild)
Per operator: the firewalld rule (VM->host:3000 for the Flux<-Gitea source) must be
IaC, not a manual command (R-ENV-01, R-PLAT-04).
- infra/ansible/playbooks/host-prereqs.yml: idempotent firewalld open on the libvirt
  zone (localhost play, become; no extra collection).
- bootstrap/gitea/bootstrap-gitea.sh: now fully automates admin + public repo + push
  (was manual instructions) — idempotent; creds stay 0600 on host, never committed.
Applying host-prereqs still needs host sudo (I lack non-interactive sudo); with
passwordless become a destroy/rebuild is fully unattended.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 14:52:08 -05:00
jakman85andClaude Opus 4.8 cc89b4b830 Autonomous session: Phase-2 platform authoring + Flux bootstrap progress
Author-only Phase-2 components (platform suspended; cluster untouched):
- platform/longhorn: 3-replica SC + bulk external-mount indirection (R-STATE)
- platform/monitoring: kube-prometheus-stack + Loki + Alloy (dev-trimmed) + 8 alert rules (R-OBS)
- platform/cilium: parity fix (k8sServiceHost=10.73.0.11 + masquerade) safe to adopt Phase 2
- conformance: partial L2 tests (worker hit session limit; incomplete — see DECISIONS-LOG K2)
- bootstrap/gitea: fix rootless-podman perms (host bind-mount -> named volume)

Flux installed on the cluster; host Gitea up + repo pushed; GitRepository BLOCKED by
host firewalld (VM->10.73.0.1:3000) — needs one operator sudo command (DECISIONS-LOG).
Known issue K1: cluster-vars substitution-var case mismatch (fix before un-suspend).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 14:32:47 -05:00
jakman85andClaude Opus 4.8 e6b4062310 Phase 1: host Gitea as the bootstrap GitOps source (resolves source-URL)
Run a single Gitea container on the host (bootstrap/gitea/), outside the cluster,
while the cluster is built; Flux pulls this repo from it over the k3s-dev NAT
gateway (http://10.73.0.1:3000/Home-Network/home-network-k3s-impl.git, public repo
=> no creds, NAT-only). Wired the flux-system GitRepository url in clusters/dev.
Migrates to in-cluster Gitea in Phase 6. Pins confirmed. Author-only; gate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 01:26:47 -05:00
jakman85andClaude Opus 4.8 84a2d762f0 Phase 1 (author-only): Flux + SOPS bootstrap + interface wiring (T-1.4)
bootstrap/sops (age-keygen refusing in-repo, .sops.yaml, install-age-key.sh
glue installing the sops-age Secret) + bootstrap/flux (bootstrap.sh: flux
install -> apply -k clusters/dev; decryption/url patches). PM wiring: place
.sops.yaml at repo root (SOPS walks to root); inline the 3 flux patches into
clusters/dev (avoids kustomize ../ load-restriction). clusters/dev builds with
decryption enabled; full CI gate PASS. No key material; nothing executed.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 01:21:01 -05:00
jakman85andClaude Opus 4.8 5ba9238cc1 Scope Phase 1: dev/test cluster bring-up on local KVM
Add infra/PHASE1-SCOPE.md (grounded in local recon: 24c/22GiB-avail/137G,
AMD-V+libvirt ready; OpenTofu/age/sops/flux missing) + infra/ and bootstrap/
structure stubs. Maps T-1.1..T-1.5; resource/network plan; risks; acceptance
gates; operator decisions. No spec edits; gate green.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 00:48:48 -05:00