Commit Graph
100 Commits
Author SHA1 Message Date
jakman85andClaude Opus 4.8 2e5fdd29d7 fix(vpn): gluetun update needs -enduser flag in the init container
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 23:17:16 -05:00
jakman85andClaude Opus 4.8 a735784f94 fix(vpn): initContainer fetches PIA's current server list (bundled list stale)
UPDATER_PERIOD only refreshes AFTER the tunnel is ready (chicken-and-egg: can't
connect with stale servers), and gluetun's bundled PIA Sweden list is retired
46.246.x (current is 158.173.x). Add an `update-servers` initContainer running
`gluetun update -providers "private internet access"` into a shared /gluetun
emptyDir the main container reads, so it dials current servers. UPDATER_PERIOD
raised to 720h (docs-compliant) for ongoing in-pod refresh.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 23:14:42 -05:00
jakman85 16bf3e3218 Merge feature/vpn-updater-stale-servers into main 2026-08-22 23:08:20 -05:00
jakman85andClaude Opus 4.8 457edb7a38 fix(vpn): enable gluetun server-list updater (bundled list was stale) + re-enable
CORRECTION: the connection failures were gluetun dialing RETIRED PIA IPs (46.246.x
= dead → timeouts, which I mis-read as a network block). A fresh PIA .ovpn connects
from the same host, and PIA's current SE Stockholm servers (158.173.x) are reachable.
Cilium was independently cleared (default-allow, host-fw off, Hubble FORWARDED).

Fix: UPDATER_PERIOD=24h so gluetun fetches PIA's current servers; keep
OPENVPN_MTU/MSSFIX=1350 for the real Cilium-VXLAN 1450 pod MTU (data channel);
PIA default UDP port. Re-enable vpn-egress. Full correction in DECISIONS-LOG.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 23:08:18 -05:00
jakman85 50a416077c Merge feature/vpn-network-blocks-pia into main 2026-08-22 18:15:14 -05:00
jakman85andClaude Opus 4.8 95a55952cc fix(vpn): disable vpn-egress — network blackholes PIA IPs (not ports/MTU/cluster)
Decisive diagnosis: OpenVPN's outbound init packets are 14 bytes with zero
replies (so MTU was never the blocker), pod UDP egress is healthy (NTP replies),
but TCP 501/443 to PIA IPs time out while Google/Cloudflare are OPEN — across many
PIA servers. This 192.168.2.x WiFi segment blackholes PIA's IP ranges (and
intercepts DNS). Not code-fixable. Reverted the port/MTU red-herring env to PIA
defaults; disabled vpn-egress pending an operator network fix (move to the
unfiltered 192.168.1.x LAN, or unblock PIA on the router). Full trace in
DECISIONS-LOG.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 18:15:11 -05:00
jakman85 9fc117e06d Merge feature/vpn-openvpn-mtu into main 2026-08-22 18:03:47 -05:00
jakman85andClaude Opus 4.8 f17c46b767 fix(vpn): shrink OpenVPN tun-mtu/mssfix to 1350 (Cilium VXLAN pod MTU 1450)
Actual root cause (operator-directed layered MTU check): Cilium runs VXLAN
(routing-mode tunnel) so pods get MTU 1450 (1500-50 encap) while node, hypervisor,
and the Google Fiber WAN are all 1500. OpenVPN's ~1500-byte handshake packets
exceed 1450 and are dropped (PMTUD broken). The ports were a red herring.

Fix: OPENVPN_MTU=1350 + OPENVPN_MSSFIX=1350 so encrypted packets (~1402) fit under
1450. Keeps OpenVPN + VXLAN unchanged. Re-enable vpn-egress. Full layered trace
(pod/node/host) in DECISIONS-LOG.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 18:03:46 -05:00
jakman85 5a8fd4f20a Merge feature/vpn-escalate-wireguard into main 2026-08-22 16:47:18 -05:00
jakman85andClaude Opus 4.8 0aab1702e0 chore(vpn): disable vpn-egress, escalate OpenVPN→WireGuard decision
OpenVPN can't connect on this network (each port fails differently, so it's
OpenVPN-specific not a UDP block — NTP/123 works): 1197 = home OpenVPN-server
conflict; 53 = DNS interception; 8080 = TLS handshake fails across servers (MTU
on WiFi+triple-NAT). Recommend pivoting to WireGuard via thrnz/docker-wireguard-pia
(UDP 1337, MTU-robust, ADR-0013 fallback, operator's original preference) — a
spec-level (ADR-0013) change, so escalated. Disabled meanwhile. Full trace in
DECISIONS-LOG.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 16:47:18 -05:00
jakman85 5674458754 Merge feature/vpn-port-8080 into main 2026-08-22 16:39:30 -05:00
jakman85andClaude Opus 4.8 4cde690008 fix(vpn): PIA OpenVPN port 53 -> 8080 (53 is DNS-intercepted on this network)
Port 53 also got no PIA replies: this network transparently intercepts outbound
:53, so dig works but OpenVPN-over-53 is swallowed. An NTP/123 probe from the host
returned fine, proving arbitrary non-53 UDP works bidirectionally (so it's
port-specific: 1197 conflict + 53 DNS-intercept, not a UDP block). Move to 8080
(community-verified for PIA, non-DNS, non-1197).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 16:39:30 -05:00
jakman85 7a88863f71 Merge feature/vpn-port-string-fix into main 2026-08-22 16:33:38 -05:00
jakman85andClaude Opus 4.8 734093cf92 fix(vpn): hardcode OPENVPN_ENDPOINT_PORT "53" as string (Flux var int-coercion)
Wiring the port via "${PIA_OPENVPN_PORT}" made Flux/kustomize emit an unquoted
53 (int) after substitution, which k8s rejects for an env value ("expected
string, got 53"). Same gotcha as the Longhorn numberOfReplicas fix. Hardcode the
quoted string; cluster-var removed with a NOTE explaining why.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 16:33:38 -05:00
jakman85 f73719a474 Merge feature/vpn-openvpn-port-53 into main 2026-08-22 16:28:38 -05:00
jakman85andClaude Opus 4.8 ec5ceb328f fix(vpn): PIA OpenVPN on UDP/53 (avoid home WAN:1197 conflict); re-enable vpn-egress
Operator identified the block: the home network NATs its own OpenVPN server on
WAN:1197, colliding with the cluster's outbound PIA-on-1197 flow (port-specific,
not a general UDP block — UDP/53 worked in tcpdump). PIA accepts OpenVPN on
53/1194/1197/1198/8080/9201; chose 53 (empirically proven to traverse this net).

- cluster-vars: add PIA_OPENVPN_PORT="53"
- gluetun: OPENVPN_ENDPOINT_PORT=${PIA_OPENVPN_PORT}
- re-enable vpn-egress; DECISIONS-LOG resolution noted

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 16:28:37 -05:00
jakman85 347f7a54b6 Merge feature/vpn-blocked-network into main 2026-08-22 15:05:21 -05:00
jakman85andClaude Opus 4.8 9d761872ea fix(vpn): disable vpn-egress — tunnel blocked by home network (env), config kept
Phase-3 VPN config is complete + correct (OpenVPN-UDP, SE Stockholm), but the
OpenVPN UDP/1197 handshake to PIA never gets a reply. Traced with tcpdump: packets
leave pod→node→host→WiFi (SNAT 192.168.2.103) fine, but 0 replies return to the
host, while host UDP/53 works. The 192.168.2.x home router/ISP drops OpenVPN UDP —
upstream of everything in this repo. Reverted to disabled to avoid a crashloop
firing false KubePodCrashLooping alerts; one-line re-enable once the network is
fixed. Full trace in DECISIONS-LOG.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 15:05:20 -05:00
jakman85 e4d856fd83 Merge feature/pia-region-fix into main 2026-08-22 14:51:29 -05:00
jakman85andClaude Opus 4.8 9a108443e8 fix(vpn): PIA_REGION Sweden -> "SE Stockholm" (Gluetun OpenVPN region name)
Gluetun's PIA OpenVPN provider rejects "Sweden"; the valid PF-capable Sweden
region identifier is "SE Stockholm" ("SE Streaming Optimized" lacks port
forwarding). Preserves the operator's Sweden choice.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 14:51:29 -05:00
jakman85 b50433d705 Merge feature/vpn-openvpn-enable into main 2026-08-22 14:49:17 -05:00
jakman85andClaude Opus 4.8 015b423353 feat(vpn): enable vpn-egress on OpenVPN-UDP; align docs to ADR-0013 amendment
Gluetun cannot do PIA over WireGuard (upstream unsupported; live "provider name
is not valid for Wireguard"). Operator approved OpenVPN-UDP and ADR-0013 was
amended in the spec. This aligns the impl:

- platform/kustomization.yaml: re-enable `- vpn-egress` (Phase 3, OpenVPN-UDP)
- gluetun-gateway.yaml: clean stale WireGuard comments (header, image note,
  VPN_TYPE/creds comments); config already VPN_TYPE=openvpn
- README.md: PIA-over-OpenVPN, status=enabled, steering iface wg0 -> tun0
- DECISIONS-LOG.md: disk/drive incident (k3s -> NVMe; failing Seagate) + Phase-3
  OpenVPN resolution

Stale WireGuard comments inside the operator's encrypted pia-credentials.sops.yaml
header left untouched (editing breaks the SOPS MAC; cosmetic).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 14:49:16 -05:00
jakman85 41da3d96e5 fix(vpn): Attempting to try OpenVPN protocol with PIA 2026-08-21 23:49:49 -05:00
jakman85 6a2c7c3020 Merge Gluetun v3.41.0 (PIA WireGuard support) 2026-08-21 23:10:48 -05:00
jakman85andClaude Opus 4.8 13a8df4e4a fix(vpn): bump Gluetun to v3.41.0 (v3.40.0 lacks PIA WireGuard support)
The deployed v3.40.0 rejected PIA+WireGuard ('private internet access is not valid for
Wireguard'). Verified v3.41.0 accepts the PIA+WireGuard provider/protocol (passes config
validation; ADR-0013's model holds — v3.40.0 was just too old). Bump the pin.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-21 23:10:47 -05:00
jakman85andClaude Opus 4.8 cbcfae01e1 feat(phase3): enable vpn-egress (deploy the Gluetun PIA/WireGuard gateway)
PIA creds provisioned (SOPS) + wired. Enable vpn-egress in platform/kustomization.yaml
to deploy the shared Gluetun gateway (PIA over WireGuard, Sweden region, kill-switch).
Egress-class steering (vpn-class pods -> tunnel) still to be validated against the live
tunnel next. Gate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-21 23:06:27 -05:00
jakman85 643c3225ac Merge PIA_REGION=Sweden default 2026-08-21 23:00:51 -05:00
jakman85andClaude Opus 4.8 0d55c793f3 fix(vpn): default PIA_REGION to Sweden (operator choice, PF-capable)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-21 23:00:51 -05:00
jakman85 9667dce419 Merge VPN cluster-vars definition + pod-CIDR fix 2026-08-21 22:56:15 -05:00
jakman85andClaude Opus 4.8 6591e11ad1 fix(vpn): define PIA_REGION + CLUSTER_POD_CIDR cluster-vars; correct pod CIDR + doc where
gluetun-gateway.yaml referenced ${PIA_REGION}/${CLUSTER_POD_CIDR} but they were never defined
in cluster-vars, and the README didn't say where to set them. Add both to
clusters/base/cluster-vars.yaml (ConfigMap data:, string values, Flux postBuild substitution).
CLUSTER_POD_CIDR is 10.0.0.0/8 (Cilium cluster-pool — pods get 10.x, confirmed live), NOT the
k3s 10.42.0.0/16 I'd assumed. README now names the file + format + per-env override + PF-region note.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-21 22:56:14 -05:00
jakman85 140cb651d2 Merge Phase-3/4 + infra-recovery log 2026-08-21 22:40:18 -05:00
jakman85 2223e8403d docs: Phase-4 monitoring deployed + Phase-3 VPN authored + infra recovery + F-MON1 2026-08-21 22:40:17 -05:00
jakman85 4f17c5334e Merge monitoring PrometheusRule ordering fix 2026-08-21 22:30:02 -05:00
jakman85andClaude Opus 4.8 eeb97208fa fix(monitoring): move PrometheusRule alerts to platform-config (CRD ordering)
Enabling monitoring deadlocked the platform Kustomization: alerts.yaml ships
PrometheusRule CRs whose monitoring.coreos.com CRDs are installed by the
kube-prometheus-stack HelmRelease in the SAME pass → 'no matches for kind
PrometheusRule' → atomic apply failure → NO monitoring HelmReleases created.
Same controllers/configs fix as cert-manager/kyverno: drop alerts.yaml from
platform/monitoring and apply it via platform-config (dependsOn platform), so the
monitoring CRDs exist first. Verified: platform has 0 PrometheusRule (keeps the 3
monitoring HRs); platform-config has the 8 alert rules; gate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-21 22:30:01 -05:00
jakman85 3bfd055df2 Merge Phase-3 VPN egress gateway (author-first) 2026-08-21 22:19:24 -05:00
jakman85andClaude Opus 4.8 1a98429434 feat(phase3): author shared VPN egress gateway (Gluetun/PIA/WireGuard) — author-first
Per ADR-0013: platform/vpn-egress/ = a single shared Gluetun gateway (PIA over WireGuard,
built-in kill-switch FIREWALL=on, native port forwarding), namespace (non-app, NET_ADMIN +
/dev/net/tun), control-API Service, and a pre-wired (commented) pia-credentials SOPS line.
Defined the PIA secret location (platform/vpn-egress, keys username/password → OPENVPN_USER/
OPENVPN_PASSWORD) and updated the SOPS template accordingly.

NOT wired into platform/kustomization.yaml (Phase 3, author-first) — deploy needs PIA creds +
validation of the egress-steering mechanism (Cilium Egress Gateway → the Gluetun node's wg0;
documented in README with options + a fail-closed rationale, deliberately NOT committed as an
untested CiliumEgressGatewayPolicy). WireGuard chosen (UDP, protocol != TCP); PIA supports it on
all regions and the secret stays username/password.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-21 22:19:22 -05:00
jakman85 e6c61c97b3 Merge Phase-4 monitoring deploy 2026-08-21 22:06:24 -05:00
jakman85andClaude Opus 4.8 dfde5292d7 feat(phase4): deploy monitoring (kube-prometheus-stack + Loki + Alloy + Alertmanager)
Enable monitoring in platform/kustomization.yaml (Phase 4). Grafana admin +
Alertmanager SMTP secrets are provisioned (SOPS) and wired; Longhorn SC is up for
Prometheus/Loki PVCs. Set the Alertmanager receiver 'to:' to a valid inbox
([email protected]) so the config loads (REPLACE-ME is not a valid address);
operator can change it. NOTE: smtp_from ([email protected]) differs from
the Gmail auth user — Gmail may reject/rewrite unless a send-as alias is configured;
that only affects email delivery at alert time, not stack health.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-21 22:06:23 -05:00
jakman85 75764580d8 Merge fail-closed green log 2026-08-17 17:43:27 -05:00
jakman85 e8b5cae34d docs: R-NET fail-closed (L3) green; conformance status 2026-08-17 17:43:27 -05:00
jakman85 cde3335086 Merge L3 default-deny-failclosed (R-NET fail-closed green) 2026-08-17 17:42:35 -05:00
jakman85andClaude Opus 4.8 1bf8e3bfc4 feat(conformance): implement L3 default-deny-failclosed (R-NET-01/02, fault injection)
Was a scaffold (fixture TODO, fault-injection a no-op echo). Implemented against the
live datapath: fixtures/allowed-pair.yaml = default-deny baseline + source + dest(:8080)
+ the sole explicit allow. The test verifies source->dest works (ALLOWED_OK), then
REVOKES both halves of the allow (real fault: kubectl delete networkpolicy), and asserts
connectivity fails CLOSED (FAIL_CLOSED_OK) while DNS still resolves (DNS_OK). Proves
default-deny is datapath-enforced and fails closed on revocation. DNS check asserts the
resolved answer, since busybox nslookup exits non-zero on a successful forward lookup.
PASSES (74.8s).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 17:42:34 -05:00
jakman85 1631a7bde8 Merge R-STATE-01 + harness fix log 2026-08-17 17:02:40 -05:00
jakman85andClaude Opus 4.8 ce55332028 docs: R-STATE-01 green + chainsaw =~ harness fix (F-TEST1)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 17:02:39 -05:00
jakman85 76478f2651 Merge chainsaw check block-style fix 2026-08-17 17:02:05 -05:00
jakman85andClaude Opus 4.8 70c0701d6f fix(conformance): block-style chainsaw checks (flow-map broke on contains comma)
My previous commit converted =~ to contains() but kept flow style
'check: { (contains($stdout, 'X')): true }' — the comma inside contains() is a
YAML flow-map separator, so 9 tests failed yamllint ('expected , or }'). Convert
those checks to block style (matching data-survival), where a comma in the key
scalar is fine. Gate PASS; data-survival still green.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 17:02:05 -05:00
jakman85 3440bbad92 Merge chainsaw check contains() fix (data-survival R-STATE-01 green) 2026-08-17 16:58:38 -05:00
jakman85andClaude Opus 4.8 e0c5c58710 fix(conformance): chainsaw checks use contains(), not unsupported =~
Every chainsaw `check` used the regex operator `=~` (`($stdout =~ 'X'): true`),
which chainsaw's expression engine rejects ('SyntaxError: Unknown char: ~') — so
EVERY such test failed on the assertion even when the behavior passed. Proven with
data-survival (R-STATE-01): the Longhorn canary genuinely SURVIVED pod destruction
(DATA_SURVIVED_OK) but the test FAILED on the `=~` syntax. Converted all 10 tests to
JMESPath `contains($stdout, 'X')` (data-survival's alternation → contains||contains).
data-survival now PASSES; the other 9 (later-phase components) are syntactically fixed
for when their phases deploy.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 16:58:38 -05:00
jakman85 54fd963cc4 Merge F-OPS1 resolved log 2026-08-17 16:51:30 -05:00
jakman85andClaude Opus 4.8 429708f0a7 docs: F-OPS1 resolved (self-managing root)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 16:51:30 -05:00
jakman85 4381a71964 Merge self-managing root Kustomization 2026-08-17 16:48:58 -05:00
jakman85andClaude Opus 4.8 d06947cbfd feat(gitops): self-managing root Kustomization (fixes F-OPS1)
The on-cluster cluster-vars ConfigMap wasn't reconciled by Flux — this bootstrap
model had no Kustomization watching clusters/dev, so cluster-vars/overlay changes
needed a manual 'kubectl apply -k clusters/dev' (e.g. DNS_ZONE substituted empty
until re-applied). Add clusters/dev/sync.yaml: a self-managing root Kustomization
(flux2 gotk-sync pattern, name flux-system) that reconciles ./clusters/dev from git
— cluster-vars, the GitRepository, platform/platform-config/apps, and itself. A git
push now suffices. Bootstrap unchanged ('kubectl apply -k clusters/dev' creates it
once). Verified: dev build contains all 6 objects (root + existing), prod unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 16:48:57 -05:00
jakman85 3c8b339463 Merge kyverno image fix log 2026-08-17 08:43:08 -05:00
jakman85andClaude Opus 4.8 892fdc6a15 docs: log kyverno bitnami/kubectl cleanup-image fix (F-KY1)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 08:43:08 -05:00
jakman85 928cd5e122 Merge kyverno cleanup image fix (bitnamilegacy) 2026-08-17 08:38:53 -05:00
jakman85andClaude Opus 4.8 5521059c04 fix(kyverno): replace purged bitnami/kubectl cleanup image with bitnamilegacy
The kyverno chart's cleanup jobs (policyReportsCleanup, webhooksCleanup) default to
bitnami/kubectl:1.30.2, which Bitnami removed from Docker Hub (2024 catalog purge) ->
ImagePullBackOff. policyReportsCleanup is a Helm post-upgrade HOOK, so its failure
stalls every helm upgrade (5m timeout -> HelmRelease not-Ready -> platform Kustomization
blocked -> platform-config/apps cascade). Override both to Bitnami's relocated
bitnamilegacy/kubectl (identical image: /bin/bash + awk/wc/xargs + kubectl). Verified
via helm template that no bitnami/kubectl reference remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 08:38:53 -05:00
jakman85 de81bafdc1 Merge Phase-2 conformance exit decision log 2026-08-17 08:33:24 -05:00
jakman85andClaude Opus 4.8 f78ce4ebe7 docs: Phase-2 conformance exit achieved (R-NET/R-TLS/R-DNS green) + findings
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 08:33:24 -05:00
jakman85 17622069fc Merge Phase-2 TLS/DNS conformance tests + ExternalDNS zone fix 2026-08-17 08:24:53 -05:00
jakman85andClaude Opus 4.8 20de78f5eb feat(conformance): author R-TLS + R-DNS L2 tests; fix ExternalDNS zone filter
Phase-2 exit tests (were Phase-0 scaffolds), authored against the live platform:
- trusted-tls (R-TLS-01/03): in-cluster probe dials the gateway Service (env-agnostic
  via --connect-to) with wildcard SNI and verifies the served cert CHAINS TO THE LE
  STAGING ROOT (operator decision A; dev uses LE-staging per ADR-0006). Fixture embeds
  the '(STAGING) Pretend Pear X1' root. Probes public + internal-only hostnames (R-TLS-03).
  PASSES.
- dns-exposure (R-DNS-02): a LoadBalancer Service with an external-dns hostname; polls
  public DNS until ExternalDNS auto-publishes the record; teardown + policy=sync removes it.

ExternalDNS fix (root cause of the DNS test): domain-filter was the platform SUBDOMAIN
home.keithjackson.dev, which filters OUT the actual Cloudflare zone keithjackson.dev
('no hosted zone matching record DNS Name' — record generated but skipped). Add a
DNS_ZONE cluster-var (keithjackson.dev) and set domainFilters to it. policy=sync +
txt-registry still only manage ExternalDNS-owned records (the pre-existing wildcard is
untouched). Resolves the file's WS-NET R-DNS-02 TODO.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 08:24:51 -05:00
jakman85 15048f3889 Merge autonomous session 2 decision log 2026-08-17 07:41:32 -05:00
jakman85andClaude Opus 4.8 8732c8f4fd docs: autonomous session 2 (R-NET conformance green, TLS verified, SC + report fixes)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 07:41:32 -05:00
jakman85 ee5ad90258 Merge conformance report-path CWD fix 2026-08-17 07:40:04 -05:00
jakman85andClaude Opus 4.8 416e16bdbf fix(conformance): pass absolute --report-path so run.sh is CWD-robust
run.sh exited non-zero (breaking the L2 gate) when invoked from the repo root:
chainsaw's config report.path is relative (_artifacts), so it resolved against
$PWD and failed to write ('open _artifacts/chainsaw-report.xml: no such file')
even though all tests passed. Pass --report-path ${ARTIFACTS} (absolute) +
mkdir -p first. Verified: R-NET-01 from repo root now exits 0 with report written.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 07:40:03 -05:00
jakman85 7a45eb105a Merge k3s local-storage disable (single default SC) 2026-08-17 07:34:32 -05:00
jakman85andClaude Opus 4.8 4cc04147a2 fix(k3s): disable local-storage so Longhorn is the sole default StorageClass
The deployed cluster has TWO default StorageClasses — k3s's built-in local-path
AND longhorn — which is ambiguous for PVCs without an explicit class. Longhorn
(ADR-0003) is the cluster storage, so add local-storage to k3s --disable. Effective
on fresh installs (the k3s install task is idempotent → no restart of the running
cluster); the running dev cluster keeps both defaults harmlessly (all platform PVCs
pin storageClassName: longhorn). Rebuilds get a single default.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 07:34:31 -05:00
jakman85 a07151ab7c Merge Phase-2 deployed decision log 2026-08-17 01:53:13 -05:00
jakman85andClaude Opus 4.8 0811e2533b docs: Phase-2 platform fully deployed + green (fixes 4-6, follow-ups)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 01:53:13 -05:00
jakman85 59dd3d9526 Merge cert-manager DNS-01 public-resolver fix 2026-08-17 01:44:29 -05:00
jakman85andClaude Opus 4.8 cdb546bc6e fix(cert-manager): DNS-01 self-check via public resolvers (split-horizon)
The wildcard cert stalled: challenge Presented=true (Cloudflare TXT created OK —
token works) but propagation self-check looped 'DNS record not yet propagated'.
Verified the record IS publicly resolvable (1.1.1.1/8.8.8.8); the NAT'd k3s-dev
cluster's own DNS just can't see it. Set dns01RecursiveNameservers=1.1.1.1:53,
8.8.8.8:53 + dns01RecursiveNameserversOnly=true so the self-check bypasses cluster
DNS. Realises the pre-existing TODO in this file.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 01:44:28 -05:00
jakman85andClaude Opus 4.8 c6331c84bf fix(kyverno): correct chart webhook schema (map + forceFailurePolicyIgnore)
Kyverno HelmRelease install failed at Helm templating: config.webhooks was a
LIST [{failurePolicy, namespaceSelector}], but the kyverno 3.3.9 chart expects
config.webhooks to be a MAP (webhooks.namespaceSelector) — the chart helper reads
.namespaceSelector off the value directly ('can't evaluate field namespaceSelector
in type []interface {}'). Also, failurePolicy is not a per-webhook field here; the
chart's global switch is config.forceFailurePolicyIgnore.enabled (bool).

Fix: config.webhooks -> map with namespaceSelector (system+platform excludes);
add config.forceFailurePolicyIgnore.enabled = ${KYVERNO_FORCE_FAILURE_POLICY_IGNORE}.
Renamed the cluster-var KYVERNO_WEBHOOK_FAILURE_POLICY (Fail/Ignore) ->
KYVERNO_FORCE_FAILURE_POLICY_IGNORE (false base / true dev). Verified with
helm template (renders 73 objects, no error); gate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 01:26:04 -05:00
jakman85 827d82f0e5 Merge autonomous-session decision log 2026-08-17 01:08:35 -05:00
jakman85andClaude Opus 4.8 4260487af5 docs: log Phase-2 un-suspend + fixes (longhorn coercion, controllers/configs split, cilium L2)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 01:08:35 -05:00
jakman85 458a867cfc Merge platform controllers/configs split into main 2026-08-17 01:06:24 -05:00
jakman85andClaude Opus 4.8 ca982864fc fix(platform): controllers/configs split so CRDs precede their CRs
Un-suspend deadlocked: the single platform Kustomization applies HelmReleases
(which register CRDs) AND custom resources of those CRDs (ClusterIssuer,
Certificate, Kyverno ClusterPolicies) in ONE pass. Flux's server-side apply
fails atomically ('no matches for kind ClusterIssuer') → nothing installs, so
the CRD-registering HelmReleases never run either. Classic ordering deadlock.

Fix (flux2 controllers/configs pattern, applies to dev+prod equally — no
R-ENV-02 divergence): add a platform-config Flux Kustomization (dependsOn
platform, wait) carrying the CRs; platform keeps only the controllers.
- New clusters/base/platform-config.yaml (path ./platform-config, substituteFrom
  cluster-vars, no decryption — no SOPS content); wired into base + apps now
  dependsOn platform-config (transitively waits for platform).
- platform-config/ references the CR files IN PLACE via ../platform/... (Flux
  builds LoadRestrictionsNone; Kyverno policies must stay under platform/kyverno/
  for the CI L0  gate). CR resources dropped from the platform
  component kustomizations (cert-manager, gateway, kyverno).
- Authored the missing Cilium L2 resources (were only a TODO): CiliumLoadBalancerIPPool
  (dev NAT 10.73.0.240-.250) + CiliumL2AnnouncementPolicy (ens*), so Gateway
  LoadBalancer listeners get a VIP. Prod range must be parameterized later.

Verified: platform build has zero CRs; platform-config build has 2 ClusterIssuers,
6 ClusterPolicies, Certificate, Gateway, both Cilium L2 CRs; dev/prod build; gate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 01:06:23 -05:00
jakman85 c04aea85aa Merge Longhorn StorageClass numeric-coercion fix into main 2026-08-17 00:53:04 -05:00
jakman85andClaude Opus 4.8 dafe0e79bb fix(longhorn): hardcode StorageClass numberOfReplicas to quoted "3"
The un-suspend reconcile stalled: platform Kustomization Ready=False with
'StorageClass/longhorn ... .parameters.numberOfReplicas: expected string, got 3'.
StorageClass.parameters is map[string]string, but kustomize strips the quotes
around the bare "${LONGHORN_DEFAULT_REPLICA_COUNT}" placeholder, so Flux postBuild
substitution produced an unquoted numeric that server-side apply rejects as int.
Hardcode the quoted numeric literal "3" (kustomize preserves those, as "30" shows);
topology is locked at 3 nodes. The replica knob still drives the global default via
defaultSettings.defaultReplicaCount in the helmrelease (a Helm int value, unaffected).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 00:53:03 -05:00
jakman85 7afc3ff55e Merge Cloudflare secret wiring + yamllint sops exemption into main 2026-08-17 00:35:46 -05:00
jakman85andClaude Opus 4.8 275ac466b2 wire Cloudflare DNS token secrets + exempt *.sops.yaml from yamllint
- Rename the committed encrypted secrets to canonical cloudflare-api-token.sops.yaml
  (cert-manager + external-dns namespaces) and uncomment the resource lines in both
  kustomizations, so cert-manager DNS-01 + ExternalDNS get the token at reconcile.
- Exempt **/*.sops.yaml from ci/yamllint.yaml: SOPS re-emits encrypted files with its
  own indentation + long ENC[...] lines (never hand-edited), which was failing the lint
  gate on main after the secrets were committed. Same treatment as **/vault.yml.

Both Secrets confirmed present + encrypted in the platform build; gate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-17 00:35:45 -05:00
jakman85 06fa317063 Added fixes before token usage cutoff. Added required credentials for dns api for cert-manager and external-dns 2026-08-16 23:12:25 -05:00
jakman85 a63c841342 Merge Cloudflare token doc clarification 2026-08-16 22:41:02 -05:00
jakman85andClaude Opus 4.8 4f3be7ef79 docs(sops): exact Cloudflare token creation steps (Zone scope, not Account)
The terse 'Zone->DNS->Edit + Zone->Read' sent the operator to the Account
permission scope. Spell out switching the scope dropdown to Zone + the exact
two Zone-scoped rows + Specific-zone resource.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 22:41:02 -05:00
jakman85 c0df86c87d Merge devcontainer KUBECONFIG + SOPS age-key persistence fix into main 2026-08-16 22:32:09 -05:00
jakman85andClaude Opus 4.8 f089adf261 devcontainer: set KUBECONFIG + persist SOPS age key to host
Two dev.sh gaps surfaced while running bootstrap/sops/install-age-key.sh:
- kubectl defaulted to localhost:8080 (connection refused) because KUBECONFIG
  was never set in the toolbox. Set KUBECONFIG=/work/infra/ansible/.cache/dev.kubeconfig
  (override via DEV_KUBECONFIG); --network host already reaches 10.73.0.11:6443.
- age-keygen.sh writes the PRIVATE key to ~/.config/sops/age/keys.txt, but that
  path was not mounted in the --rm container, so the key was lost on exit. Mount
  ${HOME}/.config/sops -> /root/.config/sops (created first) so the key persists
  on the host and SOPS_AGE_KEY_FILE resolves to it.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 22:31:51 -05:00
jakman85 ae36d67dc3 Merge credential-forward correction + SOPS activation into main 2026-08-16 22:09:21 -05:00
jakman85andClaude Opus 4.8 a108d5ab98 correction: credential-forward + unified graph; activate SOPS (author-first)
Operator directive: don't diverge architecture to avoid asking for credentials;
provision them upfront via SOPS and keep the existing (unified) graph.

Roadmap reconciliation: Phase 2 = Network/Storage/TLS/DNS/Ingress, so
cert-manager/externaldns/gateway (the credential-gated pieces) ARE Phase 2;
monitoring is Phase 4. Earlier A2/rollout scope was inverted by credential-avoidance.

Reverts (credential-driven): remove clusters/dev/rollout/ + the $patch:delete's;
restore unified base graph (dev = ../base + cluster-vars + SOPS decryption,
R-ENV-02 intact); restore monitoring files (Phase 4). Kept non-credential
improvements: tofu for_each/rejoin, Kyverno parametrization + webhook excludes.

platform/kustomization.yaml now phased (single shared list): Phase-2 set =
kyverno, cilium, longhorn, cert-manager, externaldns, gateway; later components
commented with phase markers.

SOPS activated: .sops.yaml at repo root; bootstrap/sops/secret-templates/*.yaml.example
for all secrets (cloudflare x2, grafana, alertmanager-smtp, keycloak, oauth2-proxy, pia);
bootstrap/sops/CREDENTIALS.md checklist; pre-wired commented cloudflare-api-token lines
in cert-manager + externaldns; filled non-secret ACME email. Agents never touch real
secret values (operator encrypts).

Validated: kubectl kustomize dev(unified)/prod/platform(Phase-2 only), templates inert,
make -C ci gate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 22:09:20 -05:00
jakman85 e1c738fa63 Merge Phase-2 no-secret rollout prep into main 2026-08-16 21:34:58 -05:00
jakman85andClaude Opus 4.8 d9a80e5350 phase2-prep: per-component no-secret rollout + component hardening (author-first)
Prep only — nothing deployed; running cluster untouched.

Reconciliation graph (operator-approved Option A): author clusters/dev/rollout/
= 4 per-component Flux Kustomizations (kyverno->cilium->longhorn->monitoring,
monitoring dependsOn longhorn). Dev overlay $patch:delete's the monolithic base
platform/apps and adds ./rollout, so only the no-secret tier reconciles; the
secret-blocked components are simply not created until hand-offs arrive. The
no-secret tier carries no SOPS content, so it needs no sops-age key. prod is
untouched (still builds the monolith). Convergence path coded in (delete rollout/
+ the two deletes) — R-ENV-02 divergence flagged for architect review.

Component hardening via cluster-vars (prod keeps strict posture):
- KYVERNO_VALIDATION_ACTION Enforce->Audit(dev), _WEBHOOK_FAILURE_POLICY
  Fail->Ignore(dev), _ADMISSION_REPLICAS 3->1(dev); webhook namespaceSelector
  excludes system+platform namespaces.
- MONITORING_STORAGE_CLASS var; GRAFANA_ADMIN_SECRET var (dev ''=chart default);
  Alertmanager SMTP/secret deferred to Phase-4 with verbatim re-enable markers.

Cilium parity verified against the live CNI (no-op adopt). Validated:
kubectl kustomize dev/prod/all-components, envsubst->YAML parse, make -C ci gate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 21:34:57 -05:00
jakman85 99eef2fe4c Merge IaC hardening (tofu for_each + self-contained rejoin) into main 2026-08-16 19:42:12 -05:00
jakman85andClaude Opus 4.8 95e3eafac6 iac: harden node lifecycle — tofu for_each + self-contained ansible rejoin
Implements T-1.5 findings F1/F2 (author-first, no cluster change).

F1 (tofu): refactor libvirt node resources count -> for_each keyed by
node name so single-node -replace/destroy cannot cascade to siblings.
Add ordered-list/primary helper locals; update outputs.tf off the map;
add moved.tf (12 moved blocks). tofu plan proves a pure state-address
migration: 0 to add, 0 to change, 0 to destroy (all 12 'has moved to').
State-persist via a 0-change apply is operator-permission-gated.

F2 (ansible): k3s_server secondary fetches the join token via
delegate_to the primary (published fact as fast path), so a single-node
rejoin with --limit k3s-dev-N is self-contained and no longer depends on
a fact an earlier play had to set. Syntax-check + CI gate green.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 19:42:12 -05:00
jakman85 ef33865a53 Merge T-1.5 destroy/rebuild findings into main 2026-08-16 19:35:02 -05:00
jakman85andClaude Opus 4.8 1eeb754bee docs(t-1.5): log destroy/rebuild proof outcome + IaC findings
T-1.5 destructive IaC-validity test PASSED: dev cluster torn down and
reconstituted from IaC + preserved disks + Flux git source with zero
state loss (3/3 Ready, Cilium OK, Flux READY=True). Post-rebuild
conformance L2 R-NET-01 + R-NET-02 both PASS.

Two IaC-hardening findings for Phase 2:
- F1: tofu destroy -target on a count-indexed resource over-destroyed
  all 3 domains; refactor libvirt_domain.node count -> for_each keyed by
  node name; never use destroy -target for single-node lifecycle.
- F2: single-node ansible rejoin needs the primary in --limit to publish
  the k3s join-token fact; make the join self-contained via delegate_to
  the primary so --limit k3s-dev-N is hands-off.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 19:35:01 -05:00
jakman85 6c3f9eddbd Merge feature/k2-conformance-l2-green into main 2026-08-16 15:46:01 -05:00
jakman85andClaude Opus 4.8 e20f17b896 K2 DONE: L2 network behavior tests pass against the live cluster
R-NET-01 default-deny + R-NET-02 deny-undeclared PASS via run.sh (JUnit+coverage,
exit 0). Fixed 3 real bugs found by running them:
- chainsaw strict list-matching: assert scalar 'phase: Running' not a conditions[] element.
- default-deny blocks the kubelet readiness probe -> drop probes (dest pods just need to run).
- chainsaw report path ../_artifacts resolved to repo-root -> fix to _artifacts (was exit 1).
- swap agnhost->busybox httpd listeners (cached everywhere; cache-timing-proof).
Logged the kubelet-probe finding for WS-NET's T-2.2 default-deny baseline design.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 15:46:00 -05:00
jakman85 1b6e59c074 Merge docs/log-update into main 2026-08-16 15:17:36 -05:00
jakman85andClaude Opus 4.8 53e7fbcefc docs: mark K1 resolved in decision log
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 15:17:36 -05:00
jakman85 0588faedcf Merge fix/k1-substitution-var-case into main 2026-08-16 15:16:30 -05:00
jakman85andClaude Opus 4.8 9743b50c3b fix(K1): uppercase cluster-vars keys to match component substitution refs
Flux postBuild.substituteFrom is case-sensitive; platform/ components reference
${UPPERCASE} tokens but cluster-vars keys were lowercase -> none would resolve
(literal ${...} would reach the HelmReleases). Uppercased the ConfigMap data
keys + the dev/prod overlay patch paths. Every component ref now resolves;
clusters/dev & clusters/prod build; gate green. Clears the K1 landmine before
any platform un-suspend.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 15:16:30 -05:00
jakman85 6a32fc76a5 Merge feature/t1.4-flux-reconciling into main 2026-08-16 15:14:09 -05:00
jakman85andClaude Opus 4.8 ab0a7977bf Phase 1 T-1.4 DONE: Flux reconciling clusters/dev from host Gitea
host-prereqs.yml opened the firewall (via the operator's ansible-vault become
password); Flux GitRepository READY=True (main@sha1:a2f3bdc). Commit the
encrypted hypervisor vault (ansible_become_password) so rebuild is fully
hands-off; passphrase stays out-of-repo. platform/apps still not reconciled.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 15:14:09 -05:00