PIA-provider server selection is flaky here (random dead IPs); the operator's .ovpn
uses sweden.privacy.network (load-balanced, reliable). Switch to custom mode
(VPN_SERVICE_PROVIDER=custom, OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn), mount the
pia-custom-ovpn Secret, keep creds + kill-switch + tun-mtu 1300. Drops PIA-provider
server list/updater and native PF (Phase-6 concern). Pending the operator-provisioned
pia-custom-ovpn Secret (not merged until it exists).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CORRECTION: the connection failures were gluetun dialing RETIRED PIA IPs (46.246.x
= dead → timeouts, which I mis-read as a network block). A fresh PIA .ovpn connects
from the same host, and PIA's current SE Stockholm servers (158.173.x) are reachable.
Cilium was independently cleared (default-allow, host-fw off, Hubble FORWARDED).
Fix: UPDATER_PERIOD=24h so gluetun fetches PIA's current servers; keep
OPENVPN_MTU/MSSFIX=1350 for the real Cilium-VXLAN 1450 pod MTU (data channel);
PIA default UDP port. Re-enable vpn-egress. Full correction in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Decisive diagnosis: OpenVPN's outbound init packets are 14 bytes with zero
replies (so MTU was never the blocker), pod UDP egress is healthy (NTP replies),
but TCP 501/443 to PIA IPs time out while Google/Cloudflare are OPEN — across many
PIA servers. This 192.168.2.x WiFi segment blackholes PIA's IP ranges (and
intercepts DNS). Not code-fixable. Reverted the port/MTU red-herring env to PIA
defaults; disabled vpn-egress pending an operator network fix (move to the
unfiltered 192.168.1.x LAN, or unblock PIA on the router). Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Actual root cause (operator-directed layered MTU check): Cilium runs VXLAN
(routing-mode tunnel) so pods get MTU 1450 (1500-50 encap) while node, hypervisor,
and the Google Fiber WAN are all 1500. OpenVPN's ~1500-byte handshake packets
exceed 1450 and are dropped (PMTUD broken). The ports were a red herring.
Fix: OPENVPN_MTU=1350 + OPENVPN_MSSFIX=1350 so encrypted packets (~1402) fit under
1450. Keeps OpenVPN + VXLAN unchanged. Re-enable vpn-egress. Full layered trace
(pod/node/host) in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
OpenVPN can't connect on this network (each port fails differently, so it's
OpenVPN-specific not a UDP block — NTP/123 works): 1197 = home OpenVPN-server
conflict; 53 = DNS interception; 8080 = TLS handshake fails across servers (MTU
on WiFi+triple-NAT). Recommend pivoting to WireGuard via thrnz/docker-wireguard-pia
(UDP 1337, MTU-robust, ADR-0013 fallback, operator's original preference) — a
spec-level (ADR-0013) change, so escalated. Disabled meanwhile. Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Port 53 also got no PIA replies: this network transparently intercepts outbound
:53, so dig works but OpenVPN-over-53 is swallowed. An NTP/123 probe from the host
returned fine, proving arbitrary non-53 UDP works bidirectionally (so it's
port-specific: 1197 conflict + 53 DNS-intercept, not a UDP block). Move to 8080
(community-verified for PIA, non-DNS, non-1197).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator identified the block: the home network NATs its own OpenVPN server on
WAN:1197, colliding with the cluster's outbound PIA-on-1197 flow (port-specific,
not a general UDP block — UDP/53 worked in tcpdump). PIA accepts OpenVPN on
53/1194/1197/1198/8080/9201; chose 53 (empirically proven to traverse this net).
- cluster-vars: add PIA_OPENVPN_PORT="53"
- gluetun: OPENVPN_ENDPOINT_PORT=${PIA_OPENVPN_PORT}
- re-enable vpn-egress; DECISIONS-LOG resolution noted
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Phase-3 VPN config is complete + correct (OpenVPN-UDP, SE Stockholm), but the
OpenVPN UDP/1197 handshake to PIA never gets a reply. Traced with tcpdump: packets
leave pod→node→host→WiFi (SNAT 192.168.2.103) fine, but 0 replies return to the
host, while host UDP/53 works. The 192.168.2.x home router/ISP drops OpenVPN UDP —
upstream of everything in this repo. Reverted to disabled to avoid a crashloop
firing false KubePodCrashLooping alerts; one-line re-enable once the network is
fixed. Full trace in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gluetun cannot do PIA over WireGuard (upstream unsupported; live "provider name
is not valid for Wireguard"). Operator approved OpenVPN-UDP and ADR-0013 was
amended in the spec. This aligns the impl:
- platform/kustomization.yaml: re-enable `- vpn-egress` (Phase 3, OpenVPN-UDP)
- gluetun-gateway.yaml: clean stale WireGuard comments (header, image note,
VPN_TYPE/creds comments); config already VPN_TYPE=openvpn
- README.md: PIA-over-OpenVPN, status=enabled, steering iface wg0 -> tun0
- DECISIONS-LOG.md: disk/drive incident (k3s -> NVMe; failing Seagate) + Phase-3
OpenVPN resolution
Stale WireGuard comments inside the operator's encrypted pia-credentials.sops.yaml
header left untouched (editing breaks the SOPS MAC; cosmetic).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Prep only — nothing deployed; running cluster untouched.
Reconciliation graph (operator-approved Option A): author clusters/dev/rollout/
= 4 per-component Flux Kustomizations (kyverno->cilium->longhorn->monitoring,
monitoring dependsOn longhorn). Dev overlay $patch:delete's the monolithic base
platform/apps and adds ./rollout, so only the no-secret tier reconciles; the
secret-blocked components are simply not created until hand-offs arrive. The
no-secret tier carries no SOPS content, so it needs no sops-age key. prod is
untouched (still builds the monolith). Convergence path coded in (delete rollout/
+ the two deletes) — R-ENV-02 divergence flagged for architect review.
Component hardening via cluster-vars (prod keeps strict posture):
- KYVERNO_VALIDATION_ACTION Enforce->Audit(dev), _WEBHOOK_FAILURE_POLICY
Fail->Ignore(dev), _ADMISSION_REPLICAS 3->1(dev); webhook namespaceSelector
excludes system+platform namespaces.
- MONITORING_STORAGE_CLASS var; GRAFANA_ADMIN_SECRET var (dev ''=chart default);
Alertmanager SMTP/secret deferred to Phase-4 with verbatim re-enable markers.
Cilium parity verified against the live CNI (no-op adopt). Validated:
kubectl kustomize dev/prod/all-components, envsubst->YAML parse, make -C ci gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Implements T-1.5 findings F1/F2 (author-first, no cluster change).
F1 (tofu): refactor libvirt node resources count -> for_each keyed by
node name so single-node -replace/destroy cannot cascade to siblings.
Add ordered-list/primary helper locals; update outputs.tf off the map;
add moved.tf (12 moved blocks). tofu plan proves a pure state-address
migration: 0 to add, 0 to change, 0 to destroy (all 12 'has moved to').
State-persist via a 0-change apply is operator-permission-gated.
F2 (ansible): k3s_server secondary fetches the join token via
delegate_to the primary (published fact as fast path), so a single-node
rejoin with --limit k3s-dev-N is self-contained and no longer depends on
a fact an earlier play had to set. Syntax-check + CI gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
T-1.5 destructive IaC-validity test PASSED: dev cluster torn down and
reconstituted from IaC + preserved disks + Flux git source with zero
state loss (3/3 Ready, Cilium OK, Flux READY=True). Post-rebuild
conformance L2 R-NET-01 + R-NET-02 both PASS.
Two IaC-hardening findings for Phase 2:
- F1: tofu destroy -target on a count-indexed resource over-destroyed
all 3 domains; refactor libvirt_domain.node count -> for_each keyed by
node name; never use destroy -target for single-node lifecycle.
- F2: single-node ansible rejoin needs the primary in --limit to publish
the k3s join-token fact; make the join self-contained via delegate_to
the primary so --limit k3s-dev-N is hands-off.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
host-prereqs.yml opened the firewall (via the operator's ansible-vault become
password); Flux GitRepository READY=True (main@sha1:a2f3bdc). Commit the
encrypted hypervisor vault (ansible_become_password) so rebuild is fully
hands-off; passphrase stays out-of-repo. platform/apps still not reconciled.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per operator: the firewalld rule (VM->host:3000 for the Flux<-Gitea source) must be
IaC, not a manual command (R-ENV-01, R-PLAT-04).
- infra/ansible/playbooks/host-prereqs.yml: idempotent firewalld open on the libvirt
zone (localhost play, become; no extra collection).
- bootstrap/gitea/bootstrap-gitea.sh: now fully automates admin + public repo + push
(was manual instructions) — idempotent; creds stay 0600 on host, never committed.
Applying host-prereqs still needs host sudo (I lack non-interactive sudo); with
passwordless become a destroy/rebuild is fully unattended.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85andClaude Opus 4.8