PIA-provider server selection is flaky here (random dead IPs); the operator's .ovpn
uses sweden.privacy.network (load-balanced, reliable). Switch to custom mode
(VPN_SERVICE_PROVIDER=custom, OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn), mount the
pia-custom-ovpn Secret, keep creds + kill-switch + tun-mtu 1300. Drops PIA-provider
server list/updater and native PF (Phase-6 concern). Pending the operator-provisioned
pia-custom-ovpn Secret (not merged until it exists).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
gluetun silently ignored PIA_ENCRYPTION_PRESET (renamed) and defaulted to strong
(RSA-4096) the whole time. The correct v3.x var is
PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET=normal → RSA-2048/AES-128-CBC/1198,
the smaller handshake that fits the 1450 pod MTU.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
RSA-4096 strong handshake wouldn't hold over the pod's 1450 MTU (Cilium VXLAN);
MTU tuning is exhausted (no --tls-mtu in OpenVPN 2.6, PIA rejects --fragment).
Operator downloaded the lower-encryption PIA config (port 1198, RSA-2048,
AES-128-CBC, SHA1) = gluetun's 'normal' preset. Smaller handshake fits 1450.
Dropped the strong-handshake band-aid flags; kept OPENVPN_MTU 1300 for the data channel.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
RSA-4096 strong-preset handshake is marginal over the 1450 pod MTU (Cilium VXLAN).
--mtu-disc no clears DF so oversized control packets fragment through instead of
being silently dropped. Last MTU-path lever (OpenVPN 2.6 has no --tls-mtu).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
OpenVPN 2.6.16 rejects --tls-mtu (not a real option) and PIA doesn't support
--fragment, so control-channel packets can't be shrunk. Keep --hand-window 120 to
give the RSA-4096 handshake more time over a lossy/marginal path, plus tun-mtu 1300.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
RSA-4096 strong-preset handshake is marginal over the pod's 1450 MTU (Cilium VXLAN).
Lower tun-mtu to 1300 and add --tls-mtu 1200 (smaller control packets) + --hand-window
120 (more time to complete the handshake over a lossy path). Empirical test of the
MTU-tuning path (operator's choice over Cilium native routing).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator's working config is sweden.privacy.network:1197 with RSA-4096, AES-256-CBC,
SHA256 = PIA's 'strong' preset. gluetun defaults PIA to 'normal' (AES-128-GCM), a
cipher mismatch against the 1197 (strong) endpoint, so the handshake never completed
(silent after UDP link). PIA_ENCRYPTION_PRESET=strong matches it and selects 1197.
Reverts the 8080 port red herring.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
UPDATER_PERIOD only refreshes AFTER the tunnel is ready (chicken-and-egg: can't
connect with stale servers), and gluetun's bundled PIA Sweden list is retired
46.246.x (current is 158.173.x). Add an `update-servers` initContainer running
`gluetun update -providers "private internet access"` into a shared /gluetun
emptyDir the main container reads, so it dials current servers. UPDATER_PERIOD
raised to 720h (docs-compliant) for ongoing in-pod refresh.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
CORRECTION: the connection failures were gluetun dialing RETIRED PIA IPs (46.246.x
= dead → timeouts, which I mis-read as a network block). A fresh PIA .ovpn connects
from the same host, and PIA's current SE Stockholm servers (158.173.x) are reachable.
Cilium was independently cleared (default-allow, host-fw off, Hubble FORWARDED).
Fix: UPDATER_PERIOD=24h so gluetun fetches PIA's current servers; keep
OPENVPN_MTU/MSSFIX=1350 for the real Cilium-VXLAN 1450 pod MTU (data channel);
PIA default UDP port. Re-enable vpn-egress. Full correction in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Decisive diagnosis: OpenVPN's outbound init packets are 14 bytes with zero
replies (so MTU was never the blocker), pod UDP egress is healthy (NTP replies),
but TCP 501/443 to PIA IPs time out while Google/Cloudflare are OPEN — across many
PIA servers. This 192.168.2.x WiFi segment blackholes PIA's IP ranges (and
intercepts DNS). Not code-fixable. Reverted the port/MTU red-herring env to PIA
defaults; disabled vpn-egress pending an operator network fix (move to the
unfiltered 192.168.1.x LAN, or unblock PIA on the router). Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Actual root cause (operator-directed layered MTU check): Cilium runs VXLAN
(routing-mode tunnel) so pods get MTU 1450 (1500-50 encap) while node, hypervisor,
and the Google Fiber WAN are all 1500. OpenVPN's ~1500-byte handshake packets
exceed 1450 and are dropped (PMTUD broken). The ports were a red herring.
Fix: OPENVPN_MTU=1350 + OPENVPN_MSSFIX=1350 so encrypted packets (~1402) fit under
1450. Keeps OpenVPN + VXLAN unchanged. Re-enable vpn-egress. Full layered trace
(pod/node/host) in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
OpenVPN can't connect on this network (each port fails differently, so it's
OpenVPN-specific not a UDP block — NTP/123 works): 1197 = home OpenVPN-server
conflict; 53 = DNS interception; 8080 = TLS handshake fails across servers (MTU
on WiFi+triple-NAT). Recommend pivoting to WireGuard via thrnz/docker-wireguard-pia
(UDP 1337, MTU-robust, ADR-0013 fallback, operator's original preference) — a
spec-level (ADR-0013) change, so escalated. Disabled meanwhile. Full trace in
DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Port 53 also got no PIA replies: this network transparently intercepts outbound
:53, so dig works but OpenVPN-over-53 is swallowed. An NTP/123 probe from the host
returned fine, proving arbitrary non-53 UDP works bidirectionally (so it's
port-specific: 1197 conflict + 53 DNS-intercept, not a UDP block). Move to 8080
(community-verified for PIA, non-DNS, non-1197).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Wiring the port via "${PIA_OPENVPN_PORT}" made Flux/kustomize emit an unquoted
53 (int) after substitution, which k8s rejects for an env value ("expected
string, got 53"). Same gotcha as the Longhorn numberOfReplicas fix. Hardcode the
quoted string; cluster-var removed with a NOTE explaining why.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator identified the block: the home network NATs its own OpenVPN server on
WAN:1197, colliding with the cluster's outbound PIA-on-1197 flow (port-specific,
not a general UDP block — UDP/53 worked in tcpdump). PIA accepts OpenVPN on
53/1194/1197/1198/8080/9201; chose 53 (empirically proven to traverse this net).
- cluster-vars: add PIA_OPENVPN_PORT="53"
- gluetun: OPENVPN_ENDPOINT_PORT=${PIA_OPENVPN_PORT}
- re-enable vpn-egress; DECISIONS-LOG resolution noted
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Phase-3 VPN config is complete + correct (OpenVPN-UDP, SE Stockholm), but the
OpenVPN UDP/1197 handshake to PIA never gets a reply. Traced with tcpdump: packets
leave pod→node→host→WiFi (SNAT 192.168.2.103) fine, but 0 replies return to the
host, while host UDP/53 works. The 192.168.2.x home router/ISP drops OpenVPN UDP —
upstream of everything in this repo. Reverted to disabled to avoid a crashloop
firing false KubePodCrashLooping alerts; one-line re-enable once the network is
fixed. Full trace in DECISIONS-LOG.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Gluetun cannot do PIA over WireGuard (upstream unsupported; live "provider name
is not valid for Wireguard"). Operator approved OpenVPN-UDP and ADR-0013 was
amended in the spec. This aligns the impl:
- platform/kustomization.yaml: re-enable `- vpn-egress` (Phase 3, OpenVPN-UDP)
- gluetun-gateway.yaml: clean stale WireGuard comments (header, image note,
VPN_TYPE/creds comments); config already VPN_TYPE=openvpn
- README.md: PIA-over-OpenVPN, status=enabled, steering iface wg0 -> tun0
- DECISIONS-LOG.md: disk/drive incident (k3s -> NVMe; failing Seagate) + Phase-3
OpenVPN resolution
Stale WireGuard comments inside the operator's encrypted pia-credentials.sops.yaml
header left untouched (editing breaks the SOPS MAC; cosmetic).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The deployed v3.40.0 rejected PIA+WireGuard ('private internet access is not valid for
Wireguard'). Verified v3.41.0 accepts the PIA+WireGuard provider/protocol (passes config
validation; ADR-0013's model holds — v3.40.0 was just too old). Bump the pin.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
PIA creds provisioned (SOPS) + wired. Enable vpn-egress in platform/kustomization.yaml
to deploy the shared Gluetun gateway (PIA over WireGuard, Sweden region, kill-switch).
Egress-class steering (vpn-class pods -> tunnel) still to be validated against the live
tunnel next. Gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
gluetun-gateway.yaml referenced ${PIA_REGION}/${CLUSTER_POD_CIDR} but they were never defined
in cluster-vars, and the README didn't say where to set them. Add both to
clusters/base/cluster-vars.yaml (ConfigMap data:, string values, Flux postBuild substitution).
CLUSTER_POD_CIDR is 10.0.0.0/8 (Cilium cluster-pool — pods get 10.x, confirmed live), NOT the
k3s 10.42.0.0/16 I'd assumed. README now names the file + format + per-env override + PF-region note.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Enabling monitoring deadlocked the platform Kustomization: alerts.yaml ships
PrometheusRule CRs whose monitoring.coreos.com CRDs are installed by the
kube-prometheus-stack HelmRelease in the SAME pass → 'no matches for kind
PrometheusRule' → atomic apply failure → NO monitoring HelmReleases created.
Same controllers/configs fix as cert-manager/kyverno: drop alerts.yaml from
platform/monitoring and apply it via platform-config (dependsOn platform), so the
monitoring CRDs exist first. Verified: platform has 0 PrometheusRule (keeps the 3
monitoring HRs); platform-config has the 8 alert rules; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Per ADR-0013: platform/vpn-egress/ = a single shared Gluetun gateway (PIA over WireGuard,
built-in kill-switch FIREWALL=on, native port forwarding), namespace (non-app, NET_ADMIN +
/dev/net/tun), control-API Service, and a pre-wired (commented) pia-credentials SOPS line.
Defined the PIA secret location (platform/vpn-egress, keys username/password → OPENVPN_USER/
OPENVPN_PASSWORD) and updated the SOPS template accordingly.
NOT wired into platform/kustomization.yaml (Phase 3, author-first) — deploy needs PIA creds +
validation of the egress-steering mechanism (Cilium Egress Gateway → the Gluetun node's wg0;
documented in README with options + a fail-closed rationale, deliberately NOT committed as an
untested CiliumEgressGatewayPolicy). WireGuard chosen (UDP, protocol != TCP); PIA supports it on
all regions and the secret stays username/password.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Enable monitoring in platform/kustomization.yaml (Phase 4). Grafana admin +
Alertmanager SMTP secrets are provisioned (SOPS) and wired; Longhorn SC is up for
Prometheus/Loki PVCs. Set the Alertmanager receiver 'to:' to a valid inbox
([email protected]) so the config loads (REPLACE-ME is not a valid address);
operator can change it. NOTE: smtp_from ([email protected]) differs from
the Gmail auth user — Gmail may reject/rewrite unless a send-as alias is configured;
that only affects email delivery at alert time, not stack health.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The kyverno chart's cleanup jobs (policyReportsCleanup, webhooksCleanup) default to
bitnami/kubectl:1.30.2, which Bitnami removed from Docker Hub (2024 catalog purge) ->
ImagePullBackOff. policyReportsCleanup is a Helm post-upgrade HOOK, so its failure
stalls every helm upgrade (5m timeout -> HelmRelease not-Ready -> platform Kustomization
blocked -> platform-config/apps cascade). Override both to Bitnami's relocated
bitnamilegacy/kubectl (identical image: /bin/bash + awk/wc/xargs + kubectl). Verified
via helm template that no bitnami/kubectl reference remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Phase-2 exit tests (were Phase-0 scaffolds), authored against the live platform:
- trusted-tls (R-TLS-01/03): in-cluster probe dials the gateway Service (env-agnostic
via --connect-to) with wildcard SNI and verifies the served cert CHAINS TO THE LE
STAGING ROOT (operator decision A; dev uses LE-staging per ADR-0006). Fixture embeds
the '(STAGING) Pretend Pear X1' root. Probes public + internal-only hostnames (R-TLS-03).
PASSES.
- dns-exposure (R-DNS-02): a LoadBalancer Service with an external-dns hostname; polls
public DNS until ExternalDNS auto-publishes the record; teardown + policy=sync removes it.
ExternalDNS fix (root cause of the DNS test): domain-filter was the platform SUBDOMAIN
home.keithjackson.dev, which filters OUT the actual Cloudflare zone keithjackson.dev
('no hosted zone matching record DNS Name' — record generated but skipped). Add a
DNS_ZONE cluster-var (keithjackson.dev) and set domainFilters to it. policy=sync +
txt-registry still only manage ExternalDNS-owned records (the pre-existing wildcard is
untouched). Resolves the file's WS-NET R-DNS-02 TODO.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The wildcard cert stalled: challenge Presented=true (Cloudflare TXT created OK —
token works) but propagation self-check looped 'DNS record not yet propagated'.
Verified the record IS publicly resolvable (1.1.1.1/8.8.8.8); the NAT'd k3s-dev
cluster's own DNS just can't see it. Set dns01RecursiveNameservers=1.1.1.1:53,
8.8.8.8:53 + dns01RecursiveNameserversOnly=true so the self-check bypasses cluster
DNS. Realises the pre-existing TODO in this file.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Kyverno HelmRelease install failed at Helm templating: config.webhooks was a
LIST [{failurePolicy, namespaceSelector}], but the kyverno 3.3.9 chart expects
config.webhooks to be a MAP (webhooks.namespaceSelector) — the chart helper reads
.namespaceSelector off the value directly ('can't evaluate field namespaceSelector
in type []interface {}'). Also, failurePolicy is not a per-webhook field here; the
chart's global switch is config.forceFailurePolicyIgnore.enabled (bool).
Fix: config.webhooks -> map with namespaceSelector (system+platform excludes);
add config.forceFailurePolicyIgnore.enabled = ${KYVERNO_FORCE_FAILURE_POLICY_IGNORE}.
Renamed the cluster-var KYVERNO_WEBHOOK_FAILURE_POLICY (Fail/Ignore) ->
KYVERNO_FORCE_FAILURE_POLICY_IGNORE (false base / true dev). Verified with
helm template (renders 73 objects, no error); gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Un-suspend deadlocked: the single platform Kustomization applies HelmReleases
(which register CRDs) AND custom resources of those CRDs (ClusterIssuer,
Certificate, Kyverno ClusterPolicies) in ONE pass. Flux's server-side apply
fails atomically ('no matches for kind ClusterIssuer') → nothing installs, so
the CRD-registering HelmReleases never run either. Classic ordering deadlock.
Fix (flux2 controllers/configs pattern, applies to dev+prod equally — no
R-ENV-02 divergence): add a platform-config Flux Kustomization (dependsOn
platform, wait) carrying the CRs; platform keeps only the controllers.
- New clusters/base/platform-config.yaml (path ./platform-config, substituteFrom
cluster-vars, no decryption — no SOPS content); wired into base + apps now
dependsOn platform-config (transitively waits for platform).
- platform-config/ references the CR files IN PLACE via ../platform/... (Flux
builds LoadRestrictionsNone; Kyverno policies must stay under platform/kyverno/
for the CI L0 gate). CR resources dropped from the platform
component kustomizations (cert-manager, gateway, kyverno).
- Authored the missing Cilium L2 resources (were only a TODO): CiliumLoadBalancerIPPool
(dev NAT 10.73.0.240-.250) + CiliumL2AnnouncementPolicy (ens*), so Gateway
LoadBalancer listeners get a VIP. Prod range must be parameterized later.
Verified: platform build has zero CRs; platform-config build has 2 ClusterIssuers,
6 ClusterPolicies, Certificate, Gateway, both Cilium L2 CRs; dev/prod build; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The un-suspend reconcile stalled: platform Kustomization Ready=False with
'StorageClass/longhorn ... .parameters.numberOfReplicas: expected string, got 3'.
StorageClass.parameters is map[string]string, but kustomize strips the quotes
around the bare "${LONGHORN_DEFAULT_REPLICA_COUNT}" placeholder, so Flux postBuild
substitution produced an unquoted numeric that server-side apply rejects as int.
Hardcode the quoted numeric literal "3" (kustomize preserves those, as "30" shows);
topology is locked at 3 nodes. The replica knob still drives the global default via
defaultSettings.defaultReplicaCount in the helmrelease (a Helm int value, unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Rename the committed encrypted secrets to canonical cloudflare-api-token.sops.yaml
(cert-manager + external-dns namespaces) and uncomment the resource lines in both
kustomizations, so cert-manager DNS-01 + ExternalDNS get the token at reconcile.
- Exempt **/*.sops.yaml from ci/yamllint.yaml: SOPS re-emits encrypted files with its
own indentation + long ENC[...] lines (never hand-edited), which was failing the lint
gate on main after the secrets were committed. Same treatment as **/vault.yml.
Both Secrets confirmed present + encrypted in the platform build; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Prep only — nothing deployed; running cluster untouched.
Reconciliation graph (operator-approved Option A): author clusters/dev/rollout/
= 4 per-component Flux Kustomizations (kyverno->cilium->longhorn->monitoring,
monitoring dependsOn longhorn). Dev overlay $patch:delete's the monolithic base
platform/apps and adds ./rollout, so only the no-secret tier reconciles; the
secret-blocked components are simply not created until hand-offs arrive. The
no-secret tier carries no SOPS content, so it needs no sops-age key. prod is
untouched (still builds the monolith). Convergence path coded in (delete rollout/
+ the two deletes) — R-ENV-02 divergence flagged for architect review.
Component hardening via cluster-vars (prod keeps strict posture):
- KYVERNO_VALIDATION_ACTION Enforce->Audit(dev), _WEBHOOK_FAILURE_POLICY
Fail->Ignore(dev), _ADMISSION_REPLICAS 3->1(dev); webhook namespaceSelector
excludes system+platform namespaces.
- MONITORING_STORAGE_CLASS var; GRAFANA_ADMIN_SECRET var (dev ''=chart default);
Alertmanager SMTP/secret deferred to Phase-4 with verbatim re-enable markers.
Cilium parity verified against the live CNI (no-op adopt). Validated:
kubectl kustomize dev/prod/all-components, envsubst->YAML parse, make -C ci gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Adopt the spec's Phase-0 clarifications (spec now at pin a7fc19f) into the impl:
- schema v1.1: enforce exposure:public => auth.mode!=none (positive+negative tested).
- Kyverno: add longhorn=>size and longhorn=>NOT-reacquirable invariants (R-STATE-02/03).
- Ratify the homelab.keithjackson.dev/* label-key convention (now in spec docs/03).
- R-TEST-06 spec-immutability gate: ci/SPEC_PIN + ci/spec-immutability.sh + gate.sh
stage + Makefile target + a Gitea job (checks spec repo clean & at pin).
- touched-requirements.sh: conformance tags are the authoritative source.
Reconcile 3 CI seams the CI worker flagged (gate was RED, now GREEN):
- create schemas/validate.sh (SCHEMA_VALIDATE_CMD entrypoint) — L0 schema gate runs.
- fix l1.sh/conformance.sh to run.sh's real interface (fixed _artifacts/ paths; no
--junit/--coverage) and collect artifacts.
- exec bits on new scripts.
make -C ci gate => PASS (l0 skips locally w/o kyverno binary; green in CI).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The prior integration commit enabled kyverno in the platform render, but (a) the
admission bundle used ../policies refs that fail default kustomize load
restrictions, and (b) admission enforcement is Phase 2 (T-2.7), not Phase 0.
Fix: platform/kyverno/kustomization.yaml now lists resources with downward paths
(builds standalone); platform/kustomization.yaml re-defers kyverno with a Phase-2
note. All Phase-0 gates green: kustomize (platform, platform/kyverno, dev),
schema validation, conformance scaffold.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Add platform/kyverno/kustomization.yaml (renders the admission enforcing bundle)
and enable the kyverno component in platform/kustomization.yaml (first, so
admission policies precede other components). kustomize build verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Phase 0 foundation shell (read-only spec at ../home-network-k3s). Top-level
tree + rules; workstream agents fill their subtrees on feature branches.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
