Add catalog/plex.yaml (kind: container) and its rendered apps/plex.yaml,
and wire apps/kustomization.yaml. Plex = docs/13 row 1, the catalog's
largest service; validates the public-exposure + config-on-Longhorn path.
- image plexinc/pms-docker (pinned recent stable tag), port 32400
- exposure: public at plex.home.keithjackson.dev; egress: internet
- auth: native (Plex accounts, not Keycloak — docs/13/15 exception)
- config PVC 20Gi on longhorn; off-cluster backup (daily-offcluster) as the
longhorn tier forbids reacquirable under L0 (R-STATE-03)
- observability up-SLI; update channel stable
Bulk media (R-STATE-04) deferred: the `bulk` StorageClass is a documented
stub (OQ-2 unresolved, NFS CSI not installed) so a bulk PVC would leave the
pod Pending. Media mount omitted; attach once bulk provisions.
Traces: R-GITOPS-02, R-NET-01/02/05, R-STATE-01/02/03, R-TLS/R-DNS,
R-OBS-01, R-UPD-01/03, R-IDP-02, R-PLAT-02.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
New runnable chainsaw test gateway-ingress-reachable (R-NET-01/02/03): stands up an
exposed-app fixture (namespace default-deny + the renderer's additive gateway-ingress
CiliumNetworkPolicy allowing fromEntities:[ingress] to the declared port 8080 only, a
backend listening on 8080 + an undeclared 9091, and declared/undeclared HTTPRoutes on
the shared Gateway) and proves BEHAVIOR through the REAL Gateway VIP:
* then-reachable-through-gateway -> the Gateway (ingress identity) reaches backend:8080
through default-deny (backend marker returned) [R-NET-02/03]
* then-undeclared-port-denied -> port 9091 (no allow) is NOT reachable through the
Gateway (marker never returned; envoy 503) [R-NET-02]
* then-east-west-still-denied -> a non-ingress in-namespace peer is DROPPED on 8080;
the allow is least-privilege, east-west default-deny intact (hermetic) [R-NET-01]
Verified live: reachable=REACHED (200, marker), undeclared=DENIED, east-west=DENIED.
Self-skipping: the gateway probes classify each attempt REACHED / DENIED / NO-GATEWAY
(curl 000). NO-GATEWAY self-skips (green + note) because a node-local Cilium L7LB/envoy
desync can make pods on an affected node unable to reach the Gateway VIP at all — a
SEPARATE platform fault, not default-deny — so this test never false-fails on it. The
east-west denial is hermetic and always asserts. VIP discovered per-step via a kubectl
output binding (env-agnostic; chainsaw bindings are step-scoped). L2 layer stays gated by
CLUSTER_AVAILABLE in run.sh. chainsaw v0.2.12 quirks honored (scalar/JMESPath asserts,
no readinessProbe under default-deny, step-level finally delete).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Exposed apps got a default-deny NetworkPolicy with ingress: [] (deny all), so the
Cilium Gateway (Envoy, reserved identity `ingress`) could not reach the backend — an
Accepted HTTPRoute was still unreachable. render.py now emits an ADDITIVE, least-
privilege CiliumNetworkPolicy for any ServiceDefinition that renders an HTTPRoute:
ingress:
- fromEntities: [ingress] # the shared Cilium Gateway/Envoy proxy identity
toPorts: [<exposed service port(s)> TCP]
Verified against the live cluster (Cilium 1.16.5): with default-deny in place, Hubble
shows the Gateway proxy reaching pihole as source identity `(ingress)` and the flow
`Policy denied DROPPED` on :80 — so `fromEntities: [ingress]` is exactly the source the
Gateway uses, scoped to the service targetPort. Default-deny stays intact (ingress: []
unchanged); the allow only opens the north-south path (docs/09), east-west stays denied.
- render.py: add r_gateway_ingress_policy(); emit it after dependency policies; selftest
asserts the allow is additive (default-deny ingress stays []), least-privilege (only the
exposed port, only the ingress identity), and absent for non-exposed services.
- L1 goldens updated for the two exposed fixtures (container-workload, egress-class);
default-deny-baseline golden unchanged (no exposure). --selftest + run.sh --layer l1 green.
- apps/pihole.yaml re-rendered (--no-image-automation) to carry the allow.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
First app onboarded through catalog/render.py -> apps/. Added --no-image-automation so apps
deploy before the Flux image controllers exist (L1 goldens keep the default, unchanged).
KNOWN GAP surfaced: exposed apps get default-deny (ingress: []) with no gateway-ingress allow,
so the HTTPRoute path is blocked — renderer/WS-NET follow-up to allow Cilium gateway ingress
to services with an exposure/HTTPRoute.
busybox wget has no --max-redirect flag → it printed usage, so the probe never saw the
302 and R-IDP-01 falsely failed. Grafana verified live: /login/generic_oauth 302s to the
Keycloak homelab authorize endpoint. Use 'wget -S' (dumps the 302 Location before follow).
VMs did not boot on the dev cluster. Three distinct live-verified causes,
all fixed GitOps-side:
1. containerDisk root failed at OCI init:
exec "/container-disk-binary/usr/bin/container-disk": no such file.
KubeVirt v1.9 supplies that binary to the volumerootdisk init container
via a k8s-native `image` volume (VolumeSource.Image), gated by the ALPHA
`ImageVolume` feature gate. k3s v1.31 has it DISABLED, so the apiserver
silently prunes the image volume + its mount -> binary missing -> crash.
(Proved: a server-side dry-run of a pod with an `image` volume drops it.)
Fix: convert the VM roots in all three tests to CDI DataVolumes imported
onto Longhorn PVCs (mounted disks bypass the container-disk/image-volume
path). This is also the spec-required disk model (docs/06, R-STATE-01).
vm-lifecycle now PASSES green live (VMI Running on start + restart).
2. Guest dropped to emergency mode: the containerdisks/fedora:41 image is
UEFI-native (ESP + /boot mounted via systemd GPT auto-discovery); under
KubeVirt's SeaBIOS default the ESP+/boot by-uuid mounts time out ->
local-fs.target fails -> cloud-init runcmd/poweroff never runs.
Fix: boot the VMs with EFI (OVMF, secureBoot:false) in all three tests
and fixtures/vm.yaml. Confirmed live: emergency mode gone with EFI.
3. VMs ran under TCG software emulation (guests ~10x slow, saturating nodes
until one went NotReady and VMs were evicted). Cause: useEmulation:true
in the KubeVirt CR makes virt-controller NOT request devices.kubevirt.io/
kvm, so /dev/kvm is never mounted and qemu falls back to TCG even though
all nodes advertise kvm=1k / hypervisor=true. (Proved: `ls /dev/kvm` in a
virt-launcher compute container = "No such file"; qemu accel = tcg.)
Fix: useEmulation:false so the kvm device is requested and VMs run on real
(nested) KVM. Requires Flux reconcile to take effect live (PM action).
The virt-handler clientcertificates tls.crt error is UNRELATED: node-labeller
(294 labels, hypervisor=true), device plugins, and VMI scheduling all work.
Also raised test exec timeouts to 30m (CDI import + boot) and, in
vm-default-deny, import the root BEFORE applying the default-deny NetworkPolicy
so the CDI importer's egress to quay.io is not severed by the policy.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Grafana's pod predated the grafana-oidc-credentials secret, so generic_oauth started with
an empty client_secret (disabled) → /login/generic_oauth didn't redirect to Keycloak.
Add a podAnnotation to force one roll; bump on future OIDC secret changes.
oauth2-proxy + Grafana OIDC discovery to keycloak.home.keithjackson.dev failed x509
'unknown authority' — the wildcard was LE-staging (untrusted). Prod issuer is registered
+ Ready; switching ACME_CLUSTER_ISSUER dev override staging->prod re-issues a publicly
trusted cert. Needed for functional SSO (R-IDP) + R-TLS-01/03 'trusted'.
NOTE: the trusted-tls conformance test embeds the LE-staging root and must be updated to
the LE production root (ISRG Root X1) / system trust — follow-up.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add dns-internal-resolution (L2 behavioral): an in-cluster Job resolves
keycloak.home.keithjackson.dev via the default cluster resolver and asserts the answer
is the in-cluster Gateway VIP (10.73.0.24x), not a public/off-cluster hairpin. Complements
dns-exposure (R-DNS-02, external record). coverage.py now maps R-DNS-01/R-DNS-03 to this
test (covered); chainsaw v0.2.12 lint clean; strict hygiene gate green.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
In-cluster clients resolving a platform hostname over its PUBLIC name got the
off-cluster answer, so pod->keycloak.home.keithjackson.dev hit a path that does not
serve the *.home.keithjackson.dev SNI -> `tls: unrecognized name` (oauth2-proxy OIDC
discovery crash; Grafana generic_oauth would hit the same wall).
Add a GitOps-managed coredns-custom ConfigMap (kube-system) that k3s CoreDNS imports as
a `*.server` block. The template plugin answers the whole home.keithjackson.dev zone with
${GATEWAY_VIP} (the shared Gateway VIP), so pod->public-hostname traffic hairpins through
the Gateway with the correct SNI -> wildcard cert validates. CoreDNS reloads the ConfigMap
live (no pod restart). Wired into platform/kustomization.yaml (Phase 2 network core).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Parameterize the in-cluster split-horizon target as GATEWAY_VIP (R-ENV-02): base
default = dev L2-announced Gateway VIP (10.73.0.240, first IP of the Cilium LB pool).
coredns-custom answers *.home.keithjackson.dev with ${GATEWAY_VIP} so the unified graph
holds across dev/prod via cluster-vars only. Prod overlay carries a TODO to set the prod
LAN VIP alongside the prod L2 pool range when hardware is built.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The renderer's PaaS path (kind: container) had no verifying conformance test — the
R-PLAT-02 token appears only in renderer comments, so a change touching catalog/
surfaced it as an uncovered touched-requirement (blocking the merge gate). Adds the
container render analog of vm-lifecycle (R-PLAT-03): asserts a kind: container
ServiceDefinition renders a Deployment (running the declared image) + a Service,
with the egress-class/auth-mode labels admission requires. Offline render-check.sh,
wrapped by a chainsaw Test for in-cluster runs.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Implements the two L1 rendering tests that were failing scaffolds:
- egress-class-render (R-NET-05/06/07): renders vpn + internet fixtures, diffs each
against a committed golden, asserts the vpn steering label is present on vpn and
absent on internet (fail-closed), and that flipping the class changes ONLY the
egress path (R-NET-07). This fixes the --requirements R-NET-05 merge gate.
- default-deny-baseline-render (R-NET-01): asserts every render carries the
namespaced default-deny baseline, shaped deny-all + both directions.
Each test carries a pure-offline render-check.sh (renderer + golden diff, no
cluster). run.sh runs L1 render checks offline and emits a REAL pass/fail JUnit
(docs/14: L1 rendering assertions run in CI without a cluster) instead of skipping
L1 with the cluster-gated layers; the chainsaw-test.yaml wrappers invoke the same
render-check.sh so an in-cluster run executes the identical assertion.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
catalog/render.py compiles a ServiceDefinition into the manifest set defined by
schemas/renderer-contract.md: Namespace + default-deny baseline (R-NET-01),
Deployment/Service or KubeVirt VM + CDI DataVolume (R-PLAT-02/03), PVCs (R-STATE),
per-dependency CiliumNetworkPolicy allows (R-NET-02), egress-class steering label
(R-NET-05), HTTPRoutes on the shared Gateway (R-TLS/R-DNS), ServiceMonitor + SLI
alerts (R-OBS-01), Flux image-automation (R-UPD), auth-mode/backup hints (R-IDP).
Deterministic (golden-diffable) with a built-in --selftest; single dependency
(PyYAML) so the conformance harness stays stdlib-only. Documents the mechanism
choice in catalog/README.md and the script header.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy read the base64 cookie value as raw bytes (44) → 'must be 16/24/32 bytes'
crashloop. Regenerated cookie-secret as openssl rand -hex 16 (32 bytes). client-id +
client-secret unchanged (still match the realm oauth2-proxy client). Value tracked in
~/.config/home-k3s/keycloak-sso-secrets.txt.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Runnable, self-skipping (green until image.toolkit CRDs exist). Proves: an ImagePolicy
auto-advances to the newest STABLE tag on schedule with no edit and never a pre-release
(R-UPD-01/02, via podinfo ImageRepository/ImagePolicy status); a deliberately bad update
is health-gated and AUTO-ROLLS-BACK to the last-good revision while the old ReplicaSet
keeps serving, and the UpdateHealthGateFailed alert is wired (R-UPD-02/03); and an
exact-range (pinned) policy does not advance past its pin (R-UPD-03). Hermetic to the
chainsaw per-test namespace; block-style contains() checks per chainsaw v0.2.12.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add platform-health.automated-updates PrometheusRule group carried by platform-config:
ImageUpdateAutomationFailing (write-back broken -> advances silently stop),
ImagePolicyStalled (registry/range can't resolve a stable tag), and UpdateHealthGateFailed
(a bad update health-gated and auto-rolling-back via HelmRelease remediation). Routes to
Alertmanager email so a bad update is visible.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Map a ServiceDefinition `update` block to ImageRepository/ImagePolicy + the
$imagepolicy setter marker: stable-only (semver range + bare-X.Y.Z filterTags,
pre-release excluded, R-UPD-01), pin -> fixed image with no marker / suspended
policy (R-UPD-03). Worked pihole example under per-service/ (reference only, not
deployed; WS-GITOPS emits the equivalent into apps/<svc>/). Add renovate.json for
Helm chart / dependency / operator-manifest bumps: auto-merge minor/patch/digest on
green CI, major needs review, image managers disabled so Renovate never fights Flux
over a tag.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Install image-reflector-controller + image-automation-controller as part of the
flux-system install (bootstrap.sh --components-extra) so the image.toolkit CRDs
pre-exist before reconcile — no F-MON1 split needed for the ImageUpdateAutomation CR.
Refine the cluster-wide ImageUpdateAutomation for machine-authored git write-back
(setter strategy, path ./apps) reusing the flux-system GitRepository secretRef, and
document the operator git-write-credential hand-off + SOPS secret template
(gitea-image-automation-credentials). image-automation stays deferred in
platform/kustomization.yaml until that credential is provisioned.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Generate the 3 OIDC client secrets (SOPS-encrypted, minimal manifests) so the realm
import can inject the grafana + oauth2-proxy client secrets and both consumers match:
- platform/keycloak/keycloak-realm-clients.sops.yaml (grafana + oauth2-proxy client secrets)
- platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml (client-id/secret/cookie)
- platform/monitoring/grafana-oidc-credentials.sops.yaml (grafana client secret)
Wire each into its component kustomization and re-enable oauth2-proxy in platform +
platform-config (reverses the temp deferral now that its secret exists).
Values generated by PM per operator instruction (random; recorded out-of-repo at
~/.config/home-k3s/keycloak-sso-secrets.txt for later rotation). Encrypted via sops in the
dev container to the age recipient in .sops.yaml; plaintext never committed.
Serves: R-IDP-01, R-IDP-02, R-IDP-04, R-SEC-01.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
oauth2-proxy's HelmRelease mounts existingSecret oauth2-proxy-credentials, an operator
hand-off not yet filled. Under the unified graph (platform wait:true), the not-ready HR
blocks platform -> platform-config -> the whole downstream graph including KubeVirt.
Deferring it (comment its line in platform/kustomization.yaml + its two entries in
platform-config) lets the no-secret components (KubeVirt/CDI, CNPG, Keycloak operator +
instance) reconcile and be certified now. Re-enable both together once the operator fills
platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml.
Serves: R-IDP-02 (deferred). PM sequencing correction — enable per phase AND readiness.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Both components authored by worktree-isolated agents, reviewed, integrated to main,
offline-green (MERGE GATE: PASS, dev+prod overlays render). Deploy blocked on the
push-to-gitea permission + 3 operator SOPS hand-offs. Full runbook + verify-at-deploy
escalations recorded.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Adopt the architect's spec advance (ADR-0013 amendment + ADR-0015 egress
steering/invariants) into ci/SPEC_PIN; the impl main pin was stale (a7fc19f) so the
R-TEST-06 spec-immutability gate was red on main. Verified the a7fc19f..b633b7f range
is spec-only architect commits.
Add R-ENV-02 and R-GITOPS-01 to conformance/run.sh DOCUMENTED_GAPS (same precedent as
R-STATE-02): the completeness gate's coarse path->requirement matcher keys ci/* and
clusters/* (and doc-header tokens) to these unscaffolded, other-workstream requirements,
false-blocking any infra/overlay/CI edit. Documented + reviewable, not silent. Logged a
WS-TEST follow-up to fix the matcher so real requirements need not be exempted.
Serves: R-TEST-06. PM/Integration-Release governance change.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
pia-credentials and pia-custom-ovpn were gluetun's PIA creds; gluetun is retired and the node
tunnel uses host-side Ansible vault creds (docs/07), so these cluster Secrets are unused. Remove
them from the kustomization + delete the encrypted files (recoverable from git history). Flux
will prune the Secrets from the vpn-egress namespace.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The node-tunnel PIA credentials (username/password/.ovpn) live host-side in Ansible vault per
docs/07. The encrypted vault.yml was provisioned but never committed, so the vpn.egress.gateway
playbook was not reproducible from a fresh checkout. Commit it (ansible-vault encrypted; the
vault password stays out of the repo via ansible.cfg vault_password_file).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The policy is wired into the kustomization, reconciled by Flux, and conformance-green; gluetun
is retired. Update the header banner + bottom checklist to reflect that. Comment-only — the
CiliumEgressGatewayPolicy spec is unchanged.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
README rewrite + ADR-0013->0015 comment drift across kustomization, kyverno egress-class
policy, servicedefinition schema, cluster-vars, the now-unused pia-* secret headers, and the
vpn-egress namespace note. Comments/docs only — no runtime, alert-expr, resource-list, or
SOPS-payload changes (both .sops.yaml verified: zero data/sops/mac diffs).
The in-cluster Gluetun gateway is retired; VPN egress is now a node-level host
OpenVPN split-tunnel (Ansible role vpn_egress_gateway) steered by a Cilium
Egress Gateway policy with a host nftables kill-switch (ADR-0015). Credentials
moved host-side to the Ansible vault. Update docs/comments only:
- platform/vpn-egress/README.md: rewrite for ADR-0015 (live + conformance-green);
drop the resolved "option 1 not viable with Gluetun" finding to a brief history note.
- platform/kustomization.yaml: Phase 3 comment -> node tunnel + Cilium steering.
- platform/kyverno/.../require-egress-class-label.yaml: ADR-0013 -> ADR-0015 steering.
- schemas/servicedefinition.schema.json: egress=vpn description -> node tunnel + Cilium.
- clusters/base/cluster-vars.yaml: reframe PIA_REGION / CLUSTER_POD_CIDR WHY comments
to ADR-0015 (Cilium excludedCIDRs); both keys retained (still referenced/documented).
- platform/vpn-egress/namespace.yaml: no privileged gateway pod under ADR-0015.
- pia-credentials.sops.yaml / pia-custom-ovpn.sops.yaml: comment headers only note
they are now UNUSED (creds host-side), retained pending cleanup; payloads untouched.
No runtime manifest behavior, alert exprs, kustomization resources, or SOPS payloads
changed. No spec-repo edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator-approved. Removes the vestigial in-cluster gluetun Deployment/Service (the node-level
host tunnel steered by Cilium Egress Gateway is validated + conformance-green) and updates the
VPNTunnelDown alert to key on node tun0 presence (absent(node_network_receive_bytes_total{device=tun0}))
instead of the now-gone gluetun_vpn_up gauge; drops the obsolete pod-based EgressGatewayUnhealthy.
Unused pia-* SOPS secrets retained pending a separate cleanup.
gluetun retirement removes the gluetun_vpn_up gauge and the in-cluster gateway pod. Point
VPNTunnelDown at absent(node_network_receive_bytes_total{device=tun0}) — tun0 exists iff the
host OpenVPN tunnel is up (node_network_up unreliable: tun0 operstate=unknown). This also
fires if the gateway node/node-exporter is down (series goes stale). Remove the pod-readiness
EgressGatewayUnhealthy alert — there is no gateway pod under ADR-0015.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Prepared on a feature branch for operator review — NOT merged/pushed (retiring the running
fallback needs explicit approval). The node-level host OpenVPN tunnel steered by Cilium Egress
Gateway is validated end-to-end and gated by green conformance (egress-class + vpn-killswitch).
A Gluetun pod's tun0 was never steerable by Cilium egress gateway (the reason ADR-0015 moved the
tunnel to the node), so gluetun serves no steering role. The now-unused pia-* SOPS secrets are
retained pending a separate cleanup (creds are host-side in the Ansible vault).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
chainsaw v0.2.12 has no test-level `spec.finally` (finally exists only on a
step), and a step-level `finally` accepts only catch-style ops (script/command/
delete/wait), NOT `apply`/`assert`. Both tests failed to parse ("unknown field").
- egress-class: drop the redundant test-level `spec.finally` delete-pods block —
chainsaw auto-deletes applied fixtures on cleanup (skipDelete: false). The
valid step-level `finally` that restores the egress-class label is kept.
- vpn-killswitch: move the SAFETY-CRITICAL self-healing tunnel restart out of the
invalid `spec.finally` and into the STEP-level `finally` of the fault step
`when-force-tunnel-down-then-no-leak`, implemented as a `script` (finally can't
use apply/assert) that `kubectl apply`s the privileged node-pinned restart Job
(name `tunnel-restart`, distinct from `tunnel-stop`) and `kubectl wait`s for it
to complete — the equivalent of `assert succeeded:1`, blocking teardown until
the host confirms the tunnel is up. This finally runs even if the no-leak /
internet-unaffected asserts in the same step's `try` fail, so no path leaves the
gateway tunnel down. The recovery step is now poll-only (restart already
happened in the prior step's finally).
Validation with the pinned binary (~/.local/bin/chainsaw v0.2.12):
`chainsaw lint test --file ...` => "The document is valid" for both; `chainsaw
test ... --no-cluster` loads both past the parse stage (no "unknown field"), then
fails only at runtime for lack of a cluster (expected). yamllint + run.sh
--strict-coverage clean (RC=0). No live apply / fault injection, no spec edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Turn the vpn-killswitch scaffold into a runnable SAFETY-CRITICAL L3 test for
R-NET-06 under ADR-0015 (node-level host OpenVPN tunnel steered by Cilium Egress
Gateway with a host nftables kill-switch), replacing the obsolete in-cluster
gluetun design.
- fixtures/vpn-and-internet-pods.yaml: two egress-class probe pods + documented
RBAC/PodSecurity/namespace requirements for node-level fault injection.
- vpn-killswitch/chainsaw-test.yaml: GIVEN healthy tunnel (vpn egress != WAN);
WHEN the host `vpn-egress-tunnel` systemd unit is STOPPED via a privileged,
hostPID, node-pinned nsenter Job THEN vpn-class egress is blackholed (curl
fails, never returns the WAN IP = no leak) while internet-class is unaffected;
WHEN restarted THEN vpn egress recovers. spec-level `finally` ALWAYS runs an
idempotent start Job (self-healing) so a mid-test failure never leaves the
gateway tunnel down.
- incluster/rbac.yaml: grant the runner least-privilege for the delivered tests
(namespaces create/delete/patch for the PSA relabel, pods/services/configmaps/
pods-exec/pods-log, batch/jobs) and replace the gluetun-scale rule with the
ADR-0015 host-systemd mechanism note (privilege via PodSecurity, not RBAC).
Offline validation: python yaml parse (all docs), `run.sh --lint` (structural),
`run.sh --coverage-only --strict-coverage` (no tag/annotation drift; R-NET-05/06/
07 covered, R-NET-06 has its L3 test), yamllint. No live apply, no cluster run.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Session-limit interrupted the WS-TEST agent. egress-class test updated; vpn-killswitch
test + host fault-injection fixture still outstanding. Preserved for review/continuation.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The down-script previously flushed the kill-switch source set on tunnel-down, which
disarmed the 'ip saddr @vpn_egress_gateway_src oifname != tun0 drop' rule exactly when
the tunnel was down — leaving fail-closed to ride only on the stale SNAT source being
non-routable (could leak if the edge NATed it). Retain the set instead: the drop rule
stays armed so any vpn-class packet leaving the WAN while the tunnel is down is EXPLICITLY
dropped (edge-independent). Safe because the up-script flush+re-adds the source on every
reconnect. Validated behavior: WS-NET fault-injection on k3s-dev-3 (2026-08-24).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
The CiliumEgressGatewayPolicy authored in 38fcfad was inadvertently removed from main when
feature/vpn-egress-gateway-role (which carried a git-rm of the duplicated file) was merged.
The kustomization now references it, so restore the file. Content identical to 38fcfad.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Host OpenVPN split-tunnel on k3s-dev-3 validated 2026-08-24: tun0 up, VPN egress
158.173.241.36, node default unchanged (WAN), scoped nft kill-switch active. Wire the
CiliumEgressGatewayPolicy into the runtime kustomization so egress-class=vpn pods steer
to the node tun0.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author the vpn_egress_gateway role + playbook that stands up the NODE-level
PIA OpenVPN tunnel on egress-gateway node(s) — the host half of ADR-0015. The
CiliumEgressGatewayPolicy steering vpn-class pods is separate GitOps against the
same contract (group vpn_egress_gateway, node label
homelab.keithjackson.dev/egress-gateway=true, host iface tun0).
Honors all four ADR-0015 invariants:
- Split-tunnel (inv 1&2): OpenVPN route-noexec + pull-filter ignore
redirect-gateway => installs no routes; policy routing in a dedicated table
(51820) with `ip rule from <tun0-addr>` steers ONLY Cilium-SNAT'd vpn-class
traffic. Node default route stays on the WAN, tunnel up or down.
- Scoped nftables kill-switch (inv 3&4): own inet table, policy accept, drops
only tunnel-sourced (@vpn_egress_gateway_src set) packets not leaving tun0 =>
vpn-class fails closed (blackhole), node/internet-class fail open (WAN),
never matched.
PIA creds + the proven sweden-aes-128-cbc-udp-ip .ovpn come from Ansible vault
(host-side per ADR-0015/docs-07); vault keys documented, no secret values
committed. ansible-lint (production) + yamllint clean. Not deployed: green
conformance is gated on the live tunnel (operator host creds) — see role README.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add platform/vpn-egress/egress-steering.yaml — a CiliumEgressGatewayPolicy
that steers homelab.keithjackson.dev/egress-class=vpn pods out a gateway
node's host tun0 split-tunnel interface, per ADR-0015 (node-level tunnel +
Cilium Egress Gateway). Honors all four ADR-0015 invariants: only vpn-class
pods are selected (default/internet + node stay on the local/WAN path),
RFC1918 + cluster ranges are excludedCIDRs so only public internet is
tunneled and tun0 is never a default route, and the host nftables kill-switch
(Ansible) fails vpn-class CLOSED while internet-class fails OPEN.
Interface contract shared verbatim with the parallel Ansible host-tunnel role:
node label homelab.keithjackson.dev/egress-gateway=true, interface tun0,
client label homelab.keithjackson.dev/egress-class=vpn.
Cilium Egress Gateway prerequisites confirmed enabled (egressGateway.enabled,
bpf.masquerade, kubeProxyReplacement all true, [parity]) — no HelmRelease change.
Intentionally NOT added to platform/vpn-egress/kustomization.yaml: the datapath
stays out of the runtime until the live tunnel exists and the egress-class /
vpn-killswitch conformance tests pass (README rule; DoD). Gluetun is retained
as the validated fallback until then. Serves R-NET-05/06/07.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Author the internet/vpn egress-class fixture pods for the egress-class + vpn-killswitch
conformance tests. Record the datapath finding: ADR-0013's preferred steering (Cilium
egress gateway -> gluetun tun0) is NOT viable because gluetun must run hostNetwork to
put tun0 on the node, and gluetun then redirects the NODE's default route through the
VPN (breaks k3s/etcd). Escalated: remaining viable path is option 2 (gateway-pod
routing). No steering datapath committed until the mechanism is chosen + validated.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
prometheus-...-0 was OOMKilled (exit 137) every ~4min during TSDB compaction/checkpoint
(261 restarts/37h). Raise limit to 2Gi + request to 512Mi; nodes have headroom.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Once the tunnel redirects the pod default route via tun0, the pod's response to the
kubelet's httpGet liveness probe egresses the tunnel and never returns → probe times
out → pod restarts ~every 2min despite the tunnel being up (PIA egress IP confirmed).
Switch to a local wget exec probe (stays on lo).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
Operator provides a new .ovpn with an IP remote (gluetun's custom parser rejects
hostnames), so drop the resolve-remote initContainer/VPN_ENDPOINT_IP attempts and
mount the .ovpn directly (key pia.ovpn). Bump image to latest v3.41.3.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
gluetun's custom parser rejects the .ovpn's 'remote sweden.privacy.network' (needs
an IP). sweden.privacy.network LB pool = 158.173.241.{22,34,55,112,239}; pin .22
(verified to handshake) via VPN_ENDPOINT_IP/PORT while the .ovpn supplies cipher/CA.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
jakman85andClaude Opus 4.8