Commit Graph
100 Commits
Author SHA1 Message Date
jakman85 3b97f5f058 Merge fix/split-horizon-dns: in-cluster split-horizon DNS (R-DNS-01/03) — unblocks SSO 2026-08-24 22:29:59 -05:00
jakman85andClaude Opus 4.8 8b199a6780 test(dns): internal split-horizon resolution conformance (R-DNS-01/R-DNS-03)
Add dns-internal-resolution (L2 behavioral): an in-cluster Job resolves
keycloak.home.keithjackson.dev via the default cluster resolver and asserts the answer
is the in-cluster Gateway VIP (10.73.0.24x), not a public/off-cluster hairpin. Complements
dns-exposure (R-DNS-02, external record). coverage.py now maps R-DNS-01/R-DNS-03 to this
test (covered); chainsaw v0.2.12 lint clean; strict hygiene gate green.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:28:53 -05:00
jakman85andClaude Opus 4.8 3a1e0e6887 feat(dns): split-horizon in-cluster DNS via coredns-custom (R-DNS-01/R-DNS-03)
In-cluster clients resolving a platform hostname over its PUBLIC name got the
off-cluster answer, so pod->keycloak.home.keithjackson.dev hit a path that does not
serve the *.home.keithjackson.dev SNI -> `tls: unrecognized name` (oauth2-proxy OIDC
discovery crash; Grafana generic_oauth would hit the same wall).

Add a GitOps-managed coredns-custom ConfigMap (kube-system) that k3s CoreDNS imports as
a `*.server` block. The template plugin answers the whole home.keithjackson.dev zone with
${GATEWAY_VIP} (the shared Gateway VIP), so pod->public-hostname traffic hairpins through
the Gateway with the correct SNI -> wildcard cert validates. CoreDNS reloads the ConfigMap
live (no pod restart). Wired into platform/kustomization.yaml (Phase 2 network core).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:28:46 -05:00
jakman85andClaude Opus 4.8 bba67d03ce feat(dns): add GATEWAY_VIP cluster-var for split-horizon target (R-DNS-03)
Parameterize the in-cluster split-horizon target as GATEWAY_VIP (R-ENV-02): base
default = dev L2-announced Gateway VIP (10.73.0.240, first IP of the Cilium LB pool).
coredns-custom answers *.home.keithjackson.dev with ${GATEWAY_VIP} so the unified graph
holds across dev/prod via cluster-vars only. Prod overlay carries a TODO to set the prod
LAN VIP alongside the prod L2 pool range when hardware is built.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:28:37 -05:00
jakman85 186ff23098 Merge feature/servicedefinition-renderer: ServiceDefinition renderer T-0.3 (R-GITOPS-02) + L1 render tests green 2026-08-24 22:23:25 -05:00
jakman85andClaude Opus 4.8 a79f24f75e Add container-workload-render L1 test (R-PLAT-02)
The renderer's PaaS path (kind: container) had no verifying conformance test — the
R-PLAT-02 token appears only in renderer comments, so a change touching catalog/
surfaced it as an uncovered touched-requirement (blocking the merge gate). Adds the
container render analog of vm-lifecycle (R-PLAT-03): asserts a kind: container
ServiceDefinition renders a Deployment (running the declared image) + a Service,
with the egress-class/auth-mode labels admission requires. Offline render-check.sh,
wrapped by a chainsaw Test for in-cluster runs.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:22:04 -05:00
jakman85andClaude Opus 4.8 4a017c590e Wire L1 render conformance tests to the renderer (R-NET-01, R-NET-05/06/07)
Implements the two L1 rendering tests that were failing scaffolds:
- egress-class-render (R-NET-05/06/07): renders vpn + internet fixtures, diffs each
  against a committed golden, asserts the vpn steering label is present on vpn and
  absent on internet (fail-closed), and that flipping the class changes ONLY the
  egress path (R-NET-07). This fixes the --requirements R-NET-05 merge gate.
- default-deny-baseline-render (R-NET-01): asserts every render carries the
  namespaced default-deny baseline, shaped deny-all + both directions.

Each test carries a pure-offline render-check.sh (renderer + golden diff, no
cluster). run.sh runs L1 render checks offline and emits a REAL pass/fail JUnit
(docs/14: L1 rendering assertions run in CI without a cluster) instead of skipping
L1 with the cluster-gated layers; the chainsaw-test.yaml wrappers invoke the same
render-check.sh so an in-cluster run executes the identical assertion.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:19:08 -05:00
jakman85andClaude Opus 4.8 6b2c04ed5c Add ServiceDefinition renderer (T-0.3, R-GITOPS-02)
catalog/render.py compiles a ServiceDefinition into the manifest set defined by
schemas/renderer-contract.md: Namespace + default-deny baseline (R-NET-01),
Deployment/Service or KubeVirt VM + CDI DataVolume (R-PLAT-02/03), PVCs (R-STATE),
per-dependency CiliumNetworkPolicy allows (R-NET-02), egress-class steering label
(R-NET-05), HTTPRoutes on the shared Gateway (R-TLS/R-DNS), ServiceMonitor + SLI
alerts (R-OBS-01), Flux image-automation (R-UPD), auth-mode/backup hints (R-IDP).

Deterministic (golden-diffable) with a built-in --selftest; single dependency
(PyYAML) so the conformance harness stays stdlib-only. Documents the mechanism
choice in catalog/README.md and the script header.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:18:58 -05:00
jakman85 01bcb99337 Merge feature/phase4-image-automation: Phase 4 hands-off updates (R-UPD-01/02/03) 2026-08-24 22:17:44 -05:00
jakman85 0fb4ef2baf Merge fix/oauth2-proxy-cookie-secret (32-byte cookie) 2026-08-24 22:17:20 -05:00
jakman85andClaude Opus 4.8 951973b74d fix(oauth2-proxy): cookie-secret must be 32 raw bytes (hex16), not base64-44
oauth2-proxy read the base64 cookie value as raw bytes (44) → 'must be 16/24/32 bytes'
crashloop. Regenerated cookie-secret as openssl rand -hex 16 (32 bytes). client-id +
client-secret unchanged (still match the realm oauth2-proxy client). Value tracked in
~/.config/home-k3s/keycloak-sso-secrets.txt.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:17:20 -05:00
jakman85andClaude Opus 4.8 cbe5fab040 test(updates): complete hands-off-update L3 conformance (R-UPD-01/02/03)
Runnable, self-skipping (green until image.toolkit CRDs exist). Proves: an ImagePolicy
auto-advances to the newest STABLE tag on schedule with no edit and never a pre-release
(R-UPD-01/02, via podinfo ImageRepository/ImagePolicy status); a deliberately bad update
is health-gated and AUTO-ROLLS-BACK to the last-good revision while the old ReplicaSet
keeps serving, and the UpdateHealthGateFailed alert is wired (R-UPD-02/03); and an
exact-range (pinned) policy does not advance past its pin (R-UPD-03). Hermetic to the
chainsaw per-test namespace; block-style contains() checks per chainsaw v0.2.12.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:15:03 -05:00
jakman85andClaude Opus 4.8 bfec9d869e feat(updates): alert on failed / rolled-back updates (R-UPD-02 + R-OBS)
Add platform-health.automated-updates PrometheusRule group carried by platform-config:
ImageUpdateAutomationFailing (write-back broken -> advances silently stop),
ImagePolicyStalled (registry/range can't resolve a stable tag), and UpdateHealthGateFailed
(a bad update health-gated and auto-rolling-back via HelmRelease remediation). Routes to
Alertmanager email so a bad update is visible.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:14:54 -05:00
jakman85andClaude Opus 4.8 f538f8ff38 feat(updates): per-service update-policy render contract + Renovate config (R-UPD-01/R-UPD-03)
Map a ServiceDefinition `update` block to ImageRepository/ImagePolicy + the
$imagepolicy setter marker: stable-only (semver range + bare-X.Y.Z filterTags,
pre-release excluded, R-UPD-01), pin -> fixed image with no marker / suspended
policy (R-UPD-03). Worked pihole example under per-service/ (reference only, not
deployed; WS-GITOPS emits the equivalent into apps/<svc>/). Add renovate.json for
Helm chart / dependency / operator-manifest bumps: auto-merge minor/patch/digest on
green CI, major needs review, image managers disabled so Renovate never fights Flux
over a tag.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:14:48 -05:00
jakman85andClaude Opus 4.8 37f4fa6767 feat(updates): install Flux image controllers + cluster-wide write-back engine (R-UPD-02)
Install image-reflector-controller + image-automation-controller as part of the
flux-system install (bootstrap.sh --components-extra) so the image.toolkit CRDs
pre-exist before reconcile — no F-MON1 split needed for the ImageUpdateAutomation CR.
Refine the cluster-wide ImageUpdateAutomation for machine-authored git write-back
(setter strategy, path ./apps) reusing the flux-system GitRepository secretRef, and
document the operator git-write-credential hand-off + SOPS secret template
(gitea-image-automation-credentials). image-automation stays deferred in
platform/kustomization.yaml until that credential is provisioned.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:14:39 -05:00
jakman85 392f458d94 Merge feature/keycloak-sso-secrets (complete Phase 6 SSO: client secrets + oauth2-proxy) 2026-08-24 22:08:06 -05:00
jakman85andClaude Opus 4.8 ed6a772599 feat(keycloak): provision SSO client secrets + re-enable oauth2-proxy (R-IDP-01/02/04)
Generate the 3 OIDC client secrets (SOPS-encrypted, minimal manifests) so the realm
import can inject the grafana + oauth2-proxy client secrets and both consumers match:
- platform/keycloak/keycloak-realm-clients.sops.yaml (grafana + oauth2-proxy client secrets)
- platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml (client-id/secret/cookie)
- platform/monitoring/grafana-oidc-credentials.sops.yaml (grafana client secret)
Wire each into its component kustomization and re-enable oauth2-proxy in platform +
platform-config (reverses the temp deferral now that its secret exists).

Values generated by PM per operator instruction (random; recorded out-of-repo at
~/.config/home-k3s/keycloak-sso-secrets.txt for later rotation). Encrypted via sops in the
dev container to the age recipient in .sops.yaml; plaintext never committed.

Serves: R-IDP-01, R-IDP-02, R-IDP-04, R-SEC-01.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 22:08:04 -05:00
jakman85 f49b8c2e02 Merge chore/defer-oauth2proxy-pending-secret (unwedge platform for KubeVirt certification) 2026-08-24 21:43:24 -05:00
jakman85andClaude Opus 4.8 ef5b7bd57b chore(phase6): temp-defer oauth2-proxy until its SOPS secret is provisioned
oauth2-proxy's HelmRelease mounts existingSecret oauth2-proxy-credentials, an operator
hand-off not yet filled. Under the unified graph (platform wait:true), the not-ready HR
blocks platform -> platform-config -> the whole downstream graph including KubeVirt.
Deferring it (comment its line in platform/kustomization.yaml + its two entries in
platform-config) lets the no-secret components (KubeVirt/CDI, CNPG, Keycloak operator +
instance) reconcile and be certified now. Re-enable both together once the operator fills
platform/oauth2-proxy/oauth2-proxy-credentials.sops.yaml.

Serves: R-IDP-02 (deferred). PM sequencing correction — enable per phase AND readiness.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 21:43:12 -05:00
jakman85 fb9ff9e783 Merge docs/pm-phase5-6-integration (PM log: Phase 5/6 integration + deploy blocker) 2026-08-24 21:18:03 -05:00
jakman85andClaude Opus 4.8 1868a47701 docs(pm): log Phase 5 (KubeVirt) + Phase 6 (Keycloak) integration + deploy blocker
Both components authored by worktree-isolated agents, reviewed, integrated to main,
offline-green (MERGE GATE: PASS, dev+prod overlays render). Deploy blocked on the
push-to-gitea permission + 3 operator SOPS hand-offs. Full runbook + verify-at-deploy
escalations recorded.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 21:17:58 -05:00
jakman85 b95b540468 Merge feature/keycloak-auth-stack: Phase 6 Keycloak IdP + oauth2-proxy + CNPG (R-IDP-01..06, R-OBS)
# Conflicts:
#	platform-config/kustomization.yaml
#	platform/kustomization.yaml
2026-08-24 21:15:46 -05:00
jakman85 cb4fdf4a55 keycloak: README — operator+CNPG design, runbook, hand-offs 2026-08-24 21:10:39 -05:00
jakman85 4bebd4b6d2 keycloak: coverage tests + dashboard + reword cross-cutting tags 2026-08-24 21:07:27 -05:00
jakman85 4ba4337c67 Merge feature/kubevirt-iaas: Phase 5 KubeVirt+CDI IaaS proof (R-PLAT-03/04, R-STATE-01, R-NET-01, R-OBS) 2026-08-24 21:02:37 -05:00
jakman85 9da209c71e Merge branch 'main' into feature/kubevirt-iaas
# Conflicts:
#	conformance/run.sh
2026-08-24 21:01:29 -05:00
jakman85 0cb25c12e3 Merge branch 'main' into feature/keycloak-auth-stack 2026-08-24 20:59:51 -05:00
jakman85 cd4c44874f Merge chore/harness-baseline-spec-pin-gap-exemptions (green baseline for Phase 5/6) 2026-08-24 20:57:38 -05:00
jakman85andClaude Opus 4.8 f5f52b0b8c chore(harness): adopt spec pin b633b7f + exempt R-ENV-02/R-GITOPS-01 coverage gaps
Adopt the architect's spec advance (ADR-0013 amendment + ADR-0015 egress
steering/invariants) into ci/SPEC_PIN; the impl main pin was stale (a7fc19f) so the
R-TEST-06 spec-immutability gate was red on main. Verified the a7fc19f..b633b7f range
is spec-only architect commits.

Add R-ENV-02 and R-GITOPS-01 to conformance/run.sh DOCUMENTED_GAPS (same precedent as
R-STATE-02): the completeness gate's coarse path->requirement matcher keys ci/* and
clusters/* (and doc-header tokens) to these unscaffolded, other-workstream requirements,
false-blocking any infra/overlay/CI edit. Documented + reviewable, not silent. Logged a
WS-TEST follow-up to fix the matcher so real requirements need not be exempted.

Serves: R-TEST-06. PM/Integration-Release governance change.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 20:57:26 -05:00
jakman85 ce83c6af3b WIP keycloak stack (gate probe) 2026-08-24 19:52:25 -05:00
jakman85andClaude Opus 4.8 dbffb8fab8 Phase 5: KubeVirt + CDI IaaS proof (R-PLAT-03/04, R-STATE-01, R-NET-01)
- platform/kubevirt: pinned KubeVirt v1.9.0 + CDI v1.66.0 operators (remote bases);
  KubeVirt/CDI CRs tuned for Longhorn; useEmulation=true (prefer real KVM, emulate fallback).
- Controllers/configs split (F-MON1): CRs + ServiceMonitors + alerts via platform-config.
- Observability: ServiceMonitors, KubeVirt platform-health alerts (R-OBS-04), VM dashboard.
- catalog/vm-demo.yaml (kind: vm) + rendered VirtualMachine/DataVolume fixture.
- Conformance: vm-lifecycle (R-PLAT-03/04), vm-data-survival (R-STATE-01, L3),
  vm-default-deny (R-NET-01) — self-skipping until KubeVirt+longhorn exist.
- ci/SPEC_PIN: adopt accepted spec b633b7f (ADR-0015 egress work already merged).
- conformance/run.sh: document R-ENV-02/R-GITOPS-01 coverage gaps (WS-GITOPS/INFRA-owned).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 19:51:30 -05:00
jakman85andClaude Opus 4.8 254d50a20c docs: pia-* cluster secrets retired + host vault committed (Phase 3 cleanup complete)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 15:04:31 -05:00
jakman85 509a421d01 Merge chore/retire-pia-cluster-secrets (ADR-0015 cleanup) 2026-08-24 15:03:22 -05:00
jakman85andClaude Opus 4.8 d920aad74a Retire unused pia-* cluster SOPS secrets (ADR-0015 cleanup)
pia-credentials and pia-custom-ovpn were gluetun's PIA creds; gluetun is retired and the node
tunnel uses host-side Ansible vault creds (docs/07), so these cluster Secrets are unused. Remove
them from the kustomization + delete the encrypted files (recoverable from git history). Flux
will prune the Secrets from the vpn-egress namespace.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 15:03:11 -05:00
jakman85 b4ac1bef5e Merge chore/commit-vpn-egress-vault (host VPN creds vault, encrypted) 2026-08-24 15:01:54 -05:00
jakman85andClaude Opus 4.8 e9d738b705 vpn_egress_gateway: commit the ansible-vault host VPN creds vault (ADR-0015)
The node-tunnel PIA credentials (username/password/.ovpn) live host-side in Ansible vault per
docs/07. The encrypted vault.yml was provisioned but never committed, so the vpn.egress.gateway
playbook was not reproducible from a fresh checkout. Commit it (ansible-vault encrypted; the
vault password stays out of the repo via ansible.cfg vault_password_file).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 15:00:58 -05:00
jakman85andClaude Opus 4.8 2d85c03e55 docs: gluetun retirement complete (docs merged, banner fixed, worktrees pruned)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 14:54:37 -05:00
jakman85 75a38da374 Merge feature/fix-egress-steering-banner (mark steering datapath live) 2026-08-24 14:53:40 -05:00
jakman85andClaude Opus 4.8 b6e8adc6c6 docs: mark egress-steering.yaml as live (remove stale NOT-WIRED banner + done TODO)
The policy is wired into the kustomization, reconciled by Flux, and conformance-green; gluetun
is retired. Update the header banner + bottom checklist to reflect that. Comment-only — the
CiliumEgressGatewayPolicy spec is unchanged.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 14:53:39 -05:00
jakman85 9bc9b07caa docs: update vpn-egress docs/comments for gluetun retirement (ADR-0015)
README rewrite + ADR-0013->0015 comment drift across kustomization, kyverno egress-class
policy, servicedefinition schema, cluster-vars, the now-unused pia-* secret headers, and the
vpn-egress namespace note. Comments/docs only — no runtime, alert-expr, resource-list, or
SOPS-payload changes (both .sops.yaml verified: zero data/sops/mac diffs).
2026-08-24 14:52:10 -05:00
jakman85andClaude Opus 4.8 aafb4ca378 docs(vpn-egress): reflect ADR-0015 node-tunnel egress, retire gluetun narrative
The in-cluster Gluetun gateway is retired; VPN egress is now a node-level host
OpenVPN split-tunnel (Ansible role vpn_egress_gateway) steered by a Cilium
Egress Gateway policy with a host nftables kill-switch (ADR-0015). Credentials
moved host-side to the Ansible vault. Update docs/comments only:

- platform/vpn-egress/README.md: rewrite for ADR-0015 (live + conformance-green);
  drop the resolved "option 1 not viable with Gluetun" finding to a brief history note.
- platform/kustomization.yaml: Phase 3 comment -> node tunnel + Cilium steering.
- platform/kyverno/.../require-egress-class-label.yaml: ADR-0013 -> ADR-0015 steering.
- schemas/servicedefinition.schema.json: egress=vpn description -> node tunnel + Cilium.
- clusters/base/cluster-vars.yaml: reframe PIA_REGION / CLUSTER_POD_CIDR WHY comments
  to ADR-0015 (Cilium excludedCIDRs); both keys retained (still referenced/documented).
- platform/vpn-egress/namespace.yaml: no privileged gateway pod under ADR-0015.
- pia-credentials.sops.yaml / pia-custom-ovpn.sops.yaml: comment headers only note
  they are now UNUSED (creds host-side), retained pending cleanup; payloads untouched.

No runtime manifest behavior, alert exprs, kustomization resources, or SOPS payloads
changed. No spec-repo edits.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 14:50:08 -05:00
jakman85andClaude Opus 4.8 05b04b8bc7 docs: record gluetun retirement done + VPN monitoring retargeted to node tun0
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 14:47:43 -05:00
jakman85 9927b73a5a Retire gluetun gateway + retarget VPN tunnel monitoring to the node tunnel (ADR-0015)
Operator-approved. Removes the vestigial in-cluster gluetun Deployment/Service (the node-level
host tunnel steered by Cilium Egress Gateway is validated + conformance-green) and updates the
VPNTunnelDown alert to key on node tun0 presence (absent(node_network_receive_bytes_total{device=tun0}))
instead of the now-gone gluetun_vpn_up gauge; drops the obsolete pod-based EgressGatewayUnhealthy.
Unused pia-* SOPS secrets retained pending a separate cleanup.
2026-08-24 14:43:03 -05:00
jakman85andClaude Opus 4.8 d519ede4e3 monitoring: retarget VPNTunnelDown to node tun0 presence; drop obsolete EgressGatewayUnhealthy
gluetun retirement removes the gluetun_vpn_up gauge and the in-cluster gateway pod. Point
VPNTunnelDown at absent(node_network_receive_bytes_total{device=tun0}) — tun0 exists iff the
host OpenVPN tunnel is up (node_network_up unreliable: tun0 operstate=unknown). This also
fires if the gateway node/node-exporter is down (series goes stale). Remove the pod-readiness
EgressGatewayUnhealthy alert — there is no gateway pod under ADR-0015.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 14:41:20 -05:00
jakman85andClaude Opus 4.8 17fc0b7f36 docs: conformance suite green live (both gates); gluetun retirement prepared, pending approval
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 14:22:52 -05:00
jakman85andClaude Opus 4.8 7b7982e28d Retire gluetun gateway Deployment (ADR-0015: node tunnel validated + conformance-green)
Prepared on a feature branch for operator review — NOT merged/pushed (retiring the running
fallback needs explicit approval). The node-level host OpenVPN tunnel steered by Cilium Egress
Gateway is validated end-to-end and gated by green conformance (egress-class + vpn-killswitch).
A Gluetun pod's tun0 was never steerable by Cilium egress gateway (the reason ADR-0015 moved the
tunnel to the node), so gluetun serves no steering role. The now-unused pia-* SOPS secrets are
retained pending a separate cleanup (creds are host-side in the Ansible vault).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 14:21:59 -05:00
jakman85 0753b2ac96 Merge conformance tests for ADR-0015 egress steering (egress-class + vpn-killswitch)
Both validated GREEN live on dev (2026-08-24): L2 egress-class PASS (internet=WAN 136.51.x,
vpn=PIA 158.173.x distinct; R-NET-07 label-flip re-routes same pod, uid unchanged), L3
vpn-killswitch PASS (219s — tunnel-down => vpn fails closed no-leak, internet unaffected,
self-heal restart, recovery to PIA). chainsaw v0.2.12.
2026-08-24 14:18:02 -05:00
jakman85andClaude Opus 4.8 4c21f9f6e3 conformance: fix invalid test-level spec.finally for chainsaw v0.2.12
chainsaw v0.2.12 has no test-level `spec.finally` (finally exists only on a
step), and a step-level `finally` accepts only catch-style ops (script/command/
delete/wait), NOT `apply`/`assert`. Both tests failed to parse ("unknown field").

- egress-class: drop the redundant test-level `spec.finally` delete-pods block —
  chainsaw auto-deletes applied fixtures on cleanup (skipDelete: false). The
  valid step-level `finally` that restores the egress-class label is kept.
- vpn-killswitch: move the SAFETY-CRITICAL self-healing tunnel restart out of the
  invalid `spec.finally` and into the STEP-level `finally` of the fault step
  `when-force-tunnel-down-then-no-leak`, implemented as a `script` (finally can't
  use apply/assert) that `kubectl apply`s the privileged node-pinned restart Job
  (name `tunnel-restart`, distinct from `tunnel-stop`) and `kubectl wait`s for it
  to complete — the equivalent of `assert succeeded:1`, blocking teardown until
  the host confirms the tunnel is up. This finally runs even if the no-leak /
  internet-unaffected asserts in the same step's `try` fail, so no path leaves the
  gateway tunnel down. The recovery step is now poll-only (restart already
  happened in the prior step's finally).

Validation with the pinned binary (~/.local/bin/chainsaw v0.2.12):
`chainsaw lint test --file ...` => "The document is valid" for both; `chainsaw
test ... --no-cluster` loads both past the parse stage (no "unknown field"), then
fails only at runtime for lack of a cluster (expected). yamllint + run.sh
--strict-coverage clean (RC=0). No live apply / fault injection, no spec edits.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 14:09:28 -05:00
jakman85andClaude Opus 4.8 0e69c63b55 conformance: complete vpn-killswitch L3 test + ADR-0015 RBAC/fixtures
Turn the vpn-killswitch scaffold into a runnable SAFETY-CRITICAL L3 test for
R-NET-06 under ADR-0015 (node-level host OpenVPN tunnel steered by Cilium Egress
Gateway with a host nftables kill-switch), replacing the obsolete in-cluster
gluetun design.

- fixtures/vpn-and-internet-pods.yaml: two egress-class probe pods + documented
  RBAC/PodSecurity/namespace requirements for node-level fault injection.
- vpn-killswitch/chainsaw-test.yaml: GIVEN healthy tunnel (vpn egress != WAN);
  WHEN the host `vpn-egress-tunnel` systemd unit is STOPPED via a privileged,
  hostPID, node-pinned nsenter Job THEN vpn-class egress is blackholed (curl
  fails, never returns the WAN IP = no leak) while internet-class is unaffected;
  WHEN restarted THEN vpn egress recovers. spec-level `finally` ALWAYS runs an
  idempotent start Job (self-healing) so a mid-test failure never leaves the
  gateway tunnel down.
- incluster/rbac.yaml: grant the runner least-privilege for the delivered tests
  (namespaces create/delete/patch for the PSA relabel, pods/services/configmaps/
  pods-exec/pods-log, batch/jobs) and replace the gluetun-scale rule with the
  ADR-0015 host-systemd mechanism note (privilege via PodSecurity, not RBAC).

Offline validation: python yaml parse (all docs), `run.sh --lint` (structural),
`run.sh --coverage-only --strict-coverage` (no tag/annotation drift; R-NET-05/06/
07 covered, R-NET-06 has its L3 test), yamllint. No live apply, no cluster run.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 13:58:40 -05:00
jakman85andClaude Opus 4.8 09569a08e6 docs: kill-switch robustness fix done+validated; conformance tests partial (rate-limited)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 13:13:45 -05:00
jakman85andClaude Opus 4.8 e7a567db0b WIP conformance(egress-class): ADR-0015 assertions (partial — killswitch test TODO)
Session-limit interrupted the WS-TEST agent. egress-class test updated; vpn-killswitch
test + host fault-injection fixture still outstanding. Preserved for review/continuation.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 13:12:07 -05:00
jakman85 b0ff689484 Merge feature/killswitch-explicit-nft-backstop (explicit fail-closed nft backstop) 2026-08-24 13:12:07 -05:00
jakman85andClaude Opus 4.8 1c4d86319c vpn_egress_gateway: retain nft src set on tunnel-down (explicit fail-closed backstop)
The down-script previously flushed the kill-switch source set on tunnel-down, which
disarmed the 'ip saddr @vpn_egress_gateway_src oifname != tun0 drop' rule exactly when
the tunnel was down — leaving fail-closed to ride only on the stale SNAT source being
non-routable (could leak if the edge NATed it). Retain the set instead: the drop rule
stays armed so any vpn-class packet leaving the WAN while the tunnel is down is EXPLICITLY
dropped (edge-independent). Safe because the up-script flush+re-adds the source on every
reconnect. Validated behavior: WS-NET fault-injection on k3s-dev-3 (2026-08-24).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 13:10:40 -05:00
jakman85andClaude Opus 4.8 e14048e7fe docs: record Phase 3 egress steering validated end-to-end on dev (both gates pass)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 08:23:57 -05:00
jakman85 8a555999ee Merge fix/restore-egress-steering-file into main 2026-08-24 08:08:35 -05:00
jakman85andClaude Opus 4.8 c74e05bda9 fix: restore egress-steering.yaml (dropped by an earlier branch merge)
The CiliumEgressGatewayPolicy authored in 38fcfad was inadvertently removed from main when
feature/vpn-egress-gateway-role (which carried a git-rm of the duplicated file) was merged.
The kustomization now references it, so restore the file. Content identical to 38fcfad.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 08:08:35 -05:00
jakman85 c9874ca57c Merge feature/enable-vpn-egress-steering into main 2026-08-24 08:07:21 -05:00
jakman85andClaude Opus 4.8 54341964fc Enable ADR-0015 vpn-egress steering (host tunnel validated on k3s-dev-3)
Host OpenVPN split-tunnel on k3s-dev-3 validated 2026-08-24: tun0 up, VPN egress
158.173.241.36, node default unchanged (WAN), scoped nft kill-switch active. Wire the
CiliumEgressGatewayPolicy into the runtime kustomization so egress-class=vpn pods steer
to the node tun0.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 08:07:20 -05:00
jakman85andClaude Opus 4.8 83ea050e82 docs: record ADR-0015 impl (merged, not deployed) + escalation ruling + operator prereq
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 07:32:58 -05:00
jakman85 33d048d379 Merge feature/vpn-egress-gateway-role into main (ADR-0015 host tunnel role, not yet deployed) 2026-08-24 07:31:27 -05:00
jakman85 af75a6ddc9 Merge feature/vpn-egress-cilium-steering into main (ADR-0015 steering policy, not yet wired) 2026-08-24 07:31:27 -05:00
jakman85andClaude Opus 4.8 2a2ead8b28 chore: drop egress-steering.yaml from this branch (owned by feature/vpn-egress-cilium-steering; working-tree race)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 07:31:26 -05:00
jakman85andClaude Opus 4.8 2c109e2a31 feat(vpn): node-level split-tunnel egress gateway Ansible role (ADR-0015)
Author the vpn_egress_gateway role + playbook that stands up the NODE-level
PIA OpenVPN tunnel on egress-gateway node(s) — the host half of ADR-0015. The
CiliumEgressGatewayPolicy steering vpn-class pods is separate GitOps against the
same contract (group vpn_egress_gateway, node label
homelab.keithjackson.dev/egress-gateway=true, host iface tun0).

Honors all four ADR-0015 invariants:
- Split-tunnel (inv 1&2): OpenVPN route-noexec + pull-filter ignore
  redirect-gateway => installs no routes; policy routing in a dedicated table
  (51820) with `ip rule from <tun0-addr>` steers ONLY Cilium-SNAT'd vpn-class
  traffic. Node default route stays on the WAN, tunnel up or down.
- Scoped nftables kill-switch (inv 3&4): own inet table, policy accept, drops
  only tunnel-sourced (@vpn_egress_gateway_src set) packets not leaving tun0 =>
  vpn-class fails closed (blackhole), node/internet-class fail open (WAN),
  never matched.

PIA creds + the proven sweden-aes-128-cbc-udp-ip .ovpn come from Ansible vault
(host-side per ADR-0015/docs-07); vault keys documented, no secret values
committed. ansible-lint (production) + yamllint clean. Not deployed: green
conformance is gated on the live tunnel (operator host creds) — see role README.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 01:09:19 -05:00
jakman85andClaude Opus 4.8 38fcfadbf0 phase3(vpn): author Cilium Egress Gateway steering policy (ADR-0015)
Add platform/vpn-egress/egress-steering.yaml — a CiliumEgressGatewayPolicy
that steers homelab.keithjackson.dev/egress-class=vpn pods out a gateway
node's host tun0 split-tunnel interface, per ADR-0015 (node-level tunnel +
Cilium Egress Gateway). Honors all four ADR-0015 invariants: only vpn-class
pods are selected (default/internet + node stay on the local/WAN path),
RFC1918 + cluster ranges are excludedCIDRs so only public internet is
tunneled and tun0 is never a default route, and the host nftables kill-switch
(Ansible) fails vpn-class CLOSED while internet-class fails OPEN.

Interface contract shared verbatim with the parallel Ansible host-tunnel role:
  node label homelab.keithjackson.dev/egress-gateway=true, interface tun0,
  client label homelab.keithjackson.dev/egress-class=vpn.

Cilium Egress Gateway prerequisites confirmed enabled (egressGateway.enabled,
bpf.masquerade, kubeProxyReplacement all true, [parity]) — no HelmRelease change.

Intentionally NOT added to platform/vpn-egress/kustomization.yaml: the datapath
stays out of the runtime until the live tunnel exists and the egress-class /
vpn-killswitch conformance tests pass (README rule; DoD). Gluetun is retained
as the validated fallback until then. Serves R-NET-05/06/07.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 00:59:36 -05:00
jakman85 272a7d7d2f Merge feature/phase3-egress-steering-groundwork into main 2026-08-24 00:26:27 -05:00
jakman85andClaude Opus 4.8 34f41a0fba phase3(vpn): egress-class conformance fixtures + steering design finding
Author the internet/vpn egress-class fixture pods for the egress-class + vpn-killswitch
conformance tests. Record the datapath finding: ADR-0013's preferred steering (Cilium
egress gateway -> gluetun tun0) is NOT viable because gluetun must run hostNetwork to
put tun0 on the node, and gluetun then redirects the NODE's default route through the
VPN (breaks k3s/etcd). Escalated: remaining viable path is option 2 (gateway-pod
routing). No steering datapath committed until the mechanism is chosen + validated.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 00:26:26 -05:00
jakman85 f650a4dc8e Merge fix/prometheus-oom into main 2026-08-24 00:04:18 -05:00
jakman85andClaude Opus 4.8 7b619f8bb0 fix(monitoring): Prometheus memory limit 1Gi -> 2Gi (OOMKilled on compaction)
prometheus-...-0 was OOMKilled (exit 137) every ~4min during TSDB compaction/checkpoint
(261 restarts/37h). Raise limit to 2Gi + request to 512Mi; nodes have headroom.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 00:04:17 -05:00
jakman85andClaude Opus 4.8 03768ff6b0 docs: Phase 3 VPN resolved (working + stable, PIA Stockholm egress)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-24 00:02:47 -05:00
jakman85 db39879e1c Merge feature/vpn-liveness-exec into main 2026-08-23 23:57:32 -05:00
jakman85andClaude Opus 4.8 4a781ace49 fix(vpn): exec (localhost) liveness probe — VPN routing breaks kubelet HTTP probe
Once the tunnel redirects the pod default route via tun0, the pod's response to the
kubelet's httpGet liveness probe egresses the tunnel and never returns → probe times
out → pod restarts ~every 2min despite the tunnel being up (PIA egress IP confirmed).
Switch to a local wget exec probe (stays on lo).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-23 23:57:31 -05:00
jakman85 7e41cc0282 Merge feature/vpn-custom-ovpn-ip into main 2026-08-23 01:01:20 -05:00
jakman85andClaude Opus 4.8 63f6e15d0e feat(vpn): new IP-based .ovpn secret + map its key to /config/pia.ovpn
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-23 01:01:19 -05:00
jakman85andClaude Opus 4.8 c8b9653e33 fix(vpn): custom mode with IP-based .ovpn; gluetun v3.41.0 -> v3.41.3
Operator provides a new .ovpn with an IP remote (gluetun's custom parser rejects
hostnames), so drop the resolve-remote initContainer/VPN_ENDPOINT_IP attempts and
mount the .ovpn directly (key pia.ovpn). Bump image to latest v3.41.3.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-23 00:57:01 -05:00
jakman85 3787644c48 Merge feature/vpn-endpoint-ip into main 2026-08-23 00:50:54 -05:00
jakman85andClaude Opus 4.8 66a76e6e33 fix(vpn): VPN_ENDPOINT_IP for custom mode (.ovpn hostname not parseable)
gluetun's custom parser rejects the .ovpn's 'remote sweden.privacy.network' (needs
an IP). sweden.privacy.network LB pool = 158.173.241.{22,34,55,112,239}; pin .22
(verified to handshake) via VPN_ENDPOINT_IP/PORT while the .ovpn supplies cipher/CA.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-23 00:50:54 -05:00
jakman85 3c359e2937 Merge feature/vpn-ovpn-keyname into main 2026-08-23 00:46:31 -05:00
jakman85andClaude Opus 4.8 998b46df3d fix(vpn): map .ovpn secret key (filename) to /config/pia.ovpn
Operator's kubectl create secret keyed the file by its name
(sweden-aes-128-cbc-udp-dns.ovpn); mount item now references that key -> pia.ovpn.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-23 00:46:31 -05:00
jakman85 6c34fb636d Merge feature/vpn-custom-ovpn into main 2026-08-23 00:45:39 -05:00
jakman85andClaude Opus 4.8 47b6cb6afd feat(vpn): add operator-provisioned pia-custom-ovpn Secret (SOPS)
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-23 00:45:39 -05:00
jakman85andClaude Opus 4.8 91aa7acab8 wip(vpn): gluetun custom-provider mode with operator's PIA .ovpn
PIA-provider server selection is flaky here (random dead IPs); the operator's .ovpn
uses sweden.privacy.network (load-balanced, reliable). Switch to custom mode
(VPN_SERVICE_PROVIDER=custom, OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn), mount the
pia-custom-ovpn Secret, keep creds + kill-switch + tun-mtu 1300. Drops PIA-provider
server list/updater and native PF (Phase-6 concern). Pending the operator-provisioned
pia-custom-ovpn Secret (not merged until it exists).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-23 00:40:08 -05:00
jakman85 ee349c6d94 Merge feature/vpn-preset-envname into main 2026-08-23 00:18:03 -05:00
jakman85andClaude Opus 4.8 be1e317456 fix(vpn): correct env name PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET
gluetun silently ignored PIA_ENCRYPTION_PRESET (renamed) and defaulted to strong
(RSA-4096) the whole time. The correct v3.x var is
PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET=normal → RSA-2048/AES-128-CBC/1198,
the smaller handshake that fits the 1450 pod MTU.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-23 00:18:03 -05:00
jakman85 93010a5287 Merge feature/vpn-normal-preset into main 2026-08-23 00:09:48 -05:00
jakman85andClaude Opus 4.8 6a44d973b4 fix(vpn): PIA normal preset (RSA-2048/AES-128-CBC/SHA1, 1198) for MTU reliability
RSA-4096 strong handshake wouldn't hold over the pod's 1450 MTU (Cilium VXLAN);
MTU tuning is exhausted (no --tls-mtu in OpenVPN 2.6, PIA rejects --fragment).
Operator downloaded the lower-encryption PIA config (port 1198, RSA-2048,
AES-128-CBC, SHA1) = gluetun's 'normal' preset. Smaller handshake fits 1450.
Dropped the strong-handshake band-aid flags; kept OPENVPN_MTU 1300 for the data channel.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-23 00:09:47 -05:00
jakman85 8bcccf518e Merge feature/vpn-mtu-disc into main 2026-08-22 23:56:28 -05:00
jakman85andClaude Opus 4.8 772d032d4c tune(vpn): --mtu-disc no so oversized handshake packets fragment (not drop)
RSA-4096 strong-preset handshake is marginal over the 1450 pod MTU (Cilium VXLAN).
--mtu-disc no clears DF so oversized control packets fragment through instead of
being silently dropped. Last MTU-path lever (OpenVPN 2.6 has no --tls-mtu).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 23:56:28 -05:00
jakman85 605b99409d Merge feature/vpn-mtu-tune2 into main 2026-08-22 23:50:01 -05:00
jakman85andClaude Opus 4.8 0e2289a2a8 fix(vpn): valid OpenVPN flags only (--hand-window 120; --tls-mtu invalid in 2.6)
OpenVPN 2.6.16 rejects --tls-mtu (not a real option) and PIA doesn't support
--fragment, so control-channel packets can't be shrunk. Keep --hand-window 120 to
give the RSA-4096 handshake more time over a lossy/marginal path, plus tun-mtu 1300.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 23:50:00 -05:00
jakman85 bc42c4bee9 Merge feature/vpn-mtu-tune into main 2026-08-22 23:46:37 -05:00
jakman85andClaude Opus 4.8 b859e52697 tune(vpn): shrink OpenVPN control-channel MTU + widen handshake window
RSA-4096 strong-preset handshake is marginal over the pod's 1450 MTU (Cilium VXLAN).
Lower tun-mtu to 1300 and add --tls-mtu 1200 (smaller control packets) + --hand-window
120 (more time to complete the handshake over a lossy path). Empirical test of the
MTU-tuning path (operator's choice over Cilium native routing).

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 23:46:36 -05:00
jakman85 49dca0e8bb Merge feature/vpn-pf-cidr-narrow into main 2026-08-22 23:36:51 -05:00
jakman85andClaude Opus 4.8 7d2cea8402 fix(vpn): narrow FIREWALL_OUTBOUND_SUBNETS to 10.0.0.0/16 (PF gateway overlap)
PIA's PF gateway 10.31.128.1 sits inside the /8 cluster-pool, so
FIREWALL_OUTBOUND_SUBNETS=10.0.0.0/8 made gluetun route it around the tunnel and
port forwarding timed out (getSignature to 10.31.128.1:19999). /16 covers actual
pod allocations and excludes PIA's 10.31.x.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 23:36:51 -05:00
jakman85 1899a54c07 Merge feature/vpn-pia-strong-preset into main 2026-08-22 23:30:47 -05:00
jakman85andClaude Opus 4.8 41234b85f5 fix(vpn): PIA strong encryption preset (RSA-4096/AES-256-CBC/SHA256, port 1197)
Operator's working config is sweden.privacy.network:1197 with RSA-4096, AES-256-CBC,
SHA256 = PIA's 'strong' preset. gluetun defaults PIA to 'normal' (AES-128-GCM), a
cipher mismatch against the 1197 (strong) endpoint, so the handshake never completed
(silent after UDP link). PIA_ENCRYPTION_PRESET=strong matches it and selects 1197.
Reverts the 8080 port red herring.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 23:30:46 -05:00
jakman85 7bdcd6edda Merge feature/vpn-port-8080-live into main 2026-08-22 23:26:17 -05:00
jakman85andClaude Opus 4.8 b76de52b7f test(vpn): use port 8080 (avoid home 1197 return-path conflict), live servers
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 23:26:17 -05:00
jakman85 b4bd18d716 Merge feature/vpn-updater-enduser into main 2026-08-22 23:17:16 -05:00
jakman85andClaude Opus 4.8 2e5fdd29d7 fix(vpn): gluetun update needs -enduser flag in the init container
Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 23:17:16 -05:00
jakman85andClaude Opus 4.8 a735784f94 fix(vpn): initContainer fetches PIA's current server list (bundled list stale)
UPDATER_PERIOD only refreshes AFTER the tunnel is ready (chicken-and-egg: can't
connect with stale servers), and gluetun's bundled PIA Sweden list is retired
46.246.x (current is 158.173.x). Add an `update-servers` initContainer running
`gluetun update -providers "private internet access"` into a shared /gluetun
emptyDir the main container reads, so it dials current servers. UPDATER_PERIOD
raised to 720h (docs-compliant) for ongoing in-pod refresh.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
2026-08-22 23:14:42 -05:00