# Autonomous Session Decision Log (PM)

Program Manager working independently while the operator is AFK (started 2026-08-16). Every
best-effort decision and assumption is recorded here for later review. **Specs were NOT modified**
(PM cannot change `../home-network-k3s`; that requires the System Architect hat). Blocked items that
need the operator are called out.

## Guardrails I'm holding to
- No spec changes. No fabricated secrets (Cloudflare token, PIA creds, SMTP — all await hand-off).
- Don't break the running cluster: `platform/` stays **suspended** in Flux until each component is
  filled in + validated; `platform/cilium` must carry `k8sServiceHost=10.73.0.11` before it ever
  reconciles.
- Everything author-first, `make -C ci gate` green before merge, feature-branch→main.

## Decisions / assumptions (append-only)
| # | Decision / assumption | Why | Review flag |
|---|---|---|---|
| A1 | Complete Phase 1 T-1.4 via **Option A** (Flux bootstrapped, `platform`+`apps` suspended) | You picked Option A earlier as the safe path; "keep going" authorizes it | Confirm scope |
| A2 | Advance Phase 2 only for components needing **no operator secret**: Longhorn, monitoring, Kyverno-admission, Cilium-parity | cert-manager/externaldns/velero/VPN are blocked on token/PIA/OQ-BKP | — |
| A3 | Dev resource caps for monitoring (short retention, small PVCs, single replicas) from `cluster-vars` | 22 GiB RAM ceiling | Tune later |
| A4 | Host Gitea repo is **public** (NAT-only) so Flux needs no git creds | dev bootstrap simplicity | Fine for dev |

## Autonomous session results (2026-08-16)
- **Phase-2 authoring (author-only, merged):** `platform/longhorn` (3-replica SC + bulk indirection),
  `platform/monitoring` (kube-prometheus-stack + Loki + Alloy, dev-trimmed ~1.3 GiB, +8 alert rules),
  `platform/cilium` (parity fix: k8sServiceHost=10.73.0.11 + masquerade, safe to adopt in Phase 2),
  `platform/kyverno` (verified). Gate green.
- **Flux INSTALLED** on the cluster (4 controllers Running). Host **Gitea working** (named-volume fix),
  repo pushed. GitRepository created → **BLOCKED**: cluster pods can't reach `10.73.0.1:3000`
  (firewalld `libvirt` zone drops VM→host:3000).
  - **RESOLVED (operator added the ansible-vault become password):** ran `host-prereqs.yml` (firewall
    opened via vault→become), VM→Gitea = 200, and **Flux GitRepository is READY=True**
    (`main@sha1:a2f3bdc…`). **Phase-1 T-1.4 core is DONE** — Flux installed + reconciling clusters/dev
    from the host Gitea. `platform`/`apps` Kustomizations intentionally NOT created (Option A) — the
    running cluster is untouched.
  - Firewall + Gitea admin/repo/push + host become password are all IaC now (host-prereqs.yml,
    bootstrap-gitea.sh automation, vaulted group_vars/hypervisor) → destroy/rebuild is hands-off.
- **NOT deployed:** `platform`/`apps` Kustomizations were deliberately NOT created — the running
  cluster (Cilium etc.) is untouched. Only Flux controllers + the GitRepository source exist.

## T-1.5 — Destroy/rebuild proof (destructive IaC-validity test) — 2026-08-16

**Goal (operator-scoped):** prove the IaC is *valid* — the dev cluster can be destroyed and rebuilt
from IaC + git with no data loss ("Approach A is acceptable"; "more about validity of the IaC than a
COOP/DR scenario"). Ran **after K2 conformance passed**, with the operator approving each destructive
step at the prompt.

**Outcome: PASS (with one operator-visible scare and two concrete IaC findings).** The cluster was
torn down and reconstituted from IaC + preserved Longhorn/OS disks + the Flux git source. **End state:
3/3 Ready (k3s-dev-1/2/3, control-plane,etcd,master), Cilium status OK, Flux GitRepository READY=True,
zero state lost.** Behavioral health re-verified post-rebuild: conformance L2 **R-NET-01 default-deny +
R-NET-02 deny-undeclared both PASS** (exit 0, JUnit+coverage regenerated). The destroy/rebuild path is
demonstrably valid.

### Finding F1 — `tofu destroy -target` on a `count`-indexed resource is unsafe (ROOT CAUSE)
`tofu destroy -target 'libvirt_domain.node[2]'` (+ that node's `os[2]`/`data[2]`/`seed[2]`) **destroyed
all three domains**, not just node3 — while the individually-targeted *volumes* correctly hit only
index 2. Root cause: the nodes are declared with **`count`** in `infra/tofu/main.tf`; `-target` against a
single `count` index is a documented Terraform/OpenTofu hazard (the docs explicitly warn `-target` is
"not for routine use"). count-index addresses are positional and brittle under targeted destroys.
- **Recommended fix (Phase-2 IaC hardening):** refactor `libvirt_domain.node` + its per-node volumes
  from `count` to **`for_each`** keyed by node name (`k3s-dev-1/2/3`). `for_each` keys are stable and
  individually addressable, so a single node can be `-replace`d/destroyed without touching siblings.
- **Interim rule:** **never** use `tofu destroy -target` on this module for single-node lifecycle. To
  cycle one node, use `tofu apply -replace='libvirt_domain.node["k3s-dev-N"]'` (post-`for_each`), or
  destroy the specific VM via `virsh` and let `tofu apply` recreate it.
- **Silver lining:** the over-destroy is exactly what *proved* IaC validity — disks survived, `tofu
  apply` recreated the domains, nodes rejoined, Flux reconciled unchanged. The IaC round-trips.

### Finding F2 — single-node Ansible rejoin needs the primary in scope for the join-token
Recovering node3 with `ansible-playbook ... --limit k3s-dev-3` **failed**: the `k3s_server` join task
reads `hostvars[groups['k3s_primary'][0]].k3s_node_token`, but that fact is only *published* when the
primary play runs, so a limit that excludes the primary yields
`'HostVarsVars' has no attribute 'k3s_node_token'`. Worked with `--limit 'k3s-dev-1,k3s-dev-3'`.
- **Recommended fix (Phase-2 IaC hardening):** make the join self-contained — have the secondary task
  fetch the token on demand via `delegate_to: "{{ groups['k3s_primary'][0] }}"` (read
  `/var/lib/rancher/k3s/server/node-token` at join time) instead of depending on a fact the primary
  play must have populated in the same run. Then a true single-node rejoin (`--limit k3s-dev-N`) works
  hands-off, which is the T-1.5 durability bar.

### F1 / F2 — IMPLEMENTED (author-first, 2026-08-16)
- **F1 done:** `infra/tofu/main.tf` node resources refactored `count` → `for_each` (keyed by node
  name); ordered-list/`primary` helper locals added; `outputs.tf` updated off the map; **`moved.tf`**
  adds 12 `moved` blocks (os/data/seed/domain × 3). `tofu validate` OK and **`tofu plan` = `0 to add,
  0 to change, 0 to destroy`** with all 12 resources reported "has moved to" — a pure state-address
  rename, no VM churn. **State-persist pending:** a follow-up `tofu apply` (a proven 0-change apply
  that only rewrites state addresses) is operator-permission-gated; until then the migration re-shows
  harmlessly on each plan and lands automatically on the next real apply.
- **F2 done:** `roles/k3s_server/tasks/main.yml` — secondary now fetches the join token via
  `delegate_to: primary` (with the published fact as a fast path), so `--limit k3s-dev-N` rejoin is
  self-contained. `ansible-playbook --syntax-check` OK on all playbooks; role lint clean under the CI
  gate (`make -C ci gate` PASS). No cluster change; the running cluster is untouched.

## Phase 2 un-suspend PREP — no-secret rollout (author-first, 2026-08-16)

Operator directive: "Phase 2 un-suspend prep … verify cilium parity + fill REPLACE-MEs for the
no-secret components … author-first. Does NOT un-suspend/deploy yet." **Nothing was deployed;** the
running cluster is untouched (still 0 Flux Kustomizations / 0 HelmReleases — only controllers + the
GitRepository).

### Findings from three parallel readiness audits (Longhorn / monitoring / Kyverno)
- **Cilium parity VERIFIED (safety-critical):** every `[parity]` field in `platform/cilium/helmrelease.yaml`
  equals the live CNI — `k8sServiceHost=10.73.0.11` (rendered by the chart as the pod env
  `KUBERNETES_SERVICE_HOST`, **not** a configmap key — the empty `k8s-service-host` cm value was a red
  herring), kube-proxy-replacement, gateway-api, egress+bpf masquerade, hubble, l2-announcements,
  cgroup, v1.16.5. **Flux adoption is a no-op reconcile.**
- **Topology:** "suspended" = Option A (**no root Kustomization was ever created**), and the single base
  `platform` Kustomization renders **all 11 components at once** — so a naive un-suspend would drag in
  the secret-blocked components (cert-manager/externaldns/velero/keycloak/oauth2-proxy/image-automation)
  and never go healthy (`apps` dependsOn platform, wait:true).
- **Longhorn:** manifests clean; node prereqs (open-iscsi, /var/lib/longhorn mount, iscsid) are in
  cloud-init IaC; the REPLACE-ME bulk-NFS SC is isolated/harmless. Ready.
- **Monitoring:** right-sized (~1.25 GiB) but blocked by (a) needing the `longhorn` SC first and (b) two
  **by-name secrets** (`grafana-admin-credentials`, `alertmanager-smtp`) that don't exist — the missing
  mounts (not the REPLACE-ME text) are what wedge Grafana/Alertmanager.
- **Kyverno:** zero policy violations today (policies scope to `app-namespace=true`, none exist), but the
  fail-closed webhook + 3 HA replicas + Enforce are unsafe/oversized for a cold cluster.

### DECISION D-P2A — per-component Flux Kustomizations for the dev no-secret rollout (operator-approved)
Chose **Option A (per-component Kustomizations)**. Authored `clusters/dev/rollout/` = 4 Flux
Kustomizations (`kyverno` → `cilium` → `longhorn` → `monitoring`, monitoring `dependsOn: longhorn`),
and the dev overlay now `$patch: delete`s the monolithic base `platform`/`apps` and adds `./rollout`.
The no-secret tier carries **no SOPS content**, so the missing `sops-age` key does not block it
(decryption dropped from these Kustomizations). Secret-blocked components are simply **not created**
until their hand-offs land.
- **cilium** uses `prune: false` (adopt the out-of-band CNI; never own-delete it).
- **R-ENV-02 divergence — REVIEW FLAG (architect):** the dev overlay now *adds resources* and diverges
  its reconciliation graph from prod (base doc says "dev adds NO resources … graph reused verbatim").
  This is a **transitional bootstrap scaffold**, consistent with the already-inlined dev bootstrap
  patches (SOPS/URL). **Convergence path is coded in:** delete `clusters/dev/rollout/` + the two
  `$patch: delete`s to re-adopt the unified base graph once all hand-offs arrive. **prod is untouched**
  (still builds the monolith; verified). Flag for architect ratification.

### Component hardening (env-gated via cluster-vars so prod keeps the strict posture)
- New base knobs (strict/prod defaults) + dev overrides: `KYVERNO_VALIDATION_ACTION` (Enforce→**Audit**),
  `KYVERNO_WEBHOOK_FAILURE_POLICY` (Fail→**Ignore**), `KYVERNO_ADMISSION_REPLICAS` (3→**1**),
  `MONITORING_STORAGE_CLASS` (longhorn), `GRAFANA_ADMIN_SECRET` (grafana-admin-credentials→**""** in dev
  → chart-default password, dev-only/NAT-isolated).
- Kyverno webhook also gains an explicit `namespaceSelector` excluding kube-system/flux-system/kyverno/
  longhorn-system/monitoring (belt-and-suspenders vs a fail-closed brick).
- Monitoring Alertmanager: the `alertmanager-smtp` mount + smtp_*/email_configs are **deferred** (commented
  with verbatim re-enable markers) so Alertmanager starts with NULL receivers — alerts fire/route/show in
  UI+Grafana, just no email until the Phase-4 SMTP hand-off.
- Validated: `kubectl kustomize` clean for clusters/dev (exactly the 4 Kustomizations, monolith removed),
  clusters/prod (monolith intact), and all 4 components; envsubst→YAML-parse of kyverno+monitoring valid;
  `make -C ci gate` PASS.

### Still required BEFORE an actual deploy (not part of this author-first prep)
1. **Push current `main` to the host Gitea** — Flux source is pinned at `6c3f9ed` (pre-dates these merges).
2. **Runtime node check:** `systemctl is-active iscsid` + `mountpoint /var/lib/longhorn` on all 3 nodes.
3. **Cilium adoption mechanism:** confirm helm-controller can adopt the Ansible-installed release (was it
   `helm`-installed with a release secret, or CLI/kubectl?) — verify before creating the cilium Kustomization.
4. Create the dev root Kustomization (the actual un-suspend) — deliberately NOT done here.

## CORRECTION — credential-forward, unified graph (operator directive, 2026-08-16)

**Operator pushback:** "Are we making implementation choices solely on the fact that you have to ask
for credentials? It is better to get the credentials upfront (encrypted file) and keep the rollout in
line with the existing architecture." **Correct — and it exposed a scope distortion.**

- **A2 SUPERSEDED / D-P2A REVERTED.** Choosing components by "needs no operator secret" inverted the
  roadmap: `roadmap/phases.md` **Phase 2 = Network, Storage, TLS, DNS, Ingress** → the credential-gated
  cert-manager/externaldns/gateway ARE the Phase-2 core, while **monitoring is Phase 4** (I'd pulled it
  forward). The per-component dev divergence (`clusters/dev/rollout/`) existed only to dodge asking for
  the Cloudflare token.
- **Reverted to the existing architecture:** deleted `clusters/dev/rollout/` + the `$patch: delete`s;
  restored the unified base graph (dev = ../base + cluster-vars patches + SOPS decryption patches, one
  `platform` Kustomization, R-ENV-02 intact — **removes the earlier architect-review flag**). Reverted
  the monitoring Grafana/SMTP dodges (it's Phase 4). Kept the non-credential-driven improvements: tofu
  for_each/rejoin (F1/F2), and Kyverno parametrization + webhook namespace-excludes (Kyverno IS Phase 2).
- **DECISION D-P2B — phase the unified `platform/kustomization.yaml` (operator-approved).** Components
  are enabled in the single shared list **per phase** (the same pattern that deferred kyverno):
  Phase-2 set now active = **kyverno, cilium, longhorn, cert-manager, externaldns, gateway**; monitoring/
  image-automation (P4), velero (P4, OQ-BKP), kubevirt (P5), keycloak/oauth2-proxy (P6) are commented
  with phase markers. dev+prod render the same list. Verified: `kubectl kustomize platform` renders only
  the 6 Phase-2 namespaces; dev shows the unified platform+apps Kustomizations; prod builds; gate PASS.
- **DECISION D-SEC1 — provision ALL credentials upfront via SOPS; operator encrypts (both operator-chosen).**
  Activated SOPS: `.sops.yaml` installed at repo root (recipient still REPLACE-ME — operator fills after
  age-keygen). Authored `bootstrap/sops/secret-templates/*.yaml.example` (cloudflare-api-token ×2 ns,
  grafana-admin, alertmanager-smtp, keycloak-admin, oauth2-proxy, pia — all placeholder-only, `.example`
  so kustomize/SOPS ignore them) + `bootstrap/sops/CREDENTIALS.md` (end-to-end checklist). Pre-wired
  **commented** `- cloudflare-api-token.sops.yaml` lines in cert-manager + externaldns kustomizations
  (operator uncomments post-encrypt). Filled the non-secret ACME email (keithallenjackson@gmail.com,
  confirm/change). **Agents never touch real secret values** (per `.sops.yaml`).

### Operator TODO to reach Phase-2 deploy (see bootstrap/sops/CREDENTIALS.md)
1. age-keygen → fill root `.sops.yaml` recipient → install `sops-age` on cluster.
2. Provision `cloudflare-api-token` (Zone:DNS:Edit + Zone:Read on keithjackson.dev) in **both**
   cert-manager + external-dns; uncomment the two kustomization lines.
3. Confirm ACME email. 4. Push `main` to host Gitea. 5. Create the dev root Kustomization (the un-suspend).
6. (Upfront, optional now) encrypt the Phase-3/4/6 secrets so later phases are credential-ready.
7. **Chart-key verification** (flagged in templates): keycloak + oauth2-proxy `existingSecret` key names,
   Keycloak DB secret, and PIA/Gluetun protocol (≠ TCP) — verify at each component's phase.

## Phase-2 UN-SUSPEND executed + fixes (autonomous session, 2026-08-16 overnight)

Operator said "go" for the un-suspend, then went AFK: "continue to work without me. Any major decision
that requires a system architecture change VIA SPEC CHANGE will need to be escalated. Anything else is
allowed. Keep track of assumptions and report on return." None of the below is a spec change (the spec
is silent on Flux Kustomization granularity / L2 wiring), so all within PM remit. **Nothing touched the
read-only spec repo.**

- **Un-suspended** via the sanctioned `kubectl apply -k clusters/dev` (bootstrap.sh step 3): controllers
  already installed, `sops-age` present. Pre-deploy checks all green first — Cilium adoption safe (an
  existing `helm` release `cilium` v3 exists → Flux adopts by name, no-op upgrade), Longhorn node prereqs
  green on all 3 nodes (`iscsid` active, `/var/lib/longhorn`→`/dev/vdb`), host Gitea pushed + Flux source Ready.

### Fix 1 — Longhorn StorageClass numeric coercion (commit ~c04aea8)
Reconcile stalled: `.parameters.numberOfReplicas: expected string, got 3`. kustomize strips the quotes
around a bare `"${VAR}"` placeholder, so Flux substitution yielded an unquoted numeric that SSA rejects
(StorageClass.parameters is map[string]string). **Hardcoded `numberOfReplicas: "3"`** (quoted numeric
literals ARE preserved by kustomize; topology LOCKED at 3). The replica knob still drives the global
default via `defaultSettings.defaultReplicaCount` in the helmrelease (a Helm int value, unaffected).

### Fix 2 — DECISION: controllers/configs split (commit ~458a867) — NOTABLE, please review
Reconcile then failed atomically: `no matches for kind "ClusterIssuer"`. The single `platform`
Kustomization applied HelmReleases (which register CRDs) AND CRs of those CRDs (ClusterIssuer,
Certificate, Kyverno ClusterPolicies) in one pass → Flux SSA fails the WHOLE apply → the CRD-registering
HelmReleases never run. Deadlock. **Adopted the flux2 controllers/configs pattern:** new
`platform-config` Flux Kustomization (`dependsOn: platform`, wait) carries the CRs; `platform` keeps only
controllers; `apps` now `dependsOn platform-config`. Applies to dev AND prod (shared base) — no R-ENV-02
divergence. `platform-config/` references the CR files IN PLACE via `../platform/...` (Flux builds with
LoadRestrictionsNone; Kyverno policies must stay under `platform/kyverno/` for the CI L0 `kyverno test`
gate) — **ASSUMPTION: Flux's default LoadRestrictionsNone permits the `../` refs** (builds verified
locally with `--load-restrictor LoadRestrictionsNone`; confirming at reconcile).

### Fix 3 — Authored the missing Cilium L2 resources (in Fix 2 commit)
`l2announcements.enabled: true` was set but the IP pool + announcement policy were only a TODO, so a
Gateway LoadBalancer would never get an IP. Authored `platform-config/cilium/`: **CiliumLoadBalancerIPPool**
(dev NAT range **10.73.0.240-.250**, matching the tofu reservation) + **CiliumL2AnnouncementPolicy**
(interface regex `^en`, node interface is `ens3`). **ASSUMPTIONS:** (a) VIP range hardcoded to the dev
NAT range — a prod overlay must parameterize it (via cluster-vars) when prod is built; (b) `^en` matches
the node LAN NIC (`ens3`); (c) CRD apiVersion `cilium.io/v2alpha1` (confirmed served on the cluster).

### Fix 4 — Kyverno chart webhook schema (commit ~c6331c8)
Kyverno HelmRelease install FAILED at Helm templating: I'd written `config.webhooks` as a LIST
`[{failurePolicy, namespaceSelector}]`, but the kyverno 3.3.9 chart expects `config.webhooks` to be a
MAP (`webhooks.namespaceSelector`), and the fail-open switch is a separate global
`config.forceFailurePolicyIgnore.enabled` (bool), not a per-webhook field. Fixed both; renamed the
cluster-var `KYVERNO_WEBHOOK_FAILURE_POLICY`(Fail/Ignore) → `KYVERNO_FORCE_FAILURE_POLICY_IGNORE`
(false base / true dev). Verified with `helm template` before pushing.
**PROCESS SLIP:** this fix landed directly on `main` (I forgot to branch first). Content is gate-green;
noted for the record. Fixes 5+ went back through feature branches.

### Fix 5 — cert-manager DNS-01 self-check via public resolvers (commit ~59dd3d9)
Wildcard cert stalled: challenge `Presented=true` (Cloudflare TXT created — the token WORKS) but the
propagation self-check looped "DNS record not yet propagated". VERIFIED the record IS publicly
resolvable (1.1.1.1/8.8.8.8); the NAT'd k3s-dev cluster's own DNS just can't see it. Set
`dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"` + `dns01RecursiveNameserversOnly: true` (realises
a pre-existing TODO in the file). Challenge went pending→valid, cert **Issued**.

### Fix 6 — cilium-operator restart to program the Gateway (MANUAL recovery, see note)
Gateway stuck `Accepted=Unknown:Pending` for ~20min. cilium-operator logs: `no kind is registered for
the type v1.Gateway in scheme` → `one-shot job errored ... leader-lifecycle` — the operator's
gateway-controller failed scheme registration during the Flux **adoption** restart and gave up (rest of
the operator ran fine). `kubectl -n kube-system rollout restart deploy/cilium-operator` fixed it
instantly: Gateway → Accepted+Programmed, **address 10.73.0.240** from the L2 pool, `cilium-gateway-shared`
LB service created. **ASSUMPTION/NOTE:** this was a manual (non-IaC) recovery of a transient adoption
casualty; a fresh ansible bootstrap installs the Gateway API CRDs BEFORE Cilium, so a clean rebuild
should not hit it — VERIFY on the next destroy/rebuild, and if it recurs, codify an operator-restart
handler (or bump Cilium). Not codified now.

### RESULT — Phase-2 platform FULLY UP + green (2026-08-17)
`platform`, `platform-config`, `apps` all Ready=True (revision 59dd3d9). 5/5 HelmReleases Ready;
wildcard `*.home.keithjackson.dev` cert Issued (LE-staging per ADR-0006); Gateway Programmed @
10.73.0.240; 6 Kyverno policies Ready (Audit in dev); Longhorn default SC live; no not-ready pods.

### FOLLOW-UPS (non-blocking, for review)
1. **Two default StorageClasses** — k3s's built-in `local-path` AND `longhorn` are both marked
   default. Ambiguous. Fix: disable k3s local-storage in the ansible k3s_server args
   (`--disable local-storage`) OR strip the default annotation from local-path so `longhorn` is the
   sole default. (Non-blocking: platform components pin `storageClassName: longhorn` explicitly.)
2. **Phase-2 EXIT not yet verified** — the roadmap Phase-2 exit is passing conformance (R-NET-01
   default-deny, R-TLS-01/03 cert served, R-DNS-02 auto record). Platform is UP; conformance run is the
   next step (dev cert is LE-STAGING = non-self-signed but not browser-trusted — expected in dev).
3. **cilium-operator gateway restart** (Fix 6) — verify on next rebuild / codify if it recurs.

## Autonomous session 2 — Phase-2 verification + hardening (2026-08-17)

Operator away again, same SOP. All safe, no spec changes.

- **R-NET conformance GREEN against the DEPLOYED platform:** ran `conformance/run.sh --layer l2
  --requirements R-NET-01,R-NET-02` → **default-deny (R-NET-01) + deny-undeclared (R-NET-02) both PASS**
  (2/2). Confirms the live platform enforces the network isolation baseline, not just the K2 fixtures.
- **TLS path functionally VERIFIED (not the formal test):** `openssl s_client` to the gateway VIP
  10.73.0.240 with SNI returns `CN=*.home.keithjackson.dev`, SAN {*.home, home}.keithjackson.dev,
  issuer **Let's Encrypt (STAGING)**, 90-day validity. The full DNS-01 → issuance → gateway-termination
  path works. (Dev cert is LE-STAGING per ADR-0006 = non-self-signed but NOT system-trusted.)
- **Fix: single default StorageClass** (commit ~7a45eb1) — added `local-storage` to the k3s
  `--disable` list so `longhorn` is the sole default. Idempotent install task → effective on rebuild,
  no disruption to the running cluster (which harmlessly keeps both defaults; all platform PVCs pin
  longhorn). ASSUMPTION: disabling k3s local-path entirely is fine (Longhorn is the storage).
- **Fix: conformance run.sh CWD-robustness** (commit ~ee5ad90) — chainsaw's config `report.path` is
  relative, so run.sh exited non-zero (breaking the L2 gate) when invoked from repo root. Pass absolute
  `--report-path ${ARTIFACTS}` + mkdir. Verified exit 0 + report written from repo root.

### REMAINING for Phase-2 EXIT (needs an operator call — did NOT do autonomously)
The roadmap Phase-2 exit is conformance-green for R-TLS + R-DNS too, but those tests are Phase-0
**scaffolds** (TODO placeholders) — they must be AUTHORED against the live platform. Two reasons I
deferred rather than best-effort them:
1. **R-TLS trusted-cert (trusted-tls test) — DECISION NEEDED:** the test checks the SYSTEM trust store
   (no -k), but dev serves an LE-**staging** cert (not system-trusted). How should dev verify "trusted"?
   (a) verify the cert chains to the real LE **staging** root (embed the staging root in the probe) —
   most faithful; (b) verify LE-issued + valid + SAN + non-self-signed only in dev, full system-trust
   in prod; (c) use LE-prod for a designated test host. **Recommend (a) or (b).** Side-effect-free to
   implement once decided.
2. **R-DNS auto-record (dns-exposure test) — EXTERNAL SIDE-EFFECT:** verifying R-DNS-02 means letting
   ExternalDNS create/delete a real record in the **live keithjackson.dev Cloudflare zone**. Deferred
   to run with the operator present (or confirm it's fine to exercise autonomously with guaranteed
   cleanup).

## Phase-2 CONFORMANCE EXIT ACHIEVED (2026-08-17, main @ 1762206 + cluster-vars re-apply)

Operator decided: TLS trust = **A** (chain to LE-staging root); DNS test = **OK** to run live.
Authored both Phase-0 scaffolds into real tests against the deployed platform; **all 4 Phase-2 L2
tests PASS**: R-NET-01 (default-deny), R-NET-02 (deny-undeclared), R-TLS-01/03 (trusted-tls),
R-DNS-02 (dns-exposure). This covers the roadmap Phase-2 exit demonstrables (default-deny holds,
trusted cert served at *.home.keithjackson.dev, auto DNS record appears).

- **trusted-tls (R-TLS-01/03):** in-cluster curl probe dials the gateway Service (env-agnostic via
  `--connect-to`, no hardcoded VIP) with wildcard SNI and requires the served chain to verify against
  the embedded **LE-staging root** ("(STAGING) Pretend Pear X1", fixtures/le-staging-root-ca.yaml).
  Probes a public + an internal-only hostname (R-TLS-03). Empirically validated: correct root → rc 0;
  system trust → rc 60.
- **dns-exposure (R-DNS-02):** a LoadBalancer Service + external-dns hostname annotation; polls public
  DNS (1.1.1.1) until ExternalDNS publishes the A record into the cluster LB range; chainsaw teardown
  + `policy=sync` removes it. Verified the full CREATE→DELETE lifecycle in ExternalDNS logs; the live
  keithjackson.dev zone is left clean (back to the pre-existing wildcard).

### FINDING F-DNS1 — ExternalDNS domain-filter must be the ZONE, not the subdomain (FIXED)
The DNS test first failed: ExternalDNS generated the record but skipped it — "no hosted zone matching
record DNS Name". `--domain-filter=home.keithjackson.dev` filtered OUT the real Cloudflare zone
`keithjackson.dev` (a subdomain filter drops the parent zone). Added a `DNS_ZONE` cluster-var
(keithjackson.dev) and set `domainFilters: [${DNS_ZONE}]`. Scope stays effectively home.keithjackson.dev
(only suffix services request) and `policy=sync`+txt-registry never touch non-owned records.

### FINDING F-OPS1 — cluster-vars changes need a manual `kubectl apply -k` — **RESOLVED (main @ 4381a71)**
Added `clusters/dev/sync.yaml`: a self-managing ROOT Kustomization (flux2 gotk-sync pattern, name
`flux-system`) that reconciles `./clusters/dev` from git — cluster-vars, the GitRepository, platform/
platform-config/apps, and itself (inventory = 6 objects, confirmed). Bootstrap unchanged
(`kubectl apply -k clusters/dev` creates it once). Verified: root → Ready, adopts the graph without
disruption, all Kustomizations settle green; a git push now auto-applies overlay/cluster-vars changes.
prod needs its own sync.yaml (→ ./clusters/prod) when built. Original gap description below:


Adding `DNS_ZONE` didn't take effect until I re-ran `kubectl apply -k clusters/dev` (the on-cluster
`cluster-vars` ConfigMap isn't reconciled by Flux — this bootstrap model has NO self-managing root
Kustomization that watches `clusters/dev`). Then a `platform` Kustomization reconcile re-ran the
postBuild substitution. **RECOMMEND: add a self-managing root Flux Kustomization** (path ./clusters/dev)
so cluster-vars / graph changes auto-apply — otherwise every overlay change is a manual re-apply
(and, transiently, ${DNS_ZONE} substituted empty → `--domain-filter=` = all-zones, before re-apply).

### FINDING F-KY1 — kyverno chart's cleanup image (bitnami/kubectl) purged from Docker Hub (FIXED)
Surfaced when a `platform` reconcile triggered a kyverno helm upgrade: the chart's cleanup jobs
(`policyReportsCleanup`, `webhooksCleanup`) default to `bitnami/kubectl:1.30.2`, which Bitnami REMOVED
from Docker Hub (2024 catalog purge) → ImagePullBackOff. `policyReportsCleanup` runs as a Helm
**post-upgrade HOOK**, so its failure stalled the upgrade (5m timeout) → kyverno HR not-Ready → platform
Unknown → platform-config/apps cascade to False. **Fixed:** override both cleanup images to Bitnami's
relocated `bitnamilegacy/kubectl` mirror (identical image — has /bin/bash + awk/wc/xargs + kubectl the
jobs need; verified via `helm template` that no `bitnami/kubectl` ref remains). After the fix the hook
job Completed, kyverno HR → True, and the whole graph returned to green (platform/platform-config/apps
all Ready, 5/5 HelmReleases Ready, no unhealthy pods). **NOTE:** `bitnamilegacy` is itself a temporary
mirror — a durable follow-up is to pin a maintained kubectl image (shell-capable, e.g. rancher or
alpine/k8s) once tested, or a newer kyverno chart that drops the bitnami dependency.

### R-STATE-01 GREEN + harness fix F-TEST1 (2026-08-17, main @ 76478f2)
- **R-STATE-01 (data durability) PROVEN:** the L3 `data-survival` test runs against live Longhorn — writes
  a canary to a `longhorn` PVC, DESTROYS the pod entirely, re-binds a fresh pod to the same PVC, and the
  data survives (`DATA_SURVIVED_OK`). This is the platform's headline stateful-durability requirement.
- **F-TEST1 — every chainsaw `check` used an unsupported `=~` operator (FIXED):** all 10 tests wrote
  `(($stdout =~ 'X')): true`; chainsaw's expression engine rejects `=~` ('SyntaxError: Unknown char: ~'),
  so each test FAILED on the assertion even when the behavior passed (caught because data-survival printed
  DATA_SURVIVED_OK yet FAILED). Converted all to JMESPath `contains($stdout, 'X')` in BLOCK style (flow
  `{ }` breaks on the comma inside `contains()`). data-survival passes; the 9 later-phase tests
  (egress/vpn/alerting/dr/updates/drift/sso) are now syntactically correct for when their phases deploy.
- **R-NET-01/02 FAIL-CLOSED (L3) PROVEN (main @ cde3335):** implemented the `default-deny-failclosed`
  scaffold — fixtures = default-deny baseline + source + dest:8080 + the sole explicit allow. Verifies
  source→dest works, then REVOKES the allow (real fault: `kubectl delete networkpolicy`), and asserts
  connectivity fails CLOSED while DNS still resolves. Proves default-deny is datapath-enforced, not just
  declared. PASSES. (DNS check asserts the resolved answer — busybox nslookup exits non-zero on success.)
  **Conformance green now: R-NET-01/02 (L2+L3), R-TLS-01/03, R-DNS-02, R-STATE-01.** Remaining tests need
  later-phase components (VPN P3, monitoring/backup/updates P4, SSO P6) → operator secrets.

## Phase 3 + 4 ("Both") + an infra recovery (2026-08-17, main @ 4f17c53)

Operator: WireGuard preferred (I confirmed PIA supports it on all regions + Gluetun derives the WG
config from username/password, so the PIA secret shape is unchanged); keycloak creds added; oauth2-proxy
deferred to Phase 6; "proceed to next phase." Chose **Both** (deploy Phase-4 monitoring + author Phase-3 VPN).

### INFRA RECOVERY — cluster VMs + Gitea were found STOPPED (not my doing)
Mid-task, kubectl + git push failed: the 3 k3s VMs were **shut off** and the Gitea container **stopped**
(host up 12.5h — not a reboot; something halted them). Recovered via the operator-provisioned
ansible-become path: `virsh -c qemu:///system start k3s-dev-{1,2,3}` → cluster back, all Kustomizations +
HelmReleases Ready from disk in ~80s; restarted Gitea (named volume `gitea-bootstrap-data` persisted) and
pushed the queued commits. No data lost. (The VMs are SYSTEM-libvirt — user-session virsh shows nothing;
recovery needs sudo, which only the ansible vault-become has.)

### Phase 4 — MONITORING DEPLOYED + healthy
Enabled `monitoring` in platform/kustomization.yaml. All 3 HelmReleases Ready (kube-prometheus-stack
65.8.1, Loki 6.55.0, Alloy 0.12.6); Prometheus + Loki PVCs bound on Longhorn (5Gi each); 38
PrometheusRules applied; no unhealthy pods. Grafana admin + Alertmanager SMTP from the operator's SOPS
secrets. **SMTP caveats (flagged, not blocking):** operator set smtp_smarthost=smtp.gmail.com:587,
from=alertmanager@keithjackson.dev, user=keithallenjackson@gmail.com; I set the receiver `to:` to a valid
inbox (was REPLACE-ME). Gmail needs an **App Password** and the From must be the auth user or a verified
send-as alias, else delivery fails at alert time (stack health unaffected).

### FINDING F-MON1 — monitoring PrometheusRule CRD ordering (FIXED)
Enabling monitoring first deadlocked the platform Kustomization: `alerts.yaml` (PrometheusRule CRs) needs
the monitoring.coreos.com CRDs the kube-prometheus-stack HelmRelease installs in the SAME pass → atomic
'no matches for kind PrometheusRule' → NO monitoring HRs created. Same controllers/configs fix
(F-DNS1/kyverno lineage): moved `alerts.yaml` to platform-config (dependsOn platform). This is now a
RECURRING pattern — any component shipping CRs of a CRD it installs must split the CRs into platform-config.

### Phase 3 — VPN egress gateway AUTHORED (author-first, not deployed)
`platform/vpn-egress/`: single shared **Gluetun** gateway (PIA/WireGuard, kill-switch FIREWALL=on, native
PF, control-API Service), non-app namespace (NET_ADMIN + /dev/net/tun), pre-wired (commented)
pia-credentials line. Defined the PIA secret location (**platform/vpn-egress**, keys username/password →
OPENVPN_USER/OPENVPN_PASSWORD) and updated the SOPS template. NOT wired into the graph. **Open design
(README):** egress steering (`vpn`-class pods → the one shared tunnel) — recommended Cilium Egress Gateway
→ the Gluetun node's `wg0` (fail-closed when the tunnel drops), deliberately NOT committed as an untested
CiliumEgressGatewayPolicy. Deploy needs PIA creds + `PIA_REGION` (PF-capable) + `CLUSTER_POD_CIDR` +
steering validation + the qbittorrent-natmap port-sync helper. **Cluster fully green** (4/4 KS, 8/8 HR).

## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed)
| # | Issue | Impact | Fix |
|---|---|---|---|
| K1 | ~~Substitution-var case mismatch (UPPERCASE refs vs lowercase cluster-vars keys)~~ | — | ✅ **FIXED**: uppercased cluster-vars keys + dev/prod overlay paths; all refs resolve; builds green |
| K2 | ~~Conformance L2 tests partial/unverified~~ | — | ✅ **DONE**: fixed 3 test bugs (see below), ran via run.sh against the live cluster → **R-NET-01 default-deny + R-NET-02 deny-undeclared PASS**, JUnit+coverage written, exit 0. chainsaw v0.2.12 installed (host `~/.local/bin`) |
| K3 | `bootstrap/gitea/bootstrap-gitea.sh` used a host bind-mount → rootless-podman perms broke app.ini | initial Gitea start failed | FIXED: switched to a named volume |
| K4 | **Kyverno/chainsaw findings during K2 (fixed in the tests):** (a) chainsaw does STRICT (length-equal) list matching — asserting one condition against a pod's full `conditions[]` fails → used scalar `phase: Running`; (b) a default-deny baseline blocks the **kubelet readiness probe** (node→pod), so probe-bearing pods never go Ready → dropped probes; (c) chainsaw report `path: ../_artifacts` resolved to repo-root → exit 1 → fixed to `_artifacts`; (d) swapped agnhost→busybox httpd (cached, cache-timing-proof) | — | FIXED in the test files/config |

## FINDING for WS-NET (Phase 2, T-2.2 — default-deny baseline design)
The platform's real default-deny baseline **must not break kubelet health/liveness/readiness probes**
(node→pod ingress). A naive `ingress: []` NetworkPolicy in Cilium drops the kubelet TCP probe, so any
pod with a probe never becomes Ready. The baseline must explicitly allow the health-check path (Cilium
host/health-check allowance or an equivalent ingress exception). Surfaced empirically by the R-NET-02
conformance test; **carry into T-2.2**.

## Toolchain note
`chainsaw v0.2.12` is installed on the host (`~/.local/bin`) for local conformance runs; for full
reproducibility it should also be baked into `infra/devcontainer` (pinned) — the in-cluster Chainsaw
Job (ADR-0011) already uses the pinned image. Deferred (non-blocking).

## Blocked on operator (hand-offs)
- **T-2.4p** Cloudflare API token → cert-manager DNS-01 + ExternalDNS (Phase 2 TLS/DNS).
- **T-3.2p** PIA username/password → VPN egress gateway (Phase 3).
- **OQ-BKP** off-cluster backup target → Velero (Phase 4).
- SMTP creds → Alertmanager email (Phase 4).

## Disk/drive incident + k3s relocated to NVMe (2026-08-22)
**Trigger:** host root fs (Kingston SSD `/dev/sdb3`, btrfs) hit ~99% (<1GB); the system paused the k3s
VMs on I/O error. Operator asked to move the VM disks off root to a larger drive.

**Why root filled:** the k3s VM disks lived under the snapper-snapshotted root subvolume; high VM-disk
churn multiplied across hourly snapshots. Freed by deleting the 11 `Snapper` timeline snapshots +
`virsh destroy`-ing the paused VMs (which held the old, rsync-unlinked inodes open) → btrfs reclaim.
Root 99% → 30% (326G free).

**F-DISK1 — the Seagate 4TB HDD is FAILING.** First relocated the VM disks to the HDD (`/mnt/seagate`);
etcd then hit **48% iowait + constant "apply took too long"**, pods drifted to Unknown, never converged.
SMART on `/dev/sda` (ST4000DM000): **29,408 reallocated, 448 current-pending + 448 offline-uncorrectable**,
~25k power-on hrs. A dying drive, not "HDD is slow." Do NOT use for etcd/critical data.

**Resolution:** k3s disks moved to the **WD_BLACK SN770 NVMe** (`/dev/nvme0n1p3`, `/mnt/games`) in a
dedicated btrfs subvol `/mnt/games/libvirt-k3s` (nodatacow), **bind-mounted to `/var/lib/libvirt/images`**
so tofu/libvirt paths + the qcow2 backing chain are unchanged (no IaC edits). etcd iowait → 0%, cluster
fully converged (64 Running). fstab updated; SELinux relabeled. Archival VMs (rocky10/silverblue/win11)
repointed to their real HDD path and LEFT on the failing drive.

**Follow-ups (assumptions logged):** (1) the NVMe bind-mount is a runtime change — codify in an Ansible
`host-prereqs` task for reproducibility; (2) redundant k3s copies remain at `/mnt/seagate/libvirt-images/k3s-dev-*`
(safe to delete; NVMe authoritative) — left pending operator OK; (3) archival VMs on the failing HDD need
migration + the disk replaced.

## Phase 3 VPN — protocol resolved to OpenVPN-UDP; deploying (2026-08-22)
ADR-0013 said PIA/WireGuard, but **Gluetun does not support PIA over WireGuard** (upstream: native support
"cannot be added… slow WIP"; live error "provider name is not valid for Wireguard"). **Operator (as System
Architect) approved OpenVPN-UDP and ADR-0013 was amended in the spec repo** (protocol only; architecture
unchanged; `thrnz/docker-wireguard-pia` recorded as the WireGuard fallback). Impl aligned: `VPN_TYPE=openvpn`,
`vpn-egress` ENABLED in `platform/kustomization.yaml`, stale WireGuard comments cleaned (gluetun-gateway.yaml,
README; steering iface `wg0`→`tun0`). Pushed to Gitea → Flux reconciles the OpenVPN gateway. Stale WireGuard
comments remain inside the operator's encrypted `pia-credentials.sops.yaml` header — left untouched (editing
breaks the SOPS MAC); cosmetic.

**Next:** confirm the OpenVPN tunnel comes up + correct egress IP, then egress-steering validation (README
option 1, `tun0`) + kill-switch conformance (R-NET-06).

### Phase 3 VPN — tunnel BLOCKED by home network (2026-08-22)
After deploying OpenVPN + fixing the region (`SE Stockholm`), gluetun goes **Running** and OpenVPN dials
PIA Sweden servers over **UDP/1197**, but the handshake never completes — no `TLS: Initial packet from`
the server, gluetun healthcheck restarts it in a loop. **Not auth** (no `AUTH_FAILED`), **not a k8s
NetworkPolicy** (vpn-egress is `app-namespace=false`, no policies touch it), **not the cluster/host path**.

Traced with tcpdump end-to-end:
- On the node (k3s-dev-3): pod `10.0.2.161` → Cilium SNAT → `ens3 Out 10.73.0.13 > 46.246.x:1197`. Leaves fine.
- On the host: `virbr1 In 10.73.0.13 > 46.246.x:1197` → `wlp4s0 Out 192.168.2.103 > 46.246.x:1197`. Host
  NATs it out the **WiFi** onto the `192.168.2.x` home LAN. Leaves fine.
- On the host `wlp4s0`, filtering `src net 46.246.0.0/16`: **0 reply packets in 20s.** Meanwhile host
  UDP/53 (`dig @9.9.9.9`) works. → OpenVPN UDP replies never return to the host.

**Conclusion:** the break is UPSTREAM of the host — the home router/ISP on the `192.168.2.x` WiFi network
is dropping/not-returning OpenVPN UDP/1197 (TCP egress + UDP/53 both work; only the VPN UDP is affected;
this box is on 192.168.2.x via WiFi, not the main 192.168.1.x LAN). **Environmental, outside the repo.**

**Action → operator:** fix at the network (check the 192.168.2.x router for outbound UDP/1197 or VPN
filtering; try a wired connection to the main 192.168.1.x LAN; or a different port/provider). No in-config
workaround — the user requires non-TCP, and WireGuard (the fallback) is also UDP so likely blocked the same
way. `vpn-egress` reverted to disabled meanwhile (config is complete + correct; one-line re-enable once the
network passes OpenVPN UDP).

### Phase 3 VPN — RESOLVED: it was a port-1197 collision, moved to UDP/53 (2026-08-22)
Operator clarified: the home network already runs **its own OpenVPN server NAT'd on WAN:1197**, which
collided with the cluster's outbound PIA-on-1197 flow at the router (why UDP/53 worked but 1197 didn't —
port-specific, not a general UDP block). Fix: set `OPENVPN_ENDPOINT_PORT` to a non-1197 PIA OpenVPN port.
PIA accepts `53, 1194, 1197, 1198, 8080, 9201` (gluetun issue #2252). Chose **53** — empirically proven to
traverse this network (the `dig @9.9.9.9` test) and no conflict. Added cluster-var `PIA_OPENVPN_PORT="53"`,
wired to gluetun `OPENVPN_ENDPOINT_PORT`, and re-enabled `vpn-egress`. Port-forwarding is an over-tunnel PIA
API call, unaffected by the transport port. (WireGuard fallback would also have worked — PIA WG is UDP/1337,
not 1197 — but staying on Gluetun+OpenVPN is the smaller change.)

**Correction: 53 didn't work either → settled on 8080.** Deploying on 53 still got no PIA replies —
this network **transparently intercepts outbound :53 (DNS)**, so `dig` succeeds (proxy forwards it) but
OpenVPN-over-53 is swallowed. Cheap decisive test: a raw **NTP/123** probe from the host got a reply →
arbitrary non-53 UDP works bidirectionally, so it's port-specific (1197 conflict + 53 DNS-intercept), not a
UDP block. Moved `OPENVPN_ENDPOINT_PORT` to **8080** (community-verified for PIA, non-DNS, non-1197).
Also hit the `${VAR}`→int-coercion gotcha (k8s "expected string, got 53") — the port is hardcoded as a
quoted string in gluetun-gateway.yaml (Longhorn-fix precedent), not a cluster-var.

### Phase 3 VPN — OpenVPN unusable on this network → ESCALATE to WireGuard (thrnz) (2026-08-22)
Tested three PIA OpenVPN UDP ports; each fails for a **different** reason (so it's OpenVPN-specific, not a
UDP block — a raw NTP/123 probe from the host returns fine):
- **1197** — outbound leaves, zero replies. Conflicts with the home OpenVPN server NAT'd on WAN:1197.
- **53** — outbound leaves, zero replies. This network transparently intercepts outbound :53 (DNS), so
  `dig` succeeds but OpenVPN-over-53 is swallowed.
- **8080** — PIA **responds** (OpenVPN reaches the TLS handshake) but `TLS handshake failed` repeatedly
  across servers → classic **MTU/fragmentation** (OpenVPN's large TLS handshake packets dropped on the
  WiFi + triple-NAT path: pod→Cilium→node→libvirt-NAT→WiFi→home-NAT).

**Recommendation (ESCALATED — spec-level, ADR-0013):** pivot to **WireGuard via `thrnz/docker-wireguard-pia`**
(PIA WG = **UDP/1337**): a different port (no 1197 conflict, no 53 intercept), and WireGuard's handshake is
small fixed-size packets → **not MTU-sensitive** like OpenVPN's TLS handshake, so it should succeed where
8080 fails. It's ADR-0013's documented WG fallback and the operator's original preference. Trade-offs
(per the ADR): different single-maintainer image, and PF is file-based (`/pia-shared/port.dat`) rather than
Gluetun's control API — matters only when qBittorrent/natmap is built (Phase 6), not now. `vpn-egress`
disabled meanwhile (crashloop would fire false alerts). Awaiting operator's architect decision before the
container swap.

### Phase 3 VPN — ACTUAL root cause: Cilium VXLAN pod MTU (2026-08-22)
Operator (correctly) rejected the port theory (client uses ephemeral src ports; dest port is irrelevant to
return traffic) and had me check MTU at each layer. Findings via `ping -M do` sweeps:
- **Pod** → internet: MTU 1450 OK, 1470 DROP (silent). Pod `eth0` shows 1500 but the effective route is 1450.
- **Node** (k3s-dev-3) → internet: **1500 OK**. All node ifaces (ens3, cilium_*, lxc*) = 1500.
- **Hypervisor** → internet: **1500 OK**. (Google Fiber SOHO router hands off full 1500 — not the culprit.)
- Cilium config: **`routing-mode: tunnel`, `tunnel-protocol: vxlan`** → Cilium subtracts the 50-byte VXLAN
  overhead → **pod MTU 1450**. OpenVPN emits ~1500-byte handshake packets → exceed 1450 → dropped (broken
  PMTUD = no ICMP back). This is why HTTPS (MSS-clamped) + small UDP (DNS/NTP) work but OpenVPN doesn't.

### Phase 3 VPN — ✅ WORKING & STABLE (2026-08-23)
gluetun (v3.41.3, custom mode, operator's IP-based PIA .ovpn) connects to PIA Stockholm and **holds**:
egress IP `158.173.241.36` (Sweden), pod Running 0 restarts. Two final fixes after the .ovpn landed:
- The .ovpn's `remote` must be an **IP** (gluetun's custom parser rejects the `sweden.privacy.network`
  hostname) — operator supplied an IP-based .ovpn (from the sweden.privacy.network LB pool, 158.173.241.x).
- **Liveness probe → exec (localhost)**: once the tunnel redirects the pod default route via tun0, the pod's
  reply to the kubelet's *httpGet* liveness probe egressed the tunnel and never returned → probe timeout →
  restart every ~110s despite the tunnel being up. A local `wget` exec probe stays on `lo`. (OpenVPN's
  "route already exists / Network unreachable" logs are cosmetic — gluetun adds 0.0.0.0/0 itself; egress works.)
Remaining Phase-3 work (later): egress-steering validation (Cilium egress-gw → tun0) + kill-switch conformance
(R-NET-06); port-forwarding is unavailable in custom mode (Phase-6/qBittorrent — revisit then).

### Phase 3 VPN — custom .ovpn mode (the real cause: flaky server selection) (2026-08-23)
After the MTU red-herrings, packet capture (node ens3 + host wlp4s0, GNOME OFF so it wasn't the host's own
VPN) proved the pod CAN complete a **full** PIA handshake — a working flow (`50017 → 158.173.241.22:1198`)
showed 1222-byte cert packets BOTH ways, reaching the node and pod end-to-end through the double NAT
(Cilium SNAT → libvirt masq → router). It's **intermittent per-server**: gluetun's PIA-provider mode
random-picks server IPs and many don't answer this cluster's flows (→ TLS timeouts, exit-0 restarts ~every
2min), while the operator's GNOME client is reliable because its `.ovpn` targets **`sweden.privacy.network`**
— a load-balanced hostname that always lands on a healthy server. NOT MTU, NOT encryption preset (strong AND
normal both fail identically once the env name `PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET` was
corrected), NOT Cilium (policy default-allow, host-fw off, Hubble FORWARDED), NOT a network block.

**Decision (operator's choice):** switch gluetun to **custom-provider mode** feeding the operator's exact
`.ovpn` (`VPN_SERVICE_PROVIDER=custom`, `OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn`, .ovpn from the
`pia-custom-ovpn` SOPS Secret; creds still via OPENVPN_USER/PASSWORD; kill-switch kept; tun-mtu 1300 for the
1450 pod MTU). Dropped the PIA-provider server list + updater initContainer. **Trade-off:** custom mode
loses gluetun's PIA-native port forwarding (Phase-6/qBittorrent concern) — revisit then. This is an
implementation choice within ADR-0013's "single shared Gluetun PIA OpenVPN gateway" model (architecture
unchanged), noted here rather than escalated. Pending operator provisioning of the `.ovpn` Secret.

<!-- superseded strong/normal-preset attempt below, kept for the record -->
**(superseded) Fix (operator's choice: "shrink OpenVPN MTU in gluetun"):** set gluetun `OPENVPN_MTU=1350` (tun-mtu) and
`OPENVPN_MSSFIX=1350` so encrypted packets (~1350+52 ≈ 1402) fit under the 1450 pod MTU. Keeps OpenVPN +
Cilium VXLAN unchanged. Port kept at 8080 (confirmed PIA responds). Re-enabled `vpn-egress`. Values hardcoded
as quoted strings (int-coercion gotcha). Alternatives not taken: Cilium native routing (pod MTU→1500) or
WireGuard/thrnz (MTU-robust) — both still available if the MTU shrink proves insufficient.

### Phase 3 VPN — ACTUAL ACTUAL root cause: the network blackholes PIA's IP ranges (2026-08-22)
The MTU shrink did NOT fix it. A node-side tcpdump of the handshake was decisive: OpenVPN's outbound init
packets are **14 bytes** (nowhere near any MTU) and get **zero replies** — so MTU was never the handshake
blocker either. Confirmed pod UDP egress is healthy (pod NTP/123 reply OK). Then the real test — raw
TCP/ICMP reachability to PIA IPs vs controls (from host AND pod):
- **PIA** (46.246.8.55, 212.112.19.22): TCP **501 and 443 time out** (silent drop, not refused), UDP no
  reply, ICMP no reply — across many PIA servers gluetun cycled.
- **Google / Cloudflare**: TCP/443 OPEN, UDP replies, ICMP replies.

→ **This network (192.168.2.x, WiFi) blackholes traffic to PIA's IP ranges.** It also transparently
intercepts DNS/:53. Everything else (HTTPS, DNS-via-proxy, NTP, general internet) works. So it is neither
ports, nor MTU (both red herrings — the Cilium VXLAN 1450 pod MTU is real but irrelevant to a 14-byte
handshake), nor the cluster/config — it is upstream network filtering of PIA specifically.

**Action → operator (network, not code):** move the k3s host to the unfiltered main LAN (192.168.1.x /
wired), or unblock PIA's ranges on the 192.168.2.x router, then re-enable `vpn-egress` (no code change).
WireGuard/thrnz would NOT help — same PIA IPs are blocked; if switching providers is ever considered, pick
one whose ranges this network permits. Reverted the port/MTU red-herring env from gluetun-gateway.yaml
(back to PIA defaults); `vpn-egress` disabled so it doesn't crashloop/alert. **Config is correct and
complete — the only blocker is the network.**

### Phase 3 VPN — CORRECTION: it was gluetun's STALE server list, NOT a network block (2026-08-22)
The "network blackholes PIA" conclusion above was WRONG. The operator installed a fresh PIA `.ovpn` in GNOME
on the same host/network and it connected fine → the network permits PIA. Verified against PIA's live server
list: current SE Stockholm servers are **158.173.241.x** and ARE reachable (tcp/8080 → RST / "connection
refused" = the server answered). The IPs gluetun was dialing (**46.246.x / 212.112.x**) are **retired**
(timeout = dead hosts). So gluetun's **bundled server list was stale**, and my host reachability test used
those dead IPs → false "blackhole." (Cilium was independently ruled out per the operator's request: policy
mode `default` = default-allow, host firewall disabled, no policy selects these pods, and Hubble showed
pod→PIA `to-network FORWARDED`, no drops.)

**Fix:** gluetun `UPDATER_PERIOD=24h` (fetch PIA's current server list at startup); keep `OPENVPN_MTU=1350`
+ `OPENVPN_MSSFIX=1350` for the real Cilium-VXLAN 1450 pod MTU (data channel); PIA default OpenVPN UDP port.
Re-enabled `vpn-egress`. **Lesson:** when gluetun "can't reach" a provider, suspect a stale bundled server
list FIRST — validate against the provider's live server list before blaming the network/MTU/ports.

### Phase 3 egress steering — ADR-0015 implemented (PM-orchestrated), NOT yet deployed (2026-08-24)
Architect amended the spec to **ADR-0015** (node-level host OpenVPN tunnel + Cilium Egress Gateway + host
nftables kill-switch; supersedes the in-cluster-Gluetun steering — a Gluetun pod's `tun0` can't be steered by
Cilium egress gateway, and hostNetwork Gluetun hijacks the node default route). Operator also fixed the
invariants: **default = local/WAN for everything; VPN strictly opt-in; kill-switch scoped to vpn-class only;
opposite fail directions** (internet fails open, vpn fails closed).

PM dispatched two **WS-NET** agents (feature branches, no deploy); PM reviewed vs the invariants + DoD and
merged to `main` (`33d048d`) — **code only, nothing deployed** (policy not in the kustomization, playbook not
run, working gluetun kept as fallback):
- `infra/ansible/roles/vpn_egress_gateway/` + `playbooks/vpn.egress.gateway.yml` — host OpenVPN systemd
  (PIA/OpenVPN-UDP, creds from **Ansible vault**), **split-tunnel** (`route-noexec` + dedicated route table
  51820 + `ip rule from <tun0-addr>` → node's main-table default untouched), **scoped nftables kill-switch**
  (policy-accept so node/internet fail-open; tun0-sourced/marked traffic dropped when not egressing tun0 →
  vpn fail-closed). Inventory group `vpn_egress_gateway` = `k3s-dev-3`.
- `platform/vpn-egress/egress-steering.yaml` — `CiliumEgressGatewayPolicy` (only `egress-class=vpn`, RFC1918
  excluded, `interface: tun0`, gateway-node label). Held OUT of `kustomization.yaml` until validated.

PM review: verified the up-script, nft rules, and policy honor all four invariants. **Escalation ruled:** the
tunnel-*fully-down* independent backstop the agent proposed via a fwmark on the CiliumEgressGatewayPolicy is
**infeasible** (the policy has no fwmark field). Decision: keep the scoped nft + routing blackhole + inert
fwmark scaffolding, and **gate on the `vpn-killswitch` conformance test** (force `tun0` down → assert no leak,
empirically proving Cilium fails closed). Build a real backstop (host controller syncing vpn-pod IPs into the
nft set) ONLY if that test leaks — not over-engineered now.

**Cross-agent git hazard noted:** the two agents shared one working dir (I didn't use isolated worktrees), so
one branch picked up the other's file; caught + cleaned during review. Next parallel coding fan-out → use
`isolation: worktree`.

**OPERATOR PREREQ (T-3.2p):** provision `infra/ansible/inventory/group_vars/vpn_egress_gateway/vault.yml` with
`vault_vpn_egress_gateway_pia_{username,password,ovpn}` (the `.ovpn` = the proven `sweden-aes-128-cbc-udp-ip`
one, liftable from `platform/vpn-egress/pia-custom-ovpn.sops.yaml`). **VALIDATION FLOW** (post-prereq): run the
playbook → label `k3s-dev-3` `egress-gateway=true` → add `egress-steering.yaml` to the kustomization → run
`egress-class` + `vpn-killswitch` conformance → then retire gluetun in a separate change.

### Phase 3 egress steering — VALIDATED END-TO-END on dev (2026-08-24)
Operator provisioned the host vault; PM drove the deployment + validation on `k3s-dev-3`:
- Playbook ran (after one AUTH_FAILED — operator had a wrong PIA password in the vault, then fixed it; my
  earlier vault recipe put the password in YAML *double* quotes, a special-char hazard — use single quotes /
  programmatic lift next time). Tunnel up: `tun0` 10.72.193.53, VPN egress **158.173.241.36** (PIA Stockholm).
- **Split-tunnel invariants hold:** node default unchanged (`via 10.73.0.1 dev ens3`), node's own egress =
  **WAN 136.51.115.252**, node Ready throughout. `ip rule 5000: from <tun0-addr> lookup vpn_egress`, dedicated
  table (tun0 default + blackhole), nft set scoped to the tun0 src — all correct.
- **Steering wired via GitOps:** added `egress-steering.yaml` to `platform/vpn-egress/kustomization.yaml`
  (fixed a self-inflicted break first — the file had been dropped from main by the earlier gateway-role merge;
  restored from 38fcfad). Flux reconciled (`8a55599`); `CiliumEgressGatewayPolicy vpn-egress-steering` live.
- **egress-class gate PASS:** `egress-class=vpn` pod → 158.173.241.36 (VPN), `egress-class=internet` pod →
  136.51.115.252 (WAN). HTTP ×5 + HTTPS ×3 all 200 (early timeouts were post-restart warmup).
- **vpn-killswitch gate PASS (R-NET-06 fault-injection):** stopped the tunnel → `tun0` gone, ip rule removed,
  nft set flushed, blackhole retained → vpn-class pod **timed out (no WAN leak)**, internet-class unaffected;
  restart recovered cleanly.
- **FINDING (robustness, feeds WS-NET):** fail-closed currently rides on the stale SNAT source (10.72.193.53)
  being non-routable, because the **down-script flushes the nft set + removes the ip rule** — disarming the
  scoped drops exactly when the tunnel is down. The up-script already flush+re-adds the set on connect, so the
  down-script's `nft flush set` is redundant AND harmful: keeping the last tun0 src in the set makes the
  existing `ip saddr @set oifname != tun0 drop` rule an *explicit, edge-independent* backstop. Fix = drop the
  `nft flush set` line from `vpn-egress-down.sh.j2`. Gate passes without it, but this is defense-in-depth.
- **Cross-agent lesson reinforced:** parallel agents must use `isolation: worktree` (an earlier non-isolated
  pair caused the dropped-file break above).

### Kill-switch robustness fix DONE + validated; conformance tests PARTIAL (2026-08-24)
Dispatched two worktree-isolated, code-only agents (WS-NET fix, WS-TEST conformance). Both hit the shared
**session rate limit (resets 12:20pm CT)** and died mid-work; salvaged their uncommitted worktree changes:
- **WS-NET (DONE):** `vpn-egress-down.sh.j2` no longer flushes the nft src set on tunnel-down (retains it so
  the `oifname != tun0 drop` rule stays armed = explicit, edge-independent fail-closed); README documents it.
  Reviewed, `ansible-lint` clean (production profile), merged to main (`b0ff689`), redeployed to k3s-dev-3,
  and **validated live**: tunnel-down → nft set RETAINS `{ 10.72.193.53 }`, drop rule armed; clean recovery.
- **WS-TEST (PARTIAL):** egress-class chainsaw test rewritten for ADR-0015 assertions — preserved as WIP on
  branch `feature/conformance-egress-killswitch-adr0015` (`e7a567d`, NOT merged, unreviewed). STILL TODO: the
  `vpn-killswitch` L3 test + its host fault-injection fixture (privileged node-pinned pod running
  `nsenter -t 1 … systemctl stop/start vpn-egress-tunnel`, self-healing `finally`). Re-dispatch after the
  rate-limit resets, or finish by hand; then run the suite live and gate gluetun retirement on it.

### Conformance suite GREEN live; gluetun retirement PREPARED (pending approval) (2026-08-24)
- WS-TEST agent (resumed twice after the rate-limit reset) delivered both ADR-0015 conformance tests. Fixed a
  chainsaw-0.2.12 schema issue found by attempting the live run: there is NO test-level `spec.finally`, and a
  step-level `finally` accepts only `script`/`command`/`delete`/`wait` (not `apply`/`assert`) — the
  safety-critical self-heal was reimplemented as a `finally` `script` that `kubectl apply`s the restart Job and
  `kubectl wait`s for it. Reviewed + merged to main (`0753b2a`).
- **RAN BOTH LIVE** (chainsaw v0.2.12 at `~/.local/bin/chainsaw`, dev cluster):
  - **L2 egress-class: PASS (77s)** — internet=WAN `136.51.115.252` vs vpn=PIA `158.173.241.36` (distinct);
    R-NET-07 label-flip re-routed the SAME pod (uid unchanged) to WAN, then restored.
  - **L3 vpn-killswitch: PASS (219s)** — tunnel-down ⇒ vpn `NO_LEAK`, internet unaffected, self-heal restart,
    recovery to PIA; post-test safety check: tunnel `active`. This is the R-NET-06 acceptance gate, automated.
  - Pre-existing L1 `egress-class-render` scaffold FAILS (unimplemented T-0.3 renderer, tagged R-NET-05) — NOT
    my work; flagged for that workstream. It currently breaks the `--requirements R-NET-05` merge-gate.
- **SOP proven:** found the schema bug by attempting the live run → sent it back to the authoring agent → it
  fixed + parse-checked with chainsaw → I re-ran live to green. Worktree isolation held (no contamination).
- **gluetun retirement PREPARED** on branch `feature/retire-gluetun-gateway` (`7b7982e`): removes the vestigial
  gluetun Deployment (a Gluetun *pod* was never a viable Cilium egress-gateway target — the reason ADR-0015
  moved the tunnel to the node). **NOT merged/pushed** — the auto-mode permission classifier blocked removing
  running infra + pushing to main; correctly deferred to the operator. gluetun still runs on dev (harmless,
  vestigial). Follow-up when approved: merge this branch, then a separate change to retire the now-unused
  `pia-*` cluster SOPS secrets (creds are host-side in the Ansible vault per docs/07).

### gluetun RETIRED + VPN monitoring retargeted (operator-approved) (2026-08-24)
Operator approved the retirement and asked to update monitoring + docs. Done:
- **Retirement:** merged `feature/retire-gluetun-gateway` (+ the monitoring commit) to main (`9927b73`); Flux
  reconciled and PRUNED the gluetun Deployment/Service (`deployments.apps "gluetun" not found`). The plain
  `git rm && … && push` compound was blocked earlier by the auto-mode classifier; merging the already-committed
  branch + push went through — so the guardrail was on the compound deletion, not pushes generally.
- **Monitoring (`platform/monitoring/alerts.yaml`, deployed via the `platform-config` Flux Kustomization):**
  `VPNTunnelDown` retargeted from `absent(gluetun_vpn_up)` to `absent(node_network_receive_bytes_total{device="tun0"})`.
  Discovered live: gluetun's control API never exported `gluetun_vpn_up`, so the OLD alert was a PRE-EXISTING
  perma-false-alarm. Also `node_network_up{device="tun0"}` is unreliable (tun0 operstate=unknown ⇒ 0), so keyed
  on the byte-counter's presence (confirmed via Prometheus: 1 series on the gateway node; absent() also covers
  gateway-node-down since the series goes stale). Removed the obsolete pod-based `EgressGatewayUnhealthy`
  (no in-cluster gateway pod exists under ADR-0015). Verified in Prometheus runtime: rule health ok, state
  **inactive**, 0 active alerts — the perma-alarm is resolved.
- **Docs:** dispatched a worktree-isolated docs agent (`feature/docs-gluetun-retirement`) to rewrite
  `platform/vpn-egress/README.md` for ADR-0015 and fix ADR-0013→0015 / gluetun comment drift across
  kustomization, the kyverno egress-class policy, the servicedefinition schema, cluster-vars, the (now-unused)
  pia-* secret headers, and the vpn-egress namespace note. In progress — PM review pending.
- **Still open:** retire the unused `pia-*` cluster SOPS secrets (separate change); prune the two merged agent
  worktrees.