691 lines
57 KiB
Markdown
691 lines
57 KiB
Markdown
# Autonomous Session Decision Log (PM)
|
||
|
||
Program Manager working independently while the operator is AFK (started 2026-08-16). Every
|
||
best-effort decision and assumption is recorded here for later review. **Specs were NOT modified**
|
||
(PM cannot change `../home-network-k3s`; that requires the System Architect hat). Blocked items that
|
||
need the operator are called out.
|
||
|
||
## Guardrails I'm holding to
|
||
- No spec changes. No fabricated secrets (Cloudflare token, PIA creds, SMTP — all await hand-off).
|
||
- Don't break the running cluster: `platform/` stays **suspended** in Flux until each component is
|
||
filled in + validated; `platform/cilium` must carry `k8sServiceHost=10.73.0.11` before it ever
|
||
reconciles.
|
||
- Everything author-first, `make -C ci gate` green before merge, feature-branch→main.
|
||
|
||
## Decisions / assumptions (append-only)
|
||
| # | Decision / assumption | Why | Review flag |
|
||
|---|---|---|---|
|
||
| A1 | Complete Phase 1 T-1.4 via **Option A** (Flux bootstrapped, `platform`+`apps` suspended) | You picked Option A earlier as the safe path; "keep going" authorizes it | Confirm scope |
|
||
| A2 | Advance Phase 2 only for components needing **no operator secret**: Longhorn, monitoring, Kyverno-admission, Cilium-parity | cert-manager/externaldns/velero/VPN are blocked on token/PIA/OQ-BKP | — |
|
||
| A3 | Dev resource caps for monitoring (short retention, small PVCs, single replicas) from `cluster-vars` | 22 GiB RAM ceiling | Tune later |
|
||
| A4 | Host Gitea repo is **public** (NAT-only) so Flux needs no git creds | dev bootstrap simplicity | Fine for dev |
|
||
|
||
## Autonomous session results (2026-08-16)
|
||
- **Phase-2 authoring (author-only, merged):** `platform/longhorn` (3-replica SC + bulk indirection),
|
||
`platform/monitoring` (kube-prometheus-stack + Loki + Alloy, dev-trimmed ~1.3 GiB, +8 alert rules),
|
||
`platform/cilium` (parity fix: k8sServiceHost=10.73.0.11 + masquerade, safe to adopt in Phase 2),
|
||
`platform/kyverno` (verified). Gate green.
|
||
- **Flux INSTALLED** on the cluster (4 controllers Running). Host **Gitea working** (named-volume fix),
|
||
repo pushed. GitRepository created → **BLOCKED**: cluster pods can't reach `10.73.0.1:3000`
|
||
(firewalld `libvirt` zone drops VM→host:3000).
|
||
- **RESOLVED (operator added the ansible-vault become password):** ran `host-prereqs.yml` (firewall
|
||
opened via vault→become), VM→Gitea = 200, and **Flux GitRepository is READY=True**
|
||
(`main@sha1:a2f3bdc…`). **Phase-1 T-1.4 core is DONE** — Flux installed + reconciling clusters/dev
|
||
from the host Gitea. `platform`/`apps` Kustomizations intentionally NOT created (Option A) — the
|
||
running cluster is untouched.
|
||
- Firewall + Gitea admin/repo/push + host become password are all IaC now (host-prereqs.yml,
|
||
bootstrap-gitea.sh automation, vaulted group_vars/hypervisor) → destroy/rebuild is hands-off.
|
||
- **NOT deployed:** `platform`/`apps` Kustomizations were deliberately NOT created — the running
|
||
cluster (Cilium etc.) is untouched. Only Flux controllers + the GitRepository source exist.
|
||
|
||
## T-1.5 — Destroy/rebuild proof (destructive IaC-validity test) — 2026-08-16
|
||
|
||
**Goal (operator-scoped):** prove the IaC is *valid* — the dev cluster can be destroyed and rebuilt
|
||
from IaC + git with no data loss ("Approach A is acceptable"; "more about validity of the IaC than a
|
||
COOP/DR scenario"). Ran **after K2 conformance passed**, with the operator approving each destructive
|
||
step at the prompt.
|
||
|
||
**Outcome: PASS (with one operator-visible scare and two concrete IaC findings).** The cluster was
|
||
torn down and reconstituted from IaC + preserved Longhorn/OS disks + the Flux git source. **End state:
|
||
3/3 Ready (k3s-dev-1/2/3, control-plane,etcd,master), Cilium status OK, Flux GitRepository READY=True,
|
||
zero state lost.** Behavioral health re-verified post-rebuild: conformance L2 **R-NET-01 default-deny +
|
||
R-NET-02 deny-undeclared both PASS** (exit 0, JUnit+coverage regenerated). The destroy/rebuild path is
|
||
demonstrably valid.
|
||
|
||
### Finding F1 — `tofu destroy -target` on a `count`-indexed resource is unsafe (ROOT CAUSE)
|
||
`tofu destroy -target 'libvirt_domain.node[2]'` (+ that node's `os[2]`/`data[2]`/`seed[2]`) **destroyed
|
||
all three domains**, not just node3 — while the individually-targeted *volumes* correctly hit only
|
||
index 2. Root cause: the nodes are declared with **`count`** in `infra/tofu/main.tf`; `-target` against a
|
||
single `count` index is a documented Terraform/OpenTofu hazard (the docs explicitly warn `-target` is
|
||
"not for routine use"). count-index addresses are positional and brittle under targeted destroys.
|
||
- **Recommended fix (Phase-2 IaC hardening):** refactor `libvirt_domain.node` + its per-node volumes
|
||
from `count` to **`for_each`** keyed by node name (`k3s-dev-1/2/3`). `for_each` keys are stable and
|
||
individually addressable, so a single node can be `-replace`d/destroyed without touching siblings.
|
||
- **Interim rule:** **never** use `tofu destroy -target` on this module for single-node lifecycle. To
|
||
cycle one node, use `tofu apply -replace='libvirt_domain.node["k3s-dev-N"]'` (post-`for_each`), or
|
||
destroy the specific VM via `virsh` and let `tofu apply` recreate it.
|
||
- **Silver lining:** the over-destroy is exactly what *proved* IaC validity — disks survived, `tofu
|
||
apply` recreated the domains, nodes rejoined, Flux reconciled unchanged. The IaC round-trips.
|
||
|
||
### Finding F2 — single-node Ansible rejoin needs the primary in scope for the join-token
|
||
Recovering node3 with `ansible-playbook ... --limit k3s-dev-3` **failed**: the `k3s_server` join task
|
||
reads `hostvars[groups['k3s_primary'][0]].k3s_node_token`, but that fact is only *published* when the
|
||
primary play runs, so a limit that excludes the primary yields
|
||
`'HostVarsVars' has no attribute 'k3s_node_token'`. Worked with `--limit 'k3s-dev-1,k3s-dev-3'`.
|
||
- **Recommended fix (Phase-2 IaC hardening):** make the join self-contained — have the secondary task
|
||
fetch the token on demand via `delegate_to: "{{ groups['k3s_primary'][0] }}"` (read
|
||
`/var/lib/rancher/k3s/server/node-token` at join time) instead of depending on a fact the primary
|
||
play must have populated in the same run. Then a true single-node rejoin (`--limit k3s-dev-N`) works
|
||
hands-off, which is the T-1.5 durability bar.
|
||
|
||
### F1 / F2 — IMPLEMENTED (author-first, 2026-08-16)
|
||
- **F1 done:** `infra/tofu/main.tf` node resources refactored `count` → `for_each` (keyed by node
|
||
name); ordered-list/`primary` helper locals added; `outputs.tf` updated off the map; **`moved.tf`**
|
||
adds 12 `moved` blocks (os/data/seed/domain × 3). `tofu validate` OK and **`tofu plan` = `0 to add,
|
||
0 to change, 0 to destroy`** with all 12 resources reported "has moved to" — a pure state-address
|
||
rename, no VM churn. **State-persist pending:** a follow-up `tofu apply` (a proven 0-change apply
|
||
that only rewrites state addresses) is operator-permission-gated; until then the migration re-shows
|
||
harmlessly on each plan and lands automatically on the next real apply.
|
||
- **F2 done:** `roles/k3s_server/tasks/main.yml` — secondary now fetches the join token via
|
||
`delegate_to: primary` (with the published fact as a fast path), so `--limit k3s-dev-N` rejoin is
|
||
self-contained. `ansible-playbook --syntax-check` OK on all playbooks; role lint clean under the CI
|
||
gate (`make -C ci gate` PASS). No cluster change; the running cluster is untouched.
|
||
|
||
## Phase 2 un-suspend PREP — no-secret rollout (author-first, 2026-08-16)
|
||
|
||
Operator directive: "Phase 2 un-suspend prep … verify cilium parity + fill REPLACE-MEs for the
|
||
no-secret components … author-first. Does NOT un-suspend/deploy yet." **Nothing was deployed;** the
|
||
running cluster is untouched (still 0 Flux Kustomizations / 0 HelmReleases — only controllers + the
|
||
GitRepository).
|
||
|
||
### Findings from three parallel readiness audits (Longhorn / monitoring / Kyverno)
|
||
- **Cilium parity VERIFIED (safety-critical):** every `[parity]` field in `platform/cilium/helmrelease.yaml`
|
||
equals the live CNI — `k8sServiceHost=10.73.0.11` (rendered by the chart as the pod env
|
||
`KUBERNETES_SERVICE_HOST`, **not** a configmap key — the empty `k8s-service-host` cm value was a red
|
||
herring), kube-proxy-replacement, gateway-api, egress+bpf masquerade, hubble, l2-announcements,
|
||
cgroup, v1.16.5. **Flux adoption is a no-op reconcile.**
|
||
- **Topology:** "suspended" = Option A (**no root Kustomization was ever created**), and the single base
|
||
`platform` Kustomization renders **all 11 components at once** — so a naive un-suspend would drag in
|
||
the secret-blocked components (cert-manager/externaldns/velero/keycloak/oauth2-proxy/image-automation)
|
||
and never go healthy (`apps` dependsOn platform, wait:true).
|
||
- **Longhorn:** manifests clean; node prereqs (open-iscsi, /var/lib/longhorn mount, iscsid) are in
|
||
cloud-init IaC; the REPLACE-ME bulk-NFS SC is isolated/harmless. Ready.
|
||
- **Monitoring:** right-sized (~1.25 GiB) but blocked by (a) needing the `longhorn` SC first and (b) two
|
||
**by-name secrets** (`grafana-admin-credentials`, `alertmanager-smtp`) that don't exist — the missing
|
||
mounts (not the REPLACE-ME text) are what wedge Grafana/Alertmanager.
|
||
- **Kyverno:** zero policy violations today (policies scope to `app-namespace=true`, none exist), but the
|
||
fail-closed webhook + 3 HA replicas + Enforce are unsafe/oversized for a cold cluster.
|
||
|
||
### DECISION D-P2A — per-component Flux Kustomizations for the dev no-secret rollout (operator-approved)
|
||
Chose **Option A (per-component Kustomizations)**. Authored `clusters/dev/rollout/` = 4 Flux
|
||
Kustomizations (`kyverno` → `cilium` → `longhorn` → `monitoring`, monitoring `dependsOn: longhorn`),
|
||
and the dev overlay now `$patch: delete`s the monolithic base `platform`/`apps` and adds `./rollout`.
|
||
The no-secret tier carries **no SOPS content**, so the missing `sops-age` key does not block it
|
||
(decryption dropped from these Kustomizations). Secret-blocked components are simply **not created**
|
||
until their hand-offs land.
|
||
- **cilium** uses `prune: false` (adopt the out-of-band CNI; never own-delete it).
|
||
- **R-ENV-02 divergence — REVIEW FLAG (architect):** the dev overlay now *adds resources* and diverges
|
||
its reconciliation graph from prod (base doc says "dev adds NO resources … graph reused verbatim").
|
||
This is a **transitional bootstrap scaffold**, consistent with the already-inlined dev bootstrap
|
||
patches (SOPS/URL). **Convergence path is coded in:** delete `clusters/dev/rollout/` + the two
|
||
`$patch: delete`s to re-adopt the unified base graph once all hand-offs arrive. **prod is untouched**
|
||
(still builds the monolith; verified). Flag for architect ratification.
|
||
|
||
### Component hardening (env-gated via cluster-vars so prod keeps the strict posture)
|
||
- New base knobs (strict/prod defaults) + dev overrides: `KYVERNO_VALIDATION_ACTION` (Enforce→**Audit**),
|
||
`KYVERNO_WEBHOOK_FAILURE_POLICY` (Fail→**Ignore**), `KYVERNO_ADMISSION_REPLICAS` (3→**1**),
|
||
`MONITORING_STORAGE_CLASS` (longhorn), `GRAFANA_ADMIN_SECRET` (grafana-admin-credentials→**""** in dev
|
||
→ chart-default password, dev-only/NAT-isolated).
|
||
- Kyverno webhook also gains an explicit `namespaceSelector` excluding kube-system/flux-system/kyverno/
|
||
longhorn-system/monitoring (belt-and-suspenders vs a fail-closed brick).
|
||
- Monitoring Alertmanager: the `alertmanager-smtp` mount + smtp_*/email_configs are **deferred** (commented
|
||
with verbatim re-enable markers) so Alertmanager starts with NULL receivers — alerts fire/route/show in
|
||
UI+Grafana, just no email until the Phase-4 SMTP hand-off.
|
||
- Validated: `kubectl kustomize` clean for clusters/dev (exactly the 4 Kustomizations, monolith removed),
|
||
clusters/prod (monolith intact), and all 4 components; envsubst→YAML-parse of kyverno+monitoring valid;
|
||
`make -C ci gate` PASS.
|
||
|
||
### Still required BEFORE an actual deploy (not part of this author-first prep)
|
||
1. **Push current `main` to the host Gitea** — Flux source is pinned at `6c3f9ed` (pre-dates these merges).
|
||
2. **Runtime node check:** `systemctl is-active iscsid` + `mountpoint /var/lib/longhorn` on all 3 nodes.
|
||
3. **Cilium adoption mechanism:** confirm helm-controller can adopt the Ansible-installed release (was it
|
||
`helm`-installed with a release secret, or CLI/kubectl?) — verify before creating the cilium Kustomization.
|
||
4. Create the dev root Kustomization (the actual un-suspend) — deliberately NOT done here.
|
||
|
||
## CORRECTION — credential-forward, unified graph (operator directive, 2026-08-16)
|
||
|
||
**Operator pushback:** "Are we making implementation choices solely on the fact that you have to ask
|
||
for credentials? It is better to get the credentials upfront (encrypted file) and keep the rollout in
|
||
line with the existing architecture." **Correct — and it exposed a scope distortion.**
|
||
|
||
- **A2 SUPERSEDED / D-P2A REVERTED.** Choosing components by "needs no operator secret" inverted the
|
||
roadmap: `roadmap/phases.md` **Phase 2 = Network, Storage, TLS, DNS, Ingress** → the credential-gated
|
||
cert-manager/externaldns/gateway ARE the Phase-2 core, while **monitoring is Phase 4** (I'd pulled it
|
||
forward). The per-component dev divergence (`clusters/dev/rollout/`) existed only to dodge asking for
|
||
the Cloudflare token.
|
||
- **Reverted to the existing architecture:** deleted `clusters/dev/rollout/` + the `$patch: delete`s;
|
||
restored the unified base graph (dev = ../base + cluster-vars patches + SOPS decryption patches, one
|
||
`platform` Kustomization, R-ENV-02 intact — **removes the earlier architect-review flag**). Reverted
|
||
the monitoring Grafana/SMTP dodges (it's Phase 4). Kept the non-credential-driven improvements: tofu
|
||
for_each/rejoin (F1/F2), and Kyverno parametrization + webhook namespace-excludes (Kyverno IS Phase 2).
|
||
- **DECISION D-P2B — phase the unified `platform/kustomization.yaml` (operator-approved).** Components
|
||
are enabled in the single shared list **per phase** (the same pattern that deferred kyverno):
|
||
Phase-2 set now active = **kyverno, cilium, longhorn, cert-manager, externaldns, gateway**; monitoring/
|
||
image-automation (P4), velero (P4, OQ-BKP), kubevirt (P5), keycloak/oauth2-proxy (P6) are commented
|
||
with phase markers. dev+prod render the same list. Verified: `kubectl kustomize platform` renders only
|
||
the 6 Phase-2 namespaces; dev shows the unified platform+apps Kustomizations; prod builds; gate PASS.
|
||
- **DECISION D-SEC1 — provision ALL credentials upfront via SOPS; operator encrypts (both operator-chosen).**
|
||
Activated SOPS: `.sops.yaml` installed at repo root (recipient still REPLACE-ME — operator fills after
|
||
age-keygen). Authored `bootstrap/sops/secret-templates/*.yaml.example` (cloudflare-api-token ×2 ns,
|
||
grafana-admin, alertmanager-smtp, keycloak-admin, oauth2-proxy, pia — all placeholder-only, `.example`
|
||
so kustomize/SOPS ignore them) + `bootstrap/sops/CREDENTIALS.md` (end-to-end checklist). Pre-wired
|
||
**commented** `- cloudflare-api-token.sops.yaml` lines in cert-manager + externaldns kustomizations
|
||
(operator uncomments post-encrypt). Filled the non-secret ACME email ([email protected],
|
||
confirm/change). **Agents never touch real secret values** (per `.sops.yaml`).
|
||
|
||
### Operator TODO to reach Phase-2 deploy (see bootstrap/sops/CREDENTIALS.md)
|
||
1. age-keygen → fill root `.sops.yaml` recipient → install `sops-age` on cluster.
|
||
2. Provision `cloudflare-api-token` (Zone:DNS:Edit + Zone:Read on keithjackson.dev) in **both**
|
||
cert-manager + external-dns; uncomment the two kustomization lines.
|
||
3. Confirm ACME email. 4. Push `main` to host Gitea. 5. Create the dev root Kustomization (the un-suspend).
|
||
6. (Upfront, optional now) encrypt the Phase-3/4/6 secrets so later phases are credential-ready.
|
||
7. **Chart-key verification** (flagged in templates): keycloak + oauth2-proxy `existingSecret` key names,
|
||
Keycloak DB secret, and PIA/Gluetun protocol (≠ TCP) — verify at each component's phase.
|
||
|
||
## Phase-2 UN-SUSPEND executed + fixes (autonomous session, 2026-08-16 overnight)
|
||
|
||
Operator said "go" for the un-suspend, then went AFK: "continue to work without me. Any major decision
|
||
that requires a system architecture change VIA SPEC CHANGE will need to be escalated. Anything else is
|
||
allowed. Keep track of assumptions and report on return." None of the below is a spec change (the spec
|
||
is silent on Flux Kustomization granularity / L2 wiring), so all within PM remit. **Nothing touched the
|
||
read-only spec repo.**
|
||
|
||
- **Un-suspended** via the sanctioned `kubectl apply -k clusters/dev` (bootstrap.sh step 3): controllers
|
||
already installed, `sops-age` present. Pre-deploy checks all green first — Cilium adoption safe (an
|
||
existing `helm` release `cilium` v3 exists → Flux adopts by name, no-op upgrade), Longhorn node prereqs
|
||
green on all 3 nodes (`iscsid` active, `/var/lib/longhorn`→`/dev/vdb`), host Gitea pushed + Flux source Ready.
|
||
|
||
### Fix 1 — Longhorn StorageClass numeric coercion (commit ~c04aea8)
|
||
Reconcile stalled: `.parameters.numberOfReplicas: expected string, got 3`. kustomize strips the quotes
|
||
around a bare `"${VAR}"` placeholder, so Flux substitution yielded an unquoted numeric that SSA rejects
|
||
(StorageClass.parameters is map[string]string). **Hardcoded `numberOfReplicas: "3"`** (quoted numeric
|
||
literals ARE preserved by kustomize; topology LOCKED at 3). The replica knob still drives the global
|
||
default via `defaultSettings.defaultReplicaCount` in the helmrelease (a Helm int value, unaffected).
|
||
|
||
### Fix 2 — DECISION: controllers/configs split (commit ~458a867) — NOTABLE, please review
|
||
Reconcile then failed atomically: `no matches for kind "ClusterIssuer"`. The single `platform`
|
||
Kustomization applied HelmReleases (which register CRDs) AND CRs of those CRDs (ClusterIssuer,
|
||
Certificate, Kyverno ClusterPolicies) in one pass → Flux SSA fails the WHOLE apply → the CRD-registering
|
||
HelmReleases never run. Deadlock. **Adopted the flux2 controllers/configs pattern:** new
|
||
`platform-config` Flux Kustomization (`dependsOn: platform`, wait) carries the CRs; `platform` keeps only
|
||
controllers; `apps` now `dependsOn platform-config`. Applies to dev AND prod (shared base) — no R-ENV-02
|
||
divergence. `platform-config/` references the CR files IN PLACE via `../platform/...` (Flux builds with
|
||
LoadRestrictionsNone; Kyverno policies must stay under `platform/kyverno/` for the CI L0 `kyverno test`
|
||
gate) — **ASSUMPTION: Flux's default LoadRestrictionsNone permits the `../` refs** (builds verified
|
||
locally with `--load-restrictor LoadRestrictionsNone`; confirming at reconcile).
|
||
|
||
### Fix 3 — Authored the missing Cilium L2 resources (in Fix 2 commit)
|
||
`l2announcements.enabled: true` was set but the IP pool + announcement policy were only a TODO, so a
|
||
Gateway LoadBalancer would never get an IP. Authored `platform-config/cilium/`: **CiliumLoadBalancerIPPool**
|
||
(dev NAT range **10.73.0.240-.250**, matching the tofu reservation) + **CiliumL2AnnouncementPolicy**
|
||
(interface regex `^en`, node interface is `ens3`). **ASSUMPTIONS:** (a) VIP range hardcoded to the dev
|
||
NAT range — a prod overlay must parameterize it (via cluster-vars) when prod is built; (b) `^en` matches
|
||
the node LAN NIC (`ens3`); (c) CRD apiVersion `cilium.io/v2alpha1` (confirmed served on the cluster).
|
||
|
||
### Fix 4 — Kyverno chart webhook schema (commit ~c6331c8)
|
||
Kyverno HelmRelease install FAILED at Helm templating: I'd written `config.webhooks` as a LIST
|
||
`[{failurePolicy, namespaceSelector}]`, but the kyverno 3.3.9 chart expects `config.webhooks` to be a
|
||
MAP (`webhooks.namespaceSelector`), and the fail-open switch is a separate global
|
||
`config.forceFailurePolicyIgnore.enabled` (bool), not a per-webhook field. Fixed both; renamed the
|
||
cluster-var `KYVERNO_WEBHOOK_FAILURE_POLICY`(Fail/Ignore) → `KYVERNO_FORCE_FAILURE_POLICY_IGNORE`
|
||
(false base / true dev). Verified with `helm template` before pushing.
|
||
**PROCESS SLIP:** this fix landed directly on `main` (I forgot to branch first). Content is gate-green;
|
||
noted for the record. Fixes 5+ went back through feature branches.
|
||
|
||
### Fix 5 — cert-manager DNS-01 self-check via public resolvers (commit ~59dd3d9)
|
||
Wildcard cert stalled: challenge `Presented=true` (Cloudflare TXT created — the token WORKS) but the
|
||
propagation self-check looped "DNS record not yet propagated". VERIFIED the record IS publicly
|
||
resolvable (1.1.1.1/8.8.8.8); the NAT'd k3s-dev cluster's own DNS just can't see it. Set
|
||
`dns01RecursiveNameservers: "1.1.1.1:53,8.8.8.8:53"` + `dns01RecursiveNameserversOnly: true` (realises
|
||
a pre-existing TODO in the file). Challenge went pending→valid, cert **Issued**.
|
||
|
||
### Fix 6 — cilium-operator restart to program the Gateway (MANUAL recovery, see note)
|
||
Gateway stuck `Accepted=Unknown:Pending` for ~20min. cilium-operator logs: `no kind is registered for
|
||
the type v1.Gateway in scheme` → `one-shot job errored ... leader-lifecycle` — the operator's
|
||
gateway-controller failed scheme registration during the Flux **adoption** restart and gave up (rest of
|
||
the operator ran fine). `kubectl -n kube-system rollout restart deploy/cilium-operator` fixed it
|
||
instantly: Gateway → Accepted+Programmed, **address 10.73.0.240** from the L2 pool, `cilium-gateway-shared`
|
||
LB service created. **ASSUMPTION/NOTE:** this was a manual (non-IaC) recovery of a transient adoption
|
||
casualty; a fresh ansible bootstrap installs the Gateway API CRDs BEFORE Cilium, so a clean rebuild
|
||
should not hit it — VERIFY on the next destroy/rebuild, and if it recurs, codify an operator-restart
|
||
handler (or bump Cilium). Not codified now.
|
||
|
||
### RESULT — Phase-2 platform FULLY UP + green (2026-08-17)
|
||
`platform`, `platform-config`, `apps` all Ready=True (revision 59dd3d9). 5/5 HelmReleases Ready;
|
||
wildcard `*.home.keithjackson.dev` cert Issued (LE-staging per ADR-0006); Gateway Programmed @
|
||
10.73.0.240; 6 Kyverno policies Ready (Audit in dev); Longhorn default SC live; no not-ready pods.
|
||
|
||
### FOLLOW-UPS (non-blocking, for review)
|
||
1. **Two default StorageClasses** — k3s's built-in `local-path` AND `longhorn` are both marked
|
||
default. Ambiguous. Fix: disable k3s local-storage in the ansible k3s_server args
|
||
(`--disable local-storage`) OR strip the default annotation from local-path so `longhorn` is the
|
||
sole default. (Non-blocking: platform components pin `storageClassName: longhorn` explicitly.)
|
||
2. **Phase-2 EXIT not yet verified** — the roadmap Phase-2 exit is passing conformance (R-NET-01
|
||
default-deny, R-TLS-01/03 cert served, R-DNS-02 auto record). Platform is UP; conformance run is the
|
||
next step (dev cert is LE-STAGING = non-self-signed but not browser-trusted — expected in dev).
|
||
3. **cilium-operator gateway restart** (Fix 6) — verify on next rebuild / codify if it recurs.
|
||
|
||
## Autonomous session 2 — Phase-2 verification + hardening (2026-08-17)
|
||
|
||
Operator away again, same SOP. All safe, no spec changes.
|
||
|
||
- **R-NET conformance GREEN against the DEPLOYED platform:** ran `conformance/run.sh --layer l2
|
||
--requirements R-NET-01,R-NET-02` → **default-deny (R-NET-01) + deny-undeclared (R-NET-02) both PASS**
|
||
(2/2). Confirms the live platform enforces the network isolation baseline, not just the K2 fixtures.
|
||
- **TLS path functionally VERIFIED (not the formal test):** `openssl s_client` to the gateway VIP
|
||
10.73.0.240 with SNI returns `CN=*.home.keithjackson.dev`, SAN {*.home, home}.keithjackson.dev,
|
||
issuer **Let's Encrypt (STAGING)**, 90-day validity. The full DNS-01 → issuance → gateway-termination
|
||
path works. (Dev cert is LE-STAGING per ADR-0006 = non-self-signed but NOT system-trusted.)
|
||
- **Fix: single default StorageClass** (commit ~7a45eb1) — added `local-storage` to the k3s
|
||
`--disable` list so `longhorn` is the sole default. Idempotent install task → effective on rebuild,
|
||
no disruption to the running cluster (which harmlessly keeps both defaults; all platform PVCs pin
|
||
longhorn). ASSUMPTION: disabling k3s local-path entirely is fine (Longhorn is the storage).
|
||
- **Fix: conformance run.sh CWD-robustness** (commit ~ee5ad90) — chainsaw's config `report.path` is
|
||
relative, so run.sh exited non-zero (breaking the L2 gate) when invoked from repo root. Pass absolute
|
||
`--report-path ${ARTIFACTS}` + mkdir. Verified exit 0 + report written from repo root.
|
||
|
||
### REMAINING for Phase-2 EXIT (needs an operator call — did NOT do autonomously)
|
||
The roadmap Phase-2 exit is conformance-green for R-TLS + R-DNS too, but those tests are Phase-0
|
||
**scaffolds** (TODO placeholders) — they must be AUTHORED against the live platform. Two reasons I
|
||
deferred rather than best-effort them:
|
||
1. **R-TLS trusted-cert (trusted-tls test) — DECISION NEEDED:** the test checks the SYSTEM trust store
|
||
(no -k), but dev serves an LE-**staging** cert (not system-trusted). How should dev verify "trusted"?
|
||
(a) verify the cert chains to the real LE **staging** root (embed the staging root in the probe) —
|
||
most faithful; (b) verify LE-issued + valid + SAN + non-self-signed only in dev, full system-trust
|
||
in prod; (c) use LE-prod for a designated test host. **Recommend (a) or (b).** Side-effect-free to
|
||
implement once decided.
|
||
2. **R-DNS auto-record (dns-exposure test) — EXTERNAL SIDE-EFFECT:** verifying R-DNS-02 means letting
|
||
ExternalDNS create/delete a real record in the **live keithjackson.dev Cloudflare zone**. Deferred
|
||
to run with the operator present (or confirm it's fine to exercise autonomously with guaranteed
|
||
cleanup).
|
||
|
||
## Phase-2 CONFORMANCE EXIT ACHIEVED (2026-08-17, main @ 1762206 + cluster-vars re-apply)
|
||
|
||
Operator decided: TLS trust = **A** (chain to LE-staging root); DNS test = **OK** to run live.
|
||
Authored both Phase-0 scaffolds into real tests against the deployed platform; **all 4 Phase-2 L2
|
||
tests PASS**: R-NET-01 (default-deny), R-NET-02 (deny-undeclared), R-TLS-01/03 (trusted-tls),
|
||
R-DNS-02 (dns-exposure). This covers the roadmap Phase-2 exit demonstrables (default-deny holds,
|
||
trusted cert served at *.home.keithjackson.dev, auto DNS record appears).
|
||
|
||
- **trusted-tls (R-TLS-01/03):** in-cluster curl probe dials the gateway Service (env-agnostic via
|
||
`--connect-to`, no hardcoded VIP) with wildcard SNI and requires the served chain to verify against
|
||
the embedded **LE-staging root** ("(STAGING) Pretend Pear X1", fixtures/le-staging-root-ca.yaml).
|
||
Probes a public + an internal-only hostname (R-TLS-03). Empirically validated: correct root → rc 0;
|
||
system trust → rc 60.
|
||
- **dns-exposure (R-DNS-02):** a LoadBalancer Service + external-dns hostname annotation; polls public
|
||
DNS (1.1.1.1) until ExternalDNS publishes the A record into the cluster LB range; chainsaw teardown
|
||
+ `policy=sync` removes it. Verified the full CREATE→DELETE lifecycle in ExternalDNS logs; the live
|
||
keithjackson.dev zone is left clean (back to the pre-existing wildcard).
|
||
|
||
### FINDING F-DNS1 — ExternalDNS domain-filter must be the ZONE, not the subdomain (FIXED)
|
||
The DNS test first failed: ExternalDNS generated the record but skipped it — "no hosted zone matching
|
||
record DNS Name". `--domain-filter=home.keithjackson.dev` filtered OUT the real Cloudflare zone
|
||
`keithjackson.dev` (a subdomain filter drops the parent zone). Added a `DNS_ZONE` cluster-var
|
||
(keithjackson.dev) and set `domainFilters: [${DNS_ZONE}]`. Scope stays effectively home.keithjackson.dev
|
||
(only suffix services request) and `policy=sync`+txt-registry never touch non-owned records.
|
||
|
||
### FINDING F-OPS1 — cluster-vars changes need a manual `kubectl apply -k` — **RESOLVED (main @ 4381a71)**
|
||
Added `clusters/dev/sync.yaml`: a self-managing ROOT Kustomization (flux2 gotk-sync pattern, name
|
||
`flux-system`) that reconciles `./clusters/dev` from git — cluster-vars, the GitRepository, platform/
|
||
platform-config/apps, and itself (inventory = 6 objects, confirmed). Bootstrap unchanged
|
||
(`kubectl apply -k clusters/dev` creates it once). Verified: root → Ready, adopts the graph without
|
||
disruption, all Kustomizations settle green; a git push now auto-applies overlay/cluster-vars changes.
|
||
prod needs its own sync.yaml (→ ./clusters/prod) when built. Original gap description below:
|
||
|
||
|
||
Adding `DNS_ZONE` didn't take effect until I re-ran `kubectl apply -k clusters/dev` (the on-cluster
|
||
`cluster-vars` ConfigMap isn't reconciled by Flux — this bootstrap model has NO self-managing root
|
||
Kustomization that watches `clusters/dev`). Then a `platform` Kustomization reconcile re-ran the
|
||
postBuild substitution. **RECOMMEND: add a self-managing root Flux Kustomization** (path ./clusters/dev)
|
||
so cluster-vars / graph changes auto-apply — otherwise every overlay change is a manual re-apply
|
||
(and, transiently, ${DNS_ZONE} substituted empty → `--domain-filter=` = all-zones, before re-apply).
|
||
|
||
### FINDING F-KY1 — kyverno chart's cleanup image (bitnami/kubectl) purged from Docker Hub (FIXED)
|
||
Surfaced when a `platform` reconcile triggered a kyverno helm upgrade: the chart's cleanup jobs
|
||
(`policyReportsCleanup`, `webhooksCleanup`) default to `bitnami/kubectl:1.30.2`, which Bitnami REMOVED
|
||
from Docker Hub (2024 catalog purge) → ImagePullBackOff. `policyReportsCleanup` runs as a Helm
|
||
**post-upgrade HOOK**, so its failure stalled the upgrade (5m timeout) → kyverno HR not-Ready → platform
|
||
Unknown → platform-config/apps cascade to False. **Fixed:** override both cleanup images to Bitnami's
|
||
relocated `bitnamilegacy/kubectl` mirror (identical image — has /bin/bash + awk/wc/xargs + kubectl the
|
||
jobs need; verified via `helm template` that no `bitnami/kubectl` ref remains). After the fix the hook
|
||
job Completed, kyverno HR → True, and the whole graph returned to green (platform/platform-config/apps
|
||
all Ready, 5/5 HelmReleases Ready, no unhealthy pods). **NOTE:** `bitnamilegacy` is itself a temporary
|
||
mirror — a durable follow-up is to pin a maintained kubectl image (shell-capable, e.g. rancher or
|
||
alpine/k8s) once tested, or a newer kyverno chart that drops the bitnami dependency.
|
||
|
||
### R-STATE-01 GREEN + harness fix F-TEST1 (2026-08-17, main @ 76478f2)
|
||
- **R-STATE-01 (data durability) PROVEN:** the L3 `data-survival` test runs against live Longhorn — writes
|
||
a canary to a `longhorn` PVC, DESTROYS the pod entirely, re-binds a fresh pod to the same PVC, and the
|
||
data survives (`DATA_SURVIVED_OK`). This is the platform's headline stateful-durability requirement.
|
||
- **F-TEST1 — every chainsaw `check` used an unsupported `=~` operator (FIXED):** all 10 tests wrote
|
||
`(($stdout =~ 'X')): true`; chainsaw's expression engine rejects `=~` ('SyntaxError: Unknown char: ~'),
|
||
so each test FAILED on the assertion even when the behavior passed (caught because data-survival printed
|
||
DATA_SURVIVED_OK yet FAILED). Converted all to JMESPath `contains($stdout, 'X')` in BLOCK style (flow
|
||
`{ }` breaks on the comma inside `contains()`). data-survival passes; the 9 later-phase tests
|
||
(egress/vpn/alerting/dr/updates/drift/sso) are now syntactically correct for when their phases deploy.
|
||
- **R-NET-01/02 FAIL-CLOSED (L3) PROVEN (main @ cde3335):** implemented the `default-deny-failclosed`
|
||
scaffold — fixtures = default-deny baseline + source + dest:8080 + the sole explicit allow. Verifies
|
||
source→dest works, then REVOKES the allow (real fault: `kubectl delete networkpolicy`), and asserts
|
||
connectivity fails CLOSED while DNS still resolves. Proves default-deny is datapath-enforced, not just
|
||
declared. PASSES. (DNS check asserts the resolved answer — busybox nslookup exits non-zero on success.)
|
||
**Conformance green now: R-NET-01/02 (L2+L3), R-TLS-01/03, R-DNS-02, R-STATE-01.** Remaining tests need
|
||
later-phase components (VPN P3, monitoring/backup/updates P4, SSO P6) → operator secrets.
|
||
|
||
## Phase 3 + 4 ("Both") + an infra recovery (2026-08-17, main @ 4f17c53)
|
||
|
||
Operator: WireGuard preferred (I confirmed PIA supports it on all regions + Gluetun derives the WG
|
||
config from username/password, so the PIA secret shape is unchanged); keycloak creds added; oauth2-proxy
|
||
deferred to Phase 6; "proceed to next phase." Chose **Both** (deploy Phase-4 monitoring + author Phase-3 VPN).
|
||
|
||
### INFRA RECOVERY — cluster VMs + Gitea were found STOPPED (not my doing)
|
||
Mid-task, kubectl + git push failed: the 3 k3s VMs were **shut off** and the Gitea container **stopped**
|
||
(host up 12.5h — not a reboot; something halted them). Recovered via the operator-provisioned
|
||
ansible-become path: `virsh -c qemu:///system start k3s-dev-{1,2,3}` → cluster back, all Kustomizations +
|
||
HelmReleases Ready from disk in ~80s; restarted Gitea (named volume `gitea-bootstrap-data` persisted) and
|
||
pushed the queued commits. No data lost. (The VMs are SYSTEM-libvirt — user-session virsh shows nothing;
|
||
recovery needs sudo, which only the ansible vault-become has.)
|
||
|
||
### Phase 4 — MONITORING DEPLOYED + healthy
|
||
Enabled `monitoring` in platform/kustomization.yaml. All 3 HelmReleases Ready (kube-prometheus-stack
|
||
65.8.1, Loki 6.55.0, Alloy 0.12.6); Prometheus + Loki PVCs bound on Longhorn (5Gi each); 38
|
||
PrometheusRules applied; no unhealthy pods. Grafana admin + Alertmanager SMTP from the operator's SOPS
|
||
secrets. **SMTP caveats (flagged, not blocking):** operator set smtp_smarthost=smtp.gmail.com:587,
|
||
[email protected], [email protected]; I set the receiver `to:` to a valid
|
||
inbox (was REPLACE-ME). Gmail needs an **App Password** and the From must be the auth user or a verified
|
||
send-as alias, else delivery fails at alert time (stack health unaffected).
|
||
|
||
### FINDING F-MON1 — monitoring PrometheusRule CRD ordering (FIXED)
|
||
Enabling monitoring first deadlocked the platform Kustomization: `alerts.yaml` (PrometheusRule CRs) needs
|
||
the monitoring.coreos.com CRDs the kube-prometheus-stack HelmRelease installs in the SAME pass → atomic
|
||
'no matches for kind PrometheusRule' → NO monitoring HRs created. Same controllers/configs fix
|
||
(F-DNS1/kyverno lineage): moved `alerts.yaml` to platform-config (dependsOn platform). This is now a
|
||
RECURRING pattern — any component shipping CRs of a CRD it installs must split the CRs into platform-config.
|
||
|
||
### Phase 3 — VPN egress gateway AUTHORED (author-first, not deployed)
|
||
`platform/vpn-egress/`: single shared **Gluetun** gateway (PIA/WireGuard, kill-switch FIREWALL=on, native
|
||
PF, control-API Service), non-app namespace (NET_ADMIN + /dev/net/tun), pre-wired (commented)
|
||
pia-credentials line. Defined the PIA secret location (**platform/vpn-egress**, keys username/password →
|
||
OPENVPN_USER/OPENVPN_PASSWORD) and updated the SOPS template. NOT wired into the graph. **Open design
|
||
(README):** egress steering (`vpn`-class pods → the one shared tunnel) — recommended Cilium Egress Gateway
|
||
→ the Gluetun node's `wg0` (fail-closed when the tunnel drops), deliberately NOT committed as an untested
|
||
CiliumEgressGatewayPolicy. Deploy needs PIA creds + `PIA_REGION` (PF-capable) + `CLUSTER_POD_CIDR` +
|
||
steering validation + the qbittorrent-natmap port-sync helper. **Cluster fully green** (4/4 KS, 8/8 HR).
|
||
|
||
## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed)
|
||
| # | Issue | Impact | Fix |
|
||
|---|---|---|---|
|
||
| K1 | ~~Substitution-var case mismatch (UPPERCASE refs vs lowercase cluster-vars keys)~~ | — | ✅ **FIXED**: uppercased cluster-vars keys + dev/prod overlay paths; all refs resolve; builds green |
|
||
| K2 | ~~Conformance L2 tests partial/unverified~~ | — | ✅ **DONE**: fixed 3 test bugs (see below), ran via run.sh against the live cluster → **R-NET-01 default-deny + R-NET-02 deny-undeclared PASS**, JUnit+coverage written, exit 0. chainsaw v0.2.12 installed (host `~/.local/bin`) |
|
||
| K3 | `bootstrap/gitea/bootstrap-gitea.sh` used a host bind-mount → rootless-podman perms broke app.ini | initial Gitea start failed | FIXED: switched to a named volume |
|
||
| K4 | **Kyverno/chainsaw findings during K2 (fixed in the tests):** (a) chainsaw does STRICT (length-equal) list matching — asserting one condition against a pod's full `conditions[]` fails → used scalar `phase: Running`; (b) a default-deny baseline blocks the **kubelet readiness probe** (node→pod), so probe-bearing pods never go Ready → dropped probes; (c) chainsaw report `path: ../_artifacts` resolved to repo-root → exit 1 → fixed to `_artifacts`; (d) swapped agnhost→busybox httpd (cached, cache-timing-proof) | — | FIXED in the test files/config |
|
||
|
||
## FINDING for WS-NET (Phase 2, T-2.2 — default-deny baseline design)
|
||
The platform's real default-deny baseline **must not break kubelet health/liveness/readiness probes**
|
||
(node→pod ingress). A naive `ingress: []` NetworkPolicy in Cilium drops the kubelet TCP probe, so any
|
||
pod with a probe never becomes Ready. The baseline must explicitly allow the health-check path (Cilium
|
||
host/health-check allowance or an equivalent ingress exception). Surfaced empirically by the R-NET-02
|
||
conformance test; **carry into T-2.2**.
|
||
|
||
## Toolchain note
|
||
`chainsaw v0.2.12` is installed on the host (`~/.local/bin`) for local conformance runs; for full
|
||
reproducibility it should also be baked into `infra/devcontainer` (pinned) — the in-cluster Chainsaw
|
||
Job (ADR-0011) already uses the pinned image. Deferred (non-blocking).
|
||
|
||
## Blocked on operator (hand-offs)
|
||
- **T-2.4p** Cloudflare API token → cert-manager DNS-01 + ExternalDNS (Phase 2 TLS/DNS).
|
||
- **T-3.2p** PIA username/password → VPN egress gateway (Phase 3).
|
||
- **OQ-BKP** off-cluster backup target → Velero (Phase 4).
|
||
- SMTP creds → Alertmanager email (Phase 4).
|
||
|
||
## Disk/drive incident + k3s relocated to NVMe (2026-08-22)
|
||
**Trigger:** host root fs (Kingston SSD `/dev/sdb3`, btrfs) hit ~99% (<1GB); the system paused the k3s
|
||
VMs on I/O error. Operator asked to move the VM disks off root to a larger drive.
|
||
|
||
**Why root filled:** the k3s VM disks lived under the snapper-snapshotted root subvolume; high VM-disk
|
||
churn multiplied across hourly snapshots. Freed by deleting the 11 `Snapper` timeline snapshots +
|
||
`virsh destroy`-ing the paused VMs (which held the old, rsync-unlinked inodes open) → btrfs reclaim.
|
||
Root 99% → 30% (326G free).
|
||
|
||
**F-DISK1 — the Seagate 4TB HDD is FAILING.** First relocated the VM disks to the HDD (`/mnt/seagate`);
|
||
etcd then hit **48% iowait + constant "apply took too long"**, pods drifted to Unknown, never converged.
|
||
SMART on `/dev/sda` (ST4000DM000): **29,408 reallocated, 448 current-pending + 448 offline-uncorrectable**,
|
||
~25k power-on hrs. A dying drive, not "HDD is slow." Do NOT use for etcd/critical data.
|
||
|
||
**Resolution:** k3s disks moved to the **WD_BLACK SN770 NVMe** (`/dev/nvme0n1p3`, `/mnt/games`) in a
|
||
dedicated btrfs subvol `/mnt/games/libvirt-k3s` (nodatacow), **bind-mounted to `/var/lib/libvirt/images`**
|
||
so tofu/libvirt paths + the qcow2 backing chain are unchanged (no IaC edits). etcd iowait → 0%, cluster
|
||
fully converged (64 Running). fstab updated; SELinux relabeled. Archival VMs (rocky10/silverblue/win11)
|
||
repointed to their real HDD path and LEFT on the failing drive.
|
||
|
||
**Follow-ups (assumptions logged):** (1) the NVMe bind-mount is a runtime change — codify in an Ansible
|
||
`host-prereqs` task for reproducibility; (2) redundant k3s copies remain at `/mnt/seagate/libvirt-images/k3s-dev-*`
|
||
(safe to delete; NVMe authoritative) — left pending operator OK; (3) archival VMs on the failing HDD need
|
||
migration + the disk replaced.
|
||
|
||
## Phase 3 VPN — protocol resolved to OpenVPN-UDP; deploying (2026-08-22)
|
||
ADR-0013 said PIA/WireGuard, but **Gluetun does not support PIA over WireGuard** (upstream: native support
|
||
"cannot be added… slow WIP"; live error "provider name is not valid for Wireguard"). **Operator (as System
|
||
Architect) approved OpenVPN-UDP and ADR-0013 was amended in the spec repo** (protocol only; architecture
|
||
unchanged; `thrnz/docker-wireguard-pia` recorded as the WireGuard fallback). Impl aligned: `VPN_TYPE=openvpn`,
|
||
`vpn-egress` ENABLED in `platform/kustomization.yaml`, stale WireGuard comments cleaned (gluetun-gateway.yaml,
|
||
README; steering iface `wg0`→`tun0`). Pushed to Gitea → Flux reconciles the OpenVPN gateway. Stale WireGuard
|
||
comments remain inside the operator's encrypted `pia-credentials.sops.yaml` header — left untouched (editing
|
||
breaks the SOPS MAC); cosmetic.
|
||
|
||
**Next:** confirm the OpenVPN tunnel comes up + correct egress IP, then egress-steering validation (README
|
||
option 1, `tun0`) + kill-switch conformance (R-NET-06).
|
||
|
||
### Phase 3 VPN — tunnel BLOCKED by home network (2026-08-22)
|
||
After deploying OpenVPN + fixing the region (`SE Stockholm`), gluetun goes **Running** and OpenVPN dials
|
||
PIA Sweden servers over **UDP/1197**, but the handshake never completes — no `TLS: Initial packet from`
|
||
the server, gluetun healthcheck restarts it in a loop. **Not auth** (no `AUTH_FAILED`), **not a k8s
|
||
NetworkPolicy** (vpn-egress is `app-namespace=false`, no policies touch it), **not the cluster/host path**.
|
||
|
||
Traced with tcpdump end-to-end:
|
||
- On the node (k3s-dev-3): pod `10.0.2.161` → Cilium SNAT → `ens3 Out 10.73.0.13 > 46.246.x:1197`. Leaves fine.
|
||
- On the host: `virbr1 In 10.73.0.13 > 46.246.x:1197` → `wlp4s0 Out 192.168.2.103 > 46.246.x:1197`. Host
|
||
NATs it out the **WiFi** onto the `192.168.2.x` home LAN. Leaves fine.
|
||
- On the host `wlp4s0`, filtering `src net 46.246.0.0/16`: **0 reply packets in 20s.** Meanwhile host
|
||
UDP/53 (`dig @9.9.9.9`) works. → OpenVPN UDP replies never return to the host.
|
||
|
||
**Conclusion:** the break is UPSTREAM of the host — the home router/ISP on the `192.168.2.x` WiFi network
|
||
is dropping/not-returning OpenVPN UDP/1197 (TCP egress + UDP/53 both work; only the VPN UDP is affected;
|
||
this box is on 192.168.2.x via WiFi, not the main 192.168.1.x LAN). **Environmental, outside the repo.**
|
||
|
||
**Action → operator:** fix at the network (check the 192.168.2.x router for outbound UDP/1197 or VPN
|
||
filtering; try a wired connection to the main 192.168.1.x LAN; or a different port/provider). No in-config
|
||
workaround — the user requires non-TCP, and WireGuard (the fallback) is also UDP so likely blocked the same
|
||
way. `vpn-egress` reverted to disabled meanwhile (config is complete + correct; one-line re-enable once the
|
||
network passes OpenVPN UDP).
|
||
|
||
### Phase 3 VPN — RESOLVED: it was a port-1197 collision, moved to UDP/53 (2026-08-22)
|
||
Operator clarified: the home network already runs **its own OpenVPN server NAT'd on WAN:1197**, which
|
||
collided with the cluster's outbound PIA-on-1197 flow at the router (why UDP/53 worked but 1197 didn't —
|
||
port-specific, not a general UDP block). Fix: set `OPENVPN_ENDPOINT_PORT` to a non-1197 PIA OpenVPN port.
|
||
PIA accepts `53, 1194, 1197, 1198, 8080, 9201` (gluetun issue #2252). Chose **53** — empirically proven to
|
||
traverse this network (the `dig @9.9.9.9` test) and no conflict. Added cluster-var `PIA_OPENVPN_PORT="53"`,
|
||
wired to gluetun `OPENVPN_ENDPOINT_PORT`, and re-enabled `vpn-egress`. Port-forwarding is an over-tunnel PIA
|
||
API call, unaffected by the transport port. (WireGuard fallback would also have worked — PIA WG is UDP/1337,
|
||
not 1197 — but staying on Gluetun+OpenVPN is the smaller change.)
|
||
|
||
**Correction: 53 didn't work either → settled on 8080.** Deploying on 53 still got no PIA replies —
|
||
this network **transparently intercepts outbound :53 (DNS)**, so `dig` succeeds (proxy forwards it) but
|
||
OpenVPN-over-53 is swallowed. Cheap decisive test: a raw **NTP/123** probe from the host got a reply →
|
||
arbitrary non-53 UDP works bidirectionally, so it's port-specific (1197 conflict + 53 DNS-intercept), not a
|
||
UDP block. Moved `OPENVPN_ENDPOINT_PORT` to **8080** (community-verified for PIA, non-DNS, non-1197).
|
||
Also hit the `${VAR}`→int-coercion gotcha (k8s "expected string, got 53") — the port is hardcoded as a
|
||
quoted string in gluetun-gateway.yaml (Longhorn-fix precedent), not a cluster-var.
|
||
|
||
### Phase 3 VPN — OpenVPN unusable on this network → ESCALATE to WireGuard (thrnz) (2026-08-22)
|
||
Tested three PIA OpenVPN UDP ports; each fails for a **different** reason (so it's OpenVPN-specific, not a
|
||
UDP block — a raw NTP/123 probe from the host returns fine):
|
||
- **1197** — outbound leaves, zero replies. Conflicts with the home OpenVPN server NAT'd on WAN:1197.
|
||
- **53** — outbound leaves, zero replies. This network transparently intercepts outbound :53 (DNS), so
|
||
`dig` succeeds but OpenVPN-over-53 is swallowed.
|
||
- **8080** — PIA **responds** (OpenVPN reaches the TLS handshake) but `TLS handshake failed` repeatedly
|
||
across servers → classic **MTU/fragmentation** (OpenVPN's large TLS handshake packets dropped on the
|
||
WiFi + triple-NAT path: pod→Cilium→node→libvirt-NAT→WiFi→home-NAT).
|
||
|
||
**Recommendation (ESCALATED — spec-level, ADR-0013):** pivot to **WireGuard via `thrnz/docker-wireguard-pia`**
|
||
(PIA WG = **UDP/1337**): a different port (no 1197 conflict, no 53 intercept), and WireGuard's handshake is
|
||
small fixed-size packets → **not MTU-sensitive** like OpenVPN's TLS handshake, so it should succeed where
|
||
8080 fails. It's ADR-0013's documented WG fallback and the operator's original preference. Trade-offs
|
||
(per the ADR): different single-maintainer image, and PF is file-based (`/pia-shared/port.dat`) rather than
|
||
Gluetun's control API — matters only when qBittorrent/natmap is built (Phase 6), not now. `vpn-egress`
|
||
disabled meanwhile (crashloop would fire false alerts). Awaiting operator's architect decision before the
|
||
container swap.
|
||
|
||
### Phase 3 VPN — ACTUAL root cause: Cilium VXLAN pod MTU (2026-08-22)
|
||
Operator (correctly) rejected the port theory (client uses ephemeral src ports; dest port is irrelevant to
|
||
return traffic) and had me check MTU at each layer. Findings via `ping -M do` sweeps:
|
||
- **Pod** → internet: MTU 1450 OK, 1470 DROP (silent). Pod `eth0` shows 1500 but the effective route is 1450.
|
||
- **Node** (k3s-dev-3) → internet: **1500 OK**. All node ifaces (ens3, cilium_*, lxc*) = 1500.
|
||
- **Hypervisor** → internet: **1500 OK**. (Google Fiber SOHO router hands off full 1500 — not the culprit.)
|
||
- Cilium config: **`routing-mode: tunnel`, `tunnel-protocol: vxlan`** → Cilium subtracts the 50-byte VXLAN
|
||
overhead → **pod MTU 1450**. OpenVPN emits ~1500-byte handshake packets → exceed 1450 → dropped (broken
|
||
PMTUD = no ICMP back). This is why HTTPS (MSS-clamped) + small UDP (DNS/NTP) work but OpenVPN doesn't.
|
||
|
||
### Phase 3 VPN — ✅ WORKING & STABLE (2026-08-23)
|
||
gluetun (v3.41.3, custom mode, operator's IP-based PIA .ovpn) connects to PIA Stockholm and **holds**:
|
||
egress IP `158.173.241.36` (Sweden), pod Running 0 restarts. Two final fixes after the .ovpn landed:
|
||
- The .ovpn's `remote` must be an **IP** (gluetun's custom parser rejects the `sweden.privacy.network`
|
||
hostname) — operator supplied an IP-based .ovpn (from the sweden.privacy.network LB pool, 158.173.241.x).
|
||
- **Liveness probe → exec (localhost)**: once the tunnel redirects the pod default route via tun0, the pod's
|
||
reply to the kubelet's *httpGet* liveness probe egressed the tunnel and never returned → probe timeout →
|
||
restart every ~110s despite the tunnel being up. A local `wget` exec probe stays on `lo`. (OpenVPN's
|
||
"route already exists / Network unreachable" logs are cosmetic — gluetun adds 0.0.0.0/0 itself; egress works.)
|
||
Remaining Phase-3 work (later): egress-steering validation (Cilium egress-gw → tun0) + kill-switch conformance
|
||
(R-NET-06); port-forwarding is unavailable in custom mode (Phase-6/qBittorrent — revisit then).
|
||
|
||
### Phase 3 VPN — custom .ovpn mode (the real cause: flaky server selection) (2026-08-23)
|
||
After the MTU red-herrings, packet capture (node ens3 + host wlp4s0, GNOME OFF so it wasn't the host's own
|
||
VPN) proved the pod CAN complete a **full** PIA handshake — a working flow (`50017 → 158.173.241.22:1198`)
|
||
showed 1222-byte cert packets BOTH ways, reaching the node and pod end-to-end through the double NAT
|
||
(Cilium SNAT → libvirt masq → router). It's **intermittent per-server**: gluetun's PIA-provider mode
|
||
random-picks server IPs and many don't answer this cluster's flows (→ TLS timeouts, exit-0 restarts ~every
|
||
2min), while the operator's GNOME client is reliable because its `.ovpn` targets **`sweden.privacy.network`**
|
||
— a load-balanced hostname that always lands on a healthy server. NOT MTU, NOT encryption preset (strong AND
|
||
normal both fail identically once the env name `PRIVATE_INTERNET_ACCESS_OPENVPN_ENCRYPTION_PRESET` was
|
||
corrected), NOT Cilium (policy default-allow, host-fw off, Hubble FORWARDED), NOT a network block.
|
||
|
||
**Decision (operator's choice):** switch gluetun to **custom-provider mode** feeding the operator's exact
|
||
`.ovpn` (`VPN_SERVICE_PROVIDER=custom`, `OPENVPN_CUSTOM_CONFIG=/config/pia.ovpn`, .ovpn from the
|
||
`pia-custom-ovpn` SOPS Secret; creds still via OPENVPN_USER/PASSWORD; kill-switch kept; tun-mtu 1300 for the
|
||
1450 pod MTU). Dropped the PIA-provider server list + updater initContainer. **Trade-off:** custom mode
|
||
loses gluetun's PIA-native port forwarding (Phase-6/qBittorrent concern) — revisit then. This is an
|
||
implementation choice within ADR-0013's "single shared Gluetun PIA OpenVPN gateway" model (architecture
|
||
unchanged), noted here rather than escalated. Pending operator provisioning of the `.ovpn` Secret.
|
||
|
||
<!-- superseded strong/normal-preset attempt below, kept for the record -->
|
||
**(superseded) Fix (operator's choice: "shrink OpenVPN MTU in gluetun"):** set gluetun `OPENVPN_MTU=1350` (tun-mtu) and
|
||
`OPENVPN_MSSFIX=1350` so encrypted packets (~1350+52 ≈ 1402) fit under the 1450 pod MTU. Keeps OpenVPN +
|
||
Cilium VXLAN unchanged. Port kept at 8080 (confirmed PIA responds). Re-enabled `vpn-egress`. Values hardcoded
|
||
as quoted strings (int-coercion gotcha). Alternatives not taken: Cilium native routing (pod MTU→1500) or
|
||
WireGuard/thrnz (MTU-robust) — both still available if the MTU shrink proves insufficient.
|
||
|
||
### Phase 3 VPN — ACTUAL ACTUAL root cause: the network blackholes PIA's IP ranges (2026-08-22)
|
||
The MTU shrink did NOT fix it. A node-side tcpdump of the handshake was decisive: OpenVPN's outbound init
|
||
packets are **14 bytes** (nowhere near any MTU) and get **zero replies** — so MTU was never the handshake
|
||
blocker either. Confirmed pod UDP egress is healthy (pod NTP/123 reply OK). Then the real test — raw
|
||
TCP/ICMP reachability to PIA IPs vs controls (from host AND pod):
|
||
- **PIA** (46.246.8.55, 212.112.19.22): TCP **501 and 443 time out** (silent drop, not refused), UDP no
|
||
reply, ICMP no reply — across many PIA servers gluetun cycled.
|
||
- **Google / Cloudflare**: TCP/443 OPEN, UDP replies, ICMP replies.
|
||
|
||
→ **This network (192.168.2.x, WiFi) blackholes traffic to PIA's IP ranges.** It also transparently
|
||
intercepts DNS/:53. Everything else (HTTPS, DNS-via-proxy, NTP, general internet) works. So it is neither
|
||
ports, nor MTU (both red herrings — the Cilium VXLAN 1450 pod MTU is real but irrelevant to a 14-byte
|
||
handshake), nor the cluster/config — it is upstream network filtering of PIA specifically.
|
||
|
||
**Action → operator (network, not code):** move the k3s host to the unfiltered main LAN (192.168.1.x /
|
||
wired), or unblock PIA's ranges on the 192.168.2.x router, then re-enable `vpn-egress` (no code change).
|
||
WireGuard/thrnz would NOT help — same PIA IPs are blocked; if switching providers is ever considered, pick
|
||
one whose ranges this network permits. Reverted the port/MTU red-herring env from gluetun-gateway.yaml
|
||
(back to PIA defaults); `vpn-egress` disabled so it doesn't crashloop/alert. **Config is correct and
|
||
complete — the only blocker is the network.**
|
||
|
||
### Phase 3 VPN — CORRECTION: it was gluetun's STALE server list, NOT a network block (2026-08-22)
|
||
The "network blackholes PIA" conclusion above was WRONG. The operator installed a fresh PIA `.ovpn` in GNOME
|
||
on the same host/network and it connected fine → the network permits PIA. Verified against PIA's live server
|
||
list: current SE Stockholm servers are **158.173.241.x** and ARE reachable (tcp/8080 → RST / "connection
|
||
refused" = the server answered). The IPs gluetun was dialing (**46.246.x / 212.112.x**) are **retired**
|
||
(timeout = dead hosts). So gluetun's **bundled server list was stale**, and my host reachability test used
|
||
those dead IPs → false "blackhole." (Cilium was independently ruled out per the operator's request: policy
|
||
mode `default` = default-allow, host firewall disabled, no policy selects these pods, and Hubble showed
|
||
pod→PIA `to-network FORWARDED`, no drops.)
|
||
|
||
**Fix:** gluetun `UPDATER_PERIOD=24h` (fetch PIA's current server list at startup); keep `OPENVPN_MTU=1350`
|
||
+ `OPENVPN_MSSFIX=1350` for the real Cilium-VXLAN 1450 pod MTU (data channel); PIA default OpenVPN UDP port.
|
||
Re-enabled `vpn-egress`. **Lesson:** when gluetun "can't reach" a provider, suspect a stale bundled server
|
||
list FIRST — validate against the provider's live server list before blaming the network/MTU/ports.
|
||
|
||
### Phase 3 egress steering — ADR-0015 implemented (PM-orchestrated), NOT yet deployed (2026-08-24)
|
||
Architect amended the spec to **ADR-0015** (node-level host OpenVPN tunnel + Cilium Egress Gateway + host
|
||
nftables kill-switch; supersedes the in-cluster-Gluetun steering — a Gluetun pod's `tun0` can't be steered by
|
||
Cilium egress gateway, and hostNetwork Gluetun hijacks the node default route). Operator also fixed the
|
||
invariants: **default = local/WAN for everything; VPN strictly opt-in; kill-switch scoped to vpn-class only;
|
||
opposite fail directions** (internet fails open, vpn fails closed).
|
||
|
||
PM dispatched two **WS-NET** agents (feature branches, no deploy); PM reviewed vs the invariants + DoD and
|
||
merged to `main` (`33d048d`) — **code only, nothing deployed** (policy not in the kustomization, playbook not
|
||
run, working gluetun kept as fallback):
|
||
- `infra/ansible/roles/vpn_egress_gateway/` + `playbooks/vpn.egress.gateway.yml` — host OpenVPN systemd
|
||
(PIA/OpenVPN-UDP, creds from **Ansible vault**), **split-tunnel** (`route-noexec` + dedicated route table
|
||
51820 + `ip rule from <tun0-addr>` → node's main-table default untouched), **scoped nftables kill-switch**
|
||
(policy-accept so node/internet fail-open; tun0-sourced/marked traffic dropped when not egressing tun0 →
|
||
vpn fail-closed). Inventory group `vpn_egress_gateway` = `k3s-dev-3`.
|
||
- `platform/vpn-egress/egress-steering.yaml` — `CiliumEgressGatewayPolicy` (only `egress-class=vpn`, RFC1918
|
||
excluded, `interface: tun0`, gateway-node label). Held OUT of `kustomization.yaml` until validated.
|
||
|
||
PM review: verified the up-script, nft rules, and policy honor all four invariants. **Escalation ruled:** the
|
||
tunnel-*fully-down* independent backstop the agent proposed via a fwmark on the CiliumEgressGatewayPolicy is
|
||
**infeasible** (the policy has no fwmark field). Decision: keep the scoped nft + routing blackhole + inert
|
||
fwmark scaffolding, and **gate on the `vpn-killswitch` conformance test** (force `tun0` down → assert no leak,
|
||
empirically proving Cilium fails closed). Build a real backstop (host controller syncing vpn-pod IPs into the
|
||
nft set) ONLY if that test leaks — not over-engineered now.
|
||
|
||
**Cross-agent git hazard noted:** the two agents shared one working dir (I didn't use isolated worktrees), so
|
||
one branch picked up the other's file; caught + cleaned during review. Next parallel coding fan-out → use
|
||
`isolation: worktree`.
|
||
|
||
**OPERATOR PREREQ (T-3.2p):** provision `infra/ansible/inventory/group_vars/vpn_egress_gateway/vault.yml` with
|
||
`vault_vpn_egress_gateway_pia_{username,password,ovpn}` (the `.ovpn` = the proven `sweden-aes-128-cbc-udp-ip`
|
||
one, liftable from `platform/vpn-egress/pia-custom-ovpn.sops.yaml`). **VALIDATION FLOW** (post-prereq): run the
|
||
playbook → label `k3s-dev-3` `egress-gateway=true` → add `egress-steering.yaml` to the kustomization → run
|
||
`egress-class` + `vpn-killswitch` conformance → then retire gluetun in a separate change.
|
||
|
||
### Phase 3 egress steering — VALIDATED END-TO-END on dev (2026-08-24)
|
||
Operator provisioned the host vault; PM drove the deployment + validation on `k3s-dev-3`:
|
||
- Playbook ran (after one AUTH_FAILED — operator had a wrong PIA password in the vault, then fixed it; my
|
||
earlier vault recipe put the password in YAML *double* quotes, a special-char hazard — use single quotes /
|
||
programmatic lift next time). Tunnel up: `tun0` 10.72.193.53, VPN egress **158.173.241.36** (PIA Stockholm).
|
||
- **Split-tunnel invariants hold:** node default unchanged (`via 10.73.0.1 dev ens3`), node's own egress =
|
||
**WAN 136.51.115.252**, node Ready throughout. `ip rule 5000: from <tun0-addr> lookup vpn_egress`, dedicated
|
||
table (tun0 default + blackhole), nft set scoped to the tun0 src — all correct.
|
||
- **Steering wired via GitOps:** added `egress-steering.yaml` to `platform/vpn-egress/kustomization.yaml`
|
||
(fixed a self-inflicted break first — the file had been dropped from main by the earlier gateway-role merge;
|
||
restored from 38fcfad). Flux reconciled (`8a55599`); `CiliumEgressGatewayPolicy vpn-egress-steering` live.
|
||
- **egress-class gate PASS:** `egress-class=vpn` pod → 158.173.241.36 (VPN), `egress-class=internet` pod →
|
||
136.51.115.252 (WAN). HTTP ×5 + HTTPS ×3 all 200 (early timeouts were post-restart warmup).
|
||
- **vpn-killswitch gate PASS (R-NET-06 fault-injection):** stopped the tunnel → `tun0` gone, ip rule removed,
|
||
nft set flushed, blackhole retained → vpn-class pod **timed out (no WAN leak)**, internet-class unaffected;
|
||
restart recovered cleanly.
|
||
- **FINDING (robustness, feeds WS-NET):** fail-closed currently rides on the stale SNAT source (10.72.193.53)
|
||
being non-routable, because the **down-script flushes the nft set + removes the ip rule** — disarming the
|
||
scoped drops exactly when the tunnel is down. The up-script already flush+re-adds the set on connect, so the
|
||
down-script's `nft flush set` is redundant AND harmful: keeping the last tun0 src in the set makes the
|
||
existing `ip saddr @set oifname != tun0 drop` rule an *explicit, edge-independent* backstop. Fix = drop the
|
||
`nft flush set` line from `vpn-egress-down.sh.j2`. Gate passes without it, but this is defense-in-depth.
|
||
- **Cross-agent lesson reinforced:** parallel agents must use `isolation: worktree` (an earlier non-isolated
|
||
pair caused the dropped-file break above).
|