Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
b258e47aa6 |
No files matched your search
@@ -0,0 +1,198 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: fix/vpn-ingress-l2-lease-collision
|
||||
title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it
|
||||
was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is
|
||||
not re-derived — it is very easy to re-derive from `ip neigh` alone.
|
||||
|
||||
### The claim, and how it died
|
||||
|
||||
The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2
|
||||
announcement lease it holds itself* — the announcer does not answer its own ARP request, the
|
||||
neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because
|
||||
`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
|
||||
own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded
|
||||
that label from announcing, with two alerts and four promtool tests behind it.
|
||||
|
||||
The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*`
|
||||
leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) —
|
||||
the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five
|
||||
VIPs and the other two nodes should reach **all**.
|
||||
|
||||
**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node:
|
||||
|
||||
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|
||||
|---|---|---|
|
||||
| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
|
||||
| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms |
|
||||
| `nc -z 192.168.2.20 443` | **OK** | OK |
|
||||
| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` |
|
||||
| `192.168.2.21:8443` | FAIL | **FAIL** (both) |
|
||||
|
||||
The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the
|
||||
asserted **effect is absent**. That is sufficient to falsify the causal claim without having to
|
||||
explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**.
|
||||
|
||||
### Why it is inert
|
||||
|
||||
No traffic path resolves a LoadBalancer VIP by ARP:
|
||||
|
||||
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
|
||||
`Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath
|
||||
is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP
|
||||
is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**.
|
||||
* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's
|
||||
backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet.
|
||||
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
|
||||
which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a
|
||||
bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT,
|
||||
not wire traffic.
|
||||
|
||||
### The two probes that misled me
|
||||
|
||||
Both were cited as evidence in the previous version of this entry.
|
||||
|
||||
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing
|
||||
artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong.
|
||||
* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a
|
||||
host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including
|
||||
the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it
|
||||
discriminates nothing.
|
||||
|
||||
**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then
|
||||
routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns
|
||||
probe cannot measure the forwarded path in either direction — it reports success where the forwarded
|
||||
path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not
|
||||
sharing a hook, and that is the assumption the original diagnosis rested on.
|
||||
|
||||
### Files
|
||||
|
||||
Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`,
|
||||
`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`,
|
||||
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`,
|
||||
`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted:
|
||||
`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`.
|
||||
|
||||
Kept, rewritten from scratch:
|
||||
|
||||
* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is
|
||||
inert and which probes lie, and the internet-egress facts below.
|
||||
* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster
|
||||
work, internet does not") that works outwards from the client profile, and a short section saying
|
||||
the L2 lease is a dead end, with the unifi `.21` note.
|
||||
|
||||
**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is
|
||||
proposed by this branch any more.**
|
||||
|
||||
### The internet-routing report: no server-side fix was needed, and none was made
|
||||
|
||||
The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full
|
||||
one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has
|
||||
emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`,
|
||||
2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and
|
||||
`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel
|
||||
feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that
|
||||
already exists. **I deliberately shipped no code change here** — writing one would have meant
|
||||
inventing a fix for a symptom I could not reproduce (see below).
|
||||
|
||||
The server side was measured end to end and is capable and correct, so the cause is in the client
|
||||
profile or the client. The triage section names the two candidates that fit the symptom exactly —
|
||||
a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and
|
||||
leak public traffic outside the tunnel — each with a one-line test that settles it.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on
|
||||
2026-10-17:
|
||||
|
||||
* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1`
|
||||
(`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again.
|
||||
* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a
|
||||
real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere.
|
||||
* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three
|
||||
nodes — so the successes are service-aware DNAT, not wire delivery.
|
||||
* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with
|
||||
`REACHABLE` neighbour entries. This is the measurement that destroys the original control.
|
||||
* **The datapath facts** quoted above from `cilium-dbg status --verbose` and
|
||||
`cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`:
|
||||
`kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`,
|
||||
`routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three.
|
||||
* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` →
|
||||
`dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP.
|
||||
So the 200 is not loopback delivery by the host stack — it is BPF.
|
||||
* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1`
|
||||
(read, never written); `iptables -t nat -S POSTROUTING` carries
|
||||
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0
|
||||
packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both
|
||||
present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it;
|
||||
`curl http://1.1.1.1/` returns `301` from each node.
|
||||
* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by
|
||||
`podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod
|
||||
`egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on
|
||||
`-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the
|
||||
home WAN IP, not PIA's.
|
||||
* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned
|
||||
`142.250.113.138`.
|
||||
* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one
|
||||
enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both
|
||||
`FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The
|
||||
`wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters.
|
||||
* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f`
|
||||
(2026-10-05) and never changed.
|
||||
* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could
|
||||
not test what a real remote client does, and — as established above — **no probe runnable from the
|
||||
pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a
|
||||
VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a
|
||||
LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one
|
||||
remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in
|
||||
the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this
|
||||
unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show`
|
||||
shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the
|
||||
client loads a cluster app — and repeat it on a node that holds leases and one that does not.*
|
||||
* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same
|
||||
reason. **I need a client connected** to learn it; the router is unreadable from the cluster.
|
||||
* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP
|
||||
failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes.
|
||||
My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**,
|
||||
rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A
|
||||
competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and
|
||||
`wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is
|
||||
needed for the falsification** — the asserted cause is present today without its effect. I did not
|
||||
retro-measure anything, and I did not read Cilium's source or issue tracker.
|
||||
* **The operator's report cannot be a description of "right now."** No client has connected since
|
||||
the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster
|
||||
services and the LAN but not the internet" must describe a session before the power-cycle or one
|
||||
that has since dropped. **I did not establish when it was observed, or with which profile.**
|
||||
* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not**
|
||||
find the fault, and I shipped no fix for it. The two candidates in the triage section (a
|
||||
`--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not
|
||||
measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line
|
||||
decides split vs full and halves the search — and a connected client for step 2.
|
||||
* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and
|
||||
lives only on the operator's machine and the device; it is not in the repo, and I did not look for
|
||||
it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing.
|
||||
* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without
|
||||
a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU
|
||||
would see large transfers hang while small requests succeed. The runbook's existing MTU note
|
||||
(drop to 1280) already covers the remedy.
|
||||
* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole
|
||||
IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a
|
||||
client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an
|
||||
interface with no IPv6 address.
|
||||
* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate,
|
||||
pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1
|
||||
offline subset only. Nothing above is covered by a test that runs in CI.
|
||||
* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there
|
||||
is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating
|
||||
`kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material).
|
||||
* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).
|
||||
@@ -1,78 +0,0 @@
|
||||
---
|
||||
date: 2026-10-18
|
||||
branch: fix/renovate-invalid-config
|
||||
title: "renovate.json was invalid, so the first manual run errored out"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The operator ran the Renovate workflow manually and it **failed**. The cause is not the workflow:
|
||||
**`renovate.json` was never a valid Renovate config.**
|
||||
|
||||
It carried five pseudo-comment keys — `_comment`, `_kubernetes_comment`, `_unmanageable_comment`,
|
||||
`_ignorePaths_comment`, `_docker_comment` — because JSON has no comments. Renovate **rejects unknown
|
||||
top-level options**, so the entire config failed validation. Proven with the official validator
|
||||
rather than inferred:
|
||||
|
||||
```
|
||||
$ podman run --rm -v "$PWD/renovate.json:/tmp/renovate.json:ro,Z" \
|
||||
ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json
|
||||
ERROR: Found errors in configuration
|
||||
Invalid configuration option: _comment
|
||||
Invalid configuration option: _docker_comment
|
||||
Invalid configuration option: _ignorePaths_comment
|
||||
Invalid configuration option: _kubernetes_comment
|
||||
Invalid configuration option: _unmanageable_comment
|
||||
```
|
||||
|
||||
This is the second independent reason Renovate had never worked, on top of PR #77's cron firing
|
||||
outside `renovate.json`'s own schedule window. Either alone was sufficient to produce "nothing ever
|
||||
happens"; #73 shipped both.
|
||||
|
||||
**Converted `renovate.json` → `renovate.json5`.** JSON5 permits real `//` comments, and Renovate
|
||||
discovers `renovate.json5` natively — nothing in `.gitea/` or `ci/` pins the old filename (only
|
||||
prose inside comments), and no `RENOVATE_CONFIG_FILE` is set. Every word of the original rationale
|
||||
is preserved as comments positioned next to what it explains, which a flattened `description`
|
||||
string would have lost.
|
||||
|
||||
A header on the new file records the failure, the validator command, and the instruction to run it
|
||||
before committing — this class of bug is invisible to the repo's own gate.
|
||||
|
||||
## What I verified
|
||||
|
||||
* **The exact failure**, from the official validator, quoted above. Not read off a log, not inferred
|
||||
from the workflow.
|
||||
* **The fix validates**: the same validator against `renovate.json5` returns
|
||||
`INFO: Config validated successfully against 1 file(s)`.
|
||||
* **Nothing was dropped in the conversion**, checked programmatically against `git show
|
||||
HEAD:renovate.json`: 18 non-comment keys before, 18 after, **no missing keys, no extra keys, and
|
||||
no key whose value changed**. `docker: {enabled: true}` confirmed still present — the validator
|
||||
prints a normalisation diff that mentions it, which is its own migration suggestion, not a
|
||||
removal.
|
||||
* **74 comment lines survive** in the new file, so the rationale is intact.
|
||||
* **`RENOVATE_TOKEN` exists** as a repository Actions secret (listed by name via the Gitea API), so
|
||||
the workflow's fail-hard preflight is satisfied.
|
||||
* **A suspicion was DISPROVED:** `enabledManagers` containing bare `regex` was thought to need to be
|
||||
`custom.regex`, which would have silently disabled every inline `# renovate:` marker. The
|
||||
validator accepts `regex`, so that concern was unfounded and nothing here changes it.
|
||||
* Nothing functionally references `renovate.json` by name — `grep -rn "renovate\.json" .gitea/ ci/
|
||||
conformance/` returns only comment prose.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **Renovate has still never completed a run.** This removes one of two blockers; PR #77 removes the
|
||||
other. Neither is merged, so **no successful Renovate run has been observed** and I cannot say the
|
||||
bot works — only that these two specific defects are fixed.
|
||||
* **I did not re-run the workflow after the fix.** The honest test is a `workflow_dispatch` with
|
||||
`dry_run: true` once both this and #77 are merged. Expect ~30 dependencies behind, rate-limited to
|
||||
`prHourlyLimit: 2` / `prConcurrentLimit: 5`.
|
||||
* **Whether the managers actually extract anything.** A valid config that matches no files is still
|
||||
a no-op. The dry-run log's per-manager dependency counts are the only thing that proves the
|
||||
`managerFilePatterns` are right, and I have not seen them.
|
||||
* **Whether `renovate.json5` is honoured by the pinned Renovate version in the workflow.** JSON5
|
||||
discovery is long-standing and the validator accepted the file, but the workflow's image tag was
|
||||
not cross-checked against the version that added it.
|
||||
* **Nothing in the gate catches an invalid Renovate config.** This bug would have recurred silently.
|
||||
A one-line L0 check shelling out to `renovate-config-validator` would close it; it is not written
|
||||
here, and `conformance/**` is a sensitive path.
|
||||
* I did not re-measure branch protection or runner state (CLAUDE.md §2b).
|
||||
@@ -186,6 +186,86 @@ right way; one was a defect in my own manifest.
|
||||
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
|
||||
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
|
||||
|
||||
## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach
|
||||
|
||||
A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose
|
||||
Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from
|
||||
announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is
|
||||
real but inert, and the distinction matters enough to record, because the wrong version is very
|
||||
easy to re-derive from `ip neigh` alone.
|
||||
|
||||
Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case,
|
||||
after the UPS power-cycle — from the host netns of the `wireguard` pod on each node:
|
||||
|
||||
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|
||||
|---|---|---|
|
||||
| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
|
||||
| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms |
|
||||
| `nc -z 192.168.2.20 443` | **OK** | OK |
|
||||
| plex `192.168.2.24:32400` | **`200`** | `200` |
|
||||
|
||||
So the announcer really does **not** answer its own ARP request — that part of the observation is
|
||||
reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**:
|
||||
|
||||
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
|
||||
`Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**,
|
||||
so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before
|
||||
any route or neighbour lookup for the VIP happens**.
|
||||
* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the
|
||||
shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without
|
||||
a packet crossing the LAN at all.
|
||||
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
|
||||
which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real**
|
||||
service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes —
|
||||
that is service-aware DNAT, not wire traffic.
|
||||
|
||||
**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis:
|
||||
|
||||
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an
|
||||
artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not
|
||||
anything is wrong.
|
||||
* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the
|
||||
two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it
|
||||
discriminates nothing — and it is not the client path either (see below).
|
||||
|
||||
**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are
|
||||
**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is
|
||||
**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the
|
||||
forwarded path in either direction: it reports success where the forwarded path might fail, and
|
||||
`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with
|
||||
a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE`
|
||||
counters move.
|
||||
|
||||
## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses
|
||||
|
||||
`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the
|
||||
component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature
|
||||
to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel
|
||||
client cannot reach the internet, nothing in the list below is the reason:
|
||||
|
||||
| checked | result on all three nodes |
|
||||
|---|---|
|
||||
| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` |
|
||||
| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change |
|
||||
| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE |
|
||||
| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present |
|
||||
| default route | `default via 192.168.2.1 dev eno1 onlink`, intact |
|
||||
| node's own internet reachability | `http://1.1.1.1/` returns `301` |
|
||||
|
||||
**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is
|
||||
closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by
|
||||
`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled
|
||||
`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity
|
||||
traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway
|
||||
node. A remote client's apparent public address is therefore the **home WAN IP**, which is the
|
||||
intended behaviour and the thing to state when enrolling a device.
|
||||
|
||||
**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one
|
||||
enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and
|
||||
the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the
|
||||
power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the
|
||||
forwarded node shows a peer with an `endpoint` and a recent `latest handshake`.
|
||||
|
||||
## Prerequisites outside this directory
|
||||
|
||||
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
|
||||
|
||||
@@ -0,0 +1,99 @@
|
||||
{
|
||||
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
|
||||
"_comment": "Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17 RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags. It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009 split that gave image tags to Flux image-automation is formally closed — see .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write anyway; and enabling it would need an in-cluster git WRITE credential for the repo that defines the cluster). image-automation has been commented out at platform/kustomization.yaml:52 since it was authored and the cluster holds zero ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down. CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag` line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER: .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.",
|
||||
"extends": [
|
||||
"config:recommended",
|
||||
":dependencyDashboard",
|
||||
":semanticCommits",
|
||||
"helpers:pinGitHubActionDigests"
|
||||
],
|
||||
"timezone": "America/New_York",
|
||||
"schedule": ["after 3am and before 6am"],
|
||||
"prConcurrentLimit": 5,
|
||||
"prHourlyLimit": 2,
|
||||
"commitMessagePrefix": "chore(deps):",
|
||||
"labels": ["dependencies", "automated-update"],
|
||||
"ignoreUnstable": true,
|
||||
"respectLatest": true,
|
||||
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
|
||||
"flux": {
|
||||
"managerFilePatterns": ["/(^|/)platform/.+\\.ya?ml$/", "/(^|/)apps/.+\\.ya?ml$/"]
|
||||
},
|
||||
"kustomize": {
|
||||
"managerFilePatterns": ["/(^|/)kustomization\\.ya?ml$/"]
|
||||
},
|
||||
"_kubernetes_comment": "ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes manifests, and it is NOT auto-detected — it only looks at files matched here, which is why this block is as broad as it is. Scope covers every place this repo writes a container image: platform components, the rendered app waves, the ServiceDefinition catalog that generates them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the rendered output is reverted by the next render and would also fail ci/render-golden.sh. Renovate will raise BOTH, and both must land together.",
|
||||
"kubernetes": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)platform/.+\\.ya?ml$/",
|
||||
"/(^|/)platform-config/.+\\.ya?ml$/",
|
||||
"/(^|/)apps/.+\\.ya?ml$/",
|
||||
"/(^|/)catalog/.+\\.ya?ml$/",
|
||||
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
|
||||
"/(^|/)infra/agent/.+\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
"customManagers": [
|
||||
{
|
||||
"customType": "regex",
|
||||
"description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)kustomization\\.ya?ml$/",
|
||||
"/(^|/)versions\\.env$/"
|
||||
],
|
||||
"matchStrings": [
|
||||
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
|
||||
],
|
||||
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
|
||||
}
|
||||
],
|
||||
"packageRules": [
|
||||
{
|
||||
"description": "Auto-merge the allowed low-risk classes on green CI — no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
|
||||
"matchUpdateTypes": ["minor", "patch", "digest"],
|
||||
"automerge": true,
|
||||
"automergeType": "pr",
|
||||
"platformAutomerge": true
|
||||
},
|
||||
{
|
||||
"description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
|
||||
"matchUpdateTypes": ["major"],
|
||||
"automerge": false,
|
||||
"addLabels": ["needs-review", "breaking"]
|
||||
},
|
||||
{
|
||||
"description": "Keep the KubeVirt + CDI operator pair on a compatible track — grouped so they bump together (platform/kubevirt/kustomization.yaml).",
|
||||
"matchDepNames": ["kubevirt/kubevirt", "kubevirt/containerized-data-importer"],
|
||||
"groupName": "kubevirt-cdi"
|
||||
},
|
||||
{
|
||||
"description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface — label but do not auto-merge; a version bump there changes CI reproducibility.",
|
||||
"matchFileNames": ["ci/versions.env", "infra/devcontainer/versions.env"],
|
||||
"automerge": false,
|
||||
"addLabels": ["ci-tooling"]
|
||||
},
|
||||
{
|
||||
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check — it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
|
||||
"matchPackageNames": [
|
||||
"kured",
|
||||
"longhorn",
|
||||
"cilium",
|
||||
"cloudnative-pg",
|
||||
"kube-prometheus-stack"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": ["needs-review", "blast-radius"]
|
||||
}
|
||||
],
|
||||
"_unmanageable_comment": "WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest, lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete version before any automation can maintain them, and until then they drift on every pull with nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12, e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it has been told not to leave; raising those ceilings is deliberate per-component work and is NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.",
|
||||
"ignorePaths": [
|
||||
"**/*.example",
|
||||
"**/*.sops.yaml",
|
||||
"conformance/**",
|
||||
"platform/image-automation/per-service/**",
|
||||
"platform/kyverno/tests/**"
|
||||
],
|
||||
"_ignorePaths_comment": "platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest` and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed shape, not software to keep current. `conformance/**` was already ignored for the same reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise have opened.",
|
||||
"_docker_comment": "The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation over a running tag. That controller was never enabled and never will be here (see _comment), so the only thing those two flags were achieving was leaving every plain `image:` tag in the repo unmanaged. `docker` is left out of enabledManagers because this repo contains no Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.",
|
||||
"docker": { "enabled": true }
|
||||
}
|
||||
-187
@@ -1,187 +0,0 @@
|
||||
// =================================================================================================
|
||||
// Renovate configuration — JSON5, so the rationale can live NEXT TO what it explains.
|
||||
// =================================================================================================
|
||||
// THIS FILE WAS `renovate.json` AND IT WAS INVALID. It carried five `_comment` / `_*_comment`
|
||||
// pseudo-comment keys, because JSON has no comments. Renovate REJECTS unknown top-level options, so
|
||||
// the whole config failed validation and the first manual run of the workflow ERRORED OUT. Proven
|
||||
// with the official validator (`ghcr.io/renovatebot/renovate renovate-config-validator`):
|
||||
//
|
||||
// ERROR: Found errors in configuration
|
||||
// Invalid configuration option: _comment
|
||||
// Invalid configuration option: _docker_comment
|
||||
// Invalid configuration option: _ignorePaths_comment
|
||||
// Invalid configuration option: _kubernetes_comment
|
||||
// Invalid configuration option: _unmanageable_comment
|
||||
//
|
||||
// Renovate discovers `renovate.json5` natively (no RENOVATE_CONFIG_FILE needed, and nothing in
|
||||
// .gitea/ or ci/ pins the old filename — only prose in comments), and JSON5 permits real `//`
|
||||
// comments. So the fix keeps every word of the original rationale, positioned, instead of
|
||||
// flattening it into a `description` string.
|
||||
//
|
||||
// VALIDATE BEFORE COMMITTING — it is one command and it is the only thing that catches this class:
|
||||
// podman run --rm -v "$PWD/renovate.json5:/tmp/renovate.json5:ro,Z" \
|
||||
// ghcr.io/renovatebot/renovate renovate-config-validator /tmp/renovate.json5
|
||||
//
|
||||
// NOTE on what was NOT wrong: `enabledManagers` containing bare `regex` was suspected of needing to
|
||||
// be `custom.regex`. The validator accepts `regex`, so that suspicion was unfounded and nothing
|
||||
// here changes it.
|
||||
{ "$schema": "https://docs.renovatebot.com/renovate-schema.json",
|
||||
// Renovate config — Phase 4 automated updates (ADR-0009, R-UPD-01/02). AS OF 2026-10-17
|
||||
// RENOVATE OWNS BOTH HALVES: Helm chart/dependency bumps AND per-service container image tags.
|
||||
// It opens PRs for everything; allowed low-risk classes auto-merge on green CI. The ADR-0009
|
||||
// split that gave image tags to Flux image-automation is formally closed — see
|
||||
// .gitea/workflows/renovate.yaml for the three measured reasons (it pushes commits rather than
|
||||
// opening PRs, so it bypasses the CLAUDE.md §1 gate; `main` is protected with
|
||||
// user_can_push:false and the flux-system GitRepository has no secretRef, so it cannot write
|
||||
// anyway; and enabling it would need an in-cluster git WRITE credential for the repo that
|
||||
// defines the cluster). image-automation has been commented out at
|
||||
// platform/kustomization.yaml:52 since it was authored and the cluster holds zero
|
||||
// ImagePolicy/ImageRepository/ImageUpdateAutomation objects, so nothing is being torn down.
|
||||
// CONSEQUENCE: the `docker` and `kubernetes` managers, previously disabled ONLY to avoid
|
||||
// fighting that controller, are now ENABLED — without them every plain `image: repo/name:tag`
|
||||
// line in this repo was maintained by nothing at all, which is how the oauth2-proxy auth
|
||||
// sidecars reached 7 minors behind the oauth2-proxy running in the same cluster. RUNNER:
|
||||
// .gitea/workflows/renovate.yaml (scheduled daily); platform/endpoint are set there, not here.
|
||||
"extends": ["config:recommended", ":dependencyDashboard", ":semanticCommits", "helpers:pinGitHubActionDigests"],
|
||||
"timezone": "America/New_York",
|
||||
"schedule": ["after 3am and before 6am"],
|
||||
"prConcurrentLimit": 5,
|
||||
"prHourlyLimit": 2,
|
||||
"commitMessagePrefix": "chore(deps):",
|
||||
"labels": ["dependencies", "automated-update"],
|
||||
"ignoreUnstable": true,
|
||||
"respectLatest": true,
|
||||
"enabledManagers": ["flux", "kustomize", "helmv3", "kubernetes", "regex", "github-actions"],
|
||||
"flux": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)platform/.+\\.ya?ml$/",
|
||||
"/(^|/)apps/.+\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
"kustomize": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)kustomization\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
// ADDED 2026-10-17. The `kubernetes` manager reads plain `image:` fields out of Kubernetes
|
||||
// manifests, and it is NOT auto-detected — it only looks at files matched here, which is why
|
||||
// this block is as broad as it is. Scope covers every place this repo writes a container image:
|
||||
// platform components, the rendered app waves, the ServiceDefinition catalog that generates
|
||||
// them, the runbooks web server and the pm-agent job template. `catalog/` matters as much as
|
||||
// `apps/`: apps/wave-*/*.yaml are GENERATED by catalog/render.py, so a bump applied only to the
|
||||
// rendered output is reverted by the next render and would also fail ci/render-golden.sh.
|
||||
// Renovate will raise BOTH, and both must land together.
|
||||
"kubernetes": {
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)platform/.+\\.ya?ml$/",
|
||||
"/(^|/)platform-config/.+\\.ya?ml$/",
|
||||
"/(^|/)apps/.+\\.ya?ml$/",
|
||||
"/(^|/)catalog/.+\\.ya?ml$/",
|
||||
"/(^|/)runbooks/serving/.+\\.ya?ml$/",
|
||||
"/(^|/)infra/agent/.+\\.ya?ml$/"
|
||||
]
|
||||
},
|
||||
"customManagers": [
|
||||
{
|
||||
"customType": "regex",
|
||||
"description": "Inline `# renovate: datasource=... depName=...` markers on pinned URLs / version literals (e.g. platform/kubevirt/kustomization.yaml release URLs, ci/versions.env, infra/devcontainer/versions.env).",
|
||||
"managerFilePatterns": [
|
||||
"/(^|/)kustomization\\.ya?ml$/",
|
||||
"/(^|/)versions\\.env$/"
|
||||
],
|
||||
"matchStrings": [
|
||||
"#\\s*renovate:\\s*datasource=(?<datasource>\\S+)\\s+depName=(?<depName>\\S+)(?:\\s+versioning=(?<versioning>\\S+))?\\s*\\n.*?(?<currentValue>v?[0-9]+\\.[0-9]+\\.[0-9]+)"
|
||||
],
|
||||
"versioningTemplate": "{{#if versioning}}{{{versioning}}}{{else}}semver{{/if}}"
|
||||
}
|
||||
],
|
||||
"packageRules": [
|
||||
{
|
||||
"description": "Auto-merge the allowed low-risk classes on green CI \u2014 no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
|
||||
"matchUpdateTypes": [
|
||||
"minor",
|
||||
"patch",
|
||||
"digest"
|
||||
],
|
||||
"automerge": true,
|
||||
"automergeType": "pr",
|
||||
"platformAutomerge": true
|
||||
},
|
||||
{
|
||||
"description": "MAJOR / potentially-breaking bumps require human review (docs/11 safety rail). Never auto-merged.",
|
||||
"matchUpdateTypes": [
|
||||
"major"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": [
|
||||
"needs-review",
|
||||
"breaking"
|
||||
]
|
||||
},
|
||||
{
|
||||
"description": "Keep the KubeVirt + CDI operator pair on a compatible track \u2014 grouped so they bump together (platform/kubevirt/kustomization.yaml).",
|
||||
"matchDepNames": [
|
||||
"kubevirt/kubevirt",
|
||||
"kubevirt/containerized-data-importer"
|
||||
],
|
||||
"groupName": "kubevirt-cdi"
|
||||
},
|
||||
{
|
||||
"description": "Pinned tool versions (ci/versions.env, infra/devcontainer/versions.env) are a deliberate change surface \u2014 label but do not auto-merge; a version bump there changes CI reproducibility.",
|
||||
"matchFileNames": [
|
||||
"ci/versions.env",
|
||||
"infra/devcontainer/versions.env"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": [
|
||||
"ci-tooling"
|
||||
]
|
||||
},
|
||||
{
|
||||
"description": "ADDED 2026-10-17. NEVER AUTO-MERGE THE BLAST-RADIUS COMPONENTS, not even a patch. The default rule above auto-merges minor/patch/digest on a green gate, which is right for an app image: a bad Sonarr bump inconveniences one pod. It is wrong for these five, because the gate is a STATIC check \u2014 it renders manifests and lints them, it does not boot a node, drain a volume or form an etcd quorum (vars.CLUSTER_AVAILABLE is unset, so L2/L3 conformance does not run in CI at all). So for anything whose failure mode is the CLUSTER rather than a service, a green gate is not evidence and a human must read the changelog. kured is here because it reboots control-plane nodes; longhorn because a storage-engine bump migrates live volume data; cilium because it is the CNI and a bad bump is a total partition (and its version is DUAL-PINNED in infra/ansible/roles/cilium/defaults/main.yml, which Renovate does not see, so the two would silently desync); cloudnative-pg because it drives Postgres switchovers; kube-prometheus-stack because ci/versions.env asserts an exact promtool/Prometheus lockstep and a chart bump that moves Prometheus fails the gate until that file moves with it.",
|
||||
"matchPackageNames": [
|
||||
"kured",
|
||||
"longhorn",
|
||||
"cilium",
|
||||
"cloudnative-pg",
|
||||
"kube-prometheus-stack"
|
||||
],
|
||||
"automerge": false,
|
||||
"addLabels": [
|
||||
"needs-review",
|
||||
"blast-radius"
|
||||
]
|
||||
}
|
||||
],
|
||||
// WHAT RENOVATE STILL CANNOT FIX, so nobody mistakes a working bot for full coverage. A
|
||||
// FLOATING tag has no version to compare, so Renovate skips it silently — it does not pin it
|
||||
// and does not warn. As of 2026-10-17 that is: plexinc/pms-docker:latest,
|
||||
// lscr.io/linuxserver/jackett:latest, pihole/pihole (NO TAG AT ALL), quay.io/keycloak/keycloak
|
||||
// (no tag), nextcloud:31-apache and docker:27-dind. Those must be pinned BY HAND to a concrete
|
||||
// version before any automation can maintain them, and until then they drift on every pull with
|
||||
// nothing recording what is running. Separately, the HelmRelease minor-locking ranges (9 of 12,
|
||||
// e.g. `>=88.0.0 <89.0.0` on kube-prometheus-stack) cap Renovate at patches inside a line it
|
||||
// has been told not to leave; raising those ceilings is deliberate per-component work and is
|
||||
// NOT part of enabling the bot. Both lists are in decisions/2026-10-17-automated-upgrades.md.
|
||||
// platform/kyverno/tests/** ADDED 2026-10-17: those Kyverno policy fixtures pin `alpine:latest`
|
||||
// and a `conformance-runner:latest`, which are test inputs whose whole point is to be a fixed
|
||||
// shape, not software to keep current. `conformance/**` was already ignored for the same
|
||||
// reason; this closes the matching hole that enabling the `kubernetes` manager would otherwise
|
||||
// have opened.
|
||||
"ignorePaths": [
|
||||
"**/*.example",
|
||||
"**/*.sops.yaml",
|
||||
"conformance/**",
|
||||
"platform/image-automation/per-service/**",
|
||||
"platform/kyverno/tests/**"
|
||||
],
|
||||
// The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were
|
||||
// BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation
|
||||
// over a running tag. That controller was never enabled and never will be here (see _comment),
|
||||
// so the only thing those two flags were achieving was leaving every plain `image:` tag in the
|
||||
// repo unmanaged. `docker` is left out of enabledManagers because this repo contains no
|
||||
// Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.
|
||||
"docker": {
|
||||
"enabled": true
|
||||
}
|
||||
}
|
||||
@@ -273,6 +273,84 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
|
||||
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
|
||||
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
|
||||
|
||||
## Triage — "the client reaches the LAN and cluster apps, but not the internet"
|
||||
|
||||
Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom
|
||||
almost always lives, and because the server side has been measured and is not the cause.
|
||||
|
||||
**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.**
|
||||
|
||||
| the profile says | meaning |
|
||||
|---|---|
|
||||
| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. |
|
||||
| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. |
|
||||
|
||||
`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit —
|
||||
`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client
|
||||
profile only**, so an already-issued profile keeps whatever it was rendered with until it is
|
||||
re-rendered and re-imported on the device. The two forms differ by one flag:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet>' # full tunnel
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet> --split' # LAN only
|
||||
```
|
||||
|
||||
Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into
|
||||
`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is
|
||||
editing `AllowedIPs` in the profile **on the device** — the server does not care.
|
||||
|
||||
**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the
|
||||
client connected, then watch the two counters that only forwarded traffic moves:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c "
|
||||
wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5;
|
||||
iptables -t nat -L POSTROUTING -n -v | head -5"'
|
||||
```
|
||||
|
||||
* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which
|
||||
node the router's `udp/51820` forward points at (the router is unreadable from the cluster).
|
||||
* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is
|
||||
upstream of the cluster (router forward, client, or carrier), not here.
|
||||
* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side
|
||||
failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there,
|
||||
which breaks internet while node-local cluster VIPs still work. It has not been observed.
|
||||
|
||||
**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly:
|
||||
|
||||
* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack
|
||||
client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel —
|
||||
so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4:
|
||||
`curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it,
|
||||
and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is
|
||||
blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it).
|
||||
* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified
|
||||
2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a
|
||||
blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on
|
||||
a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all.
|
||||
|
||||
**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the
|
||||
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain
|
||||
**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route
|
||||
`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the
|
||||
PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled
|
||||
`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the
|
||||
**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6).
|
||||
|
||||
## Triage — which L2 announcement lease a node holds does NOT matter
|
||||
|
||||
Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node
|
||||
could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`:
|
||||
that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh`
|
||||
for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer
|
||||
VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend
|
||||
is a node-local socket. Full measurements and the two misleading probes
|
||||
(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`.
|
||||
|
||||
`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a
|
||||
separate, pre-existing fault in that service and has nothing to do with the VPN.
|
||||
|
||||
## Revoking a device
|
||||
|
||||
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
|
||||
|
||||
Reference in new issue
Block a user