jakman85andClaude Opus 5
4656b07178
A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
holds itself: the announcer does not answer its own ARP request, so on the
lease holder the neighbour entry settles at FAILED and packets to that VIP die
before leaving the node. Measured on prod 2026-10-08, symmetric across 3 nodes
x 5 VIPs -- each node reached every VIP EXCEPT the ones it alone announced.
platform/vpn-ingress is uniquely exposed: it runs WireGuard on hostNetwork and
MASQUERADEs remote clients to the node's own address, so a VPN client takes
the node's LAN path and inherits its ARP failures. The router's udp/51820
forward pointed at k3s-prod-1, which held the shared Gateway lease, so all 16
Gateway hostnames were dead over the VPN -- while PiHole DNS kept answering
(its VIP has two Services and k3s-prod-2 held the other lease). The tunnel
handshook, names resolved, every app hung, and nothing reported it.
on k3s-prod-1: ip neigh show 192.168.2.20 -> FAILED (its own VIP)
on prod-2/3: ip neigh show 192.168.2.20 -> lladdr ... (prod-1), 443 OK
Mark the node and exclude it from announcing:
* node_baseline labels the forwarded node (node_baseline_vpn_ingress_node,
k3s-prod-1 on prod) homelab.keithjackson.dev/vpn-ingress=true, and removes
the label from any node that no longer matches so moving it MOVES the label.
* l2-announcement-policy.yaml excludes that label, so the node ARP-resolves
every VIP. Fixes all VIPs for VPN clients, not just the one that collided.
Cost: announcement candidacy drops from 3 nodes to 2.
* VpnIngressNodeAnnouncesL2VIP joins kube_lease_owner against kube_node_labels
(label added to the kube-state-metrics allowlist), with
VpnIngressNodeLabelMissing as its watchdog -- NotIn on an absent key matches,
so a missing label would silently re-arm the whole failure AND leave the
critical alert inert. Four promtool cases, including that inertness.
A node LABEL rather than a cluster-vars knob because the alert must read it
too, and extract_rules.py rightly refuses a ${...} placeholder in an alert
expr -- the rule unit-tested would not be the rule deployed. One source of
truth, and it touches no sensitive path.
Runbook/docs: step 3 of vpn-remote-access.md said "pick any node" and
"failover is this one line". Both false; the first caused this.
UNAPPLIED and unproven end to end; the Ansible tasks have never run. See the
handoff entry's third section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
decisions/ — the handoff log, one file per entry
This directory is the decision log. ../DECISIONS-LOG.md is the frozen archive of
everything written before 2026-09-29 and is never edited again (see "The archive is
frozen" below).
Why this exists
DECISIONS-LOG.md was append-only in a single file, and every agent appended to its
tail. Two branches that both append hit the same last hunk, so git cannot
auto-merge them: the second one to be reviewed conflicts, and so does the third, and
the fourth. Measured on 2026-09-29: four of the five open PRs were unmergeable, and
DECISIONS-LOG.md was the only conflicting file in all four:
#22 fix/crafty-oauth2-proxy-signin CONFLICT (content): DECISIONS-LOG.md
#20 feat/conformance-job-template CONFLICT (content): DECISIONS-LOG.md
#19 feat/conformance-trigger-kubeconfig CONFLICT (content): DECISIONS-LOG.md
#18 feat/conformance-runner-image CONFLICT (content): DECISIONS-LOG.md
#17 feat/ci-is-the-approver clean (it touched no log entry)
Nothing was wrong with any of those changes. The log's storage format serialised work that was otherwise parallel, and it got worse with every concurrent branch: N branches produce N-1 rebases, each of which invalidates any gate run that preceded it.
The fix is the obvious one for concurrent appenders: stop sharing a file. Each
handoff is its own file. Two agents writing handoffs touch disjoint paths, so there is
nothing for git to merge and nothing to conflict. This scales to any number of parallel
branches with zero coordination — which is the point, since the agents are deliberately
isolated from each other in separate worktrees (required by CLAUDE.md §0) and cannot
coordinate.
There is deliberately no index file and no counter. An index would be a shared
mutable file — exactly the thing that just failed — and a sequence number would have
two agents racing for 0043. The directory listing is the index.
Writing an entry
One file per handoff, named:
decisions/YYYY-MM-DD-<slug>.md
<slug> is lowercase [a-z0-9-]. Use your branch name's last segment — it is
already unique per unit of work, which is precisely the property the filename needs.
A branch fix/crafty-oauth2-proxy-signin handed off on 2026-09-29 writes:
decisions/2026-09-29-crafty-oauth2-proxy-signin.md
The file is YAML frontmatter followed by three required sections:
---
date: 2026-09-29
branch: fix/crafty-oauth2-proxy-signin
title: The Keycloak gate works — Crafty's own login was the wall
---
## What changed
Prose. What you actually did, and why.
## What I verified
The measurements, with their numbers. Quote the field you read, not the prose you
believed. Name the gate result and its exit code.
## What I did NOT verify
Everything else. An honest "UNVERIFIED" is worth more than an optimistic "green" —
this repo has been burned by the difference more than once.
All three headings are required and enforced (ci/decisions-log.sh). They are
CLAUDE.md §7's contract; making them a checked structure is what stops an entry
degrading into "done ✅".
Reading the log
The whole log, oldest first:
ls decisions/[0-9]*.md | sort
The current PM handoff — the newest entry. Ties within a day are broken by what landed
on main last, which is the honest answer, so ask git rather than the filename:
git log --diff-filter=A --format='%H' --name-only -- 'decisions/[0-9]*.md' \
| grep '^decisions/' | head -1
For a quick skim of who said what:
grep -H '^title:' decisions/[0-9]*.md | sort
The archive is frozen
DECISIONS-LOG.md is read-only history. Its SHA-256 is pinned in
decisions/ARCHIVE.sha256 and ci/decisions-log.sh fails the merge gate if the file
changes. That is not ceremony: without it, the next agent to read runbooks/ or an old
CLAUDE.md habit appends to the tail again and re-creates the conflict for everybody
else, silently, one PR at a time.
If the archive genuinely must change (a leaked credential to redact, say), that is a
deliberate, reviewed act: change it, run sha256sum DECISIONS-LOG.md > decisions/ARCHIVE.sha256,
and say in your handoff entry why. Re-pinning is visible in the diff. Appending is not.