jakman85andClaude Opus 5
6a8953e288
Minimal prod delivery, authored directly on main (= prod's live revision d1a6628)
rather than by merging work/prod-standup-campaign. That branch carries the DNS apex
inversion, which would have set prod's BASE_DOMAIN to home.keithjackson.dev; prod
runs external-dns with policy: sync and txtOwnerId: prod, so it would have begun
rewriting apex records dev owns while the non-k3s prod still serves 443 there.
Operator sequencing is "dev migrates first". Nothing under clusters/ is touched
here at all, which is what makes this safe to land now.
Also deliberately EXCLUDED, because prod's live cluster-vars has no CNPG_BACKUP_PATH
key and StrictPostBuildSubstitutions would fail the whole tier: the R2 rework of
platform/velero/ and the CNPG destinationPath changes. Those land as
feat/r2-backup-prod together with the endpoint and bucket knobs.
WHAT THIS FIXES
1. Nothing was watching CNPG. All three prod databases have run since bootstrap with
ContinuousArchiving=False and firstRecoverabilityPoint=null, and no alert existed
anywhere. Because archive_mode=on and the archive command fails, Postgres refuses
to recycle WAL: keycloak-db holds every segment ever written (257 MiB on a 5 GiB
volume, ~6.8 MiB/h) — roughly 29 days to a disk-full PANIC on the cluster's SSO
database. Found by hand, 38h late. CNPGWALArchiveBacklog would have fired within
the hour, and evaluates to 3 firing series on prod today.
Rules key on cnpg_collector_* rather than cnpg_pg_stat_archiver_*, because
nextcloud-db's exporter cannot log in to its own database (permission denied for
database "nextcloud", both clusters) so the pg_stat_archiver family does not exist
for it — the obvious metric would have been blind to one database in three.
2. No PVC outside namespace=monitoring had any fill alert. New rules are partitioned
by namespace so they cannot double-fire with MonitoringVolumeFillingUp, pinned to
job="kubelet" (the apiserver copy of the same series reports namespace=default),
and guarded by inodes > 0 — block-mode PVCs such as unifi-rootdisk report
available=0 forever and would otherwise have fired critical permanently.
3. VPNTunnelDown could never clear on a cluster with no gateway node; it has been
firing critical on prod since 2026-09-05. Now guarded so prod self-arms the moment
a node carries the egress-gateway label, with dev's behaviour unchanged today.
4. The Keycloak realm import fails OPEN: an unresolved placeholder is stored
LITERALLY and the import still reports success, then sits at Done=True and never
re-runs. On 2026-09-06 all 12 prod OIDC clients held the literal
$(env:..._CLIENT_SECRET) string and every SSO login failed, while every signal was
green. The CronJob is the level-triggered correction; two alerts make its silence
observable. Repaired live already — this stops it recurring.
Also: the R-NET-04 conformance scan descended into nested agent worktrees under
.claude/, counting 4 egress policies where 1 is authored.
Co-Authored-By: Claude Opus 5 <[email protected]>
95 lines
6.1 KiB
YAML
95 lines
6.1 KiB
YAML
# =============================================================================
|
|
# L0 STATIC — two egress paths, and the rule set that selects between them (R-NET-04) [EXECUTABLE]
|
|
# Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
|
|
# Spec: docs/00 R-NET-04 ("The platform MUST support **multiple egress paths** selected by
|
|
# routing rules ... realized as two paths — the local `internet` path and the
|
|
# in-cluster `vpn` path — not multi-ISP routing"), elaborated by docs/03 §"Egress
|
|
# routing: two paths, one physical WAN": "the `internet` path is simply the default
|
|
# route out"; "**Default is the WAN; the VPN is strictly opt-in.** Every pod — and
|
|
# the nodes themselves — egress the local/private WAN by default and **never** the
|
|
# VPN. Only services labeled `egress-class=vpn` are steered to the tunnel, and the
|
|
# kill-switch blackholes **only** that class."
|
|
#
|
|
# WHAT THIS ADDS OVER R-NET-05/07 (already covered, and NOT re-tagged here)
|
|
# ------------------------------------------------------------------------
|
|
# R-NET-05/07 are about a SERVICE — it declares a class, enforcement is outside it, the class
|
|
# flips via GitOps — and are proven by tests/l1-rendering/egress-class-render and
|
|
# tests/l2-behavioral/egress-class. R-NET-04 is about the PLATFORM's PATH SET and the RULES
|
|
# that select among them, which nothing else asserts:
|
|
# * exactly TWO paths, the same closed set in the schema, the admission policy and the
|
|
# renderer (a third value would be a path nothing routes; "multi-ISP" is out of scope);
|
|
# * selection is by a ROUTING RULE and the rule set is TOTAL AND DISJOINT — exactly one
|
|
# Cilium steering rule, selecting only `egress-class=vpn`, and on the gateway node one
|
|
# `ip rule` scoped to the tunnel address. NOTHING selects the `internet` class, because
|
|
# absence of a matching rule IS the default path;
|
|
# * the two paths COEXIST per DESTINATION: private space is excluded from the tunnel, so a
|
|
# vpn-class pod's cluster/LAN traffic still takes the local path;
|
|
# * the vpn path never becomes the DEFAULT — its default route lives in a dedicated routing
|
|
# table, `route-noexec` is set and a pushed `redirect-gateway` is ignored;
|
|
# * the kill-switch blackholes ONLY that class — the nft chain's policy is accept and every
|
|
# DROP is scoped to the vpn class's own source set/fwmark, so `internet` fails OPEN.
|
|
# (R-NET-06's opposite claim, that `vpn` fails CLOSED, is L3's vpn-killswitch.)
|
|
#
|
|
# WHY L0: every clause above is a statement about ALL rules — including rules that must be
|
|
# ABSENT (nothing selects the default path; no unscoped rule; no unqualified drop; no default
|
|
# route in the main table). A cluster can show which path two pods took; it cannot show that
|
|
# no third rule exists or that the vocabulary is closed.
|
|
#
|
|
# ENVIRONMENT-AGNOSTIC: no address appears. The private ranges the tunnel must exclude are
|
|
# checked as CONTAINMENT of the addresses each environment's own Flux overlay declares (pod
|
|
# CIDR, Gateway VIP, LB pool, API host), so a new environment on a different LAN is checked
|
|
# against its own values.
|
|
#
|
|
# GIVEN the repository as git tracks it,
|
|
# WHEN the egress vocabulary, every CiliumEgressGatewayPolicy, the gateway node's policy
|
|
# routing / OpenVPN / nftables templates, and every environment's overlay addresses are
|
|
# parsed and cross-checked,
|
|
# THEN there are exactly two paths with `internet` as the default; exactly one steering rule
|
|
# selects exactly the vpn class by label; the gateway is selected by node label with a
|
|
# named egress interface; all RFC1918 space and every address any environment assigns
|
|
# itself is excluded from the tunnel; the tunnel's default route and its fail-closed
|
|
# blackhole live only in a dedicated routing table behind a scoped `ip rule`; OpenVPN
|
|
# cannot take the main table; and no nftables DROP is unscoped.
|
|
#
|
|
# STATUS: EXECUTABLE. `check.sh` is a pure offline assertion script (git tree + PyYAML; no
|
|
# cluster, no network); conformance/run.sh --layer l0 and ci/l0.sh run it directly, and this
|
|
# Chainsaw Test wraps the SAME script.
|
|
#
|
|
# KNOWN, DELIBERATE GAPS (enumerated in check.sh's header; this test claims none of them):
|
|
# * BEHAVIOUR. That an unlabelled pod really takes the WAN path is asserted at L2 by
|
|
# tests/l2-behavioral/egress-default-path (same requirement label); the two labelled
|
|
# classes are R-NET-05's L2 test; the kill-switch firing is R-NET-06's L3 test.
|
|
# * The RUNNING gateway node's routing tables / nftables ruleset. This reads the templates
|
|
# that generate them; a hand-added rule on the box is invisible here.
|
|
# * PHYSICAL MULTI-WAN's absence is NOT asserted — it is out of scope per OQ-1 and no
|
|
# honest repository signal distinguishes "one WAN, correctly" from "one WAN, by default".
|
|
# * IPv6 egress.
|
|
# =============================================================================
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: egress-paths-selected-by-rules
|
|
labels:
|
|
conformance.home.k3s/R-NET-04: "true"
|
|
annotations:
|
|
conformance.home.k3s/requirements: "R-NET-04"
|
|
conformance.home.k3s/layer: "L0"
|
|
conformance.home.k3s/safety-critical: "false"
|
|
conformance.home.k3s/executable: "true"
|
|
conformance.home.k3s/intent: >-
|
|
Given the tracked tree, when the egress vocabulary, every steering rule and the gateway
|
|
node's policy-routing/OpenVPN/nftables templates are parsed, then the platform offers
|
|
exactly two egress paths with the WAN as the default, exactly one rule selects exactly
|
|
the vpn class, local destinations stay on the local path in every environment, the
|
|
tunnel never becomes the default route, and nothing unscoped can blackhole the default
|
|
path.
|
|
spec:
|
|
steps:
|
|
- name: then-two-paths-exist-and-the-rules-select-between-them-totally-and-disjointly
|
|
description: >-
|
|
Run the offline assertion script over the tracked tree (same script ci/l0.sh and
|
|
conformance/run.sh --layer l0 execute).
|
|
try:
|
|
- script:
|
|
content: bash ./check.sh
|