Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 jakman85
|
0913ea9450 | ||
 jakman85andClaude Opus 5
|
f5353cbd30 |
No files matched your search
@@ -1,311 +0,0 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: feat/keycloak-passkeys
|
||||
title: Passkeys declared for the homelab realm — and the realm import will not apply them
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`platform/keycloak/keycloak-realm-import.yaml` now declares passkey (WebAuthn Passwordless)
|
||||
login for the `homelab` realm, in two parts:
|
||||
|
||||
1. **The WebAuthn Passwordless policy.** Ten `webAuthnPolicyPasswordless*` fields on the realm:
|
||||
RP ID `${BASE_DOMAIN}`, RP entity name `Homelab SSO`, signature algorithms `[ES256, RS256]`,
|
||||
`requireResidentKey: "Yes"`, `userVerificationRequirement: required`,
|
||||
`authenticatorAttachment: "not specified"`, `attestationConveyancePreference: none`,
|
||||
`acceptableAaguids: []`, `createTimeout: 300`, `avoidSameAuthenticatorRegister: true`.
|
||||
Deliberately the **passwordless** family only; the two-factor `webAuthnPolicy*` family is left
|
||||
at Keycloak's defaults because nothing in this realm uses `webauthn-authenticator`.
|
||||
|
||||
2. **The browser flow.** `browser-mfa` keeps its alias and stays bound as `browserFlow`; its
|
||||
interior is restructured so a passkey is an **ALTERNATIVE** beside the password form:
|
||||
|
||||
```
|
||||
browser-mfa (top level, unchanged)
|
||||
browser-mfa-forms ALTERNATIVE
|
||||
auth-username-form REQUIRED <- was auth-username-password-form
|
||||
browser-mfa-passkey-or-password REQUIRED <- NEW sub-flow
|
||||
webauthn-authenticator-passwordless ALTERNATIVE <- NEW: the passkey
|
||||
browser-mfa-password-and-otp ALTERNATIVE <- NEW sub-flow
|
||||
auth-password-form REQUIRED <- was half of the combined form
|
||||
browser-mfa-conditional-otp CONDITIONAL (unchanged, relocated intact)
|
||||
```
|
||||
|
||||
The design decisions, with their reasoning, are written into the manifest comments in this repo's
|
||||
usual density. The ones that matter:
|
||||
|
||||
* **Additive, not replacing. Enrolment optional, not forced.** This IdP gates Grafana, Gitea,
|
||||
Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the \*arr stack *and* `kubectl` via the
|
||||
`kubernetes` OIDC client. A mandatory passkey flow that misfires is a total lock-out from the
|
||||
platform and the cluster API at once, leaving only the Keycloak bootstrap admin and the
|
||||
break-glass cert kubeconfig. The gain from forcing it (one human stops typing a password) does
|
||||
not price that risk.
|
||||
* **No `requiredActions` block was added, on purpose.** Declaring one would have been the obvious
|
||||
way to add `webauthn-register-passwordless`, and it would have been actively dangerous — see
|
||||
"What I verified".
|
||||
* **RP ID is `${BASE_DOMAIN}`, not `keycloak.${BASE_DOMAIN}`.** A passkey's RP ID is fixed at
|
||||
enrolment and can never be re-keyed, so it must not be bound to a name that can move. This realm
|
||||
already declares an `auth.${BASE_DOMAIN}` redirect URI, and this repo already owns a reconciler
|
||||
built because a hostname move silently failed to reach the realm. Residual risk, unavoidable:
|
||||
changing `BASE_DOMAIN` itself invalidates every enrolled passkey.
|
||||
* **A passkey login is not also asked for TOTP.** `browser-mfa-conditional-otp` hangs off the
|
||||
password branch only. The justification is `userVerificationRequirement: required` — the
|
||||
ceremony already proves possession plus a verified human, phishing-resistantly and origin-bound.
|
||||
Stacking a replayable shared secret on top of that buys friction. The manifest names the exact
|
||||
one-line edit to reverse this decision.
|
||||
* **Login becomes two screens** (username, then password) for every user, passkey or not, because
|
||||
the server must know who is logging in before it can decide whether a passkey exists. Accepted
|
||||
cost; it is the enabling change.
|
||||
* **Usernameless login deliberately not used**, although Keycloak 26.0.6 supports it — see "What I
|
||||
verified" for the measured reason it would have made every login open on a passkey prompt.
|
||||
|
||||
`platform/keycloak/README.md` gains a `## Passkeys` section: the decision table, the fact that
|
||||
this does not reach the live realm, the two operator paths that would apply it, and the four-step
|
||||
browser verification nobody has run.
|
||||
|
||||
No secret, key, credential or AAGUID was minted or committed. No file under `clusters/**`,
|
||||
`conformance/**`, `ci/**`, `platform/kyverno/**`, `**/rbac.yaml` or `**/*.sops.yaml` was touched,
|
||||
so this should not trip `ci/approval-policy.sh`'s sensitive-path review requirement.
|
||||
|
||||
## What I verified
|
||||
|
||||
**The live realm import has already completed and will never re-run.** Measured on prod,
|
||||
2026-10-17:
|
||||
|
||||
```
|
||||
$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
|
||||
{"conditions":[{"message":"","status":"True","type":"Done"},
|
||||
{"message":"","status":"False","type":"Started"},
|
||||
{"message":"","status":"False","type":"HasErrors"}]}
|
||||
```
|
||||
|
||||
`Done=True` / `Started=False`, exactly the one-shot state `platform/keycloak/README.md` already
|
||||
documents for client secrets and client URLs. **So the change in this PR has zero effect on how
|
||||
anyone logs in today**, and the client-secret/client-URL reconciler does not and cannot close the
|
||||
gap: its hard endpoint allow-list makes `/authentication` unreachable by design, and realm-level
|
||||
policy fields are not client fields.
|
||||
|
||||
**The CRD accepts every field I used.** Measured against the live CRD (not the upstream docs):
|
||||
`kubectl --context k3s-prod get crd keycloakrealmimports.k8s.keycloak.org` lists all twelve
|
||||
`webAuthnPolicyPasswordless*` properties with the types I wrote (`RequireResidentKey`,
|
||||
`RpEntityName`, `RpId`, `UserVerificationRequirement`, `AttestationConveyancePreference`,
|
||||
`AuthenticatorAttachment` as `string`; `SignatureAlgorithms`, `AcceptableAaguids`, `ExtraOrigins`
|
||||
as `array/string`; `CreateTimeout` as `integer`; `AvoidSameAuthenticatorRegister` as `boolean`),
|
||||
plus `authenticationFlows`, `authenticatorConfig` and `requiredActions`. The `spec.realm` schema
|
||||
has **no** `x-kubernetes-preserve-unknown-fields`, so a misspelled field would have been pruned
|
||||
silently rather than rejected — which is why this was checked rather than assumed.
|
||||
|
||||
**Running Keycloak is 26.0.6.** `quay.io/keycloak/keycloak:26.0.6` on `keycloak-0`/`keycloak-1`,
|
||||
operator `quay.io/keycloak/keycloak-operator:26.0.6`. Every source claim below is read from the
|
||||
`26.0.6` tag, not from memory or from docs.
|
||||
|
||||
**Every provider ID is quoted from upstream source, not guessed.**
|
||||
|
||||
| String I used | Where it is defined at tag 26.0.6 |
|
||||
|---|---|
|
||||
| `webauthn-authenticator-passwordless` | `WebAuthnPasswordlessAuthenticatorFactory.PROVIDER_ID`, display "WebAuthn Passwordless Authenticator" |
|
||||
| `auth-username-form` | `UsernameFormFactory.PROVIDER_ID`, display "Username Form" |
|
||||
| `auth-password-form` | `PasswordFormFactory.PROVIDER_ID`, display "Password Form" |
|
||||
| `webauthn-register-passwordless` | `WebAuthnPasswordlessRegisterFactory.PROVIDER_ID` (referenced in comments only) |
|
||||
|
||||
**`requiredActions` is REPLACE, not merge — measured, and it is why I did not declare one.**
|
||||
`model/storage-private/.../DefaultExportImportManager.java` lines 327-338:
|
||||
|
||||
```java
|
||||
if (rep.getRequiredActions() != null) {
|
||||
for (RequiredActionProviderRepresentation action : rep.getRequiredActions()) { ... }
|
||||
DefaultRequiredActions.addDeleteAccountAction(newRealm);
|
||||
} else {
|
||||
DefaultRequiredActions.addActions(newRealm);
|
||||
}
|
||||
```
|
||||
|
||||
Declaring one required action therefore skips the entire default set. `RealmManager` is no
|
||||
backstop: `setupRequiredActions` is `if (realm.getRequiredActionProvidersStream().count() == 0)`,
|
||||
and the count is already non-zero by then (`RepresentationToModel.importRealm` runs at
|
||||
`RealmManager.java:585`, the backstop at `:612`). A realm built that way would have had the passkey
|
||||
action plus `delete_account` and nothing else — **no `CONFIGURE_TOTP`**, which would leave
|
||||
`auth-otp-form` demanding a TOTP that no platform-admin could ever enrol: a hard lock-out
|
||||
delivered by a change whose purpose was to *add* a login method.
|
||||
|
||||
**And declaring it would have bought nothing.** `DefaultRequiredActions.addWebAuthnPasswordlessRegisterAction`
|
||||
registers `webauthn-register-passwordless` with `setEnabled(true)`, `setDefaultAction(false)`,
|
||||
priority 80, gated on `Profile.Feature.WEB_AUTHN` — which is `Type.DEFAULT` in
|
||||
`common/.../Profile.java:74`, i.e. on unless explicitly disabled. `enabled=true` /
|
||||
`defaultAction=false` *is* the additive-and-optional posture this change wants.
|
||||
|
||||
**`authenticationFlows` has the opposite semantics, which is why declaring flows is safe.**
|
||||
`importAuthenticationFlows` adds the declared flows, then backfills each unbound binding
|
||||
(browser, registration, direct grant, reset credentials, client auth, first broker login, docker
|
||||
auth) from the built-in defaults individually (`DefaultExportImportManager.java` ~1320-1410).
|
||||
|
||||
**The graceful degradation to password is measured from the flow engine, not assumed.** For a user
|
||||
with no passkey: `WebAuthnPasswordlessAuthenticator.requiresUser()` returns `false`, so
|
||||
`DefaultAuthenticationFlow.processSingleFlowExecutionModel` takes the `else` branch (`daf:456-462`),
|
||||
whose four conditions all hold — a user is set (from `auth-username-form`), `configuredFor` is
|
||||
false, `WebAuthnPasswordlessAuthenticatorFactory.isUserSetupAllowed()` is `false`, and
|
||||
`WebAuthnAuthenticator implements Authenticator, CredentialValidator<...>` — so it raises
|
||||
`CREDENTIAL_SETUP_REQUIRED`. The ALTERNATIVE loop (`daf:289-302`) catches exactly that, marks the
|
||||
execution `ATTEMPTED` and proceeds to the next alternative, with the upstream comment *"the user,
|
||||
since he has alternatives, should be able to go to another alternative and continue"*. Hence: no
|
||||
passkey, no passkey screen, straight to the password form.
|
||||
|
||||
**Why usernameless was rejected, measured.** The `configuredFor` guard sits inside the
|
||||
`requiresUser()` branch. With `requiresUser()` false and no user yet in context, that guard is
|
||||
skipped and the engine just calls `authenticate(context)` — so a passkey execution placed before
|
||||
the username form is offered to *everyone*, including users who will never own one.
|
||||
|
||||
**The policy sentinels are read from the shipped code, and one of my initial assumptions was
|
||||
wrong.** `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED = "not specified"` is the
|
||||
realm-representation sentinel — but `WebAuthnConstants.OPTION_NOT_SPECIFIED = ""` (empty string) is
|
||||
the WebAuthn-API-level one. I checked both because they disagree. The login theme's
|
||||
`webauthnRegister.js` at 26.0.6 settles the behaviour:
|
||||
|
||||
```js
|
||||
if (input.requireResidentKey !== 'not specified') {
|
||||
if (input.requireResidentKey === 'Yes') { authenticatorSelection.requireResidentKey = true; }
|
||||
else { authenticatorSelection.requireResidentKey = false; }
|
||||
}
|
||||
...
|
||||
if (input.createTimeout !== 0) { publicKey.timeout = input.createTimeout * 1000; }
|
||||
rp: {id: input.rpId, name: input.rpEntityName}
|
||||
```
|
||||
|
||||
So `"Yes"` is a literal string compare (any other non-sentinel value means *false*), `"not
|
||||
specified"` is compared literally for four fields, and `createTimeout` is **seconds** with `0`
|
||||
meaning "send no timeout". It also shows 26.0.6 emits only the legacy `requireResidentKey` member,
|
||||
never `residentKey: "required"` — the client maps that per WebAuthn L2, but the policy cannot
|
||||
express `residentKey: "preferred"` at all.
|
||||
|
||||
`[ES256, RS256]` is exactly `WebAuthnPolicy.DEFAULT_POLICY` /
|
||||
`Constants.DEFAULT_WEBAUTHN_POLICY_SIGNATURE_ALGORITHMS` — declared explicitly, not changed.
|
||||
`getWebAuthnPolicyPasswordless` was read field by field to confirm `[]` for `acceptableAaguids`
|
||||
means "no restriction" and that `RpEntityName` must be non-empty or the **update** path
|
||||
(`DefaultExportImportManager.java:842`) drops the whole policy silently.
|
||||
|
||||
**The manifest parses and renders.** `yaml.safe_load` round-trips the file and the flow graph reads
|
||||
back exactly as designed (five flows, the executions and priorities above, `browserFlow:
|
||||
browser-mfa`, `requiredActions` absent). `python3 ci/render-keycloak-client-urls.py --check` →
|
||||
`client-urls.yaml is in sync`, rc 0 (no client URL changed, so no regeneration was needed).
|
||||
`yamllint -c ci/yamllint.yaml platform/keycloak/keycloak-realm-import.yaml` → rc 0; no line exceeds
|
||||
the 160-column ceiling.
|
||||
|
||||
**`browser-mfa` keeps its alias deliberately.** `conformance/tests/l2-behavioral/keycloak-config-as-code`
|
||||
asserts `kubectl get keycloakrealmimport homelab -o jsonpath={.spec.realm.browserFlow} | grep -q
|
||||
'browser-mfa'`. Renaming the top-level flow would have failed that test.
|
||||
|
||||
**Envsubst safety.** `platform-config`'s Flux Kustomization declares
|
||||
`postBuild.substituteFrom: cluster-vars` (`clusters/base/platform-config.yaml:26-29`), so this file
|
||||
is substituted. The only brace token I added is `${BASE_DOMAIN}`, which is declared at
|
||||
`clusters/base/cluster-vars.yaml:42` (`home.keithjackson.dev`). Every comment I wrote avoids braces
|
||||
entirely — the RP-ID discussion says `keycloak.<BASE_DOMAIN>` with angle brackets precisely so an
|
||||
unparseable token cannot fail the whole Kustomization.
|
||||
|
||||
**Gate result: rc 3, zero failures, `gitea-settings` the only stage the gate counts as not-run** —
|
||||
the documented honest best local outcome. Run in the pinned toolbox from this worktree with all
|
||||
three §3 workarounds (`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main
|
||||
checkout's `.git` and the sibling spec repo with `SPEC_REPO=/home-network-k3s`):
|
||||
|
||||
```
|
||||
[ ok ] stage spec-immutability: pass <- NOT skipped; both shim mounts worked, R-TEST-06 covered
|
||||
[ ok ] stage decisions-log: pass <- archive frozen, 48 entries well-formed
|
||||
[ ok ] stage lint: pass
|
||||
[ ok ] stage l0: pass <- incl. flux-postbuild-substitution + keycloak-client-secret-reconciler
|
||||
[ ok ] stage render-golden: pass
|
||||
[ ok ] stage l1: pass
|
||||
[ ok ] stage approval-policy: pass <- no sensitive path touched, so no non-author review needed
|
||||
[skip] stage gitea-settings: SKIPPED <- needs SETTINGS_API_TOKEN, CI-only; expected
|
||||
[skip] stage conformance: SKIPPED <- cluster-gated L2/L3; expected
|
||||
MERGE GATE: UNDETERMINED — Stages that did not run: gitea-settings
|
||||
```
|
||||
|
||||
No lint finding and no check failure names either file I changed. Touched requirements resolved to
|
||||
`R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06`; coverage 51/51, 0 gaps.
|
||||
|
||||
**A note on how that number was obtained, because the first attempt was worthless.** My first gate
|
||||
run returned the same rc 3 and the same seven passes — against a tree that did not contain this
|
||||
change at all. The gate's `tracked_files()` reads **git**, and my edits were still uncommitted, so
|
||||
every stage gated `origin/main`. The tell was in the log and I nearly missed it: the
|
||||
touched-requirement subset reported UNDETERMINED with *"check out the feature branch you mean to
|
||||
gate (not the base branch)"*, and `decisions-log` counted **47** entries rather than 48. **A green
|
||||
gate run on an uncommitted worktree is a gate run on the base branch.** The numbers above are from
|
||||
the second run, after committing and rebasing.
|
||||
|
||||
**The base moved under me mid-run.** `origin/main` advanced from `9a41a8e` to `1a0b208` (PR #70,
|
||||
oauth2-proxy OIDC startup race) while I was working. This branch is rebased onto `1a0b208`, so the
|
||||
gate above ran on the base CI will actually merge into. The rebase was clean — PR #70 touched
|
||||
`platform/oauth2-proxy/helmrelease.yaml` and its own decisions entry, disjoint from these files.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
**This change is unapplied. It changes nothing about the live realm, and I did not attempt to make
|
||||
it.** `Done=True` / `Started=False` was measured, not inferred, and I ran no mutating `kubectl`,
|
||||
no `flux reconcile`, no admin-API call and no `kcadm`. Whether the operator's eventual
|
||||
admin-console reconstruction of this flow matches what is declared here is, by construction,
|
||||
unverifiable from git.
|
||||
|
||||
**I never enrolled a passkey and never pointed a browser at any of this.** Nothing here has been
|
||||
exercised by a real WebAuthn ceremony on any device. Specifically unverified:
|
||||
|
||||
* **That the RP ID works.** `home.keithjackson.dev` is a registrable-domain suffix of
|
||||
`keycloak.home.keithjackson.dev`, so it *should* satisfy the browser's check — but I verified
|
||||
that by reading the WebAuthn rule, not by watching a browser accept it. A wrong RP ID fails
|
||||
**silently, in the browser, before Keycloak is involved**, with no server-side log line. If
|
||||
enrolment produces no OS prompt and no error, this is the first suspect.
|
||||
* **That `"not specified"` is the right literal for `authenticatorAttachment` in a realm import.**
|
||||
I traced it to `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED` and to the theme's string
|
||||
compare, but I have not seen a realm import accept it. Omitting the field would default to the
|
||||
same value; if an import ever rejects the literal, drop the line.
|
||||
* **That `createTimeout: 300` behaves.** Seconds-vs-milliseconds is read from the theme JS. The
|
||||
claim that 300 is long enough for the cross-device QR/hybrid flow, and that Chrome clamps rather
|
||||
than rejects it, is reasoning about browser behaviour, not a measurement.
|
||||
* **Whether a passkey user can still reach the password form.** I argued it from
|
||||
`createAuthenticationSelectionList` producing a "Try another way" option, and I noted that for a
|
||||
user holding both credentials the *first* screen is chosen by the user's own credential ordering
|
||||
rather than by my priorities. Neither was observed.
|
||||
* **The two-screen login regression for ordinary users.** `auth-username-form` +
|
||||
`auth-password-form` replacing `auth-username-password-form` is a visible change for every user
|
||||
including non-passkey family accounts, and nobody has logged in through it.
|
||||
* **That conditional OTP for platform-admins still fires on the password branch.** The sub-flow and
|
||||
its `conditional-otp-platform-admin` config are byte-identical and merely relocated one level
|
||||
deeper, but "identical YAML" is not "still enforced at runtime". This is the highest-value
|
||||
regression check and it is untested. The previous breakage of this exact step
|
||||
(`condUserRole` vs a non-existent property key) 403'd every platform-admin login.
|
||||
* **That a platform-admin authenticating by passkey is not challenged for TOTP.** That is the
|
||||
intended behaviour of the shape I wrote; it is unobserved, and it is also a deliberate
|
||||
relaxation of R-IDP-04 as previously implemented, which the operator may want to reverse.
|
||||
|
||||
**Whether the realm import reconciles into an existing realm: measured as NO for the completed
|
||||
one-shot, but I did not test a re-import.** I did not delete and recreate the
|
||||
`KeycloakRealmImport`, so I cannot say what a re-run would do to the existing `homelab` realm —
|
||||
whether it would merge the new policy and flows, duplicate flow aliases, or fail. The 2026-09-06
|
||||
total SSO outage documented in `platform/keycloak/README.md` is the reason I did not experiment.
|
||||
**Treat the re-import path in the README as untested advice, not a runbook.**
|
||||
|
||||
**Nothing was proved about the live realm's current state.** I did not read the realm's existing
|
||||
required actions, its existing WebAuthn policies, or its live flow executions — doing so needs the
|
||||
admin credential and I did not decrypt or use it. So I cannot confirm that the live realm's
|
||||
required-action set actually contains `webauthn-register-passwordless` today; I can only show that
|
||||
Keycloak 26.0.6's defaults register it and that this repo's import never declared any required
|
||||
actions to suppress them. **That inference has not been checked against the realm.**
|
||||
|
||||
**L2/L3 conformance did not run** (no cluster from the gate container; `vars.CLUSTER_AVAILABLE` is
|
||||
unset in CI anyway), so the existing `keycloak-config-as-code` behavioural test did not re-execute
|
||||
against this change. I reasoned about its assertion by reading it.
|
||||
|
||||
**No new conformance test was added.** Nothing in `conformance/` asserts the passkey policy or the
|
||||
new flow shape, so a future edit can silently remove either and no gate stage will notice. That is
|
||||
a real gap this PR leaves open; closing it would touch `conformance/**`, which is a sensitive path
|
||||
needing non-author review.
|
||||
|
||||
**The quoted gate run predates this file's own gate-result paragraph.** It gated commit `207f5db`;
|
||||
the gate numbers and the rebase note above were then written into this entry, so the pushed tree
|
||||
differs from the gated one by this handoff's prose only — no manifest, no check, no generated file.
|
||||
`decisions-log` is the only stage that reads this file, and it validates structure (frontmatter,
|
||||
the three headings, non-stub bodies), which is unchanged. I did not re-run the full gate for a
|
||||
prose edit; CI will gate the pushed commit.
|
||||
|
||||
**`gitea-settings` did not run locally** — it needs `SETTINGS_API_TOKEN`, which exists only in CI.
|
||||
I did not re-measure branch protection, the runner, or `required_approvals` against the Gitea API
|
||||
this session; the claims in `CLAUDE.md` §2 are taken on trust here and are explicitly flagged there
|
||||
as needing re-measurement.
|
||||
@@ -0,0 +1,137 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: fix/vpn-ingress-l2-lease-collision
|
||||
title: "A node cannot reach a VIP it announces — so the VPN ingress node must announce nothing"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
The first device enrolled on the remote-access VPN handshook, resolved names, and could not open a
|
||||
single connection to any cluster web app. The tunnel was not at fault. The fault is a property of
|
||||
Cilium L2 announcement that nothing in this repo knew about:
|
||||
|
||||
**A node cannot reach a LoadBalancer VIP whose `CiliumL2AnnouncementPolicy` lease it holds itself.**
|
||||
The announcer answers ARP for the VIP on behalf of other LAN hosts but not for its own request, so
|
||||
on the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP is dropped
|
||||
before it leaves the node.
|
||||
|
||||
`platform/vpn-ingress` is uniquely exposed because it runs WireGuard with `hostNetwork: true` and
|
||||
MASQUERADEs remote clients to the node's own address — a VPN client's packets take the node's LAN
|
||||
path and inherit its ARP failures. Ordinary pods never do. The router's `udp/51820` forward pointed
|
||||
at `k3s-prod-1`, which held `cilium-l2announce-gateway-cilium-gateway-shared`, so **all 16 `shared`
|
||||
Gateway hostnames were dead over the VPN** — while PiHole DNS kept answering, because
|
||||
`192.168.2.23` is announced by *two* Services and `k3s-prod-2` held the other lease. Names resolved
|
||||
and every app hung on connect.
|
||||
|
||||
Seven files:
|
||||
|
||||
* **`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`** — new
|
||||
`node_baseline_vpn_ingress_node`; the matching node gets
|
||||
`homelab.keithjackson.dev/vpn-ingress=true`, and a second task **removes** the label from any node
|
||||
that no longer matches, so moving the value moves the label rather than adding a second one. Same
|
||||
shape as `roles/vpn_egress_gateway`'s `egress-gateway` label. Tagged `vpn-ingress`; skipped
|
||||
entirely when the value is empty.
|
||||
* **`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`** — set to `k3s-prod-1`, with a
|
||||
`[parity]` note that it must equal the router's forward.
|
||||
* **`platform-config/cilium/l2-announcement-policy.yaml`** — `nodeSelector` becomes a
|
||||
`matchExpressions` list keeping `kubernetes.io/os: linux` and adding
|
||||
`homelab.keithjackson.dev/vpn-ingress NotIn ["true"]`. The VPN node then holds no lease, so it
|
||||
ARP-resolves every VIP. This fixes **all** VIPs for VPN clients (Gateway, crafty, plex, pihole,
|
||||
unifi), not just the one that happened to collide.
|
||||
* **`platform/monitoring/kube-prometheus-stack.yaml`** — the label added to kube-state-metrics'
|
||||
`metricLabelsAllowlist`, which is what lets the invariant be watched at all.
|
||||
* **`platform/monitoring/alerts.yaml`** — `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) joins
|
||||
`kube_lease_owner` against `kube_node_labels` via `label_replace`, plus
|
||||
`VpnIngressNodeLabelMissing` (warning, 30m) as its watchdog.
|
||||
* **`platform/monitoring/tests/vpn-ingress-l2.test.yaml`** — four promtool cases: fires on the real
|
||||
collision, quiet in the fixed state, quiet across a transient 11-minute lease move (pinning the
|
||||
`for: 15m`), and the label-absent case asserting the critical alert is inert *and* the watchdog
|
||||
fires.
|
||||
* **`runbooks/vpn-ingress-l2-collision.md`** (new, registered in `runbooks/kustomization.yaml`),
|
||||
**`runbooks/vpn-remote-access.md`**, **`platform/vpn-ingress/{daemonset.yaml,README.md}`** — step 3
|
||||
of the setup runbook said *"pick any node — all three run an identical listener"* and *"node loss =
|
||||
failover is this one line"*. Both were false and the first caused this. Failover is now documented
|
||||
as two ordered changes.
|
||||
|
||||
**Design note — why a node label and not a `cluster-vars` knob.** The first attempt added
|
||||
`WG_INGRESS_NODE` to `clusters/base/cluster-vars.yaml` and substituted it into both the policy and
|
||||
the alert. The alert half is **impossible**: `platform/monitoring/tests/extract_rules.py` rejects any
|
||||
alert whose `expr` contains a `${...}` placeholder, because the rule it would unit-test is not the
|
||||
rule deployed. Rather than duplicate the node name into the alert as a literal (two sources that can
|
||||
skew), the label is readable from both sides — the selector matches on it and `kube_node_labels`
|
||||
carries it into Prometheus. One source of truth, and as a side effect the change touches no
|
||||
sensitive path, so it merges on a green gate rather than needing a human review.
|
||||
|
||||
The DaemonSet deliberately gained **no** probe for this. A readinessProbe on VIP reachability would
|
||||
wedge a rollout behind an unrelated Gateway outage (`maxUnavailable: 1`); a liveness probe would tear
|
||||
down a tunnel carrying LAN traffic correctly.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured live against prod, `--context k3s-prod`, on 2026-10-08:
|
||||
|
||||
* **The whole tunnel is healthy**, which is what made this hard: `wg show` on `k3s-prod-1` shows the
|
||||
peer with a real endpoint and bidirectional transfer (346 KiB in / 200 KiB out);
|
||||
`net.ipv4.ip_forward=1`; `wg0` up at `10.90.0.1/24` MTU 1420; `iptables -t nat -S POSTROUTING`
|
||||
carries the `MASQUERADE` rule with 26 packets; `FORWARD -i/-o wg0` ACCEPT with 108 / 171 packets;
|
||||
DaemonSet generation 2, 3/3 ready on the content-hashed `wg-peers-m9kcckcb5t`.
|
||||
* **The ARP failure and its symmetry.** On `k3s-prod-1`: `ip neigh show 192.168.2.20` → `FAILED`,
|
||||
while `.21`/`.23` resolve to prod-2's MAC `04:0e:3c:96:b0:94`. On prod-2/prod-3, `.20` resolves to
|
||||
prod-1's `bc:e9:2f:7e:c4:a6`. TCP agrees: `.20:443` and `.20:80` FAIL from prod-1, OK from the
|
||||
other two. Repeated across 3 nodes × 5 VIPs — each node reached every VIP except the ones it alone
|
||||
announced (prod-2 ✗ `.22`/`.21`; prod-3 ✗ `.24`).
|
||||
* **Not an artefact of probing from the host netns.** `curl --interface wg0` from prod-1 fails the
|
||||
same way, and a neighbour entry is node-global state, so the forwarded path cannot differ in ARP
|
||||
resolution.
|
||||
* **The Gateway itself was never broken:** from the workstation (192.168.2.133)
|
||||
`home.keithjackson.dev` → `200`, `plex.home.keithjackson.dev` → `401`, both at `192.168.2.20`.
|
||||
From prod-1 the router (`.1`) and TrueNAS (`.30`) are `REACHABLE`.
|
||||
* **`NotIn` semantics, empirically** — the basis for the fail-back claim in the policy's comment:
|
||||
`kubectl get nodes -l 'kubernetes.io/hostname notin (k3s-prod-1)'` returns prod-2 and prod-3, and
|
||||
`NotIn` on a key no node carries returns **all three**. So an unapplied label excludes nothing and
|
||||
degrades to the old behaviour rather than taking every VIP off the LAN.
|
||||
* **The alert metric is really scraped:** `kube_lease_owner` is served by kube-state-metrics (55
|
||||
series, every `cilium-l2announce-*` lease with its holder in `lease_holder`), and `kube_node_labels`
|
||||
already carries the two `egress-*` labels through the same allowlist (`k3s-prod-3` shows
|
||||
`egress-gateway=true`), so the third entry follows a proven path.
|
||||
* **The four promtool cases pass** (`platform/monitoring/tests/run.sh` → `SUCCESS`, 55 rules
|
||||
checked), including that `VpnIngressNodeAnnouncesL2VIP` is inert when the label is absent.
|
||||
* All edited YAML parses; the rendered `nodeSelector` is the intended two-expression list.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **The fix is UNAPPLIED and therefore UNPROVEN end to end.** Applying it needs a mutating
|
||||
`kubectl`/`flux reconcile` plus an Ansible run, which are operator actions (CLAUDE.md §5). I have
|
||||
**not** observed `k3s-prod-1` resolve `192.168.2.20` after the exclusion lands, nor a VPN client
|
||||
reaching an app. The prediction rests on the measured symmetry (non-announcing nodes resolve every
|
||||
VIP, 15/15 data points), not on a test of the changed state. **Verify after merge, in this order:**
|
||||
re-run `node_baseline --tags vpn-ingress`; `kubectl get nodes -L
|
||||
homelab.keithjackson.dev/vpn-ingress` (exactly one node); `kubectl -n kube-system get lease | grep
|
||||
l2announce` (no holder is that node); `ip neigh show 192.168.2.20` from the prod-1 wireguard pod
|
||||
(expect a `lladdr`, not `FAILED`); then a client.
|
||||
* **That Cilium actually releases a lease when its holder stops matching the `nodeSelector`.** I
|
||||
assumed re-election and did not test it. If Cilium keeps renewing, the affected Leases must be
|
||||
deleted once by hand to force an election. Check this first if the fix appears not to work.
|
||||
* **The brief interruption on apply.** Moving the Gateway and `pihole-dns-udp` leases off prod-1 will
|
||||
blip `192.168.2.20` and `192.168.2.23` for as long as re-election plus gratuitous ARP takes. Not
|
||||
measured, and I did not test whether in-flight LAN sessions survive it.
|
||||
* **The Ansible tasks have never been executed.** They parse as YAML and follow
|
||||
`roles/vpn_egress_gateway`'s shape, but neither the label task nor the label-removal task has run,
|
||||
so the `changed_when`/`failed_when` conditions and the `k3s kubectl label ... -` removal path are
|
||||
unproven. The removal task's `failed_when` tolerates `'not found'` in stderr; that string was
|
||||
assumed from kubectl's behaviour, not observed.
|
||||
* **The residual off-repo coupling is NOT closed and cannot be, as built.** The router's forward is
|
||||
unreadable from the cluster. If someone re-points it without moving the label, the alert watches
|
||||
the wrong node and stays silent while the VPN breaks identically against a different set of VIPs.
|
||||
Only the runbook guards it. Closing it needs a WireGuard exporter publishing which node is
|
||||
actually terminating peer traffic (`wg show` knows — only the forwarded node has a peer
|
||||
`endpoint`); listed as "Not built" in the component README and **recommended follow-up, not done
|
||||
here**.
|
||||
* **Why Cilium behaves this way** — intended, a known upstream issue, or fixed in a later version. I
|
||||
measured the behaviour and did not read Cilium's source or issue tracker, so I cannot say whether
|
||||
an upgrade would remove the need for this exclusion.
|
||||
* **`192.168.2.21:8443` (unifi) fails from all three nodes**, including ones where ARP resolves, so
|
||||
it is a different, pre-existing fault. Not investigated, not addressed.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so CI proves the L0/L1 offline
|
||||
subset only; none of the above is covered by a test that runs in the gate.
|
||||
* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).
|
||||
@@ -420,3 +420,22 @@ flux_bootstrap_env_path: "clusters/prod"
|
||||
# rather than leaving an inert drop-in behind, so this group-wide `true` is still correct
|
||||
# for it.
|
||||
etcd_io_priority_enabled: true
|
||||
|
||||
# --- The node terminating remote-access WireGuard — DECIDED (measured 2026-10-08) ---------------
|
||||
# k3s-prod-1 (192.168.2.10) is what the SOHO router forwards WAN udp/51820 to, confirmed from the
|
||||
# cluster rather than assumed: only prod-1's `wg show` had a peer with an `endpoint` and a recent
|
||||
# handshake, the other two listeners were idle.
|
||||
#
|
||||
# node_baseline labels it `homelab.keithjackson.dev/vpn-ingress=true`, and
|
||||
# platform-config/cilium/l2-announcement-policy.yaml excludes that label from announcing any
|
||||
# LoadBalancer VIP -- because a node CANNOT reach a VIP whose Cilium L2 announcement lease it holds
|
||||
# itself (the announcer does not answer its own ARP request; the neighbour entry goes to `FAILED`).
|
||||
# prod-1 held the `shared` Gateway lease, and since vpn-ingress MASQUERADEs clients to the node's
|
||||
# own address, all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved.
|
||||
#
|
||||
# [parity] MUST equal the node named by the router's forward (runbooks/vpn-remote-access.md step 3).
|
||||
# Nothing in CI can read that forward, so re-pointing it means changing BOTH: this value plus a
|
||||
# node_baseline re-run, and the router line. `VpnIngressNodeAnnouncesL2VIP` catches the case where
|
||||
# the label lands on a node that then takes a lease; it CANNOT catch the forward being moved
|
||||
# without this value following, which is why the runbook spells the two-step out.
|
||||
node_baseline_vpn_ingress_node: k3s-prod-1
|
||||
@@ -259,3 +259,30 @@ node_baseline_dns_options:
|
||||
# Written as ONE marked block, so removing an entry here removes it from the node, and
|
||||
# an empty list removes the block entirely.
|
||||
node_baseline_hosts_pins: []
|
||||
|
||||
# --- The node that terminates remote-access WireGuard (tags: vpn-ingress) ----------------------
|
||||
# Hostname of the node the SOHO router forwards WAN udp/51820 to, i.e. the one platform/vpn-ingress
|
||||
# actually serves remote clients from. Set it in group_vars for the environment; EMPTY by default,
|
||||
# because dev has no such forward and because a wrong value is worse than none.
|
||||
#
|
||||
# WHAT IT IS FOR, AND WHY IT IS NOT COSMETIC. The matching node gets
|
||||
# `homelab.keithjackson.dev/vpn-ingress=true`, and
|
||||
# platform-config/cilium/l2-announcement-policy.yaml EXCLUDES that label from announcing any
|
||||
# LoadBalancer VIP. MEASURED on prod 2026-10-08: a node cannot reach a VIP whose Cilium L2
|
||||
# announcement lease it holds itself (the announcer does not answer its own ARP request -- the
|
||||
# neighbour entry goes to `FAILED`). Because vpn-ingress runs on `hostNetwork` and MASQUERADEs
|
||||
# clients to the node's own address, a VPN client inherits that: with the forward on the node
|
||||
# holding the `shared` Gateway lease, all 16 Gateway hostnames were dead over the VPN while PiHole
|
||||
# DNS still resolved. Names worked, every app hung, nothing alerted.
|
||||
#
|
||||
# [parity] MUST name the node the router's forward actually points at. The forward is off-repo and
|
||||
# no check can read it (runbooks/vpn-remote-access.md step 3). Changing the forward is therefore a
|
||||
# two-part change: this value (then re-run the role), and the router line.
|
||||
#
|
||||
# Only ONE node should ever carry the label: it removes that node from VIP announcement candidacy,
|
||||
# so labelling all three would leave nothing announcing any VIP at all. The task below removes the
|
||||
# label from a node that no longer matches, so moving the value moves the label.
|
||||
node_baseline_vpn_ingress_node: ""
|
||||
node_baseline_vpn_ingress_label_key: homelab.keithjackson.dev/vpn-ingress
|
||||
# k3s ships its kubeconfig here; every prod node is a k3s server, so `k3s kubectl` works locally.
|
||||
node_baseline_vpn_ingress_kubeconfig: /etc/rancher/k3s/k3s.yaml
|
||||
@@ -1109,3 +1109,45 @@
|
||||
state: "{{ 'present' if node_baseline_hosts_pins | length > 0 else 'absent' }}"
|
||||
backup: true
|
||||
tags: [dns, hosts]
|
||||
|
||||
# ── vpn-ingress node label — tag `vpn-ingress` ────────────────────────────────────────────────
|
||||
# Marks the node terminating remote-access WireGuard so Cilium's L2 announcement policy can EXCLUDE
|
||||
# it from announcing LoadBalancer VIPs (a node cannot reach a VIP it announces itself -- see
|
||||
# defaults/main.yml). Same shape as roles/vpn_egress_gateway's egress-gateway label, and for the
|
||||
# same reason: the label is the one place BOTH a nodeSelector and a PromQL alert can read it from.
|
||||
#
|
||||
# Skipped entirely when node_baseline_vpn_ingress_node is empty (dev, or before the forward exists),
|
||||
# so this adds nothing to an environment that has no WAN forward.
|
||||
- name: Label this node as the vpn-ingress node
|
||||
ansible.builtin.command:
|
||||
cmd: >-
|
||||
k3s kubectl label node {{ inventory_hostname }}
|
||||
{{ node_baseline_vpn_ingress_label_key }}=true --overwrite
|
||||
environment:
|
||||
KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}"
|
||||
register: node_baseline_vpn_ingress_label_apply
|
||||
changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_apply.stdout"
|
||||
when:
|
||||
- node_baseline_vpn_ingress_node | length > 0
|
||||
- inventory_hostname == node_baseline_vpn_ingress_node
|
||||
tags: [vpn-ingress]
|
||||
|
||||
# The other half: moving the knob must MOVE the label, not add a second one. Two labelled nodes
|
||||
# would remove two of three nodes from announcement candidacy; three would leave no announcer at
|
||||
# all and take every VIP off the LAN.
|
||||
- name: Remove the vpn-ingress label from a node that no longer holds the forward
|
||||
ansible.builtin.command:
|
||||
cmd: >-
|
||||
k3s kubectl label node {{ inventory_hostname }}
|
||||
{{ node_baseline_vpn_ingress_label_key }}-
|
||||
environment:
|
||||
KUBECONFIG: "{{ node_baseline_vpn_ingress_kubeconfig }}"
|
||||
register: node_baseline_vpn_ingress_label_remove
|
||||
changed_when: "'not labeled' not in node_baseline_vpn_ingress_label_remove.stdout"
|
||||
failed_when:
|
||||
- node_baseline_vpn_ingress_label_remove.rc != 0
|
||||
- "'not found' not in node_baseline_vpn_ingress_label_remove.stderr"
|
||||
when:
|
||||
- node_baseline_vpn_ingress_node | length > 0
|
||||
- inventory_hostname != node_baseline_vpn_ingress_node
|
||||
tags: [vpn-ingress]
|
||||
@@ -16,6 +16,71 @@ spec:
|
||||
externalIPs: false
|
||||
interfaces:
|
||||
- ^e(n|th)
|
||||
# ===============================================================================================
|
||||
# WHY THIS IS A matchExpressions LIST AND NOT `matchLabels: {kubernetes.io/os: linux}`
|
||||
# ===============================================================================================
|
||||
# The second expression excludes every node labelled `homelab.keithjackson.dev/vpn-ingress=true`
|
||||
# from announcing ANY LoadBalancer VIP. That is the node the SOHO router forwards WAN udp/51820 to.
|
||||
# It is not tuning; it is the fix for a measured, silent, total failure of remote access.
|
||||
#
|
||||
# MEASURED ON PROD 2026-10-08: A NODE CANNOT REACH A VIP IT ANNOUNCES ITSELF. Cilium's L2
|
||||
# announcer answers ARP for the VIP on behalf of OTHER LAN hosts, but not for the announcing node
|
||||
# itself. On the lease holder the neighbour entry settles at `FAILED` and every packet to that VIP
|
||||
# is dropped before it leaves the node:
|
||||
#
|
||||
# k3s-prod-1, holding cilium-l2announce-gateway-cilium-gateway-shared:
|
||||
# ip neigh show 192.168.2.20 -> 192.168.2.20 dev eno1 FAILED (its own VIP)
|
||||
# nc -z 192.168.2.20 443 -> FAIL nc -z 192.168.2.20 80 -> FAIL
|
||||
# k3s-prod-2 and k3s-prod-3, holding no lease for it:
|
||||
# ip neigh show 192.168.2.20 -> lladdr bc:e9:2f:7e:c4:a6 (prod-1) nc 443/80 -> OK
|
||||
#
|
||||
# Symmetric across all 3 nodes x 5 VIPs: every node reached every VIP EXCEPT the ones it alone
|
||||
# announced (prod-2 could not reach crafty 192.168.2.22 or unifi 192.168.2.21; prod-3 could not
|
||||
# reach plex 192.168.2.24). So this is a property of L2 announcement, not of one Service.
|
||||
#
|
||||
# WHY IT ONLY HURTS THE VPN NODE. For an ordinary workload this is harmless: pods reach Services
|
||||
# through the cluster datapath and never ARP for a VIP. platform/vpn-ingress is the exception --
|
||||
# it runs WireGuard with `hostNetwork: true` and MASQUERADEs remote clients to the node's own
|
||||
# address, so a VPN client's packets take the node's LAN path and inherit exactly this ARP
|
||||
# failure. On 2026-10-08 the forward pointed at k3s-prod-1, which held the Gateway lease, and all
|
||||
# 16 Gateway hostnames were unreachable over the VPN while PiHole DNS still answered -- so names
|
||||
# resolved and every app hung. Nothing in the cluster reported it; the alerts below now do.
|
||||
#
|
||||
# WHY A NODE LABEL AND NOT A cluster-vars KNOB. A knob would put the node's identity in git, which
|
||||
# is more reviewable -- but it CANNOT BE USED BY THE ALERT that guards this, and an unguarded
|
||||
# invariant here is worth less than a reviewable one. The alert harness
|
||||
# (platform/monitoring/tests/extract_rules.py) REFUSES any alert whose `expr` contains a
|
||||
# dollar-brace postBuild placeholder (spelled without braces here ON PURPOSE: this file is itself
|
||||
# envsubst'd, and a literal dollar-brace-dot-dot-dot is UNPARSEABLE to envsubst, which fails the
|
||||
# WHOLE Kustomization -- ci: flux-postbuild-substitution caught exactly this comment), because the
|
||||
# rule it would unit-test is not the rule that gets
|
||||
# deployed. A label is readable from BOTH sides: this selector matches on it, and
|
||||
# `kube_node_labels` carries it into Prometheus so `VpnIngressNodeAnnouncesL2VIP` can join lease
|
||||
# holders against it. One source of truth, no git-vs-cluster skew, and the policy stays
|
||||
# env-agnostic as its header above claims. The label is set by
|
||||
# infra/ansible/roles/node_baseline (node_baseline_vpn_ingress_node), the same shape
|
||||
# roles/vpn_egress_gateway already uses for `egress-gateway`.
|
||||
#
|
||||
# THE COST, STATED: VIP announcement candidacy drops from 3 nodes to 2. That is still redundant,
|
||||
# and the trade is deliberate -- the excluded node is the one whose *outbound* reach to every VIP
|
||||
# matters most, because it carries every remote client. If BOTH remaining nodes are down the VIPs
|
||||
# go dark where previously one node might have served them, but at that point the cluster has one
|
||||
# node left and far worse problems.
|
||||
#
|
||||
# `NotIn` with an ABSENT key MATCHES (k8s apimachinery: a missing label is "not in the set"), so
|
||||
# the other two nodes need no label and nothing has to be applied to them. If the label is never
|
||||
# applied anywhere, this excludes nothing and the policy degrades to its pre-2026-10-08 behaviour
|
||||
# -- the status quo, not an outage: it fails back, not down. That silent-degradation path is
|
||||
# exactly why `VpnIngressNodeLabelMissing` exists as a watchdog: without it, the alert below
|
||||
# would be structurally INERT (its join would match nothing) and would look identical to healthy.
|
||||
nodeSelector:
|
||||
matchLabels:
|
||||
kubernetes.io/os: linux
|
||||
matchExpressions:
|
||||
- key: kubernetes.io/os
|
||||
operator: In
|
||||
values:
|
||||
- linux
|
||||
# The node terminating remote-access WireGuard must announce nothing -- see above.
|
||||
- key: homelab.keithjackson.dev/vpn-ingress
|
||||
operator: NotIn
|
||||
values:
|
||||
- "true"
|
||||
@@ -154,74 +154,6 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
|
||||
(it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
|
||||
a realm edit landed.
|
||||
|
||||
## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)
|
||||
|
||||
`keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
|
||||
policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
|
||||
**ALTERNATIVE** beside the password form. Design decisions and their reasoning are in the manifest
|
||||
comments, not repeated here. The headline choices:
|
||||
|
||||
| Decision | Value | One-line reason |
|
||||
|---|---|---|
|
||||
| Additive or replacing? | **Additive** — passkey *or* password | A misfiring mandatory flow locks the single operator out of every SSO app *and* kubectl |
|
||||
| Enrolment | **Optional**, never a default action | Same reason; `webauthn-register-passwordless` is already enabled-not-forced by default |
|
||||
| Relying-party ID | `${BASE_DOMAIN}` (not `keycloak.${BASE_DOMAIN}`) | An unre-keyable credential must not be bound to a hostname that can move |
|
||||
| User verification | `required` | Without it, mere possession of an unlocked device is a complete login |
|
||||
| Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
|
||||
| OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
|
||||
|
||||
### This change does NOT reach the live realm. At all.
|
||||
|
||||
Same one-shot failure this file already documents twice above, third field over. Measured on prod
|
||||
2026-10-17:
|
||||
|
||||
```
|
||||
$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
|
||||
{"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
|
||||
{"status":"False","type":"HasErrors"}]}
|
||||
```
|
||||
|
||||
`Done=True` / `Started=False` — the import has completed and **will not re-run**, so a policy block
|
||||
and three new sub-flows added to this file change nothing about how anyone logs in. The
|
||||
client-secret/client-URL reconciler does **not** close this gap and was never meant to: its hard
|
||||
endpoint allow-list makes `/authentication` unreachable **by design** (see the table above), and
|
||||
realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
|
||||
mapper or a changed scope still needs the realm import to actually re-run" case named above.
|
||||
|
||||
**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
|
||||
an operator applies it by hand.** Two paths, both operator actions:
|
||||
|
||||
1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
|
||||
for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
|
||||
match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
|
||||
into Username Form + Password Form and add the two sub-flows exactly as declared.
|
||||
2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
|
||||
the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
|
||||
is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
|
||||
|
||||
Either way the realm is then configured by hand and git is only the record of intent, which is a
|
||||
live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.
|
||||
|
||||
### Verifying it actually works (nobody has, as of 2026-10-17)
|
||||
|
||||
No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
|
||||
it, in order:
|
||||
|
||||
```sh
|
||||
# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
|
||||
# Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
|
||||
# 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
|
||||
# -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
|
||||
# is wrong for the origin and NOTHING will say so.
|
||||
# 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
|
||||
# prompt. A platform-admin must NOT be asked for TOTP on this path.
|
||||
# 4. regression (the important one): a user with NO passkey must still get the password screen,
|
||||
# and a platform-admin must still be asked for TOTP there.
|
||||
```
|
||||
|
||||
A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak
|
||||
is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.
|
||||
|
||||
## Exceptions that keep native auth (do not force SSO) — ADR-0014
|
||||
|
||||
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
|
||||
@@ -74,142 +74,6 @@ spec:
|
||||
bruteForceProtected: true
|
||||
permanentLockout: false
|
||||
sslRequired: external
|
||||
# ===========================================================================================
|
||||
# PASSKEYS — the WebAuthn PASSWORDLESS policy (R-IDP-04). 2026-10-17.
|
||||
# ===========================================================================================
|
||||
# Read this together with the browser-mfa flow at the bottom of the file: THIS block decides
|
||||
# what a passkey must BE, the FLOW decides when one is OFFERED. Keycloak keeps TWO independent
|
||||
# WebAuthn policies per realm and they are not interchangeable:
|
||||
#
|
||||
# webAuthnPolicy* -> the TWO-FACTOR policy, consumed by the `webauthn-authenticator`
|
||||
# authenticator (a security key presented AFTER a password).
|
||||
# webAuthnPolicyPasswordless* -> the PASSWORDLESS policy, consumed by
|
||||
# `webauthn-authenticator-passwordless` (a passkey presented
|
||||
# INSTEAD of a password). This is the one passkeys use.
|
||||
#
|
||||
# Setting the wrong family is a silent no-op: the passwordless authenticator reads
|
||||
# getWebAuthnPolicyPasswordless() only (WebAuthnPasswordlessAuthenticator.getWebAuthnPolicy),
|
||||
# so a policy written into the two-factor fields is simply never consulted by a passkey login.
|
||||
# Only the passwordless family is declared here; the two-factor family is deliberately left at
|
||||
# Keycloak's defaults because nothing in this realm uses `webauthn-authenticator` — TOTP
|
||||
# (auth-otp-form) is the second factor for platform-admins, see the flow section.
|
||||
#
|
||||
# ADDITIVE AND OPTIONAL — THE DECISION THIS BLOCK EXISTS TO MAKE
|
||||
# -------------------------------------------------------------
|
||||
# Passkeys are offered ALONGSIDE the password, never instead of it, and enrolment is NOT forced.
|
||||
# This is a single-operator homelab whose IdP gates Grafana, Gitea, Nextcloud, Headlamp,
|
||||
# Semaphore, PiHole, Crafty, the *arr stack AND kubectl itself (the `kubernetes` client below).
|
||||
# A mandatory passkey flow that misfires is therefore not an inconvenience, it is a total
|
||||
# lock-out from the whole platform AND from the cluster API in one step — and the only repair
|
||||
# paths left would be the Keycloak bootstrap admin and the break-glass cert kubeconfig. The
|
||||
# expected gain from forcing enrolment (one human stops typing a password) is small; the
|
||||
# expected loss is the platform. So: additive, optional, and password login keeps working for
|
||||
# every user who never enrols.
|
||||
#
|
||||
# Concretely that means: no `requiredActions` block is declared anywhere in this file (see the
|
||||
# long note in the flow section for why declaring one would be actively DANGEROUS), and the
|
||||
# passkey step sits as an ALTERNATIVE beside the password form rather than ahead of it.
|
||||
#
|
||||
# RELYING-PARTY ID — THE FIELD THAT SILENTLY BREAKS EVERY PASSKEY IF IT IS WRONG
|
||||
# ------------------------------------------------------------------------------
|
||||
# The RP ID is baked into every credential at enrolment and can never be changed afterwards:
|
||||
# a passkey created under one RP ID is unusable under another, with no error the user can act
|
||||
# on — the browser simply reports that no credential is available. It is also browser-enforced,
|
||||
# so it cannot be "fixed" server-side: per WebAuthn, the RP ID must be equal to, or a
|
||||
# registrable-domain suffix of, the origin's effective domain. Keycloak is served at
|
||||
# keycloak.<BASE_DOMAIN> (httproute.yaml, and the Keycloak CR's hostname), so the legal choices
|
||||
# are keycloak.<BASE_DOMAIN>, <BASE_DOMAIN>, or the registrable domain one level up. It is NOT
|
||||
# the full URL and NOT a scheme — "https://..." here would fail every ceremony.
|
||||
#
|
||||
# We choose BASE_DOMAIN, the shared platform suffix, NOT the keycloak host, for one concrete
|
||||
# reason drawn from this repo's own history: a credential that cannot be re-keyed must not be
|
||||
# bound to a name that can move. The oauth2-proxy client below ALREADY declares a redirect URI
|
||||
# on auth.<BASE_DOMAIN>, i.e. this platform already contemplates serving auth on a second
|
||||
# hostname; and the client-URL reconciler exists precisely because moving a hostname in git
|
||||
# silently fails to reach the realm (see README, "Client URLs: the same one-shot failure").
|
||||
# Binding passkeys to the keycloak subdomain would mean that relocating Keycloak invalidates
|
||||
# every enrolled passkey; binding them to BASE_DOMAIN survives it. The cost is scope: any
|
||||
# future WebAuthn relying party under this base domain could assert the same RP ID. In a
|
||||
# homelab where every such party is already behind THIS IdP that is not a trust boundary worth
|
||||
# buying. Note the residual, unavoidable risk: changing BASE_DOMAIN itself invalidates every
|
||||
# passkey, which is a re-enrolment, not a repair.
|
||||
#
|
||||
# Written as the cluster-var so it is substituted by the same Flux postBuild pass, with the same
|
||||
# value, as every hostname in this file — there is no second place the domain is decided.
|
||||
webAuthnPolicyPasswordlessRpId: ${BASE_DOMAIN}
|
||||
# The human-readable name the browser/OS shows in the passkey prompt and stores beside the
|
||||
# credential ("Save a passkey for Homelab SSO?"). Matches the realm displayName above so the
|
||||
# OS keychain entry is recognisable. It is also load-bearing on the UPDATE path: Keycloak's
|
||||
# updateRealm applies the passwordless policy ONLY when this field is non-empty
|
||||
# (DefaultExportImportManager line ~842), so an admin-API realm update that omits it drops the
|
||||
# whole policy silently.
|
||||
webAuthnPolicyPasswordlessRpEntityName: "Homelab SSO"
|
||||
# ES256 then RS256 — exactly Keycloak's own DEFAULT_POLICY, declared explicitly rather than
|
||||
# inherited so the realm's crypto is auditable from git. ES256 is what every platform
|
||||
# authenticator (Touch ID/Face ID, Windows Hello, Android) and every FIDO2 key produces;
|
||||
# RS256 is the fallback for older Windows Hello TPM-backed credentials. EdDSA/Ed25519 is
|
||||
# deliberately NOT offered: authenticator and browser support is uneven, and an authenticator
|
||||
# that picks an algorithm the verifier handles badly fails at enrolment, which is the worst
|
||||
# moment to discover it. Order is a preference list, not an allow-list ordering requirement.
|
||||
webAuthnPolicyPasswordlessSignatureAlgorithms:
|
||||
- ES256
|
||||
- RS256
|
||||
# DISCOVERABLE CREDENTIAL REQUIRED. "Yes" is a Keycloak sentinel, not a boolean, and the login
|
||||
# theme compares it as the literal string: webauthnRegister.js maps 'Yes' to
|
||||
# authenticatorSelection.requireResidentKey=true and ANY other non-sentinel value to false.
|
||||
# This is not optional for passkeys — a non-discoverable credential cannot be found by the
|
||||
# browser without the server first naming it, so it is a second factor, not a passkey.
|
||||
# (Keycloak 26.0.6 emits only the LEGACY requireResidentKey member, not the newer
|
||||
# residentKey:"required"; per WebAuthn L2 the client maps requireResidentKey=true onto
|
||||
# residentKey="required", so discoverable credentials are genuinely requested — but it also
|
||||
# means this policy cannot express residentKey:"preferred" at all.)
|
||||
webAuthnPolicyPasswordlessRequireResidentKey: "Yes"
|
||||
# USER VERIFICATION REQUIRED. The authenticator must verify the human (biometric or PIN) and
|
||||
# say so in the assertion. This is the field that makes a passkey login at least as strong as
|
||||
# the password it replaces: with verification merely "preferred", possession of an unlocked
|
||||
# phone or an unattended security key would be a COMPLETE login to every SSO-gated app — a
|
||||
# single-factor possession check, strictly weaker than the password+TOTP path beside it.
|
||||
# Required makes the passkey branch inherently two-factor (something you have + something you
|
||||
# are/know), which is the premise the flow section relies on when it does NOT ask a
|
||||
# passkey-authenticated platform-admin for TOTP as well.
|
||||
webAuthnPolicyPasswordlessUserVerificationRequirement: required
|
||||
# "not specified" is Keycloak's sentinel for "either kind of authenticator"
|
||||
# (Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED; the theme JS omits the WebAuthn member
|
||||
# entirely when it sees this exact string). Deliberately NOT narrowed: "platform" would reject
|
||||
# every YubiKey and every phone used as a cross-device passkey, and "cross-platform" would
|
||||
# reject Touch ID and Windows Hello — i.e. either choice silently halves the usable hardware.
|
||||
# The operator should be able to enrol a laptop AND a phone AND a hardware key.
|
||||
webAuthnPolicyPasswordlessAuthenticatorAttachment: "not specified"
|
||||
# ATTESTATION: none. Requesting "direct" would hand Keycloak an attestation statement it has
|
||||
# no trust anchors to validate here (no metadata service is configured) and nothing to compare
|
||||
# it against, since the AAGUID allow-list below is deliberately empty — so the statement would
|
||||
# be collected, unverifiable, and would add an extra "share model information?" consent prompt
|
||||
# on Safari/iOS for no security gain. Attestation is how you ENFORCE an approved-hardware list;
|
||||
# we do not have one, so asking for it is cost without benefit.
|
||||
webAuthnPolicyPasswordlessAttestationConveyancePreference: none
|
||||
# Empty = accept any authenticator model. An AAGUID allow-list is a hardware whitelist: it
|
||||
# would have to be maintained by hand, and every new phone, laptop or key the operator buys
|
||||
# would be rejected at enrolment with a message that does not explain itself. It is also moot
|
||||
# while attestation is "none" (above) — with no attestation statement there is no trustworthy
|
||||
# AAGUID to match against in the first place.
|
||||
webAuthnPolicyPasswordlessAcceptableAaguids: []
|
||||
# Ceremony timeout in SECONDS (the login theme multiplies by 1000; 0 means "send no timeout and
|
||||
# let the browser decide"). 300s is chosen over a tighter value because the slowest LEGITIMATE
|
||||
# path is the cross-device hybrid flow — Keycloak renders a QR code, the operator unlocks a
|
||||
# phone, scans it, and completes a biometric check — which routinely takes over a minute. A
|
||||
# 60-120s timeout looks tidy and converts that flow into an unexplained failure. Browsers clamp
|
||||
# this value (Chrome to at most 10 minutes), so 300 is honoured rather than reinterpreted.
|
||||
webAuthnPolicyPasswordlessCreateTimeout: 300
|
||||
# Send the user's existing passkeys as excludeCredentials at enrolment, so re-enrolling an
|
||||
# authenticator the user already registered is refused by the authenticator itself instead of
|
||||
# producing a second, indistinguishable credential in the account console. Cheap, and it keeps
|
||||
# "which passkey is which" answerable — which matters when the recovery story is "delete the
|
||||
# lost one".
|
||||
webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister: true
|
||||
# webAuthnPolicyPasswordlessExtraOrigins is intentionally NOT set. It exists to admit non-web
|
||||
# origins (native app facet IDs, e.g. android:apk-key-hash:...). Every ceremony here happens in
|
||||
# a browser on Keycloak's own origin, which is already covered by the RP ID above; adding
|
||||
# origins is only ever a widening of who may ask for these credentials.
|
||||
# --- Authorization model (R-IDP-06): groups drive access; platform-admins carry a realm role. ---
|
||||
roles:
|
||||
realm:
|
||||
@@ -664,105 +528,14 @@ spec:
|
||||
"access.token.claim": "true"
|
||||
"userinfo.token.claim": "true"
|
||||
"claim.name": groups
|
||||
# ===========================================================================================
|
||||
# MFA for platform-admins (R-IDP-04) + PASSKEY login (2026-10-17). The realm browserFlow is
|
||||
# bound to browser-mfa at the bottom of this file.
|
||||
# ===========================================================================================
|
||||
# Non-admins log in without OTP; platform-admins must enroll/present TOTP. VERIFY at bring-up
|
||||
# (see runbook): a malformed flow fails the whole import — if so, the fallback is to comment
|
||||
# this block + browserFlow and configure the flow via the admin console.
|
||||
#
|
||||
# THE SHAPE, AND WHY IT IS THIS SHAPE
|
||||
# -----------------------------------
|
||||
# browser-mfa (top level, bound as browserFlow)
|
||||
# auth-cookie ALTERNATIVE already-authenticated SSO cookie
|
||||
# identity-provider-redirector ALTERNATIVE (no IdPs configured; kept for parity)
|
||||
# browser-mfa-forms ALTERNATIVE the interactive path:
|
||||
# auth-username-form REQUIRED WHO are you (no password yet)
|
||||
# browser-mfa-passkey-or-password REQUIRED ...and PROVE it, one of two ways:
|
||||
# webauthn-authenticator-passwordless ALTERNATIVE a passkey, or
|
||||
# browser-mfa-password-and-otp ALTERNATIVE the password path:
|
||||
# auth-password-form REQUIRED password, then
|
||||
# browser-mfa-conditional-otp CONDITIONAL TOTP iff platform-admin
|
||||
#
|
||||
# WHY THE USERNAME/PASSWORD SCREEN HAD TO BE SPLIT IN TWO. The previous flow used the combined
|
||||
# `auth-username-password-form`, which collects both on one page. A passkey branch cannot live
|
||||
# behind that: the server must know WHICH user is logging in before it can decide whether a
|
||||
# passkey is even available, and the combined form has already taken the password by the time
|
||||
# it knows. So it is replaced by the `auth-username-form` / `auth-password-form` pair — the
|
||||
# same decomposition Keycloak's own passwordless recipe uses. THIS IS A VISIBLE UX CHANGE FOR
|
||||
# EVERY USER, passkey or not: login becomes two screens (username, then password) instead of
|
||||
# one. That cost is accepted deliberately — it is the enabling change, it is what Google and
|
||||
# Microsoft logins already do, and the alternative (below) is worse.
|
||||
#
|
||||
# WHY NOT USERNAMELESS ("just tap your passkey", no username screen at all). Keycloak 26.0.6
|
||||
# does support it: WebAuthnPasswordlessAuthenticator.requiresUser() returns FALSE, so the
|
||||
# authenticator can legally run as a first step with no user in context. We deliberately do not
|
||||
# do that, because of how the flow engine then behaves. In DefaultAuthenticationFlow the
|
||||
# "does this user actually have this credential?" guard sits INSIDE the requiresUser() branch;
|
||||
# with requiresUser() false and no user yet set, the engine skips that guard and simply invokes
|
||||
# the authenticator. A passkey execution placed before the username form is therefore offered
|
||||
# to EVERYONE, so every login — including every password-only user and every family member who
|
||||
# will never own a passkey — would open on a passkey prompt they must dismiss. That is the
|
||||
# opposite of additive. Keeping the passkey step AFTER auth-username-form means a user is in
|
||||
# context, which is what makes the graceful degradation below actually work.
|
||||
#
|
||||
# WHY THE PASSKEY BRANCH DEGRADES CLEANLY TO A PASSWORD (the property the whole "additive"
|
||||
# claim rests on). With a user in context, the engine takes the non-requiresUser branch and
|
||||
# raises CREDENTIAL_SETUP_REQUIRED when the user has no credential of this type and the
|
||||
# authenticator's factory does not allow self-setup — and WebAuthnPasswordlessAuthenticatorFactory
|
||||
# .isUserSetupAllowed() is false, and the authenticator is a CredentialValidator, so both
|
||||
# conditions hold. That exception is raised INSIDE the ALTERNATIVE loop, which catches it,
|
||||
# marks the execution ATTEMPTED and proceeds to the next alternative. Net effect: a user with
|
||||
# no passkey never sees a passkey screen and lands on the password form; a user WITH one gets
|
||||
# the passkey screen and keeps the password as a "Try another way" option. Note that for a
|
||||
# user holding BOTH, which screen appears FIRST is decided by Keycloak's authentication
|
||||
# selection list (the user's own credential ordering), not by the priorities below.
|
||||
#
|
||||
# WHY A PASSKEY LOGIN IS NOT ALSO ASKED FOR TOTP. browser-mfa-conditional-otp hangs off the
|
||||
# PASSWORD branch only, so a platform-admin who authenticates with a passkey is not challenged
|
||||
# for TOTP. That is intentional, and it rests on the policy above: userVerification is
|
||||
# `required`, so the authenticator has already proved possession of hardware AND verified the
|
||||
# human (biometric or PIN) in one phishing-resistant, origin-bound ceremony. Stacking TOTP on
|
||||
# top would add a replayable, phishable shared secret to a credential that is neither, for
|
||||
# friction on every login. If a future operator disagrees, the change is small and local: move
|
||||
# the browser-mfa-conditional-otp reference out of browser-mfa-password-and-otp and into
|
||||
# browser-mfa-passkey-or-password's parent as a REQUIRED step, so it applies to both branches.
|
||||
#
|
||||
# WHY THERE IS NO `requiredActions` BLOCK IN THIS FILE — DO NOT ADD ONE
|
||||
# --------------------------------------------------------------------
|
||||
# Enrolment uses Keycloak's `webauthn-register-passwordless` required action, and the obvious
|
||||
# move is to declare it here. DO NOT. In Keycloak 26.0.6 the realm import treats
|
||||
# requiredActions as REPLACE, NOT MERGE: DefaultExportImportManager.importRealm branches on
|
||||
# `if (rep.getRequiredActions() != null)` and, when the representation declares ANY, imports
|
||||
# exactly those plus delete_account — the `else` branch that installs the full default set
|
||||
# (DefaultRequiredActions.addActions) never runs. RealmManager.setupRequiredActions is no
|
||||
# backstop either: it fires only when the realm has ZERO required-action providers.
|
||||
#
|
||||
# So declaring one action would create this realm with that action and delete_account and
|
||||
# NOTHING ELSE — no CONFIGURE_TOTP, no UPDATE_PASSWORD, no VERIFY_EMAIL, no delete_credential,
|
||||
# no UPDATE_PROFILE. The first casualty would be the MFA requirement this very flow enforces:
|
||||
# with CONFIGURE_TOTP unregistered, a platform-admin who does not yet have TOTP could never
|
||||
# enrol one, while auth-otp-form below still demands it — a hard lock-out of every
|
||||
# platform-admin from every SSO-gated app, including Headlamp and kubectl, delivered by a
|
||||
# change whose stated purpose was to ADD a login method.
|
||||
#
|
||||
# And the declaration buys nothing, because the default set ALREADY CONTAINS IT.
|
||||
# DefaultRequiredActions registers `webauthn-register-passwordless` with enabled=true,
|
||||
# defaultAction=false, priority 80, gated on Profile.Feature.WEB_AUTHN — which is Type.DEFAULT
|
||||
# in 26.0.6, i.e. on unless explicitly disabled. enabled=true / defaultAction=false IS the
|
||||
# additive-and-optional posture this change wants: the action exists and can be triggered for
|
||||
# a user on demand, but it is not forced on anyone at next login. Declaring it would restate
|
||||
# the default while deleting ten siblings.
|
||||
#
|
||||
# (The symmetric question for authenticationFlows has the opposite answer, which is why THIS
|
||||
# block is safe: importAuthenticationFlows adds the declared flows and then backfills each
|
||||
# unbound binding — browser, registration, direct grant, reset credentials, client auth, first
|
||||
# broker login, docker auth — from the built-in defaults individually. Declaring flows does not
|
||||
# suppress the built-ins; declaring required actions does suppress them.)
|
||||
# --- MFA for platform-admins (R-IDP-04): a browser flow whose OTP step is CONDITIONAL on the
|
||||
# platform-admin role. Non-admins log in without OTP; platform-admins must enroll/present TOTP.
|
||||
# Set as the realm browserFlow below. VERIFY at bring-up (see runbook): a malformed flow fails
|
||||
# the whole import — if so, the fallback is to comment this block + browserFlow and configure
|
||||
# the conditional-OTP-by-role flow via the admin console. ---
|
||||
authenticationFlows:
|
||||
- alias: browser-mfa
|
||||
description: "Browser flow: passkey or password, with conditional OTP for platform-admins."
|
||||
description: "Browser flow with conditional OTP required for platform-admins."
|
||||
providerId: basic-flow
|
||||
topLevel: true
|
||||
builtIn: false
|
||||
@@ -782,63 +555,13 @@ spec:
|
||||
priority: 30
|
||||
flowAlias: browser-mfa-forms
|
||||
userSetupAllowed: false
|
||||
# Identify the user FIRST (username only), then prove it. Both steps are REQUIRED: the
|
||||
# username is never optional, and neither is proving possession of the account.
|
||||
- alias: browser-mfa-forms
|
||||
description: "Username, then passkey or password (+ conditional OTP)."
|
||||
description: "Username/password then conditional OTP."
|
||||
providerId: basic-flow
|
||||
topLevel: false
|
||||
builtIn: false
|
||||
authenticationExecutions:
|
||||
# auth-username-form = Keycloak's "Username Form" (UsernameFormFactory) — resolves the
|
||||
# user WITHOUT taking a password, which is the prerequisite for offering a passkey.
|
||||
# Replaces the previous auth-username-password-form; see the header for the UX cost.
|
||||
- authenticator: auth-username-form
|
||||
authenticatorFlow: false
|
||||
requirement: REQUIRED
|
||||
priority: 10
|
||||
userSetupAllowed: false
|
||||
- authenticatorFlow: true
|
||||
requirement: REQUIRED
|
||||
priority: 20
|
||||
flowAlias: browser-mfa-passkey-or-password
|
||||
userSetupAllowed: false
|
||||
# THE FORK. Two ALTERNATIVEs, so satisfying either one completes the flow. Priority puts the
|
||||
# passkey first for users who have one; users who do not fall through to the password branch
|
||||
# automatically (CREDENTIAL_SETUP_REQUIRED is caught by the ALTERNATIVE loop — see header).
|
||||
- alias: browser-mfa-passkey-or-password
|
||||
description: "A passkey (WebAuthn passwordless), or password with conditional OTP."
|
||||
providerId: basic-flow
|
||||
topLevel: false
|
||||
builtIn: false
|
||||
authenticationExecutions:
|
||||
# webauthn-authenticator-passwordless = "WebAuthn Passwordless Authenticator"
|
||||
# (WebAuthnPasswordlessAuthenticatorFactory). It reads the realm's PASSWORDLESS policy
|
||||
# declared above — NOT the two-factor webAuthnPolicy* family. Keep it ALTERNATIVE: as
|
||||
# REQUIRED it would force a passkey on every user, which is the lock-out this change is
|
||||
# explicitly designed not to create.
|
||||
- authenticator: webauthn-authenticator-passwordless
|
||||
authenticatorFlow: false
|
||||
requirement: ALTERNATIVE
|
||||
priority: 10
|
||||
userSetupAllowed: false
|
||||
- authenticatorFlow: true
|
||||
requirement: ALTERNATIVE
|
||||
priority: 20
|
||||
flowAlias: browser-mfa-password-and-otp
|
||||
userSetupAllowed: false
|
||||
# The unchanged pre-passkey behaviour, relocated intact into the password branch: password,
|
||||
# then TOTP iff the user holds the platform-admin role. Nothing about the OTP condition
|
||||
# changed; it simply no longer sits on the path a passkey login takes.
|
||||
- alias: browser-mfa-password-and-otp
|
||||
description: "Password, then conditional OTP for platform-admins."
|
||||
providerId: basic-flow
|
||||
topLevel: false
|
||||
builtIn: false
|
||||
authenticationExecutions:
|
||||
# auth-password-form = Keycloak's "Password Form" (PasswordFormFactory) — the password
|
||||
# half of the old combined form, now that the username has already been collected.
|
||||
- authenticator: auth-password-form
|
||||
- authenticator: auth-username-password-form
|
||||
authenticatorFlow: false
|
||||
requirement: REQUIRED
|
||||
priority: 10
|
||||
|
||||
@@ -1147,6 +1147,97 @@ spec:
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-egress-stale"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# REMOTE-ACCESS VPN (platform/vpn-ingress) -- NOT the egress tunnel above. Different
|
||||
# direction, different workload: this is inbound WireGuard for remote devices.
|
||||
# -----------------------------------------------------------------------------------------
|
||||
- alert: VpnIngressNodeAnnouncesL2VIP
|
||||
# THE FAILURE THIS CATCHES. MEASURED on prod 2026-10-08: a node CANNOT reach a
|
||||
# LoadBalancer VIP whose Cilium L2 announcement lease it holds itself -- the announcer
|
||||
# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and
|
||||
# packets die before leaving the node (`ip neigh show 192.168.2.20` -> FAILED on the
|
||||
# lease holder, resolved on both others). platform/vpn-ingress runs WireGuard on
|
||||
# `hostNetwork` and MASQUERADEs remote clients to the node's own address, so if the node
|
||||
# the router forwards udp/51820 to is also a lease holder, VPN clients cannot reach the
|
||||
# VIPs it announces. On 2026-10-08 that was the `shared` Gateway: all 16 app hostnames
|
||||
# dead over the VPN, while PiHole DNS kept answering (its VIP has two Services and the
|
||||
# other lease sat on prod-2) -- so the tunnel handshook, names resolved, and every app
|
||||
# hung. Nothing reported it for ~2 days.
|
||||
#
|
||||
# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
|
||||
# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so this should be
|
||||
# permanently unsatisfiable. The alert exists because the exclusion has ways to stop
|
||||
# working that no static check on that file's TEXT can see: the label being removed or
|
||||
# moved to the wrong node, a Cilium upgrade changing `NotIn` semantics or the
|
||||
# nodeSelector field, and the selector silently failing to match. This watches the
|
||||
# OUTCOME -- who actually holds the lease -- so all of them surface the same way.
|
||||
#
|
||||
# WHY THE LABEL JOIN RATHER THAN A NODE NAME. The node's identity is NOT written into
|
||||
# this expr, because the harness (tests/extract_rules.py) refuses any alert whose expr
|
||||
# contains a dollar-brace placeholder -- spelled in words because a literal one here is
|
||||
# unparseable to envsubst and would fail this whole Kustomization. Rightly so: the rule it
|
||||
# unit-tests would not be the
|
||||
# rule deployed. Joining `kube_lease_owner` against `kube_node_labels` keeps ONE source
|
||||
# of truth (the label itself, which is also what the policy selects on), so this alert
|
||||
# cannot drift out of agreement with the exclusion it guards. `label_replace` is needed
|
||||
# because kube_lease_owner names the holder `lease_holder` while kube_node_labels names
|
||||
# it `node`; the join is then on a shared `node` label. The label is in
|
||||
# kube-state-metrics' metricLabelsAllowlist (kube-prometheus-stack.yaml) -- without that
|
||||
# line this rule is inert, which `VpnIngressNodeLabelMissing` below exists to catch.
|
||||
#
|
||||
# `for: 15m` because leases legitimately move during a node reboot or a Cilium roll and
|
||||
# Cilium re-elects within seconds; a real collision persists, since the lease holder
|
||||
# keeps renewing and it never self-heals.
|
||||
expr: |
|
||||
max by (lease) (
|
||||
label_replace(
|
||||
kube_lease_owner{namespace="kube-system", lease=~"cilium-l2announce-.*"},
|
||||
"node", "$1", "lease_holder", "(.+)"
|
||||
)
|
||||
* on (node) group_left ()
|
||||
kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"}
|
||||
) == 1
|
||||
for: 15m
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "VPN ingress node is announcing a LoadBalancer VIP ({{ $labels.lease }})"
|
||||
description: >-
|
||||
The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
|
||||
WireGuard, and it now holds the Cilium L2 announcement lease `{{ $labels.lease }}`. A
|
||||
node cannot reach a VIP it announces itself, so every remote VPN client has just lost
|
||||
all access to that VIP's Service -- silently, with DNS still resolving. Check that
|
||||
platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
|
||||
the label is still on the node the router forwards udp/51820 to.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
|
||||
- alert: VpnIngressNodeLabelMissing
|
||||
# THE WATCHDOG for the rule above, and the reason it is not optional. The exclusion in
|
||||
# l2-announcement-policy.yaml is a `NotIn` on this label, and `NotIn` with an ABSENT key
|
||||
# MATCHES -- so if the label is on NO node, the policy excludes nothing, every node is an
|
||||
# announcement candidate again, and the 2026-10-08 failure is fully re-armed. The alert
|
||||
# above would not fire either: its join would match nothing and return no series. Present,
|
||||
# evaluating, and incapable of firing -- the exact "inert critical alert" shape that
|
||||
# VPNEgressUplinkLabelMissing exists for on the egress side, and that this repo has been
|
||||
# burned by before.
|
||||
#
|
||||
# The label is applied by infra/ansible/roles/node_baseline
|
||||
# (node_baseline_vpn_ingress_node); the most likely cause of it going absent is a node
|
||||
# rebuild where that role has not been re-run.
|
||||
expr: absent(kube_node_labels{label_homelab_keithjackson_dev_vpn_ingress="true"})
|
||||
for: 30m
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
|
||||
description: >-
|
||||
No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
|
||||
l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
|
||||
any node may now announce a LoadBalancer VIP, including the one terminating
|
||||
remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
|
||||
the warning matters more than its severity suggests. Re-run the node_baseline role on
|
||||
the forwarded node, and confirm the label is in kube-state-metrics'
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
|
||||
# -----------------------------------------------------------------------------------------
|
||||
# (ADR-0015) The former `EgressGatewayUnhealthy` pod-readiness alert was REMOVED with the gluetun
|
||||
# retirement: there is no in-cluster egress-gateway pod anymore (the tunnel is a host systemd
|
||||
# service, steering is Cilium eBPF). Tunnel + gateway-node health is covered by `VPNTunnelDown`
|
||||
|
||||
@@ -281,8 +281,20 @@ spec:
|
||||
# without anyone having to remember to re-enable it later. Scoped to one label on one resource:
|
||||
# 3 extra series, no cardinality concern. Label name is sanitized by KSM to
|
||||
# `label_homelab_keithjackson_dev_egress_gateway` (`label_` + non-alphanumerics -> `_`).
|
||||
# `vpn-ingress` (added 2026-10-17) marks the node the SOHO router forwards WAN udp/51820 to.
|
||||
# platform-config/cilium/l2-announcement-policy.yaml excludes that node from announcing any
|
||||
# LoadBalancer VIP, because a node cannot reach a VIP whose L2 announcement lease it holds
|
||||
# itself and vpn-ingress MASQUERADEs remote clients to the node's own address (measured
|
||||
# 2026-10-08: all 16 Gateway hostnames dead over the VPN while DNS still resolved). It is
|
||||
# allowlisted HERE because that is the only way the invariant can be WATCHED:
|
||||
# `VpnIngressNodeAnnouncesL2VIP` joins `kube_lease_owner` against this label, and the alert
|
||||
# unit-test harness refuses a dollar-brace placeholder in an alert expr (written out in words:
|
||||
# a literal one here is unparseable to envsubst and fails the whole Kustomization), so a knob
|
||||
# could not have been used. Without this line BOTH that alert and its
|
||||
# `VpnIngressNodeLabelMissing` watchdog are structurally inert. Same reasoning, and the same
|
||||
# 3-extra-series cost, as the two egress labels below.
|
||||
metricLabelsAllowlist:
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink]"
|
||||
- "nodes=[homelab.keithjackson.dev/egress-gateway,homelab.keithjackson.dev/egress-uplink,homelab.keithjackson.dev/vpn-ingress]"
|
||||
# --- VELERO SCHEDULE OBJECTS AS METRICS (added 2026-09-13) ---------------------------------
|
||||
# Velero exports NOTHING that enumerates Schedule objects and nothing that exposes
|
||||
# `spec.paused`. Verified on prod against velero v1.18.1: all 35 velero_* metric names that
|
||||
|
||||
@@ -0,0 +1,203 @@
|
||||
# promtool rule-unit tests for VpnIngressNodeAnnouncesL2VIP + VpnIngressNodeLabelMissing
|
||||
# (platform/monitoring/alerts.yaml)
|
||||
#
|
||||
# Run with: platform/monitoring/tests/run.sh
|
||||
# (the merge gate runs the same harness via
|
||||
# conformance/tests/l0-static/alert-rules-unit-tested)
|
||||
#
|
||||
# WHAT THESE RULES CLAIM, AND WHY IT NEEDS PINNING IN BOTH DIRECTIONS
|
||||
# MEASURED on prod 2026-10-08: a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement
|
||||
# lease it holds ITSELF -- the announcer does not answer its own ARP request, so on the lease holder
|
||||
# the neighbour entry settles at `FAILED` and packets to that VIP die before leaving the node.
|
||||
# platform/vpn-ingress runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
|
||||
# own address, so a VPN client takes that node's LAN path and inherits the failure. With the
|
||||
# router's udp/51820 forward on k3s-prod-1, which held the `shared` Gateway lease, all 16 Gateway
|
||||
# hostnames were dead over the VPN -- while PiHole DNS kept answering, because its VIP has two
|
||||
# Services and k3s-prod-2 held the other lease. The tunnel handshook, names resolved, every app
|
||||
# hung, and nothing reported it for ~2 days.
|
||||
#
|
||||
# platform-config/cilium/l2-announcement-policy.yaml now excludes the node labelled
|
||||
# `homelab.keithjackson.dev/vpn-ingress=true` from announcing, so VpnIngressNodeAnnouncesL2VIP
|
||||
# should be permanently unsatisfiable. THAT IS EXACTLY WHY IT NEEDS A TEST: an alert expected never
|
||||
# to fire is indistinguishable, in production, from an alert that CANNOT fire. Nothing would ever
|
||||
# reveal a mistyped label, a broken `label_replace`, or a join on the wrong label. Test 1 proves it
|
||||
# fires on the real failure; test 2 proves it is quiet in the fixed state, so it is not a rule that
|
||||
# fires on anything; test 3 pins the `for: 15m` against routine lease churn.
|
||||
#
|
||||
# THE JOIN IS THE FRAGILE PART AND IS TESTED DELIBERATELY. `kube_lease_owner` names the holder in
|
||||
# `lease_holder`; `kube_node_labels` names it `node`. The rule bridges them with `label_replace`
|
||||
# and joins `on (node) group_left ()`. Fixtures therefore carry BOTH series shapes with real label
|
||||
# sets, because a join that silently matches nothing returns no series -- which looks exactly like
|
||||
# "healthy" and is the failure mode this whole group exists to prevent.
|
||||
#
|
||||
# Fixture values are prod's real shape (measured 2026-10-08): leases
|
||||
# `cilium-l2announce-gateway-cilium-gateway-shared` (k3s-prod-1) and
|
||||
# `cilium-l2announce-pihole-pihole-dns-tcp` (k3s-prod-2), in namespace kube-system, with
|
||||
# k3s-prod-1 the vpn-ingress node.
|
||||
|
||||
rule_files:
|
||||
- generated.rules.yaml
|
||||
|
||||
evaluation_interval: 1m
|
||||
|
||||
tests:
|
||||
# =============================================================================================
|
||||
# 1. THE BUG, EXACTLY AS IT HAPPENED -- the vpn-ingress node holds the Gateway lease: FIRES.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: vpn-ingress node holds the shared Gateway L2 lease - VpnIngressNodeAnnouncesL2VIP fires
|
||||
input_series:
|
||||
# The collision. A lease holder keeps renewing, so this series does not lapse on its own --
|
||||
# which is why the condition never self-heals and why a 15m `for:` is safe.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
|
||||
values: '1+0x40'
|
||||
# A second lease held by a DIFFERENT node, present throughout. This is the discriminator that
|
||||
# stops the rule from being "any l2announce lease exists anywhere": if the join were dropped,
|
||||
# this series alone would fire the alert, and test 2 would fail.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-pihole-pihole-dns-tcp",lease_holder="k3s-prod-2"}'
|
||||
values: '1+0x40'
|
||||
# k3s-prod-1 is the vpn-ingress node. Only this node carries the label.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
|
||||
values: '1+0x40'
|
||||
# The other two nodes publish kube_node_labels WITHOUT the vpn-ingress label. Present so the
|
||||
# join is exercised against non-matching nodes rather than against nothing at all.
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
|
||||
values: '1+0x40'
|
||||
alert_rule_test:
|
||||
# Held 15m, so nothing before then.
|
||||
- eval_time: 14m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
- eval_time: 20m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts:
|
||||
# EXACTLY ONE alert: the pihole lease on prod-2 must NOT produce a second one.
|
||||
- exp_labels:
|
||||
severity: critical
|
||||
lease: cilium-l2announce-gateway-cilium-gateway-shared
|
||||
exp_annotations:
|
||||
summary: "VPN ingress node is announcing a LoadBalancer VIP (cilium-l2announce-gateway-cilium-gateway-shared)"
|
||||
description: >-
|
||||
The node labelled homelab.keithjackson.dev/vpn-ingress=true terminates remote-access
|
||||
WireGuard, and it now holds the Cilium L2 announcement lease `cilium-l2announce-gateway-cilium-gateway-shared`. A
|
||||
node cannot reach a VIP it announces itself, so every remote VPN client has just lost
|
||||
all access to that VIP's Service -- silently, with DNS still resolving. Check that
|
||||
platform-config/cilium/l2-announcement-policy.yaml still excludes that label and that
|
||||
the label is still on the node the router forwards udp/51820 to.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
|
||||
# The label IS present, so the watchdog must stay quiet here.
|
||||
- eval_time: 40m
|
||||
alertname: VpnIngressNodeLabelMissing
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 2. THE FIXED STATE -- every lease is held by a node that is NOT the vpn-ingress node: QUIET.
|
||||
# This is what l2-announcement-policy.yaml's exclusion produces, and the assertion that keeps
|
||||
# test 1 meaningful rather than an alert that fires on anything.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: all l2 leases held by other nodes - VpnIngressNodeAnnouncesL2VIP stays quiet
|
||||
input_series:
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-2"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-plex-plex-l4-tcp",lease_holder="k3s-prod-3"}'
|
||||
values: '1+0x40'
|
||||
# A lease the vpn-ingress node DOES legitimately hold, in a different family -- proves the
|
||||
# `cilium-l2announce-.*` matcher is doing work. Every node holds operator/controller leases;
|
||||
# only the ANNOUNCEMENT ones matter here, and a matcher broad enough to catch this would page
|
||||
# on a healthy cluster forever.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-operator-resource-lock",lease_holder="k3s-prod-1"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
|
||||
values: '1+0x40'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
|
||||
values: '1+0x40'
|
||||
alert_rule_test:
|
||||
- eval_time: 40m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
- eval_time: 40m
|
||||
alertname: VpnIngressNodeLabelMissing
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 3. A LEGITIMATE, TRANSIENT LEASE MOVE -- the vpn-ingress node holds it for 11m during a node
|
||||
# reboot or a Cilium roll, then Cilium re-elects away: QUIET. This pins the `for: 15m` as
|
||||
# deliberate. Without it, every rolling restart of the cilium DaemonSet would page, and a paging
|
||||
# alert that cries wolf on routine maintenance gets muted -- which is how the real collision
|
||||
# would go unseen again.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: transient lease move shorter than for 15m - VpnIngressNodeAnnouncesL2VIP stays quiet
|
||||
input_series:
|
||||
# Held by the vpn-ingress node for 11 samples, then the series goes absent (the lease moved,
|
||||
# so kube-state-metrics retires this label set and starts a new one).
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
|
||||
values: '1+0x10 _x30'
|
||||
# Re-elected onto prod-3 from t=11m. A label VALUE change retires one series and starts
|
||||
# another, which is why this is a second series and not a changing value.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-3"}'
|
||||
values: '_x11 1+0x29'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1",label_homelab_keithjackson_dev_vpn_ingress="true"}'
|
||||
values: '1+0x40'
|
||||
alert_rule_test:
|
||||
# Inside the transient window: pending at most, never firing.
|
||||
- eval_time: 10m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
# Well past it, with the lease settled on another node.
|
||||
- eval_time: 30m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
|
||||
# =============================================================================================
|
||||
# 4. THE WATCHDOG -- no node carries the label at all. The policy's `NotIn` then excludes
|
||||
# NOTHING (a missing key matches), so the 2026-10-08 failure is fully re-armed, AND
|
||||
# VpnIngressNodeAnnouncesL2VIP is structurally INERT: its join matches nothing and returns no
|
||||
# series, which is indistinguishable from healthy. Both halves are asserted here, because an
|
||||
# inert critical alert is the exact shape this repo has been burned by before.
|
||||
# =============================================================================================
|
||||
- interval: 1m
|
||||
name: vpn-ingress label absent everywhere - watchdog fires and the critical alert is inert
|
||||
input_series:
|
||||
# A lease held by the node that IS (in reality) the forwarded one -- but with no label
|
||||
# anywhere, nothing can tell. This is the dangerous state.
|
||||
- series: 'kube_lease_owner{job="kube-state-metrics",namespace="kube-system",lease="cilium-l2announce-gateway-cilium-gateway-shared",lease_holder="k3s-prod-1"}'
|
||||
values: '1+0x60'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-1"}'
|
||||
values: '1+0x60'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-2"}'
|
||||
values: '1+0x60'
|
||||
- series: 'kube_node_labels{job="kube-state-metrics",node="k3s-prod-3"}'
|
||||
values: '1+0x60'
|
||||
alert_rule_test:
|
||||
# The critical alert CANNOT fire -- this is the inertness being pinned, not an aside.
|
||||
- eval_time: 45m
|
||||
alertname: VpnIngressNodeAnnouncesL2VIP
|
||||
exp_alerts: []
|
||||
# ... which is precisely why the watchdog must.
|
||||
- eval_time: 45m
|
||||
alertname: VpnIngressNodeLabelMissing
|
||||
exp_alerts:
|
||||
- exp_labels:
|
||||
severity: warning
|
||||
# absent() propagates the matcher's own labels onto the series it returns, so the
|
||||
# alert inherits this one. Asserted rather than worked around: it is harmless, and
|
||||
# pinning it means a future change to the matcher cannot alter the alert's identity
|
||||
# (and therefore its Alertmanager grouping) unnoticed.
|
||||
label_homelab_keithjackson_dev_vpn_ingress: "true"
|
||||
exp_annotations:
|
||||
summary: "No node carries the vpn-ingress label -- VPN VIP protection is OFF"
|
||||
description: >-
|
||||
No node publishes homelab.keithjackson.dev/vpn-ingress=true, so
|
||||
l2-announcement-policy.yaml's `NotIn` exclusion matches every node and excludes none:
|
||||
any node may now announce a LoadBalancer VIP, including the one terminating
|
||||
remote-access WireGuard. `VpnIngressNodeAnnouncesL2VIP` is INERT while this fires, so
|
||||
the warning matters more than its severity suggests. Re-run the node_baseline role on
|
||||
the forwarded node, and confirm the label is in kube-state-metrics'
|
||||
metricLabelsAllowlist.
|
||||
runbook_url: "https://${BASE_DOMAIN}/runbooks/vpn-ingress-l2-collision"
|
||||
@@ -117,8 +117,19 @@ boundary around this port is WireGuard's peer authentication, full stop.
|
||||
The WAN forward names **one node address**, by hand, off-repo (same shape as the plex tcp/32400
|
||||
forward). A single-replica Deployment would float on every reschedule and silently invalidate it.
|
||||
A DaemonSet instead puts an identical listener on all three nodes — every node answers the same
|
||||
keys — so **losing the forwarded node is a one-line router change**, with no cluster work and no
|
||||
re-issued client profiles. The two idle listeners cost ~15 MiB each.
|
||||
keys — so losing the forwarded node never costs a re-issued client profile. The two idle listeners
|
||||
cost ~15 MiB each.
|
||||
|
||||
> **This section used to say "losing the forwarded node is a one-line router change, with no cluster
|
||||
> work". It was wrong, and it cost two days of silently broken remote access on 2026-10-08.** The
|
||||
> nodes are warm, but they are **not interchangeable**: the node the forward points at must be the
|
||||
> one labelled `homelab.keithjackson.dev/vpn-ingress=true`, because that is the node
|
||||
> `platform-config/cilium/l2-announcement-policy.yaml` excludes from announcing LoadBalancer VIPs.
|
||||
> **A node cannot reach a VIP whose Cilium L2 announcement lease it holds itself** (see "Measured on
|
||||
> prod, 2026-10-08" below), and since this pod MASQUERADEs clients to the node's own address, a VPN
|
||||
> client inherits that. Failover is therefore **two** changes: move
|
||||
> `node_baseline_vpn_ingress_node` and re-run that role, then re-point the router —
|
||||
> `runbooks/vpn-remote-access.md` step 3.
|
||||
|
||||
A pinned `LoadBalancer` VIP was the considered alternative (it would make the forward immune to node
|
||||
replacement). Rejected: it reintroduces the L2-announcement lease, SNAT, and an extra node-to-node
|
||||
@@ -186,6 +197,45 @@ right way; one was a defect in my own manifest.
|
||||
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
|
||||
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
|
||||
|
||||
## Measured on prod, 2026-10-08 — why the forwarded node cannot be any node
|
||||
|
||||
The first device enrolled, handshook, resolved names, and could not open a single connection to any
|
||||
cluster web app. The tunnel was not at fault — every part of it measured healthy:
|
||||
`net.ipv4.ip_forward=1`, `wg0` up at `10.90.0.1/24` MTU 1420, the peer authorised with real
|
||||
bidirectional transfer (346 KiB in / 200 KiB out), `MASQUERADE` present and counting 26 flows,
|
||||
`FORWARD -i/-o wg0 ACCEPT` counting 108 / 171 packets, and PiHole DNS answering from the node.
|
||||
|
||||
**The fault was that a node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it
|
||||
holds itself.** Cilium's announcer answers ARP for the VIP on behalf of other LAN hosts, but not for
|
||||
the announcing node's own request, so the neighbour entry settles at `FAILED`:
|
||||
|
||||
| from | `192.168.2.20` (Gateway) | `192.168.2.22` (crafty) | `192.168.2.24` (plex) | announces |
|
||||
|---|---|---|---|---|
|
||||
| `k3s-prod-1` | **FAILED** | — | — | `.20` |
|
||||
| `k3s-prod-2` | ok | **INCOMPLETE** | ok | `.21`, `.22`, `.23-tcp` |
|
||||
| `k3s-prod-3` | ok | ok | **INCOMPLETE** | `.24` |
|
||||
|
||||
Symmetric across 3 nodes × 5 VIPs: each node reached every VIP **except** the ones it alone
|
||||
announced. It is a property of L2 announcement, not of any one Service, and it is harmless for
|
||||
ordinary pods — they reach Services through the cluster datapath and never ARP for a VIP. This
|
||||
component is the exception, because `hostNetwork` + `MASQUERADE` put remote clients on the node's
|
||||
own LAN path.
|
||||
|
||||
The forward pointed at `k3s-prod-1`, which held the `shared` Gateway lease, so **all 16 Gateway
|
||||
hostnames were dead over the VPN**. Two things hid it:
|
||||
|
||||
* **PiHole survived by accident.** `192.168.2.23` is announced by *two* Services
|
||||
(`pihole-dns-udp` on prod-1, `pihole-dns-tcp` on prod-2), so prod-2 answered the ARP prod-1 could
|
||||
not. Names resolved perfectly and every app hung — the hardest shape to diagnose from a client.
|
||||
* **Nothing reported it.** No probe in this repo compares the forwarded node against the lease
|
||||
holders, and from outside WireGuard is silent to unauthenticated traffic anyway.
|
||||
|
||||
Closed by labelling the forwarded node `homelab.keithjackson.dev/vpn-ingress=true` and excluding
|
||||
that label in `platform-config/cilium/l2-announcement-policy.yaml` (the node then ARP-resolves every
|
||||
VIP), with `VpnIngressNodeAnnouncesL2VIP` watching the lease holders so a recurrence is loud and
|
||||
`VpnIngressNodeLabelMissing` watching for the label going absent, which would silently re-arm the
|
||||
whole failure. Cost: VIP announcement candidacy drops from three nodes to two.
|
||||
|
||||
## Prerequisites outside this directory
|
||||
|
||||
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
|
||||
|
||||
@@ -62,9 +62,39 @@
|
||||
# as the plex tcp/32400 forward — runbooks/plex-remote-access.md). A Deployment would float to
|
||||
# whichever node the scheduler picked and silently invalidate that forward on every reschedule. A
|
||||
# DaemonSet instead puts an identical warm listener on ALL THREE nodes: the forward names one of
|
||||
# them, and if that node is lost, FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work and no
|
||||
# new client profiles — every node answers the same keys. Each node binds udp/51820 in its own
|
||||
# netns, so there is no conflict, and the two idle listeners cost ~15 MiB each.
|
||||
# them, and if that node is lost every other node is already warm and answers the same keys, so no
|
||||
# client profile ever changes. Each node binds udp/51820 in its own netns, so there is no conflict,
|
||||
# and the two idle listeners cost ~15 MiB each.
|
||||
#
|
||||
# BUT THE NODES ARE NOT INTERCHANGEABLE, AND THIS COMMENT USED TO CLAIM THEY WERE.
|
||||
# It said "FAILOVER IS A ONE-LINE ROUTER CHANGE with no cluster work". That was wrong, and the
|
||||
# wording cost two days of silently broken remote access on 2026-10-08. MEASURED that day: a node
|
||||
# cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds ITSELF — the announcer
|
||||
# does not answer its own ARP request, so the neighbour entry settles at `FAILED` and packets to
|
||||
# that VIP die before leaving the node:
|
||||
#
|
||||
# k3s-prod-1, holding the shared Gateway lease: ip neigh show 192.168.2.20 -> FAILED
|
||||
# k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr …(prod-1)
|
||||
#
|
||||
# Because this pod MASQUERADEs clients to the node's own address (see PostUp below), a remote client
|
||||
# inherits exactly that failure. With the forward on the Gateway's announcing node, all 16 Gateway
|
||||
# hostnames were unreachable over the VPN while PiHole DNS still resolved — a tunnel that
|
||||
# handshakes, resolves names, and then hangs on every single app.
|
||||
#
|
||||
# THE FIX IS DECLARATIVE AND LIVES OUTSIDE THIS FILE: the forwarded node is labelled
|
||||
# homelab.keithjackson.dev/vpn-ingress=true by infra/ansible/roles/node_baseline (from
|
||||
# node_baseline_vpn_ingress_node), and platform-config/cilium/l2-announcement-policy.yaml excludes
|
||||
# that label from announcing ANY VIP, so the node can ARP-resolve all of them. A label rather than a
|
||||
# cluster-vars knob because the alert that guards this has to read it too, and the alert unit-test
|
||||
# harness refuses a postBuild placeholder in an expr. Failover is therefore TWO steps — move
|
||||
# node_baseline_vpn_ingress_node and re-run the role, THEN re-point the router
|
||||
# (runbooks/vpn-remote-access.md step 3).
|
||||
# `VpnIngressNodeAnnouncesL2VIP` (platform/monitoring/alerts.yaml) fires if the two collide again.
|
||||
#
|
||||
# This pod deliberately does NOT probe that itself. A readinessProbe on VIP reachability would wedge
|
||||
# a DaemonSet rollout behind an unrelated Gateway outage (maxUnavailable: 1 below), and a liveness
|
||||
# probe would tear down a tunnel that is carrying LAN traffic perfectly well. The condition is a
|
||||
# property of the CLUSTER's announcement layout, so it is watched there.
|
||||
#
|
||||
# All three nodes carry no taints (measured 2026-10-05), so no tolerations are needed for the
|
||||
# DaemonSet to cover the cluster; add them if a node is ever tainted.
|
||||
|
||||
@@ -119,6 +119,11 @@ configMapGenerator:
|
||||
- update-rollback=update-rollback.md
|
||||
- vpn-down=vpn-down.md
|
||||
- vpn-egress-stale=vpn-egress-stale.md
|
||||
# `VpnIngressNodeAnnouncesL2VIP` — the INBOUND remote-access VPN, not the egress tunnel the two
|
||||
# pages above cover. A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease
|
||||
# it holds itself, and vpn-ingress MASQUERADEs clients to the node's own address, so the
|
||||
# forwarded node must announce nothing (measured 2026-10-08).
|
||||
- vpn-ingress-l2-collision=vpn-ingress-l2-collision.md
|
||||
# Operator procedure, not an alert page. Served because the component it completes is INERT
|
||||
# until three off-repo manual steps are done (kernel module, server key, WAN port-forward) —
|
||||
# and because the one check that actually proves it cannot be run from inside the house.
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
# vpn-ingress-l2-collision — the VPN ingress node is announcing a LoadBalancer VIP
|
||||
|
||||
**Alerts:** `VpnIngressNodeAnnouncesL2VIP` (critical, 15m) and `VpnIngressNodeLabelMissing`
|
||||
(warning, 30m)
|
||||
**File:** `platform/monitoring/alerts.yaml`
|
||||
**Related:** `/runbooks/vpn-remote-access` is the setup and enrolment runbook for this component.
|
||||
`/runbooks/vpn-egress-stale` and `/runbooks/vpn-down` are the **other** VPN — the outbound PIA
|
||||
tunnel — and have nothing to do with this.
|
||||
|
||||
## What this means here
|
||||
|
||||
**A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
|
||||
announcer answers ARP for the VIP on behalf of other LAN hosts, but not for the announcing node's
|
||||
own request. On the lease holder the neighbour entry settles at `FAILED`, and every packet to that
|
||||
VIP is dropped before it leaves the node. Measured on prod 2026-10-08, symmetric across all three
|
||||
nodes and five VIPs — each node reached every VIP **except** the ones it alone announced.
|
||||
|
||||
For an ordinary workload this is harmless: pods reach Services through the cluster datapath and
|
||||
never ARP for a VIP. `platform/vpn-ingress` is the exception. It runs WireGuard with
|
||||
`hostNetwork: true` and MASQUERADEs remote clients to the node's own address, so a remote client's
|
||||
packets take that node's LAN path and inherit its ARP failures. **If the node the router forwards
|
||||
`udp/51820` to is also an L2 lease holder, every remote VPN client silently loses all access to the
|
||||
Services behind the VIPs it announces.**
|
||||
|
||||
The node terminating WireGuard is marked with `homelab.keithjackson.dev/vpn-ingress=true`, and
|
||||
`platform-config/cilium/l2-announcement-policy.yaml` excludes that label from announcing anything,
|
||||
so this should be unreachable.
|
||||
|
||||
**The symptom your users report will not sound like a network fault.** On 2026-10-08 the forward sat
|
||||
on `k3s-prod-1`, which held the `shared` Gateway lease, so all 16 Gateway hostnames were dead over
|
||||
the VPN — while PiHole DNS kept answering, because `192.168.2.23` is announced by *two* Services and
|
||||
`k3s-prod-2` held the other lease. So the tunnel handshakes, `wg show` looks perfect, names resolve,
|
||||
and **every app hangs on connect**. Expect "the VPN connects but nothing loads", not "the VPN is
|
||||
down".
|
||||
|
||||
Which Services are affected depends on which lease moved — read `{{ $labels.lease }}` from the
|
||||
alert. `cilium-l2announce-gateway-cilium-gateway-shared` is the worst case: it is every web app at
|
||||
once.
|
||||
|
||||
### If it is `VpnIngressNodeLabelMissing` that fired
|
||||
|
||||
That is the watchdog, and it means the protection is **switched off**, not that something is merely
|
||||
untidy. The exclusion is a `NotIn` on the label, and `NotIn` with an absent key **matches** — so
|
||||
with the label on no node, the policy excludes nothing and every node is an announcement candidate
|
||||
again. `VpnIngressNodeAnnouncesL2VIP` is simultaneously **inert**: its join matches nothing and
|
||||
returns no series, which is indistinguishable from healthy. Treat a quiet critical alert as
|
||||
meaningless until this warning is cleared. Usual cause: a node was rebuilt and `node_baseline` has
|
||||
not been re-run on it.
|
||||
|
||||
## First commands
|
||||
|
||||
Run cluster commands through the pinned toolbox, which supplies the kube context:
|
||||
|
||||
```bash
|
||||
# 1. WHO HOLDS WHAT, and WHO IS THE VPN NODE — the whole question, in two lines.
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n kube-system get lease | grep l2announce'
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod get nodes -L homelab.keithjackson.dev/vpn-ingress'
|
||||
# Exactly ONE node should carry vpn-ingress=true, and NO l2announce lease should name it.
|
||||
|
||||
# 2. IS THE EXCLUSION STILL IN THE LIVE POLICY? Expect a matchExpressions list with a
|
||||
# homelab.keithjackson.dev/vpn-ingress NotIn [true] entry — not a bare matchLabels.
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod get ciliuml2announcementpolicy default-l2 -o jsonpath="{.spec.nodeSelector}"; echo'
|
||||
|
||||
# 3. IS THE LABEL REACHING PROMETHEUS? If this is empty the alerts are inert even when
|
||||
# the label is on the node (kube-state-metrics metricLabelsAllowlist).
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n monitoring exec deploy/kube-prometheus-stack-kube-state-metrics -- \
|
||||
wget -qO- localhost:8080/metrics | grep vpn_ingress | head'
|
||||
```
|
||||
|
||||
**Confirm the actual damage**, which is the one measurement that is unambiguous. From the WireGuard
|
||||
pod on the affected node (it shares the host netns, so this is the path a VPN client takes):
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress get pods -o wide' # pick the pod on that node
|
||||
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec <pod> -c wireguard -- \
|
||||
sh -c "ping -c1 -W1 192.168.2.20 >/dev/null 2>&1; ip neigh show 192.168.2.20"'
|
||||
# BROKEN: 192.168.2.20 dev eno1 FAILED
|
||||
# HEALTHY: 192.168.2.20 dev eno1 lladdr bc:e9:2f:7e:c4:a6 STALE|REACHABLE|DELAY
|
||||
```
|
||||
|
||||
The `ping` is only there to trigger ARP — the VIP does not answer ICMP even when healthy, so
|
||||
**ignore its 100% packet loss**; the neighbour state is the answer. Substitute the VIP that belongs
|
||||
to `{{ $labels.lease }}` if the Gateway is not the one that moved.
|
||||
|
||||
**Which node actually holds the forward** (the router cannot be queried, but the cluster knows):
|
||||
only the forwarded node's `wg show` has a peer with an `endpoint` and a recent `latest handshake`.
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'for p in $(kubectl --context k3s-prod -n vpn-ingress get pods -o name); do
|
||||
echo "== $p"; kubectl --context k3s-prod -n vpn-ingress exec ${p#pod/} -c wireguard -- wg show; done'
|
||||
```
|
||||
|
||||
## Likely causes, most likely first
|
||||
|
||||
1. **The label is on the wrong node** — someone re-pointed the router's forward during a node
|
||||
failover and did not move `node_baseline_vpn_ingress_node`. This is the most likely cause by a
|
||||
wide margin, because the router is off-repo and nothing can check it. Confirm with the `wg show`
|
||||
loop above: the node with the live peer endpoint is the real forward target.
|
||||
*Fix:* merge a PR setting `node_baseline_vpn_ingress_node`
|
||||
(`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`) to that node, then re-run the role —
|
||||
`ansible-playbook -i inventory/hosts.ini <playbook> -e k3s_hosts=k3s_prod --tags vpn-ingress`.
|
||||
The role labels the new node and **removes** the label from the old one. Then confirm Cilium
|
||||
re-elected the lease off the newly-labelled node.
|
||||
|
||||
2. **The exclusion was edited out of `platform-config/cilium/l2-announcement-policy.yaml`**, or the
|
||||
file was reverted to a bare `matchLabels: {kubernetes.io/os: linux}`. Command 2 above shows this
|
||||
directly. *Fix:* restore the `matchExpressions` form; the file's own header explains why it is
|
||||
shaped that way.
|
||||
|
||||
3. **The label is missing entirely** (`VpnIngressNodeLabelMissing` firing) — a node rebuild without
|
||||
a `node_baseline` re-run. *Fix:* re-run the role as above.
|
||||
|
||||
4. **The label is on the node but not in Prometheus** — `metricLabelsAllowlist` in
|
||||
`platform/monitoring/kube-prometheus-stack.yaml` lost the `vpn-ingress` entry. Then the critical
|
||||
alert is inert *and* the watchdog fires. Command 3 above distinguishes this from cause 3.
|
||||
|
||||
5. **Cilium kept renewing a lease its holder is no longer eligible for.** The exclusion is correct
|
||||
and in place, but the lease never re-elected. This was never verified when the fix was authored
|
||||
(see `decisions/2026-10-17-vpn-ingress-l2-lease-collision.md`). *Fix:* delete that one Lease
|
||||
object to force an election — `kubectl -n kube-system delete lease <name>` — and watch it come
|
||||
back on a different node.
|
||||
|
||||
6. **A Cilium upgrade changed `NotIn` semantics or the `nodeSelector` field.** Least likely, but it
|
||||
is the reason this is a runtime alert rather than a static check on the policy text.
|
||||
|
||||
## When to escalate or stop
|
||||
|
||||
**Act now, don't wait for a window.** Remote access is fully broken for the affected VIPs while this
|
||||
is firing, and it does **not** self-heal: a lease holder keeps renewing, so this condition persists
|
||||
indefinitely once entered. That is also why the 15m `for:` is safe — a transient move during a node
|
||||
reboot or a Cilium roll resolves on its own well inside it.
|
||||
|
||||
**The fastest honest mitigation is to move the forward, not the leases.** Re-pointing the router's
|
||||
`udp/51820` at a node that holds no lease for the affected VIP restores service in seconds, with no
|
||||
cluster change and no new client profiles — all three nodes answer the same keys. Follow it with the
|
||||
`node_baseline_vpn_ingress_node` PR and role re-run so the two agree again; until you do, this alert
|
||||
is watching the wrong node. Note each node's own announcements when choosing: as of 2026-10-08,
|
||||
prod-2 announced crafty `192.168.2.22` and unifi `192.168.2.21`, prod-3 announced plex
|
||||
`192.168.2.24`, so neither is a cost-free target.
|
||||
|
||||
**Do not** `kubectl edit` the `CiliumL2AnnouncementPolicy` to fix this — Flux owns it and will revert
|
||||
you, and the drift is the kind that outlives the memory of making it. The two acceptable hand
|
||||
actions are deleting a single Lease to force re-election (cause 5), and `kubectl label node` as a
|
||||
*temporary* stopgap while the PR lands — note that a node rebuild or a `node_baseline` run will undo
|
||||
a hand-applied label, so it is not a fix.
|
||||
|
||||
**Stop and escalate to the operator if** the leases will not leave the excluded node after a forced
|
||||
re-election, or if the intended forward target is unclear. Both need a decision about which node
|
||||
carries remote access, which is an operator call, not a runbook step.
|
||||
|
||||
**Known gap, so you are not misled:** if someone re-points the router without moving the label,
|
||||
`VpnIngressNodeAnnouncesL2VIP` watches the wrong node and **stays silent** while the VPN breaks in
|
||||
exactly the way described above. There is no in-cluster signal for that today; closing it needs a
|
||||
WireGuard exporter publishing which node is actually terminating peer traffic
|
||||
(`platform/vpn-ingress/README.md`, "Not built") — the `wg show` loop above is the manual substitute.
|
||||
Until then, if remote access is broken and both alerts are quiet, **check the forward against the
|
||||
label by hand before believing the alerts.**
|
||||
@@ -171,14 +171,57 @@ server without a `wg0` "ready".
|
||||
On the SOHO router at `192.168.2.1`, forward **WAN udp/51820** to **one node address**, e.g.
|
||||
`192.168.2.10:51820`. This is hand-made and off-repo; it is the half CI can never check.
|
||||
|
||||
Pick any node — all three run an identical listener. Note which one you chose here:
|
||||
> ### You may NOT pick any node. The forward target must be the `vpn-ingress`-labelled node.
|
||||
>
|
||||
> This step used to say "pick any node — all three run an identical listener". **That sentence
|
||||
> caused a two-day silent outage of all remote access on 2026-10-08**, and it is why there is now a
|
||||
> cluster-vars knob instead of a free choice here.
|
||||
>
|
||||
> **A node cannot reach a LoadBalancer VIP whose Cilium L2 announcement lease it holds itself.** The
|
||||
> announcer answers ARP for the VIP on behalf of other LAN hosts but not for itself, so on the lease
|
||||
> holder the neighbour entry settles at `FAILED` and packets to that VIP are dropped before they
|
||||
> leave the node. Measured on prod 2026-10-08:
|
||||
>
|
||||
> ```text
|
||||
> on k3s-prod-1 (held the shared Gateway lease): ip neigh show 192.168.2.20 -> FAILED
|
||||
> on k3s-prod-2 / k3s-prod-3: ip neigh show 192.168.2.20 -> lladdr … (prod-1)
|
||||
> ```
|
||||
>
|
||||
> WireGuard runs on `hostNetwork` and MASQUERADEs remote clients to the node's own address, so a VPN
|
||||
> client inherits that node's ARP failure. With the forward on `k3s-prod-1`, which held the Gateway
|
||||
> lease, **all 16 Gateway hostnames were dead over the VPN while PiHole DNS still resolved** — so
|
||||
> the tunnel handshook, names looked fine, and every single app hung on connect. Nothing reported
|
||||
> it.
|
||||
>
|
||||
`infra/ansible/roles/node_baseline` now labels the forwarded node
|
||||
> `homelab.keithjackson.dev/vpn-ingress=true` (from `node_baseline_vpn_ingress_node` in
|
||||
> `group_vars/k3s_prod/vars.yml`), and `platform-config/cilium/l2-announcement-policy.yaml` excludes
|
||||
> that label from announcing any VIP. **The forward must point at the node carrying that label.** CI
|
||||
> cannot read your router, so this parity is yours to hold — `VpnIngressNodeAnnouncesL2VIP` catches a
|
||||
> labelled node taking a lease, but it CANNOT catch the forward being moved without the label.
|
||||
|
||||
Forward to the node named by `node_baseline_vpn_ingress_node` (`k3s-prod-1` → `192.168.2.10`).
|
||||
Note it here:
|
||||
|
||||
```
|
||||
# WAN udp/51820 -> 192.168.2.__ (k3s-prod-_) set <date>
|
||||
# MUST equal node_baseline_vpn_ingress_node in group_vars/k3s_prod/vars.yml
|
||||
```
|
||||
|
||||
**Node loss = failover is this one line.** Re-point the forward at another node address; every
|
||||
device keeps working with the same profile, because all three nodes answer the same keys.
|
||||
**Node loss = failover is TWO changes, not one.** It used to be advertised as one router line; that
|
||||
was wrong, because re-pointing at another node lands the VPN on a node that announces VIPs and
|
||||
reintroduces the failure above against a different set of addresses (prod-2 announces crafty and
|
||||
unifi; prod-3 announces plex). Do both, in this order:
|
||||
|
||||
1. **Merge a PR** setting `node_baseline_vpn_ingress_node` to the new node, then re-run the role
|
||||
(`--tags vpn-ingress`). It labels the new node and removes the label from the old one; Cilium
|
||||
then moves the leases off the newly-labelled node and onto its peers.
|
||||
2. **Re-point the router** forward at the new node's address.
|
||||
|
||||
Every device keeps working with the same profile either way; all three nodes answer the same keys,
|
||||
and the listener stays warm on all three. Doing step 2 without step 1 gives you a tunnel that
|
||||
handshakes and routes nothing — and `VpnIngressNodeAnnouncesL2VIP` will **not** catch it, because
|
||||
that alert watches the node the knob names, not the node your router actually points at.
|
||||
|
||||
Do **not** also expose this on tcp, and do not change the port hoping for obscurity — WireGuard is
|
||||
already silent to unauthenticated probes (README, "Why this is safe to expose").
|
||||
|
||||
Reference in new issue
Block a user