Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
16894b7b73 |
No files matched your search
@@ -0,0 +1,311 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: feat/keycloak-passkeys
|
||||
title: Passkeys declared for the homelab realm — and the realm import will not apply them
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`platform/keycloak/keycloak-realm-import.yaml` now declares passkey (WebAuthn Passwordless)
|
||||
login for the `homelab` realm, in two parts:
|
||||
|
||||
1. **The WebAuthn Passwordless policy.** Ten `webAuthnPolicyPasswordless*` fields on the realm:
|
||||
RP ID `${BASE_DOMAIN}`, RP entity name `Homelab SSO`, signature algorithms `[ES256, RS256]`,
|
||||
`requireResidentKey: "Yes"`, `userVerificationRequirement: required`,
|
||||
`authenticatorAttachment: "not specified"`, `attestationConveyancePreference: none`,
|
||||
`acceptableAaguids: []`, `createTimeout: 300`, `avoidSameAuthenticatorRegister: true`.
|
||||
Deliberately the **passwordless** family only; the two-factor `webAuthnPolicy*` family is left
|
||||
at Keycloak's defaults because nothing in this realm uses `webauthn-authenticator`.
|
||||
|
||||
2. **The browser flow.** `browser-mfa` keeps its alias and stays bound as `browserFlow`; its
|
||||
interior is restructured so a passkey is an **ALTERNATIVE** beside the password form:
|
||||
|
||||
```
|
||||
browser-mfa (top level, unchanged)
|
||||
browser-mfa-forms ALTERNATIVE
|
||||
auth-username-form REQUIRED <- was auth-username-password-form
|
||||
browser-mfa-passkey-or-password REQUIRED <- NEW sub-flow
|
||||
webauthn-authenticator-passwordless ALTERNATIVE <- NEW: the passkey
|
||||
browser-mfa-password-and-otp ALTERNATIVE <- NEW sub-flow
|
||||
auth-password-form REQUIRED <- was half of the combined form
|
||||
browser-mfa-conditional-otp CONDITIONAL (unchanged, relocated intact)
|
||||
```
|
||||
|
||||
The design decisions, with their reasoning, are written into the manifest comments in this repo's
|
||||
usual density. The ones that matter:
|
||||
|
||||
* **Additive, not replacing. Enrolment optional, not forced.** This IdP gates Grafana, Gitea,
|
||||
Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the \*arr stack *and* `kubectl` via the
|
||||
`kubernetes` OIDC client. A mandatory passkey flow that misfires is a total lock-out from the
|
||||
platform and the cluster API at once, leaving only the Keycloak bootstrap admin and the
|
||||
break-glass cert kubeconfig. The gain from forcing it (one human stops typing a password) does
|
||||
not price that risk.
|
||||
* **No `requiredActions` block was added, on purpose.** Declaring one would have been the obvious
|
||||
way to add `webauthn-register-passwordless`, and it would have been actively dangerous — see
|
||||
"What I verified".
|
||||
* **RP ID is `${BASE_DOMAIN}`, not `keycloak.${BASE_DOMAIN}`.** A passkey's RP ID is fixed at
|
||||
enrolment and can never be re-keyed, so it must not be bound to a name that can move. This realm
|
||||
already declares an `auth.${BASE_DOMAIN}` redirect URI, and this repo already owns a reconciler
|
||||
built because a hostname move silently failed to reach the realm. Residual risk, unavoidable:
|
||||
changing `BASE_DOMAIN` itself invalidates every enrolled passkey.
|
||||
* **A passkey login is not also asked for TOTP.** `browser-mfa-conditional-otp` hangs off the
|
||||
password branch only. The justification is `userVerificationRequirement: required` — the
|
||||
ceremony already proves possession plus a verified human, phishing-resistantly and origin-bound.
|
||||
Stacking a replayable shared secret on top of that buys friction. The manifest names the exact
|
||||
one-line edit to reverse this decision.
|
||||
* **Login becomes two screens** (username, then password) for every user, passkey or not, because
|
||||
the server must know who is logging in before it can decide whether a passkey exists. Accepted
|
||||
cost; it is the enabling change.
|
||||
* **Usernameless login deliberately not used**, although Keycloak 26.0.6 supports it — see "What I
|
||||
verified" for the measured reason it would have made every login open on a passkey prompt.
|
||||
|
||||
`platform/keycloak/README.md` gains a `## Passkeys` section: the decision table, the fact that
|
||||
this does not reach the live realm, the two operator paths that would apply it, and the four-step
|
||||
browser verification nobody has run.
|
||||
|
||||
No secret, key, credential or AAGUID was minted or committed. No file under `clusters/**`,
|
||||
`conformance/**`, `ci/**`, `platform/kyverno/**`, `**/rbac.yaml` or `**/*.sops.yaml` was touched,
|
||||
so this should not trip `ci/approval-policy.sh`'s sensitive-path review requirement.
|
||||
|
||||
## What I verified
|
||||
|
||||
**The live realm import has already completed and will never re-run.** Measured on prod,
|
||||
2026-10-17:
|
||||
|
||||
```
|
||||
$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
|
||||
{"conditions":[{"message":"","status":"True","type":"Done"},
|
||||
{"message":"","status":"False","type":"Started"},
|
||||
{"message":"","status":"False","type":"HasErrors"}]}
|
||||
```
|
||||
|
||||
`Done=True` / `Started=False`, exactly the one-shot state `platform/keycloak/README.md` already
|
||||
documents for client secrets and client URLs. **So the change in this PR has zero effect on how
|
||||
anyone logs in today**, and the client-secret/client-URL reconciler does not and cannot close the
|
||||
gap: its hard endpoint allow-list makes `/authentication` unreachable by design, and realm-level
|
||||
policy fields are not client fields.
|
||||
|
||||
**The CRD accepts every field I used.** Measured against the live CRD (not the upstream docs):
|
||||
`kubectl --context k3s-prod get crd keycloakrealmimports.k8s.keycloak.org` lists all twelve
|
||||
`webAuthnPolicyPasswordless*` properties with the types I wrote (`RequireResidentKey`,
|
||||
`RpEntityName`, `RpId`, `UserVerificationRequirement`, `AttestationConveyancePreference`,
|
||||
`AuthenticatorAttachment` as `string`; `SignatureAlgorithms`, `AcceptableAaguids`, `ExtraOrigins`
|
||||
as `array/string`; `CreateTimeout` as `integer`; `AvoidSameAuthenticatorRegister` as `boolean`),
|
||||
plus `authenticationFlows`, `authenticatorConfig` and `requiredActions`. The `spec.realm` schema
|
||||
has **no** `x-kubernetes-preserve-unknown-fields`, so a misspelled field would have been pruned
|
||||
silently rather than rejected — which is why this was checked rather than assumed.
|
||||
|
||||
**Running Keycloak is 26.0.6.** `quay.io/keycloak/keycloak:26.0.6` on `keycloak-0`/`keycloak-1`,
|
||||
operator `quay.io/keycloak/keycloak-operator:26.0.6`. Every source claim below is read from the
|
||||
`26.0.6` tag, not from memory or from docs.
|
||||
|
||||
**Every provider ID is quoted from upstream source, not guessed.**
|
||||
|
||||
| String I used | Where it is defined at tag 26.0.6 |
|
||||
|---|---|
|
||||
| `webauthn-authenticator-passwordless` | `WebAuthnPasswordlessAuthenticatorFactory.PROVIDER_ID`, display "WebAuthn Passwordless Authenticator" |
|
||||
| `auth-username-form` | `UsernameFormFactory.PROVIDER_ID`, display "Username Form" |
|
||||
| `auth-password-form` | `PasswordFormFactory.PROVIDER_ID`, display "Password Form" |
|
||||
| `webauthn-register-passwordless` | `WebAuthnPasswordlessRegisterFactory.PROVIDER_ID` (referenced in comments only) |
|
||||
|
||||
**`requiredActions` is REPLACE, not merge — measured, and it is why I did not declare one.**
|
||||
`model/storage-private/.../DefaultExportImportManager.java` lines 327-338:
|
||||
|
||||
```java
|
||||
if (rep.getRequiredActions() != null) {
|
||||
for (RequiredActionProviderRepresentation action : rep.getRequiredActions()) { ... }
|
||||
DefaultRequiredActions.addDeleteAccountAction(newRealm);
|
||||
} else {
|
||||
DefaultRequiredActions.addActions(newRealm);
|
||||
}
|
||||
```
|
||||
|
||||
Declaring one required action therefore skips the entire default set. `RealmManager` is no
|
||||
backstop: `setupRequiredActions` is `if (realm.getRequiredActionProvidersStream().count() == 0)`,
|
||||
and the count is already non-zero by then (`RepresentationToModel.importRealm` runs at
|
||||
`RealmManager.java:585`, the backstop at `:612`). A realm built that way would have had the passkey
|
||||
action plus `delete_account` and nothing else — **no `CONFIGURE_TOTP`**, which would leave
|
||||
`auth-otp-form` demanding a TOTP that no platform-admin could ever enrol: a hard lock-out
|
||||
delivered by a change whose purpose was to *add* a login method.
|
||||
|
||||
**And declaring it would have bought nothing.** `DefaultRequiredActions.addWebAuthnPasswordlessRegisterAction`
|
||||
registers `webauthn-register-passwordless` with `setEnabled(true)`, `setDefaultAction(false)`,
|
||||
priority 80, gated on `Profile.Feature.WEB_AUTHN` — which is `Type.DEFAULT` in
|
||||
`common/.../Profile.java:74`, i.e. on unless explicitly disabled. `enabled=true` /
|
||||
`defaultAction=false` *is* the additive-and-optional posture this change wants.
|
||||
|
||||
**`authenticationFlows` has the opposite semantics, which is why declaring flows is safe.**
|
||||
`importAuthenticationFlows` adds the declared flows, then backfills each unbound binding
|
||||
(browser, registration, direct grant, reset credentials, client auth, first broker login, docker
|
||||
auth) from the built-in defaults individually (`DefaultExportImportManager.java` ~1320-1410).
|
||||
|
||||
**The graceful degradation to password is measured from the flow engine, not assumed.** For a user
|
||||
with no passkey: `WebAuthnPasswordlessAuthenticator.requiresUser()` returns `false`, so
|
||||
`DefaultAuthenticationFlow.processSingleFlowExecutionModel` takes the `else` branch (`daf:456-462`),
|
||||
whose four conditions all hold — a user is set (from `auth-username-form`), `configuredFor` is
|
||||
false, `WebAuthnPasswordlessAuthenticatorFactory.isUserSetupAllowed()` is `false`, and
|
||||
`WebAuthnAuthenticator implements Authenticator, CredentialValidator<...>` — so it raises
|
||||
`CREDENTIAL_SETUP_REQUIRED`. The ALTERNATIVE loop (`daf:289-302`) catches exactly that, marks the
|
||||
execution `ATTEMPTED` and proceeds to the next alternative, with the upstream comment *"the user,
|
||||
since he has alternatives, should be able to go to another alternative and continue"*. Hence: no
|
||||
passkey, no passkey screen, straight to the password form.
|
||||
|
||||
**Why usernameless was rejected, measured.** The `configuredFor` guard sits inside the
|
||||
`requiresUser()` branch. With `requiresUser()` false and no user yet in context, that guard is
|
||||
skipped and the engine just calls `authenticate(context)` — so a passkey execution placed before
|
||||
the username form is offered to *everyone*, including users who will never own one.
|
||||
|
||||
**The policy sentinels are read from the shipped code, and one of my initial assumptions was
|
||||
wrong.** `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED = "not specified"` is the
|
||||
realm-representation sentinel — but `WebAuthnConstants.OPTION_NOT_SPECIFIED = ""` (empty string) is
|
||||
the WebAuthn-API-level one. I checked both because they disagree. The login theme's
|
||||
`webauthnRegister.js` at 26.0.6 settles the behaviour:
|
||||
|
||||
```js
|
||||
if (input.requireResidentKey !== 'not specified') {
|
||||
if (input.requireResidentKey === 'Yes') { authenticatorSelection.requireResidentKey = true; }
|
||||
else { authenticatorSelection.requireResidentKey = false; }
|
||||
}
|
||||
...
|
||||
if (input.createTimeout !== 0) { publicKey.timeout = input.createTimeout * 1000; }
|
||||
rp: {id: input.rpId, name: input.rpEntityName}
|
||||
```
|
||||
|
||||
So `"Yes"` is a literal string compare (any other non-sentinel value means *false*), `"not
|
||||
specified"` is compared literally for four fields, and `createTimeout` is **seconds** with `0`
|
||||
meaning "send no timeout". It also shows 26.0.6 emits only the legacy `requireResidentKey` member,
|
||||
never `residentKey: "required"` — the client maps that per WebAuthn L2, but the policy cannot
|
||||
express `residentKey: "preferred"` at all.
|
||||
|
||||
`[ES256, RS256]` is exactly `WebAuthnPolicy.DEFAULT_POLICY` /
|
||||
`Constants.DEFAULT_WEBAUTHN_POLICY_SIGNATURE_ALGORITHMS` — declared explicitly, not changed.
|
||||
`getWebAuthnPolicyPasswordless` was read field by field to confirm `[]` for `acceptableAaguids`
|
||||
means "no restriction" and that `RpEntityName` must be non-empty or the **update** path
|
||||
(`DefaultExportImportManager.java:842`) drops the whole policy silently.
|
||||
|
||||
**The manifest parses and renders.** `yaml.safe_load` round-trips the file and the flow graph reads
|
||||
back exactly as designed (five flows, the executions and priorities above, `browserFlow:
|
||||
browser-mfa`, `requiredActions` absent). `python3 ci/render-keycloak-client-urls.py --check` →
|
||||
`client-urls.yaml is in sync`, rc 0 (no client URL changed, so no regeneration was needed).
|
||||
`yamllint -c ci/yamllint.yaml platform/keycloak/keycloak-realm-import.yaml` → rc 0; no line exceeds
|
||||
the 160-column ceiling.
|
||||
|
||||
**`browser-mfa` keeps its alias deliberately.** `conformance/tests/l2-behavioral/keycloak-config-as-code`
|
||||
asserts `kubectl get keycloakrealmimport homelab -o jsonpath={.spec.realm.browserFlow} | grep -q
|
||||
'browser-mfa'`. Renaming the top-level flow would have failed that test.
|
||||
|
||||
**Envsubst safety.** `platform-config`'s Flux Kustomization declares
|
||||
`postBuild.substituteFrom: cluster-vars` (`clusters/base/platform-config.yaml:26-29`), so this file
|
||||
is substituted. The only brace token I added is `${BASE_DOMAIN}`, which is declared at
|
||||
`clusters/base/cluster-vars.yaml:42` (`home.keithjackson.dev`). Every comment I wrote avoids braces
|
||||
entirely — the RP-ID discussion says `keycloak.<BASE_DOMAIN>` with angle brackets precisely so an
|
||||
unparseable token cannot fail the whole Kustomization.
|
||||
|
||||
**Gate result: rc 3, zero failures, `gitea-settings` the only stage the gate counts as not-run** —
|
||||
the documented honest best local outcome. Run in the pinned toolbox from this worktree with all
|
||||
three §3 workarounds (`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main
|
||||
checkout's `.git` and the sibling spec repo with `SPEC_REPO=/home-network-k3s`):
|
||||
|
||||
```
|
||||
[ ok ] stage spec-immutability: pass <- NOT skipped; both shim mounts worked, R-TEST-06 covered
|
||||
[ ok ] stage decisions-log: pass <- archive frozen, 48 entries well-formed
|
||||
[ ok ] stage lint: pass
|
||||
[ ok ] stage l0: pass <- incl. flux-postbuild-substitution + keycloak-client-secret-reconciler
|
||||
[ ok ] stage render-golden: pass
|
||||
[ ok ] stage l1: pass
|
||||
[ ok ] stage approval-policy: pass <- no sensitive path touched, so no non-author review needed
|
||||
[skip] stage gitea-settings: SKIPPED <- needs SETTINGS_API_TOKEN, CI-only; expected
|
||||
[skip] stage conformance: SKIPPED <- cluster-gated L2/L3; expected
|
||||
MERGE GATE: UNDETERMINED — Stages that did not run: gitea-settings
|
||||
```
|
||||
|
||||
No lint finding and no check failure names either file I changed. Touched requirements resolved to
|
||||
`R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06`; coverage 51/51, 0 gaps.
|
||||
|
||||
**A note on how that number was obtained, because the first attempt was worthless.** My first gate
|
||||
run returned the same rc 3 and the same seven passes — against a tree that did not contain this
|
||||
change at all. The gate's `tracked_files()` reads **git**, and my edits were still uncommitted, so
|
||||
every stage gated `origin/main`. The tell was in the log and I nearly missed it: the
|
||||
touched-requirement subset reported UNDETERMINED with *"check out the feature branch you mean to
|
||||
gate (not the base branch)"*, and `decisions-log` counted **47** entries rather than 48. **A green
|
||||
gate run on an uncommitted worktree is a gate run on the base branch.** The numbers above are from
|
||||
the second run, after committing and rebasing.
|
||||
|
||||
**The base moved under me mid-run.** `origin/main` advanced from `9a41a8e` to `1a0b208` (PR #70,
|
||||
oauth2-proxy OIDC startup race) while I was working. This branch is rebased onto `1a0b208`, so the
|
||||
gate above ran on the base CI will actually merge into. The rebase was clean — PR #70 touched
|
||||
`platform/oauth2-proxy/helmrelease.yaml` and its own decisions entry, disjoint from these files.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
**This change is unapplied. It changes nothing about the live realm, and I did not attempt to make
|
||||
it.** `Done=True` / `Started=False` was measured, not inferred, and I ran no mutating `kubectl`,
|
||||
no `flux reconcile`, no admin-API call and no `kcadm`. Whether the operator's eventual
|
||||
admin-console reconstruction of this flow matches what is declared here is, by construction,
|
||||
unverifiable from git.
|
||||
|
||||
**I never enrolled a passkey and never pointed a browser at any of this.** Nothing here has been
|
||||
exercised by a real WebAuthn ceremony on any device. Specifically unverified:
|
||||
|
||||
* **That the RP ID works.** `home.keithjackson.dev` is a registrable-domain suffix of
|
||||
`keycloak.home.keithjackson.dev`, so it *should* satisfy the browser's check — but I verified
|
||||
that by reading the WebAuthn rule, not by watching a browser accept it. A wrong RP ID fails
|
||||
**silently, in the browser, before Keycloak is involved**, with no server-side log line. If
|
||||
enrolment produces no OS prompt and no error, this is the first suspect.
|
||||
* **That `"not specified"` is the right literal for `authenticatorAttachment` in a realm import.**
|
||||
I traced it to `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED` and to the theme's string
|
||||
compare, but I have not seen a realm import accept it. Omitting the field would default to the
|
||||
same value; if an import ever rejects the literal, drop the line.
|
||||
* **That `createTimeout: 300` behaves.** Seconds-vs-milliseconds is read from the theme JS. The
|
||||
claim that 300 is long enough for the cross-device QR/hybrid flow, and that Chrome clamps rather
|
||||
than rejects it, is reasoning about browser behaviour, not a measurement.
|
||||
* **Whether a passkey user can still reach the password form.** I argued it from
|
||||
`createAuthenticationSelectionList` producing a "Try another way" option, and I noted that for a
|
||||
user holding both credentials the *first* screen is chosen by the user's own credential ordering
|
||||
rather than by my priorities. Neither was observed.
|
||||
* **The two-screen login regression for ordinary users.** `auth-username-form` +
|
||||
`auth-password-form` replacing `auth-username-password-form` is a visible change for every user
|
||||
including non-passkey family accounts, and nobody has logged in through it.
|
||||
* **That conditional OTP for platform-admins still fires on the password branch.** The sub-flow and
|
||||
its `conditional-otp-platform-admin` config are byte-identical and merely relocated one level
|
||||
deeper, but "identical YAML" is not "still enforced at runtime". This is the highest-value
|
||||
regression check and it is untested. The previous breakage of this exact step
|
||||
(`condUserRole` vs a non-existent property key) 403'd every platform-admin login.
|
||||
* **That a platform-admin authenticating by passkey is not challenged for TOTP.** That is the
|
||||
intended behaviour of the shape I wrote; it is unobserved, and it is also a deliberate
|
||||
relaxation of R-IDP-04 as previously implemented, which the operator may want to reverse.
|
||||
|
||||
**Whether the realm import reconciles into an existing realm: measured as NO for the completed
|
||||
one-shot, but I did not test a re-import.** I did not delete and recreate the
|
||||
`KeycloakRealmImport`, so I cannot say what a re-run would do to the existing `homelab` realm —
|
||||
whether it would merge the new policy and flows, duplicate flow aliases, or fail. The 2026-09-06
|
||||
total SSO outage documented in `platform/keycloak/README.md` is the reason I did not experiment.
|
||||
**Treat the re-import path in the README as untested advice, not a runbook.**
|
||||
|
||||
**Nothing was proved about the live realm's current state.** I did not read the realm's existing
|
||||
required actions, its existing WebAuthn policies, or its live flow executions — doing so needs the
|
||||
admin credential and I did not decrypt or use it. So I cannot confirm that the live realm's
|
||||
required-action set actually contains `webauthn-register-passwordless` today; I can only show that
|
||||
Keycloak 26.0.6's defaults register it and that this repo's import never declared any required
|
||||
actions to suppress them. **That inference has not been checked against the realm.**
|
||||
|
||||
**L2/L3 conformance did not run** (no cluster from the gate container; `vars.CLUSTER_AVAILABLE` is
|
||||
unset in CI anyway), so the existing `keycloak-config-as-code` behavioural test did not re-execute
|
||||
against this change. I reasoned about its assertion by reading it.
|
||||
|
||||
**No new conformance test was added.** Nothing in `conformance/` asserts the passkey policy or the
|
||||
new flow shape, so a future edit can silently remove either and no gate stage will notice. That is
|
||||
a real gap this PR leaves open; closing it would touch `conformance/**`, which is a sensitive path
|
||||
needing non-author review.
|
||||
|
||||
**The quoted gate run predates this file's own gate-result paragraph.** It gated commit `207f5db`;
|
||||
the gate numbers and the rebase note above were then written into this entry, so the pushed tree
|
||||
differs from the gated one by this handoff's prose only — no manifest, no check, no generated file.
|
||||
`decisions-log` is the only stage that reads this file, and it validates structure (frontmatter,
|
||||
the three headings, non-stub bodies), which is unchanged. I did not re-run the full gate for a
|
||||
prose edit; CI will gate the pushed commit.
|
||||
|
||||
**`gitea-settings` did not run locally** — it needs `SETTINGS_API_TOKEN`, which exists only in CI.
|
||||
I did not re-measure branch protection, the runner, or `required_approvals` against the Gitea API
|
||||
this session; the claims in `CLAUDE.md` §2 are taken on trust here and are explicitly flagged there
|
||||
as needing re-measurement.
|
||||
@@ -1,198 +0,0 @@
|
||||
---
|
||||
date: 2026-10-17
|
||||
branch: fix/vpn-ingress-l2-lease-collision
|
||||
title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong"
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it
|
||||
was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is
|
||||
not re-derived — it is very easy to re-derive from `ip neigh` alone.
|
||||
|
||||
### The claim, and how it died
|
||||
|
||||
The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2
|
||||
announcement lease it holds itself* — the announcer does not answer its own ARP request, the
|
||||
neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because
|
||||
`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
|
||||
own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded
|
||||
that label from announcing, with two alerts and four promtool tests behind it.
|
||||
|
||||
The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*`
|
||||
leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) —
|
||||
the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five
|
||||
VIPs and the other two nodes should reach **all**.
|
||||
|
||||
**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node:
|
||||
|
||||
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|
||||
|---|---|---|
|
||||
| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
|
||||
| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms |
|
||||
| `nc -z 192.168.2.20 443` | **OK** | OK |
|
||||
| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` |
|
||||
| `192.168.2.21:8443` | FAIL | **FAIL** (both) |
|
||||
|
||||
The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the
|
||||
asserted **effect is absent**. That is sufficient to falsify the causal claim without having to
|
||||
explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**.
|
||||
|
||||
### Why it is inert
|
||||
|
||||
No traffic path resolves a LoadBalancer VIP by ARP:
|
||||
|
||||
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
|
||||
`Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath
|
||||
is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP
|
||||
is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**.
|
||||
* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's
|
||||
backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet.
|
||||
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
|
||||
which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a
|
||||
bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT,
|
||||
not wire traffic.
|
||||
|
||||
### The two probes that misled me
|
||||
|
||||
Both were cited as evidence in the previous version of this entry.
|
||||
|
||||
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing
|
||||
artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong.
|
||||
* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a
|
||||
host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including
|
||||
the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it
|
||||
discriminates nothing.
|
||||
|
||||
**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then
|
||||
routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns
|
||||
probe cannot measure the forwarded path in either direction — it reports success where the forwarded
|
||||
path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not
|
||||
sharing a hook, and that is the assumption the original diagnosis rested on.
|
||||
|
||||
### Files
|
||||
|
||||
Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`,
|
||||
`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`,
|
||||
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`,
|
||||
`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted:
|
||||
`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`.
|
||||
|
||||
Kept, rewritten from scratch:
|
||||
|
||||
* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is
|
||||
inert and which probes lie, and the internet-egress facts below.
|
||||
* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster
|
||||
work, internet does not") that works outwards from the client profile, and a short section saying
|
||||
the L2 lease is a dead end, with the unifi `.21` note.
|
||||
|
||||
**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is
|
||||
proposed by this branch any more.**
|
||||
|
||||
### The internet-routing report: no server-side fix was needed, and none was made
|
||||
|
||||
The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full
|
||||
one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has
|
||||
emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`,
|
||||
2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and
|
||||
`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel
|
||||
feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that
|
||||
already exists. **I deliberately shipped no code change here** — writing one would have meant
|
||||
inventing a fix for a symptom I could not reproduce (see below).
|
||||
|
||||
The server side was measured end to end and is capable and correct, so the cause is in the client
|
||||
profile or the client. The triage section names the two candidates that fit the symptom exactly —
|
||||
a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and
|
||||
leak public traffic outside the tunnel — each with a one-line test that settles it.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on
|
||||
2026-10-17:
|
||||
|
||||
* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1`
|
||||
(`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again.
|
||||
* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a
|
||||
real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere.
|
||||
* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three
|
||||
nodes — so the successes are service-aware DNAT, not wire delivery.
|
||||
* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with
|
||||
`REACHABLE` neighbour entries. This is the measurement that destroys the original control.
|
||||
* **The datapath facts** quoted above from `cilium-dbg status --verbose` and
|
||||
`cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`:
|
||||
`kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`,
|
||||
`routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three.
|
||||
* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` →
|
||||
`dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP.
|
||||
So the 200 is not loopback delivery by the host stack — it is BPF.
|
||||
* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1`
|
||||
(read, never written); `iptables -t nat -S POSTROUTING` carries
|
||||
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0
|
||||
packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both
|
||||
present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it;
|
||||
`curl http://1.1.1.1/` returns `301` from each node.
|
||||
* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by
|
||||
`podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod
|
||||
`egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on
|
||||
`-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the
|
||||
home WAN IP, not PIA's.
|
||||
* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned
|
||||
`142.250.113.138`.
|
||||
* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one
|
||||
enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both
|
||||
`FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The
|
||||
`wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters.
|
||||
* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f`
|
||||
(2026-10-05) and never changed.
|
||||
* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could
|
||||
not test what a real remote client does, and — as established above — **no probe runnable from the
|
||||
pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a
|
||||
VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a
|
||||
LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one
|
||||
remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in
|
||||
the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this
|
||||
unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show`
|
||||
shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the
|
||||
client loads a cluster app — and repeat it on a node that holds leases and one that does not.*
|
||||
* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same
|
||||
reason. **I need a client connected** to learn it; the router is unreadable from the cluster.
|
||||
* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP
|
||||
failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes.
|
||||
My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**,
|
||||
rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A
|
||||
competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and
|
||||
`wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is
|
||||
needed for the falsification** — the asserted cause is present today without its effect. I did not
|
||||
retro-measure anything, and I did not read Cilium's source or issue tracker.
|
||||
* **The operator's report cannot be a description of "right now."** No client has connected since
|
||||
the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster
|
||||
services and the LAN but not the internet" must describe a session before the power-cycle or one
|
||||
that has since dropped. **I did not establish when it was observed, or with which profile.**
|
||||
* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not**
|
||||
find the fault, and I shipped no fix for it. The two candidates in the triage section (a
|
||||
`--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not
|
||||
measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line
|
||||
decides split vs full and halves the search — and a connected client for step 2.
|
||||
* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and
|
||||
lives only on the operator's machine and the device; it is not in the repo, and I did not look for
|
||||
it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing.
|
||||
* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without
|
||||
a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU
|
||||
would see large transfers hang while small requests succeed. The runbook's existing MTU note
|
||||
(drop to 1280) already covers the remedy.
|
||||
* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole
|
||||
IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a
|
||||
client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an
|
||||
interface with no IPv6 address.
|
||||
* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate,
|
||||
pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such.
|
||||
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1
|
||||
offline subset only. Nothing above is covered by a test that runs in CI.
|
||||
* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there
|
||||
is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating
|
||||
`kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material).
|
||||
* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).
|
||||
@@ -154,6 +154,74 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
|
||||
(it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
|
||||
a realm edit landed.
|
||||
|
||||
## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)
|
||||
|
||||
`keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
|
||||
policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
|
||||
**ALTERNATIVE** beside the password form. Design decisions and their reasoning are in the manifest
|
||||
comments, not repeated here. The headline choices:
|
||||
|
||||
| Decision | Value | One-line reason |
|
||||
|---|---|---|
|
||||
| Additive or replacing? | **Additive** — passkey *or* password | A misfiring mandatory flow locks the single operator out of every SSO app *and* kubectl |
|
||||
| Enrolment | **Optional**, never a default action | Same reason; `webauthn-register-passwordless` is already enabled-not-forced by default |
|
||||
| Relying-party ID | `${BASE_DOMAIN}` (not `keycloak.${BASE_DOMAIN}`) | An unre-keyable credential must not be bound to a hostname that can move |
|
||||
| User verification | `required` | Without it, mere possession of an unlocked device is a complete login |
|
||||
| Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
|
||||
| OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
|
||||
|
||||
### This change does NOT reach the live realm. At all.
|
||||
|
||||
Same one-shot failure this file already documents twice above, third field over. Measured on prod
|
||||
2026-10-17:
|
||||
|
||||
```
|
||||
$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
|
||||
{"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
|
||||
{"status":"False","type":"HasErrors"}]}
|
||||
```
|
||||
|
||||
`Done=True` / `Started=False` — the import has completed and **will not re-run**, so a policy block
|
||||
and three new sub-flows added to this file change nothing about how anyone logs in. The
|
||||
client-secret/client-URL reconciler does **not** close this gap and was never meant to: its hard
|
||||
endpoint allow-list makes `/authentication` unreachable **by design** (see the table above), and
|
||||
realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
|
||||
mapper or a changed scope still needs the realm import to actually re-run" case named above.
|
||||
|
||||
**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
|
||||
an operator applies it by hand.** Two paths, both operator actions:
|
||||
|
||||
1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
|
||||
for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
|
||||
match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
|
||||
into Username Form + Password Form and add the two sub-flows exactly as declared.
|
||||
2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
|
||||
the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
|
||||
is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
|
||||
|
||||
Either way the realm is then configured by hand and git is only the record of intent, which is a
|
||||
live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.
|
||||
|
||||
### Verifying it actually works (nobody has, as of 2026-10-17)
|
||||
|
||||
No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
|
||||
it, in order:
|
||||
|
||||
```sh
|
||||
# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
|
||||
# Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
|
||||
# 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
|
||||
# -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
|
||||
# is wrong for the origin and NOTHING will say so.
|
||||
# 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
|
||||
# prompt. A platform-admin must NOT be asked for TOTP on this path.
|
||||
# 4. regression (the important one): a user with NO passkey must still get the password screen,
|
||||
# and a platform-admin must still be asked for TOTP there.
|
||||
```
|
||||
|
||||
A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak
|
||||
is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.
|
||||
|
||||
## Exceptions that keep native auth (do not force SSO) — ADR-0014
|
||||
|
||||
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
|
||||
@@ -74,6 +74,142 @@ spec:
|
||||
bruteForceProtected: true
|
||||
permanentLockout: false
|
||||
sslRequired: external
|
||||
# ===========================================================================================
|
||||
# PASSKEYS — the WebAuthn PASSWORDLESS policy (R-IDP-04). 2026-10-17.
|
||||
# ===========================================================================================
|
||||
# Read this together with the browser-mfa flow at the bottom of the file: THIS block decides
|
||||
# what a passkey must BE, the FLOW decides when one is OFFERED. Keycloak keeps TWO independent
|
||||
# WebAuthn policies per realm and they are not interchangeable:
|
||||
#
|
||||
# webAuthnPolicy* -> the TWO-FACTOR policy, consumed by the `webauthn-authenticator`
|
||||
# authenticator (a security key presented AFTER a password).
|
||||
# webAuthnPolicyPasswordless* -> the PASSWORDLESS policy, consumed by
|
||||
# `webauthn-authenticator-passwordless` (a passkey presented
|
||||
# INSTEAD of a password). This is the one passkeys use.
|
||||
#
|
||||
# Setting the wrong family is a silent no-op: the passwordless authenticator reads
|
||||
# getWebAuthnPolicyPasswordless() only (WebAuthnPasswordlessAuthenticator.getWebAuthnPolicy),
|
||||
# so a policy written into the two-factor fields is simply never consulted by a passkey login.
|
||||
# Only the passwordless family is declared here; the two-factor family is deliberately left at
|
||||
# Keycloak's defaults because nothing in this realm uses `webauthn-authenticator` — TOTP
|
||||
# (auth-otp-form) is the second factor for platform-admins, see the flow section.
|
||||
#
|
||||
# ADDITIVE AND OPTIONAL — THE DECISION THIS BLOCK EXISTS TO MAKE
|
||||
# -------------------------------------------------------------
|
||||
# Passkeys are offered ALONGSIDE the password, never instead of it, and enrolment is NOT forced.
|
||||
# This is a single-operator homelab whose IdP gates Grafana, Gitea, Nextcloud, Headlamp,
|
||||
# Semaphore, PiHole, Crafty, the *arr stack AND kubectl itself (the `kubernetes` client below).
|
||||
# A mandatory passkey flow that misfires is therefore not an inconvenience, it is a total
|
||||
# lock-out from the whole platform AND from the cluster API in one step — and the only repair
|
||||
# paths left would be the Keycloak bootstrap admin and the break-glass cert kubeconfig. The
|
||||
# expected gain from forcing enrolment (one human stops typing a password) is small; the
|
||||
# expected loss is the platform. So: additive, optional, and password login keeps working for
|
||||
# every user who never enrols.
|
||||
#
|
||||
# Concretely that means: no `requiredActions` block is declared anywhere in this file (see the
|
||||
# long note in the flow section for why declaring one would be actively DANGEROUS), and the
|
||||
# passkey step sits as an ALTERNATIVE beside the password form rather than ahead of it.
|
||||
#
|
||||
# RELYING-PARTY ID — THE FIELD THAT SILENTLY BREAKS EVERY PASSKEY IF IT IS WRONG
|
||||
# ------------------------------------------------------------------------------
|
||||
# The RP ID is baked into every credential at enrolment and can never be changed afterwards:
|
||||
# a passkey created under one RP ID is unusable under another, with no error the user can act
|
||||
# on — the browser simply reports that no credential is available. It is also browser-enforced,
|
||||
# so it cannot be "fixed" server-side: per WebAuthn, the RP ID must be equal to, or a
|
||||
# registrable-domain suffix of, the origin's effective domain. Keycloak is served at
|
||||
# keycloak.<BASE_DOMAIN> (httproute.yaml, and the Keycloak CR's hostname), so the legal choices
|
||||
# are keycloak.<BASE_DOMAIN>, <BASE_DOMAIN>, or the registrable domain one level up. It is NOT
|
||||
# the full URL and NOT a scheme — "https://..." here would fail every ceremony.
|
||||
#
|
||||
# We choose BASE_DOMAIN, the shared platform suffix, NOT the keycloak host, for one concrete
|
||||
# reason drawn from this repo's own history: a credential that cannot be re-keyed must not be
|
||||
# bound to a name that can move. The oauth2-proxy client below ALREADY declares a redirect URI
|
||||
# on auth.<BASE_DOMAIN>, i.e. this platform already contemplates serving auth on a second
|
||||
# hostname; and the client-URL reconciler exists precisely because moving a hostname in git
|
||||
# silently fails to reach the realm (see README, "Client URLs: the same one-shot failure").
|
||||
# Binding passkeys to the keycloak subdomain would mean that relocating Keycloak invalidates
|
||||
# every enrolled passkey; binding them to BASE_DOMAIN survives it. The cost is scope: any
|
||||
# future WebAuthn relying party under this base domain could assert the same RP ID. In a
|
||||
# homelab where every such party is already behind THIS IdP that is not a trust boundary worth
|
||||
# buying. Note the residual, unavoidable risk: changing BASE_DOMAIN itself invalidates every
|
||||
# passkey, which is a re-enrolment, not a repair.
|
||||
#
|
||||
# Written as the cluster-var so it is substituted by the same Flux postBuild pass, with the same
|
||||
# value, as every hostname in this file — there is no second place the domain is decided.
|
||||
webAuthnPolicyPasswordlessRpId: ${BASE_DOMAIN}
|
||||
# The human-readable name the browser/OS shows in the passkey prompt and stores beside the
|
||||
# credential ("Save a passkey for Homelab SSO?"). Matches the realm displayName above so the
|
||||
# OS keychain entry is recognisable. It is also load-bearing on the UPDATE path: Keycloak's
|
||||
# updateRealm applies the passwordless policy ONLY when this field is non-empty
|
||||
# (DefaultExportImportManager line ~842), so an admin-API realm update that omits it drops the
|
||||
# whole policy silently.
|
||||
webAuthnPolicyPasswordlessRpEntityName: "Homelab SSO"
|
||||
# ES256 then RS256 — exactly Keycloak's own DEFAULT_POLICY, declared explicitly rather than
|
||||
# inherited so the realm's crypto is auditable from git. ES256 is what every platform
|
||||
# authenticator (Touch ID/Face ID, Windows Hello, Android) and every FIDO2 key produces;
|
||||
# RS256 is the fallback for older Windows Hello TPM-backed credentials. EdDSA/Ed25519 is
|
||||
# deliberately NOT offered: authenticator and browser support is uneven, and an authenticator
|
||||
# that picks an algorithm the verifier handles badly fails at enrolment, which is the worst
|
||||
# moment to discover it. Order is a preference list, not an allow-list ordering requirement.
|
||||
webAuthnPolicyPasswordlessSignatureAlgorithms:
|
||||
- ES256
|
||||
- RS256
|
||||
# DISCOVERABLE CREDENTIAL REQUIRED. "Yes" is a Keycloak sentinel, not a boolean, and the login
|
||||
# theme compares it as the literal string: webauthnRegister.js maps 'Yes' to
|
||||
# authenticatorSelection.requireResidentKey=true and ANY other non-sentinel value to false.
|
||||
# This is not optional for passkeys — a non-discoverable credential cannot be found by the
|
||||
# browser without the server first naming it, so it is a second factor, not a passkey.
|
||||
# (Keycloak 26.0.6 emits only the LEGACY requireResidentKey member, not the newer
|
||||
# residentKey:"required"; per WebAuthn L2 the client maps requireResidentKey=true onto
|
||||
# residentKey="required", so discoverable credentials are genuinely requested — but it also
|
||||
# means this policy cannot express residentKey:"preferred" at all.)
|
||||
webAuthnPolicyPasswordlessRequireResidentKey: "Yes"
|
||||
# USER VERIFICATION REQUIRED. The authenticator must verify the human (biometric or PIN) and
|
||||
# say so in the assertion. This is the field that makes a passkey login at least as strong as
|
||||
# the password it replaces: with verification merely "preferred", possession of an unlocked
|
||||
# phone or an unattended security key would be a COMPLETE login to every SSO-gated app — a
|
||||
# single-factor possession check, strictly weaker than the password+TOTP path beside it.
|
||||
# Required makes the passkey branch inherently two-factor (something you have + something you
|
||||
# are/know), which is the premise the flow section relies on when it does NOT ask a
|
||||
# passkey-authenticated platform-admin for TOTP as well.
|
||||
webAuthnPolicyPasswordlessUserVerificationRequirement: required
|
||||
# "not specified" is Keycloak's sentinel for "either kind of authenticator"
|
||||
# (Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED; the theme JS omits the WebAuthn member
|
||||
# entirely when it sees this exact string). Deliberately NOT narrowed: "platform" would reject
|
||||
# every YubiKey and every phone used as a cross-device passkey, and "cross-platform" would
|
||||
# reject Touch ID and Windows Hello — i.e. either choice silently halves the usable hardware.
|
||||
# The operator should be able to enrol a laptop AND a phone AND a hardware key.
|
||||
webAuthnPolicyPasswordlessAuthenticatorAttachment: "not specified"
|
||||
# ATTESTATION: none. Requesting "direct" would hand Keycloak an attestation statement it has
|
||||
# no trust anchors to validate here (no metadata service is configured) and nothing to compare
|
||||
# it against, since the AAGUID allow-list below is deliberately empty — so the statement would
|
||||
# be collected, unverifiable, and would add an extra "share model information?" consent prompt
|
||||
# on Safari/iOS for no security gain. Attestation is how you ENFORCE an approved-hardware list;
|
||||
# we do not have one, so asking for it is cost without benefit.
|
||||
webAuthnPolicyPasswordlessAttestationConveyancePreference: none
|
||||
# Empty = accept any authenticator model. An AAGUID allow-list is a hardware whitelist: it
|
||||
# would have to be maintained by hand, and every new phone, laptop or key the operator buys
|
||||
# would be rejected at enrolment with a message that does not explain itself. It is also moot
|
||||
# while attestation is "none" (above) — with no attestation statement there is no trustworthy
|
||||
# AAGUID to match against in the first place.
|
||||
webAuthnPolicyPasswordlessAcceptableAaguids: []
|
||||
# Ceremony timeout in SECONDS (the login theme multiplies by 1000; 0 means "send no timeout and
|
||||
# let the browser decide"). 300s is chosen over a tighter value because the slowest LEGITIMATE
|
||||
# path is the cross-device hybrid flow — Keycloak renders a QR code, the operator unlocks a
|
||||
# phone, scans it, and completes a biometric check — which routinely takes over a minute. A
|
||||
# 60-120s timeout looks tidy and converts that flow into an unexplained failure. Browsers clamp
|
||||
# this value (Chrome to at most 10 minutes), so 300 is honoured rather than reinterpreted.
|
||||
webAuthnPolicyPasswordlessCreateTimeout: 300
|
||||
# Send the user's existing passkeys as excludeCredentials at enrolment, so re-enrolling an
|
||||
# authenticator the user already registered is refused by the authenticator itself instead of
|
||||
# producing a second, indistinguishable credential in the account console. Cheap, and it keeps
|
||||
# "which passkey is which" answerable — which matters when the recovery story is "delete the
|
||||
# lost one".
|
||||
webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister: true
|
||||
# webAuthnPolicyPasswordlessExtraOrigins is intentionally NOT set. It exists to admit non-web
|
||||
# origins (native app facet IDs, e.g. android:apk-key-hash:...). Every ceremony here happens in
|
||||
# a browser on Keycloak's own origin, which is already covered by the RP ID above; adding
|
||||
# origins is only ever a widening of who may ask for these credentials.
|
||||
# --- Authorization model (R-IDP-06): groups drive access; platform-admins carry a realm role. ---
|
||||
roles:
|
||||
realm:
|
||||
@@ -528,14 +664,105 @@ spec:
|
||||
"access.token.claim": "true"
|
||||
"userinfo.token.claim": "true"
|
||||
"claim.name": groups
|
||||
# --- MFA for platform-admins (R-IDP-04): a browser flow whose OTP step is CONDITIONAL on the
|
||||
# platform-admin role. Non-admins log in without OTP; platform-admins must enroll/present TOTP.
|
||||
# Set as the realm browserFlow below. VERIFY at bring-up (see runbook): a malformed flow fails
|
||||
# the whole import — if so, the fallback is to comment this block + browserFlow and configure
|
||||
# the conditional-OTP-by-role flow via the admin console. ---
|
||||
# ===========================================================================================
|
||||
# MFA for platform-admins (R-IDP-04) + PASSKEY login (2026-10-17). The realm browserFlow is
|
||||
# bound to browser-mfa at the bottom of this file.
|
||||
# ===========================================================================================
|
||||
# Non-admins log in without OTP; platform-admins must enroll/present TOTP. VERIFY at bring-up
|
||||
# (see runbook): a malformed flow fails the whole import — if so, the fallback is to comment
|
||||
# this block + browserFlow and configure the flow via the admin console.
|
||||
#
|
||||
# THE SHAPE, AND WHY IT IS THIS SHAPE
|
||||
# -----------------------------------
|
||||
# browser-mfa (top level, bound as browserFlow)
|
||||
# auth-cookie ALTERNATIVE already-authenticated SSO cookie
|
||||
# identity-provider-redirector ALTERNATIVE (no IdPs configured; kept for parity)
|
||||
# browser-mfa-forms ALTERNATIVE the interactive path:
|
||||
# auth-username-form REQUIRED WHO are you (no password yet)
|
||||
# browser-mfa-passkey-or-password REQUIRED ...and PROVE it, one of two ways:
|
||||
# webauthn-authenticator-passwordless ALTERNATIVE a passkey, or
|
||||
# browser-mfa-password-and-otp ALTERNATIVE the password path:
|
||||
# auth-password-form REQUIRED password, then
|
||||
# browser-mfa-conditional-otp CONDITIONAL TOTP iff platform-admin
|
||||
#
|
||||
# WHY THE USERNAME/PASSWORD SCREEN HAD TO BE SPLIT IN TWO. The previous flow used the combined
|
||||
# `auth-username-password-form`, which collects both on one page. A passkey branch cannot live
|
||||
# behind that: the server must know WHICH user is logging in before it can decide whether a
|
||||
# passkey is even available, and the combined form has already taken the password by the time
|
||||
# it knows. So it is replaced by the `auth-username-form` / `auth-password-form` pair — the
|
||||
# same decomposition Keycloak's own passwordless recipe uses. THIS IS A VISIBLE UX CHANGE FOR
|
||||
# EVERY USER, passkey or not: login becomes two screens (username, then password) instead of
|
||||
# one. That cost is accepted deliberately — it is the enabling change, it is what Google and
|
||||
# Microsoft logins already do, and the alternative (below) is worse.
|
||||
#
|
||||
# WHY NOT USERNAMELESS ("just tap your passkey", no username screen at all). Keycloak 26.0.6
|
||||
# does support it: WebAuthnPasswordlessAuthenticator.requiresUser() returns FALSE, so the
|
||||
# authenticator can legally run as a first step with no user in context. We deliberately do not
|
||||
# do that, because of how the flow engine then behaves. In DefaultAuthenticationFlow the
|
||||
# "does this user actually have this credential?" guard sits INSIDE the requiresUser() branch;
|
||||
# with requiresUser() false and no user yet set, the engine skips that guard and simply invokes
|
||||
# the authenticator. A passkey execution placed before the username form is therefore offered
|
||||
# to EVERYONE, so every login — including every password-only user and every family member who
|
||||
# will never own a passkey — would open on a passkey prompt they must dismiss. That is the
|
||||
# opposite of additive. Keeping the passkey step AFTER auth-username-form means a user is in
|
||||
# context, which is what makes the graceful degradation below actually work.
|
||||
#
|
||||
# WHY THE PASSKEY BRANCH DEGRADES CLEANLY TO A PASSWORD (the property the whole "additive"
|
||||
# claim rests on). With a user in context, the engine takes the non-requiresUser branch and
|
||||
# raises CREDENTIAL_SETUP_REQUIRED when the user has no credential of this type and the
|
||||
# authenticator's factory does not allow self-setup — and WebAuthnPasswordlessAuthenticatorFactory
|
||||
# .isUserSetupAllowed() is false, and the authenticator is a CredentialValidator, so both
|
||||
# conditions hold. That exception is raised INSIDE the ALTERNATIVE loop, which catches it,
|
||||
# marks the execution ATTEMPTED and proceeds to the next alternative. Net effect: a user with
|
||||
# no passkey never sees a passkey screen and lands on the password form; a user WITH one gets
|
||||
# the passkey screen and keeps the password as a "Try another way" option. Note that for a
|
||||
# user holding BOTH, which screen appears FIRST is decided by Keycloak's authentication
|
||||
# selection list (the user's own credential ordering), not by the priorities below.
|
||||
#
|
||||
# WHY A PASSKEY LOGIN IS NOT ALSO ASKED FOR TOTP. browser-mfa-conditional-otp hangs off the
|
||||
# PASSWORD branch only, so a platform-admin who authenticates with a passkey is not challenged
|
||||
# for TOTP. That is intentional, and it rests on the policy above: userVerification is
|
||||
# `required`, so the authenticator has already proved possession of hardware AND verified the
|
||||
# human (biometric or PIN) in one phishing-resistant, origin-bound ceremony. Stacking TOTP on
|
||||
# top would add a replayable, phishable shared secret to a credential that is neither, for
|
||||
# friction on every login. If a future operator disagrees, the change is small and local: move
|
||||
# the browser-mfa-conditional-otp reference out of browser-mfa-password-and-otp and into
|
||||
# browser-mfa-passkey-or-password's parent as a REQUIRED step, so it applies to both branches.
|
||||
#
|
||||
# WHY THERE IS NO `requiredActions` BLOCK IN THIS FILE — DO NOT ADD ONE
|
||||
# --------------------------------------------------------------------
|
||||
# Enrolment uses Keycloak's `webauthn-register-passwordless` required action, and the obvious
|
||||
# move is to declare it here. DO NOT. In Keycloak 26.0.6 the realm import treats
|
||||
# requiredActions as REPLACE, NOT MERGE: DefaultExportImportManager.importRealm branches on
|
||||
# `if (rep.getRequiredActions() != null)` and, when the representation declares ANY, imports
|
||||
# exactly those plus delete_account — the `else` branch that installs the full default set
|
||||
# (DefaultRequiredActions.addActions) never runs. RealmManager.setupRequiredActions is no
|
||||
# backstop either: it fires only when the realm has ZERO required-action providers.
|
||||
#
|
||||
# So declaring one action would create this realm with that action and delete_account and
|
||||
# NOTHING ELSE — no CONFIGURE_TOTP, no UPDATE_PASSWORD, no VERIFY_EMAIL, no delete_credential,
|
||||
# no UPDATE_PROFILE. The first casualty would be the MFA requirement this very flow enforces:
|
||||
# with CONFIGURE_TOTP unregistered, a platform-admin who does not yet have TOTP could never
|
||||
# enrol one, while auth-otp-form below still demands it — a hard lock-out of every
|
||||
# platform-admin from every SSO-gated app, including Headlamp and kubectl, delivered by a
|
||||
# change whose stated purpose was to ADD a login method.
|
||||
#
|
||||
# And the declaration buys nothing, because the default set ALREADY CONTAINS IT.
|
||||
# DefaultRequiredActions registers `webauthn-register-passwordless` with enabled=true,
|
||||
# defaultAction=false, priority 80, gated on Profile.Feature.WEB_AUTHN — which is Type.DEFAULT
|
||||
# in 26.0.6, i.e. on unless explicitly disabled. enabled=true / defaultAction=false IS the
|
||||
# additive-and-optional posture this change wants: the action exists and can be triggered for
|
||||
# a user on demand, but it is not forced on anyone at next login. Declaring it would restate
|
||||
# the default while deleting ten siblings.
|
||||
#
|
||||
# (The symmetric question for authenticationFlows has the opposite answer, which is why THIS
|
||||
# block is safe: importAuthenticationFlows adds the declared flows and then backfills each
|
||||
# unbound binding — browser, registration, direct grant, reset credentials, client auth, first
|
||||
# broker login, docker auth — from the built-in defaults individually. Declaring flows does not
|
||||
# suppress the built-ins; declaring required actions does suppress them.)
|
||||
authenticationFlows:
|
||||
- alias: browser-mfa
|
||||
description: "Browser flow with conditional OTP required for platform-admins."
|
||||
description: "Browser flow: passkey or password, with conditional OTP for platform-admins."
|
||||
providerId: basic-flow
|
||||
topLevel: true
|
||||
builtIn: false
|
||||
@@ -555,13 +782,63 @@ spec:
|
||||
priority: 30
|
||||
flowAlias: browser-mfa-forms
|
||||
userSetupAllowed: false
|
||||
# Identify the user FIRST (username only), then prove it. Both steps are REQUIRED: the
|
||||
# username is never optional, and neither is proving possession of the account.
|
||||
- alias: browser-mfa-forms
|
||||
description: "Username/password then conditional OTP."
|
||||
description: "Username, then passkey or password (+ conditional OTP)."
|
||||
providerId: basic-flow
|
||||
topLevel: false
|
||||
builtIn: false
|
||||
authenticationExecutions:
|
||||
- authenticator: auth-username-password-form
|
||||
# auth-username-form = Keycloak's "Username Form" (UsernameFormFactory) — resolves the
|
||||
# user WITHOUT taking a password, which is the prerequisite for offering a passkey.
|
||||
# Replaces the previous auth-username-password-form; see the header for the UX cost.
|
||||
- authenticator: auth-username-form
|
||||
authenticatorFlow: false
|
||||
requirement: REQUIRED
|
||||
priority: 10
|
||||
userSetupAllowed: false
|
||||
- authenticatorFlow: true
|
||||
requirement: REQUIRED
|
||||
priority: 20
|
||||
flowAlias: browser-mfa-passkey-or-password
|
||||
userSetupAllowed: false
|
||||
# THE FORK. Two ALTERNATIVEs, so satisfying either one completes the flow. Priority puts the
|
||||
# passkey first for users who have one; users who do not fall through to the password branch
|
||||
# automatically (CREDENTIAL_SETUP_REQUIRED is caught by the ALTERNATIVE loop — see header).
|
||||
- alias: browser-mfa-passkey-or-password
|
||||
description: "A passkey (WebAuthn passwordless), or password with conditional OTP."
|
||||
providerId: basic-flow
|
||||
topLevel: false
|
||||
builtIn: false
|
||||
authenticationExecutions:
|
||||
# webauthn-authenticator-passwordless = "WebAuthn Passwordless Authenticator"
|
||||
# (WebAuthnPasswordlessAuthenticatorFactory). It reads the realm's PASSWORDLESS policy
|
||||
# declared above — NOT the two-factor webAuthnPolicy* family. Keep it ALTERNATIVE: as
|
||||
# REQUIRED it would force a passkey on every user, which is the lock-out this change is
|
||||
# explicitly designed not to create.
|
||||
- authenticator: webauthn-authenticator-passwordless
|
||||
authenticatorFlow: false
|
||||
requirement: ALTERNATIVE
|
||||
priority: 10
|
||||
userSetupAllowed: false
|
||||
- authenticatorFlow: true
|
||||
requirement: ALTERNATIVE
|
||||
priority: 20
|
||||
flowAlias: browser-mfa-password-and-otp
|
||||
userSetupAllowed: false
|
||||
# The unchanged pre-passkey behaviour, relocated intact into the password branch: password,
|
||||
# then TOTP iff the user holds the platform-admin role. Nothing about the OTP condition
|
||||
# changed; it simply no longer sits on the path a passkey login takes.
|
||||
- alias: browser-mfa-password-and-otp
|
||||
description: "Password, then conditional OTP for platform-admins."
|
||||
providerId: basic-flow
|
||||
topLevel: false
|
||||
builtIn: false
|
||||
authenticationExecutions:
|
||||
# auth-password-form = Keycloak's "Password Form" (PasswordFormFactory) — the password
|
||||
# half of the old combined form, now that the username has already been collected.
|
||||
- authenticator: auth-password-form
|
||||
authenticatorFlow: false
|
||||
requirement: REQUIRED
|
||||
priority: 10
|
||||
|
||||
@@ -186,86 +186,6 @@ right way; one was a defect in my own manifest.
|
||||
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
|
||||
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
|
||||
|
||||
## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach
|
||||
|
||||
A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose
|
||||
Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from
|
||||
announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is
|
||||
real but inert, and the distinction matters enough to record, because the wrong version is very
|
||||
easy to re-derive from `ip neigh` alone.
|
||||
|
||||
Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case,
|
||||
after the UPS power-cycle — from the host netns of the `wireguard` pod on each node:
|
||||
|
||||
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|
||||
|---|---|---|
|
||||
| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
|
||||
| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms |
|
||||
| `nc -z 192.168.2.20 443` | **OK** | OK |
|
||||
| plex `192.168.2.24:32400` | **`200`** | `200` |
|
||||
|
||||
So the announcer really does **not** answer its own ARP request — that part of the observation is
|
||||
reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**:
|
||||
|
||||
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
|
||||
`Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**,
|
||||
so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before
|
||||
any route or neighbour lookup for the VIP happens**.
|
||||
* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the
|
||||
shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without
|
||||
a packet crossing the LAN at all.
|
||||
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
|
||||
which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real**
|
||||
service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes —
|
||||
that is service-aware DNAT, not wire traffic.
|
||||
|
||||
**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis:
|
||||
|
||||
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an
|
||||
artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not
|
||||
anything is wrong.
|
||||
* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the
|
||||
two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it
|
||||
discriminates nothing — and it is not the client path either (see below).
|
||||
|
||||
**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are
|
||||
**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is
|
||||
**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the
|
||||
forwarded path in either direction: it reports success where the forwarded path might fail, and
|
||||
`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with
|
||||
a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE`
|
||||
counters move.
|
||||
|
||||
## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses
|
||||
|
||||
`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the
|
||||
component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature
|
||||
to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel
|
||||
client cannot reach the internet, nothing in the list below is the reason:
|
||||
|
||||
| checked | result on all three nodes |
|
||||
|---|---|
|
||||
| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` |
|
||||
| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change |
|
||||
| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE |
|
||||
| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present |
|
||||
| default route | `default via 192.168.2.1 dev eno1 onlink`, intact |
|
||||
| node's own internet reachability | `http://1.1.1.1/` returns `301` |
|
||||
|
||||
**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is
|
||||
closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by
|
||||
`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled
|
||||
`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity
|
||||
traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway
|
||||
node. A remote client's apparent public address is therefore the **home WAN IP**, which is the
|
||||
intended behaviour and the thing to state when enrolling a device.
|
||||
|
||||
**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one
|
||||
enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and
|
||||
the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the
|
||||
power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the
|
||||
forwarded node shows a peer with an `endpoint` and a recent `latest handshake`.
|
||||
|
||||
## Prerequisites outside this directory
|
||||
|
||||
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
|
||||
|
||||
@@ -273,84 +273,6 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
|
||||
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
|
||||
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
|
||||
|
||||
## Triage — "the client reaches the LAN and cluster apps, but not the internet"
|
||||
|
||||
Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom
|
||||
almost always lives, and because the server side has been measured and is not the cause.
|
||||
|
||||
**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.**
|
||||
|
||||
| the profile says | meaning |
|
||||
|---|---|
|
||||
| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. |
|
||||
| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. |
|
||||
|
||||
`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit —
|
||||
`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client
|
||||
profile only**, so an already-issued profile keeps whatever it was rendered with until it is
|
||||
re-rendered and re-imported on the device. The two forms differ by one flag:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet>' # full tunnel
|
||||
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet> --split' # LAN only
|
||||
```
|
||||
|
||||
Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into
|
||||
`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is
|
||||
editing `AllowedIPs` in the profile **on the device** — the server does not care.
|
||||
|
||||
**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the
|
||||
client connected, then watch the two counters that only forwarded traffic moves:
|
||||
|
||||
```bash
|
||||
./infra/devcontainer/dev.sh run bash -c \
|
||||
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c "
|
||||
wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5;
|
||||
iptables -t nat -L POSTROUTING -n -v | head -5"'
|
||||
```
|
||||
|
||||
* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which
|
||||
node the router's `udp/51820` forward points at (the router is unreadable from the cluster).
|
||||
* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is
|
||||
upstream of the cluster (router forward, client, or carrier), not here.
|
||||
* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side
|
||||
failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there,
|
||||
which breaks internet while node-local cluster VIPs still work. It has not been observed.
|
||||
|
||||
**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly:
|
||||
|
||||
* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack
|
||||
client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel —
|
||||
so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4:
|
||||
`curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it,
|
||||
and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is
|
||||
blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it).
|
||||
* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified
|
||||
2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a
|
||||
blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on
|
||||
a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all.
|
||||
|
||||
**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the
|
||||
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain
|
||||
**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route
|
||||
`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the
|
||||
PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled
|
||||
`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the
|
||||
**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6).
|
||||
|
||||
## Triage — which L2 announcement lease a node holds does NOT matter
|
||||
|
||||
Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node
|
||||
could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`:
|
||||
that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh`
|
||||
for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer
|
||||
VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend
|
||||
is a node-local socket. Full measurements and the two misleading probes
|
||||
(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`.
|
||||
|
||||
`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a
|
||||
separate, pre-existing fault in that service and has nothing to do with the VPN.
|
||||
|
||||
## Revoking a device
|
||||
|
||||
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
|
||||
|
||||
Reference in new issue
Block a user