Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e82dab5df5 | ||
|
|
4656b07178 |
No files matched your search
@@ -1,311 +0,0 @@
|
|||||||
---
|
|
||||||
date: 2026-10-17
|
|
||||||
branch: feat/keycloak-passkeys
|
|
||||||
title: Passkeys declared for the homelab realm — and the realm import will not apply them
|
|
||||||
---
|
|
||||||
|
|
||||||
## What changed
|
|
||||||
|
|
||||||
`platform/keycloak/keycloak-realm-import.yaml` now declares passkey (WebAuthn Passwordless)
|
|
||||||
login for the `homelab` realm, in two parts:
|
|
||||||
|
|
||||||
1. **The WebAuthn Passwordless policy.** Ten `webAuthnPolicyPasswordless*` fields on the realm:
|
|
||||||
RP ID `${BASE_DOMAIN}`, RP entity name `Homelab SSO`, signature algorithms `[ES256, RS256]`,
|
|
||||||
`requireResidentKey: "Yes"`, `userVerificationRequirement: required`,
|
|
||||||
`authenticatorAttachment: "not specified"`, `attestationConveyancePreference: none`,
|
|
||||||
`acceptableAaguids: []`, `createTimeout: 300`, `avoidSameAuthenticatorRegister: true`.
|
|
||||||
Deliberately the **passwordless** family only; the two-factor `webAuthnPolicy*` family is left
|
|
||||||
at Keycloak's defaults because nothing in this realm uses `webauthn-authenticator`.
|
|
||||||
|
|
||||||
2. **The browser flow.** `browser-mfa` keeps its alias and stays bound as `browserFlow`; its
|
|
||||||
interior is restructured so a passkey is an **ALTERNATIVE** beside the password form:
|
|
||||||
|
|
||||||
```
|
|
||||||
browser-mfa (top level, unchanged)
|
|
||||||
browser-mfa-forms ALTERNATIVE
|
|
||||||
auth-username-form REQUIRED <- was auth-username-password-form
|
|
||||||
browser-mfa-passkey-or-password REQUIRED <- NEW sub-flow
|
|
||||||
webauthn-authenticator-passwordless ALTERNATIVE <- NEW: the passkey
|
|
||||||
browser-mfa-password-and-otp ALTERNATIVE <- NEW sub-flow
|
|
||||||
auth-password-form REQUIRED <- was half of the combined form
|
|
||||||
browser-mfa-conditional-otp CONDITIONAL (unchanged, relocated intact)
|
|
||||||
```
|
|
||||||
|
|
||||||
The design decisions, with their reasoning, are written into the manifest comments in this repo's
|
|
||||||
usual density. The ones that matter:
|
|
||||||
|
|
||||||
* **Additive, not replacing. Enrolment optional, not forced.** This IdP gates Grafana, Gitea,
|
|
||||||
Nextcloud, Headlamp, Semaphore, PiHole, Crafty, the \*arr stack *and* `kubectl` via the
|
|
||||||
`kubernetes` OIDC client. A mandatory passkey flow that misfires is a total lock-out from the
|
|
||||||
platform and the cluster API at once, leaving only the Keycloak bootstrap admin and the
|
|
||||||
break-glass cert kubeconfig. The gain from forcing it (one human stops typing a password) does
|
|
||||||
not price that risk.
|
|
||||||
* **No `requiredActions` block was added, on purpose.** Declaring one would have been the obvious
|
|
||||||
way to add `webauthn-register-passwordless`, and it would have been actively dangerous — see
|
|
||||||
"What I verified".
|
|
||||||
* **RP ID is `${BASE_DOMAIN}`, not `keycloak.${BASE_DOMAIN}`.** A passkey's RP ID is fixed at
|
|
||||||
enrolment and can never be re-keyed, so it must not be bound to a name that can move. This realm
|
|
||||||
already declares an `auth.${BASE_DOMAIN}` redirect URI, and this repo already owns a reconciler
|
|
||||||
built because a hostname move silently failed to reach the realm. Residual risk, unavoidable:
|
|
||||||
changing `BASE_DOMAIN` itself invalidates every enrolled passkey.
|
|
||||||
* **A passkey login is not also asked for TOTP.** `browser-mfa-conditional-otp` hangs off the
|
|
||||||
password branch only. The justification is `userVerificationRequirement: required` — the
|
|
||||||
ceremony already proves possession plus a verified human, phishing-resistantly and origin-bound.
|
|
||||||
Stacking a replayable shared secret on top of that buys friction. The manifest names the exact
|
|
||||||
one-line edit to reverse this decision.
|
|
||||||
* **Login becomes two screens** (username, then password) for every user, passkey or not, because
|
|
||||||
the server must know who is logging in before it can decide whether a passkey exists. Accepted
|
|
||||||
cost; it is the enabling change.
|
|
||||||
* **Usernameless login deliberately not used**, although Keycloak 26.0.6 supports it — see "What I
|
|
||||||
verified" for the measured reason it would have made every login open on a passkey prompt.
|
|
||||||
|
|
||||||
`platform/keycloak/README.md` gains a `## Passkeys` section: the decision table, the fact that
|
|
||||||
this does not reach the live realm, the two operator paths that would apply it, and the four-step
|
|
||||||
browser verification nobody has run.
|
|
||||||
|
|
||||||
No secret, key, credential or AAGUID was minted or committed. No file under `clusters/**`,
|
|
||||||
`conformance/**`, `ci/**`, `platform/kyverno/**`, `**/rbac.yaml` or `**/*.sops.yaml` was touched,
|
|
||||||
so this should not trip `ci/approval-policy.sh`'s sensitive-path review requirement.
|
|
||||||
|
|
||||||
## What I verified
|
|
||||||
|
|
||||||
**The live realm import has already completed and will never re-run.** Measured on prod,
|
|
||||||
2026-10-17:
|
|
||||||
|
|
||||||
```
|
|
||||||
$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
|
|
||||||
{"conditions":[{"message":"","status":"True","type":"Done"},
|
|
||||||
{"message":"","status":"False","type":"Started"},
|
|
||||||
{"message":"","status":"False","type":"HasErrors"}]}
|
|
||||||
```
|
|
||||||
|
|
||||||
`Done=True` / `Started=False`, exactly the one-shot state `platform/keycloak/README.md` already
|
|
||||||
documents for client secrets and client URLs. **So the change in this PR has zero effect on how
|
|
||||||
anyone logs in today**, and the client-secret/client-URL reconciler does not and cannot close the
|
|
||||||
gap: its hard endpoint allow-list makes `/authentication` unreachable by design, and realm-level
|
|
||||||
policy fields are not client fields.
|
|
||||||
|
|
||||||
**The CRD accepts every field I used.** Measured against the live CRD (not the upstream docs):
|
|
||||||
`kubectl --context k3s-prod get crd keycloakrealmimports.k8s.keycloak.org` lists all twelve
|
|
||||||
`webAuthnPolicyPasswordless*` properties with the types I wrote (`RequireResidentKey`,
|
|
||||||
`RpEntityName`, `RpId`, `UserVerificationRequirement`, `AttestationConveyancePreference`,
|
|
||||||
`AuthenticatorAttachment` as `string`; `SignatureAlgorithms`, `AcceptableAaguids`, `ExtraOrigins`
|
|
||||||
as `array/string`; `CreateTimeout` as `integer`; `AvoidSameAuthenticatorRegister` as `boolean`),
|
|
||||||
plus `authenticationFlows`, `authenticatorConfig` and `requiredActions`. The `spec.realm` schema
|
|
||||||
has **no** `x-kubernetes-preserve-unknown-fields`, so a misspelled field would have been pruned
|
|
||||||
silently rather than rejected — which is why this was checked rather than assumed.
|
|
||||||
|
|
||||||
**Running Keycloak is 26.0.6.** `quay.io/keycloak/keycloak:26.0.6` on `keycloak-0`/`keycloak-1`,
|
|
||||||
operator `quay.io/keycloak/keycloak-operator:26.0.6`. Every source claim below is read from the
|
|
||||||
`26.0.6` tag, not from memory or from docs.
|
|
||||||
|
|
||||||
**Every provider ID is quoted from upstream source, not guessed.**
|
|
||||||
|
|
||||||
| String I used | Where it is defined at tag 26.0.6 |
|
|
||||||
|---|---|
|
|
||||||
| `webauthn-authenticator-passwordless` | `WebAuthnPasswordlessAuthenticatorFactory.PROVIDER_ID`, display "WebAuthn Passwordless Authenticator" |
|
|
||||||
| `auth-username-form` | `UsernameFormFactory.PROVIDER_ID`, display "Username Form" |
|
|
||||||
| `auth-password-form` | `PasswordFormFactory.PROVIDER_ID`, display "Password Form" |
|
|
||||||
| `webauthn-register-passwordless` | `WebAuthnPasswordlessRegisterFactory.PROVIDER_ID` (referenced in comments only) |
|
|
||||||
|
|
||||||
**`requiredActions` is REPLACE, not merge — measured, and it is why I did not declare one.**
|
|
||||||
`model/storage-private/.../DefaultExportImportManager.java` lines 327-338:
|
|
||||||
|
|
||||||
```java
|
|
||||||
if (rep.getRequiredActions() != null) {
|
|
||||||
for (RequiredActionProviderRepresentation action : rep.getRequiredActions()) { ... }
|
|
||||||
DefaultRequiredActions.addDeleteAccountAction(newRealm);
|
|
||||||
} else {
|
|
||||||
DefaultRequiredActions.addActions(newRealm);
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Declaring one required action therefore skips the entire default set. `RealmManager` is no
|
|
||||||
backstop: `setupRequiredActions` is `if (realm.getRequiredActionProvidersStream().count() == 0)`,
|
|
||||||
and the count is already non-zero by then (`RepresentationToModel.importRealm` runs at
|
|
||||||
`RealmManager.java:585`, the backstop at `:612`). A realm built that way would have had the passkey
|
|
||||||
action plus `delete_account` and nothing else — **no `CONFIGURE_TOTP`**, which would leave
|
|
||||||
`auth-otp-form` demanding a TOTP that no platform-admin could ever enrol: a hard lock-out
|
|
||||||
delivered by a change whose purpose was to *add* a login method.
|
|
||||||
|
|
||||||
**And declaring it would have bought nothing.** `DefaultRequiredActions.addWebAuthnPasswordlessRegisterAction`
|
|
||||||
registers `webauthn-register-passwordless` with `setEnabled(true)`, `setDefaultAction(false)`,
|
|
||||||
priority 80, gated on `Profile.Feature.WEB_AUTHN` — which is `Type.DEFAULT` in
|
|
||||||
`common/.../Profile.java:74`, i.e. on unless explicitly disabled. `enabled=true` /
|
|
||||||
`defaultAction=false` *is* the additive-and-optional posture this change wants.
|
|
||||||
|
|
||||||
**`authenticationFlows` has the opposite semantics, which is why declaring flows is safe.**
|
|
||||||
`importAuthenticationFlows` adds the declared flows, then backfills each unbound binding
|
|
||||||
(browser, registration, direct grant, reset credentials, client auth, first broker login, docker
|
|
||||||
auth) from the built-in defaults individually (`DefaultExportImportManager.java` ~1320-1410).
|
|
||||||
|
|
||||||
**The graceful degradation to password is measured from the flow engine, not assumed.** For a user
|
|
||||||
with no passkey: `WebAuthnPasswordlessAuthenticator.requiresUser()` returns `false`, so
|
|
||||||
`DefaultAuthenticationFlow.processSingleFlowExecutionModel` takes the `else` branch (`daf:456-462`),
|
|
||||||
whose four conditions all hold — a user is set (from `auth-username-form`), `configuredFor` is
|
|
||||||
false, `WebAuthnPasswordlessAuthenticatorFactory.isUserSetupAllowed()` is `false`, and
|
|
||||||
`WebAuthnAuthenticator implements Authenticator, CredentialValidator<...>` — so it raises
|
|
||||||
`CREDENTIAL_SETUP_REQUIRED`. The ALTERNATIVE loop (`daf:289-302`) catches exactly that, marks the
|
|
||||||
execution `ATTEMPTED` and proceeds to the next alternative, with the upstream comment *"the user,
|
|
||||||
since he has alternatives, should be able to go to another alternative and continue"*. Hence: no
|
|
||||||
passkey, no passkey screen, straight to the password form.
|
|
||||||
|
|
||||||
**Why usernameless was rejected, measured.** The `configuredFor` guard sits inside the
|
|
||||||
`requiresUser()` branch. With `requiresUser()` false and no user yet in context, that guard is
|
|
||||||
skipped and the engine just calls `authenticate(context)` — so a passkey execution placed before
|
|
||||||
the username form is offered to *everyone*, including users who will never own one.
|
|
||||||
|
|
||||||
**The policy sentinels are read from the shipped code, and one of my initial assumptions was
|
|
||||||
wrong.** `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED = "not specified"` is the
|
|
||||||
realm-representation sentinel — but `WebAuthnConstants.OPTION_NOT_SPECIFIED = ""` (empty string) is
|
|
||||||
the WebAuthn-API-level one. I checked both because they disagree. The login theme's
|
|
||||||
`webauthnRegister.js` at 26.0.6 settles the behaviour:
|
|
||||||
|
|
||||||
```js
|
|
||||||
if (input.requireResidentKey !== 'not specified') {
|
|
||||||
if (input.requireResidentKey === 'Yes') { authenticatorSelection.requireResidentKey = true; }
|
|
||||||
else { authenticatorSelection.requireResidentKey = false; }
|
|
||||||
}
|
|
||||||
...
|
|
||||||
if (input.createTimeout !== 0) { publicKey.timeout = input.createTimeout * 1000; }
|
|
||||||
rp: {id: input.rpId, name: input.rpEntityName}
|
|
||||||
```
|
|
||||||
|
|
||||||
So `"Yes"` is a literal string compare (any other non-sentinel value means *false*), `"not
|
|
||||||
specified"` is compared literally for four fields, and `createTimeout` is **seconds** with `0`
|
|
||||||
meaning "send no timeout". It also shows 26.0.6 emits only the legacy `requireResidentKey` member,
|
|
||||||
never `residentKey: "required"` — the client maps that per WebAuthn L2, but the policy cannot
|
|
||||||
express `residentKey: "preferred"` at all.
|
|
||||||
|
|
||||||
`[ES256, RS256]` is exactly `WebAuthnPolicy.DEFAULT_POLICY` /
|
|
||||||
`Constants.DEFAULT_WEBAUTHN_POLICY_SIGNATURE_ALGORITHMS` — declared explicitly, not changed.
|
|
||||||
`getWebAuthnPolicyPasswordless` was read field by field to confirm `[]` for `acceptableAaguids`
|
|
||||||
means "no restriction" and that `RpEntityName` must be non-empty or the **update** path
|
|
||||||
(`DefaultExportImportManager.java:842`) drops the whole policy silently.
|
|
||||||
|
|
||||||
**The manifest parses and renders.** `yaml.safe_load` round-trips the file and the flow graph reads
|
|
||||||
back exactly as designed (five flows, the executions and priorities above, `browserFlow:
|
|
||||||
browser-mfa`, `requiredActions` absent). `python3 ci/render-keycloak-client-urls.py --check` →
|
|
||||||
`client-urls.yaml is in sync`, rc 0 (no client URL changed, so no regeneration was needed).
|
|
||||||
`yamllint -c ci/yamllint.yaml platform/keycloak/keycloak-realm-import.yaml` → rc 0; no line exceeds
|
|
||||||
the 160-column ceiling.
|
|
||||||
|
|
||||||
**`browser-mfa` keeps its alias deliberately.** `conformance/tests/l2-behavioral/keycloak-config-as-code`
|
|
||||||
asserts `kubectl get keycloakrealmimport homelab -o jsonpath={.spec.realm.browserFlow} | grep -q
|
|
||||||
'browser-mfa'`. Renaming the top-level flow would have failed that test.
|
|
||||||
|
|
||||||
**Envsubst safety.** `platform-config`'s Flux Kustomization declares
|
|
||||||
`postBuild.substituteFrom: cluster-vars` (`clusters/base/platform-config.yaml:26-29`), so this file
|
|
||||||
is substituted. The only brace token I added is `${BASE_DOMAIN}`, which is declared at
|
|
||||||
`clusters/base/cluster-vars.yaml:42` (`home.keithjackson.dev`). Every comment I wrote avoids braces
|
|
||||||
entirely — the RP-ID discussion says `keycloak.<BASE_DOMAIN>` with angle brackets precisely so an
|
|
||||||
unparseable token cannot fail the whole Kustomization.
|
|
||||||
|
|
||||||
**Gate result: rc 3, zero failures, `gitea-settings` the only stage the gate counts as not-run** —
|
|
||||||
the documented honest best local outcome. Run in the pinned toolbox from this worktree with all
|
|
||||||
three §3 workarounds (`DEV_KUBECONFIG=/dev/null`, plus an `ENGINE` shim bind-mounting the main
|
|
||||||
checkout's `.git` and the sibling spec repo with `SPEC_REPO=/home-network-k3s`):
|
|
||||||
|
|
||||||
```
|
|
||||||
[ ok ] stage spec-immutability: pass <- NOT skipped; both shim mounts worked, R-TEST-06 covered
|
|
||||||
[ ok ] stage decisions-log: pass <- archive frozen, 48 entries well-formed
|
|
||||||
[ ok ] stage lint: pass
|
|
||||||
[ ok ] stage l0: pass <- incl. flux-postbuild-substitution + keycloak-client-secret-reconciler
|
|
||||||
[ ok ] stage render-golden: pass
|
|
||||||
[ ok ] stage l1: pass
|
|
||||||
[ ok ] stage approval-policy: pass <- no sensitive path touched, so no non-author review needed
|
|
||||||
[skip] stage gitea-settings: SKIPPED <- needs SETTINGS_API_TOKEN, CI-only; expected
|
|
||||||
[skip] stage conformance: SKIPPED <- cluster-gated L2/L3; expected
|
|
||||||
MERGE GATE: UNDETERMINED — Stages that did not run: gitea-settings
|
|
||||||
```
|
|
||||||
|
|
||||||
No lint finding and no check failure names either file I changed. Touched requirements resolved to
|
|
||||||
`R-IDP-01 R-IDP-02 R-IDP-03 R-IDP-04 R-IDP-05 R-IDP-06`; coverage 51/51, 0 gaps.
|
|
||||||
|
|
||||||
**A note on how that number was obtained, because the first attempt was worthless.** My first gate
|
|
||||||
run returned the same rc 3 and the same seven passes — against a tree that did not contain this
|
|
||||||
change at all. The gate's `tracked_files()` reads **git**, and my edits were still uncommitted, so
|
|
||||||
every stage gated `origin/main`. The tell was in the log and I nearly missed it: the
|
|
||||||
touched-requirement subset reported UNDETERMINED with *"check out the feature branch you mean to
|
|
||||||
gate (not the base branch)"*, and `decisions-log` counted **47** entries rather than 48. **A green
|
|
||||||
gate run on an uncommitted worktree is a gate run on the base branch.** The numbers above are from
|
|
||||||
the second run, after committing and rebasing.
|
|
||||||
|
|
||||||
**The base moved under me mid-run.** `origin/main` advanced from `9a41a8e` to `1a0b208` (PR #70,
|
|
||||||
oauth2-proxy OIDC startup race) while I was working. This branch is rebased onto `1a0b208`, so the
|
|
||||||
gate above ran on the base CI will actually merge into. The rebase was clean — PR #70 touched
|
|
||||||
`platform/oauth2-proxy/helmrelease.yaml` and its own decisions entry, disjoint from these files.
|
|
||||||
|
|
||||||
## What I did NOT verify
|
|
||||||
|
|
||||||
**This change is unapplied. It changes nothing about the live realm, and I did not attempt to make
|
|
||||||
it.** `Done=True` / `Started=False` was measured, not inferred, and I ran no mutating `kubectl`,
|
|
||||||
no `flux reconcile`, no admin-API call and no `kcadm`. Whether the operator's eventual
|
|
||||||
admin-console reconstruction of this flow matches what is declared here is, by construction,
|
|
||||||
unverifiable from git.
|
|
||||||
|
|
||||||
**I never enrolled a passkey and never pointed a browser at any of this.** Nothing here has been
|
|
||||||
exercised by a real WebAuthn ceremony on any device. Specifically unverified:
|
|
||||||
|
|
||||||
* **That the RP ID works.** `home.keithjackson.dev` is a registrable-domain suffix of
|
|
||||||
`keycloak.home.keithjackson.dev`, so it *should* satisfy the browser's check — but I verified
|
|
||||||
that by reading the WebAuthn rule, not by watching a browser accept it. A wrong RP ID fails
|
|
||||||
**silently, in the browser, before Keycloak is involved**, with no server-side log line. If
|
|
||||||
enrolment produces no OS prompt and no error, this is the first suspect.
|
|
||||||
* **That `"not specified"` is the right literal for `authenticatorAttachment` in a realm import.**
|
|
||||||
I traced it to `Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED` and to the theme's string
|
|
||||||
compare, but I have not seen a realm import accept it. Omitting the field would default to the
|
|
||||||
same value; if an import ever rejects the literal, drop the line.
|
|
||||||
* **That `createTimeout: 300` behaves.** Seconds-vs-milliseconds is read from the theme JS. The
|
|
||||||
claim that 300 is long enough for the cross-device QR/hybrid flow, and that Chrome clamps rather
|
|
||||||
than rejects it, is reasoning about browser behaviour, not a measurement.
|
|
||||||
* **Whether a passkey user can still reach the password form.** I argued it from
|
|
||||||
`createAuthenticationSelectionList` producing a "Try another way" option, and I noted that for a
|
|
||||||
user holding both credentials the *first* screen is chosen by the user's own credential ordering
|
|
||||||
rather than by my priorities. Neither was observed.
|
|
||||||
* **The two-screen login regression for ordinary users.** `auth-username-form` +
|
|
||||||
`auth-password-form` replacing `auth-username-password-form` is a visible change for every user
|
|
||||||
including non-passkey family accounts, and nobody has logged in through it.
|
|
||||||
* **That conditional OTP for platform-admins still fires on the password branch.** The sub-flow and
|
|
||||||
its `conditional-otp-platform-admin` config are byte-identical and merely relocated one level
|
|
||||||
deeper, but "identical YAML" is not "still enforced at runtime". This is the highest-value
|
|
||||||
regression check and it is untested. The previous breakage of this exact step
|
|
||||||
(`condUserRole` vs a non-existent property key) 403'd every platform-admin login.
|
|
||||||
* **That a platform-admin authenticating by passkey is not challenged for TOTP.** That is the
|
|
||||||
intended behaviour of the shape I wrote; it is unobserved, and it is also a deliberate
|
|
||||||
relaxation of R-IDP-04 as previously implemented, which the operator may want to reverse.
|
|
||||||
|
|
||||||
**Whether the realm import reconciles into an existing realm: measured as NO for the completed
|
|
||||||
one-shot, but I did not test a re-import.** I did not delete and recreate the
|
|
||||||
`KeycloakRealmImport`, so I cannot say what a re-run would do to the existing `homelab` realm —
|
|
||||||
whether it would merge the new policy and flows, duplicate flow aliases, or fail. The 2026-09-06
|
|
||||||
total SSO outage documented in `platform/keycloak/README.md` is the reason I did not experiment.
|
|
||||||
**Treat the re-import path in the README as untested advice, not a runbook.**
|
|
||||||
|
|
||||||
**Nothing was proved about the live realm's current state.** I did not read the realm's existing
|
|
||||||
required actions, its existing WebAuthn policies, or its live flow executions — doing so needs the
|
|
||||||
admin credential and I did not decrypt or use it. So I cannot confirm that the live realm's
|
|
||||||
required-action set actually contains `webauthn-register-passwordless` today; I can only show that
|
|
||||||
Keycloak 26.0.6's defaults register it and that this repo's import never declared any required
|
|
||||||
actions to suppress them. **That inference has not been checked against the realm.**
|
|
||||||
|
|
||||||
**L2/L3 conformance did not run** (no cluster from the gate container; `vars.CLUSTER_AVAILABLE` is
|
|
||||||
unset in CI anyway), so the existing `keycloak-config-as-code` behavioural test did not re-execute
|
|
||||||
against this change. I reasoned about its assertion by reading it.
|
|
||||||
|
|
||||||
**No new conformance test was added.** Nothing in `conformance/` asserts the passkey policy or the
|
|
||||||
new flow shape, so a future edit can silently remove either and no gate stage will notice. That is
|
|
||||||
a real gap this PR leaves open; closing it would touch `conformance/**`, which is a sensitive path
|
|
||||||
needing non-author review.
|
|
||||||
|
|
||||||
**The quoted gate run predates this file's own gate-result paragraph.** It gated commit `207f5db`;
|
|
||||||
the gate numbers and the rebase note above were then written into this entry, so the pushed tree
|
|
||||||
differs from the gated one by this handoff's prose only — no manifest, no check, no generated file.
|
|
||||||
`decisions-log` is the only stage that reads this file, and it validates structure (frontmatter,
|
|
||||||
the three headings, non-stub bodies), which is unchanged. I did not re-run the full gate for a
|
|
||||||
prose edit; CI will gate the pushed commit.
|
|
||||||
|
|
||||||
**`gitea-settings` did not run locally** — it needs `SETTINGS_API_TOKEN`, which exists only in CI.
|
|
||||||
I did not re-measure branch protection, the runner, or `required_approvals` against the Gitea API
|
|
||||||
this session; the claims in `CLAUDE.md` §2 are taken on trust here and are explicitly flagged there
|
|
||||||
as needing re-measurement.
|
|
||||||
@@ -0,0 +1,198 @@
|
|||||||
|
---
|
||||||
|
date: 2026-10-17
|
||||||
|
branch: fix/vpn-ingress-l2-lease-collision
|
||||||
|
title: "WITHDRAWN: a node CAN reach a VIP it announces. My L2-lease diagnosis was wrong"
|
||||||
|
---
|
||||||
|
|
||||||
|
## What changed
|
||||||
|
|
||||||
|
**This branch previously proposed a fix. That fix has been reverted in full. The diagnosis behind it
|
||||||
|
was wrong.** What is left is documentation of what was actually measured, so the wrong conclusion is
|
||||||
|
not re-derived — it is very easy to re-derive from `ip neigh` alone.
|
||||||
|
|
||||||
|
### The claim, and how it died
|
||||||
|
|
||||||
|
The earlier version of this entry claimed: *a node cannot reach a LoadBalancer VIP whose Cilium L2
|
||||||
|
announcement lease it holds itself* — the announcer does not answer its own ARP request, the
|
||||||
|
neighbour entry settles at `FAILED`, and packets to that VIP die before leaving the node. Because
|
||||||
|
`platform/vpn-ingress` runs WireGuard on `hostNetwork` and MASQUERADEs remote clients to the node's
|
||||||
|
own address, a VPN client was said to inherit that. The fix labelled the forwarded node and excluded
|
||||||
|
that label from announcing, with two alerts and four promtool tests behind it.
|
||||||
|
|
||||||
|
The claim makes a sharp prediction. After today's UPS power-cycle **all eight** `cilium-l2announce-*`
|
||||||
|
leases are held by `k3s-prod-1` (gateway, pihole tcp+udp, crafty tcp+udp, unifi tcp+udp, plex) —
|
||||||
|
the extreme case. The prediction was therefore that `k3s-prod-1` should reach **none** of the five
|
||||||
|
VIPs and the other two nodes should reach **all**.
|
||||||
|
|
||||||
|
**The prediction failed.** Measured from the host netns of the `wireguard` pod on each node:
|
||||||
|
|
||||||
|
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|
||||||
|
|---|---|---|
|
||||||
|
| `ip neigh show 192.168.2.20` | `FAILED`, no lladdr | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
|
||||||
|
| `curl -k https://192.168.2.20/` | **`200` in 18 ms** | `200` in 13–40 ms |
|
||||||
|
| `nc -z 192.168.2.20 443` | **OK** | OK |
|
||||||
|
| `192.168.2.22:25565`, `.23:53`, `.24:32400` | **all OK / `200`** | all OK / `200` |
|
||||||
|
| `192.168.2.21:8443` | FAIL | **FAIL** (both) |
|
||||||
|
|
||||||
|
The asserted **cause is still present** today — `FAILED` on the holder, for all five VIPs — while the
|
||||||
|
asserted **effect is absent**. That is sufficient to falsify the causal claim without having to
|
||||||
|
explain anything else. The ARP asymmetry is real and reproducible; it is also **inert**.
|
||||||
|
|
||||||
|
### Why it is inert
|
||||||
|
|
||||||
|
No traffic path resolves a LoadBalancer VIP by ARP:
|
||||||
|
|
||||||
|
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
|
||||||
|
`Masquerading: BPF [eno1, wg0]` (Cilium v1.20.1, TCX attach, tunnel/vxlan). Cilium's LB datapath
|
||||||
|
is attached to **`wg0` as well as `eno1`**, so a remote client's packet arriving on `wg0` for a VIP
|
||||||
|
is DNAT'd **in BPF, before any route or neighbour lookup for the VIP**.
|
||||||
|
* `cilium-dbg service list` programs `192.168.2.20:443 => 127.0.0.1:13606` — the shared Gateway's
|
||||||
|
backend is a **node-local** Envoy socket, so every node serves that VIP without a LAN packet.
|
||||||
|
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
|
||||||
|
which is why `curl` succeeds on the holder. Discriminator: a real service port connects while a
|
||||||
|
bogus one (`nc -z 192.168.2.20 9999`) fails, identically on all three nodes — service-aware DNAT,
|
||||||
|
not wire traffic.
|
||||||
|
|
||||||
|
### The two probes that misled me
|
||||||
|
|
||||||
|
Both were cited as evidence in the previous version of this entry.
|
||||||
|
|
||||||
|
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless, and is also a probing
|
||||||
|
artefact: VIPs answer no ICMP, so a `ping` leaves a failed entry whether or not anything is wrong.
|
||||||
|
* **`curl --interface wg0`.** The previous entry used this as the control that ruled out a
|
||||||
|
host-netns artefact. It is not a control: it fails (`rc=28`) from **all three** nodes, including
|
||||||
|
the two whose ARP for the VIP resolves perfectly. It is uncorrelated with lease holding, so it
|
||||||
|
discriminates nothing.
|
||||||
|
|
||||||
|
**The deeper error:** a MASQUERADEd client's packets are **forwarded** (netfilter `FORWARD`, then
|
||||||
|
routing); everything runnable from the pod is **socket-originated**. Different hooks. A host-netns
|
||||||
|
probe cannot measure the forwarded path in either direction — it reports success where the forwarded
|
||||||
|
path might fail, and `--interface wg0` reports failure where it does not. Sharing a netns is not
|
||||||
|
sharing a hook, and that is the assumption the original diagnosis rested on.
|
||||||
|
|
||||||
|
### Files
|
||||||
|
|
||||||
|
Reverted to `origin/main`: `platform-config/cilium/l2-announcement-policy.yaml`,
|
||||||
|
`infra/ansible/roles/node_baseline/{defaults,tasks}/main.yml`,
|
||||||
|
`infra/ansible/inventory/group_vars/k3s_prod/vars.yml`, `platform/monitoring/alerts.yaml`,
|
||||||
|
`platform/monitoring/kube-prometheus-stack.yaml`, `runbooks/kustomization.yaml`. Deleted:
|
||||||
|
`platform/monitoring/tests/vpn-ingress-l2.test.yaml`, `runbooks/vpn-ingress-l2-collision.md`.
|
||||||
|
|
||||||
|
Kept, rewritten from scratch:
|
||||||
|
|
||||||
|
* **`platform/vpn-ingress/README.md`** — two measured sections: the L2 table above with why it is
|
||||||
|
inert and which probes lie, and the internet-egress facts below.
|
||||||
|
* **`runbooks/vpn-remote-access.md`** — a triage section for the reported symptom ("LAN and cluster
|
||||||
|
work, internet does not") that works outwards from the client profile, and a short section saying
|
||||||
|
the L2 lease is a dead end, with the unifi `.21` note.
|
||||||
|
|
||||||
|
**Nothing in `platform/vpn-ingress` or `infra/vpn-ingress` changed behaviourally. No code fix is
|
||||||
|
proposed by this branch any more.**
|
||||||
|
|
||||||
|
### The internet-routing report: no server-side fix was needed, and none was made
|
||||||
|
|
||||||
|
The brief's prior was that `new-peer.sh` renders a split tunnel and that the operator wants a full
|
||||||
|
one, to be added as a documented option. **That prior does not match the tree.** `new-peer.sh` has
|
||||||
|
emitted `AllowedIPs = 0.0.0.0/0` **by default since the component's first commit** (`f92707f`,
|
||||||
|
2026-10-05, confirmed with `git log -S`); `--split` is the opt-in, and
|
||||||
|
`runbooks/vpn-remote-access.md:252-253` already documents both forms. So there is no full-tunnel
|
||||||
|
feature to add, and adding a `WG_FULL_TUNNEL` knob would be a second switch for a default that
|
||||||
|
already exists. **I deliberately shipped no code change here** — writing one would have meant
|
||||||
|
inventing a fix for a symptom I could not reproduce (see below).
|
||||||
|
|
||||||
|
The server side was measured end to end and is capable and correct, so the cause is in the client
|
||||||
|
profile or the client. The triage section names the two candidates that fit the symptom exactly —
|
||||||
|
a `--split` profile, and the IPv4-only `AllowedIPs` letting a dual-stack client prefer AAAA and
|
||||||
|
leak public traffic outside the tunnel — each with a one-line test that settles it.
|
||||||
|
|
||||||
|
## What I verified
|
||||||
|
|
||||||
|
Measured live against prod, `kubectl --context k3s-prod --request-timeout=15s`, read-only, on
|
||||||
|
2026-10-17:
|
||||||
|
|
||||||
|
* **The lease distribution.** All eight `cilium-l2announce-*` leases held by `k3s-prod-1`
|
||||||
|
(`.spec.holderIdentity`). The brief's observation reproduced; it had not moved again.
|
||||||
|
* **The falsification, in full**, as tabulated above: 3 nodes x 5 VIPs, `ip neigh` + `nc -z` + a
|
||||||
|
real `curl` for each. The holder reaches four of five VIPs; `.21` fails from everywhere.
|
||||||
|
* **The socket-LB discriminator.** Real service port OK, bogus port FAIL, identically on all three
|
||||||
|
nodes — so the successes are service-aware DNAT, not wire delivery.
|
||||||
|
* **`curl --interface wg0` fails on all three nodes**, including non-announcing ones with
|
||||||
|
`REACHABLE` neighbour entries. This is the measurement that destroys the original control.
|
||||||
|
* **The datapath facts** quoted above from `cilium-dbg status --verbose` and
|
||||||
|
`cilium-dbg service list` on `cilium-55jv8` (`k3s-prod-1`), plus `cilium-config`:
|
||||||
|
`kube-proxy-replacement=true`, `enable-l2-announcements=true`, `enable-bpf-masquerade=true`,
|
||||||
|
`routing-mode=tunnel`, `devices=` (empty, auto-detected). Cilium `v1.20.1` on all three.
|
||||||
|
* **The VIP is not a local address and routes on-link.** `ip route get 192.168.2.20` →
|
||||||
|
`dev eno1 src 192.168.2.10` on the holder; nothing in `table local`; `ip -4 addr` shows no VIP.
|
||||||
|
So the 200 is not loopback delivery by the host stack — it is BPF.
|
||||||
|
* **The internet path, server side, on all three nodes:** `/proc/sys/net/ipv4/ip_forward` = `1`
|
||||||
|
(read, never written); `iptables -t nat -S POSTROUTING` carries
|
||||||
|
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE`; the preceding `CILIUM_POST_nat` chain is **empty with 0
|
||||||
|
packets**, so it cannot short-circuit that rule; `FORWARD -i wg0` and `-o wg0` ACCEPT both
|
||||||
|
present; `default via 192.168.2.1 dev eno1 onlink` intact; `ip route get 1.1.1.1` resolves via it;
|
||||||
|
`curl http://1.1.1.1/` returns `301` from each node.
|
||||||
|
* **The PIA / egress-gateway answer.** `platform/vpn-egress/egress-steering.yaml` selects by
|
||||||
|
`podSelector` on `egress-class: vpn`; `platform/vpn-ingress/daemonset.yaml:130` labels this pod
|
||||||
|
`egress-class: internet`; `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`), absent on
|
||||||
|
`-1` and `-2`. So a full tunnel egresses the **house WAN**, and a client's apparent address is the
|
||||||
|
home WAN IP, not PIA's.
|
||||||
|
* **PiHole resolves public names** — `nslookup google.com 192.168.2.23` from the pod returned
|
||||||
|
`142.250.113.138`.
|
||||||
|
* **No client is connected, and none has been since the power-cycle.** `wg show wg0` lists the one
|
||||||
|
enrolled peer (`10.90.0.2/32`) with **no endpoint and no handshake** on all three nodes, and both
|
||||||
|
`FORWARD -i wg0` and the `MASQUERADE` rule read **0 packets / 0 bytes** on all three. The
|
||||||
|
`wireguard` pods restarted 153 min ago (2 restarts, 11d old), which reset the counters.
|
||||||
|
* **The full-tunnel default's age**, from `git log -S` on `new-peer.sh`: introduced in `f92707f`
|
||||||
|
(2026-10-05) and never changed.
|
||||||
|
* **The branch rebased cleanly** onto `origin/main` (`1a0b208`, PR #70 merged) with no conflicts.
|
||||||
|
|
||||||
|
## What I did NOT verify
|
||||||
|
|
||||||
|
* **THE FORWARDED PATH IS STILL UNMEASURED, in both directions.** No client is connected, so I could
|
||||||
|
not test what a real remote client does, and — as established above — **no probe runnable from the
|
||||||
|
pod can substitute for it**. Specifically I did **not** prove that a forwarded client packet to a
|
||||||
|
VIP is DNAT'd by Cilium's `wg0` program. That inference rests on `cilium-dbg` listing `wg0` as a
|
||||||
|
LB device plus the node-local Gateway backend, not on an observed client packet. **It is the one
|
||||||
|
remaining way the withdrawn diagnosis could be partly right** — if `wg0` BPF were somehow not in
|
||||||
|
the forwarded path, forwarded traffic would need ARP and would die on the holder. I judge this
|
||||||
|
unlikely but it is unproven. *To close it: connect a client, then from the node whose `wg show`
|
||||||
|
shows the peer endpoint, watch `FORWARD -i wg0` and the `MASQUERADE` counters move while the
|
||||||
|
client loads a cluster app — and repeat it on a node that holds leases and one that does not.*
|
||||||
|
* **WHICH NODE THE ROUTER FORWARDS `udp/51820` TO IS UNDETERMINABLE RIGHT NOW**, for the same
|
||||||
|
reason. **I need a client connected** to learn it; the router is unreadable from the cluster.
|
||||||
|
* **I cannot reproduce or explain the 2026-10-08 measurements.** That entry recorded `nc -z` TCP
|
||||||
|
failures correlated with lease holding across 15 node x VIP pairs. Today the same test passes.
|
||||||
|
My leading hypothesis is **stale or incorrectly-programmed BPF LB state on `k3s-prod-1`**,
|
||||||
|
rebuilt by today's full power-cycle — which would make the lease correlation coincidental. A
|
||||||
|
competing hypothesis is that Cilium had not auto-detected `wg0` then (`devices` is empty, and
|
||||||
|
`wg0` is created by this DaemonSet after Cilium starts). **Both are unproven, and neither is
|
||||||
|
needed for the falsification** — the asserted cause is present today without its effect. I did not
|
||||||
|
retro-measure anything, and I did not read Cilium's source or issue tracker.
|
||||||
|
* **The operator's report cannot be a description of "right now."** No client has connected since
|
||||||
|
the power-cycle (0 forwarded packets, no handshake on any node), so "clients can reach cluster
|
||||||
|
services and the LAN but not the internet" must describe a session before the power-cycle or one
|
||||||
|
that has since dropped. **I did not establish when it was observed, or with which profile.**
|
||||||
|
* **THE INTERNET ROOT CAUSE IS NOT DIAGNOSED.** I proved the server side is *capable*; I did **not**
|
||||||
|
find the fault, and I shipped no fix for it. The two candidates in the triage section (a
|
||||||
|
`--split` profile; IPv6 preference leaking outside an IPv4-only `AllowedIPs`) are **reasoned, not
|
||||||
|
measured**. *What I need:* the `AllowedIPs` line from the client's own profile — that one line
|
||||||
|
decides split vs full and halves the search — and a connected client for step 2.
|
||||||
|
* **Whether `--split` was in fact used for the enrolled peer.** The profile holds a private key and
|
||||||
|
lives only on the operator's machine and the device; it is not in the repo, and I did not look for
|
||||||
|
it. Server-side `AllowedIPs` is a `/32` source filter and says nothing about the client's routing.
|
||||||
|
* **Whether `MTU = 1420` holds on a full-tunnel WAN path.** Not measured, and not measurable without
|
||||||
|
a client. It is the standard 1500 − 60; a client behind PPPoE or CGNAT with a smaller path MTU
|
||||||
|
would see large transfers hang while small requests succeed. The runbook's existing MTU note
|
||||||
|
(drop to 1280) already covers the remedy.
|
||||||
|
* **Adding `::/0` to `AllowedIPs` is untested advice.** The triage section offers it to blackhole
|
||||||
|
IPv6 rather than leak it, and states that the tunnel has no IPv6 egress. I did not try it on a
|
||||||
|
client, and I do not know how each WireGuard client behaves when handed an IPv6 route on an
|
||||||
|
interface with no IPv6 address.
|
||||||
|
* **`192.168.2.21:8443` (unifi) fails from all three nodes**, as it did on 2026-10-08 — a separate,
|
||||||
|
pre-existing fault. Not investigated, not addressed, and now recorded in the runbook as such.
|
||||||
|
* **No runtime conformance.** `vars.CLUSTER_AVAILABLE` is still unset, so the gate proves the L0/L1
|
||||||
|
offline subset only. Nothing above is covered by a test that runs in CI.
|
||||||
|
* **Nothing was applied, and nothing needs to be.** The branch is now documentation-only, so there
|
||||||
|
is no Flux reconcile, Ansible run or `kubectl` step to perform after merge. I ran no mutating
|
||||||
|
`kubectl`, `flux` or Ansible, and did not run `new-peer.sh` (it mints key material).
|
||||||
|
* I did not re-measure branch protection, runner state, or any other platform claim (CLAUDE.md §2b).
|
||||||
@@ -154,74 +154,6 @@ NEW client, a new protocol mapper or a changed scope still needs the realm impor
|
|||||||
(it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
|
(it will not on its own) or a deliberate admin-API/`kcadm` change. `Done=True` is never evidence that
|
||||||
a realm edit landed.
|
a realm edit landed.
|
||||||
|
|
||||||
## Passkeys (WebAuthn Passwordless) — DECLARED IN GIT, NOT YET IN THE REALM (2026-10-17)
|
|
||||||
|
|
||||||
`keycloak-realm-import.yaml` now declares passkey login: the realm's **WebAuthn Passwordless**
|
|
||||||
policy (`webAuthnPolicyPasswordless*`), and a `browser-mfa` flow that offers a passkey as an
|
|
||||||
**ALTERNATIVE** beside the password form. Design decisions and their reasoning are in the manifest
|
|
||||||
comments, not repeated here. The headline choices:
|
|
||||||
|
|
||||||
| Decision | Value | One-line reason |
|
|
||||||
|---|---|---|
|
|
||||||
| Additive or replacing? | **Additive** — passkey *or* password | A misfiring mandatory flow locks the single operator out of every SSO app *and* kubectl |
|
|
||||||
| Enrolment | **Optional**, never a default action | Same reason; `webauthn-register-passwordless` is already enabled-not-forced by default |
|
|
||||||
| Relying-party ID | `${BASE_DOMAIN}` (not `keycloak.${BASE_DOMAIN}`) | An unre-keyable credential must not be bound to a hostname that can move |
|
|
||||||
| User verification | `required` | Without it, mere possession of an unlocked device is a complete login |
|
|
||||||
| Resident key | `Yes` | A non-discoverable credential is a second factor, not a passkey |
|
|
||||||
| OTP after a passkey? | **No** — conditional OTP stays on the password branch | A UV-required passkey is already two factors, and a phishing-resistant pair at that |
|
|
||||||
|
|
||||||
### This change does NOT reach the live realm. At all.
|
|
||||||
|
|
||||||
Same one-shot failure this file already documents twice above, third field over. Measured on prod
|
|
||||||
2026-10-17:
|
|
||||||
|
|
||||||
```
|
|
||||||
$ kubectl --context k3s-prod -n keycloak get keycloakrealmimport homelab -o jsonpath='{.status}'
|
|
||||||
{"conditions":[{"status":"True","type":"Done"},{"status":"False","type":"Started"},
|
|
||||||
{"status":"False","type":"HasErrors"}]}
|
|
||||||
```
|
|
||||||
|
|
||||||
`Done=True` / `Started=False` — the import has completed and **will not re-run**, so a policy block
|
|
||||||
and three new sub-flows added to this file change nothing about how anyone logs in. The
|
|
||||||
client-secret/client-URL reconciler does **not** close this gap and was never meant to: its hard
|
|
||||||
endpoint allow-list makes `/authentication` unreachable **by design** (see the table above), and
|
|
||||||
realm-level policy fields are not client fields. This is exactly the "a NEW client, a new protocol
|
|
||||||
mapper or a changed scope still needs the realm import to actually re-run" case named above.
|
|
||||||
|
|
||||||
**So the manifest is the declared source of truth and a DR-rebuild input, and nothing more, until
|
|
||||||
an operator applies it by hand.** Two paths, both operator actions:
|
|
||||||
|
|
||||||
1. **Admin console (expected path).** *Authentication → Policies → WebAuthn Passwordless Policy*
|
|
||||||
for the policy block, then *Authentication → Flows → browser-mfa* to rebuild the executions to
|
|
||||||
match the manifest. The flow edit is the fiddly half: split the existing Username Password Form
|
|
||||||
into Username Form + Password Form and add the two sub-flows exactly as declared.
|
|
||||||
2. **A deliberate re-import.** Deleting and recreating the `KeycloakRealmImport` re-runs it against
|
|
||||||
the **existing** realm. Do not reach for this casually — it is the operation whose failure mode
|
|
||||||
is the 2026-09-06 total SSO outage documented above, and it has not been rehearsed here.
|
|
||||||
|
|
||||||
Either way the realm is then configured by hand and git is only the record of intent, which is a
|
|
||||||
live exception to R-IDP-03 (config as code) — not a new one, but one more instance of it.
|
|
||||||
|
|
||||||
### Verifying it actually works (nobody has, as of 2026-10-17)
|
|
||||||
|
|
||||||
No passkey has been enrolled and no browser has been pointed at this. The checks that would prove
|
|
||||||
it, in order:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
# 1. the policy is live (RP ID must read as the base domain, NOT the keycloak host, NOT a URL)
|
|
||||||
# Authentication -> Policies -> WebAuthn Passwordless Policy, in the admin console.
|
|
||||||
# 2. enrol: account console (https://keycloak.${BASE_DOMAIN}/realms/homelab/account) -> Signing in
|
|
||||||
# -> Passwordless -> set up. A browser/OS passkey prompt must appear; if it does not, the RP ID
|
|
||||||
# is wrong for the origin and NOTHING will say so.
|
|
||||||
# 3. log in: a fresh private window to any SSO-gated app. Expect username screen, then a passkey
|
|
||||||
# prompt. A platform-admin must NOT be asked for TOTP on this path.
|
|
||||||
# 4. regression (the important one): a user with NO passkey must still get the password screen,
|
|
||||||
# and a platform-admin must still be asked for TOTP there.
|
|
||||||
```
|
|
||||||
|
|
||||||
A failed passkey ceremony is usually silent in the server log — the browser refuses before Keycloak
|
|
||||||
is involved — so step 2 failing with no log entry is the expected signature of a wrong RP ID.
|
|
||||||
|
|
||||||
## Exceptions that keep native auth (do not force SSO) — ADR-0014
|
## Exceptions that keep native auth (do not force SSO) — ADR-0014
|
||||||
|
|
||||||
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
|
Plex/Seerr (Plex auth), Minecraft (Mojang/Microsoft), UniFi (native).
|
||||||
@@ -74,142 +74,6 @@ spec:
|
|||||||
bruteForceProtected: true
|
bruteForceProtected: true
|
||||||
permanentLockout: false
|
permanentLockout: false
|
||||||
sslRequired: external
|
sslRequired: external
|
||||||
# ===========================================================================================
|
|
||||||
# PASSKEYS — the WebAuthn PASSWORDLESS policy (R-IDP-04). 2026-10-17.
|
|
||||||
# ===========================================================================================
|
|
||||||
# Read this together with the browser-mfa flow at the bottom of the file: THIS block decides
|
|
||||||
# what a passkey must BE, the FLOW decides when one is OFFERED. Keycloak keeps TWO independent
|
|
||||||
# WebAuthn policies per realm and they are not interchangeable:
|
|
||||||
#
|
|
||||||
# webAuthnPolicy* -> the TWO-FACTOR policy, consumed by the `webauthn-authenticator`
|
|
||||||
# authenticator (a security key presented AFTER a password).
|
|
||||||
# webAuthnPolicyPasswordless* -> the PASSWORDLESS policy, consumed by
|
|
||||||
# `webauthn-authenticator-passwordless` (a passkey presented
|
|
||||||
# INSTEAD of a password). This is the one passkeys use.
|
|
||||||
#
|
|
||||||
# Setting the wrong family is a silent no-op: the passwordless authenticator reads
|
|
||||||
# getWebAuthnPolicyPasswordless() only (WebAuthnPasswordlessAuthenticator.getWebAuthnPolicy),
|
|
||||||
# so a policy written into the two-factor fields is simply never consulted by a passkey login.
|
|
||||||
# Only the passwordless family is declared here; the two-factor family is deliberately left at
|
|
||||||
# Keycloak's defaults because nothing in this realm uses `webauthn-authenticator` — TOTP
|
|
||||||
# (auth-otp-form) is the second factor for platform-admins, see the flow section.
|
|
||||||
#
|
|
||||||
# ADDITIVE AND OPTIONAL — THE DECISION THIS BLOCK EXISTS TO MAKE
|
|
||||||
# -------------------------------------------------------------
|
|
||||||
# Passkeys are offered ALONGSIDE the password, never instead of it, and enrolment is NOT forced.
|
|
||||||
# This is a single-operator homelab whose IdP gates Grafana, Gitea, Nextcloud, Headlamp,
|
|
||||||
# Semaphore, PiHole, Crafty, the *arr stack AND kubectl itself (the `kubernetes` client below).
|
|
||||||
# A mandatory passkey flow that misfires is therefore not an inconvenience, it is a total
|
|
||||||
# lock-out from the whole platform AND from the cluster API in one step — and the only repair
|
|
||||||
# paths left would be the Keycloak bootstrap admin and the break-glass cert kubeconfig. The
|
|
||||||
# expected gain from forcing enrolment (one human stops typing a password) is small; the
|
|
||||||
# expected loss is the platform. So: additive, optional, and password login keeps working for
|
|
||||||
# every user who never enrols.
|
|
||||||
#
|
|
||||||
# Concretely that means: no `requiredActions` block is declared anywhere in this file (see the
|
|
||||||
# long note in the flow section for why declaring one would be actively DANGEROUS), and the
|
|
||||||
# passkey step sits as an ALTERNATIVE beside the password form rather than ahead of it.
|
|
||||||
#
|
|
||||||
# RELYING-PARTY ID — THE FIELD THAT SILENTLY BREAKS EVERY PASSKEY IF IT IS WRONG
|
|
||||||
# ------------------------------------------------------------------------------
|
|
||||||
# The RP ID is baked into every credential at enrolment and can never be changed afterwards:
|
|
||||||
# a passkey created under one RP ID is unusable under another, with no error the user can act
|
|
||||||
# on — the browser simply reports that no credential is available. It is also browser-enforced,
|
|
||||||
# so it cannot be "fixed" server-side: per WebAuthn, the RP ID must be equal to, or a
|
|
||||||
# registrable-domain suffix of, the origin's effective domain. Keycloak is served at
|
|
||||||
# keycloak.<BASE_DOMAIN> (httproute.yaml, and the Keycloak CR's hostname), so the legal choices
|
|
||||||
# are keycloak.<BASE_DOMAIN>, <BASE_DOMAIN>, or the registrable domain one level up. It is NOT
|
|
||||||
# the full URL and NOT a scheme — "https://..." here would fail every ceremony.
|
|
||||||
#
|
|
||||||
# We choose BASE_DOMAIN, the shared platform suffix, NOT the keycloak host, for one concrete
|
|
||||||
# reason drawn from this repo's own history: a credential that cannot be re-keyed must not be
|
|
||||||
# bound to a name that can move. The oauth2-proxy client below ALREADY declares a redirect URI
|
|
||||||
# on auth.<BASE_DOMAIN>, i.e. this platform already contemplates serving auth on a second
|
|
||||||
# hostname; and the client-URL reconciler exists precisely because moving a hostname in git
|
|
||||||
# silently fails to reach the realm (see README, "Client URLs: the same one-shot failure").
|
|
||||||
# Binding passkeys to the keycloak subdomain would mean that relocating Keycloak invalidates
|
|
||||||
# every enrolled passkey; binding them to BASE_DOMAIN survives it. The cost is scope: any
|
|
||||||
# future WebAuthn relying party under this base domain could assert the same RP ID. In a
|
|
||||||
# homelab where every such party is already behind THIS IdP that is not a trust boundary worth
|
|
||||||
# buying. Note the residual, unavoidable risk: changing BASE_DOMAIN itself invalidates every
|
|
||||||
# passkey, which is a re-enrolment, not a repair.
|
|
||||||
#
|
|
||||||
# Written as the cluster-var so it is substituted by the same Flux postBuild pass, with the same
|
|
||||||
# value, as every hostname in this file — there is no second place the domain is decided.
|
|
||||||
webAuthnPolicyPasswordlessRpId: ${BASE_DOMAIN}
|
|
||||||
# The human-readable name the browser/OS shows in the passkey prompt and stores beside the
|
|
||||||
# credential ("Save a passkey for Homelab SSO?"). Matches the realm displayName above so the
|
|
||||||
# OS keychain entry is recognisable. It is also load-bearing on the UPDATE path: Keycloak's
|
|
||||||
# updateRealm applies the passwordless policy ONLY when this field is non-empty
|
|
||||||
# (DefaultExportImportManager line ~842), so an admin-API realm update that omits it drops the
|
|
||||||
# whole policy silently.
|
|
||||||
webAuthnPolicyPasswordlessRpEntityName: "Homelab SSO"
|
|
||||||
# ES256 then RS256 — exactly Keycloak's own DEFAULT_POLICY, declared explicitly rather than
|
|
||||||
# inherited so the realm's crypto is auditable from git. ES256 is what every platform
|
|
||||||
# authenticator (Touch ID/Face ID, Windows Hello, Android) and every FIDO2 key produces;
|
|
||||||
# RS256 is the fallback for older Windows Hello TPM-backed credentials. EdDSA/Ed25519 is
|
|
||||||
# deliberately NOT offered: authenticator and browser support is uneven, and an authenticator
|
|
||||||
# that picks an algorithm the verifier handles badly fails at enrolment, which is the worst
|
|
||||||
# moment to discover it. Order is a preference list, not an allow-list ordering requirement.
|
|
||||||
webAuthnPolicyPasswordlessSignatureAlgorithms:
|
|
||||||
- ES256
|
|
||||||
- RS256
|
|
||||||
# DISCOVERABLE CREDENTIAL REQUIRED. "Yes" is a Keycloak sentinel, not a boolean, and the login
|
|
||||||
# theme compares it as the literal string: webauthnRegister.js maps 'Yes' to
|
|
||||||
# authenticatorSelection.requireResidentKey=true and ANY other non-sentinel value to false.
|
|
||||||
# This is not optional for passkeys — a non-discoverable credential cannot be found by the
|
|
||||||
# browser without the server first naming it, so it is a second factor, not a passkey.
|
|
||||||
# (Keycloak 26.0.6 emits only the LEGACY requireResidentKey member, not the newer
|
|
||||||
# residentKey:"required"; per WebAuthn L2 the client maps requireResidentKey=true onto
|
|
||||||
# residentKey="required", so discoverable credentials are genuinely requested — but it also
|
|
||||||
# means this policy cannot express residentKey:"preferred" at all.)
|
|
||||||
webAuthnPolicyPasswordlessRequireResidentKey: "Yes"
|
|
||||||
# USER VERIFICATION REQUIRED. The authenticator must verify the human (biometric or PIN) and
|
|
||||||
# say so in the assertion. This is the field that makes a passkey login at least as strong as
|
|
||||||
# the password it replaces: with verification merely "preferred", possession of an unlocked
|
|
||||||
# phone or an unattended security key would be a COMPLETE login to every SSO-gated app — a
|
|
||||||
# single-factor possession check, strictly weaker than the password+TOTP path beside it.
|
|
||||||
# Required makes the passkey branch inherently two-factor (something you have + something you
|
|
||||||
# are/know), which is the premise the flow section relies on when it does NOT ask a
|
|
||||||
# passkey-authenticated platform-admin for TOTP as well.
|
|
||||||
webAuthnPolicyPasswordlessUserVerificationRequirement: required
|
|
||||||
# "not specified" is Keycloak's sentinel for "either kind of authenticator"
|
|
||||||
# (Constants.DEFAULT_WEBAUTHN_POLICY_NOT_SPECIFIED; the theme JS omits the WebAuthn member
|
|
||||||
# entirely when it sees this exact string). Deliberately NOT narrowed: "platform" would reject
|
|
||||||
# every YubiKey and every phone used as a cross-device passkey, and "cross-platform" would
|
|
||||||
# reject Touch ID and Windows Hello — i.e. either choice silently halves the usable hardware.
|
|
||||||
# The operator should be able to enrol a laptop AND a phone AND a hardware key.
|
|
||||||
webAuthnPolicyPasswordlessAuthenticatorAttachment: "not specified"
|
|
||||||
# ATTESTATION: none. Requesting "direct" would hand Keycloak an attestation statement it has
|
|
||||||
# no trust anchors to validate here (no metadata service is configured) and nothing to compare
|
|
||||||
# it against, since the AAGUID allow-list below is deliberately empty — so the statement would
|
|
||||||
# be collected, unverifiable, and would add an extra "share model information?" consent prompt
|
|
||||||
# on Safari/iOS for no security gain. Attestation is how you ENFORCE an approved-hardware list;
|
|
||||||
# we do not have one, so asking for it is cost without benefit.
|
|
||||||
webAuthnPolicyPasswordlessAttestationConveyancePreference: none
|
|
||||||
# Empty = accept any authenticator model. An AAGUID allow-list is a hardware whitelist: it
|
|
||||||
# would have to be maintained by hand, and every new phone, laptop or key the operator buys
|
|
||||||
# would be rejected at enrolment with a message that does not explain itself. It is also moot
|
|
||||||
# while attestation is "none" (above) — with no attestation statement there is no trustworthy
|
|
||||||
# AAGUID to match against in the first place.
|
|
||||||
webAuthnPolicyPasswordlessAcceptableAaguids: []
|
|
||||||
# Ceremony timeout in SECONDS (the login theme multiplies by 1000; 0 means "send no timeout and
|
|
||||||
# let the browser decide"). 300s is chosen over a tighter value because the slowest LEGITIMATE
|
|
||||||
# path is the cross-device hybrid flow — Keycloak renders a QR code, the operator unlocks a
|
|
||||||
# phone, scans it, and completes a biometric check — which routinely takes over a minute. A
|
|
||||||
# 60-120s timeout looks tidy and converts that flow into an unexplained failure. Browsers clamp
|
|
||||||
# this value (Chrome to at most 10 minutes), so 300 is honoured rather than reinterpreted.
|
|
||||||
webAuthnPolicyPasswordlessCreateTimeout: 300
|
|
||||||
# Send the user's existing passkeys as excludeCredentials at enrolment, so re-enrolling an
|
|
||||||
# authenticator the user already registered is refused by the authenticator itself instead of
|
|
||||||
# producing a second, indistinguishable credential in the account console. Cheap, and it keeps
|
|
||||||
# "which passkey is which" answerable — which matters when the recovery story is "delete the
|
|
||||||
# lost one".
|
|
||||||
webAuthnPolicyPasswordlessAvoidSameAuthenticatorRegister: true
|
|
||||||
# webAuthnPolicyPasswordlessExtraOrigins is intentionally NOT set. It exists to admit non-web
|
|
||||||
# origins (native app facet IDs, e.g. android:apk-key-hash:...). Every ceremony here happens in
|
|
||||||
# a browser on Keycloak's own origin, which is already covered by the RP ID above; adding
|
|
||||||
# origins is only ever a widening of who may ask for these credentials.
|
|
||||||
# --- Authorization model (R-IDP-06): groups drive access; platform-admins carry a realm role. ---
|
# --- Authorization model (R-IDP-06): groups drive access; platform-admins carry a realm role. ---
|
||||||
roles:
|
roles:
|
||||||
realm:
|
realm:
|
||||||
@@ -664,105 +528,14 @@ spec:
|
|||||||
"access.token.claim": "true"
|
"access.token.claim": "true"
|
||||||
"userinfo.token.claim": "true"
|
"userinfo.token.claim": "true"
|
||||||
"claim.name": groups
|
"claim.name": groups
|
||||||
# ===========================================================================================
|
# --- MFA for platform-admins (R-IDP-04): a browser flow whose OTP step is CONDITIONAL on the
|
||||||
# MFA for platform-admins (R-IDP-04) + PASSKEY login (2026-10-17). The realm browserFlow is
|
# platform-admin role. Non-admins log in without OTP; platform-admins must enroll/present TOTP.
|
||||||
# bound to browser-mfa at the bottom of this file.
|
# Set as the realm browserFlow below. VERIFY at bring-up (see runbook): a malformed flow fails
|
||||||
# ===========================================================================================
|
# the whole import — if so, the fallback is to comment this block + browserFlow and configure
|
||||||
# Non-admins log in without OTP; platform-admins must enroll/present TOTP. VERIFY at bring-up
|
# the conditional-OTP-by-role flow via the admin console. ---
|
||||||
# (see runbook): a malformed flow fails the whole import — if so, the fallback is to comment
|
|
||||||
# this block + browserFlow and configure the flow via the admin console.
|
|
||||||
#
|
|
||||||
# THE SHAPE, AND WHY IT IS THIS SHAPE
|
|
||||||
# -----------------------------------
|
|
||||||
# browser-mfa (top level, bound as browserFlow)
|
|
||||||
# auth-cookie ALTERNATIVE already-authenticated SSO cookie
|
|
||||||
# identity-provider-redirector ALTERNATIVE (no IdPs configured; kept for parity)
|
|
||||||
# browser-mfa-forms ALTERNATIVE the interactive path:
|
|
||||||
# auth-username-form REQUIRED WHO are you (no password yet)
|
|
||||||
# browser-mfa-passkey-or-password REQUIRED ...and PROVE it, one of two ways:
|
|
||||||
# webauthn-authenticator-passwordless ALTERNATIVE a passkey, or
|
|
||||||
# browser-mfa-password-and-otp ALTERNATIVE the password path:
|
|
||||||
# auth-password-form REQUIRED password, then
|
|
||||||
# browser-mfa-conditional-otp CONDITIONAL TOTP iff platform-admin
|
|
||||||
#
|
|
||||||
# WHY THE USERNAME/PASSWORD SCREEN HAD TO BE SPLIT IN TWO. The previous flow used the combined
|
|
||||||
# `auth-username-password-form`, which collects both on one page. A passkey branch cannot live
|
|
||||||
# behind that: the server must know WHICH user is logging in before it can decide whether a
|
|
||||||
# passkey is even available, and the combined form has already taken the password by the time
|
|
||||||
# it knows. So it is replaced by the `auth-username-form` / `auth-password-form` pair — the
|
|
||||||
# same decomposition Keycloak's own passwordless recipe uses. THIS IS A VISIBLE UX CHANGE FOR
|
|
||||||
# EVERY USER, passkey or not: login becomes two screens (username, then password) instead of
|
|
||||||
# one. That cost is accepted deliberately — it is the enabling change, it is what Google and
|
|
||||||
# Microsoft logins already do, and the alternative (below) is worse.
|
|
||||||
#
|
|
||||||
# WHY NOT USERNAMELESS ("just tap your passkey", no username screen at all). Keycloak 26.0.6
|
|
||||||
# does support it: WebAuthnPasswordlessAuthenticator.requiresUser() returns FALSE, so the
|
|
||||||
# authenticator can legally run as a first step with no user in context. We deliberately do not
|
|
||||||
# do that, because of how the flow engine then behaves. In DefaultAuthenticationFlow the
|
|
||||||
# "does this user actually have this credential?" guard sits INSIDE the requiresUser() branch;
|
|
||||||
# with requiresUser() false and no user yet set, the engine skips that guard and simply invokes
|
|
||||||
# the authenticator. A passkey execution placed before the username form is therefore offered
|
|
||||||
# to EVERYONE, so every login — including every password-only user and every family member who
|
|
||||||
# will never own a passkey — would open on a passkey prompt they must dismiss. That is the
|
|
||||||
# opposite of additive. Keeping the passkey step AFTER auth-username-form means a user is in
|
|
||||||
# context, which is what makes the graceful degradation below actually work.
|
|
||||||
#
|
|
||||||
# WHY THE PASSKEY BRANCH DEGRADES CLEANLY TO A PASSWORD (the property the whole "additive"
|
|
||||||
# claim rests on). With a user in context, the engine takes the non-requiresUser branch and
|
|
||||||
# raises CREDENTIAL_SETUP_REQUIRED when the user has no credential of this type and the
|
|
||||||
# authenticator's factory does not allow self-setup — and WebAuthnPasswordlessAuthenticatorFactory
|
|
||||||
# .isUserSetupAllowed() is false, and the authenticator is a CredentialValidator, so both
|
|
||||||
# conditions hold. That exception is raised INSIDE the ALTERNATIVE loop, which catches it,
|
|
||||||
# marks the execution ATTEMPTED and proceeds to the next alternative. Net effect: a user with
|
|
||||||
# no passkey never sees a passkey screen and lands on the password form; a user WITH one gets
|
|
||||||
# the passkey screen and keeps the password as a "Try another way" option. Note that for a
|
|
||||||
# user holding BOTH, which screen appears FIRST is decided by Keycloak's authentication
|
|
||||||
# selection list (the user's own credential ordering), not by the priorities below.
|
|
||||||
#
|
|
||||||
# WHY A PASSKEY LOGIN IS NOT ALSO ASKED FOR TOTP. browser-mfa-conditional-otp hangs off the
|
|
||||||
# PASSWORD branch only, so a platform-admin who authenticates with a passkey is not challenged
|
|
||||||
# for TOTP. That is intentional, and it rests on the policy above: userVerification is
|
|
||||||
# `required`, so the authenticator has already proved possession of hardware AND verified the
|
|
||||||
# human (biometric or PIN) in one phishing-resistant, origin-bound ceremony. Stacking TOTP on
|
|
||||||
# top would add a replayable, phishable shared secret to a credential that is neither, for
|
|
||||||
# friction on every login. If a future operator disagrees, the change is small and local: move
|
|
||||||
# the browser-mfa-conditional-otp reference out of browser-mfa-password-and-otp and into
|
|
||||||
# browser-mfa-passkey-or-password's parent as a REQUIRED step, so it applies to both branches.
|
|
||||||
#
|
|
||||||
# WHY THERE IS NO `requiredActions` BLOCK IN THIS FILE — DO NOT ADD ONE
|
|
||||||
# --------------------------------------------------------------------
|
|
||||||
# Enrolment uses Keycloak's `webauthn-register-passwordless` required action, and the obvious
|
|
||||||
# move is to declare it here. DO NOT. In Keycloak 26.0.6 the realm import treats
|
|
||||||
# requiredActions as REPLACE, NOT MERGE: DefaultExportImportManager.importRealm branches on
|
|
||||||
# `if (rep.getRequiredActions() != null)` and, when the representation declares ANY, imports
|
|
||||||
# exactly those plus delete_account — the `else` branch that installs the full default set
|
|
||||||
# (DefaultRequiredActions.addActions) never runs. RealmManager.setupRequiredActions is no
|
|
||||||
# backstop either: it fires only when the realm has ZERO required-action providers.
|
|
||||||
#
|
|
||||||
# So declaring one action would create this realm with that action and delete_account and
|
|
||||||
# NOTHING ELSE — no CONFIGURE_TOTP, no UPDATE_PASSWORD, no VERIFY_EMAIL, no delete_credential,
|
|
||||||
# no UPDATE_PROFILE. The first casualty would be the MFA requirement this very flow enforces:
|
|
||||||
# with CONFIGURE_TOTP unregistered, a platform-admin who does not yet have TOTP could never
|
|
||||||
# enrol one, while auth-otp-form below still demands it — a hard lock-out of every
|
|
||||||
# platform-admin from every SSO-gated app, including Headlamp and kubectl, delivered by a
|
|
||||||
# change whose stated purpose was to ADD a login method.
|
|
||||||
#
|
|
||||||
# And the declaration buys nothing, because the default set ALREADY CONTAINS IT.
|
|
||||||
# DefaultRequiredActions registers `webauthn-register-passwordless` with enabled=true,
|
|
||||||
# defaultAction=false, priority 80, gated on Profile.Feature.WEB_AUTHN — which is Type.DEFAULT
|
|
||||||
# in 26.0.6, i.e. on unless explicitly disabled. enabled=true / defaultAction=false IS the
|
|
||||||
# additive-and-optional posture this change wants: the action exists and can be triggered for
|
|
||||||
# a user on demand, but it is not forced on anyone at next login. Declaring it would restate
|
|
||||||
# the default while deleting ten siblings.
|
|
||||||
#
|
|
||||||
# (The symmetric question for authenticationFlows has the opposite answer, which is why THIS
|
|
||||||
# block is safe: importAuthenticationFlows adds the declared flows and then backfills each
|
|
||||||
# unbound binding — browser, registration, direct grant, reset credentials, client auth, first
|
|
||||||
# broker login, docker auth — from the built-in defaults individually. Declaring flows does not
|
|
||||||
# suppress the built-ins; declaring required actions does suppress them.)
|
|
||||||
authenticationFlows:
|
authenticationFlows:
|
||||||
- alias: browser-mfa
|
- alias: browser-mfa
|
||||||
description: "Browser flow: passkey or password, with conditional OTP for platform-admins."
|
description: "Browser flow with conditional OTP required for platform-admins."
|
||||||
providerId: basic-flow
|
providerId: basic-flow
|
||||||
topLevel: true
|
topLevel: true
|
||||||
builtIn: false
|
builtIn: false
|
||||||
@@ -782,63 +555,13 @@ spec:
|
|||||||
priority: 30
|
priority: 30
|
||||||
flowAlias: browser-mfa-forms
|
flowAlias: browser-mfa-forms
|
||||||
userSetupAllowed: false
|
userSetupAllowed: false
|
||||||
# Identify the user FIRST (username only), then prove it. Both steps are REQUIRED: the
|
|
||||||
# username is never optional, and neither is proving possession of the account.
|
|
||||||
- alias: browser-mfa-forms
|
- alias: browser-mfa-forms
|
||||||
description: "Username, then passkey or password (+ conditional OTP)."
|
description: "Username/password then conditional OTP."
|
||||||
providerId: basic-flow
|
providerId: basic-flow
|
||||||
topLevel: false
|
topLevel: false
|
||||||
builtIn: false
|
builtIn: false
|
||||||
authenticationExecutions:
|
authenticationExecutions:
|
||||||
# auth-username-form = Keycloak's "Username Form" (UsernameFormFactory) — resolves the
|
- authenticator: auth-username-password-form
|
||||||
# user WITHOUT taking a password, which is the prerequisite for offering a passkey.
|
|
||||||
# Replaces the previous auth-username-password-form; see the header for the UX cost.
|
|
||||||
- authenticator: auth-username-form
|
|
||||||
authenticatorFlow: false
|
|
||||||
requirement: REQUIRED
|
|
||||||
priority: 10
|
|
||||||
userSetupAllowed: false
|
|
||||||
- authenticatorFlow: true
|
|
||||||
requirement: REQUIRED
|
|
||||||
priority: 20
|
|
||||||
flowAlias: browser-mfa-passkey-or-password
|
|
||||||
userSetupAllowed: false
|
|
||||||
# THE FORK. Two ALTERNATIVEs, so satisfying either one completes the flow. Priority puts the
|
|
||||||
# passkey first for users who have one; users who do not fall through to the password branch
|
|
||||||
# automatically (CREDENTIAL_SETUP_REQUIRED is caught by the ALTERNATIVE loop — see header).
|
|
||||||
- alias: browser-mfa-passkey-or-password
|
|
||||||
description: "A passkey (WebAuthn passwordless), or password with conditional OTP."
|
|
||||||
providerId: basic-flow
|
|
||||||
topLevel: false
|
|
||||||
builtIn: false
|
|
||||||
authenticationExecutions:
|
|
||||||
# webauthn-authenticator-passwordless = "WebAuthn Passwordless Authenticator"
|
|
||||||
# (WebAuthnPasswordlessAuthenticatorFactory). It reads the realm's PASSWORDLESS policy
|
|
||||||
# declared above — NOT the two-factor webAuthnPolicy* family. Keep it ALTERNATIVE: as
|
|
||||||
# REQUIRED it would force a passkey on every user, which is the lock-out this change is
|
|
||||||
# explicitly designed not to create.
|
|
||||||
- authenticator: webauthn-authenticator-passwordless
|
|
||||||
authenticatorFlow: false
|
|
||||||
requirement: ALTERNATIVE
|
|
||||||
priority: 10
|
|
||||||
userSetupAllowed: false
|
|
||||||
- authenticatorFlow: true
|
|
||||||
requirement: ALTERNATIVE
|
|
||||||
priority: 20
|
|
||||||
flowAlias: browser-mfa-password-and-otp
|
|
||||||
userSetupAllowed: false
|
|
||||||
# The unchanged pre-passkey behaviour, relocated intact into the password branch: password,
|
|
||||||
# then TOTP iff the user holds the platform-admin role. Nothing about the OTP condition
|
|
||||||
# changed; it simply no longer sits on the path a passkey login takes.
|
|
||||||
- alias: browser-mfa-password-and-otp
|
|
||||||
description: "Password, then conditional OTP for platform-admins."
|
|
||||||
providerId: basic-flow
|
|
||||||
topLevel: false
|
|
||||||
builtIn: false
|
|
||||||
authenticationExecutions:
|
|
||||||
# auth-password-form = Keycloak's "Password Form" (PasswordFormFactory) — the password
|
|
||||||
# half of the old combined form, now that the username has already been collected.
|
|
||||||
- authenticator: auth-password-form
|
|
||||||
authenticatorFlow: false
|
authenticatorFlow: false
|
||||||
requirement: REQUIRED
|
requirement: REQUIRED
|
||||||
priority: 10
|
priority: 10
|
||||||
|
|||||||
@@ -186,6 +186,86 @@ right way; one was a defect in my own manifest.
|
|||||||
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
|
| Is the image's **iptables** backend compatible with the host's? | **Yes — both nft.** The image ships `iptables v1.8.13 (nf_tables)` via `xtables-nft-multi`, and Debian 13 is nf_tables. This was the documented worst fear (rules silently landing in a table the kernel does not consult); it is closed. |
|
||||||
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
|
| Can the pod set **`net.ipv4.ip_forward`**? | **No, and it must not try.** `/proc/sys` is mounted **read-only** under `hostNetwork` (`proc /proc/sys proc ro,…`), because those are the host's globals; `sysctl -w` fails with EPERM whatever the capabilities. A failed `PostUp` is fatal to `wg-quick`, so this one line took the whole tunnel down. The node already has it set to `1`; the init container now **verifies** rather than writes. |
|
||||||
|
|
||||||
|
## Measured on prod, 2026-10-17 — L2 announcement does NOT affect what a client can reach
|
||||||
|
|
||||||
|
A change was proposed (PR #69) on the belief that **a node cannot reach a LoadBalancer VIP whose
|
||||||
|
Cilium L2 announcement lease it holds**, and that the VPN node therefore had to be excluded from
|
||||||
|
announcing. **That belief is wrong, and the change was withdrawn.** The half of it that is true is
|
||||||
|
real but inert, and the distinction matters enough to record, because the wrong version is very
|
||||||
|
easy to re-derive from `ip neigh` alone.
|
||||||
|
|
||||||
|
Measured with all **eight** `cilium-l2announce-*` leases held by `k3s-prod-1` — the extreme case,
|
||||||
|
after the UPS power-cycle — from the host netns of the `wireguard` pod on each node:
|
||||||
|
|
||||||
|
| | `k3s-prod-1` (holds all 8) | `k3s-prod-2` / `-3` (hold none) |
|
||||||
|
|---|---|---|
|
||||||
|
| `ip neigh show 192.168.2.20` | `FAILED` (no lladdr) | `lladdr bc:e9:2f:7e:c4:a6 REACHABLE` |
|
||||||
|
| `curl -k https://192.168.2.20/` | **`200`, 18 ms** | `200`, 13–40 ms |
|
||||||
|
| `nc -z 192.168.2.20 443` | **OK** | OK |
|
||||||
|
| plex `192.168.2.24:32400` | **`200`** | `200` |
|
||||||
|
|
||||||
|
So the announcer really does **not** answer its own ARP request — that part of the observation is
|
||||||
|
reproducible. It changes nothing, because **no traffic path resolves a LoadBalancer VIP by ARP**:
|
||||||
|
|
||||||
|
* `cilium-dbg status --verbose` reports `KubeProxyReplacement: True [eno1 …, wg0 10.90.0.1]` and
|
||||||
|
`Masquerading: BPF [eno1, wg0]`. Cilium's LB datapath is attached to **`wg0` as well as `eno1`**,
|
||||||
|
so a remote client's packet arriving on `wg0` for a VIP is DNAT'd to a backend **in BPF, before
|
||||||
|
any route or neighbour lookup for the VIP happens**.
|
||||||
|
* `cilium-dbg service list` on `k3s-prod-1` programs `192.168.2.20:443 => 127.0.0.1:13606` — the
|
||||||
|
shared Gateway's backend is a **node-local** Envoy socket. Every node can serve that VIP without
|
||||||
|
a packet crossing the LAN at all.
|
||||||
|
* Host-originated traffic is socket-LB'd (`Socket LB: Enabled`, `bpf-lb-sock-hostns-only: true`),
|
||||||
|
which is why `curl` succeeds on the lease holder despite `FAILED`. The discriminator: a **real**
|
||||||
|
service port connects, a **bogus** one (`nc -z 192.168.2.20 9999`) fails, on all three nodes —
|
||||||
|
that is service-aware DNAT, not wire traffic.
|
||||||
|
|
||||||
|
**Two tests that look like proof and are not.** Both were used to support the withdrawn diagnosis:
|
||||||
|
|
||||||
|
* **`ip neigh` on its own.** `FAILED` on the holder is permanent and harmless. It is also an
|
||||||
|
artefact of probing: VIPs answer no ICMP, so a `ping` leaves a failed entry behind whether or not
|
||||||
|
anything is wrong.
|
||||||
|
* **`curl --interface wg0`.** This fails (`rc=28`, timeout) from **all three** nodes, including the
|
||||||
|
two whose ARP for the VIP resolves perfectly. It is not correlated with lease holding, so it
|
||||||
|
discriminates nothing — and it is not the client path either (see below).
|
||||||
|
|
||||||
|
**Why neither probe can see the client path.** A MASQUERADEd remote client's packets are
|
||||||
|
**forwarded** — netfilter `FORWARD`, then routing — whereas everything runnable from the pod is
|
||||||
|
**socket-originated**. The two traverse different hooks, so a host-netns probe cannot measure the
|
||||||
|
forwarded path in either direction: it reports success where the forwarded path might fail, and
|
||||||
|
`--interface wg0` reports failure where it does not. **The forwarded path can only be measured with
|
||||||
|
a client actually connected**, by watching the `FORWARD -i wg0` and `POSTROUTING … MASQUERADE`
|
||||||
|
counters move.
|
||||||
|
|
||||||
|
## Measured on prod, 2026-10-17 — where a full tunnel's internet traffic actually egresses
|
||||||
|
|
||||||
|
`new-peer.sh` has emitted a **full tunnel** (`AllowedIPs = 0.0.0.0/0`) **by default since the
|
||||||
|
component's first commit**; `--split` is the opt-in for LAN-only. There is no full-tunnel feature
|
||||||
|
to add. The server side was checked end to end and is **capable and correct** — if a full-tunnel
|
||||||
|
client cannot reach the internet, nothing in the list below is the reason:
|
||||||
|
|
||||||
|
| checked | result on all three nodes |
|
||||||
|
|---|---|
|
||||||
|
| `net.ipv4.ip_forward` (read from `/proc/sys`, never written) | `1` |
|
||||||
|
| `iptables -t nat -S POSTROUTING` | carries `-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` — NATs to **any** outbound interface, so internet NAT needs no change |
|
||||||
|
| `CILIUM_POST_nat`, which precedes it in the chain | **empty, 0 packets** — it cannot short-circuit the MASQUERADE |
|
||||||
|
| `iptables -L FORWARD` | `-i wg0` and `-o wg0` ACCEPT both present |
|
||||||
|
| default route | `default via 192.168.2.1 dev eno1 onlink`, intact |
|
||||||
|
| node's own internet reachability | `http://1.1.1.1/` returns `301` |
|
||||||
|
|
||||||
|
**A full tunnel egresses the HOUSE WAN, not PIA.** This was the trap worth checking, and it is
|
||||||
|
closed: `platform/vpn-egress/egress-steering.yaml`'s `CiliumEgressGatewayPolicy` selects by
|
||||||
|
`podSelector` on `egress-class: vpn`, and this DaemonSet is deliberately labelled
|
||||||
|
`egress-class: internet`, so it is not selected. Forwarded `wg0` traffic is not pod-identity
|
||||||
|
traffic at all, and `tun0` exists on **`k3s-prod-3` only** (`10.150.192.3/18`) — the egress-gateway
|
||||||
|
node. A remote client's apparent public address is therefore the **home WAN IP**, which is the
|
||||||
|
intended behaviour and the thing to state when enrolling a device.
|
||||||
|
|
||||||
|
**Not measurable without a connected client.** At the time of writing `wg show wg0` listed the one
|
||||||
|
enrolled peer with **no endpoint and no handshake** on every node, and both the `FORWARD -i wg0` and
|
||||||
|
the `MASQUERADE` counters read **0 packets** — so no client traffic had been forwarded since the
|
||||||
|
power-cycle, and which node the router forwards `udp/51820` to was **undeterminable**. Only the
|
||||||
|
forwarded node shows a peer with an `endpoint` and a recent `latest handshake`.
|
||||||
|
|
||||||
## Prerequisites outside this directory
|
## Prerequisites outside this directory
|
||||||
|
|
||||||
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
|
1. **The `wireguard` kernel module on every node.** Added to `node_baseline_kernel_modules`
|
||||||
|
|||||||
@@ -273,6 +273,84 @@ bytes of WireGuard encapsulation); drop the client to `1280` and re-test.
|
|||||||
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
|
by design. Work inwards: `wg show` on the node holding the forward will show *zero* received bytes
|
||||||
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
|
if the router is not delivering the packets, and a `handshake` with no completed session if it is.
|
||||||
|
|
||||||
|
## Triage — "the client reaches the LAN and cluster apps, but not the internet"
|
||||||
|
|
||||||
|
Reported 2026-10-17. Work **outwards from the client profile**, because that is where this symptom
|
||||||
|
almost always lives, and because the server side has been measured and is not the cause.
|
||||||
|
|
||||||
|
**1. Read `AllowedIPs` in the client's own `[Peer]` stanza. This one line settles it.**
|
||||||
|
|
||||||
|
| the profile says | meaning |
|
||||||
|
|---|---|
|
||||||
|
| `0.0.0.0/0` | **full tunnel** — the default. Internet *should* be carried. Go to step 2. |
|
||||||
|
| `192.168.2.0/24, 10.90.0.0/24` | **split tunnel** — `--split` was passed, or the profile predates the script. Internet never enters the tunnel **by design**; it uses the device's own uplink, and if that is broken the VPN cannot fix it. Re-render without `--split` to change it. |
|
||||||
|
|
||||||
|
`new-peer.sh` emits a full tunnel **by default** and has since the component's first commit —
|
||||||
|
`--split` is the opt-in. There is no server-side switch for this: **it is a property of the client
|
||||||
|
profile only**, so an already-issued profile keeps whatever it was rendered with until it is
|
||||||
|
re-rendered and re-imported on the device. The two forms differ by one flag:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet>' # full tunnel
|
||||||
|
./infra/devcontainer/dev.sh run bash -c './infra/vpn-ingress/new-peer.sh <name> <octet> --split' # LAN only
|
||||||
|
```
|
||||||
|
|
||||||
|
Re-rendering mints a **new keypair**, so it also needs the new `[Peer]` stanza PR'd into
|
||||||
|
`peers.conf` and the old one deleted. Changing an existing device in place, without new keys, is
|
||||||
|
editing `AllowedIPs` in the profile **on the device** — the server does not care.
|
||||||
|
|
||||||
|
**2. If it is a full tunnel, prove the client traffic is reaching the node at all.** Have the
|
||||||
|
client connected, then watch the two counters that only forwarded traffic moves:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./infra/devcontainer/dev.sh run bash -c \
|
||||||
|
'kubectl --context k3s-prod -n vpn-ingress exec ds/wireguard -- sh -c "
|
||||||
|
wg show wg0 endpoints; iptables -L FORWARD -n -v | head -5;
|
||||||
|
iptables -t nat -L POSTROUTING -n -v | head -5"'
|
||||||
|
```
|
||||||
|
|
||||||
|
* Only the **forwarded node** shows a peer with an `endpoint`; that is the only way to learn which
|
||||||
|
node the router's `udp/51820` forward points at (the router is unreadable from the cluster).
|
||||||
|
* `FORWARD -i wg0` at **0 packets** on every node means nothing is arriving — the problem is
|
||||||
|
upstream of the cluster (router forward, client, or carrier), not here.
|
||||||
|
* `FORWARD -i wg0` moving while `POSTROUTING … MASQUERADE` stays at **0** is the one server-side
|
||||||
|
failure this symptom could have: un-SNATed `10.90.0.x` packets reach the router and die there,
|
||||||
|
which breaks internet while node-local cluster VIPs still work. It has not been observed.
|
||||||
|
|
||||||
|
**3. Suspect DNS and address family before suspecting routing.** Both fit this symptom exactly:
|
||||||
|
|
||||||
|
* **IPv6.** `AllowedIPs = 0.0.0.0/0` carries **IPv4 only** (see Known limitations). A dual-stack
|
||||||
|
client resolving a public name gets an AAAA, prefers IPv6, and sends it **outside** the tunnel —
|
||||||
|
so LAN and cluster names (IPv4/RFC1918) work while public sites hang. Test by forcing IPv4:
|
||||||
|
`curl -4 https://example.com` from the client. If `-4` works and plain `curl` hangs, this is it,
|
||||||
|
and the fix is on the client (disable IPv6 on the tunnel, or add `::/0` to `AllowedIPs` so it is
|
||||||
|
blackholed rather than leaked — note the tunnel has **no** IPv6 egress, so that blackholes it).
|
||||||
|
* **PiHole.** The profile sets `DNS = 192.168.2.23`. It does resolve public names — verified
|
||||||
|
2026-10-17, `google.com` answered from inside the cluster — but it is also an ad-blocker, so a
|
||||||
|
blocked name is indistinguishable from "no internet" on one site. Test a name you know is not on
|
||||||
|
a blocklist, and compare against `curl https://1.1.1.1/` by **IP**, which needs no DNS at all.
|
||||||
|
|
||||||
|
**4. What is NOT the cause.** Measured 2026-10-17, all three nodes: `ip_forward=1`; the
|
||||||
|
`-s 10.90.0.0/24 ! -o wg0 -j MASQUERADE` rule present, with the preceding `CILIUM_POST_nat` chain
|
||||||
|
**empty** so it cannot short-circuit it; `FORWARD -i/-o wg0` ACCEPT present; the default route
|
||||||
|
`via 192.168.2.1 dev eno1` intact; and the nodes themselves reaching `1.1.1.1`. Also **not** the
|
||||||
|
PIA egress gateway: it selects pods by `egress-class: vpn`, this DaemonSet is labelled
|
||||||
|
`egress-class: internet`, and `tun0` exists on `k3s-prod-3` only — so a full tunnel egresses the
|
||||||
|
**house WAN** and the client's apparent address is the **home WAN IP** (check 4 in step 6).
|
||||||
|
|
||||||
|
## Triage — which L2 announcement lease a node holds does NOT matter
|
||||||
|
|
||||||
|
Do not spend time here. It was investigated at length (PR #69, withdrawn) on the belief that a node
|
||||||
|
could not reach a VIP it announced. Measured 2026-10-17 with **all eight** leases on `k3s-prod-1`:
|
||||||
|
that node reached `192.168.2.20`, `.22`, `.23` and `.24` normally. The lease holder's `ip neigh`
|
||||||
|
for its own VIP does sit at `FAILED` **permanently and harmlessly** — Cilium DNATs LoadBalancer
|
||||||
|
VIPs in BPF on both `eno1` and `wg0` before any neighbour lookup, and the shared Gateway's backend
|
||||||
|
is a node-local socket. Full measurements and the two misleading probes
|
||||||
|
(`ip neigh`, `curl --interface wg0`) are in `platform/vpn-ingress/README.md`.
|
||||||
|
|
||||||
|
`192.168.2.21` (unifi) fails from **all three** nodes, including ones whose ARP resolves. That is a
|
||||||
|
separate, pre-existing fault in that service and has nothing to do with the VPN.
|
||||||
|
|
||||||
## Revoking a device
|
## Revoking a device
|
||||||
|
|
||||||
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
|
Delete its `[Peer]` stanza from `peers.conf`, PR, merge. Nothing else — the key stops working on the
|
||||||
|
|||||||
Reference in new issue
Block a user