a1d89f4a5fd3e5dd2f4081dc477ec98537be9381
262
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5.5
|
19097e4d4f |
chore(pm-agent): pin agent-runner to the run-394 digest (466f63e, PR #99), bump config-generation
Dispatcher image and PM_WORKER_IMAGE both move to sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2, the digest the registry returns for agent-runner:sha-466f63ec3c3d and :main. config-generation 4 -> 5. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
664ee82b9a |
fix(pm-agent): log the CLI's stdout envelope when triage fails; add PM_DECISION_MAX_TURNS
`claude --print --output-format json` reports its own failures (API error, usage limit, error_max_turns) as the result envelope on stdout and leaves stderr empty, so ask_pm()'s stderr-only message logged "claude exited 1: " with nothing after it, twice on 2026-10-31. describe_cli_failure() now logs the envelope's diagnostic fields plus stdout/stderr tails, redacted (sk-ant-, Bearer/token, URL userinfo). PM_DECISION_MAX_TURNS (default 1, unchanged) makes the max-turns remedy a ConfigMap change if the envelope turns out to say error_max_turns. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
0d3e51acc4 |
chore(pm-agent): turn PM_DRY_RUN off (operator-approved), bump config-generation to 4
The run-383 worker image (df36d9b, pinned in #96) bakes imagePullSecrets: pm-agent-registry, which admission requires, and the dispatcher's dry-run triage of #97 was read and approved. PM_REQUIRE_LABEL=agent and the worker caps are unchanged. The dry-run status text in the configmap comments, the README and the plan's follow-up 3 is updated to match. Pre-flight (read-only, k3s-prod): pm-agent-job-shape Ready+Enforce; SA/Role/RoleBinding present and can-i create jobs = yes; pm-agent-registry and pm-agent-credentials exist; both egress CNPs Valid. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
6a192ede2a |
chore(pm-agent): pin agent-runner to the run-383 digest (ddd04d4), bump config-generation
c4eba5b (run 340) bakes a worker template without imagePullSecrets, so the new admission rule refuses worker Jobs. df36d9b is agent-runner:main == :sha-ddd04d460ef6 (measured on the registry), and its /etc/pm-agent/worker-job-template.yaml carries imagePullSecrets: pm-agent-registry (read from the published layer). Renovate found df36d9b but holds it under "Awaiting Schedule" on dashboard #91: the operator's dispatch (run 385) was a real run at 12:14 New York, outside the 03:00-06:00 window. The renovate-digest markers are unchanged, so later bumps remain Renovate's. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85
|
9db20e8ca0 | Added the credentials | ||
|
jakman85
|
553d027f7c | Merge branch 'main' into feat/pm-agent-registry-pull-secret | ||
 renovate-bot
|
4ae648f6d3 | chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to c4eba5b | ||
|
jakman85andClaude Opus 5.5
|
24db269ea8 |
feat(renovate): track the Keycloak operator (the real Keycloak version), never auto-merge it
The Keycloak CR has no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the server image; the three keycloak-k8s-resources release URLs in platform/keycloak/kustomization.yaml are therefore THE Keycloak version, and Renovate could not see them (no `# renovate:` marker). They now carry markers under one depName (one branch, groupName keycloak-operator), with a packageRule: automerge false, labels needs-review + blast-radius; majors still wait on the dashboard. The untagged quay.io/keycloak/keycloak in catalog/keycloak.yaml renders apps/wave-1/keycloak.yaml, which no kustomization lists. It is pinned to the operator's 26.0.6 as documentation, re-rendered, and Renovate is disabled for that image so it cannot raise PRs against a file nothing deploys. The render is kept (not deleted): it is the fixture for apps/wave-1/tests/app-slis.test.yaml's keycloak SLI rule tests. README: an "Upgrading Keycloak" section — an operator bump must be checked against passkey-reconciler.yaml's 26.0.6 Admin API assumptions. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
7f0cbc7ba2 |
feat(pm-agent): pin the first published agent-runner digest, start dry, and fix the token scopes
Rebased #61 onto the CI image pipeline (#81). This commit makes the remaining in-git blockers to going live go away: - deployment.yaml and PM_WORKER_IMAGE pin images run 340's digest (sha256:c4eba5b3..., tag sha-523cd9118f95 = main), read from the registry. Renovate markers unchanged. - PM_DRY_RUN=true and PM_REQUIRE_LABEL=agent for the first run. Dry run is required, not just cautious: the run-340 image's baked worker template predates imagePullSecrets, so admission would deny its Jobs until CI re-publishes after this merge and Renovate bumps. - Token scopes: Gitea 1.24.3 scopes GET /api/v1/user (the dispatcher's first call) under read:user. The documented list was missing it. Now read:user, write:repository, write:issue, read:package everywhere. - Org-package read documented as a team (Packages=Read, no repos), not just a scope. - sops is toolbox-only; the .examples say so and give the in-place edit. - runbook: key-name check no longer emits Secret values before the pipe. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a7eadd4054 |
feat(pm-agent): the kubelet's pull credential, and the seventh field admission had not read
The registry had two blockers. #59 fixed the first (nodes resolved the hostname to the non-hairpinned WAN and failed TLS against the router's CN=%D cert). This is the second: the pull is unauthenticated, and this registry refuses unauthenticated clients outright — measured 2026-10-05, its anonymous token carries access: null and every path 401s. pm-agent-registry is a dockerconfigjson Secret (SOPS, placeholders) referenced by imagePullSecrets on the dispatcher Deployment and on the committed worker Job template. It holds the SAME Gitea token as pm-agent-credentials, because the two consumers need two formats: the application reads a raw token from env, the kubelet reads a dockerconfigjson before any container starts. Same one-value-in-two-files pattern as the Keycloak client secret, documented in both .example files. The operator's part is the value plus one new scope: read:package. ADMISSION GAINED A SEVENTH RULE. imagePullSecrets is a pod-spec field no container references, which is exactly why it is easy to leave unread — the same shape of omission as initContainers, ephemeralContainers, env and envFrom before it. The kubelet reads the named Secret, in this namespace, before anything runs. Rule 1's pattern now REQUIRES the committed entry; rule 7 forbids any other and any second one. Both halves are needed: a foreach over an absent list yields no result. The leverage is small and the policy says so: the image is digest-pinned, so a different pull credential cannot change WHAT runs. Pinned anyway — 'only the committed shape' is the premise, and the rule costs four lines. kyverno test: 33 passed, 0 failed (was 28). Three new fixtures, each a real path: the committed entry plus a second one, a different Secret, and none at all. The gate also caught me shipping the Secret in plaintext — I never ran sops -e -i on the placeholder. Recorded in the handoff entry, because the file looked finished. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
262c6e4a97 | Merge branch 'main' into feat/keycloak-passkey-reconcile | ||
|
jakman85andClaude Opus 5.5
|
d2c4bd5194 |
fix(buildkit): allow buildkitd egress to Gitea's backend so the Gateway stops 403ing the push
Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as
|
||
|
jakman85andClaude Opus 5.5
|
04eb38584a |
fix(conformance): workload-mTLS certificates may use a private CA, if they prove they are workload-internal
R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
f65c12b395 |
feat(images): build on rootless BuildKit over mTLS; verification moves into Containerfile verify stages
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched. - platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined, read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP :1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it). No Kyverno exception: every applied policy matches app-namespace=true only. - Each Containerfile names its shipped stage and ends in a `verify` stage holding every former `docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now. - images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes. Fails loudly when the builder secrets are absent or buildkitd is unreachable. - infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
e539109830 |
feat(images): build, verify and publish the three images in CI; Renovate carries the digests
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a workstation build/push. Nothing had ever been pushed to the Gitea registry. - .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as sha-<commit> + main, reading the digest back from the registry. Only changed images build. - Each image's own build-image.sh does the build and in-image verification, now engine-neutral (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh. - Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex manager for them. The worker Job template is deliberately untracked (it would loop). - renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION. - Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
6bb8af1d70 |
docs(decisions): handoff for the passkey reconciler review fixes
Also states in bind_browser_flow()'s docstring that it is reached only when git's bind-browser-flow is "true". Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
7c4eb439b2 |
feat(monitoring): alert when the keycloak passkey reconciler's Job fails
KeycloakPasskeyReconcilerFailing (warning, for 15m) on
kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"},
following the sibling client-secret reconciler's rule. promtool unit tests pin it in
both directions: quiet when succeeding, firing after 15m on a failed Job, quiet on the
SIBLING's failure, and quiet on a single transient failure. Runbook added and projected
by the runbooks ConfigMap generator.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
||
|
jakman85andClaude Opus 5.5
|
17eed6e8ae |
fix(keycloak-passkey): bind only on git's say-so; protect real sub-flows; screen the requirement PUT by body id
Review findings on the passkey reconciler, fixed:
1. HIGH - the bind was automatic and rollback did not stick. A git-declared switch,
`bind-browser-flow` in passkey-desired-state.yaml, now decides whether pass 3 may
PUT browserFlow. Default "false": build + verify the candidate (or reuse a verified
one), log WOULD BIND <alias>, never touch the binding - including after an earlier
bind, so an operator's rollback sticks, and a hand-edited bound candidate is left
alone rather than shadowed by an -r2 build. "true" binds as before. The renderer
carries the one hand-edited line through and --check refuses anything but the
quoted strings "true"/"false". --no-bind can only narrow it.
2. HIGH - the protected-subtree guard read `alias` on sub-flow rows. Keycloak 26.0.6
AuthenticationManagementResource.recurseExecutions sets displayName =
flowRef.getAlias() and flowId on a sub-flow row and never sets alias (alias is the
CONFIG alias on authenticator rows), so the protected set was only the top-level
alias. Sub-flows are now named by displayName (getFlows() lists top-level flows
only, so flowId cannot be resolved through it); a sub-flow row without one leaves
the subtree undetermined and the flow pass fail-closed. The selftest stub now
returns 26.0.6-shaped rows, and a real run is asserted to refuse writes into
browser-mfa-forms and browser-mfa-conditional-otp.
3. MEDIUM - PUT /flows/{alias}/executions was screened on the path alias only, but
updateExecutions resolves the execution realm-wide by body id. body.id is now
screened against the protected ids AND required to be in the tree this run is
building (recorded from the build's own readback); the config POST is held to the
same build scope; and the body may carry only id/requirement/priority (the same
handler renames any flow named by flowId).
4. Stale "audit-only" docs rewritten (platform-config kustomization, chainsaw test,
desired-state header and its renderer template, README) to describe what ships.
5. backoffLimit 0 (a retried deterministic build failure burns the candidate-alias
budget); the stale "Renovate does not manage image: tags" comment corrected and the
image pinned by index digest.
6. The ~160-line header duplicated in the YAML and the script is now kept once, in the
script, with a short pointer above it.
Selftest: 259 assertions (was 216). mutations.py: 40 injected regressions (was 30),
including one per finding above, all caught.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
f5931944fa |
feat(keycloak): reconcile the passkey realm config via GitOps, the browser flow blue/green
PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa flow in the realm import. That declaration could not take effect: a KeycloakRealmImport is a ONE-SHOT, and prod's reads Done=True / Started=False, so every realm-level field and flow declared after the first import is inert -- in git, applied by Flux, reported Ready, reaching the realm never. Merging it changed nothing about how anyone logs in, and nothing anywhere said so. Same failure class as the 2026-09-06 client-secret incident, third field over. A SIBLING reconciler converges all three declared things every 15 minutes. It is a sibling, not a second pass of client-secret-reconciler.yaml, so that reconciler's allow-list -- and its guarantee that prod's real user and the media-users / platform-admins / viewers groups are structurally unreachable -- stays literally true. Its selftest still passes unchanged. pass 1 webAuthnPolicyPasswordless* -- strict read-modify-write of GET/PUT /admin/realms/homelab, readback-verified. pass 2 webauthn-register-passwordless -- strict read-modify-write of ONE literally-named alias; enabled, defaultAction false so enrolment stays opt-in. pass 3 the browser flow -- BLUE/GREEN. PASS 3 NEVER MUTATES THE FLOW USERS ARE LOGGING IN THROUGH. Converging a five-flow, ten-execution tree needs create-subflow, create-execution, set-requirement and bind-config as SEPARATE untransacted calls; doing that to the BOUND flow means a run dying between them leaves a half-built login path for every SSO app and kubectl, and this repo has had a total SSO outage from a realm change. So instead: 1. BUILD under a NEW, unused alias (declared alias + a hash of the desired shape, applied to EVERY alias in the tree -- reusing browser-mfa-forms would mean writing INTO the bound tree). A failure leaves an orphaned, unbound flow and zero user-visible effect. 2. VERIFY by readback -- levels, ordering, authenticators, requirements, and the conditional-OTP config CONTENTS -- before anything is bound, so the regression that once 403'd every platform-admin is caught BEFORE it bites. 3. BIND with ONE atomic PUT setting browserFlow. 4. ROLLBACK is one write; no DELETE exists anywhere in the allow-list, so the old flow survives as the rollback target. The widened allow-list is made safe structurally. It cannot say "but not the bound flow" -- its write entries are alias-parameterised -- so a second guard screens every write against the live browserFlow's WHOLE subtree (sub-flow aliases and execution ids) before a socket opens, FAIL-CLOSED until that binding has been read. The create-flow endpoint carries no alias in its path, so it is screened on its request BODY; the selftest caught that it otherwise escaped the guard. Idempotence is load-bearing: a rebuild-and-reflip every 15 minutes would churn the login path on a schedule. The bound flow is compared STRUCTURALLY and a match produces zero writes and no new flow. Sub-flow and config ALIASES are deliberately not compared -- a built flow carries a hash suffix, so comparing labels would make it permanently unrecognisable as converged. Single source of truth: the desired state is GENERATED from the realm import by ci/render-keycloak-passkey-policy.py and gated with --check, so the RP ID -- baked into every credential at enrolment and never re-keyable -- is declared once. The pod keeps automountServiceAccountToken: false and reads a mounted ConfigMap, so no rbac.yaml is added anywhere. Read from Keycloak 26.0.6 source before being relied on: updateRealm (DefaultExportImportManager 725-894) makes no getUsers/getClients/getGroups/ getRoles/getRequiredActions/getAuthenticationFlows call; it DOES read getBrowserFlow(), which is what makes the atomic bind possible and why the policy write pins it; the passwordless policy is applied all-or-nothing only when RpEntityName is non-empty, with defaults substituted for omitted keys; and updateExecutions compares rep.getPriority(), so a body omitting `priority` would silently reshuffle the whole execution order. Gate coverage: a new whole-repo L0 check extracts the script and runs its --selftest (216 assertions, including a build killed at four different points each asserting the old flow is still bound and the bound subtree was never written), plus mutations.py (30 injected regressions, all caught) proving the selftest can fail. An earlier iteration of this work only AUDITED the flow and left the restructure manual; decisions/2026-10-18-keycloak-passkey-reconciler.md records that design and why it was reconsidered. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
ee196c7bcd |
Merge pull request 'feat(upgrades): automate node reboots (kured) and wire Renovate, which had never run' (#73) from feat/automated-upgrades into main
Reviewed-on: #73 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85
|
24bea918ab |
Merge pull request 'fix(coredns): hold in-cluster DNS at 2 replicas via an HPA, because k3s owns the Deployment' (#74) from fix/coredns-ha into main
Reviewed-on: #74 |
||
|
jakman85andClaude Opus 5
|
5642881c4d |
fix(coredns): hold in-cluster DNS at 2 replicas via an HPA, because k3s owns the Deployment
MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster DNS was a single pod on a single node, so draining or rebooting that node was a CLUSTER-WIDE in-cluster DNS outage until the pod rescheduled -- every Service lookup, every CNPG replica finding its primary, every operator addressing the API by name. It was invisible because 1/1 Ready looks healthy. It is urgent rather than latent now: the scheduled reboot automation on feat/automated-upgrades drains and reboots one node at a time, unattended, overnight -- turning a single point of failure into a recurring one. WHY AN HPA AND NOT replicas: 2 -- the whole decision. k3s OWNS the coredns Deployment: AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml, managedFields managers deploy@k3s-prod-1 + k3s, objectset.rio.cattle.io/hash label. k3s re-stages its bundled manifests on startup, so replicas: 2 written there -- or a kubectl scale -- is reverted on the next restart or upgrade. Rejected alternatives: * coredns.yaml.skip + our own templated copy (the roles/kube_vip pattern). Deterministic, but takes ownership of coredns from k3s permanently so k3s upgrades stop updating it. Too high a price for a replica count. * A Flux patch of the Deployment. Rejected outright: Flux and k3s's deploy controller would both own one object and fight -- the default-deny NetworkPolicy drift-loop shape (~2500 events/10min) this cluster already hit. An HPA is a separate object k3s does not own, driving /scale rather than the Deployment spec, so there is no shared-ownership fight; if k3s re-stages replicas: 1 it is re-asserted within a reconcile. Disclosed cost: a brief window at 1 replica after a k3s restart. It is a FLOOR-HOLDER, not an autoscaler -- the CPU metric exists only so the autoscaling/v2 object is valid. Measured: 5m/4m against a 100m request (~5%), so the 80% target is never approached. scaleDown stabilization 600s because DNS is the worst thing to flap; scaleUp immediate, since that restores redundancy. The kustomization edit is ADDITIVE: platform/coredns/ already existed and holds the coredns-custom split-horizon ConfigMap (R-DNS-01/03), which is untouched. UNAPPLIED. The HPA has never run, and the claim that k3s reverts replicas on restart is reasoned, not observed. After merge check `describe hpa coredns` for ScalingActive=True -- an inert HPA looks identical to a working one because the hand scale already left 2 replicas. See the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a2440a3125 |
feat(upgrades): automated OS reboots (kured) + security-only unattended-upgrades, and give Renovate a runner
Two halves of one repeatable upgrade process. Neither works alone. (a) OS upgrades and reboots platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml: that tier gates platform-config and every apps wave, so a wedged reboot daemon there could stall every application. A component added to make maintenance safer must not be able to cause an outage. infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other open branches, and this is a different lifecycle), enabled for prod in group_vars. The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's rejection of automating away the gate between nodes: u-u is security-only with the kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then does only the mechanical half, including the ordering rule node_baseline has to shout. Quorum: concurrent reboots are structurally impossible, not merely unlikely. concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency), and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node holds it forever and nothing cascades. Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait, serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m (unconditional) plus kured's Prometheus alert gate (semantic). alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and under the chart default kured would have been blocked on day one by the five alerts measured firing with no end in sight, and never rebooted anything. Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it. Asserted against apt-config dump, not against the file just written. 6.12 kernels are still installed and bootable on all three nodes, and nothing was held. Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING could create the sentinel. kured alone would have idled forever at 1/1 Running. platform/kured/alerts.yaml — three rules over kured's only metric, with a companion promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md. Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this cluster's 7d Prometheus retention. (b) Image/dependency automation Renovate was aspirational: renovate.json present and complete, with no workflow, no in-cluster workload, and its docker+kubernetes managers disabled — so every plain image: tag was maintained by nothing while seven files claimed otherwise. .gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a skipped Gitea job reports success and that is how this stayed inert. Renovate now owns both halves; Flux image-automation is formally closed: it pushes commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here anyway — main is protected user_can_push:false and the flux-system GitRepository has no secretRef. Its controllers are installed and have idled 43 days with zero CRs. conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing; tightens the check for this component (requires service.create+metrics.create, both chart-default false, plus a companion NetworkPolicy requirement). Required because assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is shrink-only. Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author approving review; /approve-override deliberately NOT posted. NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role never executed, and whether CNPG switches its primary on a drain-induced eviction (the most likely way the first real reboot fails). Full list in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
16894b7b73 |
feat(keycloak): declare passkey (WebAuthn passwordless) login for the homelab realm
Adds the realm's WebAuthn Passwordless policy and restructures the browser-mfa
flow so a passkey is an ALTERNATIVE beside the password form. Additive and
optional by design: this IdP gates every SSO app plus kubectl, so a mandatory
passkey flow that misfires is a total lock-out. Enrolment is not forced.
Policy: RP ID is ${BASE_DOMAIN} (not the keycloak host) because a passkey's RP ID
is fixed at enrolment and must not be bound to a name that can move; resident key
required (a non-discoverable credential is a second factor, not a passkey); user
verification required (without it, possession alone is a complete login).
Flow: auth-username-password-form is split into auth-username-form +
auth-password-form, because the server must know who is logging in before it can
decide whether a passkey exists. Conditional OTP for platform-admins is relocated
intact onto the password branch only — a UV-required passkey is already two
phishing-resistant factors.
No requiredActions block is declared, deliberately: Keycloak 26.0.6 treats
requiredActions as REPLACE not merge, so declaring one would delete the default
set including CONFIGURE_TOTP and lock every platform-admin out of the OTP step
this flow still enforces. webauthn-register-passwordless is already registered
enabled-but-not-default by Keycloak's own defaults.
UNAPPLIED: the realm import is a one-shot resting at Done=True/Started=False, so
this reaches the live realm never. See the handoff entry and the README section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
c150327a30 |
fix(oauth2-proxy): gate startup on Keycloak discovery, or a cold boot CrashLoops the proxy
Exposed by the 2026-10-17 whole-cluster shutdown. oauth2-proxy performs OIDC
discovery ONCE at startup and exits 1 if it fails. On a cold boot Keycloak is
not serving yet, the shared Gateway answers 503, and the proxy dies:
[provider.go:55] Performing OIDC Discovery...
[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover
OIDC configuration: unexpected status "503": no healthy upstream
It then sits in exponential CrashLoopBackOff long after Keycloak is healthy --
one replica was still down minutes later and only a manual pod delete cleared
it. There is no retry flag, a readinessProbe cannot help (the container is
already dead), and CrashLoopBackOff backoff is not tunable per-pod.
Add an extraInitContainers gate (wait-for-idp) that polls the discovery doc
until it answers. A pod in Init accrues no CrashLoopBackOff backoff and retries
promptly. BOUNDED and FAILS OPEN: after ~5m it starts the proxy anyway, so the
worst case is exactly today's behaviour and never something new.
extraInitContainers is the chart's OWN hook for this -- its commented example is
literally `- name: wait-for-idp` -- and the key was CONFIRMED against the pinned
chart (helm show values, 10.7.1) rather than assumed, because a wrong Helm values
key is silently ignored. No new base image: the chart already runs an alpine init
container for its built-in waitForRedis gate; pinned to 3.22 (verified upstream)
rather than the chart's `latest`.
Rejected: Flux postRenderers (no precedent in this tree, and the chart has a
hook); --skip-oidc-discovery (removes the fetch entirely, but needs four
hardcoded endpoints and I could not verify lazy-JWKS offline -- this component
gates SSO for seven apps, so the additive option wins).
Shell vars are written BARE with no braced form: platform declares
postBuild.substituteFrom, and envsubst would blank a braced shell variable and
silently break the loop. Same convention as the gitea/nextcloud OIDC hooks.
UNAPPLIED; the init container has never run, and the race was not re-created to
prove the fix. Check the init container's log on first apply. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
0747849837 |
fix(bulk-nfs): repoint at the TrueNAS (192.168.2.30); the PVs must be recreated, not reconciled
Both ZFS pools moved off the Proxmox host onto a standalone TrueNAS on 192.168.2.30 -- prod's OWN /24, so the pfSense crossing that made this knob load-bearing is gone, along with the DNAT at 192.168.2.102 it mounted through. Measured 2026-10-07: 192.168.2.102:2049 no longer answers; .30:2049 and :111 do. BULK_NFS_SERVER -> 192.168.2.30 in prod, base and dev. The dev value is marked inert (dev is decommissioned); it carries the real address only so nothing in the tree renders a host that no longer exists. THE OPERATIONAL POINT, and why this is not just a value change: a bound PersistentVolume's NFS source is immutable. Server dry-run, 2026-10-07: spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is immutable after creation so Flux's apply FAILS rather than updating, and the pods keep their old, dead mount. The five PVs must be deleted and recreated -- safe, because Retain never touches data and render.py emits a pre-set claimRef so each PV rebinds its own PVC. The sequence is docs/BULK-NFS-STORAGE.md section 10 and is an OPERATOR action; this commit does not perform it. Pairs with the hazard already on the page: spec.nfs.server is unvalidated at create (any string binds) and immutable at update. A wrong value is accepted silently and then cannot be corrected in place. docs/BULK-NFS-STORAGE.md: sections 1 and 5 rewritten for TrueNAS -- the measured exports, the service settings that decide whether uid 1000 survives the wire, the three stale ZFS sharenfs properties that had to be cleared, and the ACL hazard (acltype=nfsv4 + aclinherit=passthrough means one click in the ACL editor propagates to the whole future library). Section 7 rewritten now that the placeholder is gone, section 9 retargeted from dev to prod, section 10 new. Sections 2/3/4/6 are unchanged by the move. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
2acfed460a |
fix(vpn-ingress): content-hash the peer register, or a merged peer never reaches the tunnel
The operator enrolled a phone, merged the peer, and `wg show` listed none. Measured on prod 2026-10-06 with the peer already on main: live wg-peers ConfigMap, active AllowedIPs .. 0 DaemonSet volume reference ................. configMap.name: wg-peers (FIXED) DaemonSet generation/observedGeneration .... 1 / 1 (never rolled) init container, all three pods ............. "0 peer(s)" Part of it was just the 10m reconcile interval. The defect is the fixed name: Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is byte-identical, so no rollout is triggered -- and the init container assembles wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old peer list indefinitely. Adding or revoking a peer had no effect until something unrelated restarted the pods. For a REVOCATION that is a security defect, not an inconvenience: delete a stanza, see a green PR, device keeps working. The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet". Never true. And this repo already documents the fix, in a file I had read -- runbooks/kustomization.yaml: "The generated name carries kustomize's content hash, so editing any runbook rolls the server automatically and a stale projection is impossible." So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a configMapGenerator. The operator's enrolled peer is carried across verbatim. Every line is WireGuard config and `#` is a legal comment there, so the documentation now lives inside the file that is actually mounted. `namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a generated ConfigMap inherits no namespace, everything else here declares vpn-ingress, and kustomize's nameReference will not rewrite across a namespace boundary -- without it the build mounts the unhashed name and the pods wedge on a missing volume. Caught only by building the output and reading the volumes. new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that after the rename the only symptom would have been duplicate detection quietly never firing again. Stale peers.yaml references swept from six files including the script's own user-facing output. Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content (edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's real allocation. No sensitive paths; merges on a green gate. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
23db97b958 | Added my phone to peers | ||
|
jakman85andClaude Opus 5
|
146dd9a906 |
fix(vpn-ingress): new-peer.sh counted its own commented-out example as an allocated address
The first real enrolment failed:
$ ./infra/vpn-ingress/new-peer.sh keith-phone 2
new-peer.sh: 10.90.0.2/32 is already allocated ... Pick another octet.
Nothing was allocated -- peers.yaml had zero peers, as the pod's init
container reported (`0 peer(s)`). The duplicate-address check grepped the
whole file, and peers.yaml line 66 is a COMMENTED-OUT example stanza I wrote
containing literally `# AllowedIPs = 10.90.0.2/32`. So .2 -- the first
address anyone would ask for, and the one my own runbook uses as its example
-- was permanently un-allocatable, with a message asserting the opposite of
the truth.
Fixed on both sides, because either alone leaves the trap half-set:
* the check strips comment lines before matching, which is the correct
predicate rather than a patch: a real stanza's AllowedIPs line is never
commented, while the template and every peer's `# <name> — <owner>` label
always are.
* the example now uses placeholders that cannot match anything real
(`10.90.0.<octet>/32`, `<device-name>`), so a future grep cannot be fooled
by it either.
The bug class is worth naming: a documentation example that is byte-identical
to real data will eventually be parsed as real data. This is its third
instance in this component -- flux-postbuild-substitution flagged a `${...}`
written inside a comment explaining that hazard, and no-plaintext-secrets
flagged the .sops.yaml.example because a kustomization named it in a comment.
Make examples structurally unmatchable, not merely commented out.
Verified by running it: .2 now renders a profile (via the real
`sops -d | wg pubkey` path, which this also proves against the operator's
actual key for the first time), and an uncommented duplicate stanza is still
correctly refused.
No sensitive paths; merges on a green gate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
5c43fbac65 |
fix(vpn-ingress): stop trying to set net.ipv4.ip_forward — a hostNetwork pod cannot, and wg-quick dies on a failed PostUp
The tunnel never came up on its first reconcile with a real key. One line of my
own manifest was the whole cause. Measured on prod 2026-10-06:
[#] ip link add dev wg0 type wireguard <- fine
[#] wg addconf wg0 /dev/fd/63 <- fine
[#] ip -4 address add 10.90.0.1/24 dev wg0 <- fine
[#] ip link set mtu 1420 up dev wg0 <- fine
[#] sysctl -q -w net.ipv4.ip_forward=1
sysctl: permission denied on key "net.ipv4.ip_forward"
[#] ip link delete dev wg0
**** Tunnel failed, will stop all others! ****
Not a missing capability: with hostNetwork the runtime mounts /proc/sys
READ-ONLY (confirmed in the pod's /proc/mounts), because those sysctls are the
HOST's globals. CAP_NET_ADMIN cannot lift a read-only mount, so the write can
never succeed here -- and wg-quick treats a failed PostUp as fatal, so it
deleted the interface it had just built.
The line was never needed. node_baseline already sets ip_forward=1, and reading
it from the pod returns 1. My header comment said exactly that ("so no pod
sysctl is needed") and then emitted the sysctl anyway, justified as "re-asserted
so this component does not silently depend on another role's sysctl file". That
defensive re-assertion is the entire outage. In a hostNetwork pod: VERIFY host
state, never write it.
Replaced with a read in the init container -- `grep -qx 1
/proc/sys/net/ipv4/ip_forward` -- failing closed with a message naming the
node_baseline role. Still fail-closed on purpose: a tunnel that comes up unable
to forward satisfies every probe here and silently drops all client traffic.
Also documents, in the manifest and the runbook, that the image's `chown:
Operation not permitted` run and its "we will not provide support for it"
warning are expected and harmless -- capabilities are dropped to NET_ADMIN so
its s6 init has no CAP_CHOWN, /config is a writable tmpfs, everything runs as
root, and wg-quick proceeds (it did; the failure was 30 lines later).
Three other unknowns closed by the same reconcile, now recorded in the README:
the kernel module reaches the pod (ip link add type wireguard works, no
CAP_SYS_MODULE needed); allowPrivilegeEscalation:false does NOT break the s6
init; and the image's iptables is nf_tables (xtables-nft-multi) matching Debian
13, which closes the worst documented fear -- rules landing in a table the
kernel never consults.
No sensitive paths; merges on a green gate with no override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85
|
be23cf3e26 | Changed secret | ||
|
jakman85
|
4791653f2b |
Merge pull request 'fix(vpn-ingress): keep the server key in source through SOPS, not as an out-of-band Secret' (#60) from fix/vpn-server-key-in-sops into main
Reviewed-on: #60 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8059297036 |
fix(devcontainer): bake wg + qrencode into the image instead of splitting the VPN procedure across host and container
Operator correction: "sop, wg, and any other tool requirement is supposed to be
baked into the devcontainer."
My previous commit discovered that `wg` existed only on the host, `sops` only in
the container, and `qrencode` on neither -- and fixed it the wrong way, by
splitting every step of the key procedure across the boundary and telling the
operator to `dnf install qrencode` on their workstation. That breaks CLAUDE.md §5
and the devcontainer README's "Don't install the toolchain on the host", produces
a procedure that cannot be reproduced in CI, and is miserable to follow.
wireguard-tools + qrencode are now distro packages in the Containerfile. Both are
in Fedora 44's repos (wireguard-tools 1.0.20260223 -- the same upstream version as
the image this component runs -- and qrencode 4.1.1), unpinned like every other
dnf package there: the pinned thing is the Fedora base, and neither is a gate tool
whose version changes what green means.
Three things got SIMPLER rather than merely relocated, which is the tell that the
rule was right:
* Step 2 is one pipeline in one shell -- `sops set ... "$(wg genkey)"` writes the
key straight into the already-encrypted document. No staged copy in .cache, no
`*.key`-suffix warning, no sha256 comparison to prove the right bytes landed.
* new-peer.sh derives the server public key from wg-server-key.sops.yaml itself
(`sops -d --extract | wg pubkey`), so there is no local key copy to keep in
step with the encrypted one, and no magic host path it must live at.
* `.cache/` is gitignored at the repo root (mirroring infra/ansible/.gitignore),
giving the container a persistent non-committable scratch dir. new-peer.sh
writes profiles to .cache/wg-peers/ -- necessary because the repo bind-mount is
the container's only persistent read-write mount, so a profile in /tmp there is
gone before it can reach a desktop. Its blanket "refuse to write inside the
checkout" guard is now a `git check-ignore` test of the real property.
Verified by execution, not reasoning: the image rebuilds; wg/qrencode/sops all
resolve in it; `sops -d | wg pubkey` runs in one shell; new-peer.sh refuses the
REPLACE-ME placeholder with an actionable message; and a full run with a throwaway
server key renders a correct profile at 0600 under a 0700 gitignored dir, prints
the QR, and refuses a non-ignored destination inside the repo.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
90a79d29d3 |
docs(node): name the exact Ansible command that applies the pin, not 'run the play'
Node configuration in this project is Ansible's — declared in the inventory, reviewed,
version-controlled — and for prod it is the ONLY surface: infra/PROD-PROVISIONING.md
records prod as 3x bare metal with the dev OpenTofu -> Proxmox -> cloud-init path
explicitly not applying, so there is no tofu/cloud-init half to keep in parity for an
/etc/hosts pin.
The pin was already in the right place; what the docs lacked was the command. All four
places now carry the scoped, idempotent one:
cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
-e k3s_hosts=k3s_prod --tags hosts
--tags hosts runs ONLY the /etc/hosts task, so applying this pin cannot drift any other
baseline setting.
Also verified the mechanism is not inert, which is the failure mode this repo keeps
meeting: roles/node_baseline/tasks/main.yml renders node_baseline_hosts_pins into a marked
blockinfile with backup: true, and state: absent when the list empties — so a removed
entry really does leave the node.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
523a463fd7 |
fix(node): pin gitea to the Gateway VIP — containerd was resolving it to the non-hairpinned WAN
The pm-agent image pull failed TLS verification, and the previous entry on this branch
recorded it honestly as not diagnosed. Diagnosed now, and it is not about the image.
containerd pulls as a NODE process, so it resolves through node_baseline_dns_servers (the
LAN gateway, deliberately not PiHole) and gets the public answer: a CNAME to
wan.home.keithjackson.dev -> 136.51.115.252, measured via both 1.1.1.1 and 8.8.8.8. This
router does not hairpin, so WAN:443 answers with its own placeholder certificate —
measured with the right SNI from the workstation:
subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
No extensions in certificate
No extensions means no subjectAltName, which is exactly 'certificate is not valid for any
names'. Meanwhile 192.168.2.20 IS the shared Gateway's VIP and its https listener serves
the real Let's Encrypt *.home.keithjackson.dev. The Gateway, the wildcard cert and the
digest were never implicated — only the node's answer.
This file already documents the identical failure for the Keycloak issuer, measured from
k3s-prod-2/-3 on 2026-10-03 (curl exit 60, ssl_verify_result=18), and already carries the
mechanism that fixes it. gitea is simply the second node process to need it; containerd is
a node process and nobody had added it. An operator must run the node_baseline play.
Also records, where an operator reads them, that the agent authenticates to Gitea with a
local account and a scoped PAT by decision — and why the federated alternative is not
merely unchosen but unavailable: Gitea implements only the Authorization Code Grant and
validates no third-party OIDC token, so a Keycloak service-account token cannot
authenticate to its API or to git-over-HTTP.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e35612dc7e |
fix(vpn-ingress): keep the server key in source through SOPS, not as an out-of-band Secret
Operator correction: secrets are kept in source through sops. #55 shipped the WireGuard server private key as a Secret to `kubectl create` by hand. Two real costs, not just a convention break: * R-SEC-01 / spec docs/07 -- secrets live in source, SOPS-encrypted, and reach the cluster only through Flux. * It broke DR for the one credential that cannot be re-derived. A rebuild restores everything in git; it does not restore a hand-made Secret, and losing this key means re-enrolling EVERY device, because each client profile pins its public half. #55 papered over that with "keep a copy in your password manager". (Note secret-provenance did NOT forbid the original shape -- it scopes to app-namespace=true and vpn-ingress is deliberately a platform namespace. The rule broken was the written requirement, not an enforced control.) An agent still must not mint a long-lived private key (CLAUDE.md §6), so the file is committed ALREADY ENCRYPTED with `privatekey: REPLACE-ME`. That makes the wiring complete and green from the first commit -- resource line plus both overlay decryption patches -- leaving the operator exactly one act: replace the value. A commented-out resource to uncomment later is the "in git, no generator entry, reviewed, merged, still broken" shape this repo keeps hitting. The init guard is now a validation rather than an emptiness check: the value is piped through `wg pubkey`, which accepts only a real curve25519 private key, and the pod names which case it hit -- placeholder, empty, mangled, or (the silent one) still `ENC[` ciphertext because an overlay is missing its decryption patch, which otherwise applies cleanly with every Flux condition green. Second, independent finding that also fixes #55's step 2: `wg` is on the HOST only and `sops` is in the CONTAINER only, so `sops -d | wg pubkey` cannot run in one shell -- the command #55 documented could not have worked. Every step is now split by where it runs, and the written key is verified by comparing sha256 digests rather than printing anything. `qrencode` is on neither, so the QR step says so and gives the install command. Touches clusters/** and adds a **/*.sops.yaml, so approval-policy will block pending the operator's /approve-override. Deliberately not posted by me. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a209a618fd |
fix(pm-agent): the agent is IN the cluster — one git host, the cluster-local service name
An in-cluster consumer reaches a service by its *.svc.cluster.local name, which is by design a different path from how a client outside the cluster reaches the same service. pm-worker's host allowlist accepted TWO values: the cluster-local name and the external hostname gitea.home.keithjackson.dev. The second is removed. It was never needed — nothing sets it — it would route in-cluster traffic out through the Gateway and back for no reason, and as an ACCEPTED value it widened the allowlist to a host reachable over the namespace's world:443 egress rule, which is precisely the injection shape the allowlist exists to refuse. probe-validation.sh now pins that: 20 refusals, including the external hostname, the cluster-local name over https, and the cluster-local name on the wrong port. Also corrects a remark in the handoff entry: I cited cluster-vars' GATEWAY_VIP (192.168.1.53) as suspicious next to the nodes' 192.168.2.x. That was noise — those base cluster-vars values are dev-era leftovers for a decommissioned cluster and say nothing about prod. None of the measured facts depended on it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6253d546f8 |
docs(pm-agent): the registry cannot be pulled from yet — name both prerequisites where the operator reads them
pm-agent reconciled exactly as designed after #57: Flux applied main@sha1:95597bd to platform-config, pm-agent and apps-wave-0 (all Ready), the ClusterPolicy pm-agent-job-shape is live with Enforce + failurePolicy Fail and all six rules, SOPS decryption works through the new overlay patch (checked by key names and value lengths, never values), and pm-agent-worker really is bound to nothing. Measuring it turned up two prerequisites for the NEXT operator step that nothing in this repository declares, and that #57's handoff could not have known — it listed the image pull as unverified, and that is exactly where it bit: - the nodes cannot verify the registry's TLS certificate (containerd: 'x509: certificate is not valid for any names'), while the same name serves a healthy Let's Encrypt wildcard from the LAN. A correct digest fails identically; - an unauthenticated client gets nothing from this Gitea's registry (its anonymous token carries no access scope) and no imagePullSecret exists anywhere in the tree. Not specific to pm-agent: conformance-runner shares the registry path and has never been pulled either. The dispatcher is the first workload in the cluster ever to reference it. Documentation only — no manifest, no policy, no code. Both fixes need an operator action and one of them is a posture decision, so the docs present both rather than pick. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
0569d47d49 |
Merge branch 'main' into feat/pm-agent-issue-to-pr
One conflict, in clusters/base/kustomization.yaml: PR #55 (vpn-ingress) and this branch both appended a resource line to the same tail. Resolved by keeping BOTH, with main's vpn-ingress first — it is a platform-config-tier leaf, while pm-agent dependsOn apps-wave-0, which is later. Nothing else conflicted. runbooks/kustomization.yaml auto-merged and carries both the vpn-remote-access and pm-agent entries; the handoff entries are separate files and had nothing to merge, which is the whole point of decisions/ being a directory. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
8b5e950baa |
test(pm-agent): a committed negative-path probe for the worker's input guards
pm-worker's command is pinned by admission with no args, so everything a caller could influence arrives in the environment — and the guards at the top of that script are what stand between an injected variable and a clone from the wrong host or a push to the wrong ref. Nothing exercised them, which makes them a claim rather than a control. probe-validation.sh asserts 17 refusals, each on BOTH a non-zero exit and the specific message, so a guard that stopped firing cannot pass by failing somewhere else. It covers a redirected git host, a real and reachable WRONG one (the egress policy permits tcp/443, so that case is not hypothetical), the right host with a smuggled path, 'main' and five other bad branch shapes, a malformed owner/repo, a bad mode, a bad issue number, and each missing credential. build-image.sh runs it inside the built image too. Writing it found three things: - the PM_BRANCH != main guard is unreachable while the pattern check above it holds, so the comment claiming it as the defence was wrong about which line does the work. Kept as an assertion independent of the regex; the comment now says so. - the EXIT trap ended in a failing test, which under set -e can replace the script's real exit code with that failure. - a provenance LOG LINE read a file through a redirect that set -o pipefail would have turned into a dead worker if the file were absent. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
f92707f38c |
feat(vpn-ingress): WireGuard remote access on udp/51820 (hostNetwork, declarative peers)
Lets the operator's Android / Linux / Windows / macOS devices dial in from anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus internet egress out the house WAN. Opposite direction to platform/vpn-egress, which sends pods OUT through PIA; the two share nothing but the word "VPN". WireGuard because it is the only option with a first-party client on all four named platforms, and because its handshake MAC is keyed with the responder's own public key: an unauthenticated packet gets NO reply, so udp/51820 is indistinguishable from a closed port. That, not "it is encrypted", is what makes an unfiltered WAN port defensible, and why there is no source allowlist. hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT clients to its own pod IP, so every remote client would carry that pod's Cilium identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway preserves the original source identity and enforces the destination's ingress policy. The obvious build would have shipped a healthy-looking tunnel on which every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve. In the host netns traffic is SNATed to the node and arrives as host/remote-node, exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the L2-announcement lease lottery, and double encapsulation (hence MTU 1420). Permitted by restrict-host-namespaces' own carve-out for platform namespaces. The cost is stated in networkpolicy.yaml rather than papered over: no NetworkPolicy constrains a hostNetwork pod. A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule; all three nodes answer the same keys, so failover is a one-line router change. Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my house" a reviewable PR diff. The only secret -- the server's private key -- is an out-of-band Secret: an agent must not mint a long-lived private key, and the two halves are joined only at pod start into a tmpfs that dies with the pod. Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain (kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md. Touches clusters/** , so ci/approval-policy.sh will block pending the operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
c5b5b8672a |
feat(pm-agent): an in-cluster AI Program Manager that turns Gitea issues into pull requests
A service that reads this repository's Gitea issues, decides what to do about each thread as the Program Manager persona, and runs one worker agent per assignment that authors a change, runs the merge gate and opens a pull request. It reacts to issue creation and to comments; the assignment and all correspondence travel through the issue thread and the PR. It authenticates with an Anthropic subscription token. It never pushes to main and never merges. Three choices carry the design, with alternatives recorded in docs/plans/: - it POLLS rather than receiving webhooks (Gitea's webhook SSRF guard, lost deliveries, and no inbound reach needed at all); - it keeps NO state: the trigger is "the newest thing in this thread was not written by me", so the issue thread is the state — no PVC, no backup obligation; - the model holds NO tools: its reply is one JSON object this code validates, so nothing in the path executes text. The boundary is platform/kyverno/admission/pm-agent-job-shape.yaml (Enforce, failurePolicy: Fail), not the RBAC: Kubernetes places no `use` check on serviceAccountName, so "may create Jobs" is "may run arbitrary code as the namespace's most privileged identity". Six rules admit only the committed shape, and the worker identity is bound to nothing with no token mounted, so there is nothing to escalate to. 28/28 offline Kyverno assertions. Nothing has run: the image has never been built or pushed, and both the image digest and the two credentials are deliberate placeholders. See the handoff entry's "What I did NOT verify". Refs CLAUDE.md §1/§3/§6, R-SEC-02, R-GITOPS-02. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
d0db63911a |
feat(cilium): roll the agents when cilium-config changes (merged config was silently inert)
A merged Cilium config change did nothing, and everything reported success. The Hubble flow exporter (#50, merged 2026-10-04) never took effect: HelmRelease cilium Ready=True, "Helm upgrade succeeded ... cilium.v6" cilium-config hubble-export-file-path: stdout + the allowlist, PRESENT cilium agent pods all three still PRE-MERGE, startTime 2026-10-04T07:11Z exported flow lines 0, 0, 0 Flux every Kustomization Ready=True Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent reads that ConfigMap only at STARTUP. Merged, believed delivered, green everywhere, doing nothing. The same class of failure this repo already guards against elsewhere: a green signal over something that never ran. Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply -- both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has rolled them since. The inertness is specific to ConfigMap-only changes. Set in all three value files to hold the [parity] contract. Cost, and it is the point: every future Cilium config change now rolls the agents instead of waiting for a manual restart. Strictly more honest -- it applies when merged or fails visibly -- but config edits become disruptive by default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge at a quiet moment. Merging this IS one rollout (the ConfigMap checksum annotation changes the pod template) -- which is what finally activates the Hubble exporter. UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable after the agents restart (check for flow lines immediately); and the rollout's actual impact on LAN DNS is unmeasured. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
81f627167d |
docs(obs): record that the gate-sidecar scrape works — four places still said it did not
PR #49 (merged
|
||
|
jakman85andClaude Opus 5
|
989649e64a |
feat(cilium): export Plex's pre-SNAT client address via Hubble, scoped to tcp/32400
The requirement is log visibility only -- which address connected to Plex.
Plex's own log cannot show it. Every LoadBalancer Service here is
externalTrafficPolicy: Cluster, so the node holding the VIP's L2 lease SNATs
the client before forwarding to the backend pod. Measured: Plex had logged
3,214 requests from 10.0.1.x (lease holder k3s-prod-3's own address, Cilium
identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No
X-Forwarded-For either -- it serves raw tcp/32400, not HTTP via the Gateway.
Nothing on the router does this; there is no hairpin NAT. It is Cilium.
The address is not lost, only unrecorded. Cilium observes the flow at ingress,
BEFORE its own translation, so Hubble already has it:
192.168.2.133 -> 10.0.0.135 real LAN client -> the Plex pod
10.0.1.78 -> 10.0.0.135 same traffic post-SNAT, what Plex sees
What was missing is RETENTION: 4095/4095 flows at ~303/s is about thirteen
seconds of history, metrics disabled, no exporter configured.
So enable the static exporter, port-scoped, to the agent's stdout -- Alloy
already tails every pod into Loki, so this needs no new mount, no rotation
handling and no Alloy change.
{namespace="kube-system", app="cilium"} |= "32400"
Scoped BY PORT, not namespace, on evidence: --to-namespace plex returned only
the post-SNAT flows; the record carrying the real client address did not match
it. A namespace filter would drop exactly what this exists to capture.
Rejected, with reasons: eTP:Local is incompatible with cilium 1.20's L2
announcer per upstream docs for this version ("may cause service IPs to be
announced on nodes without pods causing traffic drops") -- and the VIP is on
prod-3 while the pod is on prod-1, so it would black-hole 32400 immediately;
making it safe costs Plex its ability to reschedule. DSR: routing-mode is
vxlan, not native/geneve, and DSR returns bypass the ingress node, breaking
the router's stateful NAT on the remote path.
Also corrects the repo's own record: 2026-10-03 says the SNAT lands in
192.168.0.0/16 (the node's LAN address). It is 10.0.1.x, the cilium_host
inside the pod CIDR -- which is why LanNetworksBandwidth can never separate
local from remote while eTP is Cluster.
UNVERIFIED and stated in the handoff: nothing is reconciled; `filePath: stdout`
is taken from a verbatim upstream sentence but not confirmed against 1.20.1
source (a wrong notation is a silent no-op, with the file+Alloy fallback
documented); the port FlowFilter has no upstream example; and this change
ROLLS CILIUM -- prod-3 announces the Gateway, PiHole DNS and Plex VIPs, so a
lease move during the rollout could briefly interrupt LAN DNS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5.5
|
9a69f6906a |
fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root
hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing mounts cgroup2 there. The agent mounted one from inside its own private cgroup namespace, so socket-LB attached to the agent's own cgroup and host processes got no Service/LB translation. k3s-prod-1 (Gateway L2 lease holder + kube-vip VIP) therefore could not reach the Gateway VIP, and its apiserver's OIDC authenticator has failed since 2026-09-12. Co-Authored-By: Claude Opus 5.5 <[email protected]> |
||
|
jakman85andClaude Opus 5.5
|
a1e808e697 |
feat(dns): publish Gateway names to the WAN via in-cluster DDNS
Gateway-published records now CNAME to wan.${BASE_DOMAIN}, which a
favonia/cloudflare-ddns Deployment keeps at the home WAN's current public
IP. Operator decision: every Gateway-published name goes public; internal
services (controller: none) remain unpublished.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
||
|
jakman85
|
4f16e83eec |
Merge pull request 'feat(obs): make oauth2-proxy auth failures visible — the gates emit nothing' (#29) from feat/oauth2-proxy-auth-failure-visible into main
Reviewed-on: #29 Reviewed-by: Keith Jackson <[email protected]> |
||
|
jakman85andClaude Opus 5
|
4e1b169256 |
feat(gpu): Intel GPU device plugin v0.37.0, in its own Flux Kustomization
Makes the three prod nodes advertise `gpu.intel.com/i915`. This is the measured
fix for the Plex hardware-transcoding blocker, and it only makes the resource
EXIST — no consumer is switched over here.
WHY. The `/dev/dri` hostPath merged on main @
|
||
|
jakman85andClaude Opus 5
|
bce101d03f |
feat(obs): wire the gate-sidecar scrape, and fix the alert that would have gone silent doing it
PR #28 gave the five fleet oauth2-proxy sidecars --metrics-address=0.0.0.0:44180 and a named `metrics` port. This wires a scraper to them and, in the process, found a real bug in the alert this branch added. PodMonitor, not ServiceMonitor. Measured: no Service exposes 44180 on any of the five (jackett/radarr/sonarr/qbittorrent/pihole all publish only their app port + authproxy 4180), so a ServiceMonitor would select nothing. A PodMonitor targets pod ports directly and needs no change to catalog/render.py or the apps/wave-* renders and goldens. Selector is the pod label homelab.keithjackson.dev/auth-mode=forward-auth — present on exactly those five gate pods, absent from crafty's, which still has no metrics listener. The property, not an app list. THE BUG: `up{container="oauth2-proxy"}` -> `up{container="oauth2-proxy"} == 1`. A target that is discovered but FAILING still publishes `up`, with value 0. The series exists, so count by (namespace) returned 1, `unless` removed the namespace, and OAuth2ProxyGateUnobserved went QUIET while nothing was being scraped at all — this branch's own disease reproduced inside the alert written to cure it. Not hypothetical: the PodMonitor lands five targets whose scrape is dropped, so the bare form would have silently cleared on this very commit. Proven by mutation: reverting the `== 1` fails the two new unit-test scenarios (got: nil) and leaves every other assertion green. The scrape does NOT work yet, and that is documented rather than papered over. From a pod in ns monitoring to the jackett pod, :44180 AND :4180 both time out; the shared proxy's :44180 answers. ns jackett's default-deny is podSelector:{} with policyTypes:[Ingress,Egress] and NO ingress key (deny-all), and the one allowance is fromEntities:[ingress] to 4180 — the Gateway's Envoy only. The endpoint itself is healthy: port-forwarded, /metrics returns http=200, 10208 bytes, all three oauth2_proxy_* families including requests_total{code="302"}. Same bug and fingerprint as platform/longhorn/networkpolicy-metrics.yaml records for Longhorn; the fix is one narrow rule per app, and it cannot ship from this tier because oauth2-proxy and apps-wave-0 are Flux SIBLINGS (both dependsOn platform-config), so naming ns jackett here would apply today and fail on a cold rebuild. Needs an owner in the renderer. Touches conformance/** : one KNOWN_EXTERNAL_MONITORS entry, which is the mechanism the failing check explicitly instructs ("DECLARE it ... with who creates it") and which flux-podmonitor.yaml already uses. It records that the port name was measured on prod instead of resolved from a Service, and that the targets are currently down. That makes this PR require a non-author review on the sensitive-path rule. Also: ratio alert still NOT shipped. oauth2_proxy_requests_total now exists on the endpoint but not in the TSDB, so writing it today would make it inert for exactly the reason it was refused on 2026-09-30. Runbook gains policy-vs-no-target triage and the radarr /signalr/messages/negotiate worked case — one orphaned browser tab, 206 identical 302s, nobody ever re-prompted. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |