Dispatcher image and PM_WORKER_IMAGE both move to
sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2, the digest the
registry returns for agent-runner:sha-466f63ec3c3d and :main. config-generation 4 -> 5.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
`claude --print --output-format json` reports its own failures (API error, usage limit,
error_max_turns) as the result envelope on stdout and leaves stderr empty, so ask_pm()'s
stderr-only message logged "claude exited 1: " with nothing after it, twice on 2026-10-31.
describe_cli_failure() now logs the envelope's diagnostic fields plus stdout/stderr tails,
redacted (sk-ant-, Bearer/token, URL userinfo). PM_DECISION_MAX_TURNS (default 1, unchanged)
makes the max-turns remedy a ConfigMap change if the envelope turns out to say error_max_turns.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The run-383 worker image (df36d9b, pinned in #96) bakes imagePullSecrets: pm-agent-registry,
which admission requires, and the dispatcher's dry-run triage of #97 was read and approved.
PM_REQUIRE_LABEL=agent and the worker caps are unchanged. The dry-run status text in the
configmap comments, the README and the plan's follow-up 3 is updated to match.
Pre-flight (read-only, k3s-prod): pm-agent-job-shape Ready+Enforce; SA/Role/RoleBinding
present and can-i create jobs = yes; pm-agent-registry and pm-agent-credentials exist; both
egress CNPs Valid.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
c4eba5b (run 340) bakes a worker template without imagePullSecrets, so the new admission rule
refuses worker Jobs. df36d9b is agent-runner:main == :sha-ddd04d460ef6 (measured on the registry),
and its /etc/pm-agent/worker-job-template.yaml carries imagePullSecrets: pm-agent-registry (read
from the published layer).
Renovate found df36d9b but holds it under "Awaiting Schedule" on dashboard #91: the operator's
dispatch (run 385) was a real run at 12:14 New York, outside the 03:00-06:00 window. The
renovate-digest markers are unchanged, so later bumps remain Renovate's.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The Keycloak CR has no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the
server image; the three keycloak-k8s-resources release URLs in
platform/keycloak/kustomization.yaml are therefore THE Keycloak version, and Renovate
could not see them (no `# renovate:` marker). They now carry markers under one depName
(one branch, groupName keycloak-operator), with a packageRule: automerge false, labels
needs-review + blast-radius; majors still wait on the dashboard.
The untagged quay.io/keycloak/keycloak in catalog/keycloak.yaml renders
apps/wave-1/keycloak.yaml, which no kustomization lists. It is pinned to the operator's
26.0.6 as documentation, re-rendered, and Renovate is disabled for that image so it
cannot raise PRs against a file nothing deploys. The render is kept (not deleted): it is
the fixture for apps/wave-1/tests/app-slis.test.yaml's keycloak SLI rule tests.
README: an "Upgrading Keycloak" section — an operator bump must be checked against
passkey-reconciler.yaml's 26.0.6 Admin API assumptions.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Rebased #61 onto the CI image pipeline (#81). This commit makes the remaining in-git
blockers to going live go away:
- deployment.yaml and PM_WORKER_IMAGE pin images run 340's digest (sha256:c4eba5b3...,
tag sha-523cd9118f95 = main), read from the registry. Renovate markers unchanged.
- PM_DRY_RUN=true and PM_REQUIRE_LABEL=agent for the first run. Dry run is required, not
just cautious: the run-340 image's baked worker template predates imagePullSecrets, so
admission would deny its Jobs until CI re-publishes after this merge and Renovate bumps.
- Token scopes: Gitea 1.24.3 scopes GET /api/v1/user (the dispatcher's first call) under
read:user. The documented list was missing it. Now read:user, write:repository,
write:issue, read:package everywhere.
- Org-package read documented as a team (Packages=Read, no repos), not just a scope.
- sops is toolbox-only; the .examples say so and give the in-place edit.
- runbook: key-name check no longer emits Secret values before the pipe.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
The registry had two blockers. #59 fixed the first (nodes resolved the hostname to the
non-hairpinned WAN and failed TLS against the router's CN=%D cert). This is the second:
the pull is unauthenticated, and this registry refuses unauthenticated clients outright —
measured 2026-10-05, its anonymous token carries access: null and every path 401s.
pm-agent-registry is a dockerconfigjson Secret (SOPS, placeholders) referenced by
imagePullSecrets on the dispatcher Deployment and on the committed worker Job template. It
holds the SAME Gitea token as pm-agent-credentials, because the two consumers need two
formats: the application reads a raw token from env, the kubelet reads a dockerconfigjson
before any container starts. Same one-value-in-two-files pattern as the Keycloak client
secret, documented in both .example files. The operator's part is the value plus one new
scope: read:package.
ADMISSION GAINED A SEVENTH RULE. imagePullSecrets is a pod-spec field no container
references, which is exactly why it is easy to leave unread — the same shape of omission
as initContainers, ephemeralContainers, env and envFrom before it. The kubelet reads the
named Secret, in this namespace, before anything runs. Rule 1's pattern now REQUIRES the
committed entry; rule 7 forbids any other and any second one. Both halves are needed: a
foreach over an absent list yields no result.
The leverage is small and the policy says so: the image is digest-pinned, so a different
pull credential cannot change WHAT runs. Pinned anyway — 'only the committed shape' is the
premise, and the rule costs four lines.
kyverno test: 33 passed, 0 failed (was 28). Three new fixtures, each a real path: the
committed entry plus a second one, a different Secret, and none at all.
The gate also caught me shipping the Secret in plaintext — I never ran sops -e -i on the
placeholder. Recorded in the handoff entry, because the file looked finished.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Images run 337's push failed with "403 Forbidden: Access denied". Measured
read-only: Gitea never logged the request; an unauthenticated GET /v2/ via the
Gateway gets 401 from Gitea when sent from act-runner (egress unrestricted) but
403 from buildkitd; Hubble logs it as an L7 http-request DROPPED with
traffic_direction EGRESS. Cilium >= 1.16.8 (CVE-2025-30162) re-checks the
source pod's egress against the proxied backend identity, and buildkitd's
egress only allowed world and the ingress entity. Same fix as 68690e5 for
Keycloak. Gitea's ingress already admits reserved:ingress and is unchanged.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
R-TLS-01/docs/08 govern externally reachable TLS and ADR-0006 puts workload mTLS out of scope, but
tls-lifecycle-automated required ACME for every Certificate, which platform/buildkit's internal CA
cannot satisfy. Clause (5b) accepts the tls-scope=workload-mtls annotation only when the
certificate proves it: no Gateway serves it, internal names only, namespaced selfSigned/ca issuer
backed by an exempt CA certificate. Also fixes yamllint comma spacing in the buildkitd mounts.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator decision on PR #81: no CI job may reach a privileged daemon. The act runner is untouched.
- platform/buildkit (+ clusters/base/buildkit.yaml): moby/buildkit v0.34.1-rootless pinned by
digest, uid 1000, not privileged; caps drop ALL + SETUID/SETGID, seccomp/AppArmor Unconfined,
read-only rootfs (each measured under podman). cert-manager CA + server/client certs, ClusterIP
:1234, default-deny with ingress only from the act-runner pod (DinD masquerades jobs to it).
No Kyverno exception: every applied policy matches app-namespace=true only.
- Each Containerfile names its shipped stage and ends in a `verify` stage holding every former
`docker run` assertion (uid 65532, RUN --network=none, writable-path inventory standing in for
the read-only rootfs). The old `&& echo` checks that could never fail are real assertions now.
- images.sh drives buildkitd with a pinned buildx (sha256-checked) via bake; PRs build verify
targets with agent-runner FROM the in-run toolbox (target: context); main verifies, then pushes.
Fails loudly when the builder secrets are absent or buildkitd is unreachable.
- infra/gitea/buildkit-client-secret.sh copies the client cert into Actions secrets.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a
workstation build/push. Nothing had ever been pushed to the Gitea registry.
- .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main
builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as
sha-<commit> + main, reading the digest back from the registry. Only changed images build.
- Each image's own build-image.sh does the build and in-image verification, now engine-neutral
(DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh.
- Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex
manager for them. The worker Job template is deliberately untracked (it would loop).
- renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the
current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION.
- Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in
Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Also states in bind_browser_flow()'s docstring that it is reached only when git's
bind-browser-flow is "true".
Co-Authored-By: Claude Opus 5.5 <[email protected]>
KeycloakPasskeyReconcilerFailing (warning, for 15m) on
kube_job_status_failed{namespace="keycloak", job_name=~"keycloak-passkey-reconciler-.*"},
following the sibling client-secret reconciler's rule. promtool unit tests pin it in
both directions: quiet when succeeding, firing after 15m on a failed Job, quiet on the
SIBLING's failure, and quiet on a single transient failure. Runbook added and projected
by the runbooks ConfigMap generator.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Review findings on the passkey reconciler, fixed:
1. HIGH - the bind was automatic and rollback did not stick. A git-declared switch,
`bind-browser-flow` in passkey-desired-state.yaml, now decides whether pass 3 may
PUT browserFlow. Default "false": build + verify the candidate (or reuse a verified
one), log WOULD BIND <alias>, never touch the binding - including after an earlier
bind, so an operator's rollback sticks, and a hand-edited bound candidate is left
alone rather than shadowed by an -r2 build. "true" binds as before. The renderer
carries the one hand-edited line through and --check refuses anything but the
quoted strings "true"/"false". --no-bind can only narrow it.
2. HIGH - the protected-subtree guard read `alias` on sub-flow rows. Keycloak 26.0.6
AuthenticationManagementResource.recurseExecutions sets displayName =
flowRef.getAlias() and flowId on a sub-flow row and never sets alias (alias is the
CONFIG alias on authenticator rows), so the protected set was only the top-level
alias. Sub-flows are now named by displayName (getFlows() lists top-level flows
only, so flowId cannot be resolved through it); a sub-flow row without one leaves
the subtree undetermined and the flow pass fail-closed. The selftest stub now
returns 26.0.6-shaped rows, and a real run is asserted to refuse writes into
browser-mfa-forms and browser-mfa-conditional-otp.
3. MEDIUM - PUT /flows/{alias}/executions was screened on the path alias only, but
updateExecutions resolves the execution realm-wide by body id. body.id is now
screened against the protected ids AND required to be in the tree this run is
building (recorded from the build's own readback); the config POST is held to the
same build scope; and the body may carry only id/requirement/priority (the same
handler renames any flow named by flowId).
4. Stale "audit-only" docs rewritten (platform-config kustomization, chainsaw test,
desired-state header and its renderer template, README) to describe what ships.
5. backoffLimit 0 (a retried deterministic build failure burns the candidate-alias
budget); the stale "Renovate does not manage image: tags" comment corrected and the
image pinned by index digest.
6. The ~160-line header duplicated in the YAML and the script is now kept once, in the
script, with a short pointer above it.
Selftest: 259 assertions (was 216). mutations.py: 40 injected regressions (was 30),
including one per finding above, all caught.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
PR #72 declared the WebAuthn passwordless policy and a restructured browser-mfa
flow in the realm import. That declaration could not take effect: a
KeycloakRealmImport is a ONE-SHOT, and prod's reads Done=True / Started=False, so
every realm-level field and flow declared after the first import is inert -- in
git, applied by Flux, reported Ready, reaching the realm never. Merging it changed
nothing about how anyone logs in, and nothing anywhere said so. Same failure class
as the 2026-09-06 client-secret incident, third field over.
A SIBLING reconciler converges all three declared things every 15 minutes. It is a
sibling, not a second pass of client-secret-reconciler.yaml, so that reconciler's
allow-list -- and its guarantee that prod's real user and the media-users /
platform-admins / viewers groups are structurally unreachable -- stays literally
true. Its selftest still passes unchanged.
pass 1 webAuthnPolicyPasswordless* -- strict read-modify-write of
GET/PUT /admin/realms/homelab, readback-verified.
pass 2 webauthn-register-passwordless -- strict read-modify-write of ONE
literally-named alias; enabled, defaultAction false so enrolment stays
opt-in.
pass 3 the browser flow -- BLUE/GREEN.
PASS 3 NEVER MUTATES THE FLOW USERS ARE LOGGING IN THROUGH. Converging a
five-flow, ten-execution tree needs create-subflow, create-execution,
set-requirement and bind-config as SEPARATE untransacted calls; doing that to the
BOUND flow means a run dying between them leaves a half-built login path for every
SSO app and kubectl, and this repo has had a total SSO outage from a realm change.
So instead:
1. BUILD under a NEW, unused alias (declared alias + a hash of the desired
shape, applied to EVERY alias in the tree -- reusing browser-mfa-forms would
mean writing INTO the bound tree). A failure leaves an orphaned, unbound flow
and zero user-visible effect.
2. VERIFY by readback -- levels, ordering, authenticators, requirements, and the
conditional-OTP config CONTENTS -- before anything is bound, so the
regression that once 403'd every platform-admin is caught BEFORE it bites.
3. BIND with ONE atomic PUT setting browserFlow.
4. ROLLBACK is one write; no DELETE exists anywhere in the allow-list, so the old
flow survives as the rollback target.
The widened allow-list is made safe structurally. It cannot say "but not the bound
flow" -- its write entries are alias-parameterised -- so a second guard screens
every write against the live browserFlow's WHOLE subtree (sub-flow aliases and
execution ids) before a socket opens, FAIL-CLOSED until that binding has been
read. The create-flow endpoint carries no alias in its path, so it is screened on
its request BODY; the selftest caught that it otherwise escaped the guard.
Idempotence is load-bearing: a rebuild-and-reflip every 15 minutes would churn the
login path on a schedule. The bound flow is compared STRUCTURALLY and a match
produces zero writes and no new flow. Sub-flow and config ALIASES are deliberately
not compared -- a built flow carries a hash suffix, so comparing labels would make
it permanently unrecognisable as converged.
Single source of truth: the desired state is GENERATED from the realm import by
ci/render-keycloak-passkey-policy.py and gated with --check, so the RP ID -- baked
into every credential at enrolment and never re-keyable -- is declared once. The
pod keeps automountServiceAccountToken: false and reads a mounted ConfigMap, so no
rbac.yaml is added anywhere.
Read from Keycloak 26.0.6 source before being relied on: updateRealm
(DefaultExportImportManager 725-894) makes no getUsers/getClients/getGroups/
getRoles/getRequiredActions/getAuthenticationFlows call; it DOES read
getBrowserFlow(), which is what makes the atomic bind possible and why the policy
write pins it; the passwordless policy is applied all-or-nothing only when
RpEntityName is non-empty, with defaults substituted for omitted keys; and
updateExecutions compares rep.getPriority(), so a body omitting `priority` would
silently reshuffle the whole execution order.
Gate coverage: a new whole-repo L0 check extracts the script and runs its
--selftest (216 assertions, including a build killed at four different points each
asserting the old flow is still bound and the bound subtree was never written),
plus mutations.py (30 injected regressions, all caught) proving the selftest can
fail. An earlier iteration of this work only AUDITED the flow and left the
restructure manual; decisions/2026-10-18-keycloak-passkey-reconciler.md records
that design and why it was reconsidered.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
MEASURED on prod 2026-10-18: kube-system/coredns had replicas: 1. In-cluster
DNS was a single pod on a single node, so draining or rebooting that node was a
CLUSTER-WIDE in-cluster DNS outage until the pod rescheduled -- every Service
lookup, every CNPG replica finding its primary, every operator addressing the
API by name. It was invisible because 1/1 Ready looks healthy.
It is urgent rather than latent now: the scheduled reboot automation on
feat/automated-upgrades drains and reboots one node at a time, unattended,
overnight -- turning a single point of failure into a recurring one.
WHY AN HPA AND NOT replicas: 2 -- the whole decision. k3s OWNS the coredns
Deployment: AddOn/coredns, source /data/k3s/server/manifests/coredns.yaml,
managedFields managers deploy@k3s-prod-1 + k3s, objectset.rio.cattle.io/hash
label. k3s re-stages its bundled manifests on startup, so replicas: 2 written
there -- or a kubectl scale -- is reverted on the next restart or upgrade.
Rejected alternatives:
* coredns.yaml.skip + our own templated copy (the roles/kube_vip pattern).
Deterministic, but takes ownership of coredns from k3s permanently so k3s
upgrades stop updating it. Too high a price for a replica count.
* A Flux patch of the Deployment. Rejected outright: Flux and k3s's deploy
controller would both own one object and fight -- the default-deny
NetworkPolicy drift-loop shape (~2500 events/10min) this cluster already hit.
An HPA is a separate object k3s does not own, driving /scale rather than the
Deployment spec, so there is no shared-ownership fight; if k3s re-stages
replicas: 1 it is re-asserted within a reconcile. Disclosed cost: a brief
window at 1 replica after a k3s restart.
It is a FLOOR-HOLDER, not an autoscaler -- the CPU metric exists only so the
autoscaling/v2 object is valid. Measured: 5m/4m against a 100m request (~5%),
so the 80% target is never approached. scaleDown stabilization 600s because DNS
is the worst thing to flap; scaleUp immediate, since that restores redundancy.
The kustomization edit is ADDITIVE: platform/coredns/ already existed and holds
the coredns-custom split-horizon ConfigMap (R-DNS-01/03), which is untouched.
UNAPPLIED. The HPA has never run, and the claim that k3s reverts replicas on
restart is reasoned, not observed. After merge check `describe hpa coredns` for
ScalingActive=True -- an inert HPA looks identical to a working one because the
hand scale already left 2 replicas. See the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two halves of one repeatable upgrade process. Neither works alone.
(a) OS upgrades and reboots
platform/kured/ — Kubernetes Reboot Daemon, chart 6.1.0/app 1.23.0, in its own
Flux Kustomization (dependsOn platform, wait:true). NOT in platform/kustomization.yaml:
that tier gates platform-config and every apps wave, so a wedged reboot daemon there
could stall every application. A component added to make maintenance safer must not be
able to cause an outage.
infra/ansible/roles/unattended_upgrades/ — new role (node_baseline is touched by other
open branches, and this is a different lifecycle), enabled for prod in group_vars.
The human gate is the SENTINEL, which is how this honours node_baseline's "IT DOES NOT
REBOOT" (k3s-prod-2 hung on a warm reboot 2026-09-03) and cluster-upgrade-plan.md's
rejection of automating away the gate between nodes: u-u is security-only with the
kernel blacklisted, so it essentially never writes /var/run/reboot-required. That file
appears when an operator deliberately runs node-baseline.yml --tags kernel. kured then
does only the mechanical half, including the ordering rule node_baseline has to shout.
Quorum: concurrent reboots are structurally impossible, not merely unlikely.
concurrency:1 plus kured's single lock annotation (API-server optimistic concurrency),
and lockTtl:0 — an expiring lock is exactly the 3-node etcd catastrophe, so a hung node
holds it forever and nothing cascades.
Longhorn: measured 25 volumes x 3 replicas, hard anti-affinity, 600s replenishment wait,
serialized rebuild — so every drain degrades ALL volumes to 2/3 and the inter-node gate
must be volume robustness, not pod readiness. Doubled: lockReleaseDelay 30m
(unconditional) plus kured's Prometheus alert gate (semantic).
alertFilterMatchOnly:true is load-bearing — the flag inverts the regexp's meaning, and
under the chart default kured would have been blocked on day one by the five alerts
measured firing with no end in sight, and never rebooted anything.
Kernel floor (kernel >= 6.13, EFI memattr): excluded TWICE. The running kernel comes
from o=Debian Backports, which is not an allowed origin, so u-u cannot see a kernel
upgrade; and ^linux- is blacklisted anyway so widening origins later cannot re-arm it.
Asserted against apt-config dump, not against the file just written. 6.12 kernels are
still installed and bootable on all three nodes, and nothing was held.
Closes a silent no-op: update-notifier-common was absent on all three nodes, so NOTHING
could create the sentinel. kured alone would have idled forever at 1/1 Running.
platform/kured/alerts.yaml — three rules over kured's only metric, with a companion
promtool suite pinning each both ways, plus runbooks/node-reboot-automation.md.
Window is DAILY 03:00-06:00 because a weekly one forces for:8d, longer than this
cluster's 7d Prometheus retention.
(b) Image/dependency automation
Renovate was aspirational: renovate.json present and complete, with no workflow, no
in-cluster workload, and its docker+kubernetes managers disabled — so every plain
image: tag was maintained by nothing while seven files claimed otherwise.
.gitea/workflows/renovate.yaml — scheduled daily, dry_run default true for a safe
first run, and a preflight that FAILS (not skips) without RENOVATE_TOKEN, because a
skipped Gitea job reports success and that is how this stayed inert.
Renovate now owns both halves; Flux image-automation is formally closed: it pushes
commits rather than opening PRs (bypassing CLAUDE.md §1), and it cannot work here
anyway — main is protected user_can_push:false and the flux-system GitRepository has
no secretRef. Its controllers are installed and have idled 43 days with zero CRs.
conformance/: +34/-0, one new SCRAPE_OWNERS entry claiming kured_. Weakens nothing;
tightens the check for this component (requires service.create+metrics.create, both
chart-default false, plus a companion NetworkPolicy requirement). Required because
assertion (3) fails an alert over an unclaimed metric and KNOWN_UNWIRED_METRICS is
shrink-only.
Sensitive paths touched: clusters/**, conformance/**, .gitea/** — needs a non-author
approving review; /approve-override deliberately NOT posted.
NOT verified: no OS upgrade run, no node rebooted, no lock observed, the Ansible role
never executed, and whether CNPG switches its primary on a drain-induced eviction (the
most likely way the first real reboot fails). Full list in the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Adds the realm's WebAuthn Passwordless policy and restructures the browser-mfa
flow so a passkey is an ALTERNATIVE beside the password form. Additive and
optional by design: this IdP gates every SSO app plus kubectl, so a mandatory
passkey flow that misfires is a total lock-out. Enrolment is not forced.
Policy: RP ID is ${BASE_DOMAIN} (not the keycloak host) because a passkey's RP ID
is fixed at enrolment and must not be bound to a name that can move; resident key
required (a non-discoverable credential is a second factor, not a passkey); user
verification required (without it, possession alone is a complete login).
Flow: auth-username-password-form is split into auth-username-form +
auth-password-form, because the server must know who is logging in before it can
decide whether a passkey exists. Conditional OTP for platform-admins is relocated
intact onto the password branch only — a UV-required passkey is already two
phishing-resistant factors.
No requiredActions block is declared, deliberately: Keycloak 26.0.6 treats
requiredActions as REPLACE not merge, so declaring one would delete the default
set including CONFIGURE_TOTP and lock every platform-admin out of the OTP step
this flow still enforces. webauthn-register-passwordless is already registered
enabled-but-not-default by Keycloak's own defaults.
UNAPPLIED: the realm import is a one-shot resting at Done=True/Started=False, so
this reaches the live realm never. See the handoff entry and the README section.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Exposed by the 2026-10-17 whole-cluster shutdown. oauth2-proxy performs OIDC
discovery ONCE at startup and exits 1 if it fails. On a cold boot Keycloak is
not serving yet, the shared Gateway answers 503, and the proxy dies:
[provider.go:55] Performing OIDC Discovery...
[main.go:73] ERROR: Failed to initialise OAuth2 Proxy: ... failed to discover
OIDC configuration: unexpected status "503": no healthy upstream
It then sits in exponential CrashLoopBackOff long after Keycloak is healthy --
one replica was still down minutes later and only a manual pod delete cleared
it. There is no retry flag, a readinessProbe cannot help (the container is
already dead), and CrashLoopBackOff backoff is not tunable per-pod.
Add an extraInitContainers gate (wait-for-idp) that polls the discovery doc
until it answers. A pod in Init accrues no CrashLoopBackOff backoff and retries
promptly. BOUNDED and FAILS OPEN: after ~5m it starts the proxy anyway, so the
worst case is exactly today's behaviour and never something new.
extraInitContainers is the chart's OWN hook for this -- its commented example is
literally `- name: wait-for-idp` -- and the key was CONFIRMED against the pinned
chart (helm show values, 10.7.1) rather than assumed, because a wrong Helm values
key is silently ignored. No new base image: the chart already runs an alpine init
container for its built-in waitForRedis gate; pinned to 3.22 (verified upstream)
rather than the chart's `latest`.
Rejected: Flux postRenderers (no precedent in this tree, and the chart has a
hook); --skip-oidc-discovery (removes the fetch entirely, but needs four
hardcoded endpoints and I could not verify lazy-JWKS offline -- this component
gates SSO for seven apps, so the additive option wins).
Shell vars are written BARE with no braced form: platform declares
postBuild.substituteFrom, and envsubst would blank a braced shell variable and
silently break the loop. Same convention as the gitea/nextcloud OIDC hooks.
UNAPPLIED; the init container has never run, and the race was not re-created to
prove the fix. Check the init container's log on first apply. See the handoff.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Both ZFS pools moved off the Proxmox host onto a standalone TrueNAS on
192.168.2.30 -- prod's OWN /24, so the pfSense crossing that made this knob
load-bearing is gone, along with the DNAT at 192.168.2.102 it mounted through.
Measured 2026-10-07: 192.168.2.102:2049 no longer answers; .30:2049 and :111 do.
BULK_NFS_SERVER -> 192.168.2.30 in prod, base and dev. The dev value is marked
inert (dev is decommissioned); it carries the real address only so nothing in
the tree renders a host that no longer exists.
THE OPERATIONAL POINT, and why this is not just a value change: a bound
PersistentVolume's NFS source is immutable. Server dry-run, 2026-10-07:
spec.persistentvolumesource: Forbidden: spec.persistentvolumesource is
immutable after creation
so Flux's apply FAILS rather than updating, and the pods keep their old, dead
mount. The five PVs must be deleted and recreated -- safe, because Retain never
touches data and render.py emits a pre-set claimRef so each PV rebinds its own
PVC. The sequence is docs/BULK-NFS-STORAGE.md section 10 and is an OPERATOR
action; this commit does not perform it.
Pairs with the hazard already on the page: spec.nfs.server is unvalidated at
create (any string binds) and immutable at update. A wrong value is accepted
silently and then cannot be corrected in place.
docs/BULK-NFS-STORAGE.md: sections 1 and 5 rewritten for TrueNAS -- the measured
exports, the service settings that decide whether uid 1000 survives the wire,
the three stale ZFS sharenfs properties that had to be cleared, and the ACL
hazard (acltype=nfsv4 + aclinherit=passthrough means one click in the ACL editor
propagates to the whole future library). Section 7 rewritten now that the
placeholder is gone, section 9 retargeted from dev to prod, section 10 new.
Sections 2/3/4/6 are unchanged by the move.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The operator enrolled a phone, merged the peer, and `wg show` listed none.
Measured on prod 2026-10-06 with the peer already on main:
live wg-peers ConfigMap, active AllowedIPs .. 0
DaemonSet volume reference ................. configMap.name: wg-peers (FIXED)
DaemonSet generation/observedGeneration .... 1 / 1 (never rolled)
init container, all three pods ............. "0 peer(s)"
Part of it was just the 10m reconcile interval. The defect is the fixed name:
Flux updates a fixed-name ConfigMap IN PLACE, so the pod template is
byte-identical, so no rollout is triggered -- and the init container assembles
wg0.conf ONCE at pod start into a tmpfs, so the running tunnel keeps the old
peer list indefinitely. Adding or revoking a peer had no effect until something
unrelated restarted the pods. For a REVOCATION that is a security defect, not
an inconvenience: delete a stanza, see a green PR, device keeps working.
The runbook and README both claimed "PR -> merge -> Flux rolls the DaemonSet".
Never true. And this repo already documents the fix, in a file I had read --
runbooks/kustomization.yaml: "The generated name carries kustomize's content
hash, so editing any runbook rolls the server automatically and a stale
projection is impossible."
So: peers.yaml (inline ConfigMap) -> peers.conf (plain file) + a
configMapGenerator. The operator's enrolled peer is carried across verbatim.
Every line is WireGuard config and `#` is a legal comment there, so the
documentation now lives inside the file that is actually mounted.
`namespace: vpn-ingress` on the kustomization is REQUIRED, not tidiness: a
generated ConfigMap inherits no namespace, everything else here declares
vpn-ingress, and kustomize's nameReference will not rewrite across a namespace
boundary -- without it the build mounts the unhashed name and the pods wedge on
a missing volume. Caught only by building the output and reading the volumes.
new-peer.sh: PEERS_FILE repointed, and a missing register is now FATAL rather
than skipped -- the old `[ -r "$PEERS_FILE" ] && grep ...` guard meant that
after the rename the only symptom would have been duplicate detection quietly
never firing again. Stale peers.yaml references swept from six files including
the script's own user-facing output.
Verified: the DaemonSet now mounts wg-peers-m9kcckcb5t; the hash tracks content
(edit -> dgg7ht9c74 -> restore -> m9kcckcb5t); the enrolled peer is byte-identical
to main's; and `new-peer.sh dupe 2` is now correctly refused by the operator's
real allocation.
No sensitive paths; merges on a green gate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The first real enrolment failed:
$ ./infra/vpn-ingress/new-peer.sh keith-phone 2
new-peer.sh: 10.90.0.2/32 is already allocated ... Pick another octet.
Nothing was allocated -- peers.yaml had zero peers, as the pod's init
container reported (`0 peer(s)`). The duplicate-address check grepped the
whole file, and peers.yaml line 66 is a COMMENTED-OUT example stanza I wrote
containing literally `# AllowedIPs = 10.90.0.2/32`. So .2 -- the first
address anyone would ask for, and the one my own runbook uses as its example
-- was permanently un-allocatable, with a message asserting the opposite of
the truth.
Fixed on both sides, because either alone leaves the trap half-set:
* the check strips comment lines before matching, which is the correct
predicate rather than a patch: a real stanza's AllowedIPs line is never
commented, while the template and every peer's `# <name> — <owner>` label
always are.
* the example now uses placeholders that cannot match anything real
(`10.90.0.<octet>/32`, `<device-name>`), so a future grep cannot be fooled
by it either.
The bug class is worth naming: a documentation example that is byte-identical
to real data will eventually be parsed as real data. This is its third
instance in this component -- flux-postbuild-substitution flagged a `${...}`
written inside a comment explaining that hazard, and no-plaintext-secrets
flagged the .sops.yaml.example because a kustomization named it in a comment.
Make examples structurally unmatchable, not merely commented out.
Verified by running it: .2 now renders a profile (via the real
`sops -d | wg pubkey` path, which this also proves against the operator's
actual key for the first time), and an uncommented duplicate stanza is still
correctly refused.
No sensitive paths; merges on a green gate.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The tunnel never came up on its first reconcile with a real key. One line of my
own manifest was the whole cause. Measured on prod 2026-10-06:
[#] ip link add dev wg0 type wireguard <- fine
[#] wg addconf wg0 /dev/fd/63 <- fine
[#] ip -4 address add 10.90.0.1/24 dev wg0 <- fine
[#] ip link set mtu 1420 up dev wg0 <- fine
[#] sysctl -q -w net.ipv4.ip_forward=1
sysctl: permission denied on key "net.ipv4.ip_forward"
[#] ip link delete dev wg0
**** Tunnel failed, will stop all others! ****
Not a missing capability: with hostNetwork the runtime mounts /proc/sys
READ-ONLY (confirmed in the pod's /proc/mounts), because those sysctls are the
HOST's globals. CAP_NET_ADMIN cannot lift a read-only mount, so the write can
never succeed here -- and wg-quick treats a failed PostUp as fatal, so it
deleted the interface it had just built.
The line was never needed. node_baseline already sets ip_forward=1, and reading
it from the pod returns 1. My header comment said exactly that ("so no pod
sysctl is needed") and then emitted the sysctl anyway, justified as "re-asserted
so this component does not silently depend on another role's sysctl file". That
defensive re-assertion is the entire outage. In a hostNetwork pod: VERIFY host
state, never write it.
Replaced with a read in the init container -- `grep -qx 1
/proc/sys/net/ipv4/ip_forward` -- failing closed with a message naming the
node_baseline role. Still fail-closed on purpose: a tunnel that comes up unable
to forward satisfies every probe here and silently drops all client traffic.
Also documents, in the manifest and the runbook, that the image's `chown:
Operation not permitted` run and its "we will not provide support for it"
warning are expected and harmless -- capabilities are dropped to NET_ADMIN so
its s6 init has no CAP_CHOWN, /config is a writable tmpfs, everything runs as
root, and wg-quick proceeds (it did; the failure was 30 lines later).
Three other unknowns closed by the same reconcile, now recorded in the README:
the kernel module reaches the pod (ip link add type wireguard works, no
CAP_SYS_MODULE needed); allowPrivilegeEscalation:false does NOT break the s6
init; and the image's iptables is nf_tables (xtables-nft-multi) matching Debian
13, which closes the worst documented fear -- rules landing in a table the
kernel never consults.
No sensitive paths; merges on a green gate with no override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator correction: "sop, wg, and any other tool requirement is supposed to be
baked into the devcontainer."
My previous commit discovered that `wg` existed only on the host, `sops` only in
the container, and `qrencode` on neither -- and fixed it the wrong way, by
splitting every step of the key procedure across the boundary and telling the
operator to `dnf install qrencode` on their workstation. That breaks CLAUDE.md §5
and the devcontainer README's "Don't install the toolchain on the host", produces
a procedure that cannot be reproduced in CI, and is miserable to follow.
wireguard-tools + qrencode are now distro packages in the Containerfile. Both are
in Fedora 44's repos (wireguard-tools 1.0.20260223 -- the same upstream version as
the image this component runs -- and qrencode 4.1.1), unpinned like every other
dnf package there: the pinned thing is the Fedora base, and neither is a gate tool
whose version changes what green means.
Three things got SIMPLER rather than merely relocated, which is the tell that the
rule was right:
* Step 2 is one pipeline in one shell -- `sops set ... "$(wg genkey)"` writes the
key straight into the already-encrypted document. No staged copy in .cache, no
`*.key`-suffix warning, no sha256 comparison to prove the right bytes landed.
* new-peer.sh derives the server public key from wg-server-key.sops.yaml itself
(`sops -d --extract | wg pubkey`), so there is no local key copy to keep in
step with the encrypted one, and no magic host path it must live at.
* `.cache/` is gitignored at the repo root (mirroring infra/ansible/.gitignore),
giving the container a persistent non-committable scratch dir. new-peer.sh
writes profiles to .cache/wg-peers/ -- necessary because the repo bind-mount is
the container's only persistent read-write mount, so a profile in /tmp there is
gone before it can reach a desktop. Its blanket "refuse to write inside the
checkout" guard is now a `git check-ignore` test of the real property.
Verified by execution, not reasoning: the image rebuilds; wg/qrencode/sops all
resolve in it; `sops -d | wg pubkey` runs in one shell; new-peer.sh refuses the
REPLACE-ME placeholder with an actionable message; and a full run with a throwaway
server key renders a correct profile at 0600 under a 0700 gitignored dir, prints
the QR, and refuses a non-ignored destination inside the repo.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Node configuration in this project is Ansible's — declared in the inventory, reviewed,
version-controlled — and for prod it is the ONLY surface: infra/PROD-PROVISIONING.md
records prod as 3x bare metal with the dev OpenTofu -> Proxmox -> cloud-init path
explicitly not applying, so there is no tofu/cloud-init half to keep in parity for an
/etc/hosts pin.
The pin was already in the right place; what the docs lacked was the command. All four
places now carry the scoped, idempotent one:
cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
-e k3s_hosts=k3s_prod --tags hosts
--tags hosts runs ONLY the /etc/hosts task, so applying this pin cannot drift any other
baseline setting.
Also verified the mechanism is not inert, which is the failure mode this repo keeps
meeting: roles/node_baseline/tasks/main.yml renders node_baseline_hosts_pins into a marked
blockinfile with backup: true, and state: absent when the list empties — so a removed
entry really does leave the node.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The pm-agent image pull failed TLS verification, and the previous entry on this branch
recorded it honestly as not diagnosed. Diagnosed now, and it is not about the image.
containerd pulls as a NODE process, so it resolves through node_baseline_dns_servers (the
LAN gateway, deliberately not PiHole) and gets the public answer: a CNAME to
wan.home.keithjackson.dev -> 136.51.115.252, measured via both 1.1.1.1 and 8.8.8.8. This
router does not hairpin, so WAN:443 answers with its own placeholder certificate —
measured with the right SNI from the workstation:
subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
No extensions in certificate
No extensions means no subjectAltName, which is exactly 'certificate is not valid for any
names'. Meanwhile 192.168.2.20 IS the shared Gateway's VIP and its https listener serves
the real Let's Encrypt *.home.keithjackson.dev. The Gateway, the wildcard cert and the
digest were never implicated — only the node's answer.
This file already documents the identical failure for the Keycloak issuer, measured from
k3s-prod-2/-3 on 2026-10-03 (curl exit 60, ssl_verify_result=18), and already carries the
mechanism that fixes it. gitea is simply the second node process to need it; containerd is
a node process and nobody had added it. An operator must run the node_baseline play.
Also records, where an operator reads them, that the agent authenticates to Gitea with a
local account and a scoped PAT by decision — and why the federated alternative is not
merely unchosen but unavailable: Gitea implements only the Authorization Code Grant and
validates no third-party OIDC token, so a Keycloak service-account token cannot
authenticate to its API or to git-over-HTTP.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Operator correction: secrets are kept in source through sops. #55 shipped the
WireGuard server private key as a Secret to `kubectl create` by hand. Two real
costs, not just a convention break:
* R-SEC-01 / spec docs/07 -- secrets live in source, SOPS-encrypted, and
reach the cluster only through Flux.
* It broke DR for the one credential that cannot be re-derived. A rebuild
restores everything in git; it does not restore a hand-made Secret, and
losing this key means re-enrolling EVERY device, because each client
profile pins its public half. #55 papered over that with "keep a copy in
your password manager".
(Note secret-provenance did NOT forbid the original shape -- it scopes to
app-namespace=true and vpn-ingress is deliberately a platform namespace. The
rule broken was the written requirement, not an enforced control.)
An agent still must not mint a long-lived private key (CLAUDE.md §6), so the
file is committed ALREADY ENCRYPTED with `privatekey: REPLACE-ME`. That makes
the wiring complete and green from the first commit -- resource line plus both
overlay decryption patches -- leaving the operator exactly one act: replace the
value. A commented-out resource to uncomment later is the "in git, no generator
entry, reviewed, merged, still broken" shape this repo keeps hitting.
The init guard is now a validation rather than an emptiness check: the value is
piped through `wg pubkey`, which accepts only a real curve25519 private key, and
the pod names which case it hit -- placeholder, empty, mangled, or (the silent
one) still `ENC[` ciphertext because an overlay is missing its decryption patch,
which otherwise applies cleanly with every Flux condition green.
Second, independent finding that also fixes#55's step 2: `wg` is on the HOST
only and `sops` is in the CONTAINER only, so `sops -d | wg pubkey` cannot run in
one shell -- the command #55 documented could not have worked. Every step is now
split by where it runs, and the written key is verified by comparing sha256
digests rather than printing anything. `qrencode` is on neither, so the QR step
says so and gives the install command.
Touches clusters/** and adds a **/*.sops.yaml, so approval-policy will block
pending the operator's /approve-override. Deliberately not posted by me.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
An in-cluster consumer reaches a service by its *.svc.cluster.local name, which is by
design a different path from how a client outside the cluster reaches the same service.
pm-worker's host allowlist accepted TWO values: the cluster-local name and the external
hostname gitea.home.keithjackson.dev. The second is removed.
It was never needed — nothing sets it — it would route in-cluster traffic out through the
Gateway and back for no reason, and as an ACCEPTED value it widened the allowlist to a
host reachable over the namespace's world:443 egress rule, which is precisely the
injection shape the allowlist exists to refuse.
probe-validation.sh now pins that: 20 refusals, including the external hostname, the
cluster-local name over https, and the cluster-local name on the wrong port.
Also corrects a remark in the handoff entry: I cited cluster-vars' GATEWAY_VIP
(192.168.1.53) as suspicious next to the nodes' 192.168.2.x. That was noise — those base
cluster-vars values are dev-era leftovers for a decommissioned cluster and say nothing
about prod. None of the measured facts depended on it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
pm-agent reconciled exactly as designed after #57: Flux applied main@sha1:95597bd to
platform-config, pm-agent and apps-wave-0 (all Ready), the ClusterPolicy
pm-agent-job-shape is live with Enforce + failurePolicy Fail and all six rules, SOPS
decryption works through the new overlay patch (checked by key names and value lengths,
never values), and pm-agent-worker really is bound to nothing.
Measuring it turned up two prerequisites for the NEXT operator step that nothing in this
repository declares, and that #57's handoff could not have known — it listed the image
pull as unverified, and that is exactly where it bit:
- the nodes cannot verify the registry's TLS certificate (containerd: 'x509: certificate
is not valid for any names'), while the same name serves a healthy Let's Encrypt
wildcard from the LAN. A correct digest fails identically;
- an unauthenticated client gets nothing from this Gitea's registry (its anonymous token
carries no access scope) and no imagePullSecret exists anywhere in the tree.
Not specific to pm-agent: conformance-runner shares the registry path and has never been
pulled either. The dispatcher is the first workload in the cluster ever to reference it.
Documentation only — no manifest, no policy, no code. Both fixes need an operator action
and one of them is a posture decision, so the docs present both rather than pick.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
One conflict, in clusters/base/kustomization.yaml: PR #55 (vpn-ingress) and this branch
both appended a resource line to the same tail. Resolved by keeping BOTH, with main's
vpn-ingress first — it is a platform-config-tier leaf, while pm-agent dependsOn
apps-wave-0, which is later.
Nothing else conflicted. runbooks/kustomization.yaml auto-merged and carries both the
vpn-remote-access and pm-agent entries; the handoff entries are separate files and had
nothing to merge, which is the whole point of decisions/ being a directory.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
pm-worker's command is pinned by admission with no args, so everything a caller could
influence arrives in the environment — and the guards at the top of that script are what
stand between an injected variable and a clone from the wrong host or a push to the
wrong ref. Nothing exercised them, which makes them a claim rather than a control.
probe-validation.sh asserts 17 refusals, each on BOTH a non-zero exit and the specific
message, so a guard that stopped firing cannot pass by failing somewhere else. It covers
a redirected git host, a real and reachable WRONG one (the egress policy permits tcp/443,
so that case is not hypothetical), the right host with a smuggled path, 'main' and five
other bad branch shapes, a malformed owner/repo, a bad mode, a bad issue number, and each
missing credential. build-image.sh runs it inside the built image too.
Writing it found three things:
- the PM_BRANCH != main guard is unreachable while the pattern check above it holds, so
the comment claiming it as the defence was wrong about which line does the work. Kept
as an assertion independent of the regex; the comment now says so.
- the EXIT trap ended in a failing test, which under set -e can replace the script's
real exit code with that failure.
- a provenance LOG LINE read a file through a redirect that set -o pipefail would have
turned into a dead worker if the file were absent.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Lets the operator's Android / Linux / Windows / macOS devices dial in from
anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus
internet egress out the house WAN. Opposite direction to platform/vpn-egress,
which sends pods OUT through PIA; the two share nothing but the word "VPN".
WireGuard because it is the only option with a first-party client on all four
named platforms, and because its handshake MAC is keyed with the responder's
own public key: an unauthenticated packet gets NO reply, so udp/51820 is
indistinguishable from a closed port. That, not "it is encrypted", is what
makes an unfiltered WAN port defensible, and why there is no source allowlist.
hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT
clients to its own pod IP, so every remote client would carry that pod's Cilium
identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from
Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway
preserves the original source identity and enforces the destination's ingress
policy. The obvious build would have shipped a healthy-looking tunnel on which
every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve.
In the host netns traffic is SNATed to the node and arrives as host/remote-node,
exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the
L2-announcement lease lottery, and double encapsulation (hence MTU 1420).
Permitted by restrict-host-namespaces' own carve-out for platform namespaces.
The cost is stated in networkpolicy.yaml rather than papered over: no
NetworkPolicy constrains a hostNetwork pod.
A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule;
all three nodes answer the same keys, so failover is a one-line router change.
Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my
house" a reviewable PR diff. The only secret -- the server's private key -- is
an out-of-band Secret: an agent must not mint a long-lived private key, and the
two halves are joined only at pod start into a tmpfs that dies with the pod.
Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain
(kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md.
Touches clusters/** , so ci/approval-policy.sh will block pending the
operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A service that reads this repository's Gitea issues, decides what to do about each
thread as the Program Manager persona, and runs one worker agent per assignment that
authors a change, runs the merge gate and opens a pull request. It reacts to issue
creation and to comments; the assignment and all correspondence travel through the
issue thread and the PR. It authenticates with an Anthropic subscription token. It
never pushes to main and never merges.
Three choices carry the design, with alternatives recorded in docs/plans/:
- it POLLS rather than receiving webhooks (Gitea's webhook SSRF guard, lost
deliveries, and no inbound reach needed at all);
- it keeps NO state: the trigger is "the newest thing in this thread was not
written by me", so the issue thread is the state — no PVC, no backup obligation;
- the model holds NO tools: its reply is one JSON object this code validates, so
nothing in the path executes text.
The boundary is platform/kyverno/admission/pm-agent-job-shape.yaml (Enforce,
failurePolicy: Fail), not the RBAC: Kubernetes places no `use` check on
serviceAccountName, so "may create Jobs" is "may run arbitrary code as the
namespace's most privileged identity". Six rules admit only the committed shape, and
the worker identity is bound to nothing with no token mounted, so there is nothing to
escalate to. 28/28 offline Kyverno assertions.
Nothing has run: the image has never been built or pushed, and both the image digest
and the two credentials are deliberate placeholders. See the handoff entry's
"What I did NOT verify".
Refs CLAUDE.md §1/§3/§6, R-SEC-02, R-GITOPS-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A merged Cilium config change did nothing, and everything reported success.
The Hubble flow exporter (#50, merged 2026-10-04) never took effect:
HelmRelease cilium Ready=True, "Helm upgrade succeeded ... cilium.v6"
cilium-config hubble-export-file-path: stdout + the allowlist, PRESENT
cilium agent pods all three still PRE-MERGE, startTime 2026-10-04T07:11Z
exported flow lines 0, 0, 0
Flux every Kustomization Ready=True
Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the
ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent
reads that ConfigMap only at STARTUP. Merged, believed delivered, green
everywhere, doing nothing. The same class of failure this repo already guards
against elsewhere: a green signal over something that never ran.
Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply --
both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed
the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why
the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has
rolled them since. The inertness is specific to ConfigMap-only changes.
Set in all three value files to hold the [parity] contract.
Cost, and it is the point: every future Cilium config change now rolls the
agents instead of waiting for a manual restart. Strictly more honest -- it
applies when merged or fails visibly -- but config edits become disruptive by
default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding
node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge
at a quiet moment.
Merging this IS one rollout (the ConfigMap checksum annotation changes the pod
template) -- which is what finally activates the Hubble exporter.
UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable
after the agents restart (check for flow lines immediately); and the rollout's
actual impact on LAN DNS is unmeasured.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod
2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s
against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for
these namespaces for the first time (5 metric names, 148 series).
Four places in the tree still asserted the pre-fix state as current, in the present tense,
in the files a reader consults first:
platform/oauth2-proxy/podmonitor.yaml "READ THIS BEFORE BELIEVING THE TARGETS ARE UP:
THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ
up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to
disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the
diagnosis kept below it under an explicit HISTORY heading because it is why the fix is
shaped the way it is.
catalog/render.py:159 "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED".
catalog/render.py:1365 the same claim on the sidecar arg.
platform/oauth2-proxy/README.md its subsections are dated and so are honest records, but
a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits
under the heading and an Update 2026-10-05 section carries the before/after.
The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT
deliberately, because that part is still true and "flag without ingress allowance" is
exactly what caused the 63.7h outage.
Two substantive errors fixed while there, not just tense. The README's "three things are
needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports
directly, which is why no Service exposes 44180 and the renderer needed no Service change;
following it would have sent someone to edit Service rendering for nothing. And
podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" —
no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section
explaining the one namespace this file cannot help.
Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no
golden changes, and therefore no sensitive path and no human approval needed — the opposite
of the PR this documents.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
renovate-bot
jakman85andClaude Opus 5.5