Node configuration in this project is Ansible's — declared in the inventory, reviewed,
version-controlled — and for prod it is the ONLY surface: infra/PROD-PROVISIONING.md
records prod as 3x bare metal with the dev OpenTofu -> Proxmox -> cloud-init path
explicitly not applying, so there is no tofu/cloud-init half to keep in parity for an
/etc/hosts pin.
The pin was already in the right place; what the docs lacked was the command. All four
places now carry the scoped, idempotent one:
cd infra/ansible && ansible-playbook playbooks/node-baseline.yml \
-e k3s_hosts=k3s_prod --tags hosts
--tags hosts runs ONLY the /etc/hosts task, so applying this pin cannot drift any other
baseline setting.
Also verified the mechanism is not inert, which is the failure mode this repo keeps
meeting: roles/node_baseline/tasks/main.yml renders node_baseline_hosts_pins into a marked
blockinfile with backup: true, and state: absent when the list empties — so a removed
entry really does leave the node.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The pm-agent image pull failed TLS verification, and the previous entry on this branch
recorded it honestly as not diagnosed. Diagnosed now, and it is not about the image.
containerd pulls as a NODE process, so it resolves through node_baseline_dns_servers (the
LAN gateway, deliberately not PiHole) and gets the public answer: a CNAME to
wan.home.keithjackson.dev -> 136.51.115.252, measured via both 1.1.1.1 and 8.8.8.8. This
router does not hairpin, so WAN:443 answers with its own placeholder certificate —
measured with the right SNI from the workstation:
subject=C=ZZ, ST=Somewhere, L=Unknown, O=%D342e7e6b, CN=%D
No extensions in certificate
No extensions means no subjectAltName, which is exactly 'certificate is not valid for any
names'. Meanwhile 192.168.2.20 IS the shared Gateway's VIP and its https listener serves
the real Let's Encrypt *.home.keithjackson.dev. The Gateway, the wildcard cert and the
digest were never implicated — only the node's answer.
This file already documents the identical failure for the Keycloak issuer, measured from
k3s-prod-2/-3 on 2026-10-03 (curl exit 60, ssl_verify_result=18), and already carries the
mechanism that fixes it. gitea is simply the second node process to need it; containerd is
a node process and nobody had added it. An operator must run the node_baseline play.
Also records, where an operator reads them, that the agent authenticates to Gitea with a
local account and a scoped PAT by decision — and why the federated alternative is not
merely unchosen but unavailable: Gitea implements only the Authorization Code Grant and
validates no third-party OIDC token, so a Keycloak service-account token cannot
authenticate to its API or to git-over-HTTP.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
An in-cluster consumer reaches a service by its *.svc.cluster.local name, which is by
design a different path from how a client outside the cluster reaches the same service.
pm-worker's host allowlist accepted TWO values: the cluster-local name and the external
hostname gitea.home.keithjackson.dev. The second is removed.
It was never needed — nothing sets it — it would route in-cluster traffic out through the
Gateway and back for no reason, and as an ACCEPTED value it widened the allowlist to a
host reachable over the namespace's world:443 egress rule, which is precisely the
injection shape the allowlist exists to refuse.
probe-validation.sh now pins that: 20 refusals, including the external hostname, the
cluster-local name over https, and the cluster-local name on the wrong port.
Also corrects a remark in the handoff entry: I cited cluster-vars' GATEWAY_VIP
(192.168.1.53) as suspicious next to the nodes' 192.168.2.x. That was noise — those base
cluster-vars values are dev-era leftovers for a decommissioned cluster and say nothing
about prod. None of the measured facts depended on it.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
pm-agent reconciled exactly as designed after #57: Flux applied main@sha1:95597bd to
platform-config, pm-agent and apps-wave-0 (all Ready), the ClusterPolicy
pm-agent-job-shape is live with Enforce + failurePolicy Fail and all six rules, SOPS
decryption works through the new overlay patch (checked by key names and value lengths,
never values), and pm-agent-worker really is bound to nothing.
Measuring it turned up two prerequisites for the NEXT operator step that nothing in this
repository declares, and that #57's handoff could not have known — it listed the image
pull as unverified, and that is exactly where it bit:
- the nodes cannot verify the registry's TLS certificate (containerd: 'x509: certificate
is not valid for any names'), while the same name serves a healthy Let's Encrypt
wildcard from the LAN. A correct digest fails identically;
- an unauthenticated client gets nothing from this Gitea's registry (its anonymous token
carries no access scope) and no imagePullSecret exists anywhere in the tree.
Not specific to pm-agent: conformance-runner shares the registry path and has never been
pulled either. The dispatcher is the first workload in the cluster ever to reference it.
Documentation only — no manifest, no policy, no code. Both fixes need an operator action
and one of them is a posture decision, so the docs present both rather than pick.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Both reported by the operator on first use of the runbook merged in #55.
Step 1 said `cd infra/ansible && ./site.sh --tags system --limit k3s_prod`.
There is no site.sh anywhere in this repo. It was wrong four ways at once:
this repo drives Ansible with `ansible-playbook <playbook>` and never a
wrapper (playbooks/site.yml is the full bring-up chain, not this); the module
work is in playbooks/node-baseline.yml; targeting here is `-e k3s_hosts=...`,
not `--limit`; and Ansible exists only in the pinned toolbox, with ansible.cfg
read only from infra/ansible. Replaced with the real command plus why
`-e k3s_hosts=k3s_prod` matters (the default group is decommissioned dev) and
why `--tags system` must never be dropped (an untagged run also executes the
`disks` tasks, which assert on prod's deliberate placeholder devices).
Step 2 told the operator to stage the server key at /work/.cache/server.key
"inside the repo's gitignored cache". .cache/ is NOT gitignored -- the file was
ignored only incidentally by `*.key` (.gitignore:11). So the path was safe for
a reason I had not identified, and renaming it (server.pem, wg-server) would
have left a committable WireGuard private key in the checkout, which
no-plaintext-secrets does not recognise. The step now names the real rule,
marks the suffix as the load-bearing part, and gives a check-ignore
verification to run before trusting it.
Docs only -- no manifest and no behaviour change. Touches no sensitive path.
Also records the post-merge measurements for #55, whose own entry predates the
merge: Flux applied ad3f56e by the real path, all 18 Kustomizations Ready,
hostNetwork confirmed (pod IPs are the node IPs), and the designed fail-closed
state (Init:0/1 on the missing wg-server-key Secret).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
I recorded that merging main in would dismiss the operator's approval and that
approval-policy would therefore go red until they re-approved. Measured: wrong.
dismiss_stale_approvals is true, so the review did go stale — but
ci/approval-policy.sh selects on .state == "APPROVED" and a non-author login and reads
nothing about staleness, and required_approvals is 0 so Gitea never required one either.
CI run #257 on the merged head: "approved by 'jakman85' (author 'gitea-admin') —
sensitive change is human-reviewed", approval-policy pass, gitea-settings pass, required
check green, PR mergeable.
The 'push again and you must override again' rule applies to the /approve-override
directive, which IS staleness-checked against the head commit — not to a review.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PR #56 added §0 (work in your own worktree, never the shared checkout) while this branch
was open. A worker agent reads CLAUDE.md first, and §0's instruction is to create a git
worktree outside the shared checkout — which does not exist in a worker's container. It
holds a fresh clone of its own, made for one assignment and thrown away with the pod,
which is what §0 is actually asking for.
A worker that obeyed §0 literally would 'git worktree add', edit somewhere the wrapper
that started it does not look, and have its work committed by nobody while the issue got
a 'changed nothing' report. worker-rules.md now says §0 is satisfied and not to do that.
Also records in the handoff entry: the merge and its one conflict, and that this push
dismisses the operator's approving review — measured, dismiss_stale_approvals=true — so
the approval-policy stage needs them to re-approve. That is theirs to do, not an agent's.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
One conflict, in clusters/base/kustomization.yaml: PR #55 (vpn-ingress) and this branch
both appended a resource line to the same tail. Resolved by keeping BOTH, with main's
vpn-ingress first — it is a platform-config-tier leaf, while pm-agent dependsOn
apps-wave-0, which is later.
Nothing else conflicted. runbooks/kustomization.yaml auto-merged and carries both the
vpn-remote-access and pm-agent entries; the handoff entries are separate files and had
nothing to merge, which is the whole point of decisions/ being a directory.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The PR gate ran and BLOCKED on approval-policy alone — the sensitive-path rule, with no
non-author review — while every technical stage passed, gitea-settings included (the one
stage a workstation cannot run). That is the gate working, and it is the one thing an
agent must not clear for itself.
Also recorded: the commit-status API reported 'pending — Waiting to run' for a run that
had already completed, so the log is the source of truth here. §2b again.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
pm-worker's command is pinned by admission with no args, so everything a caller could
influence arrives in the environment — and the guards at the top of that script are what
stand between an injected variable and a clone from the wrong host or a push to the
wrong ref. Nothing exercised them, which makes them a claim rather than a control.
probe-validation.sh asserts 17 refusals, each on BOTH a non-zero exit and the specific
message, so a guard that stopped firing cannot pass by failing somewhere else. It covers
a redirected git host, a real and reachable WRONG one (the egress policy permits tcp/443,
so that case is not hypothetical), the right host with a smuggled path, 'main' and five
other bad branch shapes, a malformed owner/repo, a bad mode, a bad issue number, and each
missing credential. build-image.sh runs it inside the built image too.
Writing it found three things:
- the PM_BRANCH != main guard is unreachable while the pattern check above it holds, so
the comment claiming it as the defence was wrong about which line does the work. Kept
as an assertion independent of the regex; the comment now says so.
- the EXIT trap ended in a failing test, which under set -e can replace the script's
real exit code with that failure.
- a provenance LOG LINE read a file through a redirect that set -o pipefail would have
turned into a dead worker if the file were absent.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Agents have been working in the operator's checkout and clobbering each other.
Nothing in the tree told them not to: worktrees appeared in CLAUDE.md only as
§3's gate workarounds, and decisions/README.md described worktree isolation as
something agents already do rather than as a rule.
New CLAUDE.md §0 states the rule, with the commands and an explicit do-not list
for the shared checkout. Placed before §1 so every existing §-reference in the
repo keeps its meaning.
Measured in the shared checkout on 2026-10-05, each row a distinct failure mode:
107 `checkout: moving` reflog entries; sitting on feat/pm-agent-issue-to-pr, not
main; two other runs' untracked trees left in it; local main two commits behind
origin/main; 42 registered worktrees of which 31 are on already-merged branches;
9 gate scripts carrying a `.claude/` exclusion because an in-tree worktree made
find-based discovery count the same manifest up to 18 times.
Corrects a claim carried in earlier briefs: `git worktree remove` is NOT blocked
by the agent sandbox. Provoked on a throwaway worktree — rc 0.
§3 reframed as the normal path and grown from two breakages to three:
spec-immutability's skip is avoidable, not merely tolerated, with the spec repo
mounted read-only in the same ENGINE shim plus -e SPEC_REPO=/home-network-k3s.
CLAUDE.md is a sensitive path, so this PR needs a non-author review or an
administrator /approve-override posted as a PR review. Not posted by me: §1
forbids an agent clearing that on its own PR.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
last_activity() is the whole trigger: the PM acts iff the newest thing in a thread was
not written by it. It found that newest comment by comparing Gitea's RFC 3339
created_at STRINGS, which is only correct while every offset is identical — true today,
silently wrong after a server timezone change, and the failure would be the PM answering
its own last comment forever.
The comments endpoint returns them in creation order, so the last element IS the newest.
Using that removes the dependency entirely. Found by self-review, before anything ran.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Lets the operator's Android / Linux / Windows / macOS devices dial in from
anywhere and behave like LAN machines: full reach to 192.168.2.0/24 plus
internet egress out the house WAN. Opposite direction to platform/vpn-egress,
which sends pods OUT through PIA; the two share nothing but the word "VPN".
WireGuard because it is the only option with a first-party client on all four
named platforms, and because its handshake MAC is keyed with the responder's
own public key: an unauthenticated packet gets NO reply, so udp/51820 is
indistinguishable from a closed port. That, not "it is encrypted", is what
makes an unfiltered WAN port defensible, and why there is no source allowlist.
hostNetwork: true is REQUIRED, not convenient. A pod-networked server must SNAT
clients to its own pod IP, so every remote client would carry that pod's Cilium
identity -- and a pod-sourced request to the shared Gateway VIP returns 403 from
Envoy for ALL 16 hostnames (measured 2026-10-04), because Cilium's Gateway
preserves the original source identity and enforces the destination's ingress
policy. The obvious build would have shipped a healthy-looking tunnel on which
every cluster web UI was 403 and *.home.keithjackson.dev did not even resolve.
In the host netns traffic is SNATed to the node and arrives as host/remote-node,
exactly like a LAN client. It also sidesteps the unsafe ip_forward sysctl, the
L2-announcement lease lottery, and double encapsulation (hence MTU 1420).
Permitted by restrict-host-namespaces' own carve-out for platform namespaces.
The cost is stated in networkpolicy.yaml rather than papered over: no
NetworkPolicy constrains a hostNetwork pod.
A DaemonSet so the hand-made WAN forward cannot be invalidated by a reschedule;
all three nodes answer the same keys, so failover is a one-line router change.
Peer PUBLIC keys are plaintext in peers.yaml, which makes "who can get into my
house" a reviewable PR diff. The only secret -- the server's private key -- is
an out-of-band Secret: an agent must not mint a long-lived private key, and the
two halves are joined only at pod start into a tmpfs that dies with the pod.
Ships INERT-BUT-HEALTHY with zero peers. THREE off-repo manual steps remain
(kernel module, server key, WAN forward) -- runbooks/vpn-remote-access.md.
Touches clusters/** , so ci/approval-policy.sh will block pending the
operator's /approve-override. Deliberately not posted by me (CLAUDE.md §1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A service that reads this repository's Gitea issues, decides what to do about each
thread as the Program Manager persona, and runs one worker agent per assignment that
authors a change, runs the merge gate and opens a pull request. It reacts to issue
creation and to comments; the assignment and all correspondence travel through the
issue thread and the PR. It authenticates with an Anthropic subscription token. It
never pushes to main and never merges.
Three choices carry the design, with alternatives recorded in docs/plans/:
- it POLLS rather than receiving webhooks (Gitea's webhook SSRF guard, lost
deliveries, and no inbound reach needed at all);
- it keeps NO state: the trigger is "the newest thing in this thread was not
written by me", so the issue thread is the state — no PVC, no backup obligation;
- the model holds NO tools: its reply is one JSON object this code validates, so
nothing in the path executes text.
The boundary is platform/kyverno/admission/pm-agent-job-shape.yaml (Enforce,
failurePolicy: Fail), not the RBAC: Kubernetes places no `use` check on
serviceAccountName, so "may create Jobs" is "may run arbitrary code as the
namespace's most privileged identity". Six rules admit only the committed shape, and
the worker identity is bound to nothing with no token mounted, so there is nothing to
escalate to. 28/28 offline Kyverno assertions.
Nothing has run: the image has never been built or pushed, and both the image digest
and the two credentials are deliberate placeholders. See the handoff entry's
"What I did NOT verify".
Refs CLAUDE.md §1/§3/§6, R-SEC-02, R-GITOPS-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
A merged Cilium config change did nothing, and everything reported success.
The Hubble flow exporter (#50, merged 2026-10-04) never took effect:
HelmRelease cilium Ready=True, "Helm upgrade succeeded ... cilium.v6"
cilium-config hubble-export-file-path: stdout + the allowlist, PRESENT
cilium agent pods all three still PRE-MERGE, startTime 2026-10-04T07:11Z
exported flow lines 0, 0, 0
Flux every Kustomization Ready=True
Cause: the chart's rollOutCiliumPods defaults to false, so Helm rewrites the
ConfigMap and leaves the DaemonSet's pod template untouched -- and cilium-agent
reads that ConfigMap only at STARTUP. Merged, believed delivered, green
everywhere, doing nothing. The same class of failure this repo already guards
against elsewhere: a green signal over something that never ran.
Confirmed by contrast rather than reasoned: the cgroup fix (#42) DID apply --
both the DaemonSet and the running pod mount /sys/fs/cgroup -- because it changed
the POD TEMPLATE, which rolls the DaemonSet on its own. That also explains why
the agents are dated 2026-10-04T07:11Z: that was #42's rollout, and nothing has
rolled them since. The inertness is specific to ConfigMap-only changes.
Set in all three value files to hold the [parity] contract.
Cost, and it is the point: every future Cilium config change now rolls the
agents instead of waiting for a manual restart. Strictly more honest -- it
applies when merged or fails visibly -- but config edits become disruptive by
default. maxUnavailable: 1 bounds it to one node at a time. The lease-holding
node briefly loses the Gateway, PiHole-DNS and Plex VIPs as it cycles, so merge
at a quiet moment.
Merging this IS one rollout (the ConfigMap checksum annotation changes the pod
template) -- which is what finally activates the Hubble exporter.
UNVERIFIED: unreconciled; `filePath: stdout` still unproven and only answerable
after the agents restart (check for flow lines immediately); and the rollout's
actual impact on LAN DNS is unmeasured.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
PR #49 (merged 21c88b0) fixed the blocked scrape and it is verified working on prod
2026-10-05: all 6 targets up == 1, health=up, lastError='', lastScrapeDuration 0.001-0.006s
against 10.0s and "context deadline exceeded" before. oauth2_proxy_* is in the TSDB for
these namespaces for the first time (5 metric names, 148 series).
Four places in the tree still asserted the pre-fix state as current, in the present tense,
in the files a reader consults first:
platform/oauth2-proxy/podmonitor.yaml "READ THIS BEFORE BELIEVING THE TARGETS ARE UP:
THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY" and "SO THE TARGETS BELOW WILL READ
up == 0 UNTIL AN INGRESS ALLOWANCE LANDS". The first actively instructs the reader to
disbelieve a correct up == 1. Now a STATUS block with the post-fix numbers, with the
diagnosis kept below it under an explicit HISTORY heading because it is why the fix is
shaped the way it is.
catalog/render.py:159 "THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED".
catalog/render.py:1365 the same claim on the sidecar arg.
platform/oauth2-proxy/README.md its subsections are dated and so are honest records, but
a reader hits the stale body long before any correction. A SUPERSEDED pointer now sits
under the heading and an Update 2026-10-05 section carries the before/after.
The "not scraped" claims are corrected; the "a flag alone is not enough" warning is KEPT
deliberately, because that part is still true and "flag without ingress allowance" is
exactly what caused the 63.7h outage.
Two substantive errors fixed while there, not just tense. The README's "three things are
needed" listed a Service port and a ServiceMonitor — wrong, a PodMonitor targets pod ports
directly, which is why no Service exposes 44180 and the renderer needed no Service change;
following it would have sent someone to edit Service rendering for nothing. And
podmonitor.yaml deferred the fix to "the catalog/renderer, in flight on another branch" —
no such branch existed; it now names r_monitoring_ingress_policy, with a new CRAFTY section
explaining the one namespace this file cannot help.
Comments only in render.py, so the rendered output is byte-identical: no apps/ file and no
golden changes, and therefore no sensitive path and no human approval needed — the opposite
of the PR this documents.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
I took Plex down for the whole LAN. ADVERTISE_IP was
http://plex-vip.${BASE_DOMAIN}:32400 -- a name that exists ONLY in PiHole's
split-horizon answers and in no public zone.
A Plex client resolves the advertised host ITSELF, with whatever resolver it
uses, and on a real LAN that is frequently not PiHole: browsers ship DoH on by
default, phones have "private DNS", guest/IoT devices take the router's DNS.
Those clients could not resolve the name, the router answered the failed lookup
with ITSELF, and they all landed on its web UI -- hence the %D certificate.
Measured 2026-10-05:
192.168.2.1:443 -> C=ZZ, O=%D342e7e6b, CN=%D (router, LAN side)
136.51.115.252:443 -> the same %D cert (router, WAN side)
pihole-0 AND pihole-1 -> plex -> .20, plex-vip -> .24 (both CORRECT)
gateway .20:443 -> CN=*.home.keithjackson.dev, valid
http://192.168.2.24:32400/identity -> 200
Plex's published connections: all port 32400, nothing on 443
The DNS was right; the DEPENDENCY on it was the bug. And since nothing Plex
advertises listens on 443, the clients had left the Plex path entirely.
Fix: <dashed-ip>.<serverhash>.plex.direct, which is RESOLVER-INDEPENDENT -- a
public zone returning the private address embedded in the label, so every
resolver (PiHole, the router, 1.1.1.1, a DoH browser) answers it with
192.168.2.24. No split-horizon entry, no Cloudflare record and no certificate
work needed: the wildcard cert Plex already serves on 32400 covers that name.
Re-verified today with full validation (no curl -k): 200, TLS verified.
The nicer hostname cannot be made correct for clients that bypass PiHole -- a
public record resolves to the WAN address and there is no NAT hairpin, so it is
right only for PiHole users and wrong for the rest. There is no configuration of
it that is right for both. PiHole's plex-vip answer is KEPT; it is just not what
Plex advertises.
Cost restated, not buried: the value embeds the VIP and the cert hash as a
literal and nothing checks they still agree. Accepted because an unresolvable
name is a worse failure -- silent, LAN-wide, and it points users at the router.
UNVERIFIED: unreconciled, and the client-side resolver path is INFERRED -- my
public-resolver probe was broken (NXDOMAIN even for a name that resolves) so I
discarded it. The chain is the only explanation consistent with the
measurements, but the NXDOMAIN-hijack step was not observed directly.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
I broke the app tier with #48. Two faults, both mine, both measured after merge:
1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key
Secrets (plex-vip-<random>) were denied by
secret-provenance/secrets-must-be-flux-managed, which matches every Secret in
a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes
only those labelled secret-managed-by=controller or carrying a Flux label.
`plex` HAS that label; `gateway` -- where the working wildcard cert lives --
does NOT. That asymmetry is why the existing Certificate was never affected
and mine was, and I did not check it. secretTemplate cannot fix it: those
Secrets come from the keymanager controller, not the Certificate template. So
no CertificateRequest, Order or Challenge was ever created.
2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod
rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so
Flux waits on the Certificate OBJECT:
timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']
apps-wave-0 never went Ready; wave-1/-2 and everything downstream of
platform-config sat in DependencyNotReady behind it. I guarded the pod and
missed the object. That distinction is the whole bug.
Third, independent: the advertised URL was https with no servable certificate --
Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip --
the exact failure mode of the outage earlier the same day.
So: delete the Certificate (pruning it clears the health gate) and advertise
http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the
name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there.
http is explicitly temporary and only tolerable because remote access does not
work yet; the restore condition is written at the call site.
DELETED, not merely unregistered: my first attempt kept the file as "tracked but
not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02)
fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a
TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization
+ the https flip, as ONE change) is recorded in catalog/plex.yaml.
Kept and verified working from #48: the hostname design, PiHole's split-horizon
answer, and the postStart hook -- LanNetworksBandwidth is now set live.
Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
The requirement is log visibility only -- which address connected to Plex.
Plex's own log cannot show it. Every LoadBalancer Service here is
externalTrafficPolicy: Cluster, so the node holding the VIP's L2 lease SNATs
the client before forwarding to the backend pod. Measured: Plex had logged
3,214 requests from 10.0.1.x (lease holder k3s-prod-3's own address, Cilium
identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No
X-Forwarded-For either -- it serves raw tcp/32400, not HTTP via the Gateway.
Nothing on the router does this; there is no hairpin NAT. It is Cilium.
The address is not lost, only unrecorded. Cilium observes the flow at ingress,
BEFORE its own translation, so Hubble already has it:
192.168.2.133 -> 10.0.0.135 real LAN client -> the Plex pod
10.0.1.78 -> 10.0.0.135 same traffic post-SNAT, what Plex sees
What was missing is RETENTION: 4095/4095 flows at ~303/s is about thirteen
seconds of history, metrics disabled, no exporter configured.
So enable the static exporter, port-scoped, to the agent's stdout -- Alloy
already tails every pod into Loki, so this needs no new mount, no rotation
handling and no Alloy change.
{namespace="kube-system", app="cilium"} |= "32400"
Scoped BY PORT, not namespace, on evidence: --to-namespace plex returned only
the post-SNAT flows; the record carrying the real client address did not match
it. A namespace filter would drop exactly what this exists to capture.
Rejected, with reasons: eTP:Local is incompatible with cilium 1.20's L2
announcer per upstream docs for this version ("may cause service IPs to be
announced on nodes without pods causing traffic drops") -- and the VIP is on
prod-3 while the pod is on prod-1, so it would black-hole 32400 immediately;
making it safe costs Plex its ability to reschedule. DSR: routing-mode is
vxlan, not native/geneve, and DSR returns bypass the ingress node, breaking
the router's stateful NAT on the remote path.
Also corrects the repo's own record: 2026-10-03 says the SNAT lands in
192.168.0.0/16 (the node's LAN address). It is 10.0.1.x, the cilium_host
inside the pod CIDR -- which is why LanNetworksBandwidth can never separate
local from remote while eTP is Cluster.
UNVERIFIED and stated in the handoff: nothing is reconciled; `filePath: stdout`
is taken from a verbatim upstream sentence but not confirmed against 1.20.1
source (a wrong notation is a silent no-op, with the file+Alloy fallback
documented); the port FlowFilter has no upstream example; and this change
ROLLS CILIUM -- prod-3 announces the Gateway, PiHole DNS and Plex VIPs, so a
lease move during the rollout could briefly interrupt LAN DNS.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that
have never once scraped. Measured on prod 2026-10-04: all six read health=down with
lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the
configured scrapeTimeout, i.e. packets DROPPED rather than refused.
Cause: every app namespace ships a default-deny NetworkPolicy with
policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only
ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180.
Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr,
qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object.
render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None
unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this
catalog declares one. It was wired in at the call site and had never emitted a single
object on any branch: correct, and entirely latent.
It now also fires for the forward-auth gate sidecar's own metrics port, keyed on
_needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since
podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the
sidecar exists. Keying on observability.metrics instead would let metrics: false produce a
discovered-but-blocked target: the same "two halves authored apart" failure this function
exists to prevent.
Rendered here rather than in platform/oauth2-proxy because that tier would have to name app
namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster
and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering
per-app dissolves it.
Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring.
apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified
byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved.
Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an
object necessarily drifts the goldens that pin the renderer's output. The first gate run
caught container-workload-render and egress-class-render as FAIL with exactly this CNP as
the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the
evidence that the regen is faithful and not a rubber stamp.
That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author
approving review. The local gate cannot see that — offline it reports approval-policy green
(N/A).
Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went
FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no
GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements.
Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Two symptoms after the publishing fix landed:
1. LAN clients were reported as Remote. Measured root cause:
LanNetworksBandwidth was ABSENT, so Plex had no notion of a local
network and classified everything WAN -- 468 requests logged
"Allowed Network (WAN)", ZERO "(LAN)", including Seerr from inside the
cluster. allowedNetworks only waives AUTH; it does not make a client
local. Two settings, and only one has an env hook.
2. The advertised URL hard-coded both the VIP and the server's cert hash
into a cluster-var -- two values nothing kept in sync.
Operator design: ONE hostname, plex-vip.${BASE_DOMAIN}, split-horizon --
PiHole answers it with ${PLEX_L4_VIP} on the LAN, Cloudflare answers it
publicly with the WAN address NATed to the same VIP.
- catalog/plex.yaml: ADVERTISE_IP -> https://plex-vip.${BASE_DOMAIN}:32400,
plus a level-triggered postStart hook (pihole/qbittorrent idiom) that
re-asserts LanNetworksBandwidth and the three customCertificate* prefs
every start, converting PEM -> PKCS#12 so cert renewals need no extra part.
- apps/wave-0/plex-cert.yaml: cert-manager Certificate for exactly that
name from letsencrypt-prod, so the advertised https URL is servable.
- schema 1.11.0: `optional` on secretKeyRef/configMapKeyRef. Load-bearing --
the cert is issued INTO the plex namespace, which the rendered plex.yaml
itself creates, so it lands in the same reconcile; hard refs would have
held Plex in CreateContainerConfigError and, since apps-wave-0 is
wait:true, stalled the whole app tier on a slow DNS01 challenge.
- catalog/pihole.yaml: the plex-vip split-horizon answer. Required --
measured, the name resolved to the Gateway VIP where nothing listens
on 32400 (http=000).
- PLEX_ADVERTISE_URL deleted; the hostname removes the cert-hash fragility.
Corrects my own earlier claim that a SAN match is REQUIRED: it is not
(httpsRequired=0, secureConnections defaults to Preferred, so clients can
fall back to cleartext). A matching cert keeps the remote leg ENCRYPTED.
Hence: obtain a real cert for the advertised name rather than pick the name
to suit Plex's built-in one.
UNVERIFIED, and stated plainly in the handoff: nothing here has run. Cert
issuance for this name is inferred from the wildcard's success, and whether
Plex accepts an OpenSSL-3 PKCS#12 bundle is untested -- the largest risk.
Touches clusters/** -> sensitive paths -> needs a non-author approval or the
operator's /approve-override; deliberately not self-posted (CLAUDE.md 1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
customConnections held two hand-typed URLs and neither could ever complete, so
Plex had nothing publishable and the server disappeared from the account:
- https://192.168.2.24:32400 -- the server's cert is
*.130072f20c334d6c8f63d0dfef87309b.plex.direct with NO IP SAN, so bare-IP
https fails TLS for every client. Measured: a continuous storm of
"CERT: incomplete TLS handshake ... certificate unknown" from real LAN
clients arriving through the plex-l4-tcp VIP.
- https://plex.home.keithjackson.dev:32400 -- nothing listens; the Gateway
serves that host on 443 (measured http=000 against 192.168.2.20:32400).
With no completable connection plex.tv published nothing ("Mapped - Not
Published"), Plex fell back to Relay, and clients reported "not authorized".
Fix: advertise <dashed-ip>.<serverhash>.plex.direct -- public DNS that resolves
to the embedded address, so it points at the LAN VIP and matches the wildcard
cert. Verified with full TLS validation (no curl -k): 200 + verify OK, where
the bare-IP form fails to connect.
Declared as ADVERTISE_IP so git owns the value instead of a web form: the
image's 40-plex-first-run writes it into customConnections on every pod start,
outside the claim guard.
Does NOT fix off-LAN access -- the advertised address is RFC1918 and the WAN
tcp/32400 forward remains unverified. ManualPortMappingMode=1 with an absent
port is left as-is (no GitOps path exists for it).
Touches clusters/** -> sensitive paths -> needs the operator's
/approve-override, deliberately not self-posted (CLAUDE.md 1).
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
No manifests change. The three entries merged earlier today each say, correctly
at the time, that nothing had been reconciled -- which is now stale in exactly
the way CLAUDE.md section 2b warns about: a reader would conclude the work is
unproven. This entry is the current state and does not edit the earlier files.
Measured on live prod after dce20b7: apps-wave-0 reconciled in ~90s (inside the
5m timeout the multi-replica entry flagged as a risk); pihole-0 on k3s-prod-3
and pihole-1 on k3s-prod-2, so the required anti-affinity was satisfied; both
per-replica claims Bound; the old Deployment and the old unused PVC both pruned;
PDB healthy with one disruption allowed.
The mountPath fix is confirmed: /etc/pihole is now on /dev/longhorn with 6.9M
used on both pods, where it was previously on the overlay.
Phase 1's declarations reproduce onto a FRESH volume, which was the whole reason
for sequencing it first: 72,234 gravity domains rebuilt on each replica from the
declared adlist, and the declared SOPS secret authenticates on BOTH pods
("valid":true via each container's own env, value never printed).
FAILOVER DRILL, observed by the operator: a replica was deleted, recreated in
~30 seconds, and DNS never dropped a request. First direct evidence the replica
count delivers what it was added for.
Re-deriving the SLI from the rendered workload kind mattered concretely: the
StatefulSet series returns 2 while the old Deployment series now returns ZERO
series, so a stale Deployment-shaped rule would have matched its own absent()
branch and fired permanently -- a standing false page.
Still unverified and stated as such: node failure (the drill deleted a pod, so
the 300s toleration path is untouched), DNS during the cutover itself, two-node
loss, and the cluster-wide dependency that replicas cannot address.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Phase 2 of making LAN DNS survive a node failure. Pihole was one pod on one
node; measured 2026-10-04, losing that node meant roughly 6-10 minutes with no
LAN resolution (tolerationSeconds 300 + NotReady detection + a Longhorn RWO
force-detach), while losing the node holding the pihole-dns L2 lease -- a
DIFFERENT node -- meant tens of seconds. Two of three nodes were single points
of failure with wildly different blast radii.
Adds `availability.replicas` to the ServiceDefinition schema (1.10.0). Intent
only; the mechanism is render.py's, documented in the renderer contract. 1 or
absent renders a Deployment exactly as before -- verified byte-identical across
the whole apps/ tree. >1 renders a StatefulSet (a longhorn PVC is RWO, so a
multi-replica Deployment never schedules the second pod), per-replica
volumeClaimTemplates, REQUIRED hostname anti-affinity, podManagementPolicy
Parallel, a minAvailable:1 PodDisruptionBudget, and an SLI that follows the
rendered kind (a Deployment-shaped rule against a StatefulSet absent()s forever).
Pinned by ~20 new --selftest assertions rather than a conformance change.
ALSO FIXES A LIVE BUG. Pihole's config PVC was mounted at the renderer default
/data/config, but pihole writes to /etc/pihole. Measured: /etc/pihole (7.1M --
gravity.db, pihole.toml, query DB) was on the OVERLAY, while the 2Gi Longhorn
volume held lost+found and 24K of nothing. So every restart discarded all state
and re-downloaded 72k gravity domains, and backup.policy has been backing up an
empty volume while reporting success. Fixed here because otherwise the new
per-replica volumes would hold nothing while state stayed ephemeral -- and
because persisting gravity.db is what makes a restart cheap.
The /etc/pihole mount is safe on the image's own evidence: start.sh is PID 1 as
uid 0 and chowns the directory recursively (no fsGroup needed), and it calls
migrate_gravity which creates the database when absent.
The gate caught a real regression on the first run: the wave-0 alert unit test
fed pihole a Deployment-shaped series. Updated; promtool SUCCESS.
Touches no sensitive paths, so this one needs no override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Prerequisite for running more than one PiHole replica. A second replica with a
fresh PVC would have come up serving DNS and blocking nothing, with a different
admin password -- answers varying by which pod replied.
pihole.toml's own ### CHANGED markers gave the exact inventory: four settings
already came from env (declared), one came from the image, and exactly ONE
operator-set value lived only on the PVC -- webserver.api.pwhash, a one-way hash
nobody could recover. Separately the adlists were undeclared: one list and 72,234
gravity domains that nothing in this repo named.
Declares both:
- A SOPS secret feeding FTLCONF_webserver_api_password. webserver.api.password
takes cleartext and FTL hashes it itself (verified: pihole setpassword IS
setFTLConfigValue on that key), so the PVC's pwhash becomes derived state.
- A level-triggered postStart hook that reconciles the declared adlist set into
gravity.db -- adds, re-enables, and disables (never deletes) undeclared lists,
running pihole -g only when something moved. A file would not work: v6's
adlists.list import is migration-only, guarded on the DB not existing.
The rendered hook was run in the live pod against throwaway copies of gravity.db,
exercising all four paths. A fresh DB built from the image's own schema template
has ZERO adlists, confirming this is load-bearing rather than cosmetic.
Password kept rather than set empty: the pihole Service still publishes port 80
past the oauth2-proxy, closed only by policy, and default-deny has drifted here
before.
Multi-replica is phase 2 and deliberately not in this branch -- it needs a schema
field, a StatefulSet (the config PVC is RWO), anti-affinity and a PDB.
Adds a *.sops.yaml -- a sensitive path -- so approval-policy will block pending
an operator /approve-override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Remote Plex clients had no path in, for two measured reasons: Plex advertised
no remote connection at all (customConnections and ManualPortMappingMode both
ABSENT while PublishServerOnPlexOnlineKey=1), and its Service was ClusterIP, so
32400 had no off-cluster path. The L7 Gateway was NOT the problem -- it serves
Plex correctly (200, claimed=1, 401 on protected endpoints) -- but Plex clients
discover servers through plex.tv, not DNS, so an HTTPRoute cannot put this
server on a client's connection list.
Operator chose the native path over ADVERTISE_IP: Plex terminates its own TLS
with *.plex.direct and streams bypass the Envoy L7 hop.
Adds a hand-written non-catalog unit (the SD schema cannot express a
LoadBalancer Service or a pinned lbipam VIP), registered in render.py's
EXTRA_UNITS so --emit-waves places it in plex's own wave.
externalTrafficPolicy: Cluster, inheriting crafty's 8-day-outage lesson rather
than re-deriving it. Known consequence, documented: SNAT puts remote clients
inside ALLOWED_NETWORKS, so Plex meters them as LAN. Not an auth change.
Remote access does NOT work until the operator does two manual steps that
cannot live in this repo -- the WAN tcp/32400 forward and Plex's "manually
specify public port" toggle (no env var exists for it; verified NONE in the
image's init surface). runbooks/plex-remote-access.md is the procedure.
Touches clusters/** -- a sensitive path -- so the approval-policy stage will
block pending an operator /approve-override.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing
mounts cgroup2 there. The agent mounted one from inside its own private
cgroup namespace, so socket-LB attached to the agent's own cgroup and
host processes got no Service/LB translation. k3s-prod-1 (Gateway L2
lease holder + kube-vip VIP) therefore could not reach the Gateway VIP,
and its apiserver's OIDC authenticator has failed since 2026-09-12.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
After public DNS moved to the WAN IP (#40), prod nodes resolve
keycloak.home.keithjackson.dev to a WAN address the router does not
hairpin, so the apiserver could not reach its OIDC issuer. Adds an opt-in
node_baseline_hosts_pins /etc/hosts block; prod pins keycloak to
GATEWAY_VIP.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Gateway-published records now CNAME to wan.${BASE_DOMAIN}, which a
favonia/cloudflare-ddns Deployment keeps at the home WAN's current public
IP. Operator decision: every Gateway-published name goes public; internal
services (controller: none) remain unpublished.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
jakman85andClaude Opus 5