27d0ef332416d3ca7c6fee3dd75e846bfbfe0dee
707
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 gitea-admin
|
27d0ef3324 | Merge pull request 'chore(plex): fresh plex.tv claim token' (#4) from feat/plex-claim-token-013124 into main | ||
 jakman85
|
bbd4f682a6 |
chore(plex): fresh plex.tv claim token
Injected inside the plex.tv claim window per runbooks/plex-claim.md. |
||
|
jakman85
|
0391139240 |
Merge pull request 'feat(plex): wire PLEX_CLAIM from a SOPS Secret (placeholder value)' (#1) from feat/plex-claim-sops-secret into main
Reviewed-on: #1 |
||
|
jakman85
|
e9f441b5ce |
Merge pull request 'docs(gitea): a skipped job reports SUCCESS, not pending' (#3) from fix/conformance-context-is-fail-open into main
Reviewed-on: #3 |
||
 jakman85andClaude Opus 5
|
234cc6a71a |
docs(gitea): a skipped job reports SUCCESS, not pending -- correct the deadlock claim
The runner came up on 2026-09-29 and the first CI jobs this repo has ever run went green, which falsified a claim merged in #2. MEASURED: gate / gate (pull_request) success 1m58s (PR 1), 1m34s (PR 2) gate / gate (push) success 1m32s (PR 2 only) gate / conformance (pull_request) success "Has been skipped" #2 said listing the conformance context as a required check would deadlock every PR, because the job is `if: false` and never runs. That is wrong. Gitea reports a skipped job as SUCCESS. Requiring it would not deadlock anything -- it would pass VACUOUSLY, forever, while nothing ran. The conclusion is unchanged (never require it) but the reason inverts, and the corrected reason matters more: under HARD RULE 4 a permanently-green check that verifies nothing is worse than a deadlock. A deadlock is loud. This is the exact fail-open shape the 2026-09-18 audit and this one both keep finding. Also records that the (push) contexts only appear for branches matching gate.yml's push trigger -- feature/**, fix/**, docs/**, chore/** -- and NOT feat/**, which is why PR 1's feat/ branch produced only pull_request contexts. Require the pull_request context alone, or PRs from non-matching prefixes never satisfy the rule. CLAUDE.md's enforcement table is updated: the runner half is genuinely fixed and observed green, the branch-protection half is not. Adds the warning that a green tick is not by itself evidence that anything ran. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
69a0d1a1c7 |
Merge pull request 'docs: delivery contract for agents + record that the merge gate does not exist' (#2) from docs/ci-delivery-flow-for-agents into main
Reviewed-on: #2 |
||
|
jakman85
|
d3a0d587a4 | Added gitea action runner secret | ||
|
jakman85andClaude Opus 5
|
fe862ab527 |
infra(gitea): recover the Actions runner before anything requires the gate
No runner is registered at repo, org or site level, so gate.yml has never executed. PR #1 sat pending for 30.6 minutes across 122 polls and never reported. Requiring the gate as a status check in that state would lock the repo permanently. Adds runner-token-rotate.sh: mints a fresh instance-level registration token from the Gitea API and writes it into the SOPS secret via sops set, without ever printing the value. Records the likely root cause, which a new token alone does NOT fix. act_runner registers once and persists its identity in /data/.runner on the act-runner-data PVC. If Gitea no longer holds a matching runner row, the pod starts, sees .runner, SKIPS registration entirely, and polls with an identity Gitea does not recognise. Symptom: zero runners, pod running, nothing obviously broken. Clearing the PVC is what forces a genuine re-registration. The README now sequences runner-first, enforcement-second, and requires CONFIRMING the runner reports online rather than assuming it. That assumption is the whole defect: DECISIONS-LOG recorded the runner as online on 2026-08-31 and nobody checked since. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a2665c0511 |
infra(gitea): script the branch protection that was never applied
Gitea repo settings live in its own database, not in a manifest Flux reconciles, so they are drift-prone and invisible to the conformance gate -- which is how main came to accept direct pushes while the log recorded it as protected. Adds infra/gitea/branch-protection.sh (blocks direct pushes to main) and a README carrying the measured before-state and the ORDER the remaining steps must happen in. The script deliberately leaves enable_status_check:false and required_approvals:0. Both would deadlock today: there is no runner to satisfy a required context, and the only human account is not yet a repo collaborator, so no one could supply an approval. The README sequences both. It prints a push probe to run afterwards. The API response is not proof -- Gitea may let site admins bypass protection, and that has to be observed, not assumed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
5a9e455149 |
docs: delivery contract for agents + record that the merge gate does not exist
Going to land the plex claim wiring through the documented PR flow surfaced that the flow has never run. PR #1 is the first pull request this repo has ever had. Verified against the live Gitea, not inferred from the docs: * branch_protections is [] -- main is protected:false, enable_status_check:false, status_check_contexts:[], required_approvals:0, user_can_push:true * ZERO Actions runners registered at repo, org and site level DECISIONS-LOG 2026-08-31 asserts a "required status check" and a runner "incluster-dind online". Both are false of the live system. The gate is neither enforced nor able to run, and that went unnoticed because the log claimed otherwise -- the same fail-open class as the two holes found on 2026-09-18. Adds CLAUDE.md, the first agent-facing delivery doc here: the branch -> PR -> gate -> approval -> merge rule, stated alongside the honest fact that enforcement is currently weaker than the rule, so agents follow the rule rather than what they can get away with. Carries the toolbox gate invocation, the two git-worktree workarounds, the rc 0/1/3 contract, and secret handling. Records the ordering trap: enabling a required status check before a runner exists deadlocks every PR permanently, including the one that would fix it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
4fd75ccdaa |
feat(plex): wire PLEX_CLAIM from a SOPS Secret so the server can be claimed
Plex cannot create libraries over the already-mounted 8.33T NFS bulk share while the server is UNCLAIMED. catalog/plex.yaml carried a long argument for leaving PLEX_CLAIM unwired; the operator has decided to override it. This implements the wiring in the way that respects the half of that argument which is still true. WHAT WAS TRUE AND STILL IS: a https://plex.tv/claim token has a ~4-minute TTL, far shorter than commit -> push -> Flux reconcile -> pod-roll on its own timers. So the committed value is a documented PLACEHOLDER and the live token is injected by the operator inside that window, with the reconcile FORCED rather than waited for. runbooks/plex-claim.md is that timed procedure. WHAT WAS NOT TRUE: the old comment claimed "a placeholder/absent claim Secret would crashloop the pod". Half of that is wrong, and the difference is the whole design. Established from plexinc/pms-docker's own first-run script (branch `master`, root/etc/cont-init.d/40-plex-first-run, s6-overlay v2), cross-checked byte-identical against the same file inside the running :latest image: * the claim exchange (line 106) is gated on PLEX_CLAIM being non-empty AND the PlexOnlineToken pref in /config being empty. Once the server is claimed the block is skipped entirely, so the env is a NO-OP on every later start whatever it holds ("If server is already logged in, this parameter is ignored" -- upstream README). * with an invalid/expired token on a still-unclaimed /config, plex.tv answers 403 with an empty body; curl's status is never checked, the token sed yields empty, the success branch at line 120 does not fire, there is no else branch and no non-zero exit, and the script (no `set -e`) exits 0. The pod comes up UNCLAIMED AND SERVING. It does not crashloop. * the image never sets S6_BEHAVIOUR_IF_STAGE2_FAILS (default 0 = continue even if a cont-init script failed), so even a non-zero exit would not halt startup. An ABSENT Secret is the real hard failure (secretKeyRef has no `optional`, so it is CreateContainerConfigError), which is why the Secret is committed, encrypted and listed in the wave kustomization rather than hand-created at claim time. NO SCHEMA CHANGE. `optional` on secretKeyRef would only be needed if the pod had to tolerate a missing Secret or could not start without a VALID token. Neither holds: the Secret always exists in the Flux graph, and an invalid token is non-fatal. schemas/servicedefinition.schema.json is untouched, and so are catalog/render.py's env emission and the renderer contract. Changes: * apps/wave-0/plex-claim-secret.sops.yaml -- new Opaque Secret plex-claim/plex, key PLEX_CLAIM, SOPS-encrypted to the committed age recipient. Its header carries NO dollar-brace expressions on purpose: kustomize preserves body comments into the stream envsubst reads (per conformance/tests/l0-static/flux-postbuild-substitution), so quoting the upstream shell conditions verbatim would have been a live substitution hazard on apps-wave-0, not prose. * catalog/plex.yaml -- the PLEX_CLAIM env via valueFrom.secretKeyRef, the "deliberately NOT set" block rewritten around the TTL + one-shot design, and the stale crashloop claim in the /config storage comment corrected. * apps/wave-0/plex.yaml -- re-rendered with the canonical command (`--all catalog -o apps --wave-layout --no-image-automation`). No L1 golden under conformance/tests/l1-rendering covers plex, so this is the only rendered surface that moves. * catalog/render.py -- plan_apps_waves places `<svc>-claim-secret.sops.yaml` in the app's OWN wave, like the bootstrap secret, plus a selftest assertion. Without this, the next `--emit-waves` would have silently dropped the Secret from apps/wave-0/kustomization.yaml and put the pod into CreateContainerConfigError. apps/wave-0/kustomization.yaml is the hand-pruned subset (vm-demo/keycloak are rendered but deliberately unlisted), so its one new line was applied without re-emitting the rest. * runbooks/plex-claim.md + its kustomization.yaml generator entry + README index -- the operator procedure that fits in 4 minutes, including the single forced-reconcile command and the mandatory pod roll (a changed Secret does NOT restart anything: PLEX_CLAIM is read only at container start and the pod template is byte-identical across this change). Touched requirements: R-SEC-01 (secrets encrypted at rest), R-GITOPS-02 (ServiceDefinition onboarding contract). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
d7f24d0a9e | merge: fix/unit-test-coverage-means-asserted-not-mentioned | ||
|
jakman85andClaude Opus 5
|
712f4c9336 |
fix(conformance): "unit-tested" counted an alert MENTIONED in a comment as tested
alert-rules-unit-tested decided whether an alert had a unit test with a bare word
match over the whole test file. Any mention counted — including prose in a `#`
comment.
Hit for real while adding platform/monitoring/tests/vpn-egress.test.yaml. Its header
explains that VPNTunnelDown stays quiet during an egress-steering failure, and that
sentence alone made this check declare VPNTunnelDown "asserted on" and demand its
removal from UNTESTED_ALERTS. Removing the entry would have recorded coverage that did
not exist — the precise "looks fixed" outcome this check exists to prevent, produced
by the check itself. (In that instance I made the coverage real instead, which also
exposed that the fixtures were missing tun0's receive counter entirely.)
Now an alert counts only when a test file names it in an assertion position, with
comment lines stripped first so a commented-out assertion does not count either.
BOTH promtool idioms are accepted, because both are real assertions:
alert_rule_test: - alertname: CNPGPrimariesCoLocated
promql_expr_test: - expr: 'ALERTS{alertname="CNPGPrimariesCoLocated"}'
Matching only the first is not a stricter check, it is a WRONG one: it reported
CNPGPrimariesCoLocated as untested when platform/cnpg/alerts.promtool-test.yaml pins
it across four scenarios entirely through the second form. Believing that would have
produced a redundant duplicate test for an alert already covered better than most —
so the first version of this very fix had to be corrected before it landed.
MEASURED BEFORE LANDING, because the registers are shrink-only and a prose-only alert
would have required an entry this check forbids adding: of 61 alerts, 32 are asserted
in one of the two idioms, 29 are not mentioned at all (already declared), and ZERO were
prose-only. So this tightening failed nothing on the day it landed.
My first measurement said 31/30/0 and was wrong in a way worth recording: it scanned
only tests/*.test.yaml and missed the <manifest>.promtool-test.yaml companion
convention, which is exactly what hid the CNPGPrimariesCoLocated false negative.
Verified in both directions: 134 assertions pass with the real coverage intact, and a
temporary prose-only mention of NodeExporterDown (a registered untested alert) leaves
it correctly listed as untested instead of triggering the stale-entry failure.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e3e56af461 |
docs(decisions-log): overnight handoff — backup alert crying wolf, 49 unmapped app manifests, egress branch finished
Records the three fixes and, in each case, the thing that was reporting green or red over something other than what it claimed to measure. Includes corrections to my own numbers (the '192 unmapped files' figure was my audit script's regex matching across a newline, not the repo) and the two check defects the egress branch exposed. PM HANDOFF lists open items in priority order. R-STATE-02 remains the Phase 7 exit item and is blocked on the permission classifier, not on readiness. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a3f4ddb971 |
merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
5f3ea63b7f | merge: fix/touched-requirements-maps-apps-and-c2 | ||
|
jakman85andClaude Opus 5
|
23c335257b |
fix(ci): apps/* and the C2 UI attributed no requirement, so no behaviour test gated them
ci/touched-requirements.sh's coarse path map had no arm for apps/*. A change adding
or editing ANY workload on the cluster — 49 tracked manifests across apps/wave-0,
-1, -2 and apps/vms — attributed zero requirements, so the merge gate selected only
the whole-repo-scoped checks and none of the requirement-scoped L2 behaviour tests.
The tests that enforce default-deny, egress class and auth mode did not run on
exactly the changes that can violate them. Kyverno admission and the L0 egress-path
check are real backstops, but admission is post-merge and the gate exists to catch
it first.
platform/homepage and platform/headlamp were unmapped too — the C2 UI. That gap was
already written down in changed_diff()'s rule-3 comment ("the ONLY route to
R-C2-01/02 was rule 3", i.e. editing the C2 test itself) and had sat there since.
Twelve arms added: apps/vms/* (ordered BEFORE apps/*, because `case` takes the first
match), apps/*, platform/homepage|headlamp, platform/kubevirt, platform/gateway,
platform/coredns, platform/crafty, platform/external-snapshotter|backup, runbooks/*,
platform-config/*, platform/kustomization.yaml and .sops.yaml.
platform/kustomization.yaml earns its own arm: it is the assembly list, so one
deleted line disables an entire platform component while every file it points at
stays perfectly valid on disk. That is the most consequential single-line edit in
the repo and it attributed nothing.
Measured effect for an apps/ change, via --print-selection:
before: 0 requirement-scoped tests
after: 13 (3 L1, 8 L2, 2 L3)
Coverage stays 51/51, so the gate's "attributed IDs must be covered" assertion does
not false-block.
Ten new assertions in ci/tests/touched-requirements.test.sh (26 total, all passing),
including the ORDER property — apps/vms/* must win over apps/* — because arm order
is a real failure mode of `case`, not merely a question of both arms existing. Each
fixture cites no R-ID of its own, so any ID in the output came from the PATH and not
the content scan.
A correction on numbers I reported earlier in this session: I claimed "192 unmapped
files", then "182". Both were inflated by my own audit script rather than the repo.
Its regex used `[^)]*`, which matches across a newline, so the first arm parsed as
`case "$1" in\n catalog/*` and never matched anything; I also mis-parsed
alternation arms like `.gitea/*|ci/*`, which made ci/ look unmapped when it is
mapped. True remainder after this change is 18 files, all prose or captured
artifacts (docs/plans/, root READMEs, .standup-dns/*.txt).
Gated: offline l0,l1,meta --include-scope whole-repo, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
9339d5a79e | merge: fix/velero-skip-emptydir-volumes | ||
|
jakman85andClaude Opus 5
|
a1cc0a5556 |
fix(velero): skip emptyDir volumes by type; revert the narrower namespace exclusion
Supersedes
|
||
|
jakman85
|
6eaaaae334 | merge: fix/node-memory-swap-cnpg-placement | ||
|
jakman85andClaude Opus 5
|
cde02d19d2 |
fix(runbooks): node-memory-swap claimed the CNPG primaries sit on node 1
They do not, and the correction changes the operator's plan rather than just a detail. Verified live: all three CNPG clusters run 3 instances spread one per node by required anti-affinity, and k3s-prod-1 holds a REPLICA of each (keycloak-db-2, nextcloud-db-2, semaphore-db-2). No primary is on node 1, so draining it forces no failover at all — the best state this maintenance could ask for. The original sentence was carried over from earlier session state and was simply stale; the runbook now prints the command to re-derive it, because primaries move on their own after any restart or failover. Adds the warning that matters more. CNPGPrimariesCoLocated is currently firing for k3s-prod-2, which holds both the keycloak-db and semaphore-db primaries. The instinct is to tidy that before a maintenance window. Doing so would be actively harmful here: rebalancing means promoting a replica, the only node with spare primary capacity is node 1, and so "fixing" the alert would put a primary on the very node about to be powered off — converting a no-failover drain into a two-database failover. Rebalance after node 1 is back and caught up. Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
05a8094785 | merge: fix/velero-partialfail-on-own-maintenance-jobs | ||
|
jakman85andClaude Opus 5
|
d99d47494c |
fix(velero): nightly backup PartiallyFailed on Velero's own maintenance Jobs
VeleroNoRecentBackup has been firing CRITICAL on prod over a backup that was actually protecting everything. velero-daily-20260919030041: PartiallyFailed, 3079/3079 items backed up, exactly 2 errors — both PodVolumeBackups for the `plugins` and `scratch` volumes of homepage-default-kopia-maintain-job-1789786837269-qhsvh. Four such PVB failures had accumulated. Velero runs transient kopia repository-maintenance Jobs in its own namespace, one per BackupRepository, each with emptyDir `plugins` and `scratch` volumes. Schedule (1) runs defaultVolumesToFsBackup:true — the opt-out model — so it attempts fs-backup on those volumes too. The Job finishes and its pod disappears mid-backup, and the PodVolumeBackup fails. Nothing real was ever missed. But PartiallyFailed does not advance velero_backup_last_successful_timestamp, so the alert that exists to catch a genuine backup outage fires every night over Velero cleaning up after itself. An alert that cries wolf nightly is how a real failure gets ignored — and this one guards R-STATE-03. Fix uses the pattern already established for monitoring+unifi: exclude the namespace's volume DATA from schedule (1), keep its RESOURCES in schedule (2). Safe by inspection — ns velero holds no PVCs, so there is no volume data to lose, while the BSL, credentials and Backup CRs that DR actually needs are still captured. Deliberately narrower than the general rule. The general fix is a volume policy skipping emptyDir by type, which would also cover a transient Job in any other namespace, but it needs Velero's `volumeTypes` policy condition — unproven in this repo on 1.18.1, and a policies.yaml Velero cannot parse breaks EVERY backup. That is a much worse failure than the one being fixed, and all four observed failures were kopia-maintain pods in ns velero. Rationale recorded inline so the next person can widen it on purpose rather than by accident. Gated: L0 whole-repo 21 cases, 0 failures, 0 skips, 0 unwired. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
6f231a4311 | merge: docs/decisions-log-gate-fail-open-holes | ||
|
jakman85andClaude Opus 5
|
623fa14256 |
docs(decisions-log): PM handoff — two fail-open holes in the gate itself
Records the session's three pieces of work and, more importantly, the theme joining them: the gate was reporting green over things it had not checked. - runbooks/node-memory-swap.md, with the three claims I wrote and then removed because they were false (EtcdIsVoter does not exist here; node-drain-policy is not pinned so the runbook reads it live; the 5-vs-1 rebuild-limit confusion). - The nested-worktree scan: six discovery sites auditing 18 copies of the repo. - L1 rendering never running on a requirement-scoped gate, with the external-dns incident reproduced via --print-selection (0 tests before, 3 after). - The two defects that surfaced in the meta-check guarding all of it. PM HANDOFF lists the open items in priority order. The first is a judgement call worth flagging rather than burying: the planned memory swap on k3s-prod-1 IS a supervised single-node loss on prod, which is exactly the event R-STATE-02 describes and which is still proven only on the decommissioned dev cluster. Capturing evidence during the swap costs nothing; manufacturing the event later costs a second maintenance window. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
16cbd4a79e | merge: fix/l1-rendering-declares-whole-repo-scope | ||
|
jakman85andClaude Opus 5
|
755c08ab40 |
fix(conformance): the meta-check was layer-agnostic and reported failures silently
Declaring whole-repo scope on the L1 tests made whole-repo-invariants-wired (R-TEST-03) fail. It was right to change behaviour and wrong about why, and it could not tell anyone either way. Two independent defects. 1. LAYER-AGNOSTIC WHERE THE DESIGN IS LAYER-BOUNDED The "every scope-declared test is selected despite a requirement filter matching none" assertion compared EVERY scope-declaring dir against an `--layer l0` selection. That silently assumed only L0 tests would ever declare the scope — while ci/l1.sh was written from the opposite premise, that a whole-repo invariant could appear at any layer. So the first L1 declaration made the check blame the union for correctly excluding another layer. Now partitioned by layer, and the L1 half of the property is asserted too: every scope-declared L1 test must be selected by `--layer l1 --include-scope`, and that selection must not leak non-L1 tests. That is what makes the L1 declaration load-bearing rather than decorative — previously nothing proved the union reached L1 at all, because nothing at L1 declared the scope. 2. EVERY ASSERTION FAILED SILENTLY The idiom throughout was a bare test on one line and `chk $?` on the next, under `set -euo pipefail`. A false test aborts the script BEFORE chk can print, so the diagnostic — which is the entire value of this check — was never reached. The real failure above emitted 26 `ok` lines, rc=1, and no message at all; it read as a tooling glitch rather than a finding. All 12 assertion sites now capture the status (`chk $arc`) so the check reports and continues. Proven by forcing MIN_DECLARED=999 against the real check: the failure prints its full explanation and the run proceeds to the remaining assertions, where before it died mute. A meta-check that cannot say why it failed is worse than no meta-check. Verified: 35 assertions, rc=0; full offline l0,l1,meta 25 tests / 0 failures. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b389566641 |
fix(conformance): L1 rendering never ran on a requirement-scoped gate
All three L1 tests now declare `conformance.home.k3s/scope: "whole-repo"`.
ci/l1.sh has passed --include-scope whole-repo since it was written, with a
comment saying no L1 test declared the scope yet and it was wired ahead of time
so the next whole-repo invariant "at any layer" would be picked up by annotation
alone. Nothing ever declared it. The consequence is that the entire L1 rendering
layer only ran when a touched path happened to attribute R-PLAT-02, R-NET-01 or
R-NET-05/06/07 — a fail-open hole for every other change in the repo.
The coupling requirement-scoped selection cannot see is that all three tests
exercise ONE code path: catalog/render.py against
schemas/servicedefinition.schema.json. Anything that changes rendered output
invalidates these goldens no matter which requirement the touched path maps to.
The 2026-09-04 external-dns merge is the proof, measured rather than asserted.
ci/touched-requirements.sh attributes platform/externaldns/helmrelease.yaml to
R-ENV-02, R-OBS-01/02/04, R-STATE-03 and R-TEST-01/02/03/04 — none of the
rendering requirements — yet that change added
external-dns.alpha.kubernetes.io/controller: none
to the rendered output of all three goldens. Via --print-selection:
requirement-scoped only (the 2026-09-04 behaviour): 0 L1 tests selected
with --include-scope whole-repo (how ci/l1.sh runs): 3 L1 tests selected
So the merge gated GREEN while leaving every rendering golden stale, and the
staleness surfaced only on a later full run.
Also corrects the now-false comment in ci/l1.sh that said no L1 test declares the
scope, and records why the annotation is load-bearing there.
Verified: offline l0,l1,meta --include-scope whole-repo in the pinned toolbox —
25 tests, 0 failures, 0 errors, exit 0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85
|
7887ecbbdc | merge: docs/node-memory-swap-runbook | ||
|
jakman85
|
d717d8e979 | merge: fix/conformance-skips-nested-worktrees | ||
|
jakman85andClaude Opus 5
|
bc106caac1 |
docs(runbooks): node-memory-swap — replacing k3s-prod-1's mismatched DIMM pair
The operator has a matched 32 GB pair to install. k3s-prod-1 is the node that needs it: DIMM1 is a Micron 8ATF2G64HZ-3G2E1 (16 GB, DDR4-3200, 1Rx8) and DIMM3 a 16ATF2G64HZ-2G6E1 (16 GB, DDR4-2667, 2Rx8), so capacity is correct at 32 GB but the pair runs at the slower 2667. Nothing is faulty. k3s-prod-3 is a matched Crucial pair. The procedure exists because this node is the awkward one to take down: it held 24 Longhorn replicas, it is one of three etcd members (so the drain leaves quorum at 2 of 3 with no further margin), and its rebuilds are the load that starved etcd once already. So the runbook front-loads the checks that decide whether to start at all, and treats a BLOCKED drain as the safety property working rather than an obstacle to force past. Two things it deliberately does not assert: - node-drain-policy is not pinned in platform/longhorn/helmrelease.yaml, so the runbook has the operator READ the live Setting and set it before draining if it is not block-if-contains-last-replica, instead of assuming the default. - Configured Memory Speed, not Speed, is named as the field that proves the swap worked — a matched pair being clocked down still reports the rated Speed. It also flags the capacity decision as a decision: 2x16 GB keeps the three nodes symmetric as PROD-PROVISIONING.md §2 assumes, while 2x32 GB makes node 1 the asymmetric one on memory as well as disk. Recorded caveats: the node-health artifacts de-duplicate DIMM part numbers, so they can prove "two different modules" but cannot distinguish one module from two identical ones — verification therefore collects dmidecode live. And node 2's inventory is stale and self-contradictory (documented as the only single-DIMM node at 16 GB, yet reporting ~31.1 GiB), so it needs re-inventory before anyone claims three-way symmetry. This is NOT the §0b "do not buy RAM" case, which was an EFI memattr kernel bug on node 2. Wired into the configMapGenerator so it is served at /runbooks/node-memory-swap rather than existing only in git. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
79c703bc79 |
fix(conformance): whole-repo checks audited nested agent worktrees as source
Six discovery sites walked ROOT without pruning dot-directories, so they descended into .claude/worktrees/agent-*/ — up to 18 full copies of the repo, created for isolated agent runs and gitignored (.gitignore:43). The visible half was false FAILs: four ServiceMonitors reported as selecting no Service, and nine PrometheusRule manifests reported as "under test but not in the repo", all of them worktree paths. The whole-repo L0 gate was red for reasons that had nothing to do with authored surface. The dangerous half is quieter. A check that asks "does anything in the repo declare X" can be satisfied by a worktree copy instead of real source, so the same bug that invents failures can also manufacture a pass. That is the reason to fix all six sites rather than only the ones that happened to be failing. egress-paths-selected-by-rules already had this fix and documents it (it once saw 5 CiliumEgressGatewayPolicy objects instead of the 1 authored); this applies the same dot-directory predicate everywhere else, which skips dot-DIRECTORIES while still matching dot-FILES such as .sops.yaml. platform/monitoring/tests/run.sh is the other side of the same failure: its `--list` reported each manifest once per worktree, and the L0 check compares that listing against the tree. Both find calls now prune .claude alongside .git. `--list` returns 21 real pairs where it previously returned them ~19x over. Verified in the pinned toolbox: L0 --include-scope whole-repo is 21 cases, 0 failures, 0 skips, 0 unwired (was 4 failures, then 1). L1 3/3. The 18 stale worktrees are all on already-merged branches and remain on disk; cleaning them up is a separate change and is no longer load-bearing for the gate. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
3523617671 | Rolled unifi console password | ||
|
jakman85andClaude Opus 5
|
6a4dc0ada7 |
docs: INCIDENT — I printed the console password to the transcript
A kubectl jsonpath with {range .data}{@}{end} dumped the unifi Secret's data
map, putting the base64 console-password in the session transcript. Exactly the
constraint I put in every agent brief, broken by my own careless verification
command.
Exposure bounded: 0 commits contain it (git log -S), the capturing scratch file
was shredded, and a repo-wide grep finds nothing on disk. Critically it is not
yet a live credential — the Secret was created two days after the running VMI,
whose volume list is rootdisk and cloudinit only, so the UOSDATA disk was never
attached, install-uos never ran, and the ubuntu account is still locked.
Action recorded with the ordering that matters: rotate console-password BEFORE
recreating the VMI, so the leaked value never becomes valid.
Lesson: checking a Secret's existence never needs its values. Listing .data's
keys or .metadata.name answers the question; dumping .data answers one nobody
asked.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
14d5ccbadd |
fix(unifi): pin the MAC — an unpinned MAC is why the guest keeps losing its IP
Operator set console-password in the SOPS secret (installer-url deliberately left as REPLACE-ME, which install-uos handles by printing 'not set yet' and exiting 0; the uosserver guard exits before it is even read). Root cause of the lost address, from evidence rather than inference. KubeVirt generates a fresh MAC per VMI when none is pinned. cloud-init writes /etc/netplan/50-cloud-init.yaml matching the MAC it saw, and will not rewrite it while the NoCloud instance-id is unchanged — and that instance-id is domain.firmware.uuid, stable across restarts by design. So every VMI recreation handed the guest a MAC its retained netplan did not match, leaving enp1s0 administratively DOWN, never even attempting DHCP. What the 2026-09-12 boot log shows, and why it fits: cloud-init's own device table reads 'enp1s0 | Up: False | Address: .'; there are no apt failures because the per-instance modules were skipped; yet UOS is installed (6.3 GB of 40 GB) and qemu-guest-agent answers — both of which required working network on an EARLIER VMI. The MAC is identical between the current VMI status and that boot log, so it did not change on this boot; the mismatch is against a netplan written under a previous incarnation. Pinned to the MAC the current VMI already holds, so nothing changes for this guest. Combined with the instance-id bump already merged, the next restart re-runs cloud-init, rewrites netplan against a MAC that is now permanent, and the interface should come up — and the console password is in place as the fallback if it does not. Still unverified from outside: the netplan file's actual contents. That needs the console the operator can now reach. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
05881d5a6a |
fix(conformance): regenerate the three L1 goldens the external-dns change drifted
|
||
|
jakman85andClaude Opus 5
|
cf0d6c6beb |
docs: verified — internal services are gone from public DNS, split-horizon works
Confirmed at the authority, not a cache: all six internal names absent publicly, all five public names untouched, and Pi-hole answering for the six internally. The R-DNS-03 disclosure is closed for that tier. Records a correction to my own sequencing: I insisted this could not land before the DHCP repoint, but the router returns nothing for those six names, so the records served nobody and the dependency I asserted did not exist. One query would have shown it. Also records the operator's clarifications — Pi-hole is DNS only with the router still doing DHCP, reverse proxy by hostname rather than a second IP, and Minecraft staying open — plus the unencrypted-secret fix. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
20a28d0993 |
fix(unifi): the installer secret was committed unencrypted
Every other *.sops.yaml in this repo carries sops metadata and ENC[] values; this one shipped with neither and both fields as literal REPLACE-ME. Two consequences: 'sops <file>' fails with 'sops metadata not found', so the documented operator step could not work; and Flux would pass the file through undecrypted and create the Secret with the literal placeholder strings. Encrypted in place with the repo's age recipient, placeholders intact. Round-trip verified: sops -d returns installer-url and console-password as REPLACE-ME, so the operator's workflow is now the normal one — sops the file, set both values, commit. Both fields need setting, not just the password: installer-url is also REPLACE-ME, which I had omitted from the operator instructions. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
b184a42729 |
merge: internal services stop advertising themselves in public DNS
Six exposure:internal services published A records into the public Cloudflare zone answering 192.168.2.20, disclosing the internal subnet, the Gateway VIP and a full service inventory, and forming a DNS-rebind primitive that is why the SOHO router strips these answers in the first place. The renderer now stamps external-dns.alpha.kubernetes.io/controller: none on every HTTPRoute it renders for an internal service, derived from the exposure field alone — so a service that omits exposure defaults to internal and is likewise unpublishable. Established against the deployed v0.21.0's own source, where the controller check runs before hostname resolution in both configured sources, then proven live on a throwaway hostname including that adding the annotation to an existing record triggers the DELETE. Deletes exactly 12 records: the A and TXT pair for jackett, pihole, qbittorrent, radarr, semaphore and sonarr. A 39-record recovery baseline is committed alongside. Safe to land before the DHCP change, contrary to my earlier caution: the router returns NOTHING for all six names today, so these records serve nobody on the LAN, while Pi-hole at 192.168.2.23 already answers for anyone pointed at it. The five exposure:public services are untouched. They remain unreachable publicly, which is a separate problem: the legacy edge at 136.51.115.252 returns HTTP 403 for our hostnames and serves a placeholder certificate, so nothing is reverse-proxying them today. |
||
|
jakman85andClaude Opus 5
|
3173254cc4 |
docs: UniFi OS was never the blocker — it is installed and the guest lost its IP
Third confident claim of mine corrected in two days, and this one I took from a stale README rather than the machine. uosserver.service starts on the current boot and the root filesystem holds 6.3 of 40 GB, while enp1s0 is up with no address and does not answer ARP — so the controller runs and nothing can reach it. Records the near-miss: proceeding on my premise would have reinstalled over a working controller, which the new install-uos guard now prevents. Also records that there is no way into the guest at all, which is why the deliverable became console access, and the cloud-init instance-id discovery that explains why an earlier unattended install could never have taken effect. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
7cb209b9a0 |
merge: UniFi OS is already installed — the guest lost its IP; deliver console access
The premise I briefed was wrong and the agent checked before building on it. UniFi OS Server IS installed: the current boot starts uosserver.service and uosserver-updater.service, and the root filesystem holds 6.3 GB of 40 GB where bare Ubuntu plus our packages is about 2 GB. Verified independently. The actual fault is that the guest has no address — .status.interfaces shows enp1s0 link up with no ipAddress, and ip neigh inside virt-launcher reports 10.0.2.2 FAILED, so the guest does not answer ARP on its masquerade address. Probes from virt-launcher straight to the pod IP and to 10.0.2.2 RST on 11443/8443/8080/22, which exonerates the VIP, the traffic policy and the network policy. The cause cannot be determined from outside the guest. That made one guard non-negotiable: install-uos now bails the moment systemd knows a uosserver unit. Without it this merge would have reinstalled over a working controller. Second finding: there is no way into the guest at all — no SSH key, no password, and SSH cannot work by construction because masquerade forwards only declared ports and 22 is not one. So the deliverable is console access: a SOPS secret carrying installer-url AND console-password, mounted as a NoCloud disk and re-read every boot. The discovery that makes it work: NoCloud instance-id is spec.template.spec.domain.firmware.uuid, stamped on the VM object. Cloud-init compares it to the DMI product UUID, so editing userData alone changes nothing on a second boot — the cached datasource is restored and write_files/runcmd are skipped. That is why a previously shipped unattended install could never take effect. The UUID is now pinned in git and bumped once. Deliberately did NOT put the whole cloud-config behind secretRef: that would hide the shell in ciphertext, out of reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over the decrypted resource — the nextcloud-db shape from 2026-09-13. Script in plaintext, operator values in SOPS. Unattended install is reported honestly as third-party practice, not verified here, and the README's stale dev-era addressing is corrected throughout since it doubles as the rebuild runbook. |
||
|
jakman85andClaude Opus 5
|
12d4e0ae40 |
feat(unifi): SOPS-delivered UOS installer URL + per-boot install unit; correct the README
The operator's UOS bring-up becomes one field. apps/vms/unifi/unifi-uos-installer.sops.yaml
(SOPS/age, with a .example sibling per the crafty/headlamp convention) carries `installer-url`
and `console-password`. KubeVirt mounts it as a read-only iso9660 disk labelled UOSDATA and an
in-guest `uos-install.service` reads it on EVERY boot, so a later URL change needs only a commit
and a VM restart.
Mechanism verified against the DEPLOYED KubeVirt v1.9.0, not from memory:
* inline userData cap is exactly 2048 bytes — server-side dry-run apply accepted 2048 and
rejected 2049; the payload here is 1454.
* `cloudInitNoCloud.secretRef` and `secret.volumeLabel` both exist in the live CRD; the secret
disk + third disk were accepted by a server-side dry-run of the real manifest.
* the NoCloud seed's instance-id IS spec.template.spec.domain.firmware.uuid (read out of
noCloud.iso inside virt-launcher), stamped on the VM OBJECT, so it survives restarts. Editing
userData alone therefore changes NOTHING — cloud-init restores its cached datasource and skips
write_files/runcmd. The UUID is now pinned in git and bumped once so the new payload lands.
secretRef was deliberately NOT used for the whole cloud-config: it would put the shell in
ciphertext, out of reach of the L0 dollar-collapse guard, while Flux's envsubst still runs over
decrypted resources — the 2026-09-13 nextcloud-db shape. Script in plaintext, operator values in
SOPS. apps-vms had no `decryption` stanza (first encrypted content under apps/vms/), so one was
added to both overlays.
CORRECTION, and it is the headline: UNIFI OS SERVER IS ALREADY INSTALLED on the live guest. The
serial console log starts uosserver.service and uosserver-updater.service, and / is 6.3 GB of
40 GB used. The controller is unreachable because THE GUEST HAS NO IP: .status.interfaces shows
enp1s0 up with no address, and `ip neigh` in virt-launcher reports 10.0.2.2 FAILED. 11443/8443/
8080 and 22 all RST, from the pod IP and from 10.0.2.2 directly, so the VIP and the CNP are
exonerated. install-uos therefore bails out the moment systemd knows a uosserver.service —
without that guard this change would have reinstalled over a working controller.
There was also no way into the guest at all: no key, no password, and 22 is not a declared
masquerade port so SSH cannot work by construction. Hence console-password.
README corrections (it doubles as the rebuild runbook and was misdirecting):
* VIP 192.168.1.54 -> 192.168.2.21; pool 192.168.1.53-.60/.53 -> 192.168.2.20-.30 with the
Gateway on .20; device subnet 192.168.1.0/24 -> 192.168.2.0/24; node k3s-dev-1 -> k3s-prod-1.
All re-read live (pool blocks, cluster-vars, the Service EXTERNAL-IPs, LAN_CIDR).
* "UOS is NOT installed" -> it is; the real fault is the lost IP, with the evidence table.
* "apps-vms must NOT use postBuild.substituteFrom" -> it has since 2026-08-31 and the escape
contract is in force.
* "install-uos is no longer baked into the guest" -> it was baked in, and it was broken twice
over (an unverified firmware API, and no `install` subcommand).
* "SSH in, cloud-init installs the operator's key" -> it installs none, and 22 is not forwarded.
* externalTrafficPolicy Local -> Cluster (the 8-day outage fix already in 50-service-l4.yaml).
* 60 GB -> 40Gi (the DataVolume and the Bound PVC); "10 GB minimum" -> ~25 GB.
* userDataSecretRef -> the YAML key is `secretRef`.
* 90-httproute-optional.yaml referenced as if it existed; it was dropped.
Unattended install, honestly: Ubiquiti documents no silent flag, no answer file, no env vars.
`yes y | <installer> install` is the convergent community practice and is what ships here, with a
bare-invocation fallback. It has NOT been run end-to-end on our guest. The installer URL also
does not look signed or time-limited — it is account-derived and per-release; it is in SOPS
anyway, but nothing should depend on expiry.
The running VM was not touched: no restart, no migration, no re-import. Every cluster call was a
read or an explicit --dry-run=server; the live VM still carries firmware uuid ...c07 and the
34h-old virt-launcher pod is untouched. The change takes effect on the operator's next VM restart.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
989781b352 |
docs: Minecraft server live, EULA accepted on the operator's behalf, world backed up
Records the EULA acceptance prominently with revocation instructions, the Server List Ping proof, and Crafty's autostart trap where the create API accepts 'autostart' but stores 'auto_start'. Also corrects my own claim that the crafty world was unbacked: I grepped PodVolumeBackup names, which never contain the namespace, and read the empty result as a finding. Crafty had 2 PVBs in the last run. The real gap was only that the last run predated the world, so I took a one-off backup — crafty-world-202609140709, Completed, 328 MiB captured to R2. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
97257515e8 |
merge: a Minecraft server exists and 192.168.2.22:25565 is playable
Crafty was healthy but had never had a server created — /crafty/servers was empty, which is why the port refused. Paper 26.2 (Mojang's latest.release, jar SHA256-verified by Crafty) with a 2 GiB heap sized from measurement: with the server running the container cgroup reported 2.86Gi against a 4Gi limit, so a 3 GiB heap would have risked OOMKill. EULA ACCEPTED ON THE OPERATOR'S BEHALF — https://aka.ms/MinecraftEULA — via Crafty's agree_to_eula, writing eula=true. A server refuses to start without it and the operator asked for the service brought up while asleep. No account, key or payment involved. To revoke: stop the server and set eula=false, or delete the server. Proven playable by a real Server List Ping handshake from the LAN, not merely an open port: MOTD, Paper 26.2, protocol 776, 0/20 players, 12.3 ms RTT. Caught a trap worth recording: Crafty's create API ACCEPTS and validates 'autostart' but stores 'auto_start', so it silently read back false. Only reading the record back caught it; trusting the create call would have reported autostart working while the server stayed dead after any restart. Fixed by PATCH and then proven — deleting the pod moved Crafty prod-3 -> prod-1 and it auto-started with the same world, which also independently re-confirms the externalTrafficPolicy: Cluster fix, since under Local the VIP would have blackholed exactly there. Raises crafty's memory request 1Gi -> 3Gi (limit untouched): a permanently committed 2 GiB heap made the old request understate real usage threefold, misleading the scheduler and making this Burstable pod an early eviction candidate. |
||
|
jakman85andClaude Opus 5
|
b4bc8e1cb9 |
feat(crafty): first Minecraft server — Paper 26.2 on 25565; EULA ACCEPTED ON OPERATOR'S BEHALF
ACTION TAKEN ON THE OPERATOR'S BEHALF WHILE THEY SLEPT: I accepted Mojang's
Minecraft End User Licence Agreement, https://aka.ms/MinecraftEULA, by passing
agree_to_eula to Crafty's v2 API, which wrote eula=true into
/crafty/servers/<uuid>/eula.txt. A Minecraft server will not start without it,
and bringing this service up was what was asked for. Please review. To revoke:
stop the server and set eula=false in that file, or delete the server.
Crafty was healthy but had never had a server created, so 192.168.2.22:25565
refused connections. Created "homelab" via the API (not UI automation):
Paper 26.2, TCP 25565, heap -Xms2000M -Xmx2000M, autostart on
uuid 13c7b85d-00bb-4a58-8c86-2433ca47891c
Paper over Vanilla for tick performance and plugins, with no cost to players
(vanilla clients connect unchanged). 26.2 is Mojang's latest.release, out
2026-06-16; the 26.3-rc-2 snapshot and 26.2-rc-2 were deliberately avoided.
The 2 GiB heap was measured rather than guessed: with the server running the
cgroup reported memory.current 2.86Gi against the 4Gi limit, so a 3 GiB heap
would have risked an OOMKill.
Proven playable, all real output: Crafty reports running with 'Done (12.737s)!';
Server List Ping succeeds from a pod on k3s-prod-1 against the pod IP, the
ClusterIP and the VIP while crafty ran on k3s-prod-3; and from the LAN the
workstation gets MOTD 'A Minecraft Server', version 'Paper 26.2', protocol 776,
players 0/20 over a full handshake plus ping/pong. Deleting the pod moved Crafty
to another node; it auto-started with no manual action and the LAN ping still
succeeded, re-confirming the externalTrafficPolicy: Cluster fix from
|
||
|
jakman85andClaude Opus 5
|
83f13f4b0a |
docs: correct my 'live outage' claim — unifi and crafty were never up
I reported the UniFi controller and Minecraft server as unreachable for up to 8 days. Probing the backend pods directly, bypassing VIP and policy, returns REFUSED on all four ports while crafty's own UI and pihole's DNS answer as controls. /crafty/servers is empty and the unifi README still lists installing UOS in the guest as the open blocker. Nothing regressed. The externalTrafficPolicy defect is real and is fixed, but it was not the cause. I observed unreachable ports, found a mechanism that produces unreachable ports, and concluded it was the cause without running the one control probe that separates them — the same error shape as nearly blaming the drive for the I/O pressure spike when the local disks were idle. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
adda6dd779 |
merge: L2-announced VIPs must serve from any node, not only the backend's
Cilium's L2 announcer elects a lease holder without regard to which nodes hold a backend, so externalTrafficPolicy: Local leaves the frontend absent from the announcing node's service table entirely and the VIP answers ICMP unreachable. Proven causally with cilium-dbg service list: under Local the frontend is missing from prod-2; flipped to Cluster it appears as an active backend entry; flipped back it vanishes. Durability proven too — cordon prod-1, crafty reschedules to prod-3, and prod-1's UDP lease re-points to the new backend with no intervention, which under Local is exactly the failure mode. IMPORTANT CORRECTION to my own framing: this is NOT why unifi and crafty do not answer, and fixing it does not make them answer. There is no application behind any of those ports. /crafty/servers is empty — no Minecraft server has ever been created — and apps/vms/unifi/README.md step 5 still reads 'NEXT: install UOS in the guest', with the unattended install flagged as the open blocker. Probing the pods directly, bypassing VIP and policy, returns REFUSED on all four ports while crafty's own UI and pihole's DNS answer as controls. Nothing regressed; the 8-day-old leases are just 8-day-old Services. I called it a live outage and it was not. What this commit does is remove a landmine that would have blackholed both VIPs the moment those applications exist. Client-IP cost assessed per service rather than assumed: zero for UniFi, whose KubeVirt masquerade binding already NATs everything into the guest, so Local was protecting an address the controller could never observe; real but acceptable for Crafty, where player identity is a Mojang UUID so bans and whitelists are unaffected, with /ban-ip documented as do-not-use. loadBalancerSourceRanges deliberately NOT copied from the Pi-hole pattern: those Services never had a LAN restriction, and adding one during a fix could cut off off-LAN players. Cilium DSR mode is recommended separately, not bundled — it carries an Ansible parity contract and rolls the CNI across all three nodes. |
||
|
jakman85andClaude Opus 5
|
4664b41db3 |
fix(l4): externalTrafficPolicy Cluster on unifi + crafty L4 VIPs
Cilium's L2 announcer elects one node per Service to answer ARP for its LoadBalancer VIP, and that election is NOT backend-aware. Combined with externalTrafficPolicy: Local -- which lets a node deliver only to a backend on itself -- an elected announcer holding no backend answers ARP for the address and then has nowhere to send the packet, so the client gets ICMP port unreachable. It never self-heals: the lease holder keeps renewing after the backend moves away. Measured on prod 2026-09-14: unifi's VM ran on k3s-prod-1 with both leases on k3s-prod-2, and crafty's pod ran on k3s-prod-1 with the -tcp lease on k3s-prod-2 while its identical -udp twin happened to draw k3s-prod-1 and worked. Confirmed causally on the announcing node with `cilium-dbg service list`: under Local the frontend 192.168.2.22:25565 is absent from that node's service table entirely; under Cluster it appears as `1 => 10.0.0.11:25565 (active)`; flipped back, it vanishes again. Durability confirmed by cordoning k3s-prod-1 and moving the crafty pod to k3s-prod-3, after which the announcing node re-pointed to the new backend with no intervention. Both files also carried a comment claiming Local "pins the VIP to the node running the pod", which is the precise thing that is false here; both are rewritten. apps/wave-0/pihole-dns.yaml is the reference implementation. The client-IP cost is stated per service rather than waved away. For unifi it is ZERO: the VM's KubeVirt `masquerade` binding already SNATs everything into the guest, so Local was never preserving a client address the controller could see. For crafty it is real and accepted -- player identity is a Mojang/Xbox UUID so /ban and the whitelist are unaffected, but /ban-ip would hit every player at once and must not be used; no IP ban or per-IP rate limit is configured today. No loadBalancerSourceRanges here, deliberately departing from pihole-dns.yaml: these Services never had a source restriction (their CNPs admit `world`), so Cluster removes no protection, and adding a LAN-only filter during an outage fix could cut off off-LAN players or a remote site's device informing. No CiliumNetworkPolicy change is needed, and that was verified rather than assumed: Cilium's `cluster` entity subsumes `remote-node`, confirmed by dumping the crafty endpoint's datapath policy map with the CNP unmodified (`reserved:remote-node` was already an Allow on the game ports). Unlike pihole, whose policy omits `cluster` and must name remote-node explicitly. Adds an L0 whole-repo guard for the class. It fails on the pre-fix tree with exactly these four Services named and passes after. whole-repo scope is necessary: ci/touched-requirements.sh has no arm for apps/* or platform/crafty/*, so the change most likely to reintroduce this would attribute no requirement and be gated by nothing. HONEST LIMIT -- this does NOT end the reported outage. All four ports also have no listening backend: UniFi OS was never installed in the VM (its README flags this as the open blocker) and /crafty/servers is empty, so no Minecraft server exists. Probed directly from the node, bypassing the VIP entirely, 10.0.0.74:8443/:8080/:11443 and 10.0.0.11:25565 all refuse, while the two controls answer. This commit fixes a real latent defect that would have blackholed those VIPs once the applications exist; it cannot make an absent application answer. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85
|
4545611f4d |
merge: Pi-hole becomes the internal LAN DNS authority
The spec and README both claim a LAN split-horizon via Pi-hole; it never existed. Pi-hole's Service was ClusterIP with ports 80 and 4180 only, so LAN clients depended on hand-maintained per-client dnsmasq files — one of which went stale after the dev decommission and cost access to sonarr. Now DNS on udp/53 + tcp/53 on a pinned pool VIP 192.168.2.23, with the zone answered as a wildcard to the Gateway VIP plus longest-match overrides for the two hostname-publishing L4 Services. Declared via Pi-hole v6 FTLCONF env vars, which FTL treats as env-forced and applies at cold start — a real declarative entry point, unlike the Keycloak secret and CNPG grant whose reconcilers exist precisely because no such knob is available. Two defects found on the way. The live pod carried hand-set upstreams marked '### CHANGED' that no manifest knew about, so any restart would have left Pi-hole with upstreams = [] and no recursion at all; FTLCONF_dns_upstreams now declares them. And /etc/pihole is not on the config PVC, which mounts at /data/config, so that volume has been holding nothing. Deviates from the unifi/crafty pattern deliberately: externalTrafficPolicy Cluster, not Local. Cilium's L2 announcer does not filter by which nodes hold a backend, so with Local an announcing node without the pod answers ICMP port unreachable — measured 1-in-3 per protocol, and it does NOT self-heal: a cordoned node kept renewing its lease for over three minutes with LAN DNS dead. The LAN restriction moves to loadBalancerSourceRanges, enforced in eBPF on the VIP, and rate limiting is disabled because SNAT makes the whole LAN look like one client. Proven from the workstation in the worst case — pod and both protocol leases on three different nodes: wildcard, both overrides, recursion over udp and tcp, NXDOMAIN outside the zone, NODATA for AAAA, source-range drop when excluded, and survival of a cross-node reschedule with no intervention. |
||
|
jakman85andClaude Opus 5
|
fd43f332d6 |
feat(pihole): authoritative LAN DNS for the zone on udp/53 + tcp/53 at a pinned VIP
platform/coredns/coredns-custom.yaml has always said its split-horizon is for
PODS and deferred LAN clients to "PiHole split-horizon". PiHole served no DNS:
its Service was ClusterIP with ports 80 and 4180 only. The LAN half of R-DNS-03
that docs/09 and the README's Phase 2 row both claim did not exist, so LAN
clients ran on hand-maintained per-client dnsmasq files - one went stale after
the dev decommission and cost the operator sonarr.home.keithjackson.dev. The
SOHO router that owns 192.168.2.0/24 cannot cover for it either: it strips
RFC1918 answers for public domains as rebind protection and returns NOERROR
with no answer.
EXPOSURE (apps/wave-0/pihole-dns.yaml, hand-written, placed into wave 0 by
render.py's EXTRA_UNITS - the schema has no field for a LoadBalancer Service, a
pinned lbipam VIP, a sharing key, source ranges or UDP, the same reason
apps/vms/unifi and platform/crafty hand-write theirs):
- two Services, udp/53 and tcp/53, sharing one pinned VIP
${PIHOLE_DNS_VIP} = 192.168.2.23 on prod, the next free address in the
.20-.30 pool (pool status IPsUsed 3 of 11; .20 Gateway, .21 unifi, .22
crafty; arping from a LAN host got no reply).
- no external-dns hostname annotation: nothing public is created, and
platform/externaldns/ is untouched.
ANSWERS (catalog/pihole.yaml env - declarative, no reconciler): PiHole v6 takes
every FTL setting as an FTLCONF_ environment variable and treats it as
env-forced, applied ahead of pihole.toml on every cold start and read-only in
the UI. That is a real declarative entry point, which the Keycloak and CNPG
CronJob reconcilers exist precisely because their state lacks, so a reconciler
here would be ceremony. misc.dnsmasq_lines carries a wildcard
address=/<zone>/<GATEWAY_VIP> plus one longer-domain override per L4 service
(unifi, crafty-mc, pihole-dns); dnsmasq resolves overlaps by longest matching
domain, so precedence is structural and a newly onboarded Gateway-fronted
service needs no DNS change. dns.upstreams is declared too - the live pod
carried hand-set upstreams that no manifest knew about and that every restart
discarded.
externalTrafficPolicy Cluster, NOT Local, and loadBalancerSourceRanges for the
LAN restriction. Measured on prod, Cilium 1.20.1: the L2 announcer does not
filter lease candidates by which nodes hold a backend - the two Services landed
on k3s-prod-3 and k3s-prod-2 while the pod ran on k3s-prod-1 - and it keeps
renewing after the pod moves away, so with Local a reschedule leaves LAN DNS
dead indefinitely. Cluster serves the VIP from any node; the LAN restriction
moves to loadBalancerSourceRanges, which Cilium enforces in eBPF on the VIP
before load-balancing (verified by narrowing the range to exclude the test
client and back). The cost is SNAT: forwarded queries reach PiHole from the
ingress node, so PiHole's per-client rate limit is disabled in the catalog or
the whole LAN would count as one client against its 1000-per-60s default.
Proven from a LAN host (192.168.2.132) against 192.168.2.23 with the pod and
both leases on three DIFFERENT nodes: sonarr/grafana/nextcloud/keycloak/apex
-> 192.168.2.20, unifi -> .21, crafty-mc -> .22, pihole-dns -> .23, recursion
(cloudflare.com, github.com, debian.org) intact over both udp and tcp, AAAA
NODATA, NXDOMAIN outside the zone, and still answering after a forced
prod-1 -> prod-2 reschedule.
Co-Authored-By: Claude Opus 5 <[email protected]>
|