4a3f6dee5998f119fc63fb8418bb89c8e25fc61d
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 renovate-bot
|
b7cb70955c | chore(deps): Update semaphoreui/semaphore Docker tag to v2.19.17 | ||
  jakman85andClaude Opus 5
|
4080065aa7 |
feat(dns): exposure: internal can no longer publish a public record
docs/09 R-DNS-03 and schemas/renderer-contract.md both said an `internal` service renders no public DNS record. Nothing enforced either sentence. ExternalDNS's `gateway-httproute` source admitted EVERY HTTPRoute on the shared Gateway, so six `exposure: internal` services — jackett, pihole, qbittorrent, radarr, semaphore, sonarr — published `<name>.home.keithjackson.dev` into the public authoritative Cloudflare zone answering with the Gateway VIP 192.168.2.20. That published the internal subnet, the Gateway VIP and a full service inventory to anyone who could guess a label, and made each of those names a DNS-rebind primitive — which is why the SOHO router strips RFC1918 answers and LAN clients need per-client dnsmasq overrides to reach their own services. THE MECHANISM, established against the DEPLOYED version rather than from memory. The live prod Deployment runs registry.k8s.io/external-dns/external-dns:v0.21.0. In v0.21.0, source/annotations/processors.go IsControllerMismatch() skips a source object whose `external-dns.alpha.kubernetes.io/controller` annotation is present and != its own `dns-controller`, and source/gateway.go applies that check inside Endpoints() BEFORE any hostname is resolved; source/service.go applies the identical check, covering the second configured source. catalog/render.py now emits that annotation on every HTTPRoute it renders for an `internal` service, derived from `exposure` alone — no per-service list — so a service that omits `exposure` inherits the internal default and is unpublishable too. Verified LIVE on prod on a throwaway hostname with no existing record at risk: the annotated route published nothing, and adding the annotation to a route that already had a record produced action=DELETE for both its A record and its `a-` TXT registry entry one ~60s reconcile later, moving nothing else in the zone. WHY PER-RESOURCE AND NOT A CLUSTER-WIDE OPT-IN FILTER. `labelFilter` is the stronger default posture and v0.21.0 supports it for both configured sources, but nine publishing objects here are hand-authored platform/apps manifests with no `exposure` field, seven of which hold live records (auth, crafty, crafty-mc, grafana, headlamp, the apex, unifi). An opt-in filter deletes all seven at once, and whether each should be publicly resolvable is an operator decision per name. Per-resource exclusion has exactly the blast radius of the defect; the seven are enumerated in a shrink-only register so the unadjudicated set cannot grow silently, and the filter stays available as hardening afterwards. `policy: sync`, `txtOwnerId` and `domainFilters` are UNTOUCHED. Per-resource exclusion only ever removes an object from this instance's view, so nothing here makes prod capable of deleting a record it does not own. WHAT THIS DELETES, and it is not reversible by a revert alone: 12 records — the A record and the `a-`-prefixed TXT registry entry for each of jackett, pihole, qbittorrent, radarr, semaphore and sonarr. The zone has no `*.home.keithjackson.dev` wildcard and `*.keithjackson.dev` does not match two labels, so each becomes a public NXDOMAIN with no fallback. The complete pre-state is captured at the authority and committed as a recovery baseline (.standup-dns/dns-20260914-pre-externaldns-scope.txt), with the restore procedure for both halves of a record. MERGE ORDER IS NOT OPTIONAL: feat/pihole-lan-dns-authority must land and be verified first. Removing these records while the LAN still depends on them, with the router stripping RFC1918 and no internal authority yet, breaks name resolution outright. The five `exposure: public` services are LEFT UNCHANGED and still answer with the Gateway VIP, so they remain publicly unreachable. That is two open operator decisions, both written up in platform/externaldns/helmrelease.yaml: what address they should answer with (136.51.115.252 is the legacy non-k3s stack serving a placeholder cert, not established as correct for this cluster), and how to separate them from the non-public names that share the Gateway — because in v0.21.0 the target annotation is read only from the GATEWAY and a ListenerSet, never from an HTTPRoute, so a per-service override is not expressible while they all attach to `shared`. No knob is added for it yet: one that cannot take effect invites someone to set it and expect one. Guard: conformance/tests/l0-static/internal-services-publish-no-public-dns, a whole-repo-scoped L0 static check (no ci/l0.sh stage — the whole-repo-invariants-wired meta-test forbids one). It asserts both directions (present on internal, absent on public), the `service`-source half, that the renderer DERIVES the annotation from `exposure` and cannot emit external-dns's own claiming value, proves the shipped renderer end-to-end on internal/public/omitted-field fixtures, and ratchets the non-catalog register. Proven to fail on the pre-fix tree (9 of 53 assertions, naming all six services) and pass after (56/56). Schema 1.6.0 -> 1.7.0: no field changes, a mapping change to `exposure`. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
cf7d97e546 |
fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.
THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.
FIXED IN THE GENERATOR, not in twelve rendered files:
* `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
service's own workload object —
`kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
`kube_statefulset_status_replicas_ready` for a StatefulSet, and
`absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
exporter and no new target. The `or absent(...)` half is load-bearing: the signal
DISAPPEARING now fires the alert instead of silencing it.
* schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
scrape together: the named Service port, the monitor endpoint, and the
monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
kube-state-metrics liveness.
* schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
would have made its `absent()` half fire FOREVER — the other failure mode this repo has
shipped.
* renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.
PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
* quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
keycloak-db StatefulSet ready in the same namespace).
* mutation-checked: six mutations, each turning its suite red — reverting plex to
`up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
inverting plex's `== 0` to `!= 0`. Control run green.
* live on prod with `promtool query instant`: all ten deployed app Deployments export
`kube_deployment_status_replicas_available`,
`kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
`kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
— its rule is not applied either.
DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.
GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.
Also fixed, found on the way:
* platform/monitoring/tests/run.sh discovered the SUITE convention only inside
platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
was never run and still looked like coverage. Now discovered tree-wide; a misplaced
*.test.yaml fails loudly instead of vanishing.
* KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
* renderer-contract.md claimed the default-deny baseline carried a standing monitoring
scrape allow. It never did; the claim is corrected and the allow is now real, per declared
metrics endpoint and scoped to that one port.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
6548f9a89c |
fix(semaphore): per-environment OIDC URLs — same latent break as Headlamp
Semaphore's whole SEMAPHORE_OIDC_PROVIDERS JSON blob lived inside the SOPS-encrypted
semaphore-bootstrap Secret (key `oidc-providers-json`) because it CONTAINS the confidential
client secret. That buried two environment-specific, NON-SECRET URLs in ciphertext —
provider_url (the realm issuer) and redirect_url — where Flux's ${BASE_DOMAIN} postBuild
substitution cannot reach. Both were frozen at `home.keithjackson.dev`, so prod's Semaphore
was pointed at dev's Keycloak and dev's own redirect URI: identical root cause to the
Headlamp login timeout, not yet reported only because nobody had tried prod SSO.
FIX: split the blob. The URLs are public and now live as a plaintext literal in
catalog/semaphore.yaml (rendered to the apps/wave-1 golden), ${BASE_DOMAIN}-substituted like
every other hostname. Only the client secret stays encrypted, in its own key
`oidc-client-secret`, and the KUBELET splices it back via dependent-env-var expansion:
$(SEMAPHORE_OIDC_CLIENT_SECRET) resolves against the secretKeyRef env var declared
immediately above it (order is load-bearing — the kubelet resolves env in list order and
leaves a forward reference as literal text).
Verified: the new key holds the same value as keycloak-realm-clients/semaphore-client-secret
(the mirror invariant), Semaphore v2.19.10's util.OidcProvider consumes client_secret as
before, and `flux envsubst --strict` leaves $(...) untouched while resolving ${BASE_DOMAIN} —
dev renders keycloak.home.keithjackson.dev, prod keycloak.prod.home.keithjackson.dev.
apps/wave-1/semaphore.yaml is a render golden; regenerated with the canonical
`catalog/render.py --all catalog -o apps --wave-layout --no-image-automation`.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
68690e5019 |
fix(net): allow SSO apps egress to keycloak backend (Cilium 1.20 CVE-2025-30162)
App SSO broke cluster-wide after the Cilium 1.16.5->1.20.1 upgrade: the CVE-2025-30162 fix (in
>=1.16.8/1.17.2) makes the shared Gateway's L7 proxy re-evaluate a pod's egress policy against
Keycloak's REAL backend cluster identity (enforce_policy_on_l7lb) when it hairpins to keycloak.home.
'toEntities: world' matches the Gateway VIP at L3/L4 but not the resolved keycloak identity at L7, so
OIDC discovery/token 403'd ('Access denied') for every app with a NetworkPolicy (gitea/nextcloud/
semaphore/headlamp/crafty...); Grafana survived only because monitoring has no NetworkPolicy.
Fix: explicit toEndpoints egress allow to the keycloak backend for SSO apps.
- catalog/render.py: emit it for auth.mode in {oidc,forward-auth}; re-rendered 8 app goldens.
- platform/headlamp + platform/crafty: hand-written policies get the same allow.
Split-horizon design unchanged. Verified live: headlamp-ns -> keycloak discovery 403 -> 200.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
f163026954 |
feat(nextcloud,semaphore): wire native Keycloak OIDC SSO (break-glass admins retained)
Extends the SSO work to the two remaining `auth: oidc` apps still on local-admin
logins, each as a native confidential client of the Keycloak `homelab` realm with
its local admin kept as break-glass (docs/15 availability caveat; R-IDP-02/06).
Realm (source of truth): add the `nextcloud` and `semaphore` confidential clients
to platform/keycloak/keycloak-realm-import.yaml (modeled on grafana/gitea) + their
NEXTCLOUD_CLIENT_SECRET / SEMAPHORE_CLIENT_SECRET placeholders. As before, a
KeycloakRealmImport no-ops on an already-existing realm, so both live clients were
ALSO created via kcadm (nextcloud e95049ed-ca10-4085-97d9-4bbef495fe61, semaphore
1835bfe9-b125-4026-8f69-33557519cf4a); the YAML keeps git authoritative.
Secrets (SOPS/age): add nextcloud-client-secret + semaphore-client-secret to
keycloak-realm-clients.sops.yaml (keycloak ns). Mirror the nextcloud secret as
oidc-client-secret into nextcloud-bootstrap (nextcloud ns). For semaphore, the
whole SEMAPHORE_OIDC_PROVIDERS JSON (which embeds the client secret) is held
encrypted as oidc-providers-json in semaphore-bootstrap (semaphore ns). All values
verified == the live clients; existing keys untouched; no plaintext in git.
App config (catalog source + re-render):
- nextcloud: add NEXTCLOUD_OIDC_CLIENT_SECRET env + an idempotent postStart hook
that installs/enables user_oidc and runs `occ user_oidc:provider keycloak`
(create-or-update; secret via --clientsecret-env so it never hits argv; occ as
www-data; group provisioning on the `groups` claim). Syntax verified against the
running user_oidc 8.11.0 on Nextcloud 31.0.14; su-to-www-data env passthrough
verified. Missing-secret/not-installed guards prevent crashloops.
- semaphore: add SEMAPHORE_OIDC_PROVIDERS env sourced from the encrypted
oidc-providers-json (provider_url=realm issuer/auto-discovery, redirect
/api/auth/oidc/keycloak/redirect). Mechanism verified against semaphore v2.19.10
source (util ConfigType oidc_providers env tag + OidcProvider struct + router
redirect path); no kubelet service-link env collision.
apps/wave-1/{nextcloud,semaphore}.yaml re-rendered via
`render.py --no-image-automation` (diffs are env/lifecycle-only).
Verified: realm import Done=True/HasErrors=False and platform-config
Ready=True/Healthy=True (unchanged); edited realm YAML passes server dry-run; live
homelab clients now include gitea/nextcloud/semaphore; static gate (lint/l0/l1)
green. Browser SSO login for each still needs a human check.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
36a1a4e1f9 |
refactor(apps): split monolithic apps Kustomization into dependency waves
Replaces the single `apps` Flux Kustomization with dependency-ordered waves
(apps-wave-0..2) so a crashlooping workload isolates to its wave + downstream
instead of stalling the entire app tier. Parallel within a wave, ordered across.
- catalog/render.py: compute waves by longest dependency-path depth over the
ServiceDefinition `dependencies:` edges (a `<svc>-db` is a leaf, so its
consumer lands strictly higher — the "DB below consumer" rule falls out of
longest-path, no special case). `--emit-waves` generates the per-wave
kustomizations + Flux CRs; `--wave-layout` places renders into the right dir.
wave 0: pihole gitea jackett plex qbittorrent nextcloud-db semaphore-db
wave 1: sonarr radarr nextcloud semaphore gitea-actions
wave 2: seerr
- apps/*.yaml -> apps/wave-{0,1,2}/ (byte-identical moves; SOPS files intact).
- clusters/base: apps.yaml removed; apps-wave-{0,1,2}.yaml added (dependsOn
chain: platform-config -> wave-0 -> wave-1 -> wave-2, prune+wait each).
- clusters/dev: one SOPS decryption patch per wave (load-bearing).
- DESIGN-waves.md: wave table, topology, cutover runbook.
Cutover: preceded by disabling prune on `apps` (
|