sync-39-onto-main
24
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
8427b28b0e |
fix(obs): the deploy notification was inhibited before it could send
FluxRolloutInProgress fired correctly on the first real deploy and its mail
was suppressed before Alertmanager dispatched it. One label was wrong.
kube-prometheus-stack ships general.rules -> InfoInhibitor:
group by (namespace) (ALERTS{severity="info"} == 1)
unless on (namespace) group by (namespace)
(ALERTS{alertname!="InfoInhibitor", alertstate="firing",
severity=~"warning|critical"} == 1)
which fires for any namespace holding a firing info alert AND NOTHING ELSE
wrong in it, and the chart's third inhibit_rule (source alertname=InfoInhibitor
-> target severity="info", equal [namespace]) then drops exactly that alert.
For a deploy notification that is the worst possible interaction, because the
suppressed case is the ONLY case that matters: on a healthy rollout of a healthy
cluster flux-system holds no warning or critical alert, so the notification is
inhibited precisely when it is wanted, and would arrive only when something else
in flux-system was already broken. Measured on prod: InfoInhibitor went active
2026-09-30T04:58:48Z, the same moment the alert began firing for 12
Kustomizations, and amtool showed no FluxRolloutInProgress in Alertmanager
either active or --inhibited, with zero send failures.
Upstream's intent is explicit and fair: severity "info" means "only worth
showing next to a real problem". A deploy notice is not that, so it must not
claim that severity. FluxRolloutInProgress now carries `notification: deploy`
and no severity at all, and the route matches on that label — no inhibit_rule
in the chart targets anything but severity values, and omitting severity also
stops InfoInhibitor firing for flux-system rather than merely working around it.
FluxRolloutStuck keeps severity: warning and is unaffected.
Also verified on prod in the process, all previously UNVERIFIED: the 33 metric
series exist exactly as designed, the `revision` label IS populated (so the
lastAppliedRevision/artifact.revision paths are right), the podAnnotations roll
forced a new KSM pod, and the rule both fired for 12 lagging Kustomizations and
then CLEARED on convergence — the completion signal's whole mechanism, on real
data. Routing proven by `amtool config routes test notification=deploy` ->
email-deploy.
Still unverified: no mail observed in the inbox. The 226->227 email counter
increment during the window cannot be attributed (no receiver label, other
alerts active), and I am not claiming it either way. See the handoff entry.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
55b232e2ce |
feat(obs): notify on Flux deploy start and completion
The operator got no mail when Flux began applying a git change and none when it finished. Measured cause: notification-controller v1.4.0 is running and its CRDs serve v1beta3, but there are ZERO Provider and ZERO Alert objects in the cluster — nothing in git ever created any. The obvious fix does not work. Neither Flux event can be filtered into a deploy signal: ReconciliationSucceeded is a heartbeat (101 events in one hour across 16 Kustomizations, message byte-identical whether the apply changed anything), and Progressing is worse (five Kustomizations re-apply the same object on every tick — event counts 2505/2504/2501/2495/2490). gotk_* carries no revision label at all (gotk_reconcile_condition 120 series, gotk_suspend_status 40, none with one). The applied revision is the only signal that moves when and only when git moves, and it lives in object status alone. So export it via the kube-state-metrics custom-resource-state path that already serves the Velero Schedule metrics — kube_flux_kustomization_info/_suspended and kube_flux_gitrepository_info — and ask the question in PromQL: FluxRolloutInProgress (severity: info, no `for:`) fires the moment a non-suspended Kustomization's applied revision is not the revision the GitRepository fetched from main, and CLEARS when all have caught up. Fired mail = a deploy started; RESOLVED mail = it completed. The completion half is send_resolved, not a second rule. FluxRolloutStuck (severity: warning, for: 30m) is the same expression as the escalation for a rollout that started and never converged. Two guards, each with a test whose only job is to fail if the clause is removed: `and on () (count(gitrepository_info) > 0)` — without it a missing exporter makes `unless on (revision) <empty>` remove nothing and all 16 Kustomizations alert forever, the permanently-firing mode this repo shipped once already — and the suspended-Kustomization exclusion, without which a suspended tier holds the rollout open and the "applied" mail never arrives. A dedicated Alertmanager severity=info route + email-deploy receiver keeps deploy notices off the [WARNING] catch-all. podAnnotations carries a roll-counter because the chart sets no checksum over the custom-resource-state ConfigMap, so a config-only edit would leave KSM on the old collector set with the new alerts silently inert. Verified: promtool SUCCESS on 6 scenarios (both guards, plus the convergence test that proves the alert empties itself); lint/l0/render-golden/l1/decisions-log/ spec-immutability green in the pinned toolbox. NOT verified: no mail has been sent — the metrics do not exist until this merges. Full gap list, and the deliberately-cut notification-controller half, in the handoff entry. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
a3f4ddb971 |
merge: feat/egress-gateway-staleness-alert — finished so it can actually pass the gate
Brings the two alerts for the silent egress-gateway failure: VPNEgressSteeringStale
(critical) and VPNEgressUplinkLabelMissing (warning), plus the kube-state-metrics
allowlist entry that makes them non-inert.
WHY THE BRANCH SAT IN THE BACKLOG: it could not pass the merge gate. Three
whole-repo checks failed on it, and each one was a real gap rather than a nuisance:
* alert-runbooks-resolvable — both alerts link to /runbooks/vpn-egress-stale,
which did not exist. An alert firing now would hand the operator a 404.
* alert-rules-unit-tested — alerts.yaml is already a covered file, so new rules
in it must be tested or a "tested" file quietly grows untested rules.
* alert-metrics-have-scrape-targets — see the guard bug below.
WHAT THE ALERTS CATCH. egress-class=vpn pods leave through one gateway node's tun0;
Cilium resolves it to an ifindex and caches it. An OpenVPN reconnect destroys and
re-creates tun0 with a NEW ifindex, and the node label
homelab.keithjackson.dev/egress-uplink=<addr>-<ifindex> is what forces Cilium to
re-resolve. If the label does not follow the tunnel, the cache points at a destroyed
interface and every vpn-class pod is silently dropped — while tun0 stays up and
VPNTunnelDown says nothing. That ran ~80 minutes undetected on the dev gateway.
Verified this is live on prod, not inert: k3s-prod-3 carries egress-gateway=true and
egress-uplink=10.150.192.86-139, and vpn-egress-steering has been in place 14 days.
The label was exported on 0 series before this change, so the allowlist entry is
what switches the alerts on.
runbooks/vpn-egress-stale.md — written and wired into the configMapGenerator, with
the four required operator sections. It leads with proving egress from a pod the
policy selects, because the symptom is "outbound traffic silently disappears", and
it says plainly not to "fix" this by deleting the policy: that does not restore
egress, it sends vpn-class traffic out of the real ISP connection, which is the leak
the class exists to prevent.
platform/monitoring/tests/vpn-egress.test.yaml — four promtool fixtures. 1 and 2 are
the SAME tunnel reconnect with and without the label following it, which isolates the
mechanism the rule actually keys on: a label VALUE change is not a changing sample,
it retires one series and starts another. 3 is the inert case (gateway node, no uplink
label) where the critical rule CANNOT fire, which is why the warning exists.
Two fixes the branch needed that were defects in the checks themselves:
1. alert-metrics-have-scrape-targets asserted the KSM allowlist matched
`nodes=[...egress-gateway]` with a literal closing bracket — i.e. that the list
contained EXACTLY that one label. Adding a second label correctly (egress-uplink,
which these alerts need) therefore FAILED the check while the property it guards
was perfectly satisfied. Loosened to `[,\]]`: in the list, not the whole list. A
guard that breaks when you correctly extend the thing it guards teaches people to
edit the guard, which is how it stops guarding.
2. UNTESTED_ALERTS listed VPNTunnelDown, and the check flagged it as stale because
"asserted on" is a bare word match over the test file — my header COMMENT
mentioning VPNTunnelDown was enough to count as coverage. Rather than delete the
register entry and claim coverage that did not exist, the fixtures now assert
VPNTunnelDown QUIET in all four cases. That required making them faithful: they
had no node_network_receive_bytes_total{device="tun0"} at all, so VPNTunnelDown
would have been firing throughout unnoticed, because promtool only checks the
alertnames you name. A healthy tun0 passes traffic, and that silence IS the claim
these rules exist for.
NOTE FOR LATER, not fixed here: that word-match means any alert named in prose counts
as tested. Tightening it to `alertname:` positions could reveal other prose-only
"coverage", and the register is shrink-only, so the cascade wants doing deliberately
rather than at 01:00 unattended.
Conflicts resolved: kept HEAD's Velero custom-resource-state block and took only the
branch's one-line allowlist change (14 lines down to 1); inserted the branch's
2026-09-12 DECISIONS-LOG entry in date order rather than appending it out of sequence.
Gated: offline l0,l1 --include-scope whole-repo — 24 cases, 0 failures, 0 skips,
0 unwired. The new unit tests pass under the pinned promtool 3.14.0.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
aae63c97a3 |
fix(monitoring): node-exporter extraArgs REPLACES the chart default, so restate it
Verified against the chart (helm show values prometheus-community/prometheus-node-exporter) and against the live DaemonSet's argv on prod: the subchart's extraArgs default is NOT empty — it ships the two filesystem excludes (mount-points-exclude for /dev,/proc,/sys,containerd,docker,kubelet and the fs-types-exclude list). Setting extraArgs to just --collector.mountstats would have silently dropped both, so the filesystem collector would have begun reporting every container overlay and every kubelet bind mount: a large cardinality jump, and a behaviour change in NodeDiskFillingUp, which alerts per mountpoint. Both defaults are now restated verbatim alongside the new collector, with a note to re-reconcile them on a chart upgrade. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
b9f3143f42 |
feat(monitoring): name the CAUSE of node I/O pressure, not just the stall
NodeIOPressureCritical has now fired twice on k3s-prod-1 for opposite causes and
said the same thing both times. On 2026-09-14 02:40-02:45Z it fired at PSI(full)
20.98% — twice the threshold, four times the 2026-09-07 level — while every local
device was idle (0.09-0.27 utilisation) and fast (1.4-4.8 ms per write), NFS
retransmissions were zero and no Longhorn volume was degraded. Establishing that
took eight ad-hoc PromQL queries after the event had already recovered.
Adds the group platform-health.node-io-cause: eight rules, one per cause class,
each naming its cause in its own summary.
NodeDiskDeviceSaturated device saturation >0.85 busy, 10m
NodeDiskWriteLatencyHigh device latency >20ms/write, 10m
NodeDiskWriteLatencyCritical device latency >50ms/write, 5m
NodeDiskReadLatencyHigh device latency >250ms/read, 15m
NodeNFSRetransmissionRateHigh NFS transport >1% retransmitted, 10m
NodeIOPressureNotLocalDisk not local disk PSI>10% with all of the above absent
NodeEtcdRequestsSlow etcd >0.5% of etcd requests over 1s
ContainerIOStallHigh which workload per-container PSI >15%
Every threshold is measured on prod over 24-48h, not guessed, and each rule
records its headroom. The PSI thresholds themselves are deliberately unchanged:
the 21% stall was real, it fired twice in 24h for two real events on the one node
that has a problem, and raising the bar would also have raised it above the 11.36%
at which prod-1 was already losing etcd on 2026-09-07.
Context reaches the operator at page time three ways: the summary states the cause
class; every rule carries a diagnostic_query annotation with the exact companion
PromQL; and node-pressure-dashboard.yaml gains a WHY row (seven panels) plotting
saturation, write/read service time, NFS retransmissions, our own NFS op rate,
the etcd slow-request fraction and per-container PSI side by side. runbooks/node-io.md
gains a 10-row triage table with the real healthy / 2026-09-14 / 2026-09-07 columns,
which turns those eight queries into a two-minute read.
etcd and NFS metric availability, verified on prod rather than assumed:
* etcd SERVER disk metrics do NOT exist. etcd_disk_wal_fsync_duration_seconds and
etcd_disk_backend_commit_duration_seconds are absent (kubeEtcd disabled for k3s);
a query for etcd_disk_*/etcd_server_*/etcd_mvcc_* returns nothing at all.
NodeEtcdRequestsSlow uses the apiserver's client-side view of its own local
member instead, and says in its own description that it is a proxy.
* node_nfs_* DOES exist — the nfs collector is on by default, host-wide via
/proc/net/rpc/nfs, confirmed on all three nodes. It gives op counts and
retransmissions and NO latency, so it can see a dropping export but not a slow
one. The mountstats collector is enabled here to close that gap; NO rule
references node_mountstats_* yet, because the series do not exist until the
HelmRelease rolls out and the collector's visibility through the node-exporter
mount namespace must be verified on prod first. The runbook records the check
and says to revert if it comes back empty.
Proof, per the repo's two shipped alerting failure modes:
* platform/monitoring/tests/node-io-cause.test.yaml — 7 fixtures, all from
measured profiles: healthy (all 8 quiet), 2026-09-14 (NotLocalDisk fires and
names sonarr), 2026-09-07 re-queried at offset 164h (write-latency and etcd
fire, NotLocalDisk correctly silent), rebuild storm, NFS transport, read
latency, and the `unless` semantics.
* 10 mutations of the shipped rules, all killed. The mutation run found a real
gap: the etcd threshold could be changed without failing any test, because it
was only pinned by a promql_expr_test that restated the threshold in the test
file. An alert_rule_test now pins it.
* All 11 referenced metrics verified to return series on prod with
`promtool query instant`, and all 8 expressions verified quiet on today's state.
Also extends the alert-metrics-have-scrape-targets owner table for container_*
(the kubelet ServiceMonitor's cAdvisor path, which this entry already claimed in
its title) and for etcd_request_duration_seconds_* by EXACT NAME, deliberately not
by an etcd_ prefix, so a future rule on the unscraped etcd_disk_* names still fails
the gate.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
8b36242643 |
fix(monitoring): kube-state-metrics was OOMKilled 91 times — 13 rules went intermittent
Working set measured at 111 MiB with a 6h peak of 123.3 MiB against a 128Mi limit, so it died on every spike — exit 137, roughly every 11 minutes, 91 restarts in 16h. The consequence is worse than the restarts. Thirteen alert rules read kube_* series, so each kill made them intermittently unable to fire, which is the quiet version of the inert-alert class found elsewhere today; KubePodNotReady and KubeDeploymentReplicasMismatch were both sitting pending on it. Self-inflicted: enabling customResourceState for the Velero Schedule metrics earlier today added a CRD informer plus the mandatory CRD list/watch, and that marginal cost crossed a limit already sitting about 4 MiB above the working set. Raised to 384Mi, roughly 3x the observed peak, which costs nothing on nodes reporting 18.7-22.3 GiB available. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
7057bbb711 |
fix(monitoring): alert when a Velero schedule has NEVER backed up, or is paused
VeleroNoRecentBackup could not fire for the failure mode that actually happened.
`(time() - velero_backup_last_successful_timestamp{schedule!=""}) > 24*3600` is the
right SHAPE — per-schedule, un-aggregated — but Velero only creates that series once
a backup for the schedule SUCCEEDS, and `time() - <absent series>` yields no series.
So it detects "backed up once, then stopped" and is structurally INERT on "never
backed up". Prod ran with `velero-daily` (the schedule carrying VOLUME DATA) paused
and never-successful: no volume backups at all, nothing alerted, and it was masked
because `velero-daily-resources-only` completed cleanly every 03:00, so every other
backup signal read green while only Kubernetes objects were protected.
A PromQL-only fix is impossible, verified rather than assumed: velero v1.18.1 exports
35 velero_* metric names and every one carrying a `schedule` label is created by a
backup ATTEMPT, not by the Schedule object. A throwaway `paused: true` Schedule on
prod produced ZERO velero_* series, and nothing exposes `spec.paused`. The fact is
not in the TSDB, so the Schedule CRs are exported via kube-state-metrics'
custom-resource-state collector — the path this repo already uses for
kube_node_labels — as kube_velero_schedule_{created,paused,protects_volume_data,info}.
Proven by running KSM v2.20.0 with this exact config against prod's API: the CRD
list/watch rule the chart adds is MANDATORY, not cosmetic — without it KSM emits the
CRS metrics as an empty set, silently.
Three rules, all keyed on PROPERTIES rather than schedule names:
* VeleroScheduleNeverBackedUp (critical) — a Schedule older than 36h with no
successful backup ever. The 36h grace comes from creationTimestamp so a newly
added schedule is not paged before its first run is even due.
* VeleroSchedulePaused (warning) — the direct root-cause signal; a paused backup
schedule must never be silently indefinite.
* VeleroNoVolumeDataBackup (critical) — THE ANTI-MASKING RULE. Fires unless some
schedule with spec.template.defaultVolumesToFsBackup=true succeeded in 36h, so a
healthy resources-only schedule can never make the platform look protected.
Reading the property that DEFINES volume protection (not a name list) matches
platform/velero/volume-policies.yaml and survives renames. `or vector(0)` is
load-bearing: `count(<empty>) == 0` returns no series and would be inert —
absence of the exporter is deliberately the alarm.
Adds the repo's first Prometheus rule-unit-test harness (platform/monitoring/tests/,
wired as the `prom-rules` gate stage). It flattens the COMMITTED PrometheusRule CRs —
never a copy — and pins every rule in the group both ways: quiet on today's healthy
prod state, firing on the reconstructed bad state, with `for:` durations asserted.
Two mutation checks confirm the tests are not vacuous: dropping `or vector(0)` and
breaking the `unless on (schedule)` join each turn the suite red. This is the tool
that makes both of this repo's recurring alert bugs — permanently-firing and
structurally inert — provable offline, which neither yamllint, Kyverno, nor a healthy
cluster can do.
Proof of the fix is by construction; the live metrics cannot show it any more, because
`velero-daily` was unpaused on 2026-09-12 and has since completed 2492/2492 items.
The throwaway probe Schedule and the probe KSM deployment were deleted and the
deletion confirmed; `velero-daily` remains unpaused.
Co-Authored-By: Claude Opus 5 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
16c6234696 |
feat(monitoring): alert when egress steering goes stale, not just when tun0 dies
`VPNTunnelDown` goes green the instant a tun0 exists, which is not the same as vpn-class pods having egress: Cilium caches (egress IP, egress ifindex) at first resolution, so after a reconnect the BPF egress map points at a destroyed interface and the packets are dropped while tun0 looks perfectly healthy. Dev sat in exactly that state for ~80 minutes with nothing alerting. Two rules close it: * VPNEgressSteeringStale (critical, for: 15m) — fires when the live tun0 was re-created more times in the last hour than the gateway node's `egress-uplink` label changed, i.e. Cilium was never told about a reconnect. * VPNEgressUplinkLabelMissing (warning, for: 30m) — the watchdog for the watchdog: a gateway node whose uplink label never reaches kube-state-metrics is a cluster where the rule above is inert. Both gated on a gateway node existing, reusing `VPNTunnelDown`'s `kube_node_labels` guard — a cluster with no gateway cannot be made to fire them. The direct label-vs-map comparison is NOT available and nothing was invented to pretend otherwise: no `cilium_*`/`hubble_*` series exist on either cluster (the cilium-agent exposes no metrics port, there is no ServiceMonitor), and Cilium OSS publishes no series carrying egress-map contents at all — the map is readable only via `cilium-dbg bpf egress list`. PromQL also cannot compare a metric value (the live ifindex) against a number inside a label value, so the level-triggered equality test is not expressible; the same invariant is counted over a window instead. Both limits, and the textfile-collector/black-box-probe routes that would lift them, are written down in the rule comments and the DECISIONS-LOG. `egress-uplink` added to kube-state-metrics.metricLabelsAllowlist in the same kustomization, because without it the first rule has nothing to compare. Verified read-only against LIVE prod (port-forward + /api/v1/query), rendered expr extracted from `kubectl kustomize platform-config`: empty when healthy; empty when evaluated at a timestamp before prod had a gateway node; and FIRING when pointed at a label that genuinely failed to follow prod's real 128 -> 129 tun0 flap earlier today. Nothing on prod was disturbed to produce that. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6a8953e288 |
feat(prod): observability for the two failure modes that went unseen, plus OIDC self-healing
Minimal prod delivery, authored directly on main (= prod's live revision
|
||
|
jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
3e61f817b6 |
kps: raise Prometheus (2->3Gi) + Grafana (512->768Mi) memory limits
Both were pinned at ~98% of their limits (Prometheus 2002/2048Mi, Grafana 499/512Mi) with no headroom for TSDB compaction / dashboard-render spikes — the classic OOM setups. Nodes are 24Gi at 38-44% used (~14Gi free each; the '~22 GiB ceiling' comment predates the 16->24Gi node bump), so the extra headroom is comfortable. Grafana request 256->384Mi, Prometheus request unchanged. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
cbc2301b30 |
docs: reconcile docs/comments with post-upgrade cluster state
Text-only cleanup after the upgrade campaign. Updates stale version strings, prose, and comments to the verified current facts; no Kubernetes behavior or version pins changed except the kubectl toolchain bump (below). Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie, Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB, dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18, Grafana 13.2.0. - Version strings across infra/ and platform/ READMEs + comments - kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew) - Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh) - COMPLETED/historical banners on the three executed plan docs - Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges - Appended dated DECISIONS-LOG entry (2026-08-30) superseding the 2026-08-26 LIVE STATE line Left as revalidation items (not blind-edited): external-snapshotter v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1, conformance ImageVolume feature-gate assertions under 1.36. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
b2b1aa55e2 |
kps: bump Grafana memory limit 256Mi->512Mi (Grafana 12.x OOMKill was root cause of 65->88 upgrade timeout)
Grafana 12.x bundled in kps 88 OOMKilled (exit 137) at 256Mi during grafana-apiserver/unified-storage startup — no fatal log, just SIGKILL. The new Grafana pod never became Ready, so Helm --wait ran the full 20m and rolled back to 65.8.1 every time. 512Mi holds. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
00d895d25f |
kps: 20m timeout + no-rollback remediation for the 65->88 catch-up
The 65->88 multi-major upgrade exceeded the default 5m Helm timeout (full operator/KSM/ node-exporter + Prometheus/Alertmanager StatefulSet rollout), so Flux rolled back to 65.8.1 four times and Stalled, blocking the platform kustomization and everything downstream. CRDs pre-applied server-side at operator v0.93.1. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
037aa7fe9a |
Component catch-up: kube-prometheus-stack 65->88 (po v0.93, k8s 1.36); CRDs pre-applied
Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
eb31d2ef58 |
fix(monitoring): cap Prometheus TSDB by size to stop false absent()/==0 alerts
The 7 falsely-firing platform alerts (KeycloakDown, KeycloakDatabaseDown,
KubeAPIDown, KubeletDown, VPNTunnelDown, KubeVirtControlPlaneDown,
CDINoDefaultStorageClass) share ONE root cause, not per-rule label mismatches:
the Prometheus PVC is 100% full ("no space left on device" on every WAL commit).
Evidence from the live dev cluster:
- /prometheus is 4.8G/5Gi, 0 bytes free; every scrape logs "Scrape commit
failed ... write to WAL: no space left on device". Scrapes still return 200
(targets show healthy, lastScrape=now) but samples are never stored, so the
TSDB head froze at 2026-08-28 13:45 UTC while the pod clock reads ~19:33 UTC.
- Instant queries at "now" therefore return an EMPTY vector, so every
absent()/==0 platform rule trips. Evaluated at the last-good-sample time each
expr is correctly FALSE (keycloak-0/keycloak-db-1 ready=1, apiserver+kubelet
up=1 x3, tun0 series present, kubevirt control-plane expr empty, CDI default
present) — the rules are correct; the monitor was simply blind.
`retention` alone prunes only by AGE: this cluster is 3d2h old with 3d dev
retention, so time-pruning has NEVER run and 3d of block churn (~1.2Gi/day) plus
a WAL that ballooned past 1Gi overran the 5Gi disk. Fix adds a size-based
retention cap (new MONITORING_PROMETHEUS_RETENTION_SIZE knob) kept below each
env's PVC so Prometheus drops oldest blocks before the disk fills, and bumps the
dev PVC 5Gi->8Gi for real WAL/head/compaction headroom.
- platform/monitoring/kube-prometheus-stack.yaml: retentionSize: ${knob}
- clusters/base/cluster-vars.yaml: knob default 8GiB (base 10Gi PVC)
- clusters/dev/kustomization.yaml: PVC 5Gi->8Gi, retentionSize 5GiB
- clusters/prod/kustomization.yaml: retentionSize 45GiB (50Gi PVC)
- platform/monitoring/README.md: knob table + rationale
Deploy note (PM): the PVC is already 100% full, so Prometheus cannot self-prune
until it can write. The 5Gi->8Gi bump must land as a Longhorn online expansion
(operator PVC expansion / manual patch); once free space returns, retentionSize
keeps it healthy. Watchdog left untouched.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85
|
bc0982c213 |
fix(grafana): roll Grafana to load the OIDC client-secret (R-IDP-01)
Grafana's pod predated the grafana-oidc-credentials secret, so generic_oauth started with an empty client_secret (disabled) → /login/generic_oauth didn't redirect to Keycloak. Add a podAnnotation to force one roll; bump on future OIDC secret changes. |
||
|
jakman85
|
ce83c6af3b | WIP keycloak stack (gate probe) | ||
|
jakman85andClaude Opus 4.8
|
7b619f8bb0 |
fix(monitoring): Prometheus memory limit 1Gi -> 2Gi (OOMKilled on compaction)
prometheus-...-0 was OOMKilled (exit 137) every ~4min during TSDB compaction/checkpoint (261 restarts/37h). Raise limit to 2Gi + request to 512Mi; nodes have headroom. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
dfde5292d7 |
feat(phase4): deploy monitoring (kube-prometheus-stack + Loki + Alloy + Alertmanager)
Enable monitoring in platform/kustomization.yaml (Phase 4). Grafana admin + Alertmanager SMTP secrets are provisioned (SOPS) and wired; Longhorn SC is up for Prometheus/Loki PVCs. Set the Alertmanager receiver 'to:' to a valid inbox ([email protected]) so the config loads (REPLACE-ME is not a valid address); operator can change it. NOTE: smtp_from ([email protected]) differs from the Gmail auth user — Gmail may reject/rewrite unless a send-as alias is configured; that only affects email delivery at alert time, not stack health. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
a108d5ab98 |
correction: credential-forward + unified graph; activate SOPS (author-first)
Operator directive: don't diverge architecture to avoid asking for credentials; provision them upfront via SOPS and keep the existing (unified) graph. Roadmap reconciliation: Phase 2 = Network/Storage/TLS/DNS/Ingress, so cert-manager/externaldns/gateway (the credential-gated pieces) ARE Phase 2; monitoring is Phase 4. Earlier A2/rollout scope was inverted by credential-avoidance. Reverts (credential-driven): remove clusters/dev/rollout/ + the $patch:delete's; restore unified base graph (dev = ../base + cluster-vars + SOPS decryption, R-ENV-02 intact); restore monitoring files (Phase 4). Kept non-credential improvements: tofu for_each/rejoin, Kyverno parametrization + webhook excludes. platform/kustomization.yaml now phased (single shared list): Phase-2 set = kyverno, cilium, longhorn, cert-manager, externaldns, gateway; later components commented with phase markers. SOPS activated: .sops.yaml at repo root; bootstrap/sops/secret-templates/*.yaml.example for all secrets (cloudflare x2, grafana, alertmanager-smtp, keycloak, oauth2-proxy, pia); bootstrap/sops/CREDENTIALS.md checklist; pre-wired commented cloudflare-api-token lines in cert-manager + externaldns; filled non-secret ACME email. Agents never touch real secret values (operator encrypts). Validated: kubectl kustomize dev(unified)/prod/platform(Phase-2 only), templates inert, make -C ci gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d9a80e5350 |
phase2-prep: per-component no-secret rollout + component hardening (author-first)
Prep only — nothing deployed; running cluster untouched. Reconciliation graph (operator-approved Option A): author clusters/dev/rollout/ = 4 per-component Flux Kustomizations (kyverno->cilium->longhorn->monitoring, monitoring dependsOn longhorn). Dev overlay $patch:delete's the monolithic base platform/apps and adds ./rollout, so only the no-secret tier reconciles; the secret-blocked components are simply not created until hand-offs arrive. The no-secret tier carries no SOPS content, so it needs no sops-age key. prod is untouched (still builds the monolith). Convergence path coded in (delete rollout/ + the two deletes) — R-ENV-02 divergence flagged for architect review. Component hardening via cluster-vars (prod keeps strict posture): - KYVERNO_VALIDATION_ACTION Enforce->Audit(dev), _WEBHOOK_FAILURE_POLICY Fail->Ignore(dev), _ADMISSION_REPLICAS 3->1(dev); webhook namespaceSelector excludes system+platform namespaces. - MONITORING_STORAGE_CLASS var; GRAFANA_ADMIN_SECRET var (dev ''=chart default); Alertmanager SMTP/secret deferred to Phase-4 with verbatim re-enable markers. Cilium parity verified against the live CNI (no-op adopt). Validated: kubectl kustomize dev/prod/all-components, envsubst->YAML parse, make -C ci gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
cc89b4b830 |
Autonomous session: Phase-2 platform authoring + Flux bootstrap progress
Author-only Phase-2 components (platform suspended; cluster untouched): - platform/longhorn: 3-replica SC + bulk external-mount indirection (R-STATE) - platform/monitoring: kube-prometheus-stack + Loki + Alloy (dev-trimmed) + 8 alert rules (R-OBS) - platform/cilium: parity fix (k8sServiceHost=10.73.0.11 + masquerade) safe to adopt Phase 2 - conformance: partial L2 tests (worker hit session limit; incomplete — see DECISIONS-LOG K2) - bootstrap/gitea: fix rootless-podman perms (host bind-mount -> named volume) Flux installed on the cluster; host Gitea up + repo pushed; GitRepository BLOCKED by host firewalld (VM->10.73.0.1:3000) — needs one operator sudo command (DECISIONS-LOG). Known issue K1: cluster-vars substitution-var case mismatch (fix before un-suspend). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
af4f6c07ae |
T-0.1 Flux GitOps layout + dev/prod overlays + platform component skeletons (R-GITOPS-01/03, R-ENV-02, R-PLAT-01)
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> |