fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable
platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that have never once scraped. Measured on prod 2026-10-04: all six read health=down with lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the configured scrapeTimeout, i.e. packets DROPPED rather than refused. Cause: every app namespace ships a default-deny NetworkPolicy with policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180. Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr, qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object. render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this catalog declares one. It was wired in at the call site and had never emitted a single object on any branch: correct, and entirely latent. It now also fires for the forward-auth gate sidecar's own metrics port, keyed on _needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the sidecar exists. Keying on observability.metrics instead would let metrics: false produce a discovered-but-blocked target: the same "two halves authored apart" failure this function exists to prevent. Rendered here rather than in platform/oauth2-proxy because that tier would have to name app namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering per-app dissolves it. Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring. apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved. Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an object necessarily drifts the goldens that pin the renderer's output. The first gate run caught container-workload-render and egress-class-render as FAIL with exactly this CNP as the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the evidence that the regen is faithful and not a rubber stamp. That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author approving review. The local gate cannot see that — offline it reports approval-policy green (N/A). Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements. Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
1 parent
422834d4e2
commit
f9446be381
10 files changed
+336
-5
No files matched your search
@@ -244,6 +244,27 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): jackett-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: jackett-allow-monitoring-scrape
|
||||
namespace: jackett
|
||||
labels:
|
||||
app.kubernetes.io/name: jackett
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: jackett
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -232,6 +232,27 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: pihole-allow-monitoring-scrape
|
||||
namespace: pihole
|
||||
labels:
|
||||
app.kubernetes.io/name: pihole
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: pihole
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -319,6 +319,27 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): qbittorrent-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: qbittorrent-allow-monitoring-scrape
|
||||
namespace: qbittorrent
|
||||
labels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -347,6 +347,27 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): radarr-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: radarr-allow-monitoring-scrape
|
||||
namespace: radarr
|
||||
labels:
|
||||
app.kubernetes.io/name: radarr
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: radarr
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -345,6 +345,27 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: sonarr-allow-monitoring-scrape
|
||||
namespace: sonarr
|
||||
labels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
+47
-5
@@ -1541,12 +1541,54 @@ def r_monitoring_ingress_policy(s: dict):
|
||||
trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
|
||||
for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.
|
||||
|
||||
Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
|
||||
monitoring namespace: an allow for a port nothing serves would be a standing hole for no
|
||||
Rendered only alongside a port something actually serves, and it opens ONLY that port to ONLY
|
||||
the monitoring namespace: an allow for a port nothing serves would be a standing hole for no
|
||||
benefit.
|
||||
|
||||
TWO INDEPENDENT SOURCES OF A SERVED METRICS PORT, and this used to cover only the first:
|
||||
|
||||
1. `observability.metricsEndpoint` — the APP's own exporter. NOTE, measured 2026-10-04: no
|
||||
ServiceDefinition in this catalog declares one, so before the sidecar clause below this
|
||||
function had NEVER emitted a single object on any branch. It is correct but, so far,
|
||||
entirely latent — do not read its presence as evidence that scrape ingress is solved.
|
||||
|
||||
2. THE forward-auth GATE SIDECAR's own `/metrics` on 44180 (added 2026-10-04). Every
|
||||
`_needs_oauth2_sidecar` app gets `--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}`
|
||||
and a named `metrics` container port from `_oauth2_sidecar_container`, so the port is
|
||||
served whenever the sidecar exists — unconditionally, which is what makes it safe to
|
||||
open without a per-app declaration.
|
||||
|
||||
WHY CLAUSE 2 KEYS ON `_needs_oauth2_sidecar` AND NOT ON `observability.metrics`: it must match
|
||||
the condition the SCRAPE keys on, exactly. platform/oauth2-proxy/podmonitor.yaml selects pods
|
||||
by the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL and nothing else, so a
|
||||
target exists iff the sidecar exists. Keying the allowance on `observability.metrics` instead
|
||||
would let `metrics: false` produce a discovered-but-blocked target — the same "two halves
|
||||
authored apart" failure this function exists to prevent, just moved one field over.
|
||||
|
||||
THIS CLAUSE IS A BUG FIX, and the bug was live for 63.7h. Measured on prod 2026-10-04: all 6
|
||||
sidecar targets across jackett/radarr/sonarr/qbittorrent/pihole read `health=down`,
|
||||
`lastError: context deadline exceeded`, `lastScrapeDuration` exactly 10.0s — the scrapeTimeout,
|
||||
i.e. packets DROPPED, not refused. Each app namespace ships a `default-deny` NetworkPolicy
|
||||
with `policyTypes: [Ingress, Egress]` and NO `ingress` key at all (deny-all-ingress), and the
|
||||
only ingress allowance is `<app>-allow-gateway-ingress`: `fromEntities: [ingress]` -> 4180.
|
||||
Prometheus matched no clause. 5 x `TargetDown` + 6 x `OAuth2ProxyGateUnobserved` had been
|
||||
re-notifying at ~22 emails/day since the PodMonitor landed on 2026-10-02.
|
||||
|
||||
WHY THE POLICY BELONGS HERE AND NOT IN platform/oauth2-proxy: podmonitor.yaml's header records
|
||||
the objection — a policy in that tier would name app namespaces created by the SIBLING
|
||||
apps-wave-0/1 tiers (measured Flux graph: `oauth2-proxy` and `apps-wave-0` both
|
||||
`dependsOn=[platform-config]`, no ordering between them), so it would apply on a running
|
||||
cluster and fail on a cold bootstrap. Rendered here it ships in the same tier that creates the
|
||||
namespace, which dissolves that ordering problem entirely.
|
||||
"""
|
||||
ep = s["observability"].get("metricsEndpoint")
|
||||
if not ep or not s["observability"]["metrics"]:
|
||||
obs = s["observability"]
|
||||
ep = obs.get("metricsEndpoint")
|
||||
ports: list[str] = []
|
||||
if ep and obs["metrics"]:
|
||||
ports.append(str(ep["port"]))
|
||||
if _needs_oauth2_sidecar(s):
|
||||
ports.append(str(OAUTH2_PROXY_METRICS_PORT))
|
||||
if not ports:
|
||||
return None
|
||||
return {
|
||||
"apiVersion": "cilium.io/v2",
|
||||
@@ -1564,7 +1606,7 @@ def r_monitoring_ingress_policy(s: dict):
|
||||
{"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
|
||||
],
|
||||
"toPorts": [
|
||||
{"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
|
||||
{"ports": [{"port": p, "protocol": "TCP"} for p in ports]}
|
||||
],
|
||||
}
|
||||
],
|
||||
|
||||
@@ -171,6 +171,27 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): paas-demo-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: paas-demo-allow-monitoring-scrape
|
||||
namespace: paas-demo
|
||||
labels:
|
||||
app.kubernetes.io/name: paas-demo
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: paas-demo
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): paas-demo-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
+21
@@ -198,6 +198,27 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: sonarr-allow-monitoring-scrape
|
||||
namespace: sonarr
|
||||
labels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -198,6 +198,27 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: sonarr-allow-monitoring-scrape
|
||||
namespace: sonarr
|
||||
labels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
---
|
||||
date: 2026-10-04
|
||||
branch: fix/oauth2-sidecar-metrics-scrape-netpol
|
||||
title: The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`catalog/render.py`'s `r_monitoring_ingress_policy` now emits its CiliumNetworkPolicy for a
|
||||
**second** reason: the forward-auth gate sidecar's own `/metrics` port (44180), keyed on
|
||||
`_needs_oauth2_sidecar(s)`. Previously it fired only from `observability.metricsEndpoint`.
|
||||
|
||||
That matters more than it sounds, because **no ServiceDefinition in this catalog declares a
|
||||
`metricsEndpoint`** (measured: zero grep hits across `catalog/*.yaml`). So the function was wired
|
||||
in at the call site and had **never emitted a single object on any branch** — correct, and
|
||||
entirely latent. Its presence in the tree read as "scrape ingress is handled". It was not.
|
||||
|
||||
Re-rendered `apps/` from that change: five new CNPs, one per forward-auth app
|
||||
(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`), each opening **only** 44180 to **only**
|
||||
`fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring]`.
|
||||
|
||||
The bug being fixed: `platform/oauth2-proxy/podmonitor.yaml` landed 2026-10-02T04:58:44Z and
|
||||
discovered 6 targets that have **never once scraped successfully**. Each app namespace ships a
|
||||
`default-deny` NetworkPolicy with `policyTypes: [Ingress, Egress]` and **no `ingress` key at all**
|
||||
— deny-all-ingress in NetworkPolicy semantics — and the only ingress allowance is
|
||||
`<app>-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180. Prometheus matched no
|
||||
clause, so its packets were dropped.
|
||||
|
||||
Two deliberate choices, both recorded in the function's docstring:
|
||||
|
||||
- **Clause 2 keys on `_needs_oauth2_sidecar`, not on `observability.metrics`.** It has to match
|
||||
the condition the *scrape* keys on: `podmonitor.yaml` selects pods by the
|
||||
`homelab.keithjackson.dev/auth-mode: forward-auth` pod label and nothing else, so a target
|
||||
exists iff the sidecar exists. Keying on `observability.metrics` would let `metrics: false`
|
||||
produce a discovered-but-blocked target — the same "two halves authored apart" failure this
|
||||
function exists to prevent, moved one field over.
|
||||
- **The policy is rendered here, not added to `platform/oauth2-proxy/`.** podmonitor.yaml's own
|
||||
header records why that tier cannot host it: it would name app namespaces created by the
|
||||
*sibling* `apps-wave-0/1` tiers, so it would apply on a running cluster and fail on a cold
|
||||
bootstrap. Rendered per-app it ships in the same tier that creates the namespace.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured on prod 2026-10-04, read-only, via the pinned toolbox:
|
||||
|
||||
- **The cause, not a guess.** All 6 sidecar targets in pool
|
||||
`podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0`: `health=down`,
|
||||
`lastError: context deadline exceeded`, `lastScrapeDuration` exactly `10.0s` — the configured
|
||||
`scrapeTimeout`. Hitting the timeout precisely means packets **dropped, not refused**.
|
||||
- **The listener is fine.** jackett's `oauth2-proxy` container carries
|
||||
`--metrics-address=0.0.0.0:44180` and a named port `metrics: 44180`.
|
||||
- **The deny is universal.** All six namespaces (`jackett radarr sonarr qbittorrent pihole
|
||||
crafty`): `default-deny` with `ingress` key absent, `nIngress=0`; the only ingress CNP allowance
|
||||
is 4180 `fromEntities: [ingress]`. **Zero** namespaces had any `-allow-monitoring-scrape` object.
|
||||
- **Frequency.** Single continuous episode, no flapping: `ALERTS_FOR_STATE` = 229343s (63.7h),
|
||||
firing since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. TargetDown's
|
||||
`for: 10m`. A 15-day `query_range` shows **1 episode** per namespace, not many. Alertmanager:
|
||||
`group_by: [alertname, namespace]`, `repeat_interval: 12h` on `email-warning` → 5 TargetDown
|
||||
groups = 10 emails/day, plus 6 `OAuth2ProxyGateUnobserved` groups = 12/day, ~22/day total.
|
||||
0 silences.
|
||||
- **The re-render is deterministic and the diff is exactly the intended one.** `apps/` is
|
||||
byte-identical to `render.py --all catalog -o apps --wave-layout --no-image-automation`
|
||||
(12/12 SAME before my edit; the 11-file "drift" against a default render is solely the
|
||||
ImageRepository/ImagePolicy pair, which `apps/` is generated without). After the edit:
|
||||
5 files changed, +21 lines each, nothing else touched.
|
||||
- `render.py --selftest`: **PASS**.
|
||||
- **Three L1 goldens needed regenerating, and the diff is +63 lines / -0.** The first gate run
|
||||
caught `container-workload-render` (R-PLAT-02) and `egress-class-render` (R-NET-05/06/07) as
|
||||
FAIL with exactly the new CNP as the drift. Regenerated with the same invocations the checks
|
||||
use (`render.py <fixture>`, no flags). Evidence the regen is faithful rather than a rubber
|
||||
stamp: `default-deny-baseline-render`'s golden came back **byte-identical** (its fixture is not
|
||||
forward-auth), and the other three gained only the one CNP, port 44180, ns monitoring.
|
||||
- Toolbox gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`: **rc 3 — UNDETERMINED,
|
||||
NOT rc 0, and I am not reporting it as green.** Every stage that ran is green
|
||||
(spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1); `l1` went
|
||||
FAIL -> pass once the goldens were regenerated. The gate's own words: "Stages that did not run:
|
||||
gitea-settings" — skipped for want of `GITEA_SETTINGS_TOKEN`, which is an environment gap on
|
||||
every local run and unrelated to this change. L2/L3 conformance is a `[warn]`, not a not-run
|
||||
stage: UNVERIFIED for all 8 touched requirements (R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01
|
||||
R-NET-05 R-NET-06 R-NET-07 R-PLAT-02), no cluster.
|
||||
- `--emit-waves` was **not** run (it re-onboards `vm-demo` + `keycloak`); only the per-service
|
||||
render path was used, so no wave kustomization changed.
|
||||
|
||||
## This PR needs a human approval (sensitive path)
|
||||
|
||||
Regenerating the L1 goldens means this branch touches `conformance/**`, which is in
|
||||
`sensitive_paths` in `infra/gitea/approval-policy.json` ("the tests and the harness; changing them
|
||||
changes what green MEANS"). So `ci/approval-policy.sh` will require an approving review from a
|
||||
non-author, and the local gate **cannot** see that: it reports `approval-policy: green` because
|
||||
offline it evaluates N/A.
|
||||
|
||||
The golden update is not optional — a renderer change that adds an object necessarily drifts the
|
||||
goldens that pin the renderer's output — but it does mean the gate correctly refuses to
|
||||
self-approve this one. **An agent must not clear this with `/approve-override`**, even though the
|
||||
`gitea-admin` account holds `owner`: that is precisely the self-approval the stage exists to
|
||||
prevent. The operator posts the override as a PR *review* of type "Comment", or approves normally.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That the scrape now succeeds.** This is a git-only change and nothing is applied until Flux
|
||||
reconciles `main`. The 5 CNPs have never existed on the cluster, so `up == 1` for those targets
|
||||
is **UNVERIFIED** — it is a prediction from the measured drop signature, not an observation.
|
||||
After merge, confirm `up{container="oauth2-proxy"}` goes to 1 for all 6 targets and that the 11
|
||||
alerts resolve. `send_resolved: true` is set, so expect ~11 resolved emails.
|
||||
- **That the CNP `endpointSelector` matches pihole's pods.** It selects
|
||||
`app.kubernetes.io/name: pihole`, identical to the shape of `pihole-allow-gateway-ingress`
|
||||
which demonstrably works for 4180 — so this is inference from a working sibling, not a
|
||||
measurement of the new object.
|
||||
- **`crafty` is still unobserved and will keep firing `OAuth2ProxyGateUnobserved`.** It is
|
||||
hand-managed under `platform/crafty`, outside the renderer, and has **no `--metrics-address`
|
||||
at all** — so there is no port to open and this change cannot help it. Separate work.
|
||||
- **Whether 44180 is reachable in the Cilium data path once allowed.** Only the policy was
|
||||
reasoned about; no L2 probe was run (CI has `vars.CLUSTER_AVAILABLE` unset, so L2/L3 did not
|
||||
run here either — the gate proves the L0/L1 offline subset only).
|
||||
- **Why `OAuth2ProxyGateUnobserved` started at 2026-10-04 06:44** rather than 10-02 with
|
||||
TargetDown. Alertmanager did not restart (TargetDown's `startsAt` is still 10-02) and the rule's
|
||||
`for` is 1h, so neither explains it. Something changed on the monitoring side that morning which
|
||||
I did not identify. It does not affect this fix.
|
||||
- I did not re-run a port-forward to 44180 to prove the endpoint serves *today*; I confirmed the
|
||||
flag and named port on the live pod spec. The 200/10208-byte measurement is podmonitor.yaml's,
|
||||
from 2026-10-01.
|
||||
Reference in new issue
Block a user