Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
989649e64a |
No files matched your search
@@ -244,27 +244,6 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): jackett-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: jackett-allow-monitoring-scrape
|
||||
namespace: jackett
|
||||
labels:
|
||||
app.kubernetes.io/name: jackett
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: jackett
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -232,27 +232,6 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: pihole-allow-monitoring-scrape
|
||||
namespace: pihole
|
||||
labels:
|
||||
app.kubernetes.io/name: pihole
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: pihole
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -319,27 +319,6 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): qbittorrent-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: qbittorrent-allow-monitoring-scrape
|
||||
namespace: qbittorrent
|
||||
labels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: qbittorrent
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -347,27 +347,6 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): radarr-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: radarr-allow-monitoring-scrape
|
||||
namespace: radarr
|
||||
labels:
|
||||
app.kubernetes.io/name: radarr
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: radarr
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -345,27 +345,6 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: sonarr-allow-monitoring-scrape
|
||||
namespace: sonarr
|
||||
labels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
+5
-47
@@ -1541,54 +1541,12 @@ def r_monitoring_ingress_policy(s: dict):
|
||||
trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
|
||||
for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.
|
||||
|
||||
Rendered only alongside a port something actually serves, and it opens ONLY that port to ONLY
|
||||
the monitoring namespace: an allow for a port nothing serves would be a standing hole for no
|
||||
Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
|
||||
monitoring namespace: an allow for a port nothing serves would be a standing hole for no
|
||||
benefit.
|
||||
|
||||
TWO INDEPENDENT SOURCES OF A SERVED METRICS PORT, and this used to cover only the first:
|
||||
|
||||
1. `observability.metricsEndpoint` — the APP's own exporter. NOTE, measured 2026-10-04: no
|
||||
ServiceDefinition in this catalog declares one, so before the sidecar clause below this
|
||||
function had NEVER emitted a single object on any branch. It is correct but, so far,
|
||||
entirely latent — do not read its presence as evidence that scrape ingress is solved.
|
||||
|
||||
2. THE forward-auth GATE SIDECAR's own `/metrics` on 44180 (added 2026-10-04). Every
|
||||
`_needs_oauth2_sidecar` app gets `--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}`
|
||||
and a named `metrics` container port from `_oauth2_sidecar_container`, so the port is
|
||||
served whenever the sidecar exists — unconditionally, which is what makes it safe to
|
||||
open without a per-app declaration.
|
||||
|
||||
WHY CLAUSE 2 KEYS ON `_needs_oauth2_sidecar` AND NOT ON `observability.metrics`: it must match
|
||||
the condition the SCRAPE keys on, exactly. platform/oauth2-proxy/podmonitor.yaml selects pods
|
||||
by the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL and nothing else, so a
|
||||
target exists iff the sidecar exists. Keying the allowance on `observability.metrics` instead
|
||||
would let `metrics: false` produce a discovered-but-blocked target — the same "two halves
|
||||
authored apart" failure this function exists to prevent, just moved one field over.
|
||||
|
||||
THIS CLAUSE IS A BUG FIX, and the bug was live for 63.7h. Measured on prod 2026-10-04: all 6
|
||||
sidecar targets across jackett/radarr/sonarr/qbittorrent/pihole read `health=down`,
|
||||
`lastError: context deadline exceeded`, `lastScrapeDuration` exactly 10.0s — the scrapeTimeout,
|
||||
i.e. packets DROPPED, not refused. Each app namespace ships a `default-deny` NetworkPolicy
|
||||
with `policyTypes: [Ingress, Egress]` and NO `ingress` key at all (deny-all-ingress), and the
|
||||
only ingress allowance is `<app>-allow-gateway-ingress`: `fromEntities: [ingress]` -> 4180.
|
||||
Prometheus matched no clause. 5 x `TargetDown` + 6 x `OAuth2ProxyGateUnobserved` had been
|
||||
re-notifying at ~22 emails/day since the PodMonitor landed on 2026-10-02.
|
||||
|
||||
WHY THE POLICY BELONGS HERE AND NOT IN platform/oauth2-proxy: podmonitor.yaml's header records
|
||||
the objection — a policy in that tier would name app namespaces created by the SIBLING
|
||||
apps-wave-0/1 tiers (measured Flux graph: `oauth2-proxy` and `apps-wave-0` both
|
||||
`dependsOn=[platform-config]`, no ordering between them), so it would apply on a running
|
||||
cluster and fail on a cold bootstrap. Rendered here it ships in the same tier that creates the
|
||||
namespace, which dissolves that ordering problem entirely.
|
||||
"""
|
||||
obs = s["observability"]
|
||||
ep = obs.get("metricsEndpoint")
|
||||
ports: list[str] = []
|
||||
if ep and obs["metrics"]:
|
||||
ports.append(str(ep["port"]))
|
||||
if _needs_oauth2_sidecar(s):
|
||||
ports.append(str(OAUTH2_PROXY_METRICS_PORT))
|
||||
if not ports:
|
||||
ep = s["observability"].get("metricsEndpoint")
|
||||
if not ep or not s["observability"]["metrics"]:
|
||||
return None
|
||||
return {
|
||||
"apiVersion": "cilium.io/v2",
|
||||
@@ -1606,7 +1564,7 @@ def r_monitoring_ingress_policy(s: dict):
|
||||
{"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
|
||||
],
|
||||
"toPorts": [
|
||||
{"ports": [{"port": p, "protocol": "TCP"} for p in ports]}
|
||||
{"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
|
||||
],
|
||||
}
|
||||
],
|
||||
|
||||
@@ -171,27 +171,6 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): paas-demo-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: paas-demo-allow-monitoring-scrape
|
||||
namespace: paas-demo
|
||||
labels:
|
||||
app.kubernetes.io/name: paas-demo
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: paas-demo
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): paas-demo-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
-21
@@ -198,27 +198,6 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: sonarr-allow-monitoring-scrape
|
||||
namespace: sonarr
|
||||
labels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -198,27 +198,6 @@ spec:
|
||||
- port: '4180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
metadata:
|
||||
name: sonarr-allow-monitoring-scrape
|
||||
namespace: sonarr
|
||||
labels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
spec:
|
||||
endpointSelector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: sonarr
|
||||
ingress:
|
||||
- fromEndpoints:
|
||||
- matchLabels:
|
||||
k8s:io.kubernetes.pod.namespace: monitoring
|
||||
toPorts:
|
||||
- ports:
|
||||
- port: '44180'
|
||||
protocol: TCP
|
||||
---
|
||||
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
|
||||
apiVersion: cilium.io/v2
|
||||
kind: CiliumNetworkPolicy
|
||||
|
||||
@@ -0,0 +1,157 @@
|
||||
---
|
||||
date: 2026-10-04
|
||||
branch: feat/hubble-flow-export-plex
|
||||
title: Record the pre-SNAT client address for Plex via Hubble static flow export, instead of breaking externalTrafficPolicy to get it
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Operator requirement, narrowed explicitly: **this is 100% about log visibility** — knowing which
|
||||
address connected to Plex. Not bandwidth metering, which does not matter here.
|
||||
|
||||
**The problem, measured.** Plex's own log cannot show it. Every LoadBalancer Service in this cluster
|
||||
is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the VIP SNATs the client
|
||||
before forwarding to the backend pod:
|
||||
|
||||
| Measured 2026-10-04 | Value |
|
||||
|---|---|
|
||||
| Node announcing `192.168.2.24` | `k3s-prod-3` (also holds `.20` Gateway, `.23` pihole-dns) |
|
||||
| Node running the Plex pod | `k3s-prod-1` (`10.0.0.135`) |
|
||||
| Sources in Plex's log | **3,214 × `10.0.1.x`**, 105 × `10.0.2.x`, 120 × loopback |
|
||||
| `192.168.x` sources in Plex's log, ever | **0** |
|
||||
| `X-Forwarded-For` in Plex's log | **0** (it serves raw tcp/32400, not HTTP via the Gateway) |
|
||||
| `10.0.1.78` identity (`cilium-dbg bpf ipcache`) | `identity=1` → `reserved:host` = prod-3 itself |
|
||||
|
||||
**Nothing on the router is involved** — the operator correctly pointed out there is no hairpin NAT
|
||||
here. The rewrite is entirely in-cluster, done by Cilium's BPF load-balancer on the lease holder,
|
||||
because the backend pod is on a different node.
|
||||
|
||||
**The address is not lost, only unrecorded.** Cilium observes the flow at ingress, *before* its own
|
||||
translation, so Hubble already holds the true source:
|
||||
|
||||
192.168.2.133 -> 10.0.0.135 (real LAN client -> the Plex pod)
|
||||
10.0.1.78 -> 10.0.0.135 (the same traffic post-SNAT — what Plex sees)
|
||||
|
||||
What was missing is **retention**: the ring buffer was **4095/4095 flows at ~303 flows/s — about
|
||||
thirteen seconds of history** — with hubble metrics disabled and no exporter configured.
|
||||
|
||||
So: enable the static flow exporter, scoped to Plex's port, writing to the agent's stdout.
|
||||
|
||||
- **`platform/cilium/helmrelease.yaml`** (authoritative): `hubble.export.static.enabled: true`,
|
||||
`filePath: stdout`, `allowList` of `{"destination_port":["32400"]}` and
|
||||
`{"source_port":["32400"]}`.
|
||||
- **`platform/cilium/values.yaml`** (reference mirror) and
|
||||
**`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth):
|
||||
the same block, so a bootstrap render does not differ from what Flux reconciles — a mismatch would
|
||||
be a permanent Helm diff and an avoidable agent rollout.
|
||||
|
||||
**`filePath: stdout`, not a file,** because Grafana Alloy already runs as a DaemonSet tailing every
|
||||
pod's logs into Loki (`platform/monitoring/alloy.yaml`). stdout therefore reaches Loki with **no new
|
||||
hostPath mount, no file-rotation handling, and no Alloy change at all**. Upstream documents the
|
||||
value: *"To get Hubble flows directly exported to the logs instead of written to a rotated file,
|
||||
`stdout` can be defined as `hubble-export-file-path`."* Query:
|
||||
|
||||
{namespace="kube-system", app="cilium"} |= "32400"
|
||||
|
||||
**SCOPED BY PORT, NOT BY NAMESPACE — a measured decision, not a preference.** A
|
||||
`--to-namespace plex` query returned **only the post-SNAT flows**; the flow carrying the real client
|
||||
address did not match it, because at the point of observation the destination identity is not yet
|
||||
resolved to the plex pod. A namespace filter would have dropped exactly the record this change
|
||||
exists to capture.
|
||||
|
||||
**WHY NOT FIX THE SNAT INSTEAD — both alternatives rejected on evidence.**
|
||||
- `externalTrafficPolicy: Local` would put the real address in Plex's own log, but cilium 1.20's L2
|
||||
announcer is **explicitly incompatible** with it. Upstream docs for this exact version:
|
||||
*"incompatible with the `externalTrafficPolicy: Local` on services as it may cause service IPs to
|
||||
be announced on nodes without pods causing traffic drops"* and *"L2 announcements isn't currently
|
||||
aware of this setting"*. Live proof it is not theoretical: the VIP is announced by prod-3 while the
|
||||
pod runs on prod-1, so the flip would black-hole tcp/32400 **immediately**. Making it safe means
|
||||
pinning both the pod and the announcement to one node and surrendering Plex's ability to
|
||||
reschedule — a real availability loss to buy a log field. Same trap
|
||||
`platform/crafty/service-l4.yaml` documents from an 8-day Minecraft outage.
|
||||
- **DSR** also preserves the client address, but `routing-mode` here is **vxlan**, not
|
||||
native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO router's stateful
|
||||
NAT on the remote path. Cluster-wide blast radius across all eight LB Services for one service's
|
||||
logging.
|
||||
|
||||
**A correction to the repo's own record.** `decisions/2026-10-03-plex-remote-access-l4.md` states the
|
||||
SNAT puts the client inside `192.168.0.0/16` (the ingress node's LAN address), and I repeated that in
|
||||
the previous entry. **It is wrong**: measured, the SNAT source is `10.0.1.x` — the node's
|
||||
`cilium_host` inside the pod CIDR. This matters, because it means `LanNetworksBandwidth` can never
|
||||
discriminate local from remote while eTP is `Cluster`: `10.0.0.0/8` must be in the LAN set for the
|
||||
in-cluster consumers (Seerr) to count as local, and every external client arrives in that same range.
|
||||
PR #48's LAN definition therefore fixes the Remote badge by making *everything* local — acceptable
|
||||
only because metering is explicitly not a goal here.
|
||||
|
||||
## What I verified
|
||||
|
||||
Live prod, 2026-10-04, read-only.
|
||||
|
||||
- **Every measurement in the table above**, including the VIP-lease-to-node map
|
||||
(`cilium-dbg shell -- db/show l2-announce` on all three agents) and the pod's node.
|
||||
- **Hubble already carries the pre-SNAT source** — the single finding this whole approach rests on:
|
||||
`hubble observe --port 32400` on the lease holder returned `192.168.2.133 -> 10.0.0.135` **and**
|
||||
its return flow, alongside the post-SNAT `10.0.1.78 -> 10.0.0.135`.
|
||||
- **The retention gap is real:** `cilium-dbg status` reported `Hubble: Ok Current/Max Flows:
|
||||
4095/4095 (100.00%), Flows/s: 303.62 Metrics: Disabled`, and the `cilium-config` ConfigMap
|
||||
contains no `hubble-export-*` key at all.
|
||||
- **The namespace-filter trap**, directly: `--to-namespace plex` returned only `10.0.1.78` and
|
||||
`10.0.1.71`; the `192.168.2.133` flow was absent. This is why the allowList is port-scoped.
|
||||
- **All eight LoadBalancer Services are `eTP=Cluster`** (crafty ×2, gateway, pihole-dns ×2, plex,
|
||||
unifi ×2) — so this is a cluster-wide pattern, not a Plex-specific choice.
|
||||
- **The eTP:Local incompatibility is upstream-documented for v1.20 specifically**, quoted above from
|
||||
the docs for the running version — not inferred from the crafty incident.
|
||||
- **`routing-mode: tunnel`** and **`bpf-lb-mode-annotation: false`** in the live `cilium-config`,
|
||||
which is what rules DSR out.
|
||||
- **The chart's value paths and defaults**, read from cilium v1.20.1's own `values.yaml`:
|
||||
`hubble.export.static.{enabled,filePath,fieldMask,fieldAggregate,aggregationInterval,allowList,denyList,fileMaxSizeMb,fileMaxBackups,fileCompress}`,
|
||||
with `filePath` defaulting to `/var/run/cilium/hubble/events.log`. The keys used here exist.
|
||||
- **Alloy's pipeline shape**, read from `platform/monitoring/alloy.yaml`: a DaemonSet with
|
||||
`loki.source.kubernetes "pods"` → `loki.process` → `loki.write` at
|
||||
`http://loki.monitoring.svc.cluster.local:3100`. It already tails kube-system pods, which is why
|
||||
stdout needs no Alloy change.
|
||||
- **All three value files agree** (one `destination_port` occurrence each) and both YAML files parse,
|
||||
with `export.static` resolving to the intended object.
|
||||
- **Toolbox gate**: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
|
||||
`render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two known environmental skips
|
||||
(`gitea-settings` — no local `SETTINGS_API_TOKEN`; `conformance` L2/L3 — `CLUSTER_AVAILABLE`
|
||||
unset). **Not green. Reported as rc 3.**
|
||||
- **No sensitive paths touched** — `platform/cilium/**` and `infra/ansible/**` are not in
|
||||
`infra/gitea/approval-policy.json`, so this needs no approval override, unlike PR #48.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **THE OUTCOME. Nothing has been reconciled and no flow has ever been exported.** `cilium-config`
|
||||
still has no `hubble-export-*` key. Every claim is about inputs.
|
||||
- **Whether `filePath: stdout` is accepted as written.** This is the single largest risk. The value
|
||||
comes from a verbatim upstream doc sentence (*"`stdout` can be defined as
|
||||
`hubble-export-file-path`"*), but I did not confirm the exact notation in cilium 1.20.1's source,
|
||||
and the chart default is a real filesystem path. If the literal string is not special-cased, the
|
||||
agent may create a FILE named `stdout` in its working directory and nothing reaches Loki — a
|
||||
silent no-op, not a crash. **Fallback if so:** leave `filePath` at the chart default and add a
|
||||
`local.file_match` + `loki.source.file` target to Alloy with a hostPath mount for
|
||||
`/var/run/cilium/hubble`.
|
||||
- **Whether the FlowFilter JSON is accepted.** `{"destination_port":["32400"]}` follows the proto
|
||||
(`destination_port` is a repeated string) and the documented shape (`'{"verdict":["DROPPED"]}'`),
|
||||
but upstream gives **no port example** and I did not validate it against the running agent. A
|
||||
malformed filter could be rejected — or, worse, be silently treated as "no match" and export
|
||||
nothing.
|
||||
- **THIS CHANGE ROLLS CILIUM.** Editing the HelmRelease restarts the agent DaemonSet across all
|
||||
three nodes. BPF maps are preserved so established traffic should survive, but prod-3 currently
|
||||
announces the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP, so a lease change during the
|
||||
rollout could briefly interrupt LAN DNS for the whole house. Not measured, and worth doing at a
|
||||
quiet moment.
|
||||
- **Log volume in practice.** Estimated "a few hundred to a few thousand lines a day" from the
|
||||
observed request counts; never measured as flows. Both directions are exported and no `fieldMask`
|
||||
is set, so each record is a full flow object. If it is noisy, narrow with `fieldMask` rather than
|
||||
by dropping a filter.
|
||||
- **Loki's retention for this data**, and whether the `{namespace="kube-system", app="cilium"}` label
|
||||
set actually matches (it depends on the cilium pods carrying
|
||||
`app.kubernetes.io/name`, which Alloy maps to `app` — I did not confirm that label exists on the
|
||||
cilium DaemonSet pods). The query in the comment may need adjusting to `pod=~"cilium-.*"`.
|
||||
- **That the exported flow actually contains the pre-SNAT address.** Hubble's *live* stream does
|
||||
(measured). I am assuming the exporter sees the same flow objects; I did not confirm the exporter
|
||||
is not fed a post-translation view.
|
||||
- **The parity convention has no automated check.** I kept all three files in sync by hand because
|
||||
the comments demand it; nothing in `ci/` or `conformance/` verifies cilium Helm↔Ansible parity, so
|
||||
a future edit can still drift them silently.
|
||||
@@ -1,150 +0,0 @@
|
||||
---
|
||||
date: 2026-10-04
|
||||
branch: fix/oauth2-sidecar-metrics-scrape-netpol
|
||||
title: The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
`catalog/render.py`'s `r_monitoring_ingress_policy` now emits its CiliumNetworkPolicy for a
|
||||
**second** reason: the forward-auth gate sidecar's own `/metrics` port (44180), keyed on
|
||||
`_needs_oauth2_sidecar(s)`. Previously it fired only from `observability.metricsEndpoint`.
|
||||
|
||||
That matters more than it sounds, because **no ServiceDefinition in this catalog declares a
|
||||
`metricsEndpoint`** (measured: zero grep hits across `catalog/*.yaml`). So the function was wired
|
||||
in at the call site and had **never emitted a single object on any branch** — correct, and
|
||||
entirely latent. Its presence in the tree read as "scrape ingress is handled". It was not.
|
||||
|
||||
Re-rendered `apps/` from that change: five new CNPs, one per forward-auth app
|
||||
(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`), each opening **only** 44180 to **only**
|
||||
`fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring]`.
|
||||
|
||||
The bug being fixed: `platform/oauth2-proxy/podmonitor.yaml` landed 2026-10-02T04:58:44Z and
|
||||
discovered 6 targets that have **never once scraped successfully**. Each app namespace ships a
|
||||
`default-deny` NetworkPolicy with `policyTypes: [Ingress, Egress]` and **no `ingress` key at all**
|
||||
— deny-all-ingress in NetworkPolicy semantics — and the only ingress allowance is
|
||||
`<app>-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180. Prometheus matched no
|
||||
clause, so its packets were dropped.
|
||||
|
||||
Two deliberate choices, both recorded in the function's docstring:
|
||||
|
||||
- **Clause 2 keys on `_needs_oauth2_sidecar`, not on `observability.metrics`.** It has to match
|
||||
the condition the *scrape* keys on: `podmonitor.yaml` selects pods by the
|
||||
`homelab.keithjackson.dev/auth-mode: forward-auth` pod label and nothing else, so a target
|
||||
exists iff the sidecar exists. Keying on `observability.metrics` would let `metrics: false`
|
||||
produce a discovered-but-blocked target — the same "two halves authored apart" failure this
|
||||
function exists to prevent, moved one field over.
|
||||
- **The policy is rendered here, not added to `platform/oauth2-proxy/`.** podmonitor.yaml's own
|
||||
header records why that tier cannot host it: it would name app namespaces created by the
|
||||
*sibling* `apps-wave-0/1` tiers, so it would apply on a running cluster and fail on a cold
|
||||
bootstrap. Rendered per-app it ships in the same tier that creates the namespace.
|
||||
|
||||
## What I verified
|
||||
|
||||
Measured on prod 2026-10-04, read-only, via the pinned toolbox:
|
||||
|
||||
- **The cause, not a guess.** All 6 sidecar targets in pool
|
||||
`podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0`: `health=down`,
|
||||
`lastError: context deadline exceeded`, `lastScrapeDuration` exactly `10.0s` — the configured
|
||||
`scrapeTimeout`. Hitting the timeout precisely means packets **dropped, not refused**.
|
||||
- **The listener is fine.** jackett's `oauth2-proxy` container carries
|
||||
`--metrics-address=0.0.0.0:44180` and a named port `metrics: 44180`.
|
||||
- **The deny is universal.** All six namespaces (`jackett radarr sonarr qbittorrent pihole
|
||||
crafty`): `default-deny` with `ingress` key absent, `nIngress=0`; the only ingress CNP allowance
|
||||
is 4180 `fromEntities: [ingress]`. **Zero** namespaces had any `-allow-monitoring-scrape` object.
|
||||
- **Frequency.** Single continuous episode, no flapping: `ALERTS_FOR_STATE` = 229343s (63.7h),
|
||||
firing since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. TargetDown's
|
||||
`for: 10m`. A 15-day `query_range` shows **1 episode** per namespace, not many. Alertmanager:
|
||||
`group_by: [alertname, namespace]`, `repeat_interval: 12h` on `email-warning` → 5 TargetDown
|
||||
groups = 10 emails/day, plus 6 `OAuth2ProxyGateUnobserved` groups = 12/day, ~22/day total.
|
||||
0 silences.
|
||||
- **The re-render is deterministic and the diff is exactly the intended one.** `apps/` is
|
||||
byte-identical to `render.py --all catalog -o apps --wave-layout --no-image-automation`
|
||||
(12/12 SAME before my edit; the 11-file "drift" against a default render is solely the
|
||||
ImageRepository/ImagePolicy pair, which `apps/` is generated without). After the edit:
|
||||
5 files changed, +21 lines each, nothing else touched.
|
||||
- `render.py --selftest`: **PASS**.
|
||||
- **Three L1 goldens needed regenerating, and the diff is +63 lines / -0.** The first gate run
|
||||
caught `container-workload-render` (R-PLAT-02) and `egress-class-render` (R-NET-05/06/07) as
|
||||
FAIL with exactly the new CNP as the drift. Regenerated with the same invocations the checks
|
||||
use (`render.py <fixture>`, no flags). Evidence the regen is faithful rather than a rubber
|
||||
stamp: `default-deny-baseline-render`'s golden came back **byte-identical** (its fixture is not
|
||||
forward-auth), and the other three gained only the one CNP, port 44180, ns monitoring.
|
||||
- Toolbox gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`: **rc 3 — UNDETERMINED,
|
||||
NOT rc 0, and I am not reporting it as green.** Every stage that ran is green
|
||||
(spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1); `l1` went
|
||||
FAIL -> pass once the goldens were regenerated. The gate's own words: "Stages that did not run:
|
||||
gitea-settings" — skipped for want of `GITEA_SETTINGS_TOKEN`, which is an environment gap on
|
||||
every local run and unrelated to this change. L2/L3 conformance is a `[warn]`, not a not-run
|
||||
stage: UNVERIFIED for all 8 touched requirements (R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01
|
||||
R-NET-05 R-NET-06 R-NET-07 R-PLAT-02), no cluster.
|
||||
- `--emit-waves` was **not** run (it re-onboards `vm-demo` + `keycloak`); only the per-service
|
||||
render path was used, so no wave kustomization changed.
|
||||
|
||||
## This PR needs a human approval (sensitive path) — MEASURED, not predicted
|
||||
|
||||
CI run #230 on PR #49, head `f9446be3`: **`MERGE GATE: BLOCKED`, and `approval-policy` is the ONLY
|
||||
failing stage.** Read from the job log, not from the status tick:
|
||||
|
||||
[skip] stage spec-immutability: SKIPPED (not run) <- sibling spec repo absent in CI; expected
|
||||
[ ok ] stage decisions-log: pass
|
||||
[ ok ] stage lint: pass
|
||||
[ ok ] stage l0: pass
|
||||
[ ok ] stage render-golden: pass
|
||||
[ ok ] stage l1: pass
|
||||
[fail] stage approval-policy: FAIL
|
||||
[ ok ] stage gitea-settings: pass <- could NOT run locally; green in CI
|
||||
[skip] stage conformance: SKIPPED (not run)
|
||||
|
||||
sensitive paths touched:
|
||||
conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml
|
||||
conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml
|
||||
conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml
|
||||
|
||||
Exactly the three regenerated goldens, nothing else. `l1` and `render-golden` both pass in CI,
|
||||
which is the independent confirmation that the regenerated goldens are right rather than merely
|
||||
self-consistent with my local run.
|
||||
|
||||
One local-reproduction note worth recording, because it wasted a step: running
|
||||
`PR_NUMBER=49 SETTINGS_API_TOKEN=... ci/approval-policy.sh` with the credential from
|
||||
`git credential fill` reports `could not determine PR author — BLOCKING`. That is an **artifact**,
|
||||
not a finding — `~/.git-credentials` holds a *password*, and `Authorization: token <password>`
|
||||
returns 401 `user does not exist [uid: 0]`. The stage needs a real API token. I nearly reported
|
||||
that 401 as the CI failure; the job log is what settled it.
|
||||
|
||||
Regenerating the L1 goldens means this branch touches `conformance/**`, which is in
|
||||
`sensitive_paths` in `infra/gitea/approval-policy.json` ("the tests and the harness; changing them
|
||||
changes what green MEANS"). So `ci/approval-policy.sh` will require an approving review from a
|
||||
non-author, and the local gate **cannot** see that: it reports `approval-policy: green` because
|
||||
offline it evaluates N/A.
|
||||
|
||||
The golden update is not optional — a renderer change that adds an object necessarily drifts the
|
||||
goldens that pin the renderer's output — but it does mean the gate correctly refuses to
|
||||
self-approve this one. **An agent must not clear this with `/approve-override`**, even though the
|
||||
`gitea-admin` account holds `owner`: that is precisely the self-approval the stage exists to
|
||||
prevent. The operator posts the override as a PR *review* of type "Comment", or approves normally.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **That the scrape now succeeds.** This is a git-only change and nothing is applied until Flux
|
||||
reconciles `main`. The 5 CNPs have never existed on the cluster, so `up == 1` for those targets
|
||||
is **UNVERIFIED** — it is a prediction from the measured drop signature, not an observation.
|
||||
After merge, confirm `up{container="oauth2-proxy"}` goes to 1 for all 6 targets and that the 11
|
||||
alerts resolve. `send_resolved: true` is set, so expect ~11 resolved emails.
|
||||
- **That the CNP `endpointSelector` matches pihole's pods.** It selects
|
||||
`app.kubernetes.io/name: pihole`, identical to the shape of `pihole-allow-gateway-ingress`
|
||||
which demonstrably works for 4180 — so this is inference from a working sibling, not a
|
||||
measurement of the new object.
|
||||
- **`crafty` is still unobserved and will keep firing `OAuth2ProxyGateUnobserved`.** It is
|
||||
hand-managed under `platform/crafty`, outside the renderer, and has **no `--metrics-address`
|
||||
at all** — so there is no port to open and this change cannot help it. Separate work.
|
||||
- **Whether 44180 is reachable in the Cilium data path once allowed.** Only the policy was
|
||||
reasoned about; no L2 probe was run (CI has `vars.CLUSTER_AVAILABLE` unset, so L2/L3 did not
|
||||
run here either — the gate proves the L0/L1 offline subset only).
|
||||
- **Why `OAuth2ProxyGateUnobserved` started at 2026-10-04 06:44** rather than 10-02 with
|
||||
TargetDown. Alertmanager did not restart (TargetDown's `startsAt` is still 10-02) and the rule's
|
||||
`for` is 1h, so neither explains it. Something changed on the monitoring side that morning which
|
||||
I did not identify. It does not affect this fix.
|
||||
- I did not re-run a port-forward to 44180 to prove the endpoint serves *today*; I confirmed the
|
||||
flag and named port on the live pod spec. The 200/10208-byte measurement is podmonitor.yaml's,
|
||||
from 2026-10-01.
|
||||
@@ -45,6 +45,20 @@ hubble:
|
||||
enabled: true
|
||||
ui:
|
||||
enabled: true
|
||||
# Static flow export -> cilium-agent stdout. Kept EQUAL to the Flux values
|
||||
# (platform/cilium/helmrelease.yaml), which carry the full rationale: it records the PRE-SNAT
|
||||
# client address for Plex's raw tcp/32400, which Plex itself cannot log because every
|
||||
# LoadBalancer Service here is externalTrafficPolicy: Cluster and the VIP lease holder SNATs the
|
||||
# client. Present here so a BOOTSTRAP render does not differ from what Flux reconciles -- a
|
||||
# mismatch would show up as a permanent Helm diff and an avoidable agent rollout. Scoped by PORT,
|
||||
# not namespace: the flow carrying the real client address does not match a namespace filter.
|
||||
export:
|
||||
static:
|
||||
enabled: true
|
||||
filePath: stdout
|
||||
allowList:
|
||||
- '{"destination_port":["32400"]}'
|
||||
- '{"source_port":["32400"]}'
|
||||
|
||||
# Ingress VIP announcement on the NAT LAN. IP pool / CiliumL2AnnouncementPolicy
|
||||
# (from the reserved 10.73.0.240-.250 range) is authored by WS-NET in Phase 2.
|
||||
|
||||
@@ -101,6 +101,60 @@ spec:
|
||||
enabled: true
|
||||
ui:
|
||||
enabled: true # exposure gated behind oauth2-proxy forward-auth later (ADR-0014)
|
||||
# --- STATIC FLOW EXPORT: persist the PRE-SNAT client address for Plex (2026-10-04) ---
|
||||
# [parity] MUST equal Ansible: hubble.export.static.{enabled,filePath,allowList}
|
||||
#
|
||||
# WHY THIS EXISTS. Plex's own log cannot show who connected. Every LoadBalancer Service in
|
||||
# this cluster is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the
|
||||
# VIP SNATs the client before forwarding to the backend pod. Measured 2026-10-04: Plex had
|
||||
# logged 3,214 requests from `10.0.1.x` (the lease holder k3s-prod-3's own address, Cilium
|
||||
# identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No X-Forwarded-For
|
||||
# either -- it serves raw tcp/32400, not HTTP through the Gateway.
|
||||
#
|
||||
# THE ADDRESS IS NOT LOST, ONLY UNRECORDED. Cilium observes the flow at ingress, BEFORE its
|
||||
# own translation, so Hubble already has the true source. Measured on the lease holder:
|
||||
# 192.168.2.133 -> 10.0.0.135 (real LAN client -> the Plex pod)
|
||||
# 10.0.1.78 -> 10.0.0.135 (the same traffic, post-SNAT -- what Plex sees)
|
||||
# What was missing is RETENTION: the ring buffer was 4095/4095 flows at ~303 flows/s, i.e.
|
||||
# about THIRTEEN SECONDS of history, with hubble metrics disabled and no exporter configured.
|
||||
#
|
||||
# WHY NOT FIX THE SNAT INSTEAD. `externalTrafficPolicy: Local` would preserve the address in
|
||||
# Plex's own log, but cilium 1.20's L2 announcer is explicitly incompatible with it -- upstream
|
||||
# docs for this exact version: "incompatible with the externalTrafficPolicy: Local on services
|
||||
# as it may cause service IPs to be announced on nodes without pods causing traffic drops",
|
||||
# and "L2 announcements isn't currently aware of this setting". Live proof that this is not
|
||||
# theoretical: the VIP is announced by k3s-prod-3 while the pod runs on k3s-prod-1, so the
|
||||
# flip would black-hole tcp/32400 immediately. Making it safe would mean pinning BOTH the pod
|
||||
# and the announcement to one node and giving up Plex's ability to reschedule -- a real
|
||||
# availability loss to buy a log field. (It is the same trap platform/crafty/service-l4.yaml
|
||||
# documents from an 8-day Minecraft outage.) DSR was also rejected: routing-mode here is
|
||||
# vxlan, not native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO
|
||||
# router's stateful NAT on the remote path.
|
||||
#
|
||||
# `filePath: stdout` RATHER THAN A FILE, deliberately. Grafana Alloy already runs as a
|
||||
# DaemonSet tailing every pod's logs into Loki (platform/monitoring/alloy.yaml), so stdout
|
||||
# reaches Loki with NO new hostPath mount, no file-rotation handling and no Alloy change.
|
||||
# Upstream documents it: "To get Hubble flows directly exported to the logs instead of written
|
||||
# to a rotated file, `stdout` can be defined as `hubble-export-file-path`." fileMaxSizeMb /
|
||||
# fileMaxBackups are therefore left at chart defaults and do not apply.
|
||||
# Query: {namespace="kube-system", app="cilium"} |= "32400"
|
||||
#
|
||||
# SCOPED BY PORT, NOT BY NAMESPACE -- and that is a measured decision, not a preference. A
|
||||
# `--to-namespace plex` query returned ONLY the post-SNAT flows; the flow carrying the real
|
||||
# client address did NOT match it, because at the point of observation the destination
|
||||
# identity is not yet resolved to the plex pod. Filtering by namespace would therefore drop
|
||||
# exactly the record this change exists to capture. Port it is.
|
||||
#
|
||||
# Both directions are listed so the return path is captured too. Volume is small (a few
|
||||
# hundred to a few thousand lines a day); if it ever becomes noisy, narrow it with
|
||||
# `fieldMask` rather than by removing a filter.
|
||||
export:
|
||||
static:
|
||||
enabled: true
|
||||
filePath: stdout
|
||||
allowList:
|
||||
- '{"destination_port":["32400"]}'
|
||||
- '{"source_port":["32400"]}'
|
||||
# --- L2 announcement for the ingress VIP on the flat LAN (docs/09, OQ-1 resolved) ---
|
||||
# [parity] MUST equal Ansible: l2announcements.enabled: true. IP pool +
|
||||
# CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
|
||||
|
||||
@@ -38,6 +38,18 @@ hubble:
|
||||
enabled: true
|
||||
ui:
|
||||
enabled: true
|
||||
# [parity] static flow export -> cilium-agent stdout -> Alloy -> Loki. Records the PRE-SNAT
|
||||
# client address for Plex's raw tcp/32400, which Plex's own log cannot show (every LB Service here
|
||||
# is externalTrafficPolicy: Cluster, so the VIP lease holder SNATs the client). Scoped BY PORT,
|
||||
# not namespace: the flow carrying the real client IP does not match a namespace filter. Full
|
||||
# rationale, measurements and the rejected alternatives are in helmrelease.yaml.
|
||||
export:
|
||||
static:
|
||||
enabled: true
|
||||
filePath: stdout
|
||||
allowList:
|
||||
- '{"destination_port":["32400"]}'
|
||||
- '{"source_port":["32400"]}'
|
||||
# [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
|
||||
l2announcements:
|
||||
enabled: true
|
||||
|
||||
Reference in new issue
Block a user