# denied-flows — A spike in policy-denied network flows **Alert:** `HubbleDeniedFlowSpike` (warning, 10m) **File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking` **Expression:** `sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5` ## What this means here Cilium is dropping more than 5 flows/second on policy. East-west traffic is **default-deny** everywhere (R-NET-01): a bare workload reaches DNS and nothing else, and every allow is declared per source → destination → port (R-NET-02). So a denied-flow spike is almost always **a NetworkPolicy regression** — something legitimate that is no longer allowed — and only rarely a probe. ### Verify the alert can even fire (measured 2026-09-13) **This alert is currently INERT.** `count(hubble_drop_total)` returned **0 series** on prod: there is no Hubble/Cilium scrape job among Prometheus's 19 jobs, so Hubble metrics are not being collected and this rule evaluates against nothing. **Policy drops are unmonitored today** — you will hear about them as "service X cannot reach service Y", and you diagnose them with `hubble observe`, below. See `/runbooks/targets-down` for the full list of inert rule groups. ### The regression class that has actually bitten this cluster **Cilium's CVE-2025-30162 hairpin change (the expensive one).** After the 1.16 → 1.20 upgrade, the shared Gateway's L7 proxy re-checks a pod's egress policy against the destination's **real backend identity**. So `toEntities: world` **no longer covers an in-cluster call to a `*.home.keithjackson.dev` hostname** — the destination resolves to an in-cluster backend, not to `world`. Every app's SSO call started returning **403 with `server: envoy`**. The fix is an explicit `toEndpoints` allow to the Keycloak backend, which `catalog/render.py` now emits for every service with `auth.mode ∈ {oidc, forward-auth}`. **The durable rule that follows: any new app → app-by-hostname dependency needs its own explicit allow.** If a newly added or newly SSO-enabled service is 403ing or timing out, this is the first thing to check. A second recorded instance of the same class: Flux's `source-controller` was denied by `gitea-allow-gateway-ingress`, whose ingress rule was `fromEntities: [ingress]` only — so an in-cluster `wget` to Gitea simply timed out. And a matching diagnostic trap: **testing HTTPS from inside a pod is misleading here**; test from outside with `curl --resolve`. ## Running the commands in this runbook `kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as: infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes' Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and interfere. USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`: `longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write `backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`, `settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`. Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for agents here: every durable change goes through git. **`main` is a protected branch — a direct push is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator, and it says so. ## First commands See the actual drops — this is the tool, since the metric is not collected: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \ hubble observe --verdict DROPPED --last 200 -o compact' ``` Narrow to a suspected pair: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \ hubble observe --verdict DROPPED --from-namespace nextcloud --last 100 -o compact' ``` What policies apply to the source workload, and is the policy actually enforced? ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod get ciliumnetworkpolicies.cilium.io,networkpolicies.networking.k8s.io -A; echo; \ kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg endpoint list | head -30' ``` Cilium's own drop counters, as a cross-check: ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg metrics list | grep -i drop' ``` Is the 403 an Envoy/L7 verdict rather than an L3/L4 drop? (The hairpin class above shows up here, not in `hubble observe`.) ``` infra/devcontainer/dev.sh run bash -c \ 'kubectl --context k3s-prod -n logs --tail=50 | grep -iE "403|envoy|oidc|well-known"' ``` ## Likely causes, most likely first ### 1. A newly added or newly SSO-enabled service has no explicit allow to Keycloak The CVE-2025-30162 hairpin class described above. Signature: HTTP **403 with `server: envoy`** on a call to `keycloak.${BASE_DOMAIN}` or another `*.home` hostname from inside the cluster, while the same URL works from outside. Remediation: the allow is **rendered**, not hand-written — set `auth.mode` in `catalog/.yaml` and re-render so `render.py` emits the `toEndpoints` allow to the keycloak backend. Do not patch a CiliumNetworkPolicy by hand; it will be overwritten. ### 2. A platform component was moved or a namespace label changed `fromEntities: [ingress]`-only ingress rules (the Gitea/source-controller case) break any in-cluster client. Remediation: add the specific allow in the component's `networkpolicy.yaml`, in git. ### 3. A missing dependency declaration in the catalog `dependencies:` in a ServiceDefinition is what generates app→app allows (R-NET-02). A service talking to something it never declared is denied by design. The declaration is the fix. ### 4. A scrape target that nothing allows Prometheus selects **every** ServiceMonitor in the cluster (`serviceMonitorSelectorNilUsesHelmValues: false`), so a new workload's monitor is scraped immediately and will be denied until the component allows ingress from `monitoring`. Cross-read `/runbooks/targets-down`. ### 5. An actual probe or scan Possible, and it is why the alert is worded as a security signal (docs/07). Distinguish it by the **source**: a spike from outside the cluster or from an unexpected identity, rather than one named workload repeatedly failing one destination. Escalate rather than tuning policy. ## When to escalate or stop - **Do not widen a policy to make drops stop.** Default-deny is a safety-critical requirement here (R-NET-01 is flagged `safety_critical: true` in the requirement registry). The correct fix is the narrowest explicit allow, authored in git, usually via the catalog so it is rendered consistently. - **Do not hand-edit a rendered CiliumNetworkPolicy** — it comes from `catalog/render.py` and will be reverted on the next reconcile. - **Escalate if the source is unexpected.** An unexplained spike from outside the cluster is a security event, not a configuration bug. - If drops coincide with a Cilium upgrade, assume the hairpin/identity class above before assuming a policy authoring mistake — the upgrade changed the *evaluation*, not the policy. - Note the diagnostic trap when verifying a fix: probe from **outside** with `curl --resolve`, because in-pod HTTPS tests to the platform's own hostnames are misleading. --- Source of truth: `runbooks/denied-flows.md`. Index: `/runbooks/` Records: `DECISIONS-LOG.md` (the Cilium CVE-2025-30162 world-hairpin / envoy 403 finding and its `render.py` fix; source-controller denied by `gitea-allow-gateway-ingress`).