home-network-k3s-impl/runbooks/denied-flows.md

denied-flows — A spike in policy-denied network flows

Alert: HubbleDeniedFlowSpike (warning, 10m) File: platform/monitoring/alerts.yaml, group platform-health.networking Expression: sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5

What this means here

Cilium is dropping more than 5 flows/second on policy. East-west traffic is default-deny everywhere (R-NET-01): a bare workload reaches DNS and nothing else, and every allow is declared per source → destination → port (R-NET-02). So a denied-flow spike is almost always a NetworkPolicy regression — something legitimate that is no longer allowed — and only rarely a probe.

Verify the alert can even fire (measured 2026-09-13)

This alert is currently INERT. count(hubble_drop_total) returned 0 series on prod: there is no Hubble/Cilium scrape job among Prometheus's 19 jobs, so Hubble metrics are not being collected and this rule evaluates against nothing. Policy drops are unmonitored today — you will hear about them as "service X cannot reach service Y", and you diagnose them with hubble observe, below. See /runbooks/targets-down for the full list of inert rule groups.

The regression class that has actually bitten this cluster

Cilium's CVE-2025-30162 hairpin change (the expensive one). After the 1.16 → 1.20 upgrade, the shared Gateway's L7 proxy re-checks a pod's egress policy against the destination's real backend identity. So toEntities: world no longer covers an in-cluster call to a *.home.keithjackson.dev hostname — the destination resolves to an in-cluster backend, not to world. Every app's SSO call started returning 403 with server: envoy. The fix is an explicit toEndpoints allow to the Keycloak backend, which catalog/render.py now emits for every service with auth.mode ∈ {oidc, forward-auth}.

The durable rule that follows: any new app → app-by-hostname dependency needs its own explicit allow. If a newly added or newly SSO-enabled service is 403ing or timing out, this is the first thing to check.

A second recorded instance of the same class: Flux's source-controller was denied by gitea-allow-gateway-ingress, whose ingress rule was fromEntities: [ingress] only — so an in-cluster wget to Gitea simply timed out. And a matching diagnostic trap: testing HTTPS from inside a pod is misleading here; test from outside with curl --resolve.

Running the commands in this runbook

kubectl does not work on the operator workstation. The toolchain lives only in the pinned dev container, and the host's k3s-oidc context hangs on a browser prompt once its token expires — that hang is not the cluster being down. Every command below runs as:

infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass --context k3s-prod explicitly. dev.sh merges BOTH clusters into one kubeconfig and picks a current-context per invocation, and that default has already changed once — it was k3s-dev until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means the same thing next month as it does today; one that relies on the default does not, and the failure is silent (it answers confidently about the other cluster). CLUSTER=prod infra/devcontainer/dev.sh run kubectl ... selects it explicitly, and every invocation prints a banner naming the cluster it is about to talk to — read it. Do not run two dev.sh invocations at once; they share one container and interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind Backup: longhorn.io/v1beta2, postgresql.cnpg.io/v1 and velero.io/v1. The bare short name resolves to Longhorn's, which holds zero objects — so kubectl get backup -A prints "No resources found" on a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as "the backups are gone" and acting on it turns a false negative into real data loss. Write backups.velero.io, backups.postgresql.cnpg.io, clusters.postgresql.cnpg.io, settings.longhorn.io, volumes.longhorn.io, nodes.longhorn.io.

Mutating kubectl (apply, patch, annotate, rollout) and flux reconcile are blocked for agents here: every durable change goes through git (git push origin main is the deploy). Where this runbook gives a mutating command it is for an operator, and it says so.

First commands

See the actual drops — this is the tool, since the metric is not collected:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
     hubble observe --verdict DROPPED --last 200 -o compact'

Narrow to a suspected pair:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
     hubble observe --verdict DROPPED --from-namespace nextcloud --last 100 -o compact'

What policies apply to the source workload, and is the policy actually enforced?

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get ciliumnetworkpolicies.cilium.io,networkpolicies.networking.k8s.io -A; echo; \
   kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg endpoint list | head -30'

Cilium's own drop counters, as a cross-check:

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg metrics list | grep -i drop'

Is the 403 an Envoy/L7 verdict rather than an L3/L4 drop? (The hairpin class above shows up here, not in hubble observe.)

infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n <app-ns> logs <pod> --tail=50 | grep -iE "403|envoy|oidc|well-known"'

Likely causes, most likely first

1. A newly added or newly SSO-enabled service has no explicit allow to Keycloak

The CVE-2025-30162 hairpin class described above. Signature: HTTP 403 with server: envoy on a call to keycloak.${BASE_DOMAIN} or another *.home hostname from inside the cluster, while the same URL works from outside. Remediation: the allow is rendered, not hand-written — set auth.mode in catalog/<service>.yaml and re-render so render.py emits the toEndpoints allow to the keycloak backend. Do not patch a CiliumNetworkPolicy by hand; it will be overwritten.

2. A platform component was moved or a namespace label changed

fromEntities: [ingress]-only ingress rules (the Gitea/source-controller case) break any in-cluster client. Remediation: add the specific allow in the component's networkpolicy.yaml, in git.

3. A missing dependency declaration in the catalog

dependencies: in a ServiceDefinition is what generates app→app allows (R-NET-02). A service talking to something it never declared is denied by design. The declaration is the fix.

4. A scrape target that nothing allows

Prometheus selects every ServiceMonitor in the cluster (serviceMonitorSelectorNilUsesHelmValues: false), so a new workload's monitor is scraped immediately and will be denied until the component allows ingress from monitoring. Cross-read /runbooks/targets-down.

5. An actual probe or scan

Possible, and it is why the alert is worded as a security signal (docs/07). Distinguish it by the source: a spike from outside the cluster or from an unexpected identity, rather than one named workload repeatedly failing one destination. Escalate rather than tuning policy.

When to escalate or stop

• Do not widen a policy to make drops stop. Default-deny is a safety-critical requirement here (R-NET-01 is flagged safety_critical: true in the requirement registry). The correct fix is the narrowest explicit allow, authored in git, usually via the catalog so it is rendered consistently.
• Do not hand-edit a rendered CiliumNetworkPolicy — it comes from catalog/render.py and will be reverted on the next reconcile.
• Escalate if the source is unexpected. An unexplained spike from outside the cluster is a security event, not a configuration bug.
• If drops coincide with a Cilium upgrade, assume the hairpin/identity class above before assuming a policy authoring mistake — the upgrade changed the evaluation, not the policy.
• Note the diagnostic trap when verifying a fix: probe from outside with curl --resolve, because in-pod HTTPS tests to the platform's own hostnames are misleading.

---

Source of truth: runbooks/denied-flows.md. Index: /runbooks/ Records: DECISIONS-LOG.md (the Cilium CVE-2025-30162 world-hairpin / envoy 403 finding and its render.py fix; source-controller denied by gitea-allow-gateway-ingress).