home-network-k3s-impl/runbooks/denied-flows.md

# denied-flows — A spike in policy-denied network flows

**Alert:** `HubbleDeniedFlowSpike` (warning, 10m)
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking`
**Expression:** `sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5`

## What this means here

Cilium is dropping more than 5 flows/second on policy. East-west traffic is **default-deny** everywhere
(R-NET-01): a bare workload reaches DNS and nothing else, and every allow is declared per
source → destination → port (R-NET-02). So a denied-flow spike is almost always **a NetworkPolicy
regression** — something legitimate that is no longer allowed — and only rarely a probe.

### Verify the alert can even fire (measured 2026-09-13)

**This alert is currently INERT.** `count(hubble_drop_total)` returned **0 series** on prod: there is no
Hubble/Cilium scrape job among Prometheus's 19 jobs, so Hubble metrics are not being collected and this
rule evaluates against nothing. **Policy drops are unmonitored today** — you will hear about them as
"service X cannot reach service Y", and you diagnose them with `hubble observe`, below. See
`/runbooks/targets-down` for the full list of inert rule groups.

### The regression class that has actually bitten this cluster

**Cilium's CVE-2025-30162 hairpin change (the expensive one).** After the 1.16 → 1.20 upgrade, the
shared Gateway's L7 proxy re-checks a pod's egress policy against the destination's **real backend
identity**. So `toEntities: world` **no longer covers an in-cluster call to a `*.home.keithjackson.dev`
hostname** — the destination resolves to an in-cluster backend, not to `world`. Every app's SSO call
started returning **403 with `server: envoy`**. The fix is an explicit `toEndpoints` allow to the
Keycloak backend, which `catalog/render.py` now emits for every service with
`auth.mode ∈ {oidc, forward-auth}`.

**The durable rule that follows: any new app → app-by-hostname dependency needs its own explicit allow.**
If a newly added or newly SSO-enabled service is 403ing or timing out, this is the first thing to check.

A second recorded instance of the same class: Flux's `source-controller` was denied by
`gitea-allow-gateway-ingress`, whose ingress rule was `fromEntities: [ingress]` only — so an in-cluster
`wget` to Gitea simply timed out. And a matching diagnostic trap: **testing HTTPS from inside a pod is
misleading here**; test from outside with `curl --resolve`.
## Running the commands in this runbook

`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
that hang is not the cluster being down. Every command below runs as:

    infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'

Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
the same thing next month as it does today; one that relies on the default does not, and the failure is
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
interfere.

USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
"the backups are gone" and acting on it turns a false negative into real data loss. Write
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.

Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
agents here: every durable change goes through git (`git push origin main` **is** the deploy).
Where this runbook gives a mutating command it is for an operator, and it says so.

## First commands

See the actual drops — this is the tool, since the metric is not collected:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
     hubble observe --verdict DROPPED --last 200 -o compact'
```

Narrow to a suspected pair:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
     hubble observe --verdict DROPPED --from-namespace nextcloud --last 100 -o compact'
```

What policies apply to the source workload, and is the policy actually enforced?

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod get ciliumnetworkpolicies.cilium.io,networkpolicies.networking.k8s.io -A; echo; \
   kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg endpoint list | head -30'
```

Cilium's own drop counters, as a cross-check:

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg metrics list | grep -i drop'
```

Is the 403 an Envoy/L7 verdict rather than an L3/L4 drop? (The hairpin class above shows up here, not in
`hubble observe`.)

```
infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n <app-ns> logs <pod> --tail=50 | grep -iE "403|envoy|oidc|well-known"'
```

## Likely causes, most likely first

### 1. A newly added or newly SSO-enabled service has no explicit allow to Keycloak
The CVE-2025-30162 hairpin class described above. Signature: HTTP **403 with `server: envoy`** on a call
to `keycloak.${BASE_DOMAIN}` or another `*.home` hostname from inside the cluster, while the same URL
works from outside. Remediation: the allow is **rendered**, not hand-written — set `auth.mode` in
`catalog/<service>.yaml` and re-render so `render.py` emits the `toEndpoints` allow to the keycloak
backend. Do not patch a CiliumNetworkPolicy by hand; it will be overwritten.

### 2. A platform component was moved or a namespace label changed
`fromEntities: [ingress]`-only ingress rules (the Gitea/source-controller case) break any in-cluster
client. Remediation: add the specific allow in the component's `networkpolicy.yaml`, in git.

### 3. A missing dependency declaration in the catalog
`dependencies:` in a ServiceDefinition is what generates app→app allows (R-NET-02). A service talking to
something it never declared is denied by design. The declaration is the fix.

### 4. A scrape target that nothing allows
Prometheus selects **every** ServiceMonitor in the cluster
(`serviceMonitorSelectorNilUsesHelmValues: false`), so a new workload's monitor is scraped immediately
and will be denied until the component allows ingress from `monitoring`. Cross-read
`/runbooks/targets-down`.

### 5. An actual probe or scan
Possible, and it is why the alert is worded as a security signal (docs/07). Distinguish it by the
**source**: a spike from outside the cluster or from an unexpected identity, rather than one named
workload repeatedly failing one destination. Escalate rather than tuning policy.

## When to escalate or stop

- **Do not widen a policy to make drops stop.** Default-deny is a safety-critical requirement here
  (R-NET-01 is flagged `safety_critical: true` in the requirement registry). The correct fix is the
  narrowest explicit allow, authored in git, usually via the catalog so it is rendered consistently.
- **Do not hand-edit a rendered CiliumNetworkPolicy** — it comes from `catalog/render.py` and will be
  reverted on the next reconcile.
- **Escalate if the source is unexpected.** An unexplained spike from outside the cluster is a security
  event, not a configuration bug.
- If drops coincide with a Cilium upgrade, assume the hairpin/identity class above before assuming a
  policy authoring mistake — the upgrade changed the *evaluation*, not the policy.
- Note the diagnostic trap when verifying a fix: probe from **outside** with `curl --resolve`, because
  in-pod HTTPS tests to the platform's own hostnames are misleading.

---
Source of truth: `runbooks/denied-flows.md`. Index: `/runbooks/`
Records: `DECISIONS-LOG.md` (the Cilium CVE-2025-30162 world-hairpin / envoy 403 finding and its
`render.py` fix; source-controller denied by `gitea-allow-gateway-ingress`).