jakman85andClaude Opus 5
8e017fad91
main became a protected branch on 2026-09-29, so the direct push that 37 of 39 runbooks instructed is REJECTED. The wording was merely stale before; it is now actively wrong, and it is the line an operator reads mid-incident. Every occurrence now describes the real path: push a branch, open a PR, merge once the gate is green -- that merge is the deploy. 35 files shared one identical block; flux-reconcile.md, README.md and plex-claim.md each said it their own way and were corrected individually. plex-claim.md is rewritten around the delivery loop and records that THE SERVER IS NOW CLAIMED (PlexOnlineToken present in /config), so PLEX_CLAIM is a permanent no-op and the page only applies after a /config wipe. Adds infra/plex/claim-ship.sh, which drives branch -> commit -> push -> PR -> gate -> merge -> reconcile -> pod roll -> verdict with a stopwatch. Three steps are easy to lose under time pressure and two are SILENT when skipped: --with-source (without it the Kustomization applies the previously-fetched revision), the mandatory rollout restart (only the ciphertext changes, so the pod template is byte-identical and nothing restarts itself), and the feat/ branch prefix (act_runner capacity is 1, and a prefix inside gate.yml's push trigger fires a redundant run that serialises ahead of the PR run). Drops the "~4-minute TTL" the page asserted in four places. It was uncited, and upstream pms-docker documents no expiry at all. The countdown on plex.tv/claim is the authority. Measured loop: ~2m45s. DECISIONS-LOG carries the PM handoff: both halves of the 2026-08-31 CI claim were false, the stale-.runner root cause, the four-deep Flux dependency cascade, and the correction that a skipped Gitea job reports success rather than hanging. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
157 lines
8.8 KiB
Markdown
157 lines
8.8 KiB
Markdown
# denied-flows — A spike in policy-denied network flows
|
|
|
|
**Alert:** `HubbleDeniedFlowSpike` (warning, 10m)
|
|
**File:** `platform/monitoring/alerts.yaml`, group `platform-health.networking`
|
|
**Expression:** `sum(rate(hubble_drop_total{reason="POLICY_DENIED"}[5m])) > 5`
|
|
|
|
## What this means here
|
|
|
|
Cilium is dropping more than 5 flows/second on policy. East-west traffic is **default-deny** everywhere
|
|
(R-NET-01): a bare workload reaches DNS and nothing else, and every allow is declared per
|
|
source → destination → port (R-NET-02). So a denied-flow spike is almost always **a NetworkPolicy
|
|
regression** — something legitimate that is no longer allowed — and only rarely a probe.
|
|
|
|
### Verify the alert can even fire (measured 2026-09-13)
|
|
|
|
**This alert is currently INERT.** `count(hubble_drop_total)` returned **0 series** on prod: there is no
|
|
Hubble/Cilium scrape job among Prometheus's 19 jobs, so Hubble metrics are not being collected and this
|
|
rule evaluates against nothing. **Policy drops are unmonitored today** — you will hear about them as
|
|
"service X cannot reach service Y", and you diagnose them with `hubble observe`, below. See
|
|
`/runbooks/targets-down` for the full list of inert rule groups.
|
|
|
|
### The regression class that has actually bitten this cluster
|
|
|
|
**Cilium's CVE-2025-30162 hairpin change (the expensive one).** After the 1.16 → 1.20 upgrade, the
|
|
shared Gateway's L7 proxy re-checks a pod's egress policy against the destination's **real backend
|
|
identity**. So `toEntities: world` **no longer covers an in-cluster call to a `*.home.keithjackson.dev`
|
|
hostname** — the destination resolves to an in-cluster backend, not to `world`. Every app's SSO call
|
|
started returning **403 with `server: envoy`**. The fix is an explicit `toEndpoints` allow to the
|
|
Keycloak backend, which `catalog/render.py` now emits for every service with
|
|
`auth.mode ∈ {oidc, forward-auth}`.
|
|
|
|
**The durable rule that follows: any new app → app-by-hostname dependency needs its own explicit allow.**
|
|
If a newly added or newly SSO-enabled service is 403ing or timing out, this is the first thing to check.
|
|
|
|
A second recorded instance of the same class: Flux's `source-controller` was denied by
|
|
`gitea-allow-gateway-ingress`, whose ingress rule was `fromEntities: [ingress]` only — so an in-cluster
|
|
`wget` to Gitea simply timed out. And a matching diagnostic trap: **testing HTTPS from inside a pod is
|
|
misleading here**; test from outside with `curl --resolve`.
|
|
## Running the commands in this runbook
|
|
|
|
`kubectl` does not work on the operator workstation. The toolchain lives only in the pinned dev
|
|
container, and the host's `k3s-oidc` context hangs on a browser prompt once its token expires —
|
|
that hang is not the cluster being down. Every command below runs as:
|
|
|
|
infra/devcontainer/dev.sh run bash -c 'kubectl --context k3s-prod get nodes'
|
|
|
|
Always pass `--context k3s-prod` explicitly. `dev.sh` merges BOTH clusters into one kubeconfig and
|
|
picks a current-context per invocation, and **that default has already changed once** — it was k3s-dev
|
|
until dev was decommissioned on 2026-09-13, and is k3s-prod now. A command that names its context means
|
|
the same thing next month as it does today; one that relies on the default does not, and the failure is
|
|
silent (it answers confidently about the other cluster). `CLUSTER=prod infra/devcontainer/dev.sh run
|
|
kubectl ...` selects it explicitly, and every invocation prints a banner naming the cluster it is about
|
|
to talk to — read it. Do not run two `dev.sh` invocations at once; they share one container and
|
|
interfere.
|
|
|
|
USE FULLY-QUALIFIED RESOURCE NAMES, ALWAYS. Three CRDs on this cluster register the kind `Backup`:
|
|
`longhorn.io/v1beta2`, `postgresql.cnpg.io/v1` and `velero.io/v1`. The bare short name resolves to
|
|
Longhorn's, which holds zero objects — so `kubectl get backup -A` prints **"No resources found"** on
|
|
a cluster holding 6 Velero backups and 27 CNPG backups. Verified on prod 2026-09-13. Reading that as
|
|
"the backups are gone" and acting on it turns a false negative into real data loss. Write
|
|
`backups.velero.io`, `backups.postgresql.cnpg.io`, `clusters.postgresql.cnpg.io`,
|
|
`settings.longhorn.io`, `volumes.longhorn.io`, `nodes.longhorn.io`.
|
|
|
|
Mutating kubectl (`apply`, `patch`, `annotate`, `rollout`) and `flux reconcile` are blocked for
|
|
agents here: every durable change goes through git. **`main` is a protected branch — a direct push
|
|
is rejected.** A change lands by pushing a branch, opening a PR, and merging it once the gate is
|
|
green; that merge is the deploy. Where this runbook gives a mutating command it is for an operator,
|
|
and it says so.
|
|
|
|
## First commands
|
|
|
|
See the actual drops — this is the tool, since the metric is not collected:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
|
|
hubble observe --verdict DROPPED --last 200 -o compact'
|
|
```
|
|
|
|
Narrow to a suspected pair:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
|
|
hubble observe --verdict DROPPED --from-namespace nextcloud --last 100 -o compact'
|
|
```
|
|
|
|
What policies apply to the source workload, and is the policy actually enforced?
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod get ciliumnetworkpolicies.cilium.io,networkpolicies.networking.k8s.io -A; echo; \
|
|
kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg endpoint list | head -30'
|
|
```
|
|
|
|
Cilium's own drop counters, as a cross-check:
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- cilium-dbg metrics list | grep -i drop'
|
|
```
|
|
|
|
Is the 403 an Envoy/L7 verdict rather than an L3/L4 drop? (The hairpin class above shows up here, not in
|
|
`hubble observe`.)
|
|
|
|
```
|
|
infra/devcontainer/dev.sh run bash -c \
|
|
'kubectl --context k3s-prod -n <app-ns> logs <pod> --tail=50 | grep -iE "403|envoy|oidc|well-known"'
|
|
```
|
|
|
|
## Likely causes, most likely first
|
|
|
|
### 1. A newly added or newly SSO-enabled service has no explicit allow to Keycloak
|
|
The CVE-2025-30162 hairpin class described above. Signature: HTTP **403 with `server: envoy`** on a call
|
|
to `keycloak.${BASE_DOMAIN}` or another `*.home` hostname from inside the cluster, while the same URL
|
|
works from outside. Remediation: the allow is **rendered**, not hand-written — set `auth.mode` in
|
|
`catalog/<service>.yaml` and re-render so `render.py` emits the `toEndpoints` allow to the keycloak
|
|
backend. Do not patch a CiliumNetworkPolicy by hand; it will be overwritten.
|
|
|
|
### 2. A platform component was moved or a namespace label changed
|
|
`fromEntities: [ingress]`-only ingress rules (the Gitea/source-controller case) break any in-cluster
|
|
client. Remediation: add the specific allow in the component's `networkpolicy.yaml`, in git.
|
|
|
|
### 3. A missing dependency declaration in the catalog
|
|
`dependencies:` in a ServiceDefinition is what generates app→app allows (R-NET-02). A service talking to
|
|
something it never declared is denied by design. The declaration is the fix.
|
|
|
|
### 4. A scrape target that nothing allows
|
|
Prometheus selects **every** ServiceMonitor in the cluster
|
|
(`serviceMonitorSelectorNilUsesHelmValues: false`), so a new workload's monitor is scraped immediately
|
|
and will be denied until the component allows ingress from `monitoring`. Cross-read
|
|
`/runbooks/targets-down`.
|
|
|
|
### 5. An actual probe or scan
|
|
Possible, and it is why the alert is worded as a security signal (docs/07). Distinguish it by the
|
|
**source**: a spike from outside the cluster or from an unexpected identity, rather than one named
|
|
workload repeatedly failing one destination. Escalate rather than tuning policy.
|
|
|
|
## When to escalate or stop
|
|
|
|
- **Do not widen a policy to make drops stop.** Default-deny is a safety-critical requirement here
|
|
(R-NET-01 is flagged `safety_critical: true` in the requirement registry). The correct fix is the
|
|
narrowest explicit allow, authored in git, usually via the catalog so it is rendered consistently.
|
|
- **Do not hand-edit a rendered CiliumNetworkPolicy** — it comes from `catalog/render.py` and will be
|
|
reverted on the next reconcile.
|
|
- **Escalate if the source is unexpected.** An unexplained spike from outside the cluster is a security
|
|
event, not a configuration bug.
|
|
- If drops coincide with a Cilium upgrade, assume the hairpin/identity class above before assuming a
|
|
policy authoring mistake — the upgrade changed the *evaluation*, not the policy.
|
|
- Note the diagnostic trap when verifying a fix: probe from **outside** with `curl --resolve`, because
|
|
in-pod HTTPS tests to the platform's own hostnames are misleading.
|
|
|
|
---
|
|
Source of truth: `runbooks/denied-flows.md`. Index: `/runbooks/`
|
|
Records: `DECISIONS-LOG.md` (the Cilium CVE-2025-30162 world-hairpin / envoy 403 finding and its
|
|
`render.py` fix; source-controller denied by `gitea-allow-gateway-ingress`).
|