fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http

I broke the app tier with #48. Two faults, both mine, both measured after merge: 1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key Secrets (plex-vip-<random>) were denied by secret-provenance/secrets-must-be-flux-managed, which matches every Secret in a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes only those labelled secret-managed-by=controller or carrying a Flux label. `plex` HAS that label; `gateway` -- where the working wildcard cert lives -- does NOT. That asymmetry is why the existing Certificate was never affected and mine was, and I did not check it. secretTemplate cannot fix it: those Secrets come from the keymanager controller, not the Certificate template. So no CertificateRequest, Order or Challenge was ever created. 2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so Flux waits on the Certificate OBJECT: timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress'] apps-wave-0 never went Ready; wave-1/-2 and everything downstream of platform-config sat in DependencyNotReady behind it. I guarded the pod and missed the object. That distinction is the whole bug. Third, independent: the advertised URL was https with no servable certificate -- Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip -- the exact failure mode of the outage earlier the same day. So: delete the Certificate (pruning it clears the health gate) and advertise http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there. http is explicitly temporary and only tolerable because remote access does not work yet; the restore condition is written at the call site. DELETED, not merely unregistered: my first attempt kept the file as "tracked but not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02) fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization + the https flip, as ONE change) is recorded in catalog/plex.yaml. Kept and verified working from #48: the hostname design, PiHole's split-horizon answer, and the postStart hook -- LanNetworksBandwidth is now set live. Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Merge pull request 'feat(plex): one advertised hostname for local+remote, its own LE cert, and a LAN definition so LAN clients stop reading as remote' (#48 ) from feat/plex-vip-hostname-and-lan-networks into main
2026-10-04 17:10:25 -05:00 · 2026-10-04 21:47:32 +00:00 · 2026-10-04 21:47:20 +00:00 · 2026-10-04 16:36:12 -05:00 · 2026-10-04 21:33:28 +00:00 · 2026-10-04 16:05:44 -05:00
19 changed files with 764 additions and 91 deletions
@@ -244,6 +244,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): jackett-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: jackett-allow-monitoring-scrape
+  namespace: jackett
+  labels:
+    app.kubernetes.io/name: jackett
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: jackett
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -18,7 +18,6 @@ resources:
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
- plex-cert.yaml
 - plex-claim-secret.sops.yaml
 - plex-remote.yaml
 - plex.yaml
@@ -232,6 +232,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): pihole-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: pihole-allow-monitoring-scrape
+  namespace: pihole
+  labels:
+    app.kubernetes.io/name: pihole
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: pihole
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): pihole-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -1,66 +0,0 @@
---
-# =================================================================================================
-# PLEX TLS — a Let's Encrypt certificate for the ONE hostname Plex advertises to plex.tv.
-# =================================================================================================
-# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
-# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
-# mechanism gitea-actions, pihole-dns and plex-remote use. It is hand-written for the same reason:
-# `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has NO field for a
-# cert-manager Certificate. (`ingressRoutes` gets TLS for free from the Gateway's wildcard; this is
-# the opposite case — a certificate for a workload that terminates its OWN TLS on a raw L4 port.)
-#
-# WHY PLEX NEEDS ITS OWN CERTIFICATE AT ALL. Plex serves tcp/32400 directly (not through the L7
-# Gateway — the operator chose the native path 2026-10-03, apps/wave-0/plex-remote.yaml) and it
-# presents its own `*.<serverhash>.plex.direct` certificate there. That name cannot cover
-# `plex-vip.${BASE_DOMAIN}`, the hostname we advertise, so without this the advertised `https://`
-# URL fails hostname verification for every client. That is not hypothetical: it is the shape of
-# the 2026-10-04 outage, where Plex's log carried a continuous storm of
-# `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` from real LAN clients while
-# plex.tv could publish nothing and the server vanished from the account.
-#
-# A SAN match is not what makes Plex FUNCTION — `httpsRequired=0` and `secureConnections` default
-# to Preferred, so clients can fall back to cleartext. It is what keeps the REMOTE leg encrypted
-# rather than silently downgrading traffic that crosses the internet.
-#
-# HOW IT REACHES PLEX. catalog/plex.yaml takes `tls.crt`/`tls.key` from this Secret as env
-# (`optional: true` — see below), and its `lifecycle.postStart` hook converts the PEM into the
-# PKCS#12 bundle Plex's `customCertificatePath` requires and sets the three `customCertificate*`
-# prefs. The conversion is redone on EVERY pod start, so a cert-manager RENEWAL is picked up with
-# no extra moving part and nothing to remember.
-#
-# ORDERING, AND WHY THE ENV REFS ARE OPTIONAL. This Certificate must live in the `plex` namespace
-# for the Deployment to read its Secret, and that namespace is created by the rendered
-# apps/wave-0/plex.yaml ITSELF — so this cannot be issued in an earlier tier (platform-config runs
-# before the namespace exists) and lands in the SAME reconcile as the workload. On a cold bootstrap
-# the Secret is therefore absent for as long as a DNS01 challenge takes. With hard env refs Plex
-# would sit in CreateContainerConfigError until then, and because `apps-wave-0` is `wait: true`
-# with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE APP TIER. Hence
-# `optional: true` on both refs (schema 1.11.0, added for exactly this): no certificate means Plex
-# still starts and serves on its own plex.direct cert, the hook logs that TLS was not installed,
-# and the next start after issuance converges.
-#
-# ISSUER: ClusterIssuer `letsencrypt-prod` (platform/cert-manager/clusterissuers.yaml), the same
-# issuer behind the Gateway's wildcard (platform/gateway/wildcard-certificate.yaml, Ready 28d on
-# prod as measured 2026-10-04). A SINGLE name is requested rather than reusing the wildcard,
-# because the wildcard's Secret lives in the `gateway` namespace and a Secret cannot be read across
-# namespaces; requesting `*.${BASE_DOMAIN}` again here would also burn a duplicate-certificate slot
-# against Let's Encrypt's weekly limit for no benefit. A single name is the least-privilege shape:
-# this key signs for `plex-vip.${BASE_DOMAIN}` and nothing else.
-apiVersion: cert-manager.io/v1
-kind: Certificate
-metadata:
-  name: plex-vip
-  namespace: plex
-  labels:
-    app.kubernetes.io/name: plex
-spec:
-  # The name Plex ADVERTISES and the name on this certificate must stay identical. They are two
-  # halves of one decision: catalog/plex.yaml's ADVERTISE_IP, the postStart hook's
-  # customCertificateDomain, this dnsNames entry, and catalog/pihole.yaml's split-horizon answer.
-  # Nothing enforces that they agree.
-  dnsNames:
-    - plex-vip.${BASE_DOMAIN}
-  secretName: plex-vip-tls
-  issuerRef:
-    kind: ClusterIssuer
-    name: letsencrypt-prod
@@ -138,7 +138,7 @@ spec:
              name: plex-claim
              key: PLEX_CLAIM
        - name: ADVERTISE_IP
-          value: https://plex-vip.${BASE_DOMAIN}:32400
+          value: http://plex-vip.${BASE_DOMAIN}:32400
        - name: PLEX_TLS_CRT
          valueFrom:
            secretKeyRef:
@@ -319,6 +319,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): qbittorrent-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: qbittorrent-allow-monitoring-scrape
+  namespace: qbittorrent
+  labels:
+    app.kubernetes.io/name: qbittorrent
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: qbittorrent
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): qbittorrent-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -347,6 +347,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): radarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: radarr-allow-monitoring-scrape
+  namespace: radarr
+  labels:
+    app.kubernetes.io/name: radarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: radarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): radarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -345,6 +345,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -107,8 +107,42 @@ env:
  # OVERLAY fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not
  # a web form, owns the value. Hand-typed custom URLs are what broke this service; they are now
  # overwritten on every start.
+  #
+  # SCHEME IS `http` AND THAT IS TEMPORARY, 2026-10-04. It was `https`, which requires Plex to serve
+  # a certificate for this name — and the Certificate that would have supplied one is NOT APPLIED
+  # (see apps/wave-0/plex-cert.yaml's header). Advertising `https` without it reproduces the
+  # 2026-10-04 outage exactly: Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, every
+  # TLS-validating client rejects the hostname, and plex.tv can publish nothing. `http` works TODAY
+  # and is MEASURED to: PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (verified live
+  # against pihole-0, 2026-10-04) and Plex serves 32400 there.
+  #
+  # THE COST, STATED: the remote leg is unencrypted. That is tolerable only because remote access
+  # does not work yet at all (the WAN tcp/32400 forward is still absent), so today this carries LAN
+  # traffic only. RESTORE `https` IN THE SAME COMMIT that re-applies the Certificate — the two are
+  # one change, and splitting them is what broke this.
+  #
+  # HOW TO GET `https` BACK. The Certificate that would serve this name was applied on 2026-10-04
+  # and STALLED THE ENTIRE APP TIER; the file was DELETED rather than merely unregistered, because
+  # `tls-lifecycle-automated` (R-TLS-02) fails any Certificate left in the tree that no Flux path
+  # reconciles. Full post-mortem: decisions/2026-10-04-plex-unblock-app-tier.md. Three things must
+  # land TOGETHER:
+  #   1. EXEMPT cert-manager FROM `secret-provenance`. Its temporary private-key Secrets
+  #      (`plex-vip-<random>`) are denied by `secrets-must-be-flux-managed`, which matches every
+  #      Secret in a namespace labelled `homelab.keithjackson.dev/app-namespace: true` — `plex` HAS
+  #      that label, `gateway` (where the working wildcard cert lives) does NOT, which is why this
+  #      never bit the existing Certificate. `secretTemplate` cannot fix it: those Secrets come from
+  #      the keymanager controller, not the Certificate's template. Narrow fix: an
+  #      `exclude.any[].subjects` entry for ServiceAccount `cert-manager`/`cert-manager` (confirmed
+  #      live as the controller's SA). `platform/kyverno/**` is a SENSITIVE path — non-author approval.
+  #   2. GIVE THE CERTIFICATE A NON-GATING HOME. `apps-wave-0` is `wait: true`, so Flux health-checks
+  #      the Certificate OBJECT and a pending issuance stalls the whole tier
+  #      (`timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress']`). `optional: true`
+  #      on the env refs below protects the POD, not the Kustomization — that distinction is the
+  #      whole bug. Use a separate Kustomization with `wait: false`, the shape
+  #      clusters/base/backup.yaml already uses for exactly this reason.
+  #   3. Flip this value back to `https`.
  - name: ADVERTISE_IP
-    value: https://plex-vip.${BASE_DOMAIN}:32400
+    value: http://plex-vip.${BASE_DOMAIN}:32400
  # THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
  # `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
  # `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
@@ -1541,12 +1541,54 @@ def r_monitoring_ingress_policy(s: dict):
    trap Longhorn's own NetworkPolicies set (platform/longhorn/networkpolicy-metrics.yaml exists
    for exactly this reason), and it is a THIRD independent way for a metrics SLI to be inert.

-    Rendered only alongside a declared metricsEndpoint, and it opens ONLY that port to ONLY the
-    monitoring namespace: an allow for a port nothing serves would be a standing hole for no
+    Rendered only alongside a port something actually serves, and it opens ONLY that port to ONLY
+    the monitoring namespace: an allow for a port nothing serves would be a standing hole for no
    benefit.
+
+    TWO INDEPENDENT SOURCES OF A SERVED METRICS PORT, and this used to cover only the first:
+
+      1. `observability.metricsEndpoint` — the APP's own exporter. NOTE, measured 2026-10-04: no
+         ServiceDefinition in this catalog declares one, so before the sidecar clause below this
+         function had NEVER emitted a single object on any branch. It is correct but, so far,
+         entirely latent — do not read its presence as evidence that scrape ingress is solved.
+
+      2. THE forward-auth GATE SIDECAR's own `/metrics` on 44180 (added 2026-10-04). Every
+         `_needs_oauth2_sidecar` app gets `--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}`
+         and a named `metrics` container port from `_oauth2_sidecar_container`, so the port is
+         served whenever the sidecar exists — unconditionally, which is what makes it safe to
+         open without a per-app declaration.
+
+    WHY CLAUSE 2 KEYS ON `_needs_oauth2_sidecar` AND NOT ON `observability.metrics`: it must match
+    the condition the SCRAPE keys on, exactly. platform/oauth2-proxy/podmonitor.yaml selects pods
+    by the `homelab.keithjackson.dev/auth-mode: forward-auth` POD LABEL and nothing else, so a
+    target exists iff the sidecar exists. Keying the allowance on `observability.metrics` instead
+    would let `metrics: false` produce a discovered-but-blocked target — the same "two halves
+    authored apart" failure this function exists to prevent, just moved one field over.
+
+    THIS CLAUSE IS A BUG FIX, and the bug was live for 63.7h. Measured on prod 2026-10-04: all 6
+    sidecar targets across jackett/radarr/sonarr/qbittorrent/pihole read `health=down`,
+    `lastError: context deadline exceeded`, `lastScrapeDuration` exactly 10.0s — the scrapeTimeout,
+    i.e. packets DROPPED, not refused. Each app namespace ships a `default-deny` NetworkPolicy
+    with `policyTypes: [Ingress, Egress]` and NO `ingress` key at all (deny-all-ingress), and the
+    only ingress allowance is `<app>-allow-gateway-ingress`: `fromEntities: [ingress]` -> 4180.
+    Prometheus matched no clause. 5 x `TargetDown` + 6 x `OAuth2ProxyGateUnobserved` had been
+    re-notifying at ~22 emails/day since the PodMonitor landed on 2026-10-02.
+
+    WHY THE POLICY BELONGS HERE AND NOT IN platform/oauth2-proxy: podmonitor.yaml's header records
+    the objection — a policy in that tier would name app namespaces created by the SIBLING
+    apps-wave-0/1 tiers (measured Flux graph: `oauth2-proxy` and `apps-wave-0` both
+    `dependsOn=[platform-config]`, no ordering between them), so it would apply on a running
+    cluster and fail on a cold bootstrap. Rendered here it ships in the same tier that creates the
+    namespace, which dissolves that ordering problem entirely.
    """
-    ep = s["observability"].get("metricsEndpoint")
-    if not ep or not s["observability"]["metrics"]:
+    obs = s["observability"]
+    ep = obs.get("metricsEndpoint")
+    ports: list[str] = []
+    if ep and obs["metrics"]:
+        ports.append(str(ep["port"]))
+    if _needs_oauth2_sidecar(s):
+        ports.append(str(OAUTH2_PROXY_METRICS_PORT))
+    if not ports:
        return None
    return {
        "apiVersion": "cilium.io/v2",
@@ -1564,7 +1606,7 @@ def r_monitoring_ingress_policy(s: dict):
                        {"matchLabels": {"k8s:io.kubernetes.pod.namespace": MONITORING_NS}}
                    ],
                    "toPorts": [
-                        {"ports": [{"port": str(ep["port"]), "protocol": "TCP"}]}
+                        {"ports": [{"port": p, "protocol": "TCP"} for p in ports]}
                    ],
                }
            ],
@@ -2167,23 +2209,6 @@ EXTRA_UNITS: dict = {
            "plex-remote.yaml",
        ],
    },
-    # Plex's own TLS certificate for the single hostname it advertises to plex.tv
-    # (apps/wave-0/plex-cert.yaml). `above: False` co-locates it in plex's OWN wave, which is the
-    # only placement available: the Certificate must land in the `plex` namespace, and that
-    # namespace is created by the rendered plex.yaml in this very wave, so it CANNOT be issued
-    # earlier. The workload tolerates the resulting gap by marking the env refs `optional` rather
-    # than by ordering -- see the header of plex-cert.yaml.
-    #
-    # Non-catalog because the ServiceDefinition schema has no field for a cert-manager Certificate
-    # (additionalProperties is false); `ingressRoutes` inherits the Gateway wildcard instead, which
-    # is the opposite case to a workload terminating its own TLS on a raw L4 port.
-    "plex-cert": {
-        "anchor": "plex",
-        "above": False,
-        "files": [
-            "plex-cert.yaml",
-        ],
-    },
 }


@@ -171,6 +171,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): paas-demo-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: paas-demo-allow-monitoring-scrape
+  namespace: paas-demo
+  labels:
+    app.kubernetes.io/name: paas-demo
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: paas-demo
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): paas-demo-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -198,6 +198,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -198,6 +198,27 @@ spec:
      - port: '4180'
        protocol: TCP
 ---
+# CiliumNetworkPolicy monitoring scrape ingress allow (R-NET-02/R-OBS-01): sonarr-allow-monitoring-scrape
+apiVersion: cilium.io/v2
+kind: CiliumNetworkPolicy
+metadata:
+  name: sonarr-allow-monitoring-scrape
+  namespace: sonarr
+  labels:
+    app.kubernetes.io/name: sonarr
+spec:
+  endpointSelector:
+    matchLabels:
+      app.kubernetes.io/name: sonarr
+  ingress:
+  - fromEndpoints:
+    - matchLabels:
+        k8s:io.kubernetes.pod.namespace: monitoring
+    toPorts:
+    - ports:
+      - port: '44180'
+        protocol: TCP
+---
 # CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): sonarr-allow-internet-egress
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
@@ -0,0 +1,157 @@
+---
+date: 2026-10-04
+branch: feat/hubble-flow-export-plex
+title: Record the pre-SNAT client address for Plex via Hubble static flow export, instead of breaking externalTrafficPolicy to get it
+---
+
+## What changed
+
+Operator requirement, narrowed explicitly: **this is 100% about log visibility** — knowing which
+address connected to Plex. Not bandwidth metering, which does not matter here.
+
+**The problem, measured.** Plex's own log cannot show it. Every LoadBalancer Service in this cluster
+is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the VIP SNATs the client
+before forwarding to the backend pod:
+
+| Measured 2026-10-04 | Value |
+|---|---|
+| Node announcing `192.168.2.24` | `k3s-prod-3` (also holds `.20` Gateway, `.23` pihole-dns) |
+| Node running the Plex pod | `k3s-prod-1` (`10.0.0.135`) |
+| Sources in Plex's log | **3,214 × `10.0.1.x`**, 105 × `10.0.2.x`, 120 × loopback |
+| `192.168.x` sources in Plex's log, ever | **0** |
+| `X-Forwarded-For` in Plex's log | **0** (it serves raw tcp/32400, not HTTP via the Gateway) |
+| `10.0.1.78` identity (`cilium-dbg bpf ipcache`) | `identity=1` → `reserved:host` = prod-3 itself |
+
+**Nothing on the router is involved** — the operator correctly pointed out there is no hairpin NAT
+here. The rewrite is entirely in-cluster, done by Cilium's BPF load-balancer on the lease holder,
+because the backend pod is on a different node.
+
+**The address is not lost, only unrecorded.** Cilium observes the flow at ingress, *before* its own
+translation, so Hubble already holds the true source:
+
+    192.168.2.133 -> 10.0.0.135    (real LAN client -> the Plex pod)
+    10.0.1.78     -> 10.0.0.135    (the same traffic post-SNAT — what Plex sees)
+
+What was missing is **retention**: the ring buffer was **4095/4095 flows at ~303 flows/s — about
+thirteen seconds of history** — with hubble metrics disabled and no exporter configured.
+
+So: enable the static flow exporter, scoped to Plex's port, writing to the agent's stdout.
+
+- **`platform/cilium/helmrelease.yaml`** (authoritative): `hubble.export.static.enabled: true`,
+  `filePath: stdout`, `allowList` of `{"destination_port":["32400"]}` and
+  `{"source_port":["32400"]}`.
+- **`platform/cilium/values.yaml`** (reference mirror) and
+  **`infra/ansible/roles/cilium/templates/cilium-values.yaml.j2`** (the `[parity]` source of truth):
+  the same block, so a bootstrap render does not differ from what Flux reconciles — a mismatch would
+  be a permanent Helm diff and an avoidable agent rollout.
+
+**`filePath: stdout`, not a file,** because Grafana Alloy already runs as a DaemonSet tailing every
+pod's logs into Loki (`platform/monitoring/alloy.yaml`). stdout therefore reaches Loki with **no new
+hostPath mount, no file-rotation handling, and no Alloy change at all**. Upstream documents the
+value: *"To get Hubble flows directly exported to the logs instead of written to a rotated file,
+`stdout` can be defined as `hubble-export-file-path`."* Query:
+
+    {namespace="kube-system", app="cilium"} |= "32400"
+
+**SCOPED BY PORT, NOT BY NAMESPACE — a measured decision, not a preference.** A
+`--to-namespace plex` query returned **only the post-SNAT flows**; the flow carrying the real client
+address did not match it, because at the point of observation the destination identity is not yet
+resolved to the plex pod. A namespace filter would have dropped exactly the record this change
+exists to capture.
+
+**WHY NOT FIX THE SNAT INSTEAD — both alternatives rejected on evidence.**
+- `externalTrafficPolicy: Local` would put the real address in Plex's own log, but cilium 1.20's L2
+  announcer is **explicitly incompatible** with it. Upstream docs for this exact version:
+  *"incompatible with the `externalTrafficPolicy: Local` on services as it may cause service IPs to
+  be announced on nodes without pods causing traffic drops"* and *"L2 announcements isn't currently
+  aware of this setting"*. Live proof it is not theoretical: the VIP is announced by prod-3 while the
+  pod runs on prod-1, so the flip would black-hole tcp/32400 **immediately**. Making it safe means
+  pinning both the pod and the announcement to one node and surrendering Plex's ability to
+  reschedule — a real availability loss to buy a log field. Same trap
+  `platform/crafty/service-l4.yaml` documents from an 8-day Minecraft outage.
+- **DSR** also preserves the client address, but `routing-mode` here is **vxlan**, not
+  native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO router's stateful
+  NAT on the remote path. Cluster-wide blast radius across all eight LB Services for one service's
+  logging.
+
+**A correction to the repo's own record.** `decisions/2026-10-03-plex-remote-access-l4.md` states the
+SNAT puts the client inside `192.168.0.0/16` (the ingress node's LAN address), and I repeated that in
+the previous entry. **It is wrong**: measured, the SNAT source is `10.0.1.x` — the node's
+`cilium_host` inside the pod CIDR. This matters, because it means `LanNetworksBandwidth` can never
+discriminate local from remote while eTP is `Cluster`: `10.0.0.0/8` must be in the LAN set for the
+in-cluster consumers (Seerr) to count as local, and every external client arrives in that same range.
+PR #48's LAN definition therefore fixes the Remote badge by making *everything* local — acceptable
+only because metering is explicitly not a goal here.
+
+## What I verified
+
+Live prod, 2026-10-04, read-only.
+
+- **Every measurement in the table above**, including the VIP-lease-to-node map
+  (`cilium-dbg shell -- db/show l2-announce` on all three agents) and the pod's node.
+- **Hubble already carries the pre-SNAT source** — the single finding this whole approach rests on:
+  `hubble observe --port 32400` on the lease holder returned `192.168.2.133 -> 10.0.0.135` **and**
+  its return flow, alongside the post-SNAT `10.0.1.78 -> 10.0.0.135`.
+- **The retention gap is real:** `cilium-dbg status` reported `Hubble: Ok  Current/Max Flows:
+  4095/4095 (100.00%), Flows/s: 303.62  Metrics: Disabled`, and the `cilium-config` ConfigMap
+  contains no `hubble-export-*` key at all.
+- **The namespace-filter trap**, directly: `--to-namespace plex` returned only `10.0.1.78` and
+  `10.0.1.71`; the `192.168.2.133` flow was absent. This is why the allowList is port-scoped.
+- **All eight LoadBalancer Services are `eTP=Cluster`** (crafty ×2, gateway, pihole-dns ×2, plex,
+  unifi ×2) — so this is a cluster-wide pattern, not a Plex-specific choice.
+- **The eTP:Local incompatibility is upstream-documented for v1.20 specifically**, quoted above from
+  the docs for the running version — not inferred from the crafty incident.
+- **`routing-mode: tunnel`** and **`bpf-lb-mode-annotation: false`** in the live `cilium-config`,
+  which is what rules DSR out.
+- **The chart's value paths and defaults**, read from cilium v1.20.1's own `values.yaml`:
+  `hubble.export.static.{enabled,filePath,fieldMask,fieldAggregate,aggregationInterval,allowList,denyList,fileMaxSizeMb,fileMaxBackups,fileCompress}`,
+  with `filePath` defaulting to `/var/run/cilium/hubble/events.log`. The keys used here exist.
+- **Alloy's pipeline shape**, read from `platform/monitoring/alloy.yaml`: a DaemonSet with
+  `loki.source.kubernetes "pods"` → `loki.process` → `loki.write` at
+  `http://loki.monitoring.svc.cluster.local:3100`. It already tails kube-system pods, which is why
+  stdout needs no Alloy change.
+- **All three value files agree** (one `destination_port` occurrence each) and both YAML files parse,
+  with `export.static` resolving to the intended object.
+- **Toolbox gate**: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1` and `approval-policy` all **pass**; rc 3 is the two known environmental skips
+  (`gitea-settings` — no local `SETTINGS_API_TOKEN`; `conformance` L2/L3 — `CLUSTER_AVAILABLE`
+  unset). **Not green. Reported as rc 3.**
+- **No sensitive paths touched** — `platform/cilium/**` and `infra/ansible/**` are not in
+  `infra/gitea/approval-policy.json`, so this needs no approval override, unlike PR #48.
+
+## What I did NOT verify
+
+- **THE OUTCOME. Nothing has been reconciled and no flow has ever been exported.** `cilium-config`
+  still has no `hubble-export-*` key. Every claim is about inputs.
+- **Whether `filePath: stdout` is accepted as written.** This is the single largest risk. The value
+  comes from a verbatim upstream doc sentence (*"`stdout` can be defined as
+  `hubble-export-file-path`"*), but I did not confirm the exact notation in cilium 1.20.1's source,
+  and the chart default is a real filesystem path. If the literal string is not special-cased, the
+  agent may create a FILE named `stdout` in its working directory and nothing reaches Loki — a
+  silent no-op, not a crash. **Fallback if so:** leave `filePath` at the chart default and add a
+  `local.file_match` + `loki.source.file` target to Alloy with a hostPath mount for
+  `/var/run/cilium/hubble`.
+- **Whether the FlowFilter JSON is accepted.** `{"destination_port":["32400"]}` follows the proto
+  (`destination_port` is a repeated string) and the documented shape (`'{"verdict":["DROPPED"]}'`),
+  but upstream gives **no port example** and I did not validate it against the running agent. A
+  malformed filter could be rejected — or, worse, be silently treated as "no match" and export
+  nothing.
+- **THIS CHANGE ROLLS CILIUM.** Editing the HelmRelease restarts the agent DaemonSet across all
+  three nodes. BPF maps are preserved so established traffic should survive, but prod-3 currently
+  announces the Gateway VIP, **PiHole's DNS VIP** and Plex's VIP, so a lease change during the
+  rollout could briefly interrupt LAN DNS for the whole house. Not measured, and worth doing at a
+  quiet moment.
+- **Log volume in practice.** Estimated "a few hundred to a few thousand lines a day" from the
+  observed request counts; never measured as flows. Both directions are exported and no `fieldMask`
+  is set, so each record is a full flow object. If it is noisy, narrow with `fieldMask` rather than
+  by dropping a filter.
+- **Loki's retention for this data**, and whether the `{namespace="kube-system", app="cilium"}` label
+  set actually matches (it depends on the cilium pods carrying
+  `app.kubernetes.io/name`, which Alloy maps to `app` — I did not confirm that label exists on the
+  cilium DaemonSet pods). The query in the comment may need adjusting to `pod=~"cilium-.*"`.
+- **That the exported flow actually contains the pre-SNAT address.** Hubble's *live* stream does
+  (measured). I am assuming the exporter sees the same flow objects; I did not confirm the exporter
+  is not fed a post-translation view.
+- **The parity convention has no automated check.** I kept all three files in sync by hand because
+  the comments demand it; nothing in `ci/` or `conformance/` verifies cilium Helm↔Ansible parity, so
+  a future edit can still drift them silently.
@@ -0,0 +1,150 @@
+---
+date: 2026-10-04
+branch: fix/oauth2-sidecar-metrics-scrape-netpol
+title: The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring
+---
+
+## What changed
+
+`catalog/render.py`'s `r_monitoring_ingress_policy` now emits its CiliumNetworkPolicy for a
+**second** reason: the forward-auth gate sidecar's own `/metrics` port (44180), keyed on
+`_needs_oauth2_sidecar(s)`. Previously it fired only from `observability.metricsEndpoint`.
+
+That matters more than it sounds, because **no ServiceDefinition in this catalog declares a
+`metricsEndpoint`** (measured: zero grep hits across `catalog/*.yaml`). So the function was wired
+in at the call site and had **never emitted a single object on any branch** — correct, and
+entirely latent. Its presence in the tree read as "scrape ingress is handled". It was not.
+
+Re-rendered `apps/` from that change: five new CNPs, one per forward-auth app
+(`jackett`, `pihole`, `qbittorrent`, `radarr`, `sonarr`), each opening **only** 44180 to **only**
+`fromEndpoints: [k8s:io.kubernetes.pod.namespace: monitoring]`.
+
+The bug being fixed: `platform/oauth2-proxy/podmonitor.yaml` landed 2026-10-02T04:58:44Z and
+discovered 6 targets that have **never once scraped successfully**. Each app namespace ships a
+`default-deny` NetworkPolicy with `policyTypes: [Ingress, Egress]` and **no `ingress` key at all**
+— deny-all-ingress in NetworkPolicy semantics — and the only ingress allowance is
+`<app>-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180. Prometheus matched no
+clause, so its packets were dropped.
+
+Two deliberate choices, both recorded in the function's docstring:
+
+- **Clause 2 keys on `_needs_oauth2_sidecar`, not on `observability.metrics`.** It has to match
+  the condition the *scrape* keys on: `podmonitor.yaml` selects pods by the
+  `homelab.keithjackson.dev/auth-mode: forward-auth` pod label and nothing else, so a target
+  exists iff the sidecar exists. Keying on `observability.metrics` would let `metrics: false`
+  produce a discovered-but-blocked target — the same "two halves authored apart" failure this
+  function exists to prevent, moved one field over.
+- **The policy is rendered here, not added to `platform/oauth2-proxy/`.** podmonitor.yaml's own
+  header records why that tier cannot host it: it would name app namespaces created by the
+  *sibling* `apps-wave-0/1` tiers, so it would apply on a running cluster and fail on a cold
+  bootstrap. Rendered per-app it ships in the same tier that creates the namespace.
+
+## What I verified
+
+Measured on prod 2026-10-04, read-only, via the pinned toolbox:
+
+- **The cause, not a guess.** All 6 sidecar targets in pool
+  `podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0`: `health=down`,
+  `lastError: context deadline exceeded`, `lastScrapeDuration` exactly `10.0s` — the configured
+  `scrapeTimeout`. Hitting the timeout precisely means packets **dropped, not refused**.
+- **The listener is fine.** jackett's `oauth2-proxy` container carries
+  `--metrics-address=0.0.0.0:44180` and a named port `metrics: 44180`.
+- **The deny is universal.** All six namespaces (`jackett radarr sonarr qbittorrent pihole
+  crafty`): `default-deny` with `ingress` key absent, `nIngress=0`; the only ingress CNP allowance
+  is 4180 `fromEntities: [ingress]`. **Zero** namespaces had any `-allow-monitoring-scrape` object.
+- **Frequency.** Single continuous episode, no flapping: `ALERTS_FOR_STATE` = 229343s (63.7h),
+  firing since 2026-10-02 05:09 UTC — 11 minutes after the PodMonitor was created, i.e. TargetDown's
+  `for: 10m`. A 15-day `query_range` shows **1 episode** per namespace, not many. Alertmanager:
+  `group_by: [alertname, namespace]`, `repeat_interval: 12h` on `email-warning` → 5 TargetDown
+  groups = 10 emails/day, plus 6 `OAuth2ProxyGateUnobserved` groups = 12/day, ~22/day total.
+  0 silences.
+- **The re-render is deterministic and the diff is exactly the intended one.** `apps/` is
+  byte-identical to `render.py --all catalog -o apps --wave-layout --no-image-automation`
+  (12/12 SAME before my edit; the 11-file "drift" against a default render is solely the
+  ImageRepository/ImagePolicy pair, which `apps/` is generated without). After the edit:
+  5 files changed, +21 lines each, nothing else touched.
+- `render.py --selftest`: **PASS**.
+- **Three L1 goldens needed regenerating, and the diff is +63 lines / -0.** The first gate run
+  caught `container-workload-render` (R-PLAT-02) and `egress-class-render` (R-NET-05/06/07) as
+  FAIL with exactly the new CNP as the drift. Regenerated with the same invocations the checks
+  use (`render.py <fixture>`, no flags). Evidence the regen is faithful rather than a rubber
+  stamp: `default-deny-baseline-render`'s golden came back **byte-identical** (its fixture is not
+  forward-auth), and the other three gained only the one CNP, port 44180, ns monitoring.
+- Toolbox gate `./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`: **rc 3 — UNDETERMINED,
+  NOT rc 0, and I am not reporting it as green.** Every stage that ran is green
+  (spec-immutability, approval-policy, decisions-log, lint, l0, render-golden, l1); `l1` went
+  FAIL -> pass once the goldens were regenerated. The gate's own words: "Stages that did not run:
+  gitea-settings" — skipped for want of `GITEA_SETTINGS_TOKEN`, which is an environment gap on
+  every local run and unrelated to this change. L2/L3 conformance is a `[warn]`, not a not-run
+  stage: UNVERIFIED for all 8 touched requirements (R-GITOPS-02 R-IDP-02 R-IDP-06 R-NET-01
+  R-NET-05 R-NET-06 R-NET-07 R-PLAT-02), no cluster.
+- `--emit-waves` was **not** run (it re-onboards `vm-demo` + `keycloak`); only the per-service
+  render path was used, so no wave kustomization changed.
+
+## This PR needs a human approval (sensitive path) — MEASURED, not predicted
+
+CI run #230 on PR #49, head `f9446be3`: **`MERGE GATE: BLOCKED`, and `approval-policy` is the ONLY
+failing stage.** Read from the job log, not from the status tick:
+
+    [skip] stage spec-immutability: SKIPPED (not run)   <- sibling spec repo absent in CI; expected
+    [ ok ] stage decisions-log: pass
+    [ ok ] stage lint: pass
+    [ ok ] stage l0: pass
+    [ ok ] stage render-golden: pass
+    [ ok ] stage l1: pass
+    [fail] stage approval-policy: FAIL
+    [ ok ] stage gitea-settings: pass                   <- could NOT run locally; green in CI
+    [skip] stage conformance: SKIPPED (not run)
+
+    sensitive paths touched:
+      conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml
+      conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml
+      conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml
+
+Exactly the three regenerated goldens, nothing else. `l1` and `render-golden` both pass in CI,
+which is the independent confirmation that the regenerated goldens are right rather than merely
+self-consistent with my local run.
+
+One local-reproduction note worth recording, because it wasted a step: running
+`PR_NUMBER=49 SETTINGS_API_TOKEN=... ci/approval-policy.sh` with the credential from
+`git credential fill` reports `could not determine PR author — BLOCKING`. That is an **artifact**,
+not a finding — `~/.git-credentials` holds a *password*, and `Authorization: token <password>`
+returns 401 `user does not exist [uid: 0]`. The stage needs a real API token. I nearly reported
+that 401 as the CI failure; the job log is what settled it.
+
+Regenerating the L1 goldens means this branch touches `conformance/**`, which is in
+`sensitive_paths` in `infra/gitea/approval-policy.json` ("the tests and the harness; changing them
+changes what green MEANS"). So `ci/approval-policy.sh` will require an approving review from a
+non-author, and the local gate **cannot** see that: it reports `approval-policy: green` because
+offline it evaluates N/A.
+
+The golden update is not optional — a renderer change that adds an object necessarily drifts the
+goldens that pin the renderer's output — but it does mean the gate correctly refuses to
+self-approve this one. **An agent must not clear this with `/approve-override`**, even though the
+`gitea-admin` account holds `owner`: that is precisely the self-approval the stage exists to
+prevent. The operator posts the override as a PR *review* of type "Comment", or approves normally.
+
+## What I did NOT verify
+
+- **That the scrape now succeeds.** This is a git-only change and nothing is applied until Flux
+  reconciles `main`. The 5 CNPs have never existed on the cluster, so `up == 1` for those targets
+  is **UNVERIFIED** — it is a prediction from the measured drop signature, not an observation.
+  After merge, confirm `up{container="oauth2-proxy"}` goes to 1 for all 6 targets and that the 11
+  alerts resolve. `send_resolved: true` is set, so expect ~11 resolved emails.
+- **That the CNP `endpointSelector` matches pihole's pods.** It selects
+  `app.kubernetes.io/name: pihole`, identical to the shape of `pihole-allow-gateway-ingress`
+  which demonstrably works for 4180 — so this is inference from a working sibling, not a
+  measurement of the new object.
+- **`crafty` is still unobserved and will keep firing `OAuth2ProxyGateUnobserved`.** It is
+  hand-managed under `platform/crafty`, outside the renderer, and has **no `--metrics-address`
+  at all** — so there is no port to open and this change cannot help it. Separate work.
+- **Whether 44180 is reachable in the Cilium data path once allowed.** Only the policy was
+  reasoned about; no L2 probe was run (CI has `vars.CLUSTER_AVAILABLE` unset, so L2/L3 did not
+  run here either — the gate proves the L0/L1 offline subset only).
+- **Why `OAuth2ProxyGateUnobserved` started at 2026-10-04 06:44** rather than 10-02 with
+  TargetDown. Alertmanager did not restart (TargetDown's `startsAt` is still 10-02) and the rule's
+  `for` is 1h, so neither explains it. Something changed on the monitoring side that morning which
+  I did not identify. It does not affect this fix.
+- I did not re-run a port-forward to 44180 to prove the endpoint serves *today*; I confirmed the
+  flag and named port on the live pod spec. The 200/10208-byte measurement is podmonitor.yaml's,
+  from 2026-10-01.
@@ -0,0 +1,126 @@
+---
+date: 2026-10-04
+branch: fix/plex-unblock-app-tier
+title: Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate
+---
+
+## What changed
+
+**I broke the app tier with PR #48 and this reverts the part that did it.** Two faults, both mine,
+both measured after the merge:
+
+1. **Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets
+   (`plex-vip-<random>`) were denied by `secret-provenance/secrets-must-be-flux-managed`:
+
+       resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies
+       secret-provenance:
+         secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by
+         Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets.
+
+   The policy matches every Secret in a namespace labelled
+   `homelab.keithjackson.dev/app-namespace: true` and excludes only Secrets labelled
+   `homelab.keithjackson.dev/secret-managed-by: controller` or carrying a Flux label. **The `plex`
+   namespace has that label; the `gateway` namespace — where the working wildcard cert lives — does
+   NOT.** That asymmetry is why the existing Certificate was never affected and mine was, and I did
+   not check it. cert-manager's temp key Secrets carry neither exempting label, and `secretTemplate`
+   cannot label them (the keymanager controller creates them, not the Certificate template). Result:
+   **no CertificateRequest, Order or Challenge was ever created.**
+
+2. **Flux health-checks the Certificate — guarding the pod was not enough.** `optional: true` did
+   exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But
+   `apps-wave-0` is `wait: true`, so Flux waits on every applied object with a known health type:
+
+       health check failed after 5m0.045564329s: timeout waiting for:
+       [Certificate/plex/plex-vip status: 'InProgress']
+
+   apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config
+   sat in `DependencyNotReady` behind it. I reasoned about the pod and missed the object.
+
+A third, independent problem the same PR introduced: **the advertised URL was `https://` with no
+servable certificate.** Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, which cannot
+cover `plex-vip.${BASE_DOMAIN}` — the precise failure mode of the outage earlier that day.
+
+- **`apps/wave-0/plex-cert.yaml` DELETED**, and `plex-cert.yaml` removed from
+  `apps/wave-0/kustomization.yaml` `resources`, so Flux prunes the stuck Certificate and the health
+  gate clears. **Deleted rather than merely unregistered** — my first attempt kept the file as
+  "tracked but not onboarded" (the shape `keycloak.yaml` and `vm-demo.yaml` use) and **the repo's own
+  gate rejected it**: `tls-lifecycle-automated` (R-TLS-02) fails any Certificate in the tree that no
+  Flux path reconciles — *"nothing reconciles it, so a deleted Certificate/issuer/controller would
+  stay deleted until a human re-applied it."* That pattern is legal for a Deployment and not for a
+  TLS object, which is a distinction I had not known.
+- **`catalog/render.py`:** the now-dangling `plex-cert` EXTRA_UNITS entry removed.
+- **The re-enable recipe is recorded at the call site** — in `catalog/plex.yaml` beside the scheme
+  it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating
+  Kustomization, and the `https` flip, as one change.
+- **`catalog/plex.yaml`:** `ADVERTISE_IP` → **`http://`**`plex-vip.${BASE_DOMAIN}:32400`, explicitly
+  temporary, with the restore condition written at the call site.
+
+**Everything else from PR #48 is kept and working** — the hostname design, the PiHole split-horizon
+answer, the `LanNetworksBandwidth` hook, the `optional` schema field, and the TLS env (inert without
+the Secret, so re-enabling is additive).
+
+## What I verified
+
+Live prod, 2026-10-04, read-only, after the #48/#50 merges.
+
+- **The stall and its exact cause**: the health-check timeout message above, and the full dependency
+  cascade (`cilium` → "dependency 'flux-system/flux-system' revision is not up to date", everything
+  else `DependencyNotReady`). The root `cilium` wait was a normal revision-gated cascade; the
+  *Certificate* was the real blocker.
+- **The Kyverno denial text, verbatim**, from the cert-manager controller log, repeating every ~30s
+  across at least 15 distinct Secret names.
+- **The policy's own structure**, read from `platform/kyverno/admission/secret-provenance.yaml`:
+  match = Secrets in `app-namespace: true` namespaces; exclude = `secret-managed-by: controller` OR
+  a `kustomize.toolkit.fluxcd.io/name` label; validate = unconditional `deny`.
+- **The namespace-label asymmetry**: `plex` carries
+  `homelab.keithjackson.dev/app-namespace: "true"`; `gateway` does not.
+- **cert-manager's controller ServiceAccount is `cert-manager`** in namespace `cert-manager` — the
+  identity a future exemption should name.
+- **`optional: true` worked**: the new pod (`plex-c4f87c84d-zgjq5`) is `1/1 Running` with no Secret
+  present, and `/config/certs` was never created.
+- **The postStart hook RAN and took effect** — the thing the whole hook was written for:
+  `LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set in the live `Preferences.xml`, and
+  `customConnections="https://plex-vip.home.keithjackson.dev:32400"` was applied from the env. The
+  certificate prefs are correctly **ABSENT**, i.e. the hook's "no bundle → leave them alone" guard
+  behaved as designed.
+- **PiHole's split-horizon answer is correct**, asked of `pihole-0` directly:
+  `plex-vip.home.keithjackson.dev -> 192.168.2.24`, alongside `plex -> .20` and `crafty-mc -> .22`,
+  and the live `/etc/pihole/dnsmasq.conf` carries
+  `address=/plex-vip.home.keithjackson.dev/192.168.2.24`. **This corrects my own earlier check**,
+  which read `.20` and which I nearly reported as a failure — that test resolved via CLUSTER DNS
+  (CoreDNS) from inside a pod and never consulted PiHole at all.
+- **The gate caught my first attempt and I fixed it rather than working around it.** Keeping the
+  file unregistered produced `MERGE GATE: BLOCKED` (rc 1) with
+  `CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed`. After deleting the file and
+  the dangling EXTRA_UNITS entry: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
+  `render-golden`, `l1` and `approval-policy` all **pass** — the rc 3 being only the two known
+  environmental skips (`gitea-settings`, `conformance` L2/L3). **Not green. Reported as rc 3.**
+
+## What I did NOT verify
+
+- **Whether this actually unblocks the tier.** Unreconciled at time of writing. The reasoning is
+  that pruning the Certificate removes the only failing health check, but I have not seen
+  apps-wave-0 go Ready.
+- **Whether `http://` restores client connectivity.** PiHole resolves the name and Plex serves 32400
+  (both measured), but I have not watched a Plex client complete a connection over the advertised
+  URL, and I did not re-check whether plex.tv publishes it.
+- **THE REMOTE BADGE — the symptom that started this.** `LanNetworksBandwidth` is set, but the log
+  still tags traffic `Allowed Network (WAN)`: **82 WAN, 0 LAN**, with entries continuing after the
+  pref was written. So either Plex needs a restart for it to take effect, or that log tag is not
+  driven by this pref at all. **I do not know which, and the user-visible badge is the only real
+  test** — it is in the Plex client, which I cannot see.
+- **The Hubble flow export is NOT active.** The cilium agents are still the pre-merge pods, so the
+  `filePath: stdout` question is still open and no flow has been exported. The Cilium rollout — and
+  its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD.
+- **Whether a cert-manager exemption is the right shape**, or whether it widens
+  `secret-provenance` more than the operator wants. It is `platform/kyverno/**`, a sensitive path,
+  and deliberately not attempted here.
+- **Whether the pruned Certificate leaves residue** — I did not check for orphaned
+  `plex-vip-<random>` Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one
+  of those Secrets, so there may be nothing to clean, but that is an inference, not a check.
+- **Whether `tls-lifecycle-automated` would also have objected to the Certificate living in a
+  `wait: false` Kustomization** (the recommended re-enable shape). It demands a reconciling Flux
+  path, which that provides, but I did not test it.
+- **The unencrypted advertised URL.** `http://` is tolerable only while remote access does not work
+  (no WAN forward). Nothing enforces that assumption, and nothing will complain if `http` is still
+  there the day the port-forward appears.
@@ -45,6 +45,20 @@ hubble:
    enabled: true
  ui:
    enabled: true
+  # Static flow export -> cilium-agent stdout. Kept EQUAL to the Flux values
+  # (platform/cilium/helmrelease.yaml), which carry the full rationale: it records the PRE-SNAT
+  # client address for Plex's raw tcp/32400, which Plex itself cannot log because every
+  # LoadBalancer Service here is externalTrafficPolicy: Cluster and the VIP lease holder SNATs the
+  # client. Present here so a BOOTSTRAP render does not differ from what Flux reconciles -- a
+  # mismatch would show up as a permanent Helm diff and an avoidable agent rollout. Scoped by PORT,
+  # not namespace: the flow carrying the real client address does not match a namespace filter.
+  export:
+    static:
+      enabled: true
+      filePath: stdout
+      allowList:
+        - '{"destination_port":["32400"]}'
+        - '{"source_port":["32400"]}'

 # Ingress VIP announcement on the NAT LAN. IP pool / CiliumL2AnnouncementPolicy
 # (from the reserved 10.73.0.240-.250 range) is authored by WS-NET in Phase 2.
@@ -101,6 +101,60 @@ spec:
        enabled: true
      ui:
        enabled: true   # exposure gated behind oauth2-proxy forward-auth later (ADR-0014)
+      # --- STATIC FLOW EXPORT: persist the PRE-SNAT client address for Plex (2026-10-04) ---
+      # [parity] MUST equal Ansible: hubble.export.static.{enabled,filePath,allowList}
+      #
+      # WHY THIS EXISTS. Plex's own log cannot show who connected. Every LoadBalancer Service in
+      # this cluster is `externalTrafficPolicy: Cluster`, so the node holding the L2 lease for the
+      # VIP SNATs the client before forwarding to the backend pod. Measured 2026-10-04: Plex had
+      # logged 3,214 requests from `10.0.1.x` (the lease holder k3s-prod-3's own address, Cilium
+      # identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No X-Forwarded-For
+      # either -- it serves raw tcp/32400, not HTTP through the Gateway.
+      #
+      # THE ADDRESS IS NOT LOST, ONLY UNRECORDED. Cilium observes the flow at ingress, BEFORE its
+      # own translation, so Hubble already has the true source. Measured on the lease holder:
+      #     192.168.2.133 -> 10.0.0.135   (real LAN client -> the Plex pod)
+      #     10.0.1.78     -> 10.0.0.135   (the same traffic, post-SNAT -- what Plex sees)
+      # What was missing is RETENTION: the ring buffer was 4095/4095 flows at ~303 flows/s, i.e.
+      # about THIRTEEN SECONDS of history, with hubble metrics disabled and no exporter configured.
+      #
+      # WHY NOT FIX THE SNAT INSTEAD. `externalTrafficPolicy: Local` would preserve the address in
+      # Plex's own log, but cilium 1.20's L2 announcer is explicitly incompatible with it -- upstream
+      # docs for this exact version: "incompatible with the externalTrafficPolicy: Local on services
+      # as it may cause service IPs to be announced on nodes without pods causing traffic drops",
+      # and "L2 announcements isn't currently aware of this setting". Live proof that this is not
+      # theoretical: the VIP is announced by k3s-prod-3 while the pod runs on k3s-prod-1, so the
+      # flip would black-hole tcp/32400 immediately. Making it safe would mean pinning BOTH the pod
+      # and the announcement to one node and giving up Plex's ability to reschedule -- a real
+      # availability loss to buy a log field. (It is the same trap platform/crafty/service-l4.yaml
+      # documents from an 8-day Minecraft outage.) DSR was also rejected: routing-mode here is
+      # vxlan, not native/geneve, and DSR returns bypass the ingress node, which breaks the SOHO
+      # router's stateful NAT on the remote path.
+      #
+      # `filePath: stdout` RATHER THAN A FILE, deliberately. Grafana Alloy already runs as a
+      # DaemonSet tailing every pod's logs into Loki (platform/monitoring/alloy.yaml), so stdout
+      # reaches Loki with NO new hostPath mount, no file-rotation handling and no Alloy change.
+      # Upstream documents it: "To get Hubble flows directly exported to the logs instead of written
+      # to a rotated file, `stdout` can be defined as `hubble-export-file-path`." fileMaxSizeMb /
+      # fileMaxBackups are therefore left at chart defaults and do not apply.
+      #   Query:  {namespace="kube-system", app="cilium"} |= "32400"
+      #
+      # SCOPED BY PORT, NOT BY NAMESPACE -- and that is a measured decision, not a preference. A
+      # `--to-namespace plex` query returned ONLY the post-SNAT flows; the flow carrying the real
+      # client address did NOT match it, because at the point of observation the destination
+      # identity is not yet resolved to the plex pod. Filtering by namespace would therefore drop
+      # exactly the record this change exists to capture. Port it is.
+      #
+      # Both directions are listed so the return path is captured too. Volume is small (a few
+      # hundred to a few thousand lines a day); if it ever becomes noisy, narrow it with
+      # `fieldMask` rather than by removing a filter.
+      export:
+        static:
+          enabled: true
+          filePath: stdout
+          allowList:
+            - '{"destination_port":["32400"]}'
+            - '{"source_port":["32400"]}'
    # --- L2 announcement for the ingress VIP on the flat LAN (docs/09, OQ-1 resolved) ---
    # [parity] MUST equal Ansible: l2announcements.enabled: true. IP pool +
    # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
@@ -38,6 +38,18 @@ hubble:
    enabled: true
  ui:
    enabled: true
+  # [parity] static flow export -> cilium-agent stdout -> Alloy -> Loki. Records the PRE-SNAT
+  # client address for Plex's raw tcp/32400, which Plex's own log cannot show (every LB Service here
+  # is externalTrafficPolicy: Cluster, so the VIP lease holder SNATs the client). Scoped BY PORT,
+  # not namespace: the flow carrying the real client IP does not match a namespace filter. Full
+  # rationale, measurements and the rejected alternatives are in helmrelease.yaml.
+  export:
+    static:
+      enabled: true
+      filePath: stdout
+      allowList:
+        - '{"destination_port":["32400"]}'
+        - '{"source_port":["32400"]}'
 # [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
 l2announcements:
  enabled: true
Author	SHA1	Message	Date
jakman85andClaude Opus 5	a0e6ef145f	fix(plex): unblock the app tier — drop the Certificate that stalled apps-wave-0, advertise http gate / gate (pull_request) Successful in 1m50s Details gate / conformance (pull_request) Has been skipped Details I broke the app tier with #48. Two faults, both mine, both measured after merge: 1. KYVERNO BLOCKS cert-manager IN APP NAMESPACES. The temporary private-key Secrets (plex-vip-<random>) were denied by secret-provenance/secrets-must-be-flux-managed, which matches every Secret in a namespace labelled homelab.keithjackson.dev/app-namespace=true and excludes only those labelled secret-managed-by=controller or carrying a Flux label. `plex` HAS that label; `gateway` -- where the working wildcard cert lives -- does NOT. That asymmetry is why the existing Certificate was never affected and mine was, and I did not check it. secretTemplate cannot fix it: those Secrets come from the keymanager controller, not the Certificate template. So no CertificateRequest, Order or Challenge was ever created. 2. FLUX HEALTH-CHECKS THE CERTIFICATE. `optional: true` did its job -- the pod rolled and ran fine without the Secret -- but apps-wave-0 is wait:true, so Flux waits on the Certificate OBJECT: timeout waiting for: [Certificate/plex/plex-vip status: 'InProgress'] apps-wave-0 never went Ready; wave-1/-2 and everything downstream of platform-config sat in DependencyNotReady behind it. I guarded the pod and missed the object. That distinction is the whole bug. Third, independent: the advertised URL was https with no servable certificate -- Plex answers 32400 with its *.plex.direct cert, which cannot cover plex-vip -- the exact failure mode of the outage earlier the same day. So: delete the Certificate (pruning it clears the health gate) and advertise http://plex-vip:32400, which is MEASURED to work today -- PiHole answers the name with the Plex VIP (verified against pihole-0) and Plex serves 32400 there. http is explicitly temporary and only tolerable because remote access does not work yet; the restore condition is written at the call site. DELETED, not merely unregistered: my first attempt kept the file as "tracked but not onboarded" and the gate rejected it -- tls-lifecycle-automated (R-TLS-02) fails any Certificate no Flux path reconciles. Legal for a Deployment, not for a TLS object. The re-enable recipe (Kyverno exemption + a wait:false Kustomization + the https flip, as ONE change) is recorded in catalog/plex.yaml. Kept and verified working from #48: the hostname design, PiHole's split-horizon answer, and the postStart hook -- LanNetworksBandwidth is now set live. Gate: rc 3, all substantive stages pass (first attempt was rc 1 BLOCKED). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 17:10:25 -05:00
jakman85	c163deaef9	Merge pull request 'feat(plex): one advertised hostname for local+remote, its own LE cert, and a LAN definition so LAN clients stop reading as remote' (#48 ) from feat/plex-vip-hostname-and-lan-networks into main gate / gate (push) Successful in 1m48s Details gate / conformance (push) Has been skipped Details Reviewed-on: #48 Reviewed-by: Keith Jackson <[email protected]>	2026-10-04 21:47:32 +00:00
jakman85	642a7752d8	Merge pull request 'feat(cilium): export Plex's pre-SNAT client address via Hubble, scoped to tcp/32400' (#50 ) from feat/hubble-flow-export-plex into main gate / conformance (push) Has been cancelled Details gate / gate (push) Has been cancelled Details Reviewed-on: #50	2026-10-04 21:47:20 +00:00
jakman85andClaude Opus 5	989649e64a	feat(cilium): export Plex's pre-SNAT client address via Hubble, scoped to tcp/32400 gate / gate (pull_request) Successful in 1m49s Details gate / conformance (pull_request) Has been skipped Details The requirement is log visibility only -- which address connected to Plex. Plex's own log cannot show it. Every LoadBalancer Service here is externalTrafficPolicy: Cluster, so the node holding the VIP's L2 lease SNATs the client before forwarding to the backend pod. Measured: Plex had logged 3,214 requests from 10.0.1.x (lease holder k3s-prod-3's own address, Cilium identity 1 = reserved:host) and ZERO from any 192.168.x address, ever. No X-Forwarded-For either -- it serves raw tcp/32400, not HTTP via the Gateway. Nothing on the router does this; there is no hairpin NAT. It is Cilium. The address is not lost, only unrecorded. Cilium observes the flow at ingress, BEFORE its own translation, so Hubble already has it: 192.168.2.133 -> 10.0.0.135 real LAN client -> the Plex pod 10.0.1.78 -> 10.0.0.135 same traffic post-SNAT, what Plex sees What was missing is RETENTION: 4095/4095 flows at ~303/s is about thirteen seconds of history, metrics disabled, no exporter configured. So enable the static exporter, port-scoped, to the agent's stdout -- Alloy already tails every pod into Loki, so this needs no new mount, no rotation handling and no Alloy change. {namespace="kube-system", app="cilium"} \|= "32400" Scoped BY PORT, not namespace, on evidence: --to-namespace plex returned only the post-SNAT flows; the record carrying the real client address did not match it. A namespace filter would drop exactly what this exists to capture. Rejected, with reasons: eTP:Local is incompatible with cilium 1.20's L2 announcer per upstream docs for this version ("may cause service IPs to be announced on nodes without pods causing traffic drops") -- and the VIP is on prod-3 while the pod is on prod-1, so it would black-hole 32400 immediately; making it safe costs Plex its ability to reschedule. DSR: routing-mode is vxlan, not native/geneve, and DSR returns bypass the ingress node, breaking the router's stateful NAT on the remote path. Also corrects the repo's own record: 2026-10-03 says the SNAT lands in 192.168.0.0/16 (the node's LAN address). It is 10.0.1.x, the cilium_host inside the pod CIDR -- which is why LanNetworksBandwidth can never separate local from remote while eTP is Cluster. UNVERIFIED and stated in the handoff: nothing is reconciled; `filePath: stdout` is taken from a verbatim upstream sentence but not confirmed against 1.20.1 source (a wrong notation is a silent no-op, with the file+Alloy fallback documented); the port FlowFilter has no upstream example; and this change ROLLS CILIUM -- prod-3 announces the Gateway, PiHole DNS and Plex VIPs, so a lease move during the rollout could briefly interrupt LAN DNS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 16:36:12 -05:00
jakman85	21c88b0dc7	Merge pull request 'fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable' (#49 ) from fix/oauth2-sidecar-metrics-scrape-netpol into main gate / gate (push) Successful in 1m47s Details gate / conformance (push) Has been skipped Details Reviewed-on: #49	2026-10-04 21:33:28 +00:00
jakman85andClaude Opus 5	b156ee6cfa	fix(obs): open 44180 to ns monitoring so the forward-auth gate sidecars are actually scrapable gate / gate (pull_request) Successful in 1m51s Details gate / conformance (pull_request) Has been skipped Details platform/oauth2-proxy/podmonitor.yaml landed 2026-10-02 and discovered six targets that have never once scraped. Measured on prod 2026-10-04: all six read health=down with lastError "context deadline exceeded" and lastScrapeDuration exactly 10.0s — the configured scrapeTimeout, i.e. packets DROPPED rather than refused. Cause: every app namespace ships a default-deny NetworkPolicy with policyTypes: [Ingress, Egress] and NO ingress key at all (deny-all-ingress), and the only ingress allowance is <app>-allow-gateway-ingress: fromEntities: [ingress] -> 4180. Prometheus matched no clause. Verified identical in all six of jackett, radarr, sonarr, qbittorrent, pihole, crafty; zero namespaces had any -allow-monitoring-scrape object. render.py already had the right idiom — r_monitoring_ingress_policy — but it returned None unless the service declared observability.metricsEndpoint, and NO ServiceDefinition in this catalog declares one. It was wired in at the call site and had never emitted a single object on any branch: correct, and entirely latent. It now also fires for the forward-auth gate sidecar's own metrics port, keyed on _needs_oauth2_sidecar — deliberately the same condition the scrape keys on, since podmonitor.yaml selects pods purely by the auth-mode label, so a target exists iff the sidecar exists. Keying on observability.metrics instead would let metrics: false produce a discovered-but-blocked target: the same "two halves authored apart" failure this function exists to prevent. Rendered here rather than in platform/oauth2-proxy because that tier would have to name app namespaces created by the sibling apps-wave-0/1 tiers — it would apply on a running cluster and fail on a cold bootstrap. podmonitor.yaml's header records that objection; rendering per-app dissolves it. Five CNPs result, one per forward-auth app, each opening only 44180 to only ns monitoring. apps/ re-rendered with the invocation it was generated by (--no-image-automation, verified byte-identical beforehand); --emit-waves deliberately not run, so no wave kustomization moved. Three L1 goldens are regenerated with it (+63 lines, -0): a renderer change that adds an object necessarily drifts the goldens that pin the renderer's output. The first gate run caught container-workload-render and egress-class-render as FAIL with exactly this CNP as the drift. default-deny-baseline-render's golden regenerated byte-identical, which is the evidence that the regen is faithful and not a rubber stamp. That puts conformance/** in the diff, which is a sensitive path: this PR needs a non-author approving review. The local gate cannot see that — offline it reports approval-policy green (N/A). Local toolbox gate: rc 3 UNDETERMINED, not 0. Every stage that ran is green and l1 went FAIL -> pass with the goldens; the only not-run stage is gitea-settings (no GITEA_SETTINGS_TOKEN). L2/L3 conformance is UNVERIFIED for all 8 touched requirements. Not verified: that the scrape now succeeds. Nothing is applied until Flux reconciles main. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>	2026-10-04 16:05:44 -05:00