Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  jakman85andClaude Opus 5
|
81f627167d | ||
|
jakman85
|
4386d1453b | ||
|
jakman85andClaude Opus 5
|
874c6703f0 | ||
|
jakman85
|
bc7095677b | ||
|
jakman85andClaude Opus 5
|
a0e6ef145f |
No files matched your search
@@ -18,7 +18,6 @@ resources:
|
||||
- pihole-dns.yaml
|
||||
- pihole-oauth2-proxy-credentials.sops.yaml
|
||||
- pihole.yaml
|
||||
- plex-cert.yaml
|
||||
- plex-claim-secret.sops.yaml
|
||||
- plex-remote.yaml
|
||||
- plex.yaml
|
||||
|
||||
@@ -1,66 +0,0 @@
|
||||
---
|
||||
# =================================================================================================
|
||||
# PLEX TLS — a Let's Encrypt certificate for the ONE hostname Plex advertises to plex.tv.
|
||||
# =================================================================================================
|
||||
# HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
|
||||
# and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
|
||||
# mechanism gitea-actions, pihole-dns and plex-remote use. It is hand-written for the same reason:
|
||||
# `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has NO field for a
|
||||
# cert-manager Certificate. (`ingressRoutes` gets TLS for free from the Gateway's wildcard; this is
|
||||
# the opposite case — a certificate for a workload that terminates its OWN TLS on a raw L4 port.)
|
||||
#
|
||||
# WHY PLEX NEEDS ITS OWN CERTIFICATE AT ALL. Plex serves tcp/32400 directly (not through the L7
|
||||
# Gateway — the operator chose the native path 2026-10-03, apps/wave-0/plex-remote.yaml) and it
|
||||
# presents its own `*.<serverhash>.plex.direct` certificate there. That name cannot cover
|
||||
# `plex-vip.${BASE_DOMAIN}`, the hostname we advertise, so without this the advertised `https://`
|
||||
# URL fails hostname verification for every client. That is not hypothetical: it is the shape of
|
||||
# the 2026-10-04 outage, where Plex's log carried a continuous storm of
|
||||
# `CERT: incomplete TLS handshake ... sslv3 alert certificate unknown` from real LAN clients while
|
||||
# plex.tv could publish nothing and the server vanished from the account.
|
||||
#
|
||||
# A SAN match is not what makes Plex FUNCTION — `httpsRequired=0` and `secureConnections` default
|
||||
# to Preferred, so clients can fall back to cleartext. It is what keeps the REMOTE leg encrypted
|
||||
# rather than silently downgrading traffic that crosses the internet.
|
||||
#
|
||||
# HOW IT REACHES PLEX. catalog/plex.yaml takes `tls.crt`/`tls.key` from this Secret as env
|
||||
# (`optional: true` — see below), and its `lifecycle.postStart` hook converts the PEM into the
|
||||
# PKCS#12 bundle Plex's `customCertificatePath` requires and sets the three `customCertificate*`
|
||||
# prefs. The conversion is redone on EVERY pod start, so a cert-manager RENEWAL is picked up with
|
||||
# no extra moving part and nothing to remember.
|
||||
#
|
||||
# ORDERING, AND WHY THE ENV REFS ARE OPTIONAL. This Certificate must live in the `plex` namespace
|
||||
# for the Deployment to read its Secret, and that namespace is created by the rendered
|
||||
# apps/wave-0/plex.yaml ITSELF — so this cannot be issued in an earlier tier (platform-config runs
|
||||
# before the namespace exists) and lands in the SAME reconcile as the workload. On a cold bootstrap
|
||||
# the Secret is therefore absent for as long as a DNS01 challenge takes. With hard env refs Plex
|
||||
# would sit in CreateContainerConfigError until then, and because `apps-wave-0` is `wait: true`
|
||||
# with `apps-wave-1` dependsOn it, a failed or slow issuance would STALL THE WHOLE APP TIER. Hence
|
||||
# `optional: true` on both refs (schema 1.11.0, added for exactly this): no certificate means Plex
|
||||
# still starts and serves on its own plex.direct cert, the hook logs that TLS was not installed,
|
||||
# and the next start after issuance converges.
|
||||
#
|
||||
# ISSUER: ClusterIssuer `letsencrypt-prod` (platform/cert-manager/clusterissuers.yaml), the same
|
||||
# issuer behind the Gateway's wildcard (platform/gateway/wildcard-certificate.yaml, Ready 28d on
|
||||
# prod as measured 2026-10-04). A SINGLE name is requested rather than reusing the wildcard,
|
||||
# because the wildcard's Secret lives in the `gateway` namespace and a Secret cannot be read across
|
||||
# namespaces; requesting `*.${BASE_DOMAIN}` again here would also burn a duplicate-certificate slot
|
||||
# against Let's Encrypt's weekly limit for no benefit. A single name is the least-privilege shape:
|
||||
# this key signs for `plex-vip.${BASE_DOMAIN}` and nothing else.
|
||||
apiVersion: cert-manager.io/v1
|
||||
kind: Certificate
|
||||
metadata:
|
||||
name: plex-vip
|
||||
namespace: plex
|
||||
labels:
|
||||
app.kubernetes.io/name: plex
|
||||
spec:
|
||||
# The name Plex ADVERTISES and the name on this certificate must stay identical. They are two
|
||||
# halves of one decision: catalog/plex.yaml's ADVERTISE_IP, the postStart hook's
|
||||
# customCertificateDomain, this dnsNames entry, and catalog/pihole.yaml's split-horizon answer.
|
||||
# Nothing enforces that they agree.
|
||||
dnsNames:
|
||||
- plex-vip.${BASE_DOMAIN}
|
||||
secretName: plex-vip-tls
|
||||
issuerRef:
|
||||
kind: ClusterIssuer
|
||||
name: letsencrypt-prod
|
||||
@@ -138,7 +138,7 @@ spec:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
- name: ADVERTISE_IP
|
||||
value: https://plex-vip.${BASE_DOMAIN}:32400
|
||||
value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
|
||||
- name: PLEX_TLS_CRT
|
||||
valueFrom:
|
||||
secretKeyRef:
|
||||
|
||||
+46
-30
@@ -72,43 +72,59 @@ env:
|
||||
secretKeyRef:
|
||||
name: plex-claim
|
||||
key: PLEX_CLAIM
|
||||
# ADVERTISE_IP ("Custom server access URLs") — the ONE hostname this server publishes to plex.tv,
|
||||
# serving BOTH local and remote clients by split-horizon DNS (operator design, 2026-10-04):
|
||||
# LAN -> PiHole answers plex-vip.${BASE_DOMAIN} with ${PLEX_L4_VIP} (the tcp/32400 LoadBalancer
|
||||
# in apps/wave-0/plex-remote.yaml). See catalog/pihole.yaml's split-horizon line, which
|
||||
# needs a per-host override because the zone wildcard otherwise points at the GATEWAY
|
||||
# VIP — measured 2026-10-04: plex-vip.${BASE_DOMAIN} resolved to the Gateway, where
|
||||
# nothing listens on 32400.
|
||||
# PUBLIC -> Cloudflare answers with the WAN address, NATed to the same VIP (off-repo).
|
||||
# ADVERTISE_IP ("Custom server access URLs") — the connection Plex publishes to plex.tv.
|
||||
#
|
||||
# WHY A HOSTNAME AND NOT THE `plex.direct` FORM THIS BRIEFLY USED. The first fix for the
|
||||
# 2026-10-04 outage advertised `https://<dashed-ip>.<serverhash>.plex.direct:32400`, which works
|
||||
# (and is Plex's own mechanism), but it hard-codes BOTH the VIP and the server's certificate hash
|
||||
# into a cluster-var — two values nothing could keep in sync, with no alert if either drifted. A
|
||||
# hostname in our own zone removes that whole class of problem: ${BASE_DOMAIN} is already a knob,
|
||||
# the VIP lives only in PiHole's answer, and `PLEX_ADVERTISE_URL` is deleted outright.
|
||||
# THE VALUE MUST BE RESOLVABLE BY EVERY CLIENT, NOT JUST BY CLIENTS THAT USE OUR DNS. That single
|
||||
# property is what this field keeps getting wrong, so it is the first thing to check before
|
||||
# changing it. A Plex client resolves the advertised host ITSELF, with whatever resolver it
|
||||
# happens to use — and on a real LAN that is frequently NOT PiHole: browsers ship DoH on by
|
||||
# default, phones have "private DNS", guest and IoT devices often take the router's DNS, and some
|
||||
# clients hard-code a public resolver.
|
||||
#
|
||||
# THIS IS `https`, WHICH ONLY WORKS BECAUSE WE HAND PLEX OUR OWN CERTIFICATE. Plex serves 32400
|
||||
# with its `*.<serverhash>.plex.direct` cert, which does NOT cover this name — advertising
|
||||
# `https://plex-vip...` against the stock cert reproduces the exact failure that caused the
|
||||
# outage (measured: a storm of `CERT: incomplete TLS handshake ... certificate unknown` from real
|
||||
# clients, and `curl` without `-k` failing to connect). The `lifecycle.postStart` hook below
|
||||
# installs the Let's Encrypt cert for this name and points Plex at it, so the advertised URL and
|
||||
# the served certificate agree. Do not change this to a bare IP, and do not drop the hook and
|
||||
# leave the `https` here.
|
||||
# THAT IS NOT THEORETICAL — IT TOOK PLEX DOWN FOR THE WHOLE LAN ON 2026-10-05. This field briefly
|
||||
# advertised `plex-vip.${BASE_DOMAIN}`, a name that exists ONLY in PiHole's split-horizon answers
|
||||
# (catalog/pihole.yaml) and in no public zone. Clients that did not use PiHole could not resolve
|
||||
# it; the SOHO router answered the failed lookup with ITSELF, and every one of them landed on the
|
||||
# router's own web UI and reported a certificate error for `CN = %D`. Measured 2026-10-05: the
|
||||
# router serves that self-signed cert (`C = ZZ, O = %D342e7e6b, CN = %D`) on BOTH `192.168.2.1:443`
|
||||
# and the WAN address, while PiHole itself was answering perfectly — both replicas returned
|
||||
# `plex-vip -> ${PLEX_L4_VIP}` and `plex -> ${GATEWAY_VIP}`. The DNS was right; the DEPENDENCY on
|
||||
# it was the bug. Nothing Plex advertises listens on 443, which is how we know the clients had left
|
||||
# the Plex path entirely and were talking to the router.
|
||||
#
|
||||
# (A SAN match is not what makes Plex *work* — `httpsRequired=0` and `secureConnections` default
|
||||
# to Preferred, so clients can fall back to cleartext. It is what keeps the remote leg ENCRYPTED
|
||||
# instead of silently downgrading traffic that crosses the internet.)
|
||||
# SO: `<dashed-ip>.<serverhash>.plex.direct`, WHICH IS RESOLVER-INDEPENDENT. This is Plex's own
|
||||
# mechanism and the reason it exists: `plex.direct` is a PUBLIC zone that returns the private
|
||||
# address embedded in the label, so EVERY resolver on earth — PiHole, the router, 1.1.1.1, a DoH
|
||||
# browser — answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no
|
||||
# split-horizon entry, no Cloudflare record, and no certificate work, because the wildcard cert
|
||||
# Plex already serves on 32400 covers exactly this name. Re-verified live 2026-10-05 with FULL
|
||||
# validation (no `curl -k`): resolves to 192.168.2.24 and `/identity` returns **200, TLS verified**.
|
||||
#
|
||||
# WHY NOT THE NICER HOSTNAME, since a readable name in our own zone is obviously more pleasant:
|
||||
# it cannot be made to work for clients that bypass PiHole. A public record for it would resolve
|
||||
# to the WAN address, and with no NAT hairpin on this router a LAN client sent there fails too —
|
||||
# so the hostname is correct ONLY for clients using PiHole, and wrong for the rest. There is no
|
||||
# configuration of that name that is right for both. `plex.direct` is right for both.
|
||||
# (The PiHole `plex-vip` answer is KEPT — it is harmless, costs nothing, and is genuinely useful
|
||||
# for reaching the VIP by name from a LAN shell. It is simply not what Plex advertises.)
|
||||
#
|
||||
# THE TWO EMBEDDED VALUES, and the honest cost of this choice. The dashed IP must track
|
||||
# ${PLEX_L4_VIP} and `<serverhash>` is this server's certificate identity — read it from the live
|
||||
# listener with
|
||||
# openssl s_client -connect <vip>:32400 </dev/null | openssl x509 -noout -subject
|
||||
# Nothing in this tree can derive either, and nothing checks that they still agree: envsubst
|
||||
# cannot dash an IP, so this is a literal. That fragility is real and was the reason the hostname
|
||||
# was tried in the first place — it is accepted here because an unresolvable name is a WORSE
|
||||
# failure (silent, LAN-wide, and it points users at the router) than a stale literal, which at
|
||||
# least fails loudly and only after someone changes the VIP.
|
||||
#
|
||||
# The image's /etc/cont-init.d/40-plex-first-run runs `setPref "customConnections"
|
||||
# "${ADVERTISE_IP}"` (line 127, verified in the running image), OUTSIDE the
|
||||
# PLEX_CLAIM/PlexOnlineToken guard, and its `/.firstRunComplete` marker is on the container
|
||||
# OVERLAY fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not
|
||||
# a web form, owns the value. Hand-typed custom URLs are what broke this service; they are now
|
||||
# overwritten on every start.
|
||||
# PLEX_CLAIM/PlexOnlineToken guard, with its `/.firstRunComplete` marker on the container OVERLAY
|
||||
# fs rather than the /config PVC — so this is re-applied on EVERY pod start and git, not a web
|
||||
# form, owns the value.
|
||||
- name: ADVERTISE_IP
|
||||
value: https://plex-vip.${BASE_DOMAIN}:32400
|
||||
value: https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400
|
||||
# THE TLS MATERIAL FOR THE NAME ABOVE, as PEM text in env rather than a mounted volume.
|
||||
# `plex-vip-tls` is issued by cert-manager from ClusterIssuer `letsencrypt-prod` for exactly
|
||||
# `plex-vip.${BASE_DOMAIN}` (apps/wave-0/plex-cert.yaml, same wave => same reconcile as this
|
||||
|
||||
+14
-24
@@ -156,11 +156,17 @@ OAUTH2_COOKIE_REFRESH = "2m30s"
|
||||
# ready endpoints. The flag is confirmed present in the PINNED v7.8.1 binary (`--help`),
|
||||
# which matters because the shared proxy is on v7.15.3 and could not have vouched for it.
|
||||
#
|
||||
# THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED. A container port is not a
|
||||
# Service port: these sidecars' Services expose the app port and :4180, and each namespace's
|
||||
# default-deny baseline has `ingress: []`, so Prometheus can neither resolve nor reach this
|
||||
# port until a Service port (or PodMonitor), a monitoring-ingress allow and a ServiceMonitor
|
||||
# are added. Do not read this flag as "metrics are working".
|
||||
# THE SCRAPE WORKS AS OF 2026-10-05 — but it took THREE independent pieces, and this flag is
|
||||
# only the first. A container port is not a Service port, and each namespace's default-deny
|
||||
# baseline denies all ingress, so the flag alone bought nothing for 63.7h:
|
||||
# 1. this flag + a named `metrics` container port (PR #28)
|
||||
# 2. a monitor that targets POD ports, not Service ports — platform/oauth2-proxy/podmonitor.yaml
|
||||
# is a PodMonitor for exactly that reason (no Service exposes 44180)
|
||||
# 3. an ingress allowance from ns monitoring (PR #49, r_monitoring_ingress_policy below)
|
||||
# Verified on prod 2026-10-05: all 6 targets up == 1, scraping in 1-6ms.
|
||||
# STILL do not read this flag alone as "metrics are working" — 1 without 3 is precisely the state
|
||||
# that produced 5 x TargetDown for 63.7h. If you add a sidecar metrics port anywhere else, check
|
||||
# that r_monitoring_ingress_policy covers it.
|
||||
OAUTH2_PROXY_METRICS_PORT = 44180
|
||||
|
||||
# --- Hardware capability: `hardware.gpu: intel-quicksync` ------------------------------
|
||||
@@ -1362,8 +1368,9 @@ def _oauth2_sidecar_container(s: dict) -> dict:
|
||||
f"--cookie-refresh={OAUTH2_COOKIE_REFRESH}",
|
||||
"--pass-access-token=true",
|
||||
"--set-xauthrequest=true",
|
||||
# Start the /metrics listener. Prerequisite only — not yet reachable by Prometheus;
|
||||
# see OAUTH2_PROXY_METRICS_PORT above for exactly what is still missing.
|
||||
# Start the /metrics listener. Scraped as of 2026-10-05, but only because
|
||||
# r_monitoring_ingress_policy also opens this port to ns monitoring — the flag alone was
|
||||
# inert for 63.7h. See OAUTH2_PROXY_METRICS_PORT above for the three pieces required.
|
||||
f"--metrics-address=0.0.0.0:{OAUTH2_PROXY_METRICS_PORT}",
|
||||
]
|
||||
secret = f"{s['name']}-oauth2-proxy"
|
||||
@@ -2209,23 +2216,6 @@ EXTRA_UNITS: dict = {
|
||||
"plex-remote.yaml",
|
||||
],
|
||||
},
|
||||
# Plex's own TLS certificate for the single hostname it advertises to plex.tv
|
||||
# (apps/wave-0/plex-cert.yaml). `above: False` co-locates it in plex's OWN wave, which is the
|
||||
# only placement available: the Certificate must land in the `plex` namespace, and that
|
||||
# namespace is created by the rendered plex.yaml in this very wave, so it CANNOT be issued
|
||||
# earlier. The workload tolerates the resulting gap by marking the env refs `optional` rather
|
||||
# than by ordering -- see the header of plex-cert.yaml.
|
||||
#
|
||||
# Non-catalog because the ServiceDefinition schema has no field for a cert-manager Certificate
|
||||
# (additionalProperties is false); `ingressRoutes` inherits the Gateway wildcard instead, which
|
||||
# is the opposite case to a workload terminating its own TLS on a raw L4 port.
|
||||
"plex-cert": {
|
||||
"anchor": "plex",
|
||||
"above": False,
|
||||
"files": [
|
||||
"plex-cert.yaml",
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -4,6 +4,11 @@ branch: fix/oauth2-sidecar-metrics-scrape-netpol
|
||||
title: The forward-auth gate sidecars' /metrics was NetworkPolicy-blocked — the renderer now opens 44180 to ns monitoring
|
||||
---
|
||||
|
||||
> **Added 2026-10-05, after merge:** the top item under *What I did NOT verify* below — "that the
|
||||
> scrape now succeeds" — **has since been verified on prod**. All 6 targets read `up == 1`. The
|
||||
> original text is left exactly as written; the measurements are in
|
||||
> [[2026-10-05-oauth2-sidecar-scrape-verified]].
|
||||
|
||||
## What changed
|
||||
|
||||
`catalog/render.py`'s `r_monitoring_ingress_policy` now emits its CiliumNetworkPolicy for a
|
||||
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
date: 2026-10-04
|
||||
branch: fix/plex-unblock-app-tier
|
||||
title: Unblock the app tier — my Certificate stalled apps-wave-0, and the advertised https URL had no servable certificate
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**I broke the app tier with PR #48 and this reverts the part that did it.** Two faults, both mine,
|
||||
both measured after the merge:
|
||||
|
||||
1. **Kyverno blocks cert-manager in app namespaces.** The temporary private-key Secrets
|
||||
(`plex-vip-<random>`) were denied by `secret-provenance/secrets-must-be-flux-managed`:
|
||||
|
||||
resource Secret/plex/plex-vip-rp4qj was blocked due to the following policies
|
||||
secret-provenance:
|
||||
secrets-must-be-flux-managed: Secrets in application namespaces must be delivered by
|
||||
Flux from SOPS-encrypted git ... or be exempt controller-generated Secrets.
|
||||
|
||||
The policy matches every Secret in a namespace labelled
|
||||
`homelab.keithjackson.dev/app-namespace: true` and excludes only Secrets labelled
|
||||
`homelab.keithjackson.dev/secret-managed-by: controller` or carrying a Flux label. **The `plex`
|
||||
namespace has that label; the `gateway` namespace — where the working wildcard cert lives — does
|
||||
NOT.** That asymmetry is why the existing Certificate was never affected and mine was, and I did
|
||||
not check it. cert-manager's temp key Secrets carry neither exempting label, and `secretTemplate`
|
||||
cannot label them (the keymanager controller creates them, not the Certificate template). Result:
|
||||
**no CertificateRequest, Order or Challenge was ever created.**
|
||||
|
||||
2. **Flux health-checks the Certificate — guarding the pod was not enough.** `optional: true` did
|
||||
exactly what it was designed to do: Plex rolled and ran fine without the Secret (verified). But
|
||||
`apps-wave-0` is `wait: true`, so Flux waits on every applied object with a known health type:
|
||||
|
||||
health check failed after 5m0.045564329s: timeout waiting for:
|
||||
[Certificate/plex/plex-vip status: 'InProgress']
|
||||
|
||||
apps-wave-0 never went Ready, and apps-wave-1/-2 plus everything downstream of platform-config
|
||||
sat in `DependencyNotReady` behind it. I reasoned about the pod and missed the object.
|
||||
|
||||
A third, independent problem the same PR introduced: **the advertised URL was `https://` with no
|
||||
servable certificate.** Plex answers 32400 with its `*.<serverhash>.plex.direct` cert, which cannot
|
||||
cover `plex-vip.${BASE_DOMAIN}` — the precise failure mode of the outage earlier that day.
|
||||
|
||||
- **`apps/wave-0/plex-cert.yaml` DELETED**, and `plex-cert.yaml` removed from
|
||||
`apps/wave-0/kustomization.yaml` `resources`, so Flux prunes the stuck Certificate and the health
|
||||
gate clears. **Deleted rather than merely unregistered** — my first attempt kept the file as
|
||||
"tracked but not onboarded" (the shape `keycloak.yaml` and `vm-demo.yaml` use) and **the repo's own
|
||||
gate rejected it**: `tls-lifecycle-automated` (R-TLS-02) fails any Certificate in the tree that no
|
||||
Flux path reconciles — *"nothing reconciles it, so a deleted Certificate/issuer/controller would
|
||||
stay deleted until a human re-applied it."* That pattern is legal for a Deployment and not for a
|
||||
TLS object, which is a distinction I had not known.
|
||||
- **`catalog/render.py`:** the now-dangling `plex-cert` EXTRA_UNITS entry removed.
|
||||
- **The re-enable recipe is recorded at the call site** — in `catalog/plex.yaml` beside the scheme
|
||||
it is coupled to — rather than in the deleted file: the Kyverno exemption, the non-gating
|
||||
Kustomization, and the `https` flip, as one change.
|
||||
- **`catalog/plex.yaml`:** `ADVERTISE_IP` → **`http://`**`plex-vip.${BASE_DOMAIN}:32400`, explicitly
|
||||
temporary, with the restore condition written at the call site.
|
||||
|
||||
**Everything else from PR #48 is kept and working** — the hostname design, the PiHole split-horizon
|
||||
answer, the `LanNetworksBandwidth` hook, the `optional` schema field, and the TLS env (inert without
|
||||
the Secret, so re-enabling is additive).
|
||||
|
||||
## What I verified
|
||||
|
||||
Live prod, 2026-10-04, read-only, after the #48/#50 merges.
|
||||
|
||||
- **The stall and its exact cause**: the health-check timeout message above, and the full dependency
|
||||
cascade (`cilium` → "dependency 'flux-system/flux-system' revision is not up to date", everything
|
||||
else `DependencyNotReady`). The root `cilium` wait was a normal revision-gated cascade; the
|
||||
*Certificate* was the real blocker.
|
||||
- **The Kyverno denial text, verbatim**, from the cert-manager controller log, repeating every ~30s
|
||||
across at least 15 distinct Secret names.
|
||||
- **The policy's own structure**, read from `platform/kyverno/admission/secret-provenance.yaml`:
|
||||
match = Secrets in `app-namespace: true` namespaces; exclude = `secret-managed-by: controller` OR
|
||||
a `kustomize.toolkit.fluxcd.io/name` label; validate = unconditional `deny`.
|
||||
- **The namespace-label asymmetry**: `plex` carries
|
||||
`homelab.keithjackson.dev/app-namespace: "true"`; `gateway` does not.
|
||||
- **cert-manager's controller ServiceAccount is `cert-manager`** in namespace `cert-manager` — the
|
||||
identity a future exemption should name.
|
||||
- **`optional: true` worked**: the new pod (`plex-c4f87c84d-zgjq5`) is `1/1 Running` with no Secret
|
||||
present, and `/config/certs` was never created.
|
||||
- **The postStart hook RAN and took effect** — the thing the whole hook was written for:
|
||||
`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is now set in the live `Preferences.xml`, and
|
||||
`customConnections="https://plex-vip.home.keithjackson.dev:32400"` was applied from the env. The
|
||||
certificate prefs are correctly **ABSENT**, i.e. the hook's "no bundle → leave them alone" guard
|
||||
behaved as designed.
|
||||
- **PiHole's split-horizon answer is correct**, asked of `pihole-0` directly:
|
||||
`plex-vip.home.keithjackson.dev -> 192.168.2.24`, alongside `plex -> .20` and `crafty-mc -> .22`,
|
||||
and the live `/etc/pihole/dnsmasq.conf` carries
|
||||
`address=/plex-vip.home.keithjackson.dev/192.168.2.24`. **This corrects my own earlier check**,
|
||||
which read `.20` and which I nearly reported as a failure — that test resolved via CLUSTER DNS
|
||||
(CoreDNS) from inside a pod and never consulted PiHole at all.
|
||||
- **The gate caught my first attempt and I fixed it rather than working around it.** Keeping the
|
||||
file unregistered produced `MERGE GATE: BLOCKED` (rc 1) with
|
||||
`CHECK FAIL (tls-lifecycle-automated): 1 of 612 assertions failed`. After deleting the file and
|
||||
the dangling EXTRA_UNITS entry: **rc 3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
|
||||
`render-golden`, `l1` and `approval-policy` all **pass** — the rc 3 being only the two known
|
||||
environmental skips (`gitea-settings`, `conformance` L2/L3). **Not green. Reported as rc 3.**
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **Whether this actually unblocks the tier.** Unreconciled at time of writing. The reasoning is
|
||||
that pruning the Certificate removes the only failing health check, but I have not seen
|
||||
apps-wave-0 go Ready.
|
||||
- **Whether `http://` restores client connectivity.** PiHole resolves the name and Plex serves 32400
|
||||
(both measured), but I have not watched a Plex client complete a connection over the advertised
|
||||
URL, and I did not re-check whether plex.tv publishes it.
|
||||
- **THE REMOTE BADGE — the symptom that started this.** `LanNetworksBandwidth` is set, but the log
|
||||
still tags traffic `Allowed Network (WAN)`: **82 WAN, 0 LAN**, with entries continuing after the
|
||||
pref was written. So either Plex needs a restart for it to take effect, or that log tag is not
|
||||
driven by this pref at all. **I do not know which, and the user-visible badge is the only real
|
||||
test** — it is in the Plex client, which I cannot see.
|
||||
- **The Hubble flow export is NOT active.** The cilium agents are still the pre-merge pods, so the
|
||||
`filePath: stdout` question is still open and no flow has been exported. The Cilium rollout — and
|
||||
its LAN-DNS risk, since prod-3 holds the Gateway, PiHole-DNS and Plex VIPs — is still AHEAD.
|
||||
- **Whether a cert-manager exemption is the right shape**, or whether it widens
|
||||
`secret-provenance` more than the operator wants. It is `platform/kyverno/**`, a sensitive path,
|
||||
and deliberately not attempted here.
|
||||
- **Whether the pruned Certificate leaves residue** — I did not check for orphaned
|
||||
`plex-vip-<random>` Secrets or CertificateRequests after the prune. Note Kyverno DENIED every one
|
||||
of those Secrets, so there may be nothing to clean, but that is an inference, not a check.
|
||||
- **Whether `tls-lifecycle-automated` would also have objected to the Certificate living in a
|
||||
`wait: false` Kustomization** (the recommended re-enable shape). It demands a reconciling Flux
|
||||
path, which that provides, but I did not test it.
|
||||
- **The unencrypted advertised URL.** `http://` is tolerable only while remote access does not work
|
||||
(no WAN forward). Nothing enforces that assumption, and nothing will complain if `http` is still
|
||||
there the day the port-forward appears.
|
||||
@@ -0,0 +1,90 @@
|
||||
---
|
||||
date: 2026-10-05
|
||||
branch: docs/oauth2-sidecar-scrape-verified
|
||||
title: The gate-sidecar scrape is verified working on prod — and four places in the tree still said it wasn't
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
Documentation only. No manifest, no rendered output, no behaviour.
|
||||
|
||||
PR #49 merged as `21c88b0` and the fix works, but four places in the tree still asserted the
|
||||
*pre-fix* state as current — in the present tense, in capitals, in the files a reader would consult
|
||||
first. This is the stale-documentation failure mode this repo has been burned by twice already
|
||||
(`git push origin main` in the runbooks, "append to DECISIONS-LOG.md" in the old handoffs), so:
|
||||
|
||||
- **`platform/oauth2-proxy/podmonitor.yaml`** — the banner read *"READ THIS BEFORE BELIEVING THE
|
||||
TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY"* and *"SO THE TARGETS BELOW WILL
|
||||
READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS"*. Both now false, and the first one actively
|
||||
instructs the reader to disbelieve a correct `up == 1`. Replaced with a `STATUS: THE SCRAPE WORKS`
|
||||
block carrying the post-fix numbers; the diagnosis is kept below it under an explicit `HISTORY`
|
||||
heading, because it is why the fix is shaped the way it is. Added the drop fingerprint to
|
||||
recognise if it regresses — `lastScrapeDuration` pinned at *exactly* the `scrapeTimeout` with
|
||||
`context deadline exceeded`, versus 1–6 ms when healthy.
|
||||
- **`catalog/render.py:159`** — *"THIS IS THE PREREQUISITE ONLY, AND IT IS NOT YET SCRAPED … Do not
|
||||
read this flag as 'metrics are working'."* Rewritten to enumerate the three pieces a working
|
||||
scrape needs and which PR landed each. The warning is **kept, deliberately**: the flag alone
|
||||
really is insufficient, and "1 without 3" is exactly what caused the 63.7h outage. Only the
|
||||
"not yet scraped" claim was false.
|
||||
- **`catalog/render.py:1365`** — the sidecar arg comment, same correction, one line.
|
||||
- **`platform/oauth2-proxy/README.md`** — its two observability subsections are *dated*
|
||||
("measured 2026-09-30", "Update 2026-10-01"), so they are honest records rather than lies. But a
|
||||
reader hits the stale body long before any correction, so a `> SUPERSEDED` pointer now sits
|
||||
directly under the heading, and an `### Update 2026-10-05` section at the end carries the
|
||||
before/after table, the TSDB series counts and the alert outcome.
|
||||
|
||||
Two substantive errors fixed while there, not just tense:
|
||||
|
||||
1. The README's *"To close the gap for an app, three things are needed"* listed **a Service port
|
||||
and a ServiceMonitor**. Wrong — a PodMonitor targets pod ports directly, which is the whole
|
||||
reason no Service exposes 44180 and the renderer needed no Service change. Following that
|
||||
paragraph would have sent someone to edit `catalog/render.py`'s Service rendering for no reason.
|
||||
2. `podmonitor.yaml` ended by deferring the fix to *"the catalog/renderer, which is in flight on
|
||||
another branch"* — that branch does not exist and never did; the work happened in PR #49. It now
|
||||
names `r_monitoring_ingress_policy` directly, and a new `CRAFTY` section explains why crafty is
|
||||
the one namespace this file cannot help.
|
||||
|
||||
## What I verified
|
||||
|
||||
- **The fix itself, on prod 2026-10-05, read-only** (this is the evidence the docs now quote):
|
||||
Flux applied `21c88b0` through the real path — `apps-wave-0` `lastAppliedRevision` at 21:37:01Z,
|
||||
`apps-wave-1` at 21:37:46Z, in dependency order — and all five `<app>-allow-monitoring-scrape`
|
||||
CNPs exist, each `ports=['44180']`, `from=[['monitoring']]`.
|
||||
- **All 6 targets `up == 1`**, `health=up`, `lastError=''`, `lastScrapeDuration` 0.001–0.006s —
|
||||
against 10.0s and `context deadline exceeded` before.
|
||||
- **`oauth2_proxy_*` is in the TSDB for these namespaces for the first time:** 5 metric names,
|
||||
148 series; `sum by (namespace) (oauth2_proxy_requests_total)` = qbittorrent 35072, radarr 23560,
|
||||
sonarr 1284. (Queried with `count by (__name__)` plus a matcher, not label-values.)
|
||||
- **Alerts:** all 5 `TargetDown` and 5 of 6 `OAuth2ProxyGateUnobserved` resolved; one series
|
||||
remains, `OAuth2ProxyGateUnobserved{namespace="crafty"}`, firing.
|
||||
- **The rendered output is byte-identical.** `catalog/render.py` changed only in comments, so this
|
||||
branch touches no `apps/` file and no golden. Confirmed by `render-golden` and `l1` both passing
|
||||
with no regeneration, and by `git status` showing nothing under `apps/`.
|
||||
- **This branch touches no sensitive path**, unlike PR #49: `platform/oauth2-proxy/**`,
|
||||
`catalog/**` and `decisions/**` are none of them in `infra/gitea/approval-policy.json`. So it
|
||||
should merge on a green gate with no human approval — the opposite of the PR it documents, and
|
||||
the reason the two were kept apart.
|
||||
- `render.py --selftest`: PASS.
|
||||
- Toolbox gate: see the handoff's own rule — reported below rather than claimed here.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **I did not re-measure the 63.7h figure or the ~22 emails/day rate**; both are carried forward
|
||||
from the 2026-10-04 measurements, now quoted inside the docs as historical. The 15-day
|
||||
`query_range` that established "one episode, not flapping" was run once, on 2026-10-04.
|
||||
- **`oauth2_proxy_requests_total` is still absent for jackett and pihole.** I assert in the README
|
||||
that this is because the counter is not created until the gate serves its first request. That is
|
||||
the documented behaviour of a Prometheus counter with no initialisation, and it is consistent
|
||||
with the other three namespaces having traffic — but I did not force a request through jackett's
|
||||
gate and watch the series appear. If it is still absent after real use, that is a finding.
|
||||
- **The auth-churn rule is still unwritten.** The README now says it is "writable rather than
|
||||
decorative"; that is a claim about the metrics being present, not a rule anyone has authored or
|
||||
unit-tested. The thing the 2026-09-30 incident actually needed does not exist yet.
|
||||
- **crafty remains unobserved.** I read its container spec on 2026-10-05 and confirmed no
|
||||
`--metrics-address`; I did not attempt the fix, and nothing here changes its alert.
|
||||
- **I did not audit the whole tree for other stale "metrics are not scraped" claims** beyond the
|
||||
four sites a targeted grep found (`not yet reachable`, `UNTIL AN INGRESS`, `up == 0`,
|
||||
`not reachable by Prometheus` and neighbours). `runbooks/oauth2-proxy-auth-failures.md` is
|
||||
described in the README as "the by-hand substitute meanwhile" and I left it alone — the by-hand
|
||||
log procedure still works and is not wrong, but whether it should now be rewritten against the
|
||||
new metrics is unexamined.
|
||||
@@ -0,0 +1,105 @@
|
||||
---
|
||||
date: 2026-10-05
|
||||
branch: fix/plex-resolver-independent-advertise
|
||||
title: Plex down LAN-wide — the advertised name existed only in PiHole, so clients that bypass it landed on the router's %D cert
|
||||
---
|
||||
|
||||
## What changed
|
||||
|
||||
**I took Plex down for the whole LAN.** Reported symptom: every internal client showed a certificate
|
||||
error with `%D` as the subject name and could not connect.
|
||||
|
||||
**Root cause: the advertised connection depended on our own resolver.** `ADVERTISE_IP` was
|
||||
`http://plex-vip.${BASE_DOMAIN}:32400` — a name that exists ONLY in PiHole's split-horizon answers
|
||||
and in no public zone. A Plex client resolves the advertised host *itself*, with whatever resolver it
|
||||
happens to use, and on a real LAN that is frequently not PiHole: browsers ship DoH on by default,
|
||||
phones have "private DNS", guest/IoT devices take the router's DNS. Those clients could not resolve
|
||||
the name, the SOHO router answered the failed lookup with **itself**, and they all landed on the
|
||||
router's web UI.
|
||||
|
||||
| Measured 2026-10-05 | Result |
|
||||
|---|---|
|
||||
| `192.168.2.1:443` (router, LAN side) | `C = ZZ, ST = Somewhere, O = %D342e7e6b, CN = %D` |
|
||||
| `136.51.115.252:443` (router, WAN side) | the same `%D` cert |
|
||||
| PiHole **pihole-0** | `plex -> 192.168.2.20`, `plex-vip -> 192.168.2.24` ✅ |
|
||||
| PiHole **pihole-1** | identical ✅ (both behind the `.23` VIP) |
|
||||
| Gateway `.20:443` cert | `CN = *.home.keithjackson.dev`, valid to 2026-12-12 ✅ |
|
||||
| `http://192.168.2.24:32400/identity` | `200` ✅ |
|
||||
| Plex's published connections | all port **32400** — nothing on 443 |
|
||||
|
||||
**The DNS was right; the dependency on it was the bug.** PiHole answered correctly from both
|
||||
replicas the entire time. And because nothing Plex advertises listens on 443, the clients had plainly
|
||||
left the Plex path altogether and were talking to the router.
|
||||
|
||||
**The fix: go back to `<dashed-ip>.<serverhash>.plex.direct`, which is RESOLVER-INDEPENDENT.** This
|
||||
is Plex's own mechanism and the reason it exists — `plex.direct` is a public zone that returns the
|
||||
private address embedded in the label, so *every* resolver (PiHole, the router, 1.1.1.1, a DoH
|
||||
browser) answers `192-168-2-24.<hash>.plex.direct` with `192.168.2.24`. It needs no split-horizon
|
||||
entry, no Cloudflare record and no certificate work, because the wildcard cert Plex already serves on
|
||||
32400 covers exactly that name.
|
||||
|
||||
- **`catalog/plex.yaml`:** `ADVERTISE_IP` →
|
||||
`https://192-168-2-24.130072f20c334d6c8f63d0dfef87309b.plex.direct:32400`, with the
|
||||
resolver-independence requirement written at the top of the block as the first thing to check
|
||||
before anyone changes this field again.
|
||||
- **PiHole's `plex-vip` answer is KEPT.** It is harmless, costs nothing, and is useful for reaching
|
||||
the VIP by name from a LAN shell. It is simply not what Plex advertises.
|
||||
|
||||
**Why not keep the nicer hostname.** It cannot be made correct for clients that bypass PiHole: a
|
||||
public record for it resolves to the WAN address, and with no NAT hairpin on this router a LAN client
|
||||
sent there fails too. The name is right only for PiHole users and wrong for everyone else, and there
|
||||
is no configuration of it that is right for both. `plex.direct` is right for both.
|
||||
|
||||
**The honest cost, restated rather than buried:** the value embeds both the VIP and the server's
|
||||
certificate hash as a literal — envsubst cannot dash an IP and nothing in the tree can derive the
|
||||
hash, so nothing checks that they still agree. That fragility is exactly why the hostname was tried.
|
||||
It is accepted because an unresolvable name is a *worse* failure: silent, LAN-wide, and it points
|
||||
users at their router.
|
||||
|
||||
## What I verified
|
||||
|
||||
Live prod, 2026-10-05, read-only.
|
||||
|
||||
- **Every row of the table above**, including the router's `%D` cert on both its LAN and WAN sides —
|
||||
which is what identifies the router as what the clients were reaching.
|
||||
- **THE FIX ITSELF, with full TLS validation (no `curl -k`)**: the plex.direct name resolves to
|
||||
`192.168.2.24` and `/identity` returns **200, TLS verified**. Re-checked today, not carried over
|
||||
from yesterday's measurement.
|
||||
- **Both PiHole replicas are identical and correct**, with the same five `address=` lines live in
|
||||
`/etc/pihole/dnsmasq.conf`, and both IPs present behind the `pihole-dns-tcp`/`-udp` endpoints. I
|
||||
checked only pihole-0 at first and deliberately went back for pihole-1, because a split-brain pair
|
||||
would have produced exactly this intermittent-looking symptom.
|
||||
- **The cluster was never the problem:** all eight LoadBalancer VIPs still assigned, L2
|
||||
announcements intact (prod-3 holds `.20`/`.23`/`.24`), `wildcard-home` Certificate `Ready=True`,
|
||||
and PR #51 had reconciled (`customConnections="http://plex-vip.home.keithjackson.dev:32400"` was
|
||||
live, confirming the bad value was in effect).
|
||||
- **`LanNetworksBandwidth="10.0.0.0/8,192.168.0.0/16"` is still set**, so the postStart hook survived
|
||||
the roll and keeps re-asserting.
|
||||
- **The Cilium rollout from #50 STILL has not happened** — agents are unchanged since
|
||||
2026-10-04T07:11, so the Hubble export is inactive and `filePath: stdout` remains unproven.
|
||||
|
||||
## What I did NOT verify
|
||||
|
||||
- **THE OUTCOME. Unreconciled.** No client has been observed connecting after this change. The
|
||||
argument is that the advertised name now resolves for every resolver and serves a matching
|
||||
certificate — both measured — but I have not seen a Plex client complete a connection, and I
|
||||
cannot see the user's clients at all.
|
||||
- **THE CLIENT-SIDE RESOLVER PATH IS INFERRED, NOT OBSERVED.** I could not query a public resolver
|
||||
from here — my attempt returned NXDOMAIN even for a `plex.direct` name that certainly resolves, so
|
||||
the probe was broken (egress policy or a missing tool) and I discarded it rather than report it.
|
||||
The chain "client bypasses PiHole → NXDOMAIN → router answers with itself → `%D`" is the only
|
||||
explanation consistent with all the measurements, but the NXDOMAIN-hijack step in particular is
|
||||
deduced from the router serving `%D` on its LAN address, not from watching a lookup.
|
||||
- **Which clients bypass PiHole, or how many.** If in fact every client uses PiHole, then my
|
||||
explanation is wrong and something else sent them to the router — in which case the next thing to
|
||||
check is whether the clients were using `plex.home.keithjackson.dev` (port 443, the Gateway) rather
|
||||
than an advertised connection at all.
|
||||
- **Whether plex.tv will re-publish promptly** after the value changes, and whether stale connection
|
||||
lists are cached on clients. A client holding the old list may keep failing until it refreshes.
|
||||
- **Mixed content.** The previous `http://` value would also have been blocked by the browser for
|
||||
anyone using `https://app.plex.tv`; that is fixed incidentally by returning to `https`, but I did
|
||||
not test it.
|
||||
- **The Remote badge — the symptom that began this whole sequence — remains unconfirmed.**
|
||||
`LanNetworksBandwidth` is set, but Plex still tags traffic `Allowed Network (WAN)`. Unchanged and
|
||||
still unexplained.
|
||||
- **`ManualPortMappingMode="1"` with an absent port** is still there, untouched.
|
||||
@@ -91,6 +91,10 @@ This proves the R-IDP-02 property (unauthenticated cannot reach the upstream) on
|
||||
|
||||
## Observability: what this component can and cannot see (measured 2026-09-30)
|
||||
|
||||
> **SUPERSEDED — read "Update 2026-10-05" at the end of this section first.** The gate sidecars
|
||||
> **are** scraped now and the TSDB **does** hold `oauth2_proxy_*` for them. The two subsections
|
||||
> below are kept as the dated record of how this was diagnosed, not as current state.
|
||||
|
||||
`servicemonitor.yaml` scrapes **this namespace's** deployment and nothing else. On prod that is
|
||||
two targets, `job="oauth2-proxy"`, both `up=1`. The complete set of oauth2-proxy-specific series
|
||||
in the TSDB is:
|
||||
@@ -157,3 +161,55 @@ present `up` series. A discovered-but-failing target publishes `up` with value 0
|
||||
would have cleared `OAuth2ProxyGateUnobserved` the moment the PodMonitor landed — reporting the gates
|
||||
as observed while nothing was being scraped. The unit test pins that scenario; deleting the `== 1`
|
||||
fails it and nothing else.
|
||||
|
||||
### Update 2026-10-05: the scrape works
|
||||
|
||||
PR #49 (merged `21c88b0`) added the missing third piece. `catalog/render.py`'s
|
||||
`r_monitoring_ingress_policy` now also fires for the gate sidecar's metrics port, keyed on
|
||||
`_needs_oauth2_sidecar`, rendering `<app>-allow-monitoring-scrape` per app: ns `monitoring` → that
|
||||
app's `endpointSelector` → TCP 44180 only. Five objects, one each for jackett, pihole, qbittorrent,
|
||||
radarr, sonarr.
|
||||
|
||||
Measured on prod 2026-10-05 after Flux applied it (`apps-wave-0` 21:37:01Z, `apps-wave-1`
|
||||
21:37:46Z):
|
||||
|
||||
| | before | after |
|
||||
|---|---|---|
|
||||
| target health (6) | `down` | `up` |
|
||||
| `lastError` | `context deadline exceeded` | `''` |
|
||||
| `lastScrapeDuration` | **10.0s** — exactly the `scrapeTimeout` | **0.001–0.006s** |
|
||||
| `up` | `0` × 6 | `1` × 6 |
|
||||
|
||||
count by (__name__) ({__name__=~"oauth2_proxy_.*",
|
||||
namespace=~"jackett|radarr|sonarr|qbittorrent|pihole"})
|
||||
oauth2_proxy_requests_in_flight => 6
|
||||
oauth2_proxy_requests_total => 16
|
||||
oauth2_proxy_response_duration_seconds_bucket => 108
|
||||
oauth2_proxy_response_duration_seconds_sum => 9
|
||||
oauth2_proxy_response_duration_seconds_count => 9
|
||||
|
||||
sum by (namespace) (oauth2_proxy_requests_total)
|
||||
qbittorrent => 35072 radarr => 23560 sonarr => 1284
|
||||
|
||||
`oauth2_proxy_requests_total` is absent for jackett and pihole simply because that counter is not
|
||||
created until the gate has served its first request.
|
||||
|
||||
**So the auth-churn rule is now writable rather than decorative** — sustained `code="302"` as a
|
||||
share of `oauth2_proxy_requests_total`, held across a window longer than the session lifetime. That
|
||||
is the rule the 2026-09-30 incident above could not be caught by. It is still unwritten.
|
||||
|
||||
**Alert outcome:** all 5 `TargetDown` resolved, and 5 of 6 `OAuth2ProxyGateUnobserved` with them.
|
||||
At `repeat_interval: 12h` with `group_by: [alertname, namespace]`, those 11 groups had been
|
||||
re-notifying at roughly **22 emails/day** since 2026-10-02; that drops to 2/day.
|
||||
|
||||
**`crafty` still fires, and that is correct.** It is a hand-maintained Deployment under
|
||||
`platform/crafty`, outside the renderer, whose `oauth2-proxy` container has **no
|
||||
`--metrics-address` at all** — no listener, so no port to open and nothing for the PodMonitor's
|
||||
pod-label selector to match. Closing it needs the flag and a `metrics` containerPort on crafty's
|
||||
own manifest first, then a hand-written allowance beside it. Open follow-up work.
|
||||
|
||||
**The paragraph above headed "To close the gap for an app" is superseded**: a *Service* port and a
|
||||
ServiceMonitor are **not** required. A PodMonitor targets pod ports directly, which is why no
|
||||
Service exposes 44180 and why the renderer needed no Service change. The three things actually
|
||||
needed are `--metrics-address` + named container port, a PodMonitor, and the monitoring-ingress
|
||||
allowance.
|
||||
@@ -27,10 +27,28 @@
|
||||
# created by the apps-wave-0/1 tiers, which this tier does not and must not depend on.
|
||||
#
|
||||
# ============================================================================================
|
||||
# READ THIS BEFORE BELIEVING THE TARGETS ARE UP: THE SCRAPE IS CURRENTLY DENIED BY NETWORKPOLICY
|
||||
# STATUS: THE SCRAPE WORKS. Verified on prod 2026-10-05 (PR #49 merged as 21c88b0).
|
||||
# ============================================================================================
|
||||
# This PodMonitor is NECESSARY AND NOT SUFFICIENT. The endpoint serves, and the path from ns
|
||||
# monitoring is blocked. Both halves measured on prod 2026-10-01, read-only:
|
||||
# All 6 targets in pool podMonitor/oauth2-proxy/oauth2-proxy-sidecars/0 read `up == 1`,
|
||||
# health=up, lastError='', lastScrapeDuration 0.001-0.006s. The TSDB holds oauth2_proxy_* for
|
||||
# these namespaces for the first time: 5 metric names / 148 series, with live counters
|
||||
# (oauth2_proxy_requests_total 35072 qbittorrent, 23560 radarr, 1284 sonarr). All 5 TargetDown
|
||||
# and 5 of 6 OAuth2ProxyGateUnobserved resolved; only namespace="crafty" still fires, and that
|
||||
# is correct — see CRAFTY below.
|
||||
#
|
||||
# The missing piece was an ingress allowance, and it now ships WITH EACH APP, rendered by
|
||||
# catalog/render.py's r_monitoring_ingress_policy as `<app>-allow-monitoring-scrape`:
|
||||
# ns monitoring -> the app's endpointSelector -> TCP 44180 only. See WHY NOT IN THIS DIRECTORY
|
||||
# below for why it is rendered per-app instead of shipped from this tier.
|
||||
#
|
||||
# Keep the history below. It is why the fix is shaped the way it is, and `up == 1` in
|
||||
# auth-visibility-alerts.yaml is load-bearing precisely because of it.
|
||||
#
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# HISTORY: 2026-10-02 -> 2026-10-05, THE SCRAPE WAS DENIED BY NETWORKPOLICY FOR 63.7h
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# This PodMonitor was NECESSARY AND NOT SUFFICIENT: the endpoint served, and the path from ns
|
||||
# monitoring was blocked. Both halves measured on prod 2026-10-01, read-only:
|
||||
#
|
||||
# THE ENDPOINT IS HEALTHY — port-forward straight to the jackett pod's 44180:
|
||||
# GET /metrics -> http=200 bytes=10208
|
||||
@@ -54,19 +72,24 @@
|
||||
# `jackett-allow-gateway-ingress`: `fromEntities: [ingress]` to port 4180 — the Gateway's Envoy
|
||||
# and nothing else. Prometheus matches no clause and its packets are dropped, not refused.
|
||||
#
|
||||
# SO THE TARGETS BELOW WILL READ `up == 0` UNTIL AN INGRESS ALLOWANCE LANDS. That is a known,
|
||||
# measured, documented state — NOT a regression introduced here, and NOT something to silence.
|
||||
# It is deliberately left visible: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
|
||||
# THE TARGETS READ `up == 0` FROM 2026-10-02 05:09 UTC UNTIL THE ALLOWANCE LANDED — one
|
||||
# continuous 63.7h episode (a 15d query_range shows 1 episode per namespace, not flapping),
|
||||
# starting 11 minutes after this file was created, i.e. TargetDown's `for: 10m`. It was never
|
||||
# silenced, and that was the right call: `OAuth2ProxyGateUnobserved` in auth-visibility-alerts.yaml
|
||||
# requires `up == 1`, precisely so a discovered-but-failing target does not read as "observed"
|
||||
# (see that file's header — this exact trap is why the `== 1` is there).
|
||||
#
|
||||
# The signature to recognise if it ever returns: lastScrapeDuration pinned at EXACTLY the
|
||||
# scrapeTimeout (10.0s) with `context deadline exceeded`. That is a DROP, not a refusal — a
|
||||
# refused connection fails in milliseconds. Post-fix the same targets scrape in 1-6ms.
|
||||
#
|
||||
# THIS IS THE SAME BUG LONGHORN HAD, and the platform has a documented fix idiom for it:
|
||||
# platform/longhorn/networkpolicy-metrics.yaml, whose header records the identical fingerprint
|
||||
# measured on 2026-09-13 — ServiceMonitor in place, 3 targets DISCOVERED, `up` 0/0/0, TSDB empty,
|
||||
# `lastError: context deadline exceeded` (a drop, not a refusal). The fix there is a narrow
|
||||
# additive `networking.k8s.io/v1` NetworkPolicy: ns monitoring -> the pod label, that one port.
|
||||
#
|
||||
# WHY THAT POLICY IS NOT IN THIS DIRECTORY. It would have to be five objects, one per app
|
||||
# WHY NOT IN THIS DIRECTORY (the reasoning the fix followed). It would have to be five objects, one per app
|
||||
# namespace, and those namespaces are created by apps-wave-0/1. Measured Flux graph, 2026-10-01:
|
||||
# oauth2-proxy dependsOn=['platform-config'] <- this tier
|
||||
# apps-wave-0 dependsOn=['platform-config'] <- SIBLING, no ordering between them
|
||||
@@ -76,8 +99,19 @@
|
||||
# running cluster and break on a rebuild — the "works now, broken on restore" shape this repo
|
||||
# keeps paying for. Longhorn's precedent is safe only because longhorn-system is created by the
|
||||
# same tier that ships its policy. The allowance therefore belongs WITH THE APPS, rendered next to
|
||||
# the rest of each app's network policy — i.e. in the catalog/renderer, which is in flight on
|
||||
# another branch. Tracked in the handoff; it is one narrow rule per app, not a redesign.
|
||||
# the rest of each app's network policy — i.e. in the catalog/renderer. THAT IS WHERE IT LANDED:
|
||||
# `r_monitoring_ingress_policy`, keyed on `_needs_oauth2_sidecar`, one narrow rule per app.
|
||||
#
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# CRAFTY: STILL UNOBSERVED, AND THIS FILE CANNOT HELP IT
|
||||
# --------------------------------------------------------------------------------------------
|
||||
# `OAuth2ProxyGateUnobserved{namespace="crafty"}` still fires after the fix, correctly. crafty is
|
||||
# a hand-maintained Deployment under platform/crafty, OUTSIDE the renderer, and its oauth2-proxy
|
||||
# container has NO `--metrics-address` at all (measured 2026-10-05) — so its metrics listener
|
||||
# never starts and there is no port to open. The selector below therefore does not match it, and
|
||||
# the per-app allowance is never rendered for it. Closing crafty's gap needs the flag plus a
|
||||
# `metrics` containerPort added to its own manifest FIRST; the allowance is the second half, and
|
||||
# it would have to be written by hand there rather than rendered. Open follow-up work.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
|
||||
Reference in new issue
Block a user