Files
home-network-k3s-impl/apps/wave-0/jackett.yaml
T
jakman85andClaude Opus 5 cf7d97e546 fix(obs): twelve app SLI rules read a signal that exists; kill the dead ServiceMonitors
Twelve `<app>-down` SLI rules were STRUCTURALLY INERT. Each read `up{job="<app>"} == 0`,
and the ServiceMonitor rendered beside it named `endpoints[].port: metrics` — a port NO
rendered app Service defined. Every monitor discovered ZERO targets, so no `up{job="<app>"}`
series existed and the rule could not fire. Verified on prod before this change:
`count by (job) (up)` listed 24 jobs and not one of the twelve appeared; `up{job="plex"}`
and `up{job="gitea"}` both returned nothing. `PrometheusTargetsMissing` could not see it
either — a target that was never discovered has no `up` series to be 0.

THE PORT NAME WAS NEVER THE MISSING PIECE. Ground truth per app: Plex, Pi-hole, Jackett,
qBittorrent, Sonarr, Radarr, Seerr, Semaphore and Nextcloud export NO Prometheus metrics
without a separate exporter sidecar; Gitea can but only with app.ini metrics enabled;
Keycloak does, on the operator's management port, already scraped by
platform/keycloak/servicemonitor.yaml; vm-demo is a KubeVirt VM with no Service at all, so
its monitor selected nothing whatsoever. `metrics: true` was asserting a fact none of them
had. There was also a THIRD layer of inertness nobody had hit yet: every app namespace ships
default-deny with `ingress: []` and the renderer emitted no monitoring allow, so even a
correct port would have been dropped by policy — while renderer-contract.md claimed the
baseline carried such an allow.

FIXED IN THE GENERATOR, not in twelve rendered files:
  * `slis: [up]` now renders a `<app>-down` alert against kube-state-metrics readiness of the
    service's own workload object —
    `kube_deployment_status_replicas_available{namespace,deployment} == 0 or absent(...)`,
    `kube_statefulset_status_replicas_ready` for a StatefulSet, and
    `absent(kubevirt_vmi_info{namespace,name,phase="running"})` for a VM. The same idiom
    KeycloakDown has used since it was fixed. kube-state-metrics IS scraped, so this needs no
    exporter and no new target. The `or absent(...)` half is load-bearing: the signal
    DISAPPEARING now fires the alert instead of silencing it.
  * schema 1.6.0 adds `observability.metricsEndpoint` — WHERE metrics are served. A
    ServiceMonitor is rendered ONLY from it, and it renders all three halves of a working
    scrape together: the named Service port, the monitor endpoint, and the
    monitoring-namespace CiliumNetworkPolicy allow. No catalog entry declares one, so twelve
    dead ServiceMonitors are gone. `metrics: true` keeps meaning logs-to-Loki plus
    kube-state-metrics liveness.
  * schema 1.6.0 also adds `observability.livenessWorkload`, and catalog/keycloak.yaml uses
    it. Keycloak's server is a StatefulSet run by the operator; the default Deployment shape
    would have made its `absent()` half fire FOREVER — the other failure mode this repo has
    shipped.
  * renderer selftest gains cases 9/9a/9b/9c: no metricsEndpoint => no monitor; the up SLI
    must not read `up{job=...}`; endpoint port must exist on the Service; the metrics port
    must NOT reach the Gateway allow; an unknown livenessWorkload.kind is REFUSED.

PROOF, both directions, for all twelve (apps/wave-{0,1,2}/tests/app-slis.test.yaml, run by
platform/monitoring/tests/run.sh over the RENDERED manifests):
  * quiet on the healthy state; firing on no-ready-replica; firing when the series is ABSENT;
    the VM asserted both gone and in a non-running phase; per-app isolation asserted, and
    keycloak asserted un-maskable by its neighbours (keycloak-operator Deployment,
    keycloak-db StatefulSet ready in the same namespace).
  * mutation-checked: six mutations, each turning its suite red — reverting plex to
    `up{job=...} == 0`; dropping seerr's `absent()` half; pointing keycloak at a Deployment;
    capitalising the VMI phase to `Running`; widening keycloak's matcher to the namespace;
    inverting plex's `== 0` to `!= 0`. Control run green.
  * live on prod with `promtool query instant`: all ten deployed app Deployments export
    `kube_deployment_status_replicas_available`,
    `kube_statefulset_status_replicas_ready{namespace="keycloak",statefulset="keycloak"}` = 2,
    `kubevirt_vmi_info` present with a lowercase `phase="running"`; and evaluating the twelve
    SHIPPED expressions returns no series for eleven (quiet, not permanently firing). The
    twelfth is vm-demo, which is not in apps/wave-0/kustomization.yaml and so is not deployed
    — its rule is not applied either.

DELETION WAS NOT THE ANSWER, and the evidence is recorded rather than assumed:
kube-prometheus-stack's `kubernetes-apps` group IS live on prod (17 rules, KubePodNotReady /
KubeDeploymentReplicasMismatch among them), so app pods are not unmonitored — but that is a
generic warning keyed to a namespace, not the per-service CRITICAL SLI R-OBS-01 declares.
platform/monitoring/alerts.yaml carries no generic app-readiness rule at all.

GUARD SO IT CANNOT RECUR SILENTLY: new whole-repo L0 invariant
conformance/tests/l0-static/servicemonitor-ports-resolve. It walks the MONITORS, not the
alerts, so it needs no `up{job=...}` rule to reach one, and resolves each endpoint by SELECTOR
across the whole tree rather than by same-file adjacency. Fixing the alerts DELETED the only
thing that made the broken monitors visible to the sibling check, which is exactly why this
axis needed its own home. It declares `conformance.home.k3s/scope: whole-repo` and adds no
ci/l0.sh stage. Proven to FAIL on the pre-fix tree: 15 failures — eleven port disagreements,
vm-demo's monitor matching no Service at all, and the three generator/contract assertions —
and to pass after. Its resolver is additionally proven on in-check fixtures (agreeing pair
accepted, disagreeing pair rejected, target-less endpoint rejected), because after the fix
there are zero in-repo monitors left for it to judge and a guard whose only evidence is
"nothing failed" cannot be told apart from one that stopped working. It also asserts
r_servicemonitor does not HARDCODE a port name — the mechanism by which one edit gave twelve
services the same wrong one.

Also fixed, found on the way:
  * platform/monitoring/tests/run.sh discovered the SUITE convention only inside
    platform/monitoring/tests/ while documenting it as general — a suite added anywhere else
    was never run and still looked like coverage. Now discovered tree-wide; a misplaced
    *.test.yaml fails loudly instead of vanishing.
  * KNOWN_UNWIRED_JOBS in alert-metrics-have-scrape-targets is now EMPTY, and the twelve
    apps/* entries in alert-rules-unit-tested's UNTESTED_FILES are gone — both because the
    gaps were CLOSED, not waived. Both registers are shrink-only and a stale entry fails.
  * renderer-contract.md claimed the default-deny baseline carried a standing monitoring
    scrape allow. It never did; the claim is corrected and the allow is now real, per declared
    metrics endpoint and scoped to that one port.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-13 17:11:30 -05:00

309 lines
7.6 KiB
YAML

---
# Namespace (trust zone; default-deny + policies apply)
apiVersion: v1
kind: Namespace
metadata:
name: jackett
labels:
homelab.keithjackson.dev/app-namespace: 'true'
gateway-access: shared
annotations:
homelab.keithjackson.dev/backup-policy: daily-offcluster
---
# Default-deny baseline (R-NET-01): deny all, allow only DNS
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: jackett
labels:
homelab.keithjackson.dev/baseline: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
---
# PVC jackett-config (R-STATE)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: jackett-config
namespace: jackett
labels:
app.kubernetes.io/name: jackett
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 2Gi
---
# Deployment (R-PLAT-02)
apiVersion: apps/v1
kind: Deployment
metadata:
name: jackett
namespace: jackett
labels:
app.kubernetes.io/name: jackett
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app.kubernetes.io/name: jackett
template:
metadata:
labels:
app.kubernetes.io/name: jackett
homelab.keithjackson.dev/egress-class: vpn
homelab.keithjackson.dev/auth-mode: forward-auth
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
spec:
containers:
- name: jackett
image: lscr.io/linuxserver/jackett:latest
ports:
- containerPort: 9117
volumeMounts:
- name: config
mountPath: /config
- name: oauth2-proxy
image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
imagePullPolicy: IfNotPresent
args:
- --http-address=0.0.0.0:4180
- --reverse-proxy=true
- --provider=oidc
- --oidc-issuer-url=https://keycloak.${BASE_DOMAIN}/realms/homelab
- --upstream=http://127.0.0.1:9117
- --email-domain=*
- --scope=openid email profile
- --oidc-groups-claim=groups
- --allowed-group=media-users
- --allowed-group=platform-admins
- --skip-provider-button=true
- --redirect-url=https://jackett.${BASE_DOMAIN}/oauth2/callback
- --cookie-domain=.${BASE_DOMAIN}
- --whitelist-domain=.${BASE_DOMAIN}
- --cookie-secure=true
- --pass-access-token=true
- --set-xauthrequest=true
env:
- name: OAUTH2_PROXY_CLIENT_ID
valueFrom:
secretKeyRef:
name: jackett-oauth2-proxy
key: client-id
- name: OAUTH2_PROXY_CLIENT_SECRET
valueFrom:
secretKeyRef:
name: jackett-oauth2-proxy
key: client-secret
- name: OAUTH2_PROXY_COOKIE_SECRET
valueFrom:
secretKeyRef:
name: jackett-oauth2-proxy
key: cookie-secret
ports:
- name: http
containerPort: 4180
protocol: TCP
resources:
requests:
cpu: 25m
memory: 64Mi
limits:
memory: 128Mi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
livenessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 5
periodSeconds: 15
readinessProbe:
httpGet:
path: /ping
port: http
initialDelaySeconds: 5
periodSeconds: 15
volumes:
- name: config
persistentVolumeClaim:
claimName: jackett-config
---
# Service
apiVersion: v1
kind: Service
metadata:
name: jackett
namespace: jackett
labels:
app.kubernetes.io/name: jackett
spec:
selector:
app.kubernetes.io/name: jackett
ports:
- name: port-9117
port: 9117
targetPort: 9117
- name: authproxy
port: 4180
targetPort: 4180
---
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): jackett-allow-radarr-9117
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: jackett-allow-radarr-9117
namespace: jackett
labels:
app.kubernetes.io/name: jackett
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: jackett
ingress:
- fromEndpoints:
- matchLabels:
app.kubernetes.io/name: radarr
k8s:io.kubernetes.pod.namespace: radarr
toPorts:
- ports:
- port: '9117'
protocol: TCP
---
# CiliumNetworkPolicy reciprocal ingress allow (R-NET-02): jackett-allow-sonarr-9117
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: jackett-allow-sonarr-9117
namespace: jackett
labels:
app.kubernetes.io/name: jackett
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: jackett
ingress:
- fromEndpoints:
- matchLabels:
app.kubernetes.io/name: sonarr
k8s:io.kubernetes.pod.namespace: sonarr
toPorts:
- ports:
- port: '9117'
protocol: TCP
---
# CiliumNetworkPolicy gateway-ingress allow (R-NET-01/02): jackett-allow-gateway-ingress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: jackett-allow-gateway-ingress
namespace: jackett
labels:
app.kubernetes.io/name: jackett
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: jackett
ingress:
- fromEntities:
- ingress
toPorts:
- ports:
- port: '4180'
protocol: TCP
---
# CiliumNetworkPolicy world egress allow (egress class governs the path; R-NET-05): jackett-allow-internet-egress
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
name: jackett-allow-internet-egress
namespace: jackett
labels:
app.kubernetes.io/name: jackett
spec:
endpointSelector:
matchLabels:
app.kubernetes.io/name: jackett
egress:
- toEntities:
- world
- toEndpoints:
- matchLabels:
io.kubernetes.pod.namespace: keycloak
app: keycloak
---
# HTTPRoute jackett-9117 (R-TLS/R-DNS)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: jackett-9117
namespace: jackett
labels:
app.kubernetes.io/name: jackett
annotations:
homelab.keithjackson.dev/auth-mode: forward-auth
homelab.keithjackson.dev/auth-groups: media-users,platform-admins
spec:
parentRefs:
- name: shared
namespace: gateway
sectionName: https
hostnames:
- jackett.${BASE_DOMAIN}
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: jackett
port: 4180
---
# PrometheusRule SLI alerts (R-OBS-01)
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: jackett-slis
namespace: jackett
labels:
release: kube-prometheus-stack
app.kubernetes.io/name: jackett
spec:
groups:
- name: jackett.slis
rules:
- alert: jackett-down
expr: kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"} == 0 or absent(kube_deployment_status_replicas_available{namespace="jackett", deployment="jackett"})
for: 5m
labels:
severity: critical
service: jackett
annotations:
summary: jackett is down
description: 'jackett: its Deployment has no available/ready replica for 5m, or the signal is gone entirely (SLI: up, via kube_deployment_status_replicas_available).'