fix(cloudflared): auditable Service+ServiceMonitor scrape; placeholder-only template value
gate / gate (pull_request) Successful in 42s
gate / conformance (pull_request) Skipped

servicemonitor-ports-resolve could not audit a PodMonitor with no in-repo
Service, and no-plaintext-secrets requires a template's secret value to be a
bare placeholder.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jakman85andClaude Opus 5.5 committed 2026-11-08 12:39:56 -06:00
1 parent aabc8c60b6
commit 09f219fe7f
10 files changed
+59 -34

No files matched your search

@@ -413,13 +413,14 @@ SCRAPE_OWNERS: list[dict] = [
},
# ── ADDED 2026-11-08 (feat/cloudflare-tunnel). A NEW OWNER, NOT A RELAXED ASSERTION. ───────
# Purely additive, the same shape as the kured entry: it registers the cloudflared_* family
# and the PodMonitor that scrapes it, and makes the check STRICTER for that component.
# and the Service+ServiceMonitor that scrapes it, and makes the check STRICTER for that component.
{
"owner": "cloudflared: the PodMonitor on the `metrics` (:2000) container port",
"owner": "cloudflared: headless metrics Service + ServiceMonitor on port `metrics` (:2000)",
"prefixes": ("cloudflared_",),
"file": "platform/cloudflared/podmonitor.yaml",
"file": "platform/cloudflared/servicemonitor.yaml",
"require": [
r"^kind:\s*PodMonitor\s*$",
r"^kind:\s*ServiceMonitor\s*$",
r"^kind:\s*Service\s*$",
r"^\s*-\s*port:\s*metrics\s*$",
],
"forbid": [],
+1 -1
View File
@@ -52,7 +52,7 @@ different agent, moves DNS.
- The egress CNP allows world tcp/udp 7844 and tcp 443, plus the `ingress` entity on tcp 443.
It also allows **Gateway backends by `io.cilium.k8s.namespace.labels.gateway-access=shared`,
minus the LAN-only namespaces.**
- `podmonitor.yaml` and `alerts.yaml` (`CloudflaredTunnelDown`, `CloudflaredReplicasDegraded`)
- `servicemonitor.yaml` (headless metrics Service + ServiceMonitor) and `alerts.yaml` (`CloudflaredTunnelDown`, `CloudflaredReplicasDegraded`)
ship with `alerts.promtool-test.yaml`, and `README.md` documents the component.
- `clusters/base/cloudflared.yaml` (new) is its own Flux Kustomization: `dependsOn: platform-config`,
**`wait: false`**, prune, and postBuild `cluster-vars`. It is listed in `clusters/base/kustomization.yaml`.
+1 -1
View File
@@ -15,7 +15,7 @@ outbound-only from the cluster (no router port forward, no hairpin, no split-DNS
| `cloudflared-credentials.sops.yaml` | SOPS-encrypted tunnel credentials JSON. Ships with `REPLACE-ME`; the operator fills it. |
| `deployment.yaml` | 2 replicas, hard spread across nodes, non-root, read-only root fs, pinned digest; PDB `minAvailable: 1`. |
| `networkpolicy.yaml` | default-deny (single owner — not an app namespace) + Prometheus scrape allow + egress CNP: world 7844 tcp/udp and 443, the Gateway (`ingress` entity), and Gateway backends by `gateway-access=shared` minus the LAN-only namespaces. |
| `podmonitor.yaml`, `alerts.yaml` | Scrape of :2000 and the two alerts, both written to fire (not go quiet) when their series vanish. |
| `servicemonitor.yaml`, `alerts.yaml` | Headless metrics Service + ServiceMonitor scraping :2000, and the two alerts, both written to fire (not go quiet) when their series vanish. |
**Phase 1 only.** This component changes no live traffic: Cloudflare sends a hostname through the
tunnel only when its DNS record is a CNAME to `<UUID>.cfargotunnel.com`, and nothing here creates
@@ -33,7 +33,7 @@ tests:
name: nothing scraped at all - TunnelDown still fires (no self-silencing on absence)
input_series:
# kube-state-metrics says both pods are Available, but no cloudflared series exists — the
# broken-PodMonitor / dropped-scrape case. The rule must not read this as healthy.
# broken-ServiceMonitor / dropped-scrape case. The rule must not read this as healthy.
- series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}'
values: '2x60'
promql_expr_test:
+1 -1
View File
@@ -6,7 +6,7 @@
#
# * CloudflaredTunnelDown reads `cloudflared_tunnel_ha_connections` — the per-replica gauge of
# live edge connections — and falls back to `vector(0)` when NO such series exists. A dead
# tunnel, zero pods, a broken PodMonitor, or a NetworkPolicy that drops the scrape all make the
# tunnel, zero pods, a broken ServiceMonitor, or a NetworkPolicy that drops the scrape all make the
# series absent, and all of them FIRE rather than read as healthy.
# * CloudflaredReplicasDegraded reads kube-state-metrics, which does not depend on cloudflared
# being scrapable at all, and has the same `vector(0)` fallback.
@@ -29,6 +29,7 @@ metadata:
namespace: cloudflared
type: Opaque
stringData:
# One line of JSON, exactly as `cloudflared tunnel create` wrote it (compacted with `jq -c`).
# One line of JSON, exactly as `cloudflared tunnel create` wrote it (compacted with `jq -c`):
# {"AccountTag":"<account>","TunnelSecret":"<base64>","TunnelID":"<uuid>"}
# TunnelID MUST equal the `tunnel:` UUID in config.yaml.
credentials.json: '{"AccountTag":"REPLACE-ME","TunnelSecret":"REPLACE-ME","TunnelID":"REPLACE-ME"}'
credentials.json: REPLACE-ME
+1 -1
View File
@@ -15,7 +15,7 @@ resources:
# Namespace default-deny + the Prometheus scrape allow + the egress CiliumNetworkPolicy. Keep
# them in ONE file: a baseline without its scrape allow silently empties the TSDB.
- networkpolicy.yaml
- podmonitor.yaml
- servicemonitor.yaml # headless metrics Service + ServiceMonitor (:2000)
- alerts.yaml
# The tunnel credentials JSON — the ONE secret here. SOPS-encrypted, decrypted per-env by the
# `cloudflared` Flux Kustomization (both overlays patch it), referenced by name only. Its
+1 -1
View File
@@ -31,7 +31,7 @@ spec:
port: 53
---
# The ONLY ingress: Prometheus scraping :2000. Shipped in the same file as the baseline on purpose
# (the Longhorn/kured lesson): a deny-all without this allow leaves the PodMonitor's targets
# (the Longhorn/kured lesson): a deny-all without this allow leaves the ServiceMonitor's targets
# DISCOVERED and every scrape timing out, which is the state the CloudflaredTunnelDown rule is
# written to fire on rather than go quiet in.
apiVersion: networking.k8s.io/v1
-22
View File
@@ -1,22 +0,0 @@
---
# Scrapes cloudflared's own metrics server (:2000, `metrics:` in config.yaml) on every replica.
# kube-prometheus-stack runs with podMonitorSelectorNilUsesHelmValues: false, so Prometheus selects
# every PodMonitor in every namespace; the `release` label is kept for consistency with the
# platform's other monitors. The ingress allow that lets this scrape through the namespace's
# default-deny is cloudflared-allow-prometheus-scrape in networkpolicy.yaml.
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: cloudflared
namespace: cloudflared
labels:
app.kubernetes.io/name: cloudflared
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app.kubernetes.io/name: cloudflared
podMetricsEndpoints:
- port: metrics
path: /metrics
interval: 30s
+45
View File
@@ -0,0 +1,45 @@
---
# Scrape of cloudflared's own metrics server (:2000, `metrics:` in config.yaml) on every replica.
#
# A Service + ServiceMonitor in ONE file rather than a PodMonitor, on purpose: the repo's L0 checks
# (servicemonitor-ports-resolve, alert-metrics-have-scrape-targets assertion 4) can then prove the
# monitor's port is a port this Service actually defines. A PodMonitor would have to be declared as
# an unauditable exception. The Service is headless and carries only the metrics port — it is not a
# way into the tunnel, and nothing routes to it but Prometheus (networkpolicy.yaml admits only the
# monitoring namespace on :2000).
#
# kube-prometheus-stack runs with serviceMonitorSelectorNilUsesHelmValues: false, so this is
# selected from any namespace; the `release` label is kept for consistency.
apiVersion: v1
kind: Service
metadata:
name: cloudflared-metrics
namespace: cloudflared
labels:
app.kubernetes.io/name: cloudflared
spec:
clusterIP: None
selector:
app.kubernetes.io/name: cloudflared
ports:
- name: metrics
port: 2000
targetPort: metrics
protocol: TCP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: cloudflared
namespace: cloudflared
labels:
app.kubernetes.io/name: cloudflared
release: kube-prometheus-stack
spec:
selector:
matchLabels:
app.kubernetes.io/name: cloudflared
endpoints:
- port: metrics
path: /metrics
interval: 30s