fix(cloudflared): auditable Service+ServiceMonitor scrape; placeholder-only template value
servicemonitor-ports-resolve could not audit a PodMonitor with no in-repo Service, and no-plaintext-secrets requires a template's secret value to be a bare placeholder. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
1 parent
aabc8c60b6
commit
09f219fe7f
10 files changed
+59
-34
No files matched your search
@@ -413,13 +413,14 @@ SCRAPE_OWNERS: list[dict] = [
|
||||
},
|
||||
# ── ADDED 2026-11-08 (feat/cloudflare-tunnel). A NEW OWNER, NOT A RELAXED ASSERTION. ───────
|
||||
# Purely additive, the same shape as the kured entry: it registers the cloudflared_* family
|
||||
# and the PodMonitor that scrapes it, and makes the check STRICTER for that component.
|
||||
# and the Service+ServiceMonitor that scrapes it, and makes the check STRICTER for that component.
|
||||
{
|
||||
"owner": "cloudflared: the PodMonitor on the `metrics` (:2000) container port",
|
||||
"owner": "cloudflared: headless metrics Service + ServiceMonitor on port `metrics` (:2000)",
|
||||
"prefixes": ("cloudflared_",),
|
||||
"file": "platform/cloudflared/podmonitor.yaml",
|
||||
"file": "platform/cloudflared/servicemonitor.yaml",
|
||||
"require": [
|
||||
r"^kind:\s*PodMonitor\s*$",
|
||||
r"^kind:\s*ServiceMonitor\s*$",
|
||||
r"^kind:\s*Service\s*$",
|
||||
r"^\s*-\s*port:\s*metrics\s*$",
|
||||
],
|
||||
"forbid": [],
|
||||
|
||||
@@ -52,7 +52,7 @@ different agent, moves DNS.
|
||||
- The egress CNP allows world tcp/udp 7844 and tcp 443, plus the `ingress` entity on tcp 443.
|
||||
It also allows **Gateway backends by `io.cilium.k8s.namespace.labels.gateway-access=shared`,
|
||||
minus the LAN-only namespaces.**
|
||||
- `podmonitor.yaml` and `alerts.yaml` (`CloudflaredTunnelDown`, `CloudflaredReplicasDegraded`)
|
||||
- `servicemonitor.yaml` (headless metrics Service + ServiceMonitor) and `alerts.yaml` (`CloudflaredTunnelDown`, `CloudflaredReplicasDegraded`)
|
||||
ship with `alerts.promtool-test.yaml`, and `README.md` documents the component.
|
||||
- `clusters/base/cloudflared.yaml` (new) is its own Flux Kustomization: `dependsOn: platform-config`,
|
||||
**`wait: false`**, prune, and postBuild `cluster-vars`. It is listed in `clusters/base/kustomization.yaml`.
|
||||
|
||||
@@ -15,7 +15,7 @@ outbound-only from the cluster (no router port forward, no hairpin, no split-DNS
|
||||
| `cloudflared-credentials.sops.yaml` | SOPS-encrypted tunnel credentials JSON. Ships with `REPLACE-ME`; the operator fills it. |
|
||||
| `deployment.yaml` | 2 replicas, hard spread across nodes, non-root, read-only root fs, pinned digest; PDB `minAvailable: 1`. |
|
||||
| `networkpolicy.yaml` | default-deny (single owner — not an app namespace) + Prometheus scrape allow + egress CNP: world 7844 tcp/udp and 443, the Gateway (`ingress` entity), and Gateway backends by `gateway-access=shared` minus the LAN-only namespaces. |
|
||||
| `podmonitor.yaml`, `alerts.yaml` | Scrape of :2000 and the two alerts, both written to fire (not go quiet) when their series vanish. |
|
||||
| `servicemonitor.yaml`, `alerts.yaml` | Headless metrics Service + ServiceMonitor scraping :2000, and the two alerts, both written to fire (not go quiet) when their series vanish. |
|
||||
|
||||
**Phase 1 only.** This component changes no live traffic: Cloudflare sends a hostname through the
|
||||
tunnel only when its DNS record is a CNAME to `<UUID>.cfargotunnel.com`, and nothing here creates
|
||||
|
||||
@@ -33,7 +33,7 @@ tests:
|
||||
name: nothing scraped at all - TunnelDown still fires (no self-silencing on absence)
|
||||
input_series:
|
||||
# kube-state-metrics says both pods are Available, but no cloudflared series exists — the
|
||||
# broken-PodMonitor / dropped-scrape case. The rule must not read this as healthy.
|
||||
# broken-ServiceMonitor / dropped-scrape case. The rule must not read this as healthy.
|
||||
- series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}'
|
||||
values: '2x60'
|
||||
promql_expr_test:
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
#
|
||||
# * CloudflaredTunnelDown reads `cloudflared_tunnel_ha_connections` — the per-replica gauge of
|
||||
# live edge connections — and falls back to `vector(0)` when NO such series exists. A dead
|
||||
# tunnel, zero pods, a broken PodMonitor, or a NetworkPolicy that drops the scrape all make the
|
||||
# tunnel, zero pods, a broken ServiceMonitor, or a NetworkPolicy that drops the scrape all make the
|
||||
# series absent, and all of them FIRE rather than read as healthy.
|
||||
# * CloudflaredReplicasDegraded reads kube-state-metrics, which does not depend on cloudflared
|
||||
# being scrapable at all, and has the same `vector(0)` fallback.
|
||||
|
||||
@@ -29,6 +29,7 @@ metadata:
|
||||
namespace: cloudflared
|
||||
type: Opaque
|
||||
stringData:
|
||||
# One line of JSON, exactly as `cloudflared tunnel create` wrote it (compacted with `jq -c`).
|
||||
# One line of JSON, exactly as `cloudflared tunnel create` wrote it (compacted with `jq -c`):
|
||||
# {"AccountTag":"<account>","TunnelSecret":"<base64>","TunnelID":"<uuid>"}
|
||||
# TunnelID MUST equal the `tunnel:` UUID in config.yaml.
|
||||
credentials.json: '{"AccountTag":"REPLACE-ME","TunnelSecret":"REPLACE-ME","TunnelID":"REPLACE-ME"}'
|
||||
credentials.json: REPLACE-ME
|
||||
@@ -15,7 +15,7 @@ resources:
|
||||
# Namespace default-deny + the Prometheus scrape allow + the egress CiliumNetworkPolicy. Keep
|
||||
# them in ONE file: a baseline without its scrape allow silently empties the TSDB.
|
||||
- networkpolicy.yaml
|
||||
- podmonitor.yaml
|
||||
- servicemonitor.yaml # headless metrics Service + ServiceMonitor (:2000)
|
||||
- alerts.yaml
|
||||
# The tunnel credentials JSON — the ONE secret here. SOPS-encrypted, decrypted per-env by the
|
||||
# `cloudflared` Flux Kustomization (both overlays patch it), referenced by name only. Its
|
||||
|
||||
@@ -31,7 +31,7 @@ spec:
|
||||
port: 53
|
||||
---
|
||||
# The ONLY ingress: Prometheus scraping :2000. Shipped in the same file as the baseline on purpose
|
||||
# (the Longhorn/kured lesson): a deny-all without this allow leaves the PodMonitor's targets
|
||||
# (the Longhorn/kured lesson): a deny-all without this allow leaves the ServiceMonitor's targets
|
||||
# DISCOVERED and every scrape timing out, which is the state the CloudflaredTunnelDown rule is
|
||||
# written to fire on rather than go quiet in.
|
||||
apiVersion: networking.k8s.io/v1
|
||||
|
||||
@@ -1,22 +0,0 @@
|
||||
---
|
||||
# Scrapes cloudflared's own metrics server (:2000, `metrics:` in config.yaml) on every replica.
|
||||
# kube-prometheus-stack runs with podMonitorSelectorNilUsesHelmValues: false, so Prometheus selects
|
||||
# every PodMonitor in every namespace; the `release` label is kept for consistency with the
|
||||
# platform's other monitors. The ingress allow that lets this scrape through the namespace's
|
||||
# default-deny is cloudflared-allow-prometheus-scrape in networkpolicy.yaml.
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PodMonitor
|
||||
metadata:
|
||||
name: cloudflared
|
||||
namespace: cloudflared
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
release: kube-prometheus-stack
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
podMetricsEndpoints:
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
interval: 30s
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
# Scrape of cloudflared's own metrics server (:2000, `metrics:` in config.yaml) on every replica.
|
||||
#
|
||||
# A Service + ServiceMonitor in ONE file rather than a PodMonitor, on purpose: the repo's L0 checks
|
||||
# (servicemonitor-ports-resolve, alert-metrics-have-scrape-targets assertion 4) can then prove the
|
||||
# monitor's port is a port this Service actually defines. A PodMonitor would have to be declared as
|
||||
# an unauditable exception. The Service is headless and carries only the metrics port — it is not a
|
||||
# way into the tunnel, and nothing routes to it but Prometheus (networkpolicy.yaml admits only the
|
||||
# monitoring namespace on :2000).
|
||||
#
|
||||
# kube-prometheus-stack runs with serviceMonitorSelectorNilUsesHelmValues: false, so this is
|
||||
# selected from any namespace; the `release` label is kept for consistency.
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: cloudflared-metrics
|
||||
namespace: cloudflared
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
spec:
|
||||
clusterIP: None
|
||||
selector:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
ports:
|
||||
- name: metrics
|
||||
port: 2000
|
||||
targetPort: metrics
|
||||
protocol: TCP
|
||||
---
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
metadata:
|
||||
name: cloudflared
|
||||
namespace: cloudflared
|
||||
labels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
release: kube-prometheus-stack
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app.kubernetes.io/name: cloudflared
|
||||
endpoints:
|
||||
- port: metrics
|
||||
path: /metrics
|
||||
interval: 30s
|
||||
Reference in new issue
Block a user