fix(cloudflared): fsGroup so uid 65532 can read the 0440 credentials Secret

The pod ran as 65532:65532 with no fsGroup, so the kubelet laid the Secret's credentials.json down root:root 0440 — unreadable to the container. cloudflared logged "permission denied" and then the red-herring "Error locating origin cert". - deployment.yaml: securityContext.fsGroup 65532 (mode stays 0440) - L0 cloudflared-ingress-rules (7): Secret volumes must be readable by the container's run-as uid/groups; mutation self-tested - runbook: permission-denied troubleshooting entry, credential rotation section - decisions/2026-11-08-cloudflared-creds-permission.md Co-Authored-By: Claude Opus 5.5 <[email protected]>
Merge pull request 'Added cloudflare creds and tunnel config' (#153 ) from feature/cloudflare-tunnel-config into main
2026-11-08 15:49:56 -06:00 · 2026-11-08 21:34:24 +00:00 · 2026-11-08 15:30:19 -06:00 · 2026-11-08 18:43:42 +00:00 · 2026-11-08 12:39:56 -06:00 · 2026-11-08 12:38:22 -06:00
55 changed files with 2844 additions and 102 deletions
@@ -85,7 +85,7 @@ spec:
        - name: config
          mountPath: /config
      - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
        imagePullPolicy: IfNotPresent
        args:
        - --http-address=0.0.0.0:4180
@@ -90,7 +90,7 @@ spec:
              - -c
              - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n  echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n  exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n  have=$(sql \"select count(*) from adlist where address='$url';\")\n  if [ \"$have\" = \"0\" ]; then\n    echo \"pihole-adlists: adding $url\"\n    sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n    CHANGED=1\n  else\n    en=$(sql \"select enabled from adlist where address='$url';\")\n    if [ \"$en\" != \"1\" ]; then\n      echo \"pihole-adlists: re-enabling $url\"\n      sql \"update adlist set enabled=1 where address='$url';\"\n      CHANGED=1\n    fi\n  fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n  [ -n \"$addr\" ] || continue\n  if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n    echo \"pihole-adlists: disabling undeclared $addr\"\n    sql \"update adlist set enabled=0 where address='$addr';\"\n    echo changed >> /tmp/adlist-changed\n  fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n  echo \"pihole-adlists: declared set changed - running gravity update\"\n  pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n  echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
      - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
        imagePullPolicy: IfNotPresent
        args:
        - --http-address=0.0.0.0:4180
@@ -156,7 +156,7 @@ spec:
                , \"[Preferences]\"\n\ndef matches(raw):\n    \"\"\"True iff the stored digest is one OUR password produces (salt is read from it).\"\"\"\n    v = raw.strip().strip('\"')\n    if v.startswith(\"@ByteArray(\") and v.endswith(\")\"):\n        v = v[len(\"@ByteArray(\"):-1]\n    try:\n        salt_b64, digest_b64 = v.split(\":\", 1)\n        salt = base64.b64decode(salt_b64, validate=True)\n        digest = base64.b64decode(digest_b64, validate=True)\n    except Exception:\n        return False\n    if not salt or not digest:\n        return False\n    return hmac.compare_digest(\n        hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, len(digest)), digest)\n\ntry:\n    lines = open(conf, encoding=\"utf-8\", errors=\"replace\").read().splitlines()\nexcept FileNotFoundError:\n    lines = []\n\ndef value_of(key):\n    for i, line in enumerate(lines):\n        if line.startswith(key + \"=\"):\n            return i, line.split(\"=\", 1)[1]\n    return -1, None\n\nui, uval = value_of(UKEY)\npi, pval = value_of(PKEY)\nif uval == user and pval is not None and matches(pval):\n    raise SystemExit(0)          # converged - leave the file (and its salt) alone\n\nsalt = os.urandom(SALT_LEN)\ndigest = hashlib.pbkdf2_hmac(\"sha512\", pw.encode(), salt, ITERATIONS, DIGEST_LEN)\nstored = base64.b64encode(salt).decode() + \":\" + base64.b64encode(digest).decode()\ndesired = [UKEY + \"=\" + user, PKEY + '=\"@ByteArray(' + stored + ')\"']\n\nout = [line for i, line in enumerate(lines) if i not in (ui, pi)]\nif SECTION in out:\n    at = out.index(SECTION) + 1\nelse:\n    if out and out[-1].strip():\n        out.append(\"\")\n    out.append(SECTION)\n    at = len(out)\nout[at:at] = desired\n\ntry:\n    os.makedirs(os.path.dirname(conf) or \".\", exist_ok=True)\n    tmp = conf + \".webui-cred.new\"\n    with open(tmp, \"w\", encoding=\"utf-8\") as fh:\n        fh.write(\"\\n\".join(out) + \"\\n\")\n        fh.flush()\n        os.fsync(fh.fileno())\n    # This hook runs as ROOT; the service runs as PUID:PGID. A root-owned 0600 conf would\n    # cost the app the ability to save its own settings, so carry the ownership over.\n    try:\n        st = os.stat(conf)\n        os.chown(tmp, st.st_uid, st.st_gid)\n        os.chmod(tmp, stat.S_IMODE(st.st_mode))\n    except FileNotFoundError:\n        os.chown(tmp, int(os.environ.get(\"PUID\", \"1000\")),\n                 int(os.environ.get(\"PGID\", \"1000\")))\n        os.chmod(tmp, 0o664)\n    os.replace(tmp, conf)        # atomic: the app never reads a half-written conf\nexcept OSError as exc:\n    sys.stderr.write(\"[qbt-webui-cred] cannot write %s: %s\\n\" % (conf, exc))\n    raise SystemExit(1)\nraise SystemExit(10)\nPY\n  rc=$?\n\n  if [ \"$rc\" = 0 ]; then\n    log \"conf already holds the durable credential for '$QBT_WEBUI_USERNAME' (no write)\"\n  elif [ \"$rc\" = 10 ]; then\n    log \"asserted WebUI Username + Password_PBKDF2 into $CONF\"\n    # postStart runs CONCURRENTLY with the s6 init chain, so qbittorrent-nox may ALREADY\n    # have read the old, credential-less conf and be serving a random temporary password.\n    # Restart the SUPERVISED SERVICE (not the pod) so it re-reads what was just written.\n    # Nothing to do when the service has not started yet - it will read the new file.\n    # Its own counter, NOT the shared `waited` API budget: sharing one counter is how the\n    # 2026-09-13 failure spent the entire verification budget before the first login and\n    # then fired attempts 2 and 3 with no wait left at all.\n    sv=0\n    while [ \"$sv\" -lt 60 ] && [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" != \"true\" ]; do\n      sv=$((sv + 1)); sleep 1\n    done\n    if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n      log \"qbittorrent-nox was already running - restarting it to load the credential\"\n      s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\"\n      sleep 3\n    fi\n  else\n    log \"FAILED to write the credential into $CONF (helper exit $rc). The WebUI will keep\" >&2\n    log \"minting a random password\
                \ every start and BOTH arrs will break. Fix and restart.\" >&2\n    exit 0\n  fi\n\n  # (3) GROUND TRUTH: the WebUI itself must accept it. Wait for the API to answer at ALL\n  #     (shared ~120s budget across attempts), then spend exactly ONE login per attempt -\n  #     at most 3 rejected logins, which stays under qBittorrent's default ban threshold.\n  #\n  # \"THE API NEVER ANSWERED\" IS A DIFFERENT FAULT FROM \"THE PASSWORD WAS REJECTED\", and\n  # conflating them is what made this hook lie once already (see the PORT comment). An\n  # unreachable WebUI is reported as unreachable, and the hook stops rather than spending\n  # its remaining login attempts to produce two more identical wrong diagnoses.\n  api_ok=0\n  while [ \"$waited\" -lt 120 ]; do\n    code=$(curl -s -o /dev/null -m 3 -w '%{http_code}' \"http://127.0.0.1:$PORT/\" 2>/dev/null)\n    if [ -n \"$code\" ] && [ \"$code\" != \"000\" ]; then api_ok=1; break; fi\n    waited=$((waited + 1)); sleep 1\n  done\n  if [ \"$api_ok\" != 1 ]; then\n    log \"the WebUI never answered on 127.0.0.1:$PORT within the wait budget, so the\" >&2\n    log \"credential in $CONF is UNVERIFIED. This is NOT a rejected password - it was\" >&2\n    log \"never submitted. Do not rotate the credential on the strength of this line.\" >&2\n    exit 0\n  fi\n  body=$(curl -sS -m 10 \\\n           --data-urlencode \"username=$QBT_WEBUI_USERNAME\" \\\n           --data-urlencode \"password=$QBT_WEBUI_PASSWORD\" \\\n           \"http://127.0.0.1:$PORT/api/v2/auth/login\" 2>/dev/null)\n  if [ \"$body\" = \"Ok.\" ]; then\n    log \"VERIFIED: the WebUI accepts the durable credential for '$QBT_WEBUI_USERNAME'\"\n    exit 0\n  fi\n  log \"WebUI rejected the durable credential (login said '$body')\" >&2\n  # A rejection when the conf was ALREADY converged (rc=0, nothing written) can only mean\n  # the running qbittorrent-nox is serving something it read from an EARLIER conf, so\n  # rewriting the same bytes on the next attempt would change nothing. The remedy is to\n  # make the process RE-READ the file, so restart the supervised service and let the next\n  # attempt re-verify. Without this the retry loop could never change its own outcome.\n  if [ \"$(s6-svstat -u \"$SVC\" 2>/dev/null)\" = \"true\" ]; then\n    log \"restarting qbittorrent-nox so it re-reads $CONF, then re-verifying\" >&2\n    s6-svc -r \"$SVC\" || log \"s6-svc -r failed (continuing to verify anyway)\" >&2\n    sleep 3\n  fi\ndone\n\nlog \"GAVE UP after $attempt attempts: the WebUI does not accept the credential from git.\" >&2\n# A postStart hook MUST exit 0 - a non-zero exit makes the kubelet KILL the container\n# (FailedPostStartHook -> CrashLoopBackOff), turning a credential fault into an outage.\n# Both failure paths above therefore log LOUDLY and exit 0; the `up` SLI still watches the\n# app, and the arrs' own download-client health check surfaces a broken credential.\nexit 0\n"
      - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
        imagePullPolicy: IfNotPresent
        args:
        - --http-address=0.0.0.0:4180
@@ -77,6 +77,42 @@ spec:
    requests:
      storage: 1Gi
 ---
 # act_runner config file — DELTAS ONLY. act_runner v0.6.1 LoadDefault()
 # (internal/pkg/config/config.go) unmarshals this file onto a ZERO Config and then fills every
 # unset field with the same default it uses when there is no file at all, so a file holding
 # only `runner.capacity` changes exactly that and nothing else. Do NOT paste in the output of
 # `act_runner generate-config`: that example is NOT the defaults. Its `runner.labels` would
 # OVERRIDE the registered labels in /data/.runner (daemon.go: config labels win when non-empty)
 # and repoint ubuntu-latest at a different job image; its `force_pull: true` differs from the
 # no-file default (false); and its sample `envs` would be injected into every job.
 #
 # Using a file also makes act_runner IGNORE the legacy GITEA_RUNNER_CAPACITY / _FILE /
 # _ENVIRON / _ENV_FILE / GITEA_DEBUG / GITEA_TRACE env vars (deprecated.go). None of them is
 # set on this Deployment; the GITEA_INSTANCE_URL / _REGISTRATION_TOKEN / _NAME / _LABELS env
 # below is consumed by the image's run.sh for registration, not by that legacy path.
 #
 # A ConfigMap edit alone does NOT restart the runner (it reads the file once at start). If a
 # future change touches ONLY this ConfigMap, also bump the Deployment's pod template (e.g. the
 # config-revision annotation below) or the new value stays inert until the next restart.
 apiVersion: v1
 kind: ConfigMap
 metadata:
  name: act-runner-config
  namespace: gitea-actions
  labels:
    app.kubernetes.io/name: act-runner
 data:
  config.yaml: |
    runner:
      # Two concurrent jobs (default 1). Measured 2026-11-08: queue wait ~1s normally, but
      # 29-759s whenever gate / images / renovate runs collide (run 472 waited 759s behind a
      # renovate burst). Both jobs share the ONE dind daemon below; act_runner names each job's
      # containers GITEA-ACTIONS-TASK-<task id> and creates a network per job, so they do not
      # collide. Workflow `concurrency:` groups are enforced by Gitea server-side and are
      # unaffected: gate.yml's gate-<ref> group still cancels a superseded run on the SAME ref,
      # and renovate's group still serialises renovate. The dind resources are sized for TWO jobs.
      capacity: 2
 ---
 apiVersion: apps/v1
 kind: Deployment
 metadata:
@@ -95,6 +131,10 @@ spec:
    metadata:
      labels:
        app.kubernetes.io/name: act-runner
      annotations:
        # Bump when act-runner-config changes without any other pod-template change, so the
        # Deployment rolls and act_runner re-reads its config (see the ConfigMap header).
        homelab.keithjackson.dev/act-runner-config-revision: "1"   # capacity 2
    spec:
      # CI holds no cluster credentials — do not mount a SA token into the runner or
      # (via the shared netns) the job containers (R-SEC-02, docs/14).
@@ -146,12 +186,23 @@ spec:
              mountPath: /certs
            - name: dind-storage
              mountPath: /var/lib/docker
          # Sized for runner.capacity: 2. Every job's containers run UNDER this daemon, so this
          # container's cgroup carries the whole CI workload. Measured 2026-11-08 (Prometheus,
          # capacity 1): peak CPU 0.69 core (1m rate, 30d), peak working set 447Mi (30d), 0 OOMs.
          #   cpu request 2: two jobs at today's peak is ~1.4 core, and the gate is being
          #     parallelised, so per-job demand rises. The request is what the CFS share is
          #     weighted by under contention (100m was a 1/20 share); NO cpu limit, deliberately,
          #     so an idle node's spare cores are still usable. k3s-prod-2 (8 CPU) had 2945m
          #     requested; +1.9 core = ~4.85 (61%), and prod-1/prod-3 fit too if it reschedules.
          #   memory request 1Gi / limit 4Gi: two jobs at today's peak is ~0.9Gi. An OOM kill here
          #     takes down dockerd and with it BOTH running jobs, so the limit carries ~4x headroom
          #     over that for the parallelised gate. Nodes have ~31Gi allocatable.
          resources:
            requests:
-              cpu: 100m
+              cpu: "2"
-              memory: 256Mi
+              memory: 1Gi
            limits:
-              memory: 2Gi
+              memory: 4Gi
          startupProbe:
            exec:
              command:
@@ -195,23 +246,36 @@ spec:
              value: /certs/client
            - name: DOCKER_TLS_VERIFY
              value: "1"
            # The image's run.sh passes `--config <this>` to BOTH `act_runner register` and
            # `act_runner daemon` when CONFIG_FILE is set (act_runner v0.6.1 scripts/run.sh).
            - name: CONFIG_FILE
              value: /etc/act_runner/config.yaml
          volumeMounts:
            - name: data
              mountPath: /data
            - name: certs
              mountPath: /certs
              readOnly: true
            - name: config
              mountPath: /etc/act_runner
              readOnly: true
          # Measured 2026-11-08 at capacity 1: peak CPU 0.24 core, peak working set 314Mi (30d)
          # against the old 512Mi limit. It holds per-task state, log buffers and the embedded
          # actions/cache server for every concurrent task, so the limit is doubled for capacity 2.
          resources:
            requests:
              cpu: 100m
-              memory: 128Mi
+              memory: 256Mi
            limits:
-              memory: 512Mi
+              memory: 1Gi
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: act-runner-data
        - name: certs
          emptyDir: {}
        - name: config
          configMap:
            name: act-runner-config
        - name: dind-storage
          emptyDir: {}
@@ -138,7 +138,7 @@ spec:
        - name: media
          mountPath: /media
      - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
        imagePullPolicy: IfNotPresent
        args:
        - --http-address=0.0.0.0:4180
@@ -138,7 +138,7 @@ spec:
        - name: media
          mountPath: /media
      - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
        imagePullPolicy: IfNotPresent
        args:
        - --http-address=0.0.0.0:4180
@@ -129,7 +129,7 @@ AUTH_MODES = ("oidc", "forward-auth", "native", "none")
 # the sidecar authenticates the browser against Keycloak (homelab realm, the app's auth.groups)
 # and only THEN proxies to the co-located app UI at http://127.0.0.1:<uiport>. The gate is IN
 # the data path (fail-closed: no valid session in an authorized group => no upstream; R-IDP-02).
-OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.8.1"
+OAUTH2_PROXY_IMAGE = "quay.io/oauth2-proxy/oauth2-proxy:v7.15.6"
 OAUTH2_PROXY_PORT = 4180
 KEYCLOAK_ISSUER = f"https://keycloak.{ZONE}/realms/homelab"
@@ -25,8 +25,44 @@ if [ "${#files[@]}" -eq 0 ]; then
  exit 0
 fi
-yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${files[@]}"
+# yamllint is single-threaded, so the file list is split into LINT_JOBS CONTIGUOUS slices of the
-ok "yaml/lint passed (${#files[@]} files)"
+# sorted list, one yamllint per slice, run concurrently. Each slice's output goes to its own file
 # and the files are printed in slice order afterwards: the log is the same text, in the same
 # file order, as one serial `yamllint "${files[@]}"`, and a finding is never interleaved with
 # another process's output (yamllint's standard format puts the FILE NAME on its own line above
 # that file's `line:col` findings, so interleaving would misattribute them). The stage's exit
 # code is the highest any slice returned (yamllint: 1 = error, 2 = warning under strict), so a
 # single failing slice fails the stage exactly as the serial call did. LINT_JOBS=1 is serial.
 lint_jobs="${LINT_JOBS:-$(n="$(nproc 2>/dev/null || echo 1)"; [ "$n" -gt 8 ] && n=8; echo "$n")}"
 case "$lint_jobs" in ''|*[!0-9]*|0) err "LINT_JOBS must be a positive integer, got '${lint_jobs}'"; exit 2 ;; esac
 [ "$lint_jobs" -gt "${#files[@]}" ] && lint_jobs="${#files[@]}"
 lint_tmp="$(mktemp -d)"
 trap 'rm -rf "$lint_tmp"' EXIT
 per=$(( (${#files[@]} + lint_jobs - 1) / lint_jobs ))
 pids=()
 for (( i = 0; i < lint_jobs; i++ )); do
  slice=("${files[@]:$(( i * per )):${per}}")
  [ "${#slice[@]}" -gt 0 ] || break
  ( rc=0
    yamllint -c "${CI_DIR}/yamllint.yaml" -f "${YAMLLINT_FORMAT:-standard}" "${slice[@]}" \
      >"${lint_tmp}/${i}.out" 2>&1 || rc=$?
    echo "$rc" >"${lint_tmp}/${i}.rc" ) &
  pids+=("$!")
 done
 wait "${pids[@]}" || true
 lint_rc=0
 for (( i = 0; i < ${#pids[@]}; i++ )); do
  cat "${lint_tmp}/${i}.out"
  # A slice that left no rc file died before yamllint returned: that is a failure, not a pass.
  slice_rc="$(cat "${lint_tmp}/${i}.rc" 2>/dev/null || true)"
  case "$slice_rc" in ''|*[!0-9]*) slice_rc=1 ;; esac
  [ "$slice_rc" -gt "$lint_rc" ] && lint_rc="$slice_rc"
 done
 if [ "$lint_rc" -ne 0 ]; then
  err "yaml/lint failed (yamllint exit ${lint_rc}; ${#files[@]} files in ${#pids[@]} parallel slice(s))"
  exit "$lint_rc"
 fi
 ok "yaml/lint passed (${#files[@]} files, ${#pids[@]} parallel slice(s))"
 # A workflow's `configurationFile:` must name a file that exists. Renovate's did not after #79
 # renamed renovate.json -> renovate.json5, and every scheduled run failed with ENOENT for 11 days
@@ -0,0 +1,61 @@
 ---
 # Flux Kustomization: Cloudflare Tunnel connector — reconciles ./platform/cloudflared.
 #
 # ISOLATION IS THE POINT OF THIS FILE. The Deployment it applies CANNOT become Ready until the
 # operator enrols the tunnel (runbooks/cloudflare-tunnel.md): it ships with REPLACE-ME credentials
 # and a placeholder UUID, and a wrong or revoked credential later produces the same state. So:
 #
 #   * ITS OWN Kustomization, not a line in platform/kustomization.yaml. `platform` is wait:true and
 #     gates platform-config, which gates every apps wave; an unenrolled tunnel there would be a
 #     platform-wide stall ON MERGE, with the cause (a missing manual step) invisible from any Flux
 #     condition. Same reasoning, and same failure shape, as vpn-ingress's unfilled server key.
 #   * wait: false. Its pods are legitimately not-Ready whenever the credentials are absent or the
 #     edge is unreachable. Health is reported by the CloudflaredTunnelDown / ReplicasDegraded
 #     alerts (platform/cloudflared/alerts.yaml), not by this Kustomization's Ready condition —
 #     which reaches Ready on APPLY success.
 #   * NOTHING dependsOn THIS, and nothing should. A missing or invalid credentials Secret can then
 #     only ever affect the tunnel itself. conformance/tests/l0-static/cloudflared-ingress-rules
 #     fails the gate if any Kustomization gains `dependsOn: cloudflared`, or if this one stops
 #     being wait:false.
 #   * No Certificate, no HelmRelease, no CRD: nothing in the path for Flux's health checks to wait
 #     on even if wait were flipped (the Certificate-stall lesson).
 #
 # dependsOn: [platform-config] — A REAL EDGE.
 #   1. CRD ORDERING: PodMonitor and PrometheusRule are monitoring.coreos.com CRs whose CRDs are
 #      installed by kube-prometheus-stack in `platform`; platform-config dependsOn platform, so this
 #      edge carries that guarantee transitively (Flux aborts a whole pass on an unknown kind).
 #   2. THE FAR SIDE OF THE TUNNEL: the Gateway (and so the cilium-gateway-shared Service the ingress
 #      rules point at) is applied by platform-config. A tunnel that comes up before its origin is a
 #      support call.
 #
 # ZERO EFFECT ON LIVE TRAFFIC UNTIL DNS MOVES. Cloudflare only sends a hostname through a tunnel
 # when that hostname's DNS record is a CNAME to <uuid>.cfargotunnel.com. This change creates no
 # such record and leaves external-dns, the Gateway annotations, Pi-hole and CoreDNS untouched.
 apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
  name: cloudflared
  namespace: flux-system
 spec:
  interval: 10m
  retryInterval: 2m
  timeout: 5m
  prune: true
  wait: false
  dependsOn:
    - name: platform-config
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./platform/cloudflared
  # REQUIRED, not boilerplate: platform/cloudflared/config.yaml renders `${BASE_DOMAIN}` into every
  # ingress hostname, and alerts.yaml into both runbook_url annotations. Without this block the
  # token becomes the EMPTY STRING and the Kustomization still reports Ready (substitution fails
  # OPEN), leaving ingress rules for `plex.` and `*.` — a tunnel that 404s everything at best.
  postBuild:
    substituteFrom:
      - kind: ConfigMap
        name: cluster-vars
  # SOPS decryption is layered per-environment: clusters/dev and clusters/prod each patch THIS
  # Kustomization by name with a `decryption:` stanza for cloudflared-credentials.sops.yaml.
  # Without it the Secret applies cleanly full of ENC[...] and cloudflared cannot authenticate.
@@ -78,6 +78,13 @@ resources:
  # `wg-server-key` Secret, so in the gating `platform` tier an unfinished manual step would have
  # stalled every apps wave. Carries no SOPS secret, so neither overlay adds a decryption patch.
  - vpn-ingress.yaml     # Flux Kustomization -> ./platform/vpn-ingress (dependsOn platform-config)
  # Cloudflare Tunnel connector (north-south INGRESS from the Cloudflare edge) — POST-platform tier
  # (dependsOn platform-config, wait:false), and NOTHING dependsOn it. Non-gating is load-bearing,
  # as for vpn-ingress: the Deployment cannot become Ready until the operator enrols the tunnel
  # (runbooks/cloudflare-tunnel.md), so in a gating tier an unfinished manual step — or a revoked
  # credential later — would stall every apps wave. Carries cloudflared-credentials.sops.yaml, so
  # both overlays add a SOPS decryption patch for it (by name). See clusters/base/cloudflared.yaml.
  - cloudflared.yaml     # Flux Kustomization -> ./platform/cloudflared (dependsOn platform-config)
  # PM agent — reads Gitea issues and runs one worker agent per assignment that authors a
  # change, gates it and opens a pull request. POST-APP tier (dependsOn apps-wave-0,
  # wait:false): what it needs is GITEA, not Keycloak or the Gateway, and it must never gate
@@ -140,6 +140,23 @@ patches:
          provider: sops
          secretRef:
            name: sops-age
  # cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries
  # platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one
  # secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY
  # SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot
  # authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band
  # (name only).
  - patch: |-
      apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
        name: cloudflared
        namespace: flux-system
      spec:
        decryption:
          provider: sops
          secretRef:
            name: sops-age
  # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
  # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
  # account and the Anthropic subscription token from `claude setup-token`. Without this
@@ -131,6 +131,23 @@ patches:
          provider: sops
          secretRef:
            name: sops-age
  # cloudflared (the Cloudflare Tunnel connector, clusters/base/cloudflared.yaml) carries
  # platform/cloudflared/cloudflared-credentials.sops.yaml — the tunnel credentials JSON, the one
  # secret in that component. Without this stanza the Secret applies full of ENC[...]: the APPLY
  # SUCCEEDS, the Kustomization stays Ready, and the only symptom is a connector that cannot
  # authenticate to the edge (CloudflaredTunnelDown fires). The `sops-age` Secret is out-of-band
  # (name only).
  - patch: |-
      apiVersion: kustomize.toolkit.fluxcd.io/v1
      kind: Kustomization
      metadata:
        name: cloudflared
        namespace: flux-system
      spec:
        decryption:
          provider: sops
          secretRef:
            name: sops-age
  # pm-agent (the Program Manager agent service, clusters/base/pm-agent.yaml) carries
  # pm-agent-credentials.sops.yaml — the Gitea write token for the dedicated non-admin bot
  # account and the Anthropic subscription token from `claude setup-token`. Without this
@@ -14,7 +14,7 @@ Pure standard library (matches the harness's no-pip-dependency posture); the che
 themselves shell out (render checks to catalog/render.py, whose PyYAML dependency lives
 there; L0 checks to git + PyYAML).
-Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] <test-dir> [<test-dir> ...]
+Usage: run_l1_offline.py --junit <out.xml> [--label-prefix P] [--jobs N] <test-dir> [<test-dir> ...]
 Exit 0 iff every selected offline check passes.
 The per-dir check script is whichever of `render-check.sh` / `check.sh` the directory
@@ -43,13 +43,35 @@ Exit codes: 0 all selected checks passed / 1 at least one FAILED / 3 nothing fai
 scope-declared check was UNWIRED. conformance/run.sh propagates 3 verbatim; ci/l0.sh turns it
 into the gate's own rc 3. This is the ONE place the "unwired, not green" rule lives now,
 instead of once per hand-written stage.
 PARALLELISM (--jobs, env CONF_OFFLINE_JOBS)
 -------------------------------------------
 The checks are independent processes, so they run in a bounded thread pool (one `bash` child
 per worker). Default: min(8, usable CPUs); `--jobs 1` is the old serial runner exactly. Each
 check's stdout+stderr is CAPTURED and only reported after every check has finished, in the
 ORDER THE DIRS WERE GIVEN, so the log, the JUnit and the step summary are byte-for-byte
 independent of which check finished first (only the trailing "[N.Ns]" wall-time on each
 status line varies). The 0/1/3 aggregation runs over the same ordered case list as before.
 Parallel-safety is a property of the CHECKS, not of this runner: every check must keep its
 scratch state in a private mktemp/mkdtemp dir and must not write into the tree or a fixed
 path. The audit that established that for the current set is recorded in
 decisions/2026-11-08-parallel-l0-checks.md; a new check that needs shared state must say so.
 Nested pools: the keycloak reconciler checks run their own mutation suites in a pool
 (mutations.py, env MUTATION_JOBS). When this runner is itself parallel it exports a SMALLER
 MUTATION_JOBS (unless the caller already set one), so the two suites share the CPUs the
 other jobs-2 checks are not using instead of each taking min(8, CPUs). See inner_jobs().
 """
 from __future__ import annotations
 import argparse
 import os
 import re
 import subprocess
 import sys
 import time
 from concurrent.futures import ThreadPoolExecutor
 import xml.etree.ElementTree as ET
 from pathlib import Path
 from xml.dom import minidom
@@ -97,6 +119,46 @@ def collect_meta(test_yaml: Path, prefix: str):
    return name, reqs
 def usable_cpus() -> int:
    """CPUs this process may run on (the cgroup/affinity view, not the host's count)."""
    try:
        return max(1, len(os.sched_getaffinity(0)))
    except (AttributeError, OSError):
        return max(1, os.cpu_count() or 1)
 def positive_int(text: str) -> int:
    n = int(text)
    if n < 1:
        raise argparse.ArgumentTypeError(f"must be >= 1, got {n}")
    return n
 def inner_jobs(cpus: int, outer: int) -> int:
    """MUTATION_JOBS to hand the checks when this runner runs `outer` of them at once.
    Two checks (the keycloak client-secret and passkey reconcilers) fan out internally; the
    other outer-2 workers are single processes. Give the two suites an equal share of the CPUs
    those others leave free, never fewer than 2 each (a 1-wide pool would make the 40-mutation
    passkey suite the critical path of the whole stage), never more than 8. Peak process count
    is then about (outer-2) + 2*inner, i.e. <= cpus on any machine with cpus >= outer + 2, and
    at worst ~1.25-1.5x cpus on a small runner — CPU-bound oversubscription of that size costs
    latency per process, not throughput.
    """
    if outer <= 1:
        return min(8, cpus)
    return max(2, min(8, (cpus - outer + 2) // 2))
 def run_check(check: Path, env: dict) -> tuple[int, str, float]:
    t0 = time.monotonic()
    proc = subprocess.run(
        ["bash", str(check)], capture_output=True, text=True,
        stdin=subprocess.DEVNULL, env=env,
    )
    return proc.returncode, (proc.stdout + proc.stderr).strip(), time.monotonic() - t0
 def main() -> int:
    ap = argparse.ArgumentParser(description=__doc__)
    ap.add_argument("--junit", type=Path, required=True)
@@ -108,26 +170,54 @@ def main() -> int:
                         "(makes this runner exit 3); repeatable")
    ap.add_argument("--summary-out", type=Path, default=None,
                    help="write one Markdown line per case here, for the CI step summary")
    ap.add_argument("--jobs", type=positive_int, default=None,
                    help="checks to run at once (default: env CONF_OFFLINE_JOBS, else "
                         "min(8, usable CPUs)); 1 = serial")
    ap.add_argument("dirs", nargs="+", type=Path)
    args = ap.parse_args()
    strict_scopes = set(args.unwired_scope)
-    cases = []  # (name, reqs, dir, rc, output, scope)
+    cpus = usable_cpus()
    if args.jobs is not None:
        jobs = args.jobs
    elif os.environ.get("CONF_OFFLINE_JOBS", "").strip():
        try:
            jobs = positive_int(os.environ["CONF_OFFLINE_JOBS"].strip())
        except (ValueError, argparse.ArgumentTypeError) as e:
            ap.error(f"CONF_OFFLINE_JOBS={os.environ['CONF_OFFLINE_JOBS']!r}: {e}")
    else:
        jobs = min(8, cpus)
    child_env = dict(os.environ)
    if not child_env.get("MUTATION_JOBS", "").strip():
        child_env["MUTATION_JOBS"] = str(inner_jobs(cpus, jobs))
    print(f"[offline] running {len(args.dirs)} check(s), jobs={jobs} "
          f"(cpus={cpus}, MUTATION_JOBS={child_env['MUTATION_JOBS']})", file=sys.stderr)
    # Metadata first, in the given order; then the checks themselves through the pool. The
    # results are consumed in SUBMISSION order, so `cases` is identical to the serial runner's.
    planned = []  # (name, reqs, dir, scope, check-or-None)
    for d in args.dirs:
        check = next((d / s for s in CHECK_SCRIPTS if (d / s).exists()), None)
        name, reqs = collect_meta(d / "chainsaw-test.yaml", args.label_prefix)
        scope = scope_of(d / "chainsaw-test.yaml", args.scope_annotation)
-        if check is None:
+        planned.append((name, reqs, d, scope, check))
-            cases.append((name, reqs, d, 3,
+
-                          f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
+    cases = []  # (name, reqs, dir, rc, output, scope)
-                          scope))
+    elapsed = []  # wall seconds per case, same order (None: nothing ran)
-            continue
+    with ThreadPoolExecutor(max_workers=jobs) as pool:
-        proc = subprocess.run(
+        futures = [pool.submit(run_check, check, child_env) if check is not None else None
-            ["bash", str(check)], capture_output=True, text=True
+                   for (_n, _r, _d, _s, check) in planned]
-        )
+        for (name, reqs, d, scope, check), fut in zip(planned, futures):
-        out = (proc.stdout + proc.stderr).strip()
+            if fut is None:
-        cases.append((name, reqs, d, proc.returncode, out, scope))
+                cases.append((name, reqs, d, 3,
                              f"no offline check script ({' or '.join(CHECK_SCRIPTS)}) in {d}",
                              scope))
                elapsed.append(None)
                continue
            rc, out, secs = fut.result()
            cases.append((name, reqs, d, rc, out, scope))
            elapsed.append(secs)
    failures = sum(1 for c in cases if c[3] not in (0, 3))
    skipped = sum(1 for c in cases if c[3] == 3)
@@ -145,7 +235,7 @@ def main() -> int:
        },
    )
    summary_lines = []
-    for name, reqs, d, rc, out, scope in cases:
+    for (name, reqs, d, rc, out, scope), secs in zip(cases, elapsed):
        tc = ET.SubElement(
            suite,
            "testcase",
@@ -164,8 +254,9 @@ def main() -> int:
            fail.text = out
        status = "PASS" if rc == 0 else ("SKIP" if rc == 3 else "FAIL")
        mark = f" scope={scope}" if scope else ""
        took = f" [{secs:.1f}s]" if secs is not None else ""
        print(f"[offline] {layer_of(d):3s} {status:4s} {name} "
-              f"({','.join(reqs) or 'untagged'}){mark}",
+              f"({','.join(reqs) or 'untagged'}){mark}{took}",
              file=sys.stderr)
        if rc not in (0, 3):
            print(out, file=sys.stderr)
@@ -411,6 +411,32 @@ SCRAPE_OWNERS: list[dict] = [
                   "Keeping the two in one file makes that pair inseparable",
        },
    },
    # ── ADDED 2026-11-08 (feat/cloudflare-tunnel). A NEW OWNER, NOT A RELAXED ASSERTION. ───────
    # Purely additive, the same shape as the kured entry: it registers the cloudflared_* family
    # and the Service+ServiceMonitor that scrapes it, and makes the check STRICTER for that component.
    {
        "owner": "cloudflared: headless metrics Service + ServiceMonitor on port `metrics` (:2000)",
        "prefixes": ("cloudflared_",),
        "file": "platform/cloudflared/servicemonitor.yaml",
        "require": [
            r"^kind:\s*ServiceMonitor\s*$",
            r"^kind:\s*Service\s*$",
            r"^\s*-\s*port:\s*metrics\s*$",
        ],
        "forbid": [],
        "why": "cloudflared_tunnel_ha_connections powers CloudflaredTunnelDown, the only signal "
               "that the tunnel has lost the Cloudflare edge. That rule is written to FIRE on an "
               "absent series, so a lost scrape is loud rather than inert — but it would then be "
               "permanently firing for a wiring reason, which teaches the operator to ignore it",
        "companion": {
            "file": "platform/cloudflared/kustomization.yaml",
            "require": [r"^\s*-\s*networkpolicy\.yaml\s*$"],
            "why": "platform/cloudflared/networkpolicy.yaml ships the namespace's default-deny "
                   "(a platform namespace, outside Kyverno's generator) together with the ingress "
                   "allow for Prometheus on :2000. Keeping them in one file, applied together, is "
                   "what stops a baseline-without-allow from emptying the TSDB",
        },
    },
 ]
 # `up` is produced by EVERY scrape target, so no owner can meaningfully claim it and assertion (1)
@@ -0,0 +1,47 @@
 # =============================================================================
 # L0 STATIC — the Cloudflare Tunnel exposes only what the operator decided, and cannot stall Flux
 # Catalog: docs/14 L0 row ("Pre-apply, in CI") — runs WITHOUT a cluster.
 #
 # platform/cloudflared/config.yaml is the entire internet-facing surface of the tunnel. This test
 # asserts the operator decisions of 2026-11-08 against it on every gate run: no rule may target
 # anything but a 404 or the shared Gateway (so never the Kubernetes API); the LAN/VPN-only
 # hostnames (plex, jackett, pihole, semaphore, qbittorrent, radarr, sonarr, headlamp,
 # unifi, crafty-mc) 404 before the wildcard; the
 # egress policy admits no API/node entity; and the Flux tier is isolated (wait:false, nothing
 # dependsOn it, both overlays decrypt it). It proves itself non-vacuous by mutation. Rationale and
 # the honest NOT-proven list: check.sh.
 #
 # WHY SCOPE, NOT REQUIREMENTS: the property spans platform/cloudflared, platform/gateway (the
 # derived Service name), clusters/base and both overlays. A requirement-scoped selection would skip
 # it on, e.g., a Gateway rename — precisely a change that breaks it.
 # =============================================================================
 apiVersion: chainsaw.kyverno.io/v1alpha1
 kind: Test
 metadata:
  name: cloudflared-ingress-rules
  labels:
    conformance.home.k3s/R-NET-01: "true"
    conformance.home.k3s/R-NET-02: "true"
  annotations:
    conformance.home.k3s/requirements: "R-NET-01, R-NET-02"
    conformance.home.k3s/layer: "L0"
    conformance.home.k3s/scope: "whole-repo"
    conformance.home.k3s/safety-critical: "true"
    conformance.home.k3s/executable: "true"
    conformance.home.k3s/intent: >-
      The Cloudflare Tunnel routes only to the shared Gateway or a 404, never to the Kubernetes
      API; the LAN/VPN-only hostnames 404 ahead of the wildcard; the connector's egress admits no
      API or node entity; and its Flux tier cannot stall any other tier.
 spec:
  steps:
    - name: assert-tunnel-ingress-rules-and-isolation
      description: >-
        Parse platform/cloudflared/config.yaml with BASE_DOMAIN substituted, assert every rule's
        target against the Gateway Service derived from platform/gateway/gateway.yaml, assert the
        excluded hostnames 404 first, check the egress CiliumNetworkPolicy and the Flux wiring, and
        run the evaluator on mutated configs to prove it is not vacuous; and assert the
        credentials Secret volume is readable by the container's run-as uid/groups (fsGroup +
        mode), self-tested by mutation.
      try:
        - script:
            content: bash ./check.sh
@@ -0,0 +1,321 @@
 #!/usr/bin/env bash
 # =============================================================================
 # L0 static check (offline, no cluster) — cloudflared-ingress-rules
 # Serves: R-NET-01 / R-NET-02 (default-deny; allow rules declarative and scoped), and the four
 #         operator decisions of 2026-11-08 recorded in decisions/2026-11-08-cloudflare-tunnel.md.
 #
 # WHY THIS EXISTS. platform/cloudflared/config.yaml is the whole of what the internet can reach
 # through the Cloudflare Tunnel. A one-line edit there — a rule pointing at the Kubernetes API, a
 # LAN-only hostname moved below the wildcard, a dropped catch-all — publishes something the operator
 # decided must never be public, and nothing in the cluster turns red: the tunnel stays connected and
 # the Kustomization stays Ready. So the decisions are asserted here, on every gate run.
 #
 # GIVEN the repository as authored,
 # THEN  (1) every ingress rule's `service` is `http_status:404` or EXACTLY the shared Gateway's
 #           in-cluster Service, derived from platform/gateway/gateway.yaml (so a Gateway rename
 #           fails here rather than in production) — which by construction excludes the
 #           Kubernetes API, and a named deny-list says so explicitly when it is violated;
 #       (2) each LAN/VPN-only hostname (plex, plex-vip, jackett, pihole, pihole-dns, semaphore,
 #           qbittorrent, radarr, sonarr, headlamp, unifi, crafty-mc) has a rule mapping it to http_status:404 that comes BEFORE any
 #           Gateway-bound rule (cloudflared matches first-rule-wins);
 #       (3) the apex and the wildcard are the Gateway rules, with matchSNItoHost, and nothing
 #           disables origin TLS verification; the last rule is a hostname-less 404 catch-all;
 #       (4) the egress CiliumNetworkPolicy admits none of kube-apiserver/host/remote-node/
 #           cluster/all and no CIDR rule, and its backend selector excludes the LAN-only
 #           namespaces (the second, independent layer behind (2));
 #       (5) the Flux tier is ISOLATED: its own Kustomization, wait:false, not listed in
 #           platform/kustomization.yaml, nothing anywhere dependsOn it, and both overlays carry its
 #           SOPS decryption patch — so a missing or invalid credential can stall nothing else;
 #       (6) the rule evaluator is not vacuous: it is run against mutated copies of the real config
 #           (an API rule added, plex moved below the wildcard, the catch-all removed) and must
 #           flag each one;
 #   AND (7) every Secret volume the container mounts — the tunnel credentials above all — is
 #           READABLE by the container's run-as identity, modelled the way the kubelet lays it out:
 #           files owned by uid 0, group-owned by fsGroup if set (else gid 0), with the item mode or
 #           defaultMode (API default 0644). The identity is the container securityContext over
 #           the pod's; the process's groups are runAsGroup + fsGroup + supplementalGroups. An
 #           unset runAsUser is a FAIL, not a pass: the image default cannot be read offline. The
 #           model is deliberately conservative — it does not credit the kubelet's habit of
 #           OR-ing group-read into fsGroup-owned volumes, so 0400+fsGroup fails here though it
 #           would work. Self-tested by mutation (fsGroup removed; mode 0400; runAsUser unset).
 #           This is the 2026-11-08 CrashLoop: runAsUser 65532, no fsGroup, 0440 root:root.
 #
 # NOT proven (static): that Cloudflare routes only the hostnames whose DNS points at the tunnel,
 # that the Gateway's certificate validates from cloudflared, or that Envoy enforces (4) — those are
 # runtime properties, verified by the operator per runbooks/cloudflare-tunnel.md.
 # =============================================================================
 set -euo pipefail
 DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 ROOT="$(cd "${DIR}/../../../.." && pwd)"        # repo root
 exec python3 - "$ROOT" <<'PY'
 import copy
 import re
 import sys
 from pathlib import Path
 try:
    import yaml
 except ImportError:                                   # pragma: no cover
    print("CHECK FAIL (cloudflared-ingress-rules): PyYAML is required", file=sys.stderr)
    raise SystemExit(1)
 ROOT = Path(sys.argv[1])
 FAILURES, CHECKS = [], 0
 def check(cond, desc):
    global CHECKS
    CHECKS += 1
    if not cond:
        FAILURES.append(desc)
 def docs(path):
    return [d for d in yaml.safe_load_all(path.read_text()) if isinstance(d, dict)]
 COMP = ROOT / "platform" / "cloudflared"
 CONFIG = COMP / "config.yaml"
 if not CONFIG.is_file():
    print(f"CHECK FAIL (cloudflared-ingress-rules): {CONFIG.relative_to(ROOT)} is absent — the "
          "component this check guards was moved or deleted; re-point or delete this check in the "
          "SAME change", file=sys.stderr)
    raise SystemExit(1)
 # The hostnames the operator decided never go through the tunnel (decisions 2026-11-08).
 # headlamp: operator decision 2026-11-08 (LAN/VPN-only). qbittorrent/radarr/sonarr: catalog
 # `exposure: internal`, like jackett/pihole/semaphore.
 EXCLUDED_HOSTS = ("plex", "plex-vip", "jackett", "pihole", "pihole-dns", "semaphore",
                  "qbittorrent", "radarr", "sonarr", "headlamp", "unifi", "crafty-mc")
 # Their Gateway-backend namespaces (unifi / crafty-mc are L4 VIPs, not Gateway backends).
 EXCLUDED_NAMESPACES = ("plex", "jackett", "pihole", "semaphore", "qbittorrent", "radarr",
                       "sonarr", "headlamp")
 FORBIDDEN_ENTITIES = {"kube-apiserver", "host", "remote-node", "cluster", "all"}
 # Any of these in a `service` means a route toward the Kubernetes API (or a raw-TCP/SSH bastion).
 API_MARKERS = ("kubernetes", "kube-apiserver", "apiserver", ":6443", "10.43.0.1", "tcp://",
               "ssh://", "unix:", "bastion")
 # ── substitution: ${BASE_DOMAIN} from cluster-vars, the ONLY token allowed ──────────────────────
 cv = next(d for d in docs(ROOT / "clusters" / "base" / "cluster-vars.yaml")
          if d.get("kind") == "ConfigMap")
 base_domain = cv["data"]["BASE_DOMAIN"]
 raw = CONFIG.read_text()
 tokens = set(re.findall(r"\$\{([A-Za-z0-9_]+)\}", raw))
 check(tokens == {"BASE_DOMAIN"},
      f"(0) config.yaml uses exactly the ${{BASE_DOMAIN}} token (found {sorted(tokens)}); any other "
      f"token is substituted from cluster-vars unseen by this check")
 check(re.search(r"\$(?!\{BASE_DOMAIN\})", raw) is None,
      "(0) config.yaml contains a bare `$` — Flux envsubst would rewrite or empty it")
 cfg = yaml.safe_load(raw.replace("${BASE_DOMAIN}", base_domain))
 # ── the Gateway Service, derived rather than hard-coded ─────────────────────────────────────────
 gw = next(d for d in docs(ROOT / "platform" / "gateway" / "gateway.yaml") if d.get("kind") == "Gateway")
 GATEWAY_SVC = (f"https://cilium-gateway-{gw['metadata']['name']}."
               f"{gw['metadata']['namespace']}.svc.cluster.local:443")
 def evaluate(cfg):
    """Return the list of rule violations for a parsed cloudflared config."""
    bad = []
    rules = cfg.get("ingress") or []
    if len(rules) < len(EXCLUDED_HOSTS) + 3:
        bad.append(f"only {len(rules)} ingress rules — the scan is vacuous")
        return bad
    for i, r in enumerate(rules):
        svc = str(r.get("service", ""))
        if any(m in svc.lower() for m in API_MARKERS):
            bad.append(f"rule {i} ({r.get('hostname', '<catch-all>')}) targets {svc!r}, which "
                       f"is a route toward the Kubernetes API / a raw bastion — the API stays "
                       f"local-only (operator decision 2026-11-08)")
        if svc not in ("http_status:404", GATEWAY_SVC):
            bad.append(f"rule {i} ({r.get('hostname', '<catch-all>')}) targets {svc!r}; only "
                       f"http_status:404 or {GATEWAY_SVC} are permitted")
        if (r.get("originRequest") or {}).get("noTLSVerify"):
            bad.append(f"rule {i} disables origin TLS verification (noTLSVerify)")
    if (cfg.get("originRequest") or {}).get("noTLSVerify"):
        bad.append("top-level originRequest disables origin TLS verification (noTLSVerify)")
    gw_idx = [i for i, r in enumerate(rules) if r.get("service") == GATEWAY_SVC]
    first_gw = min(gw_idx) if gw_idx else len(rules)
    for h in EXCLUDED_HOSTS:
        fqdn = f"{h}.{base_domain}"
        hits = [i for i, r in enumerate(rules) if r.get("hostname") == fqdn]
        if not hits:
            bad.append(f"no rule for LAN/VPN-only host {fqdn}; it would fall through to the "
                       f"wildcard and reach the Gateway")
            continue
        i = hits[0]
        if rules[i].get("service") != "http_status:404":
            bad.append(f"{fqdn} maps to {rules[i].get('service')!r}, not http_status:404")
        if i > first_gw:
            bad.append(f"{fqdn}'s 404 rule (#{i}) comes AFTER the first Gateway rule (#{first_gw}); "
                       f"cloudflared is first-match, so the Gateway rule wins")
    gw_hosts = {rules[i].get("hostname") for i in gw_idx}
    if gw_hosts != {base_domain, f"*.{base_domain}"}:
        bad.append(f"the Gateway-bound hostnames are {sorted(map(str, gw_hosts))}; expected "
                   f"exactly the apex and the wildcard")
    for i in gw_idx:
        if (rules[i].get("originRequest") or {}).get("matchSNItoHost") is not True:
            bad.append(f"Gateway rule #{i} lacks originRequest.matchSNItoHost: true — the Gateway "
                       f"would not see the SNI that selects its wildcard certificate")
    last = rules[-1]
    if "hostname" in last or last.get("service") != "http_status:404":
        bad.append("the last rule is not a hostname-less http_status:404 catch-all")
    return bad
 # (1)-(3) the real config
 for v in evaluate(cfg):
    check(False, f"(1-3) {v}")
 check(str(cfg.get("credentials-file", "")).startswith("/etc/cloudflared/creds/"),
      "(3) credentials-file points into the mounted Secret volume")
 # (6) mutation self-test — the evaluator must flag each broken variant
 mutants = {}
 m = copy.deepcopy(cfg)
 m["ingress"].insert(0, {"hostname": f"k8s.{base_domain}",
                        "service": "https://kubernetes.default.svc:443"})
 mutants["API rule added"] = m
 m = copy.deepcopy(cfg)
 plex = next(r for r in m["ingress"] if r.get("hostname") == f"plex.{base_domain}")
 m["ingress"].remove(plex)
 m["ingress"].insert(len(m["ingress"]) - 1, plex)
 mutants["plex moved below the wildcard"] = m
 m = copy.deepcopy(cfg)
 m["ingress"] = m["ingress"][:-1]
 mutants["catch-all removed"] = m
 for name, mut in mutants.items():
    check(evaluate(mut), f"(6) self-test: the evaluator did NOT flag the mutant '{name}' — the "
                         f"check is vacuous")
 # (4) the egress policy
 np_docs = docs(COMP / "networkpolicy.yaml")
 cnps = [d for d in np_docs if d.get("kind") == "CiliumNetworkPolicy"]
 check(len(cnps) == 1, f"(4) exactly one CiliumNetworkPolicy in networkpolicy.yaml (found {len(cnps)})")
 for cnp in cnps:
    for i, rule in enumerate(cnp["spec"].get("egress") or []):
        ents = set(rule.get("toEntities") or [])
        check(not (ents & FORBIDDEN_ENTITIES),
              f"(4) egress rule {i} admits {sorted(ents & FORBIDDEN_ENTITIES)} — that reaches the "
              f"Kubernetes API or the nodes; the API stays local-only")
        check(not (rule.get("toCIDR") or rule.get("toCIDRSet")),
              f"(4) egress rule {i} uses a CIDR — a CIDR can name the API VIP; use entities")
        for ep in rule.get("toEndpoints") or []:
            notin = [e for e in ep.get("matchExpressions") or []
                     if e.get("key") == "io.kubernetes.pod.namespace" and e.get("operator") == "NotIn"]
            missing = set(EXCLUDED_NAMESPACES) - set(notin[0]["values"] if notin else [])
            check(not missing,
                  f"(4) backend selector in egress rule {i} does not exclude {sorted(missing)}")
 for d in np_docs:
    if d.get("kind") == "NetworkPolicy" and d["metadata"]["name"] == "default-deny":
        check(set(d["spec"].get("policyTypes", [])) == {"Ingress", "Egress"},
              "(4) default-deny covers both directions")
        break
 else:
    check(False, "(4) networkpolicy.yaml ships no default-deny baseline for the namespace")
 # (5) Flux isolation
 fk = next(d for d in docs(ROOT / "clusters" / "base" / "cloudflared.yaml")
          if d.get("kind") == "Kustomization")
 check(fk["spec"].get("wait") is False,
      "(5) clusters/base/cloudflared.yaml must be wait: false — its pods are legitimately not-Ready "
      "until the tunnel is enrolled")
 check(fk["spec"].get("path") == "./platform/cloudflared", "(5) the Kustomization reconciles ./platform/cloudflared")
 check("cloudflared.yaml" in (ROOT / "clusters" / "base" / "kustomization.yaml").read_text(),
      "(5) clusters/base/kustomization.yaml lists cloudflared.yaml")
 check("cloudflared" not in (ROOT / "platform" / "kustomization.yaml").read_text(),
      "(5) platform/kustomization.yaml must not include cloudflared — that is the gating tier")
 for f in sorted((ROOT / "clusters").rglob("*.yaml")):
    for d in docs(f):
        for dep in (d.get("spec") or {}).get("dependsOn") or []:
            check(not (isinstance(dep, dict) and dep.get("name") == "cloudflared"),
                  f"(5) {f.relative_to(ROOT)} {d.get('metadata', {}).get('name')} dependsOn "
                  f"cloudflared — that hands an unenrolled tunnel the power to stall it")
 for env in ("dev", "prod"):
    text = (ROOT / "clusters" / env / "kustomization.yaml").read_text()
    check(re.search(r"name:\s*cloudflared\s*\n\s*namespace:\s*flux-system\s*\n\s*spec:\s*\n"
                    r"\s*decryption:", text) is not None,
          f"(5) clusters/{env}/kustomization.yaml carries the cloudflared SOPS decryption patch")
 dep = next(d for d in docs(COMP / "deployment.yaml") if d.get("kind") == "Deployment")
 check(dep["spec"].get("replicas", 1) >= 2, "(5) the Deployment runs at least 2 replicas")
 check(dep["spec"]["template"]["metadata"]["labels"].get("homelab.keithjackson.dev/egress-class")
      == "internet",
      "(4) the pods are egress-class=internet — vpn would put the edge connections behind PIA")
 check(dep["spec"]["template"]["spec"].get("automountServiceAccountToken") is False,
      "(4) the pods mount no ServiceAccount token")
 # (7) the credentials Secret is readable by the run-as identity
 def unreadable_secret_volumes(pod_spec):
    """Return a description per Secret volume the containers mount but cannot read."""
    bad = []
    psc = pod_spec.get("securityContext") or {}
    vols = {v["name"]: v for v in pod_spec.get("volumes") or [] if "secret" in v}
    for c in (pod_spec.get("initContainers") or []) + (pod_spec.get("containers") or []):
        csc = c.get("securityContext") or {}
        uid = csc.get("runAsUser", psc.get("runAsUser"))
        gid = csc.get("runAsGroup", psc.get("runAsGroup"))
        fs_group = psc.get("fsGroup")
        groups = {g for g in [gid, fs_group, *(psc.get("supplementalGroups") or [])]
                  if g is not None}
        for vm in c.get("volumeMounts") or []:
            v = vols.get(vm["name"])
            if v is None:
                continue
            sec = v["secret"]
            if uid is None:
                bad.append(f"container {c['name']} mounts Secret volume {vm['name']} but sets no "
                           f"runAsUser (pod or container) — readability cannot be proven offline")
                continue
            default = sec.get("defaultMode", 0o644)
            modes = [it.get("mode", default) for it in sec.get("items") or []] or [default]
            owner_gid = fs_group if fs_group is not None else 0
            for mode in modes:
                readable = ((uid == 0 and mode & 0o400) or (owner_gid in groups and mode & 0o040)
                            or mode & 0o004)
                if not readable:
                    bad.append(f"container {c['name']} (uid {uid}, groups {sorted(groups)}) cannot "
                               f"read Secret volume {vm['name']} ({sec.get('secretName')}): files "
                               f"are 0:{owner_gid} mode {mode:04o} — set pod fsGroup to a group "
                               f"the container runs with, keeping 0440")
    return bad
 pod_spec = dep["spec"]["template"]["spec"]
 creds_vol = [v for v in pod_spec.get("volumes") or []
             if (v.get("secret") or {}).get("secretName") == "cloudflared-credentials"]
 check(len(creds_vol) == 1, "(7) the Deployment mounts Secret cloudflared-credentials as a volume")
 for v in unreadable_secret_volumes(pod_spec):
    check(False, f"(7) {v}")
 sec_mutants = {}
 m = copy.deepcopy(pod_spec)
 m["securityContext"].pop("fsGroup", None)
 m["securityContext"].pop("supplementalGroups", None)
 sec_mutants["fsGroup removed"] = m
 m = copy.deepcopy(pod_spec)
 for v in m["volumes"]:
    if "secret" in v:
        v["secret"]["defaultMode"] = 0o400
        for it in v["secret"].get("items") or []:
            it["mode"] = 0o400
 sec_mutants["mode 0400"] = m
 m = copy.deepcopy(pod_spec)
 m["securityContext"].pop("runAsUser", None)
 for c in m["containers"]:
    (c.get("securityContext") or {}).pop("runAsUser", None)
 sec_mutants["runAsUser unset"] = m
 for name, mut in sec_mutants.items():
    check(unreadable_secret_volumes(mut),
          f"(7) self-test: the readability model did NOT flag the mutant '{name}' — vacuous")
 if CHECKS < 25:
    FAILURES.append(f"only {CHECKS} assertions ran — the check degenerated")
 if FAILURES:
    print(f"CHECK FAIL (cloudflared-ingress-rules): {len(FAILURES)} of {CHECKS} assertions failed",
          file=sys.stderr)
    for f in FAILURES:
        print(f"  - {f}", file=sys.stderr)
    raise SystemExit(1)
 print(f"PASS (cloudflared-ingress-rules): {CHECKS} assertions")
 PY
@@ -25,15 +25,19 @@
 #
 #   python3 mutations.py <path-to-extracted-reconciler.py>
 #
-# Runtime is ~20s (one subprocess per mutation, each running the full selftest
+# Runtime is ~11s serial (one subprocess per mutation, each running the full selftest
-# against an in-process stub Keycloak; no network, no cluster).
+# against an in-process stub Keycloak; no network, no cluster). The selftests run in a
 # bounded pool (env MUTATION_JOBS, default min(8, CPUs); 1 = serial); output order and
 # verdicts are independent of the pool size.
 # =============================================================================
 from __future__ import annotations
 import os
 import re
 import subprocess
 import sys
 import tempfile
 from concurrent.futures import ThreadPoolExecutor
 from pathlib import Path
@@ -136,43 +140,99 @@ MUTATIONS: list[tuple[str, str, str, str]] = [
 ]
 def mutation_jobs() -> int:
    """Selftests to run at once: env MUTATION_JOBS, else min(8, usable CPUs).
    conformance/lib/run_l1_offline.py exports a smaller MUTATION_JOBS when it is itself
    running several checks in parallel, so the nested pools do not oversubscribe the host.
    MUTATION_JOBS=1 reproduces the old serial loop exactly.
    """
    raw = os.environ.get("MUTATION_JOBS", "").strip()
    if raw:
        try:
            n = int(raw)
        except ValueError:
            n = 0
        if n < 1:
            raise ValueError("MUTATION_JOBS must be a positive integer, got %r" % raw)
        return n
    try:
        cpus = len(os.sched_getaffinity(0))
    except (AttributeError, OSError):
        cpus = os.cpu_count() or 1
    return max(1, min(8, cpus))
 def run_selftest(mutated_src: str) -> subprocess.CompletedProcess:
    with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
        fh.write(mutated_src)
        path = fh.name
    try:
        return subprocess.run([sys.executable, path, "--selftest"],
                              capture_output=True, text=True, timeout=240,
                              stdin=subprocess.DEVNULL)
    finally:
        Path(path).unlink(missing_ok=True)
 def main() -> int:
    if len(sys.argv) != 2:
        print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
        return 2
    src = Path(sys.argv[1]).read_text()
-    problems: list[str] = []
+    try:
-    killed = 0
+        jobs = mutation_jobs()
    except ValueError as e:
        print("mutations.py: %s" % e, file=sys.stderr)
        return 2
    # Pass 1, serial and cheap: every anchor is checked BEFORE anything runs, exactly as the
    # serial loop did, and each mutation's source is built here. None = anchor failure.
    planned: list[tuple[str, str, str | None, str | None]] = []
    for name, why, old, new in MUTATIONS:
        old, new = dedent4(old), dedent4(new)
        # An anchor that no longer matches is a FAILURE, not a skip: it means the code
        # moved and this mutation silently stopped proving anything.
        n = src.count(old)
        if n != 1:
-            problems.append("%s: anchor matches %d times (expected exactly 1) -- the "
+            planned.append((name, why,
                            "%s: anchor matches %d times (expected exactly 1) -- the "
                            "code moved and this mutation no longer proves anything. "
-                            "Re-anchor it against the current source." % (name, n))
+                            "Re-anchor it against the current source." % (name, n), None))
            continue
-        with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
+        planned.append((name, why, None, src.replace(old, new, 1)))
-            fh.write(src.replace(old, new, 1))
+
-            path = fh.name
+    # Pass 2: the selftests, `jobs` at a time. Each child gets its own temp file, its own
-        try:
+    # mkdtemp dirs and its own 127.0.0.1:0 stub port, so they share nothing. Results are
-            proc = subprocess.run([sys.executable, path, "--selftest"],
+    # consumed in MUTATIONS order, so every line below is printed in the same order, with the
-                                  capture_output=True, text=True, timeout=240)
+    # same verdict, as the serial loop produced -- whichever child finishes first.
-        finally:
+    problems: list[str] = []
-            Path(path).unlink(missing_ok=True)
+    killed = 0
-        if proc.returncode == 0:
+    pool = ThreadPoolExecutor(max_workers=jobs)
-            problems.append("%s SURVIVED -- --selftest still passed with this regression "
+    try:
-                            "applied, so nothing in the suite actually checks it. The "
+        futures = [pool.submit(run_selftest, mutated) if mutated is not None else None
-                            "mistake it models: %s" % (name, why))
+                   for (_n, _w, _a, mutated) in planned]
-            continue
+        for (name, why, anchor_problem, _m), fut in zip(planned, futures):
-        killed += 1
+            if anchor_problem is not None:
-        out = proc.stdout + proc.stderr
+                problems.append(anchor_problem)
-        m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
+                continue
-        how = ("%s assertion(s) caught it" % m.group(1) if m
+            proc = fut.result()
-               else "the selftest aborted (caught, less cleanly)")
+            if proc.returncode == 0:
-        print("  killed  %-42s %s" % (name, how))
+                problems.append("%s SURVIVED -- --selftest still passed with this regression "
                                "applied, so nothing in the suite actually checks it. The "
                                "mistake it models: %s" % (name, why))
                continue
            killed += 1
            out = proc.stdout + proc.stderr
            m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
            how = ("%s assertion(s) caught it" % m.group(1) if m
                   else "the selftest aborted (caught, less cleanly)")
            print("  killed  %-42s %s" % (name, how))
    finally:
        # A timeout (or any other exception) still propagates and fails the check, as it did
        # serially; queued mutations are cancelled rather than run for nothing.
        pool.shutdown(wait=True, cancel_futures=True)
    print()
    if problems:
@@ -40,15 +40,19 @@
 #
 #   python3 mutations.py <path-to-extracted-reconciler.py>
 #
-# Runtime is ~60s (one subprocess per mutation, each running the full selftest
+# Runtime is ~85s serial (one subprocess per mutation, each running the full selftest
-# against an in-process stub Keycloak; no network, no cluster).
+# against an in-process stub Keycloak; no network, no cluster). The selftests run in a
 # bounded pool (env MUTATION_JOBS, default min(8, CPUs); 1 = serial); output order and
 # verdicts are independent of the pool size.
 # =============================================================================
 from __future__ import annotations
 import os
 import re
 import subprocess
 import sys
 import tempfile
 from concurrent.futures import ThreadPoolExecutor
 from pathlib import Path
 # (name, the mistake being modelled, anchor, replacement)
@@ -403,42 +407,98 @@ MUTATIONS: list[tuple[str, str, str, str]] = [
 ]
 def mutation_jobs() -> int:
    """Selftests to run at once: env MUTATION_JOBS, else min(8, usable CPUs).
    conformance/lib/run_l1_offline.py exports a smaller MUTATION_JOBS when it is itself
    running several checks in parallel, so the nested pools do not oversubscribe the host.
    MUTATION_JOBS=1 reproduces the old serial loop exactly.
    """
    raw = os.environ.get("MUTATION_JOBS", "").strip()
    if raw:
        try:
            n = int(raw)
        except ValueError:
            n = 0
        if n < 1:
            raise ValueError("MUTATION_JOBS must be a positive integer, got %r" % raw)
        return n
    try:
        cpus = len(os.sched_getaffinity(0))
    except (AttributeError, OSError):
        cpus = os.cpu_count() or 1
    return max(1, min(8, cpus))
 def run_selftest(mutated_src: str) -> subprocess.CompletedProcess:
    with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
        fh.write(mutated_src)
        path = fh.name
    try:
        return subprocess.run([sys.executable, path, "--selftest"],
                              capture_output=True, text=True, timeout=300,
                              stdin=subprocess.DEVNULL)
    finally:
        Path(path).unlink(missing_ok=True)
 def main() -> int:
    if len(sys.argv) != 2:
        print("usage: mutations.py <extracted-reconciler.py>", file=sys.stderr)
        return 2
    src = Path(sys.argv[1]).read_text()
-    problems: list[str] = []
+    try:
-    killed = 0
+        jobs = mutation_jobs()
    except ValueError as e:
        print("mutations.py: %s" % e, file=sys.stderr)
        return 2
    # Pass 1, serial and cheap: every anchor is checked BEFORE anything runs, exactly as the
    # serial loop did, and each mutation's source is built here. None = anchor failure.
    planned: list[tuple[str, str, str | None, str | None]] = []
    for name, why, old, new in MUTATIONS:
        # An anchor that no longer matches is a FAILURE, not a skip: it means the code
        # moved and this mutation silently stopped proving anything.
        n = src.count(old)
        if n != 1:
-            problems.append("%s: anchor matches %d times (expected exactly 1) -- the "
+            planned.append((name, why,
                            "%s: anchor matches %d times (expected exactly 1) -- the "
                            "code moved and this mutation no longer proves anything. "
-                            "Re-anchor it against the current source." % (name, n))
+                            "Re-anchor it against the current source." % (name, n), None))
            continue
-        with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as fh:
+        planned.append((name, why, None, src.replace(old, new, 1)))
-            fh.write(src.replace(old, new, 1))
+
-            path = fh.name
+    # Pass 2: the selftests, `jobs` at a time. Each child gets its own temp file, its own
-        try:
+    # mkdtemp dirs and its own 127.0.0.1:0 stub port, so they share nothing. Results are
-            proc = subprocess.run([sys.executable, path, "--selftest"],
+    # consumed in MUTATIONS order, so every line below is printed in the same order, with the
-                                  capture_output=True, text=True, timeout=300)
+    # same verdict, as the serial loop produced -- whichever child finishes first.
-        finally:
+    problems: list[str] = []
-            Path(path).unlink(missing_ok=True)
+    killed = 0
-        if proc.returncode == 0:
+    pool = ThreadPoolExecutor(max_workers=jobs)
-            problems.append("%s SURVIVED -- --selftest still passed with this regression "
+    try:
-                            "applied, so nothing in the suite actually checks it. The "
+        futures = [pool.submit(run_selftest, mutated) if mutated is not None else None
-                            "mistake it models: %s" % (name, why))
+                   for (_n, _w, _a, mutated) in planned]
-            continue
+        for (name, why, anchor_problem, _m), fut in zip(planned, futures):
-        killed += 1
+            if anchor_problem is not None:
-        out = proc.stdout + proc.stderr
+                problems.append(anchor_problem)
-        m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
+                continue
-        how = ("%s assertion(s) caught it" % m.group(1) if m
+            proc = fut.result()
-               else "the selftest aborted (caught, less cleanly)")
+            if proc.returncode == 0:
-        print("  killed  %-54s %s" % (name, how))
+                problems.append("%s SURVIVED -- --selftest still passed with this regression "
                                "applied, so nothing in the suite actually checks it. The "
                                "mistake it models: %s" % (name, why))
                continue
            killed += 1
            out = proc.stdout + proc.stderr
            m = re.search(r"SELFTEST FAIL: (\d+) of \d+", out)
            how = ("%s assertion(s) caught it" % m.group(1) if m
                   else "the selftest aborted (caught, less cleanly)")
            print("  killed  %-54s %s" % (name, how))
    finally:
        # A timeout (or any other exception) still propagates and fails the check, as it did
        # serially; queued mutations are cancelled rather than run for nothing.
        pool.shutdown(wait=True, cancel_futures=True)
    print()
    if problems:
@@ -62,7 +62,7 @@ spec:
        ports:
        - containerPort: 8080
      - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
        imagePullPolicy: IfNotPresent
        args:
        - --http-address=0.0.0.0:4180
@@ -85,7 +85,7 @@ spec:
        - name: config
          mountPath: /data/config
      - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
        imagePullPolicy: IfNotPresent
        args:
        - --http-address=0.0.0.0:4180
@@ -85,7 +85,7 @@ spec:
        - name: config
          mountPath: /data/config
      - name: oauth2-proxy
-        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+        image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
        imagePullPolicy: IfNotPresent
        args:
        - --http-address=0.0.0.0:4180
@@ -61,7 +61,7 @@ CONFORMANCE_RUNNER_IMAGE_TAG="chainsaw0.2.12-kubectl1.36.4-fc44"
 # chainsaw-job.yaml's @@CONFORMANCE_RUNNER_IMAGE_DIGEST@@ takes the 64 hex digits WITHOUT the
 # `sha256:` prefix this value carries — the trigger must strip it.
 # renovate-digest: image=gitea.home.keithjackson.dev/home-network/conformance-runner tag=main
-CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:65cc7c1ab576fe8cdf45e7f68c7cf4eb5ce7e0db46e62191f6190c51610b20d9"
+CONFORMANCE_RUNNER_IMAGE_DIGEST="sha256:63e1aa74289a8b77ef5cd2ec4e4d01e7fef638f965f2057329bc1991d3074f0d"
 CONFORMANCE_RUNNER_IMAGE="${CONFORMANCE_RUNNER_IMAGE_REPO}@${CONFORMANCE_RUNNER_IMAGE_DIGEST}"
 #
 # AFTERWARDS, TIGHTEN THE BOUNDARY: the header of conformance-trigger-job-shape.yaml asks for
@@ -0,0 +1,139 @@
 ---
 date: 2026-11-08
 branch: feat/cloudflare-tunnel
 title: Cloudflare Tunnel connector, Phase 1 — the component only, isolated, with zero change to live traffic
 ---
 ## What changed
 Ingress is moving to a Cloudflare Tunnel because the GFiber router cannot hairpin and hands LAN
 clients Google's IPv6 DNS, so split DNS keeps breaking. This is **Phase 1: the connector only.**
 external-dns, the Gateway annotations, Pi-hole and CoreDNS are untouched. A later change, by a
 different agent, moves DNS.
 **Operator decisions (2026-11-08), recorded here as the source for the guard:**
 1. Public hostnames stay `*.home.keithjackson.dev` and the apex `home.keithjackson.dev`. The
   operator enables Cloudflare ACM (ordered edge cert or Total TLS) for edge certificates.
 2. **Plex stays OFF the tunnel.** No plex hostname goes through cloudflared (`plex`, `plex-vip` → 404).
 3. **jackett, pihole and semaphore stay LAN/VPN-only**, never via the tunnel. The same goes for the
   non-HTTP L4 VIPs **unifi** and **crafty-mc**. After the PM's read-only audit, the other three
   catalog `exposure: internal` apps, **qbittorrent, radarr and sonarr**, are on the same list. All
   six carry the external-dns `controller: none` route annotation live.
 4. **The Kubernetes API stays local-only.** It must never be reachable through the tunnel.
 5. Pi-hole keeps serving the LAN. Split DNS gets trimmed later, by another agent.
 6. **Headlamp is LAN/VPN-only** (operator decision, relayed by the PM later the same day). It is
   permanently in the 404 list and excluded from the egress backends. Its catalog/route exposure is
   deliberately **not** changed here; the Phase 2 DNS-flip PR does that.
 **Files:**
 - `platform/cloudflared/` (new):
  - `config.yaml` is the locally-managed tunnel config, projected through a content-hashed
    `configMapGenerator`, so edits roll the pods. It has explicit `http_status:404` rules for
    plex, plex-vip, jackett, pihole, pihole-dns, semaphore, qbittorrent, radarr, sonarr, headlamp,
    unifi and crafty-mc. Then the apex and `*.${BASE_DOMAIN}` go to
    `https://cilium-gateway-shared.gateway.svc.cluster.local:443` with `matchSNItoHost: true`. Last
    comes a catch-all 404. The tunnel UUID is a **literal** in this file (`REPLACE-ME-TUNNEL-UUID`
    until enrolment), not a cluster-var, so the Phase 2 PR can match it without touching `clusters/**`.
  - `cloudflared-credentials.sops.yaml` is SOPS-encrypted and holds `REPLACE-ME` placeholders, with
    a `.example` hand-off sibling. I encrypted the placeholder (not a credential) in the toolbox
    with the root `.sops.yaml` recipient.
  - `deployment.yaml` runs 2 replicas with a hard `topologySpreadConstraints` across nodes. The image
    is `cloudflare/cloudflared:2026.10.7@sha256:4955524a…` (tag + digest, Renovate `kubernetes`
    manager). It runs as non-root 65532, with a read-only root fs, all capabilities dropped and no
    SA token. Requests are 10m/64Mi and limits 500m/256Mi. Readiness uses `/ready`. Liveness uses
    `/healthcheck`, so an edge outage cannot restart-loop the pods. The pods are labelled
    `egress-class: internet`. A PDB sets `minAvailable: 1`.
  - `networkpolicy.yaml`:
    - A namespace `default-deny` with DNS only. It has a **single owner**: the namespace is not
      `app-namespace=true`, so Kyverno's generate rule never writes it.
    - A Prometheus scrape allow on :2000.
    - The egress CNP allows world tcp/udp 7844 and tcp 443, plus the `ingress` entity on tcp 443.
      It also allows **Gateway backends by `io.cilium.k8s.namespace.labels.gateway-access=shared`,
      minus the LAN-only namespaces.**
  - `servicemonitor.yaml` (headless metrics Service + ServiceMonitor) and `alerts.yaml` (`CloudflaredTunnelDown`, `CloudflaredReplicasDegraded`)
    ship with `alerts.promtool-test.yaml`, and `README.md` documents the component.
 - `clusters/base/cloudflared.yaml` (new) is its own Flux Kustomization: `dependsOn: platform-config`,
  **`wait: false`**, prune, and postBuild `cluster-vars`. It is listed in `clusters/base/kustomization.yaml`.
 - `clusters/{dev,prod}/kustomization.yaml` each gain a SOPS `decryption` patch for `cloudflared`.
 - `conformance/tests/l0-static/cloudflared-ingress-rules/` (new, whole-repo scope) asserts several
  things:
  - Every rule targets either 404 or exactly the Gateway Service. The Service name is derived from
    `platform/gateway/gateway.yaml`, and API markers are named explicitly.
  - The excluded hosts 404 before the first Gateway rule, the Gateway rules use `matchSNItoHost`,
    there is no `noTLSVerify`, and a catch-all closes the list.
  - The CNP has no kube-apiserver, host, remote-node, cluster or `all` entity, no CIDR, and the full
    NotIn list. The pods carry egress-class internet and have no SA token.
  - The Flux tier is isolated: wait:false, not in `platform/kustomization.yaml`, nothing
    `dependsOn: cloudflared`, and both overlays decrypt it.
  - **Mutation self-test:** an added API rule, plex moved below the wildcard, and a dropped
    catch-all must each be flagged.
 - `conformance/tests/l0-static/alert-metrics-have-scrape-targets/check.sh` gains an additive
  `cloudflared_` scrape owner, with a networkpolicy companion, in the same shape as the kured entry.
 - `runbooks/cloudflare-tunnel.md` (new) is registered in `runbooks/kustomization.yaml` and the
  README index.
 **Design choices:**
 - **Isolation (the merge-before-credentials question).** The PR is safe to merge with placeholders.
  The Deployment CrashLoops, but in its own `wait:false` Kustomization that nothing depends on. Its
  Ready condition tracks the apply, not pod health, so neither `platform`, `platform-config` nor any
  apps wave can stall. There is no Certificate or HelmRelease in the path for health checks to hang
  on. Health is reported by the two alerts instead. `CloudflaredTunnelDown` **will fire from merge
  until enrolment**. That is honest, and it is `warning` because nothing routes through the tunnel
  yet. Raise it to critical in the DNS-flip change.
 - **Egress to backends, not `{}`.** Under Cilium 1.20 (CVE-2025-30162), the Gateway's Envoy
  re-checks the source pod's egress against the resolved backend. An egress rule that names only
  the Gateway would therefore make every tunnelled request `403 Access denied`
  (decisions/2026-10-30-gitea-admit-buildkit.md; catalog/render.py, `r_internet_egress`). The
  gitea-actions `- {}` pattern would also work, but it admits kube-apiserver and host, which
  decision 4 rules out. The namespace-label selector names every Gateway backend and nothing else.
  **Backends need no ingress change.** Envoy reaches them as `reserved:ingress`, and the 2026-10-30
  measurement showed the identity check sits on the source's egress: gitea-actions is in none of
  Gitea's ingress allows and still reaches it. So no app's `inClusterClients` was touched.
 - **The alerts cannot self-silence.** Both expressions use `or on() vector(0)`, so absent series
  (a broken scrape, zero pods, KSM gone) fire. The promtool tests pin the absent case and the
  working-then-stale re-fire case.
 ## What I verified
 - **The Gateway Service, measured live (read-only, `k3s-prod`, 2026-11-08).** It is
  `gateway/cilium-gateway-shared`: type LoadBalancer, ClusterIP `10.43.225.79`, EXTERNAL-IP
  `192.168.2.20`, ports 80/443, no selector, labelled `gateway.networking.k8s.io/gateway-name=shared`.
  It belongs to Gateway `gateway/shared` (class `cilium`).
 - **The namespace selector, measured live.** Cilium identities carry the label
  `k8s:io.cilium.k8s.namespace.labels.gateway-access` (read from a gitea identity). Every namespace
  with an HTTPRoute (crafty, gitea, headlamp, homepage, jackett, keycloak, monitoring, nextcloud,
  oauth2-proxy, pihole, plex, qbittorrent, radarr, runbooks, seerr, semaphore, sonarr) is
  `gateway-access=shared`.
 - **Nothing exists yet.** Namespace `cloudflared` does not exist on prod. **cloudflared is not in
  the pinned toolbox**, so the runbook runs the pinned image under podman instead.
 - The image tag and digest came from Docker Hub's API: `2026.10.7` = `latest` =
  `sha256:4955524a…c9b9c5ba`, published 2026-10-30.
 - The credentials Secret is encrypted: `stringData.credentials.json` is `ENC[…]`, with the age
  recipient from the root `.sops.yaml`.
 - The new L0 check passes on the host (47 assertions, mutation self-test included). The toolbox
  gate result is reported in the PR description. I ran it after committing.
 ## What I did NOT verify
 - **UNVERIFIED: nothing ran in a cluster.** No pod has started, and no tunnel exists. I have not
  observed placeholder CrashLoop behaviour itself. I expect cloudflared to treat
  `REPLACE-ME-TUNNEL-UUID` as a tunnel name, find no origin cert, and exit, but whatever it does,
  the pods are not Ready and the tier is non-gating.
 - **UNVERIFIED: the CVE-2025-30162 path for cloudflared specifically.** The selector form was
  checked against live identity labels, but no tunnelled request has been made. The runbook's
  step 6 `kubectl debug` probe exists to prove "real backend, no 403" from the pod's own identity.
 - **UNVERIFIED: whether the `ingress` entity rule's `toPorts: 443` restriction holds at Envoy.**
  buildkit's working rule has no port restriction. If the canary shows drops, removing the port
  restriction there is the first thing to try.
 - The metric name `cloudflared_tunnel_ha_connections` and the `/ready` / `/healthcheck` endpoints
  come from my knowledge of cloudflared, **not measured** against 2026.10.7.
 - The Cloudflare edge egress is `world` + ports, not Cloudflare CIDRs. I did not narrow it blind.
 - Egress `world:443` "for the API" follows the brief. I have not confirmed that a credentials-file
  tunnel needs it at all.
 - The Cloudflare-side settings (ACM, Total TLS / ordered cert, Bot Fight Mode, BIC/WAF, WebSockets)
  are documented for the operator but not inspected. I have no Cloudflare access.
 - The headlamp, qbittorrent, radarr and sonarr catalog exposures are deliberately unchanged
  (Phase 2).
@@ -0,0 +1,59 @@
 ---
 date: 2026-11-08
 branch: fix/cloudflared-creds-permission
 title: cloudflared CrashLoop — credentials Secret unreadable by uid 65532; add fsGroup, pin it in L0
 ---
 ## What changed
 - **`platform/cloudflared/deployment.yaml`**: pod `securityContext.fsGroup: 65532`. Root cause of
  the CrashLoop that followed #152/#153: the pod runs as `65532:65532` with no `fsGroup`, and the
  Secret volume is `defaultMode: 0440`. The kubelet therefore laid `credentials.json` down as
  `root:root 0440`, which uid 65532 (not owner, not in gid 0) cannot read. cloudflared logged
  `couldn't read tunnel credentials ... permission denied`, then fell back to origin-cert lookup
  and ended on the misleading `error parsing tunnel ID: Error locating origin cert`. With
  `fsGroup` the files are group-owned by 65532, readable by the container and no other uid; the
  mode stays 0440 (not loosened to 0444). Same fix as `platform/keycloak/client-secret-reconciler.yaml`.
  Still non-root, `runAsNonRoot: true`, unchanged otherwise.
 - **`conformance/tests/l0-static/cloudflared-ingress-rules/check.sh`**: new assertion (7). For each
  Secret volume a container mounts, it models kubelet ownership (uid 0, gid = fsGroup or 0, item
  mode / defaultMode / API default 0644) against the container's effective runAsUser and its groups
  (runAsGroup + fsGroup + supplementalGroups) and fails if the file is unreadable. An unset
  runAsUser fails (image default is not knowable offline). Conservative: it does not credit the
  kubelet OR-ing group-read into fsGroup volumes. Mutation self-test: fsGroup removed, mode 0400,
  runAsUser unset must each be flagged. Chainsaw step description updated to match.
 - **`runbooks/cloudflare-tunnel.md`**: new first "likely cause" (permission denied on
  credentials.json → the origin-cert error is a red herring) plus a "read the first error line"
  note; causes renumbered. New section **Rotating the tunnel credential**: create a new tunnel,
  swap the UUID and the encrypted credentials, merge, verify (new tunnel has two connectors, old
  has none), then `cloudflared tunnel cleanup`/`delete <old>`.
 - NOT touched, on purpose: `tunnel:` in `config.yaml` and `cloudflared-credentials.sops.yaml`. The
  operator is rotating the credential (it was exposed in a transcript) in a separate PR.
 ## What I verified
 - Live (k3s-prod, read-only, 2026-11-08 ~21:38Z): Deployment pod securityContext is
  `{"runAsGroup":65532,"runAsNonRoot":true,"runAsUser":65532,"seccompProfile":...}`, no
  `fsGroup`; creds volume `defaultMode: 288` (= 0440). All three pods CrashLoopBackOff (38
  restarts). Secret `cloudflared-credentials` has the one key `credentials.json` (key names only).
 - Pinned image `cloudflare/cloudflared@sha256:4955524a...` config (`skopeo inspect --config`):
  `User: 65532:65532`, `Entrypoint: [cloudflared]`, `Cmd: [version]`.
 - The new assertion FAILS (rc 1) on the pre-fix deployment.yaml with
  `container cloudflared (uid 65532, groups [65532]) cannot read Secret volume creds ... files are
  0:0 mode 0440`, and PASSES (51 assertions) with the fix. All three mutants are flagged.
 - Kyverno admission policies under `platform/kyverno/admission/` that mention securityContext
  fields are scoped to the conformance-trigger and pm-agent Jobs only; none constrains fsGroup here.
 - Toolbox gate: see the PR description for the rc and skipped stages.
 ## What I did NOT verify
 - **That the pods actually become Ready after merge.** Not deployed by me; Flux applies it. The
  file mode inside the running container was not observed (pods crash before an exec is possible,
  and exec would be an operator action anyway). The readability claim rests on the kubelet's
  documented fsGroup behaviour and the identical, working keycloak reconciler.
 - Whether the CURRENT credential is otherwise valid (UUID/TunnelID match, tunnel exists). Moot:
  the operator is rotating it.
 - The rotation runbook section has not been executed; `cloudflared tunnel cleanup` / `delete`
  syntax is from the CLI's documented commands, not run here.
 - The deployment comment says the image ENTRYPOINT is `cloudflared --no-autoupdate`; the pinned
  image's ENTRYPOINT is plain `cloudflared`. Left unchanged (out of scope); flagged for follow-up.
@@ -0,0 +1,74 @@
 ---
 date: 2026-11-08
 branch: fix/keycloak-operator-crds
 title: "Keycloak 26.8.2 operator CrashLoop: install its two new CRDs, raise its memory"
 ---
 ## What changed
 **Incident.** PR #121 (Renovate) bumped `platform/keycloak/kustomization.yaml`'s three
 `keycloak-k8s-resources` URLs 26.0.6 -> 26.8.2. Upstream `kubernetes/` at 26.8.2 holds **four**
 CRDs, not two (GitHub contents API, `?ref=26.8.2`):
 | file | at 26.0.6 | at 26.8.2 |
 |---|---|---|
 | `keycloaks.k8s.keycloak.org-v1.yml` | yes | yes |
 | `keycloakrealmimports.k8s.keycloak.org-v1.yml` | yes | yes |
 | `keycloakoidcclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
 | `keycloaksamlclients.k8s.keycloak.org-v1.yml` | — | **new** (v2alpha1) |
 | `kubernetes.yml` | yes | yes (now carries RBAC for both new kinds) |
 (`cluster-wide/` is a separate variant; upstream's own `kubernetes/kustomization.yml` lists exactly
 the five files above.) The 26.8.2 operator starts an informer for every controller it has and dies
 on the first missing kind: `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1
 ... GET .../namespaces/keycloak/keycloaksamlclients ... Not Found`, exit 1, CrashLoopBackOff. Flux
 `platform` (`wait: true`, 5m) sat in health checks and every dependent Kustomization reported
 `dependency ... is not ready`. The old 26.0.6 ReplicaSet kept serving (RollingUpdate 25%/25%), so SSO
 stayed up.
 **Fix.**
 - `platform/keycloak/kustomization.yaml`: added the two CRD URLs, each with the same
  `# renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources` marker (the repo's
  custom.regex manager matches all five — checked by running its regex over the file). Header
  comment updated to v26.8.2, plus a warning that Renovate cannot add a CRD a release introduces.
 - Same file: a `patches:` JSON6902 patch on Deployment `keycloak-operator`: memory limit
  450Mi -> **768Mi**, request 450Mi -> **512Mi**. 450Mi is upstream's default (unchanged at 26.8.2,
  so not something the bump changed). Evidence it is too small here: the 26.0.6 pod had **154
  restarts**, `lastState.terminated.reason: OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and
  `kubectl top` showed **421Mi / 450Mi**. 26.8.2 runs four controllers instead of two on Quarkus
  3.40.1.1 (was 3.15.1). No variable in the operator to disable watching unused kinds was used —
  installing the CRDs is upstream's supported shape.
 - `platform/keycloak/README.md` "Upgrading Keycloak": the diff-the-CRD-list step; "three URLs" ->
  "the release URLs". `renovate.json5`: same wording fix in a comment only.
 ## What I verified
 - Upstream file lists at 26.0.6 and 26.8.2 (GitHub API) and `diff` of `kubernetes.yml` between the
  two: the new kinds' ClusterRoles + RoleBindings ship in it, and the Deployment's resources are
  still 450Mi/450Mi.
 - Live, read-only (`kubectl --context k3s-prod`, 2026-11-08 ~05:57Z): only `keycloaks` and
  `keycloakrealmimports` CRDs installed; Deployment strategy RollingUpdate maxSurge/maxUnavailable
  25%; RS `keycloak-operator-58dff6c4` (26.8.2) 0/1 CrashLoopBackOff, RS `78dbb75f64` (26.0.6) 1/1;
  operator log is the informer error above; Flux `platform` `Healthy=Unknown` "Running health
  checks for revision main@sha1:e8d2666"; no NetworkPolicy and no CiliumNetworkPolicy in ns
  `keycloak`.
 - No sensitive path touched (`infra/gitea/approval-policy.json`), so this should merge on green.
 - Gate: see the PR body for the exact rc and stages of the run against the final commit.
 ## What I did NOT verify
 - **That the 26.8.2 operator becomes Ready after this merges.** Only the two missing kinds were
  observed failing; another start-up failure behind them is possible.
 - **That 768Mi is enough** — no 26.8.2 memory measurement exists yet; re-check `kubectl top` after
  a week.
 - **The Keycloak server upgrade 26.0.6 -> 26.8.2 itself.** Once the operator runs it will roll
  keycloak-0/1 and Keycloak migrates the DB schema. The CR's `spec.update.strategy` defaults to
  `RecreateOnImageChange` — expect a **full SSO outage** while both pods recreate, not a rolling
  one. Back up / confirm the latest CNPG backup first if that matters.
 - **The 26.8.2 Keycloak CRD's `spec.networkPolicy.enabled` defaults to `true`.** The operator will
  create a NetworkPolicy selecting the Keycloak pods (first policy in that namespace). With no
  `from` lists set it allows all sources on the HTTP/management ports, so Gateway, ServiceMonitor
  and the reconciler Jobs should still reach it — but this was read from the CRD text, not observed.
 - **The passkey reconciler's 26.0.6 Admin API assumptions against 26.8.2** (README "Upgrading
  Keycloak" list) — not re-checked here; PR #121 merged without it.
 - `catalog/keycloak.yaml`'s documentation-only image tag was not updated.
@@ -0,0 +1,60 @@
 ---
 date: 2026-11-08
 branch: fix/keycloak-servicemonitor-owner
 title: One owner for the keycloak ServiceMonitor — disable the 26.8 operator's (409 loop stalled platform-config)
 ---
 ## What changed
 - `platform/keycloak/keycloak-instance.yaml`: added `spec.serviceMonitor.enabled: false`.
  The 26.x Keycloak operator creates its own ServiceMonitor, named after the CR (`keycloak/keycloak`),
  whenever `metrics-enabled=true` and the ServiceMonitor CRD exists. `serviceMonitor.enabled` defaults
  to `true` in the 26.8.2 CRD, both in v2alpha1 and in v2beta1 (v2beta1 is now the storage version).
  That is the same name and namespace as the repo's Flux-owned `platform/keycloak/servicemonitor.yaml`.
  After the 26.0.6 -> 26.8.2 bump (#121/#144) the operator's PATCH against it returned
  `409 Conflict` and it ran out of retries. The CR then sat at `Ready=False`/`HasErrors=True`, and
  `platform-config` (`wait: true`, which health-checks every object it applies, this CR included)
  blocked everything that `dependsOn` it: velero, apps-vms, apps-wave-0 (and so waves 1/2 and
  pm-agent), homepage, oauth2-proxy, headlamp, conformance, vpn-ingress, crafty, runbooks.
 - `platform/keycloak/servicemonitor.yaml`: a comment saying it is the sole owner.
 - **Why ours is kept:** these are written against it: the L0 `servicemonitor-ports-resolve`
  allow-list entry `platform/keycloak/servicemonitor.yaml:keycloak`, the L2 `keycloak-monitored`
  chainsaw test, the README's ownership table, and the dashboard's `up{namespace="keycloak"}`.
  The operator's version (upstream `KeycloakServiceMonitorDependentResource` @26.8.2) scrapes the same
  target: port name `management` (9000 on `keycloak-service`), path `/metrics`, interval 30s,
  selector = the operator's instance labels. It offers nothing ours lacks, so having two owners
  would add a fight and gain nothing.
 ## What I verified
 - Live, read-only, `k3s-prod` ~06:25Z: the CRD schema field `spec.serviceMonitor.enabled`
  (`default: true`) exists in both served versions. The live `keycloak` ServiceMonitor has exactly
  **one** managedFields manager, `kustomize-controller` (Apply, 2026-09-05), **no ownerReferences**,
  and labels `kustomize.toolkit.fluxcd.io/name=platform-config`, `release=kube-prometheus-stack`. So
  the operator's apply never landed, and nothing on the object ties it to the Keycloak CR.
 - The ServiceMonitor will not be deleted when the operator is switched off: the operator's dependent
  informer is filtered on `Constants.DEFAULT_LABELS_AS_STRING`
  (`app.kubernetes.io/managed-by=keycloak-operator`). Our object does not carry that label and has
  no ownerReference, so the operator cannot see it as its secondary and cannot delete or GC it.
  This conclusion comes from reading the code. It was not observed.
 - The other operator dependents in ns `keycloak` do not collide with anything: `keycloak-service`,
  `keycloak-discovery`, the StatefulSet `keycloak`, `keycloak-initial-admin`, and
  `keycloak-network-policy` (owner Keycloak, no Flux label). The repo declares none of these names.
  It only *references* `keycloak-service` by name (HTTPRoute, reconcilers). There are no PDBs in the
  namespace, and the Ingress is already `enabled: false`. The only NetworkPolicy in the namespace is
  the operator's. The repo has no Keycloak NetworkPolicy, and no Kyverno-generated default-deny is
  present there. NetworkPolicies add up, so the operator's allow-policy cannot fight another owner.
  Left at its default.
 - Local gate: see the PR body. It records the rc and stages.
 ## What I did NOT verify
 - That the CR reaches `Ready=True`/`HasErrors=False` after merge. The status also says
  "Performing Keycloak upgrade, scaling down the deployment". I am inferring that this half clears
  once the workflow stops erroring. It is not proven.
 - That the operator stops issuing the PATCH once `enabled: false` is applied. Watch the operator
  logs for 409.
 - That `platform-config` and its downstream Kustomizations go Ready on the new revision, and that
  Prometheus keeps exactly one `serviceMonitor/keycloak/keycloak/0` pool with targets up.
 - The operator's actual label set for `DEFAULT_LABELS_AS_STRING`. I took it from the observed labels
  on operator-created objects, not from `Constants.java`.
@@ -0,0 +1,116 @@
 ---
 date: 2026-11-08
 branch: feat/parallel-l0-checks
 title: L0 offline checks, mutation suites and yamllint run in bounded parallel pools (gate 184s -> 32s locally)
 ---
 ## What changed
 The gate's wall time was dominated by serial loops over independent subprocesses. Three of
 them now run in bounded pools; none of them changes a verdict, an exit code, or the order of
 anything printed.
 - `conformance/lib/run_l1_offline.py` — the per-check loop (L0 and L1 offline checks) runs in
  a `ThreadPoolExecutor` of `bash <check>` children. `--jobs N` / env `CONF_OFFLINE_JOBS`,
  default `min(8, usable CPUs)` (`sched_getaffinity`, so a cgroup cpuset is respected);
  `--jobs 1` is the old serial runner. Results are consumed in SUBMISSION order, so `cases`,
  the JUnit, the step summary and the 0/1/3 aggregation are built from exactly the same
  ordered list as before. Additive output only: one `[offline] running N check(s), jobs=…`
  line, and a trailing `[N.Ns]` wall time on each per-check status line. Children get
  `stdin=/dev/null` (parallel children must not share the runner's stdin).
 - `conformance/tests/l0-static/keycloak-{passkey,client-secret}-reconciler/mutations.py` —
  pass 1 (serial) checks every anchor and builds each mutated source, exactly as before;
  pass 2 runs the `--selftest` children in a pool (env `MUTATION_JOBS`, default
  `min(8, CPUs)`, `1` = serial). Verdicts are consumed in `MUTATIONS` order. A timeout
  (`TimeoutExpired`) still propagates and fails the check; queued mutations are cancelled.
  An invalid `MUTATION_JOBS` exits 2 (fails the check, never skips it).
 - **Nested-pool budget.** When the outer runner is parallel it exports
  `MUTATION_JOBS = max(2, min(8, (cpus - jobs + 2) // 2))` unless the caller already set
  one. Reason: only those two checks fan out; the other `jobs-2` workers are single
  processes, so the two suites split the CPUs those leave free. Peak processes ≈
  `(jobs-2) + 2*inner` ≤ cpus whenever cpus ≥ jobs+2 (24 CPUs: 6 + 16 = 22), and ~1.25–1.5×
  cpus on a small runner (4 CPUs: 2 + 4 = 6). The floor of 2 is deliberate: a 1-wide inner
  pool would make the 40-mutation passkey suite (~85s serial) the critical path of the stage.
 - `ci/lint.sh` — yamllint runs over `LINT_JOBS` (default `min(8, nproc)`) CONTIGUOUS slices
  of the sorted file list concurrently; each slice's output is buffered to its own file and
  printed in slice order, so the text is the same as one serial call and a `file` header is
  never separated from its `line:col` findings. Stage rc = the highest slice rc; a slice that
  leaves no rc file counts as 1. Invalid `LINT_JOBS` exits 2.
 ### Parallel-safety audit (all 25 `l0-static/*/check.sh`, 3 `l1-rendering/*/render-check.sh`)
 Grepped every check for fixed `/tmp` paths, `_artifacts`/`ARTIFACTS`/`COVERAGE_OUT`,
 `write_text`/`open(…,"w")`, `mktemp`/`mkdtemp`/`TemporaryDirectory`, sockets/ports, `git`
 writes and `run.sh` invocations, and read what each writer writes to:
 - **Scratch state: all private.** Every writer uses its own `mktemp -d` / `mkdtemp` /
  `TemporaryDirectory` (placeholder-guard-coupling, spec-immutability-enforced,
  whole-repo-invariants-wired, auditability-proofs-producible, alert-rules-unit-tested and its
  `platform/monitoring/tests/run.sh`, internal-services-publish-no-public-dns,
  oauth2-proxy-{api-routes-per-app,cookie-names-distinct}, both keycloak reconcilers, the three
  L1 render checks). No fixed `/tmp` path anywhere.
 - **Git writes:** only spec-immutability-enforced, and only `git init/add/commit` inside its
  own `mktemp` repo. Everything else is `git ls-files`/`ls-tree`/`log`/`rev-parse` (read-only).
 - **Ports:** the reconciler selftests bind `127.0.0.1:0` (kernel-assigned); the passkey
  selftest also dials `127.0.0.1:1` as a deliberate refused connection. No fixed listener.
 - **Ansible** (dev-nodes-provisioned-as-code): `--syntax-check`/`--list-hosts` only,
  `retry_files_enabled = False`; its local tmp is per-process.
 - **The one shared writer:** auditability-proofs-producible runs
  `./run.sh --coverage-only --requirements …`, which REWRITES
  `conformance/_artifacts/coverage/coverage.{json,md}` (it inherits `CONF_ARTIFACTS`). That was
  already true serially. No other offline check reads or writes that directory (the only
  reader is `tests/meta/harness-selfcheck`, not an offline check), and the outer runner
  writes coverage BEFORE and the JUnit AFTER the pool, so the final state is the same as
  serial. Not serialised; left as is and recorded here. `lib/status.py` (run by that check)
  reads `_artifacts/report.junit.xml` as an advisory — written only after the pool, unchanged.
 - whole-repo-invariants-wired calls `run.sh --print-selection` (writes nothing) and the
  runner itself against synthetic dirs in its own `mktemp`.
 Result: no check needed fixing or serialising. A new check that needs shared state must say
 so; the runner's docstring now states that contract.
 ## What I verified
 All in the pinned toolbox (`dev.sh run`, `DEV_KUBECONFIG=/dev/null`, ENGINE shim adding a ro
 mount of the main checkout's `.git`; spec repo resolved as a sibling, spec-immutability
 `pass`), host 24 CPUs, every `dev.sh` call under `flock`.
 - **Gate, before** (origin/main `1a33814`, detached scratch worktree + spec mount):
  `GATE_RC=3`, in-container wall **183.9s**; `l0` stage **168.3s**, `lint` **7.8s**.
  "Stages that did not run: gitea-settings" (plus the expected on-base
  touched-set-not-determinable note, since main has no branch commits).
 - **Gate, after** (this branch, commit `a775082`): `GATE_RC=3`, in-container wall **32.1s**;
  `l0` **21.1s**, `lint` **2.4s**, `l1` 2.3s. "Stages that did not run: gitea-settings" — the
  only not-run stage. **rc 3, not green.** Caveat: the branch run selected 23 L0 checks (the
  touched-requirement scoping dropped dev-nodes-provisioned-as-code and
  placeholder-guard-coupling), the main run 25.
 - **Same-tree L0 comparison** (all 25 l0 dirs, same tree, same container):
  `CONF_OFFLINE_JOBS=1 MUTATION_JOBS=1` **164.0s / 165.3s** vs default (jobs=8,
  MUTATION_JOBS=8) **19.2s / 18.7s**. JUnit `cmp`-identical, step summary `cmp`-identical,
  stderr identical except the timing suffixes and the JUnit path in the last line.
 - **Mutation suites, host:** passkey 83.2s → 10.5s (`MUTATION_JOBS=8`), client-secret
  11.2s → 2.0s; output of `MUTATION_JOBS=1` and `=8` byte-identical to the pre-change script's,
  rc 0 all round.
 - **Falsification (not committed):** in a scratch worktree at `a775082`, made the passkey
  mutation "the protected-subtree screen is never called" a no-op (replacement == anchor).
  Host: `MUTATION_JOBS=1` and `=8` both rc 1, output byte-identical (with a second, broken
  anchor also injected: both reported). Toolbox: runner rc 1 both serial and parallel,
  `L0 FAIL keycloak-passkey-reconciler … 1 of 40 mutation(s) not proven … SURVIVED`; full
  gate **rc 1**, `stage l0: FAIL`, `MERGE GATE: BLOCKED`. Reverted; scratch worktree removed.
 - **lint.sh, host scratch repo** (31 YAML files, two broken): `LINT_JOBS=1/4/8` all rc 1,
  stdout identical, findings named `y/f07.yaml` / `2:1 … key-duplicates` etc.; clean tree rc 0;
  `LINT_JOBS=x` rc 2.
 ## What I did NOT verify
 - **CI timings at the time of writing.** The PR's CI run is measured after push and reported
  on the PR / handoff message, not here. The CI runner's CPU count is UNVERIFIED — the new
  `[offline] running … jobs=… cpus=…` line in the CI log is what will show it. On a small
  runner the speedup will be smaller than locally.
 - **approval-policy** will be red on CI: `ci/**` and `conformance/**` are sensitive paths and
  need the operator's review. I did not and must not override it.
 - Flakiness under load: the selftests have no wall-clock assertions I found, and 4 toolbox
  runs + several host runs were all stable, but that is a handful of runs, not a soak.
 - The coverage files under `conformance/_artifacts/coverage/` end in the same state as before
  only because a single check rewrites them; I did not change that pre-existing quirk.
 - L2/L3 conformance: not run (no cluster in CI, as before).
@@ -0,0 +1,65 @@
 ---
 date: 2026-11-08
 branch: chore/renovate-config-cleanup
 title: Renovate config — hand-migrate the docker key, track gitea/runner, keep HelmRelease range lower bounds
 ---
 ## What changed
 - **`renovate.json5`: the deprecated top-level `"docker": {"enabled": true}` is hand-migrated** to
  `packageRules[0] = {"matchCategories": ["docker"], "enabled": true}`, the exact shape Renovate's
  own migration produced on `origin/renovate/migrate-config` (PR #139). Its comment moved with it.
  This replaces #139, which was closed because its rewrite deleted every comment in the file. No
  other comment was removed or reflowed (diff is additions plus the moved block).
 - **`gitea/runner` added to the blast-radius rule** beside `gitea/act_runner`, with a comment:
  act_runner was renamed upstream to gitea/runner after 0.6.1; Renovate does not follow renames,
  so the actual image migration is a separate manual PR.
 - **Two new packageRules for flux/helm `>=A <B` ranges:** `rangeStrategy: "bump"`, plus
  `matchUpdateTypes: ["bump"] -> enabled: false`, so in-range releases (which Flux adopts on its
  own) do not become PRs. Cause of the bug, from the 44.167.0 source: rangeStrategy is unset
  (`auto`); the flux manager has no `getRangeStrategy`/`updateLockedDependency`, so
  `lib/modules/manager/index.ts getRangeStrategy` resolves `auto` to `replace`; helm versioning is
  the npm api verbatim, and npm `getNewValue` under `replace` turns `>=1.12.0 <1.13.0` into
  `<1.14.0` (what PRs #131, #130 and #115 did).
 - **`platform/headlamp/helmrelease.yaml`: `version: "<0.47.0"` -> `">=0.46.0 <0.47.0"`**, comment
  refreshed. The live chart is 0.46.0, so the set of installable charts is unchanged.
 ## What I verified
 - **Semantic diff of #139:** I parsed `renovate.json5` on `origin/renovate/migrate-config` and on its
  parent `5b484f6` with json5 (the copy inside the renovate image). With `docker` removed from the
  parent and `packageRules[0]` removed from the migrated file, the two are identical. So the only
  change was `{"matchCategories":["docker"],"enabled":true}`, inserted at packageRules[0].
 - **Validator, `ghcr.io/renovatebot/renovate:44.167.0`** (the tag that `.gitea/workflows/renovate.yaml`
  pins as `container.image`): `renovate-config-validator` gives `Config validated successfully`
  (rc 0), and so does `--strict` (rc 0), both as a file path and in repo mode. **`--strict` on
  origin/main's file gives rc 1** and prints the `docker` migration diff. Nothing is left to migrate.
 - **`getNewValue` called in the 44.167.0 image** (helm versioning), currentValue
  `>=1.12.0 <1.13.0`, newVersion 1.13.0: replace `<1.14.0`, widen `>=1.12.0 <1.14.0`,
  bump `>=1.13.0 <1.14.0`, pin/update-lockfile `<1.14.0`. On `<0.47.0`, bump returns the value
  unchanged, so it would never propose anything for that range. That is why the rule is scoped to
  the `>=A <B` shape.
 - **Local dry run** (`--platform=local`, `RENOVATE_DRY_RUN=lookup`, 44.167.0, live chart repos)
  against a fixture repo of four HelmReleases under `platform/`:
  - With this branch's config: `No config migration necessary`. longhorn `>=1.9.0 <1.10.0` ->
    `>=1.13.0 <1.14.0` (minor). kube-prometheus-stack `>=88.0.0 <89.0.0` -> `>=93.1.0 <94.0.0`
    (major). The in-range bump `>=88.6.5 <89.0.0` was dropped (`Filtered out 1 disabled update(s)`).
    headlamp `>=0.46.0 <0.47.0` and `<0.47.0`: no update.
  - Control run with origin/main's config: `Config migration necessary`, longhorn -> `<1.14.0`,
    kube-prometheus-stack -> `<94.0.0`. This reproduces the reported bug.
 - Live headlamp HelmRelease (read-only kubectl, k3s-prod): spec `<0.47.0`, history chartVersion
  `0.46.0`, Ready `True`.
 ## What I did NOT verify
 - **I have not seen a real scheduled Renovate run against this repo with this config.** The claim that
  Renovate stops reopening the migrate-config PR rests on the validator and on `No config migration
  necessary` in the local dry run, not on the Gitea bot. After merge, watch the next run, and if
  `renovate/migrate-config` is still on the remote, delete it.
 - **Under bump, the new lower bound is the release Renovate proposes**, e.g. `>=1.13.2 <1.14.0`, not
  `>=1.13.0`. Flux still installs the newest release in the window. Ranges with other shapes keep the
  default `replace`: `3.9.x`, `~`, ceiling-only, and exact pins.
 - **The open PRs #131, #115 and any others still carry unbounded `<X` values.** They were generated
  under the old strategy, and Renovate should rebase or regenerate them under bump on its next run.
  I have not checked that it does.
 - The gitea/runner rename itself (a new image reference and any config changes) is not done here.
@@ -0,0 +1,48 @@
 ---
 date: 2026-11-08
 branch: rework/136
 title: "PR #136 rework: regenerate L1 render goldens for the oauth2-proxy v7.15.6 sidecar bump"
 ---
 ## What changed
 - Renovate PR #136 bumps `OAUTH2_PROXY_IMAGE` in `catalog/render.py` (and the generated
  `apps/wave-*` / `platform/crafty` pins) from `v7.8.1` to `v7.15.6`. Its gate failed at L1
  because three byte-for-byte render goldens still carried the old tag:
  - `conformance/tests/l1-rendering/container-workload-render/golden/expected.yaml`
  - `conformance/tests/l1-rendering/egress-class-render/golden/egress-vpn.expected.yaml`
  - `conformance/tests/l1-rendering/egress-class-render/golden/egress-internet.expected.yaml`
    (CI only reported the vpn one because `render-check.sh` stops at the first failing diff;
    the internet golden had drifted too.)
 - Regenerated all three in the pinned toolbox with the exact invocation the check scripts use:
  `python3 catalog/render.py <fixture> > <golden>`. The diff in each is **one line**:
  `image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1` -> `...:v7.15.6`. Nothing else moved.
 - `platform/crafty/README.md`: the "Sidecar" row described the current pin; updated to v7.15.6.
 - Deliberately NOT changed (they record history, not the current pin):
  - `catalog/render.py` comments / assertion messages saying a flag was "confirmed present in the
    PINNED v7.8.1 binary" or that `--api-routes` "does not exist in v7.8.1". Rewriting them to
    v7.15.6 would assert a `--help` verification nobody has done against v7.15.6.
  - `conformance/tests/l0-static/oauth2-proxy-api-routes-per-app/check.sh`, the schema
    description, `runbooks/oauth2-proxy-session.md`, `docs/plans/...` — same reasoning.
  - `platform/oauth2-proxy/auth-visibility-alerts.promtool-test.yaml` — v7.8.1 appears in
    synthetic kube-state-metrics series (test input data), not as a pin.
  - `.gitea/workflows/renovate.yaml` comment — a dated measurement of the pre-Renovate state.
 - Branch was 55 commits behind `origin/main`, but main touches neither `catalog/render.py` nor
  the L1 goldens, and the merge is conflict-free; no merge of main was needed for the gate.
 ## What I verified
 - The golden diff is exactly the image tag in each of the three files (`git diff -U0`).
 - No remaining `oauth2-proxy:v7.8.1` in any rendered/generated/golden file (`git grep`).
 - Local toolbox gate result: see the PR #136 comment posted with this push (rc and per-stage
  results are quoted there and in the worker report).
 ## What I did NOT verify
 - That the v7.15.6 sidecar actually starts and authenticates in the cluster with the rendered
  args. Earlier research said no flag used here was removed/renamed between 7.8.1 and 7.15.6;
  I did not re-run `oauth2-proxy --help` against the v7.15.6 image.
 - L2/L3 conformance (does not run in CI: `CLUSTER_AVAILABLE` unset).
 - `approval-policy`: `conformance/**` is a sensitive path, so this PR is CI-red until the
  operator posts `/approve-override <reason>` as a PR review. The local gate cannot evaluate it.
 - Whether Renovate keeps rebasing this branch now that it carries a non-Renovate commit.
@@ -0,0 +1,155 @@
 ---
 date: 2026-11-08
 branch: feat/runner-capacity-2
 title: act_runner runs two jobs at once, and dind is sized for two
 ---
 ## What changed
 `apps/wave-1/gitea-actions.yaml` only. That file is a **hand-owned** non-catalog unit
 (`EXTRA_UNITS["gitea-actions"]` in `catalog/render.py`): the renderer only decides which wave
 it lands in, it does not generate its bytes, and `ci/render-golden.sh` compares only files the
 catalog renders. So it is edited in place, and no re-render was needed or run.
 1. **New ConfigMap `act-runner-config`** holding a config file with ONE key,
   `runner.capacity: 2`, mounted read-only at `/etc/act_runner/` and selected with
   `CONFIG_FILE=/etc/act_runner/config.yaml`. The image's `run.sh` (act_runner v0.6.1) turns
   that into `--config` for both `register` and `daemon`.
   - Deltas only, on evidence: v0.6.1 `LoadDefault()` unmarshals the file onto a zero Config and
     then applies the same defaults as the no-file path, so the other settings are unchanged.
   - The `generate-config` example was deliberately NOT used. Its `runner.labels` would override
     the registered labels in `/data/.runner` and repoint `ubuntu-latest` at
     `docker.gitea.com/runner-images`. Its `force_pull: true` is not the default. Its sample `envs`
     would be injected into every job.
   - With a file present, act_runner ignores the legacy `GITEA_RUNNER_CAPACITY` /
     `_FILE` / `_ENVIRON` / `_ENV_FILE` / `GITEA_DEBUG` / `GITEA_TRACE` env vars. None of
     them is set on this Deployment.
 2. **Pod-template annotation `homelab.keithjackson.dev/act-runner-config-revision: "1"`.**
   act_runner reads its config once, at start, so a change to the ConfigMap alone would never
   take effect. Bump this annotation whenever you change only the ConfigMap.
 3. **dind resources:** requests `cpu 100m -> 2` and `memory 256Mi -> 1Gi`; limit
   `memory 2Gi -> 4Gi`. There is still no CPU limit, on purpose.
 4. **runner resources:** request `memory 128Mi -> 256Mi`, limit `memory 512Mi -> 1Gi`. CPU stays
   at `100m`.
 Why these sizes. Measured 2026-11-08 through Prometheus, at capacity 1:
 | container | peak CPU (1m rate, 30d) | peak working set (30d) | OOM events (7d) |
 |---|---|---|---|
 | dind | 0.69 core | 447 MiB | 0 |
 | runner | 0.24 core | 314 MiB | 0 |
 At the moment of measurement the pod was idle (`top`: dind 1m/138Mi, runner 1m/83Mi), on
 k3s-prod-2.
 - **CPU request 2.** Two jobs at today's peak is about 1.4 cores. The gate is being parallelised
  in other work, so each job will use more than that. The request sets dind's CFS weight under
  contention; at 100m it got a 1/20 share. With no limit, dind can still use spare cores on an
  idle node.
 - **Memory 1Gi / 4Gi.** Two jobs at today's peak is about 0.9 GiB. If dind is OOM-killed,
  dockerd goes down and takes both running jobs with it, so the limit is about 4x that.
 - **Runner limit 512Mi -> 1Gi.** Its 30-day peak was already 61% of the old limit, and with
  capacity 2 it holds state for two tasks.
 Scheduling headroom: CPU requests per node were measured with `describe node`. The pod's total
 CPU request goes from 200m to 2100m (+1.9 cores).
 | node | CPU requested now | after the change | allocatable |
 |---|---|---|---|
 | prod-2 (current) | 2945m | ~4845m (61%) | 8 |
 | prod-1 | 3175m | ~5075m | 8 |
 | prod-3 | 3840m | ~5740m (72%) | 8 |
 The pod fits on any of the three if the RWO PVC moves it.
 Concurrency semantics. Workflow `concurrency:` groups are enforced by Gitea on the server, not by
 the runner, so capacity does not change them:
 - `gate.yml` `gate-${{ github.ref }}` with cancel-in-progress still cancels a superseded run on
  the same ref.
 - `renovate` (group `renovate`, no cancel) and `images-${{ github.ref }}` still serialise
  themselves.
 What capacity 2 does change: runs on *different* refs (a PR gate next to a renovate or images
 run, or two PRs) now run side by side instead of queueing. The required check
 `gate / gate (pull_request)` is still one job per PR head, and gate.yml's other job
 (`conformance`) is `if: false`, so it never runs.
 Both jobs share one dockerd. act_runner gives each task its own container prefix
 (`GITEA-ACTIONS-TASK-<task id>`) and its own network, and no workflow uses fixed container names
 or host ports.
 ## What I verified
 - **Source of truth.** `catalog/render.py` `EXTRA_UNITS` places `gitea-actions.yaml` by
  anchor/offset only. `ci/render-golden.sh` iterates over rendered files only, so this file is
  outside render==golden. No `--emit-waves` was run.
 - **act_runner v0.6.1 behaviour, read in its source** (tag v0.6.1):
  - `config.LoadDefault()` treats a partial file as described above.
  - `daemon.go`: config labels override `.runner` labels only when non-empty.
  - `poller.go`: a semaphore of `cfg.Runner.Capacity` bounds concurrent tasks.
  - `runner.go`: per-task container name prefix.
  - The image's `run.sh`, read from `gitea/act_runner:0.6.1`, honours `CONFIG_FILE`.
 - **The live pod has no config file today.** `/data` holds only `.runner` and `lost+found`.
  `act_runner --version` reports v0.6.1. No LimitRange or ResourceQuota exists in
  `gitea-actions`.
 - **Resource numbers.** All figures above come from read-only `kubectl top`, `describe node` and
  Prometheus queries against k3s-prod.
 - **dind disk.** The emptyDir sits on the node's 11.8G ephemeral filesystem: 4.0G used, 7.2G
  free. Image cache is 2.18GB, which is the one `catthehacker/ubuntu:act-22.04` image. Two
  concurrent jobs share that image.
 - **Embedded config.** It parses as YAML to `{'runner': {'capacity': 2}}`. The only `$` in the
  manifest is the existing `${BASE_DOMAIN}`, so nothing new is exposed to Flux postBuild
  envsubst.
 - **Approval policy.** `apps/wave-1/` matches no `sensitive_paths` glob in
  `infra/gitea/approval-policy.json`, so this PR merges on a green gate alone.
 - **Local gate.** Pinned toolbox, run from the worktree on the committed tree. The rc and skipped
  stages are given in the PR description.
 ## What I did NOT verify
 - **UNVERIFIED: that the runner actually starts with the config and runs two jobs at once.**
  Nothing is applied until Flux reconciles `main`.
 - **Capacity cannot be seen through Gitea's API.** act_runner v0.6.1 does not report capacity to
  Gitea: `Declare` sends only version and labels. `GET /api/v1/admin/actions/runners` therefore
  cannot show "capacity 2". Concurrency has to be observed instead.
 - **UNVERIFIED: whether the parallelised gate fits in these sizes.** That gate has not landed.
  Re-measure after it does, with
  `max_over_time(sum by (container)(rate(container_cpu_usage_seconds_total{namespace="gitea-actions"}[1m]))[7d:30s])`.
 - **Rollout cancels in-flight jobs.** The rollout restarts the runner pod (strategy Recreate,
  `shutdown_timeout: 0s`), and any job running at that moment is cancelled, possibly including
  the `push`-to-main gate run triggered by this very merge. Re-run it if that happens.
 - **dind disk is not bounded by a limit.** With two jobs it holds two jobs' workspaces and layers
  on the same 11.8G node filesystem. I did not add an `ephemeral-storage` limit, because an
  eviction would kill both jobs. Watch kubelet DiskPressure on the runner's node.
 - **Stale comments in sensitive paths.** `.gitea/workflows/gate.yml` (line ~52) and
  `.gitea/workflows/README.md` (line ~86) still say "`act_runner` runs at capacity 1". I left
  them alone because `.gitea/**` is a sensitive path and editing them would put this PR behind
  human approval. Fix them in a separate PR.
 ### Operator post-merge checklist
 1. Flux applied it:
   `kubectl --context k3s-prod -n flux-system get kustomization apps-wave-1 -o jsonpath='{.status.lastAppliedRevision}'`
   equals the merge commit.
 2. The pod restarted with the config:
   - `kubectl --context k3s-prod -n gitea-actions get pod -l app.kubernetes.io/name=act-runner`
     shows a new pod, 2/2 Running.
   - `kubectl ... exec deploy/act-runner -c runner -- cat /etc/act_runner/config.yaml` shows
     `capacity: 2`.
   - `kubectl ... exec deploy/act-runner -c runner -- ps` shows `act_runner daemon --config /etc/act_runner/config.yaml`.
   - The logs show "declare successfully" with the two expected labels (`ubuntu-latest`,
     `homelab` -> `catthehacker/ubuntu:act-22.04`). That confirms the labels did not change.
 3. The runner is registered and online:
   - `GET /api/v1/admin/actions/runners` lists `incluster-dind`, online, with the same ID as
     before (no duplicate row).
   - Capacity is NOT shown there; see above.
 4. Concurrency is real: trigger two runs on different refs (e.g. two PR gates, or a PR gate plus a
   `workflow_dispatch` of renovate). Then either:
   - in `GET /api/v1/repos/Home-Network/home-network-k3s-impl/actions/tasks`, confirm their
     `run_started_at` .. `updated_at` windows overlap; or
   - check
     `docker --host=tcp://127.0.0.1:2376 --tlsverify ... ps` in the dind container lists two
     `GITEA-ACTIONS-TASK-*` container sets at once.
 5. Resources: `kubectl --context k3s-prod -n gitea-actions describe pod` shows dind requests
   cpu 2 / memory 1Gi, limit memory 4Gi, and runner limit memory 1Gi.
@@ -0,0 +1,22 @@
 # platform/cloudflared
 **Cloudflare Tunnel connector**: inbound HTTPS from the Cloudflare edge to the shared Gateway,
 outbound-only from the cluster (no router port forward, no hairpin, no split-DNS dependency).
 - **Flux tier:** `clusters/base/cloudflared.yaml` — its own Kustomization, `dependsOn: [platform-config]`,
  `wait: false`, and nothing depends on it. A missing or invalid credential can affect only the tunnel.
 - **Runbook (enrolment, canary, alerts, rollback):** `runbooks/cloudflare-tunnel.md`
 - **Design record:** `decisions/2026-11-08-cloudflare-tunnel.md`
 - **Guard:** `conformance/tests/l0-static/cloudflared-ingress-rules`
 | File | What it is |
 |---|---|
 | `config.yaml` | The locally-managed tunnel: literal tunnel UUID + ingress rules. LAN/VPN-only hosts → 404, apex + wildcard → `https://cilium-gateway-shared.gateway.svc.cluster.local:443` with `matchSNItoHost`, catch-all 404. Never the Kubernetes API. |
 | `cloudflared-credentials.sops.yaml` | SOPS-encrypted tunnel credentials JSON. Ships with `REPLACE-ME`; the operator fills it. |
 | `deployment.yaml` | 2 replicas, hard spread across nodes, non-root, read-only root fs, pinned digest; PDB `minAvailable: 1`. |
 | `networkpolicy.yaml` | default-deny (single owner — not an app namespace) + Prometheus scrape allow + egress CNP: world 7844 tcp/udp and 443, the Gateway (`ingress` entity), and Gateway backends by `gateway-access=shared` minus the LAN-only namespaces. |
 | `servicemonitor.yaml`, `alerts.yaml` | Headless metrics Service + ServiceMonitor scraping :2000, and the two alerts, both written to fire (not go quiet) when their series vanish. |
 **Phase 1 only.** This component changes no live traffic: Cloudflare sends a hostname through the
 tunnel only when its DNS record is a CNAME to `<UUID>.cfargotunnel.com`, and nothing here creates
 one. Moving DNS (external-dns, Gateway annotations, Pi-hole/CoreDNS split horizon) is a later change.
@@ -0,0 +1,128 @@
 ---
 # promtool unit tests for platform/cloudflared/alerts.yaml.
 #
 # Found by platform/monitoring/tests/run.sh through the COMPANION convention
 # (`<manifest>.promtool-test.yaml` beside `<manifest>.yaml`). Run just this file with
 #   platform/monitoring/tests/run.sh platform/cloudflared/alerts.promtool-test.yaml
 # The harness flattens the PrometheusRule's `.spec.groups` into alerts.rules.yaml. Assertions query
 # the synthetic ALERTS series (not exp_alerts), so annotation wording is not pinned here.
 #
 # Each rule is pinned BOTH ways, plus the two self-silencing traps:
 #   * series ABSENT (nothing scraped) must FIRE, not read as healthy;
 #   * working-then-broken must RE-FIRE, not latch cleared.
 rule_files:
  - alerts.rules.yaml
 evaluation_interval: 1m
 tests:
  - interval: 1m
    name: healthy - two replicas, four edge connections each, both rules quiet
    input_series:
      - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}'
        values: '4x60'
      - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-b"}'
        values: '4x60'
      - series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}'
        values: '2x60'
    promql_expr_test:
      - expr: 'count(ALERTS{alertname=~"Cloudflared.*"})'
        eval_time: 50m
        exp_samples: []
  - interval: 1m
    name: nothing scraped at all - TunnelDown still fires (no self-silencing on absence)
    input_series:
      # kube-state-metrics says both pods are Available, but no cloudflared series exists — the
      # broken-ServiceMonitor / dropped-scrape case. The rule must not read this as healthy.
      - series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}'
        values: '2x60'
    promql_expr_test:
      - expr: 'ALERTS{alertname="CloudflaredTunnelDown"}'
        eval_time: 5m
        exp_samples:
          - labels: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="pending", severity="warning"}'
            value: 1
      - expr: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing"}'
        eval_time: 12m
        exp_samples:
          - labels: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing", severity="warning"}'
            value: 1
      - expr: 'ALERTS{alertname="CloudflaredReplicasDegraded"}'
        eval_time: 30m
        exp_samples: []
  - interval: 1m
    name: scraped but zero edge connections - TunnelDown fires after 10m
    input_series:
      - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}'
        values: '0x60'
      - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-b"}'
        values: '0x60'
      - series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}'
        values: '0x60'
    promql_expr_test:
      - expr: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing"}'
        eval_time: 9m
        exp_samples: []
      - expr: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing"}'
        eval_time: 12m
        exp_samples:
          - labels: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing", severity="warning"}'
            value: 1
      - expr: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing"}'
        eval_time: 17m
        exp_samples:
          - labels: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing", severity="warning"}'
            value: 1
  - interval: 1m
    name: working then the series vanish - TunnelDown re-fires instead of latching cleared
    input_series:
      # 20 healthy samples, then the series stop (stale) — the scrape broke or the pods died.
      - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}'
        values: '4x20 stale'
      - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-b"}'
        values: '4x20 stale'
      - series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}'
        values: '2x60'
    promql_expr_test:
      - expr: 'ALERTS{alertname="CloudflaredTunnelDown"}'
        eval_time: 15m
        exp_samples: []
      - expr: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing"}'
        eval_time: 35m
        exp_samples:
          - labels: 'ALERTS{alertname="CloudflaredTunnelDown", alertstate="firing", severity="warning"}'
            value: 1
  - interval: 1m
    name: one replica lost - ReplicasDegraded fires after 15m, tunnel itself still up
    input_series:
      - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}'
        values: '4x60'
      - series: 'kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"}'
        values: '1x60'
    promql_expr_test:
      - expr: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing"}'
        eval_time: 14m
        exp_samples: []
      - expr: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing"}'
        eval_time: 17m
        exp_samples:
          - labels: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing", severity="warning"}'
            value: 1
      - expr: 'ALERTS{alertname="CloudflaredTunnelDown"}'
        eval_time: 30m
        exp_samples: []
  - interval: 1m
    name: kube-state-metrics series absent - ReplicasDegraded fires rather than going quiet
    input_series:
      - series: 'cloudflared_tunnel_ha_connections{namespace="cloudflared",pod="cloudflared-a"}'
        values: '4x60'
    promql_expr_test:
      - expr: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing"}'
        eval_time: 17m
        exp_samples:
          - labels: 'ALERTS{alertname="CloudflaredReplicasDegraded", alertstate="firing", severity="warning"}'
            value: 1
@@ -0,0 +1,64 @@
 ---
 # Alert rules for the Cloudflare Tunnel connector.
 #
 # THE DESIGN CONSTRAINT: NEITHER RULE MAY GO QUIET WHEN ITS OWN DATA DISAPPEARS. The repo has
 # shipped self-silencing rules before (a bare `up{}` that clears exactly when scraping breaks), so:
 #
 #   * CloudflaredTunnelDown reads `cloudflared_tunnel_ha_connections` — the per-replica gauge of
 #     live edge connections — and falls back to `vector(0)` when NO such series exists. A dead
 #     tunnel, zero pods, a broken ServiceMonitor, or a NetworkPolicy that drops the scrape all make the
 #     series absent, and all of them FIRE rather than read as healthy.
 #   * CloudflaredReplicasDegraded reads kube-state-metrics, which does not depend on cloudflared
 #     being scrapable at all, and has the same `vector(0)` fallback.
 #
 # Both directions are pinned by the companion alerts.promtool-test.yaml (healthy is quiet; absent
 # series fires; working-then-broken re-fires).
 #
 # NOTE FOR PHASE 1: until the operator completes runbooks/cloudflare-tunnel.md steps 1-5 the
 # Deployment cannot connect and CloudflaredTunnelDown fires. That is the honest state — the tunnel
 # is not serving — and nothing routes through it yet, which is why both rules are `warning`.
 # Raise TunnelDown to `critical` in the change that points public DNS at the tunnel.
 apiVersion: monitoring.coreos.com/v1
 kind: PrometheusRule
 metadata:
  name: cloudflared
  namespace: cloudflared
  labels:
    app.kubernetes.io/name: cloudflared
 spec:
  groups:
    - name: platform-health.cloudflare-tunnel
      rules:
        - alert: CloudflaredTunnelDown
          # Sum of live edge connections across every replica. `or on() vector(0)` turns an EMPTY
          # result (nothing scraped) into 0 so it fires instead of evaluating to no series.
          expr: (sum(cloudflared_tunnel_ha_connections{namespace="cloudflared"}) or on() vector(0)) < 1
          # 10m rides out a rolling update and a Cloudflare edge reconnect, not a real outage.
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: Cloudflare Tunnel has no healthy edge connections
            description: >-
              No cloudflared replica reports a live connection to the Cloudflare edge (or none is
              being scraped at all). Every hostname routed through the tunnel is unreachable from
              the internet. Before the operator has enrolled the tunnel this is expected.
            runbook_url: "https://${BASE_DOMAIN}/runbooks/cloudflare-tunnel"
        - alert: CloudflaredReplicasDegraded
          # Fewer than the 2 replicas deployment.yaml asks for are Available — the tunnel has lost
          # its HA, and the next node drain or reboot takes it down. Keep `2` in step with
          # spec.replicas. Absent series (kube-state-metrics gone, Deployment deleted) -> 0 -> fires.
          expr: >-
            (max(kube_deployment_status_replicas_available{namespace="cloudflared",deployment="cloudflared"})
            or on() vector(0)) < 2
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: cloudflared has fewer than 2 available replicas
            description: >-
              The cloudflared Deployment has fewer than two Available replicas, so the tunnel has
              no redundancy (or is down entirely). Check for unenrolled credentials, a node that
              cannot schedule the hard-spread pod, or an image pull failure.
            runbook_url: "https://${BASE_DOMAIN}/runbooks/cloudflare-tunnel"
@@ -0,0 +1,32 @@
 # Cloudflare Tunnel credentials for platform/cloudflared — SOPS-ENCRYPTED (R-SEC-01, spec docs/07).
 # Keys stay cleartext for reviewable diffs; only `stringData` values are encrypted (.sops.yaml
 # `encrypted_regex`). Plaintext template + operator hand-off: cloudflared-credentials.sops.yaml.example.
 # Shipped with REPLACE-ME placeholders; runbooks/cloudflare-tunnel.md step 4 fills it.
 apiVersion: v1
 kind: Secret
 metadata:
    name: cloudflared-credentials
    namespace: cloudflared
 type: Opaque
 stringData:
    credentials.json: ENC[AES256_GCM,data:iTCsCQ9g4zouELNa1KzFy8ZO5eLctBdKFaWDXlejpxjWs06sGaduYRPV9kmm1JT8A8s10OevqEEXM+9unOSrmOiUvsxMYAvtwvfyUs3dVbtOB/uCBzHsGK+4cJoC1iII6WzITVe5tlWMXYtyBnDo1Mm3BfO+H/6ytLf750BoABqohO2cfWRfN9MAXL6BFe7urZdKrMou23BooDUxG/pwTpnALbUCer/PhCmrB6cgOw==,iv:ltn/fN59x320pFgvb3/g0UgzBmQcdnnk8JvKpoFn0H0=,tag:fr8s1aDsZBAQYLGXgwG8WA==,type:str]
 sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUanIxelFEUDZtWU5CMHd5
            VHl5MkJnM1JDUWhkM2daY0dtRUlFUmNUMDJrClM5UFBsdjh2Q0NIWmxpM1djYXUy
            OFlZYVdBbURLanRRNzV1RVNmbkxWR28KLS0tIFE3d1Z1MjFZMTlaTFFIWm5ndVpN
            emhHaFpEZWpMN0JXL2NVaW5PU1ZwVW8KkCDX7ED039X12BCmiSqHr0fmZKSZmfjP
            DIiR6DvWWw1TcB3CjD5q29nIthCDf9zlrpuMfQ+9ODAlRr5t6zP+cg==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2026-11-08T21:28:19Z"
    mac: ENC[AES256_GCM,data:4wD/6YSBP/e6MTevGSOFTi/EamJRZoryf+aET/opTwjf7yMgi5iObXTnRapKwAFzeXkSkFVfnfRd8Ir714Uez3z2yLxQ6Ygte9VQxHUfkKfXfu+OmuCnjFGaTjsxm6551VPFGHZuto6wZjebwozWuxzHhZi/AjCo2QGyRjmymBM=,iv:hClSAe8r6fhnQzwQY3NZSYhqElzwSRPhI8Y2fH1+r64=,tag:LviWf5oMB6NUjTFNXy62zg==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.9.4
@@ -0,0 +1,35 @@
 # =============================================================================
 # TEMPLATE — NOT APPLIED, NOT ENCRYPTED (.yaml.example is ignored by kustomize and never
 # auto-encrypted by SOPS). Reference scaffold + OPERATOR HAND-OFF.
 #
 # WHAT THIS IS: the credentials JSON of the locally-managed Cloudflare Tunnel that
 # platform/cloudflared runs. `cloudflared tunnel create` writes it; it holds the account tag, the
 # tunnel UUID and the tunnel SECRET. Anyone holding it can run a connector for this tunnel and
 # receive its traffic, so it is a real credential: SOPS-encrypted in git, never printed.
 #
 # *** THE REAL FILE IS COMMITTED WITH REPLACE-ME PLACEHOLDERS, ALREADY ENCRYPTED. ***
 # So the wiring is complete before the tunnel exists: the resource is listed in kustomization.yaml
 # and both overlays already carry the `cloudflared` decryption patch. Until you fill it, the
 # Deployment CrashLoops inside its own NON-GATING Flux Kustomization (clusters/base/cloudflared.yaml,
 # wait:false) and CloudflaredTunnelDown fires. Nothing else in the cluster is affected.
 #
 # HAND-OFF (operator — an agent must not mint a long-lived credential, CLAUDE.md §6).
 # The full procedure, with verification and rollback, is runbooks/cloudflare-tunnel.md. In short:
 #   1. `cloudflared tunnel login` + `cloudflared tunnel create --credentials-file ... home-k3s`
 #      (pinned cloudflared image under podman; the toolbox does not ship cloudflared).
 #   2. Write the JSON STRAIGHT INTO the encrypted file with `sops set`, inside the toolbox — the
 #      plaintext never reaches a terminal or the repo tree.
 #   3. Put the tunnel UUID (not secret) into platform/cloudflared/config.yaml `tunnel:`.
 #   4. Verify without printing (`sops -d --extract ... | jq -e ...`), then PR it.
 # =============================================================================
 apiVersion: v1
 kind: Secret
 metadata:
  name: cloudflared-credentials
  namespace: cloudflared
 type: Opaque
 stringData:
  # One line of JSON, exactly as `cloudflared tunnel create` wrote it (compacted with `jq -c`):
  #   {"AccountTag":"<account>","TunnelSecret":"<base64>","TunnelID":"<uuid>"}
  # TunnelID MUST equal the `tunnel:` UUID in config.yaml.
  credentials.json: REPLACE-ME
@@ -0,0 +1,84 @@
 ---
 # cloudflared configuration for the LOCALLY-MANAGED tunnel (the ingress rules live here, in git,
 # not in the Cloudflare dashboard). Projected into the pods by the configMapGenerator in
 # kustomization.yaml as the content-hashed ConfigMap `cloudflared-config`, so editing this file
 # changes the pod template and ROLLS the Deployment (the vpn-ingress peers.conf lesson: a
 # fixed-name ConfigMap updates in place and the running process never re-reads it).
 #
 # `${BASE_DOMAIN}` is substituted by Flux postBuild (clusters/base/cloudflared.yaml). It is the ONLY
 # token in this file; do not add a bare dollar sign anywhere here.
 #
 # INVARIANTS, enforced by conformance/tests/l0-static/cloudflared-ingress-rules:
 #   * every rule's `service` is either `http_status:404` or EXACTLY the shared Gateway's in-cluster
 #     Service — nothing else, and in particular NEVER the Kubernetes API (operator decision
 #     2026-11-08: the API stays local-only and must never be reachable through the tunnel);
 #   * the LAN/VPN-only hostnames below map to `http_status:404` and come BEFORE the wildcard;
 #   * `tunnel:` is a LITERAL UUID here, not a cluster-var: the Phase 2 DNS-flip change matches
 #     records against it, and keeping it out of clusters/** keeps that change off a sensitive path;
 #   * the last rule is a hostname-less `http_status:404` catch-all.
 #
 # OPERATOR STEP: replace REPLACE-ME-TUNNEL-UUID with the UUID printed by `cloudflared tunnel create`
 # (runbooks/cloudflare-tunnel.md step 3). Until then cloudflared treats the value as a tunnel NAME,
 # finds no origin certificate to resolve it with, and exits — a CrashLoop in a non-gating tier.
 tunnel: f26f1052-1bc0-4d9d-800b-4da0377472a3
 credentials-file: /etc/cloudflared/creds/credentials.json
 # Serves /metrics (Prometheus), /ready (200 only with >=1 live edge connection) and /healthcheck.
 metrics: 0.0.0.0:2000
 no-autoupdate: true
 ingress:
  # ── NEVER THROUGH THE TUNNEL (operator decisions 2026-11-08) ─────────────────────────────────
  # Plex stays off the tunnel; it keeps its own remote-access path (runbooks/plex-remote-access.md).
  - hostname: plex.${BASE_DOMAIN}
    service: http_status:404
  - hostname: plex-vip.${BASE_DOMAIN}
    service: http_status:404
  # LAN/VPN-only services. The six catalog `exposure: internal` apps (each carries the external-dns
  # `controller: none` route annotation live, so none has a public record today) ...
  - hostname: jackett.${BASE_DOMAIN}
    service: http_status:404
  - hostname: pihole.${BASE_DOMAIN}
    service: http_status:404
  - hostname: pihole-dns.${BASE_DOMAIN}
    service: http_status:404
  - hostname: semaphore.${BASE_DOMAIN}
    service: http_status:404
  - hostname: qbittorrent.${BASE_DOMAIN}
    service: http_status:404
  - hostname: radarr.${BASE_DOMAIN}
    service: http_status:404
  - hostname: sonarr.${BASE_DOMAIN}
    service: http_status:404
  # ... and Headlamp, the cluster dashboard (operator decision 2026-11-08: LAN/VPN-only). It holds a
  # route to the Kubernetes API on the user's behalf, which is one more reason it stays off the
  # tunnel alongside decision 4.
  - hostname: headlamp.${BASE_DOMAIN}
    service: http_status:404
  # Non-HTTP L4 VIPs (their own LoadBalancer addresses, not the Gateway). Listed so a DNS flip
  # that CNAMEs them at the tunnel by mistake gets a 404, not the Gateway's catch-all.
  - hostname: unifi.${BASE_DOMAIN}
    service: http_status:404
  - hostname: crafty-mc.${BASE_DOMAIN}
    service: http_status:404
  # ── EVERYTHING ELSE ON THE ZONE -> THE SHARED GATEWAY, OVER TLS ──────────────────────────────
  # cilium-gateway-shared is the LoadBalancer Service Cilium creates for Gateway gateway/shared
  # (platform/gateway/gateway.yaml); its ClusterIP frontend is redirected to the Gateway's Envoy
  # exactly like the VIP. Addressed by cluster DNS, not by a ${BASE_DOMAIN} name, so the tunnel
  # does not depend on split-horizon DNS at all.
  #
  # matchSNItoHost: cloudflared sends the request's Host as the TLS SNI, so the Gateway selects the
  # right listener and presents the existing Let's Encrypt `wildcard-home-tls` certificate, which
  # cloudflared validates against the public roots for that same name. No TLS verification is
  # disabled anywhere.
  - hostname: ${BASE_DOMAIN}
    service: https://cilium-gateway-shared.gateway.svc.cluster.local:443
    originRequest:
      matchSNItoHost: true
  - hostname: "*.${BASE_DOMAIN}"
    service: https://cilium-gateway-shared.gateway.svc.cluster.local:443
    originRequest:
      matchSNItoHost: true
  # ── CATCH-ALL (cloudflared requires the last rule to have no hostname) ───────────────────────
  - service: http_status:404
@@ -0,0 +1,150 @@
 ---
 # cloudflared — the Cloudflare Tunnel connector. Outbound-only: it dials the Cloudflare edge on
 # 7844 (QUIC, falling back to HTTP/2 over TCP) and carries requests for the tunnel's hostnames
 # back to the shared Gateway. Nothing listens on the LAN and no router port is opened.
 #
 # HA: 2 replicas, hard-spread across nodes. Each replica registers its own set of edge
 # connections for the SAME tunnel, so Cloudflare load-balances between them and losing one node
 # (a kured reboot, a drain) loses no traffic. The PDB in this directory keeps a drain from taking
 # both at once.
 apiVersion: apps/v1
 kind: Deployment
 metadata:
  name: cloudflared
  namespace: cloudflared
  labels:
    app.kubernetes.io/name: cloudflared
 spec:
  replicas: 2
  revisionHistoryLimit: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      # Never drop below the running count during a rollout; with 3 nodes and a hard spread the
      # surge pod always has a node to land on.
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: cloudflared
  template:
    metadata:
      labels:
        app.kubernetes.io/name: cloudflared
        # internet, NEVER vpn. The vpn class steers egress through the PIA tunnel (ADR-0015), which
        # would put the edge connections behind a third party's exit and its kill-switch — the
        # tunnel would drop whenever PIA does. This namespace is outside the app-namespace scope
        # of require-egress-class-label, so the label is declared for the steering, not to pass
        # admission; the L0 check cloudflared-ingress-rules pins it.
        homelab.keithjackson.dev/egress-class: internet
    spec:
      # cloudflared never talks to the Kubernetes API. No token mounted, and the egress policy
      # does not admit the kube-apiserver entity either.
      automountServiceAccountToken: false
      enableServiceLinks: false
      securityContext:
        runAsNonRoot: true
        # The image is distroless `nonroot` (65532).
        runAsUser: 65532
        runAsGroup: 65532
        # fsGroup makes the kubelet GROUP-own the Secret volume's files to 65532. Without it they
        # are root:root, and defaultMode 0440 (below) is unreadable to uid 65532 — every replica
        # CrashLooped on "couldn't read tunnel credentials ... permission denied" (2026-11-08),
        # which cloudflared then misreports as "Error locating origin cert". Same lesson as
        # platform/keycloak/client-secret-reconciler.yaml. fsGroup + 0440 keeps the credential
        # unreadable to any other uid rather than relaxing it to world-readable 0444.
        # Pinned by the L0 check cloudflared-ingress-rules, assertion (7).
        fsGroup: 65532
        seccompProfile:
          type: RuntimeDefault
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: cloudflared
      # cloudflared drains in-flight requests for up to its own 30s grace-period on SIGTERM.
      terminationGracePeriodSeconds: 45
      containers:
        - name: cloudflared
          # Tag + digest: Renovate's `kubernetes` manager (renovate.json5, platform/**) bumps both.
          image: docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba
          # The image ENTRYPOINT is `cloudflared --no-autoupdate`; these replace its CMD. The tunnel
          # UUID, credentials path, metrics address and ingress rules all come from the file.
          args:
            - tunnel
            - --config
            - /etc/cloudflared/config/config.yaml
            - run
          ports:
            - name: metrics
              containerPort: 2000
              protocol: TCP
          # /ready is 200 only while this replica holds at least one live edge connection, so an
          # unenrolled or edge-isolated replica is NotReady and the Deployment says so.
          readinessProbe:
            httpGet:
              path: /ready
              port: metrics
            initialDelaySeconds: 5
            periodSeconds: 10
            failureThreshold: 3
          # Liveness is deliberately NOT /ready: an edge outage would otherwise restart-loop every
          # replica, which reconnects nothing faster and churns the pods. /healthcheck answers
          # whenever the process is serving.
          livenessProbe:
            httpGet:
              path: /healthcheck
              port: metrics
            initialDelaySeconds: 10
            periodSeconds: 20
            failureThreshold: 3
          resources:
            requests:
              cpu: 10m
              memory: 64Mi
            limits:
              cpu: 500m
              memory: 256Mi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop:
                - ALL
          volumeMounts:
            - name: config
              mountPath: /etc/cloudflared/config
              readOnly: true
            - name: creds
              mountPath: /etc/cloudflared/creds
              readOnly: true
      volumes:
        - name: config
          configMap:
            # Rewritten to the content-hashed name by kustomize (configMapGenerator).
            name: cloudflared-config
        - name: creds
          secret:
            # cloudflared-credentials.sops.yaml — the tunnel credentials JSON, by name only.
            secretName: cloudflared-credentials
            # 0440, owner root, group fsGroup 65532 (pod securityContext) — readable by the
            # container user through its group, by nobody else.
            defaultMode: 0440
 ---
 # One replica may be disrupted at a time — enough for kured's one-node-at-a-time drain to
 # proceed, never enough to take the tunnel down. minAvailable 1 (not 2) on purpose: a 0-disruption
 # PDB wedges a drain (platform/kured/alerts.yaml, KuredRebootPending).
 apiVersion: policy/v1
 kind: PodDisruptionBudget
 metadata:
  name: cloudflared
  namespace: cloudflared
  labels:
    app.kubernetes.io/name: cloudflared
 spec:
  minAvailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: cloudflared
@@ -0,0 +1,35 @@
 ---
 # Cloudflare Tunnel connector. Reconciled by its OWN non-gating Flux Kustomization
 # (clusters/base/cloudflared.yaml: dependsOn platform-config, wait:false) — NOT a line in
 # platform/kustomization.yaml. See that file for why; in one line: this Deployment cannot become
 # Ready until the operator enrols the tunnel, and in a gating tier that would stall every app wave.
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 # REQUIRED for the configMapGenerator: a generated ConfigMap inherits no namespace, and kustomize
 # will not rewrite the Deployment's reference to its hashed name across a namespace boundary
 # (measured on platform/vpn-ingress).
 namespace: cloudflared
 resources:
  - namespace.yaml
  - deployment.yaml      # Deployment (2 replicas, hard node spread) + PodDisruptionBudget
  # Namespace default-deny + the Prometheus scrape allow + the egress CiliumNetworkPolicy. Keep
  # them in ONE file: a baseline without its scrape allow silently empties the TSDB.
  - networkpolicy.yaml
  - servicemonitor.yaml  # headless metrics Service + ServiceMonitor (:2000)
  - alerts.yaml
  # The tunnel credentials JSON — the ONE secret here. SOPS-encrypted, decrypted per-env by the
  # `cloudflared` Flux Kustomization (both overlays patch it), referenced by name only. Its
  # plaintext template sibling is deliberately NOT named in this file (no-plaintext-secrets).
  # Ships with REPLACE-ME placeholders; runbooks/cloudflare-tunnel.md fills it.
  - cloudflared-credentials.sops.yaml
 # config.yaml becomes the CONTENT-HASHED ConfigMap `cloudflared-config`, so an edit to the ingress
 # rules or the tunnel UUID changes the pod template and rolls the Deployment. Do NOT set
 # disableNameSuffixHash: the hash is the rollout mechanism.
 configMapGenerator:
  - name: cloudflared-config
    files:
      - config.yaml
    options:
      labels:
        app.kubernetes.io/name: cloudflared
@@ -0,0 +1,18 @@
 ---
 # Cloudflare Tunnel connector (north-south INGRESS from the Cloudflare edge). See README.md.
 #
 # NOT homelab.keithjackson.dev/app-namespace=true. This is platform network infrastructure (like
 # vpn-ingress / vpn-egress), not an application, so Kyverno's app-scoped policies — including the
 # `require-default-deny-baseline` GENERATE rule — do not select this namespace. That is why
 # networkpolicy.yaml ships its own `default-deny` here, and why doing so creates NO second owner
 # of that object: Kyverno never generates one in a namespace without the app-namespace label
 # (the drift loop recorded in the 2026-09-30 handoff only exists where both write the same name).
 #
 # NO `gateway-access` label. Nothing here attaches an HTTPRoute to the shared Gateway; cloudflared
 # is a CLIENT of the Gateway, not a backend behind it.
 apiVersion: v1
 kind: Namespace
 metadata:
  name: cloudflared
  labels:
    app.kubernetes.io/name: cloudflared
@@ -0,0 +1,149 @@
 ---
 # Namespace baseline: deny everything both ways, except DNS to kube-system.
 #
 # SINGLE OWNER. This namespace is not labelled app-namespace=true, so Kyverno's
 # require-default-deny-baseline generate rule does not select it and never writes this object;
 # Flux is its only writer (the same posture as kured / crafty / headlamp / homepage). Do NOT add
 # the app-namespace label to namespace.yaml without deleting this object in the same change, or
 # the two controllers fight over it every reconcile (the 2026-09-30 drift-loop root cause).
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
  name: default-deny
  namespace: cloudflared
  labels:
    homelab.keithjackson.dev/baseline: default-deny
 spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress: []
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
 ---
 # The ONLY ingress: Prometheus scraping :2000. Shipped in the same file as the baseline on purpose
 # (the Longhorn/kured lesson): a deny-all without this allow leaves the ServiceMonitor's targets
 # DISCOVERED and every scrape timing out, which is the state the CloudflaredTunnelDown rule is
 # written to fire on rather than go quiet in.
 apiVersion: networking.k8s.io/v1
 kind: NetworkPolicy
 metadata:
  name: cloudflared-allow-prometheus-scrape
  namespace: cloudflared
  labels:
    app.kubernetes.io/name: cloudflared
 spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: cloudflared
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - protocol: TCP
          port: 2000
 ---
 # Egress: the Cloudflare edge, the shared Gateway, and the Gateway's backends — nothing else.
 #
 #   1. world tcp+udp/7844 — the tunnel itself (QUIC on udp, HTTP/2 fallback on tcp).
 #      world tcp/443   — Cloudflare's API / update endpoints that cloudflared calls at start.
 #      Expressed as `world` + port rather than Cloudflare CIDRs: the edge ranges are Cloudflare's to
 #      change, and a stale CIDR list would fail as a silent outage. Narrowing to toCIDR is a
 #      possible later hardening; it is not done blind.
 #
 #   2. `ingress` entity tcp/443 — the shared Gateway's Envoy. Traffic to the
 #      cilium-gateway-shared Service is handed to Envoy, whose identity is reserved:ingress.
 #
 #   3. The Gateway's BACKENDS, by namespace. NOT optional: Cilium >= 1.16.8 (CVE-2025-30162; this
 #      cluster runs 1.20.x) makes the Gateway's Envoy re-evaluate THIS pod's egress policy against
 #      the identity of the backend it routes the request to. With only rule 2, Envoy accepts the TCP
 #      connection and then answers every request `403 Access denied` — measured for buildkit ->
 #      gitea on 2026-10-30, see platform/buildkit/networkpolicy.yaml. So the backends are named:
 #        * any namespace opted in to the shared Gateway (gateway-access=shared) — which is the set
 #          of namespaces that can serve a route at all, so this tracks new apps automatically;
 #        * MINUS the LAN/VPN-only services (operator decisions 2026-11-08): plex, jackett, pihole,
 #          semaphore, qbittorrent, radarr, sonarr, headlamp. config.yaml already 404s their
 #          hostnames; this is the second, independent layer: even a request that got past the
 #          ingress rules would be refused by Envoy for these backends.
 #      unifi and crafty-mc are not here because they are L4 LoadBalancer VIPs, not Gateway
 #      backends; the crafty WEB UI (crafty.<domain>, an HTTPRoute) is deliberately reachable.
 #
 #      WHY NOT the `- {}` allow-all egress gitea-actions uses (apps/wave-1/gitea-actions.yaml)?
 #      It would satisfy the L7 re-check just as well, but `{}` admits EVERY identity, including
 #      kube-apiserver and host — which operator decision 4 (the Kubernetes API is never reachable
 #      through the tunnel) rules out for the one pod that carries internet traffic in. The
 #      namespace-label selector is the narrowest rule that still names every possible backend.
 #      Verified 2026-11-08: Cilium identities carry `k8s:io.cilium.k8s.namespace.labels.
 #      gateway-access`, and every namespace with an HTTPRoute on the shared Gateway is labelled
 #      gateway-access=shared.
 #
 #      BACKENDS NEED NO INGRESS CHANGE. Envoy forwards to a backend as reserved:ingress, which every
 #      backend already admits; the source-identity check happens on THIS pod's egress, not on the
 #      destination's ingress. Measured 2026-10-30 (decisions/2026-10-30-gitea-admit-buildkit.md):
 #      gitea-actions is in none of Gitea's ingress allows and its requests through the Gateway reach
 #      Gitea, while Gitea's endpoint shows only a `reserved:ingress` allow row. So, unlike a
 #      ClusterIP client, cloudflared is not added to any app's inClusterClients. That also means
 #      this rule does not open a direct cloudflared -> app pod path into any app namespace whose
 #      ingress admits only reserved:ingress and named peers.
 #
 # DELIBERATELY ABSENT: kube-apiserver, host, remote-node, cluster, all. The Kubernetes API stays
 # local-only (operator decision 2026-11-08); conformance/tests/l0-static/cloudflared-ingress-rules
 # fails the gate if any of those entities appears here.
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
  name: cloudflared-allow-egress
  namespace: cloudflared
  labels:
    app.kubernetes.io/name: cloudflared
 spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: cloudflared
  egress:
    - toEntities:
        - world
      toPorts:
        - ports:
            - port: "7844"
              protocol: TCP
            - port: "7844"
              protocol: UDP
            - port: "443"
              protocol: TCP
    - toEntities:
        - ingress
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
    - toEndpoints:
        - matchExpressions:
            - key: io.cilium.k8s.namespace.labels.gateway-access
              operator: In
              values:
                - shared
            - key: io.kubernetes.pod.namespace
              operator: NotIn
              values:
                - plex
                - jackett
                - pihole
                - semaphore
                - qbittorrent
                - radarr
                - sonarr
                - headlamp
@@ -0,0 +1,45 @@
 ---
 # Scrape of cloudflared's own metrics server (:2000, `metrics:` in config.yaml) on every replica.
 #
 # A Service + ServiceMonitor in ONE file rather than a PodMonitor, on purpose: the repo's L0 checks
 # (servicemonitor-ports-resolve, alert-metrics-have-scrape-targets assertion 4) can then prove the
 # monitor's port is a port this Service actually defines. A PodMonitor would have to be declared as
 # an unauditable exception. The Service is headless and carries only the metrics port — it is not a
 # way into the tunnel, and nothing routes to it but Prometheus (networkpolicy.yaml admits only the
 # monitoring namespace on :2000).
 #
 # kube-prometheus-stack runs with serviceMonitorSelectorNilUsesHelmValues: false, so this is
 # selected from any namespace; the `release` label is kept for consistency.
 apiVersion: v1
 kind: Service
 metadata:
  name: cloudflared-metrics
  namespace: cloudflared
  labels:
    app.kubernetes.io/name: cloudflared
 spec:
  clusterIP: None
  selector:
    app.kubernetes.io/name: cloudflared
  ports:
    - name: metrics
      port: 2000
      targetPort: metrics
      protocol: TCP
 ---
 apiVersion: monitoring.coreos.com/v1
 kind: ServiceMonitor
 metadata:
  name: cloudflared
  namespace: cloudflared
  labels:
    app.kubernetes.io/name: cloudflared
    release: kube-prometheus-stack
 spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: cloudflared
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s
@@ -16,7 +16,7 @@ Cilium L2 LoadBalancer (game protocols authenticate players themselves).
 |---|---|
 | Image | `registry.gitlab.com/crafty-controller/crafty-4:4.10.8` (current stable; bare-semver tags, no `v`) |
 | Web UI | HTTPS **self-signed** on `:8443` (reached only via the sidecar / same-pod) |
-| Sidecar | `quay.io/oauth2-proxy/oauth2-proxy:v7.8.1`, reverse-proxy on `:4180` |
+| Sidecar | `quay.io/oauth2-proxy/oauth2-proxy:v7.15.6`, reverse-proxy on `:4180` |
 | Data | ONE Longhorn RWO PVC `crafty-data` (20Gi), mounted 5× by subPath |
 | PVC subPaths | `servers`→`/crafty/servers`, `backups`→`/crafty/backups`, `logs`→`/crafty/logs`, `import`→`/crafty/import`, `config`→`/crafty/app/config` |
 | Identity | `runAsUser:1000 runAsGroup:0 fsGroup:0 runAsNonRoot:true` (see below) |
@@ -130,7 +130,7 @@ spec:
        - name: oauth2-proxy
          # quay.io/oauth2-proxy/oauth2-proxy — same authenticator family as platform/oauth2-proxy, but
          # here in per-app REVERSE-PROXY mode (--upstream is the co-located Crafty UI, not static://202).
-          image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
+          image: quay.io/oauth2-proxy/oauth2-proxy:v7.15.6
          imagePullPolicy: IfNotPresent
          args:
            - --http-address=0.0.0.0:4180
@@ -40,7 +40,11 @@ spec:
      # values (-session-ttl + the -oidc-* args from the externalSecret; useCookie stays default-false
      # so `-oidc-use-cookie` is NOT emitted) is supported by the v0.41.0 backend. Renovate maintains
      # within 0.41.x; the appVersion continues to pin the Headlamp image.
-      version: "<0.47.0"
+      # 2026-11-08: NOW 0.46.x. PR #130 (Renovate, default rangeStrategy `replace`) rewrote
      # `>=0.41.0 <0.42.0` to the unbounded-below `<0.47.0`; the lower bound is restored here at the
      # chart that was live when it merged (0.46.0). renovate.json5 now bumps `>=A <B` ranges as a window,
      # so the next widen PR proposes `>=0.47.x <0.48.0` rather than `<0.48.0`.
      version: ">=0.46.0 <0.47.0"
      sourceRef:
        kind: HelmRepository
        name: headlamp
@@ -367,9 +367,17 @@ is involved — so step 2 failing with no log entry is the expected signature of
 The `Keycloak` CR has no `spec.image`, so the server image is whatever the operator's
 `RELATED_IMAGE_KEYCLOAK` names — the same tag as the operator (prod, 2026-10-30: operator
 `26.0.6`, `keycloak-0/1` on `quay.io/keycloak/keycloak:26.0.6`). Operator, CRDs and server are
-released in lockstep from `keycloak/keycloak-k8s-resources`, and the three URLs in
+released in lockstep from `keycloak/keycloak-k8s-resources`, and the release URLs in
 `kustomization.yaml` are the one knob. Renovate raises them as ONE PR (`keycloak-operator` group,
 labels `needs-review` + `blast-radius`), **never auto-merged**; majors wait for a dashboard tick.
 **Renovate bumps the URLs that exist; it cannot add a CRD a release introduced.** Before merging a
 bump, diff `kubernetes/` (and its `kustomization.yml`) at the new tag against the URL list. 26.0.6
 shipped 2 CRDs, 26.8.2 ships 4 (`keycloakoidcclients`, `keycloaksamlclients` added); PR #121 bumped
 only the existing URLs and the 26.8.2 operator CrashLooped on the missing `keycloaksamlclients`
 informer, stalling Flux `platform` and every tier below it (2026-11-08,
 `decisions/2026-11-08-keycloak-operator-crds.md`). The operator's memory is also patched in
 `kustomization.yaml` (upstream's 450Mi OOMKilled 26.0.6 here); keep the patch across bumps.
 `catalog/keycloak.yaml`'s image tag is documentation only (its render deploys nothing) and
 Renovate is disabled for it — update it by hand to match after an operator bump merges.
@@ -38,6 +38,20 @@ spec:
  ingress:
    enabled: false                         # exposed via Gateway API HTTPRoute (httproute.yaml), not the
                                           # operator's built-in Ingress
  # --- ONE OWNER for the `keycloak` ServiceMonitor: this repo (servicemonitor.yaml), not the operator.
  #     Since 26.x the operator creates its OWN ServiceMonitor, named after this CR (`keycloak`, i.e.
  #     the SAME name/namespace as ours), whenever metrics-enabled=true and the CRD exists —
  #     spec.serviceMonitor.enabled defaults TRUE. On 2026-11-08, right after the 26.0.6 -> 26.8.2
  #     bump, its server-side apply onto our Flux-owned object failed `409 Conflict`, the operator
  #     exhausted retries, the CR sat Ready=False/HasErrors=True, and `platform-config` (wait: true,
  #     so it health-checks this CR) stalled every Kustomization downstream of it. Ours is kept
  #     because the L0 servicemonitor-ports-resolve and L2 keycloak-monitored conformance tests and
  #     the dashboard's up{namespace="keycloak"} are written against it; the operator's would scrape
  #     the same thing (management port 9000, /metrics, 30s). Do NOT flip this to true without
  #     deleting servicemonitor.yaml in the same change — two owners of one object is a fight
  #     (cf. the default-deny NetworkPolicy drift loop).
  serviceMonitor:
    enabled: false
  # --- Hardening + observability options (docs/15). Brute-force detection is a REALM setting
  #     (bruteForceProtected in the realm import); these enable health + metrics endpoints. ---
  additionalOptions:
@@ -20,18 +20,32 @@ kind: Kustomization
 namespace: keycloak
 resources:
  - namespace.yaml
-  # --- Keycloak Operator v26.0.6 (pinned upstream release manifests) ---
+  # --- Keycloak Operator v26.8.2 (pinned upstream release manifests) ---
  # The operator, its CRDs and the SERVER IMAGE move in lockstep: the Keycloak CR has no
  # spec.image, so the operator's RELATED_IMAGE_KEYCLOAK (quay.io/keycloak/keycloak:<same tag>)
-  # is what runs. These three URLs are therefore THE Keycloak version. Each carries a Renovate
+  # is what runs. These URLs are therefore THE Keycloak version. Each carries a Renovate
-  # marker (custom.regex manager, renovate.json5) under ONE depName, so all three bump in ONE
+  # marker (custom.regex manager, renovate.json5) under ONE depName, so all of them bump in ONE
  # branch, never auto-merged (packageRule "keycloak-operator"). Before merging a bump, re-check
  # passkey-reconciler.yaml's 26.0.6 Admin API assumptions — see README.md "Upgrading Keycloak".
  #
  # THE URL LIST MUST MATCH UPSTREAM'S kubernetes/kustomization.yml AT THE NEW TAG — Renovate only
  # rewrites the version in the URLs that are here; it cannot add a CRD the new release introduced.
  # 26.0.6 shipped 2 CRDs; 26.8.2 ships 4 (+ keycloakoidcclients, keycloaksamlclients, both
  # v2alpha1). PR #121 bumped only the existing three URLs, and the 26.8.2 operator — which starts an
  # informer for EVERY kind it controls, used or not — CrashLooped on
  # `Couldn't start informer for keycloaksamlclients.k8s.keycloak.org/v2alpha1 ... Not Found`,
  # stalling Flux `platform` and everything downstream (2026-11-08). On every bump, diff
  #   https://github.com/keycloak/keycloak-k8s-resources/tree/<tag>/kubernetes
  # against this list before merging.
  # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaks.k8s.keycloak.org-v1.yml
  # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakrealmimports.k8s.keycloak.org-v1.yml
  # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloakoidcclients.k8s.keycloak.org-v1.yml
  # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/keycloaksamlclients.k8s.keycloak.org-v1.yml
  # renovate: datasource=github-tags depName=keycloak/keycloak-k8s-resources
  - https://raw.githubusercontent.com/keycloak/keycloak-k8s-resources/26.8.2/kubernetes/kubernetes.yml
  # --- Admin (break-glass) credential — SOPS-encrypted, referenced by name (ADR-0007, R-IDP-04) ---
  - keycloak-admin-credentials.sops.yaml
@@ -51,3 +65,23 @@ resources:
  - keycloak-db-backup-credentials-prod.sops.yaml
  # --- Grafana dashboard for Keycloak health (plain ConfigMap; R-OBS-03) ---
  - grafana-dashboard.yaml
 # --- Operator memory: upstream ships requests=limits=450Mi, which this cluster outgrew ON 26.0.6 ---
 # Measured on prod 2026-11-08: the 26.0.6 operator pod had 154 restarts in 64d, lastState
 # `OOMKilled` (exit 137, 2026-11-07T23:09:54Z), and sat at 421Mi of its 450Mi (94%) in `kubectl top`.
 # 26.8.2 keeps the same 450Mi default but runs FOUR controllers/informers instead of two (adds
 # keycloakoidcclient + keycloaksamlclient) on a newer Quarkus (3.15.1 -> 3.40.1.1), so it would
 # plausibly start above a ceiling 26.0.6 already crossed. Limit 768Mi = ~1.8x the observed 421Mi
 # working set; request 512Mi = the observed working set plus headroom, so the scheduler reserves what
 # it actually uses. Revisit with `kubectl top` once 26.8.2 has run for a week. Patched here (not in
 # the remote kubernetes.yml) so a Renovate bump keeps it.
 patches:
  - target:
      kind: Deployment
      name: keycloak-operator
    patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/resources/limits/memory
        value: 768Mi
      - op: replace
        path: /spec/template/spec/containers/0/resources/requests/memory
        value: 512Mi
@@ -8,6 +8,9 @@
 # /metrics) once metrics-enabled/health-enabled are set (see keycloak-instance.yaml additionalOptions).
 # We select the operator-created keycloak-service (which selects the Keycloak pods) and scrape the pods'
 # management port by number via targetPort.
 # SOLE OWNER: the operator's own ServiceMonitor (same name `keycloak`) is switched OFF in
 #   keycloak-instance.yaml (spec.serviceMonitor.enabled: false) — 26.x defaults it ON and its apply
 #   409-looped against this object on 2026-11-08. Keep exactly one of the two.
 # TODO(WS-SEC, bring-up): confirm the management port number/name the operator exposes (9000 default)
 #   and adjust targetPort if the pinned operator changes it. The KeycloakDown / KeycloakDatabaseDown
 #   alerts (platform/monitoring/alerts.yaml) key on kube-state-metrics and fire regardless of this SM.
@@ -18,7 +18,7 @@ spec:
  chart:
    spec:
      chart: longhorn
-      version: ">=1.12.0 <1.13.0"   # W5 upgrade 1.11->1.12 (resolves 1.12.1; supports k8s 1.36). Renovate maintains
+      version: "<1.14.0"   # W5 upgrade 1.11->1.12 (resolves 1.12.1; supports k8s 1.36). Renovate maintains
      sourceRef:
        kind: HelmRepository
        name: longhorn
@@ -67,7 +67,7 @@ data:
  # adds the CLI failure envelope to triage errors and the PM_DECISION_MAX_TURNS knob
  # (decisions/2026-10-31-pm-agent-digest-99.md).
  # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
-  PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2"
+  PM_WORKER_IMAGE: "gitea.home.keithjackson.dev/home-network/agent-runner@sha256:89ce7eead20ccadf95c2bc6a1ffd497c8c5e2bb93245c9082979742bda4c46a1"
  # --- the model -------------------------------------------------------------
  # The PM only reads a thread and answers with one JSON object, so it holds no tools and
@@ -78,7 +78,7 @@ spec:
          # the registry's manifest endpoint on 2026-10-30 and pinned by hand once so the
          # dispatcher does not wait on Renovate's first run; every later bump is Renovate's.
          # renovate-digest: image=gitea.home.keithjackson.dev/home-network/agent-runner tag=main
-          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:14a42ba8281ecd98005f5d99ecda4fb729104c783b01584097c32e104ced1bb2
+          image: gitea.home.keithjackson.dev/home-network/agent-runner@sha256:89ce7eead20ccadf95c2bc6a1ffd497c8c5e2bb93245c9082979742bda4c46a1
          imagePullPolicy: IfNotPresent
          command:
            - /usr/local/bin/pm-dispatch
@@ -37,6 +37,8 @@
 //     renovate-config-validator /tmp/renovate.json5
 // (Not `--strict`: that also fails on a pending migration of the pre-existing top-level
 // `"docker": {"enabled": true}` block at the bottom, which is a warning, not an error.)
 // RESOLVED 2026-11-08: that block is now hand-migrated to the first packageRule (`matchCategories:
 // ["docker"]`), the exact shape Renovate's own migration produced, so no migration is pending.
 //
 // `regex` vs `custom.regex` in enabledManagers, RESOLVED FROM SOURCE (Renovate 39.264.0 and
 // later): both work. lib/config/migrations/custom/enabled-managers-migration.ts rewrites
@@ -198,6 +200,22 @@
    }
  ],
  "packageRules": [
    // The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were
    // BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation
    // over a running tag. That controller was never enabled and never will be here (see _comment),
    // so the only thing those two flags were achieving was leaving every plain `image:` tag in the
    // repo unmanaged. `docker` is left out of enabledManagers because this repo contains no
    // Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.
    // MOVED HERE 2026-11-08 from a top-level `"docker": {"enabled": true}` key, which Renovate
    // deprecated (the top-level language objects became `matchCategories` rules). Hand-migrated to
    // the exact output of Renovate's own config migration (branch renovate/migrate-config, PR #139:
    // a json5 semantic diff of that file against its parent shows THIS rule, inserted at
    // packageRules[0], as the ONLY change). #139 itself was closed unmerged because its rewrite
    // dropped every comment in this file. With nothing left to migrate, Renovate stops reopening it.
    {
      "matchCategories": ["docker"],
      "enabled": true
    },
    {
      "description": "Auto-merge the allowed low-risk classes on green CI \u2014 no human authoring (R-UPD-02). The Gitea merge gate (make -C ci gate) is the required check.",
      "matchUpdateTypes": [
@@ -254,6 +272,11 @@
        // Keycloak's (the IdP) included, now that imageName is tracked.
        "gitea/gitea",
        "gitea/act_runner",
        // ADDED 2026-11-08. act_runner was RENAMED upstream to gitea/runner after 0.6.1
        // (https://gitea.com/gitea/runner/releases). Renovate does not follow a rename — it will
        // never propose gitea/act_runner -> gitea/runner — so that migration is a manual PR. Listed
        // now so the new name is blast-radius from its first reference, not after an auto-merge.
        "gitea/runner",
        "ghcr.io/cloudnative-pg/postgresql"
      ],
      "automerge": false,
@@ -266,7 +289,7 @@
    // and the server image are released in lockstep from keycloak/keycloak-k8s-resources, and the
    // Keycloak CR carries no spec.image, so the operator's RELATED_IMAGE_KEYCLOAK picks the server
    // (measured on prod 2026-10-30: operator 26.0.6 -> keycloak-0/1 run quay.io/keycloak/keycloak:
-    // 26.0.6). The three release URLs in platform/keycloak/kustomization.yaml carry `# renovate:`
+    // 26.0.6). The release URLs in platform/keycloak/kustomization.yaml carry `# renovate:`
    // markers under this ONE depName, so they move together in one branch (groupName makes that
    // explicit). NEVER AUTO-MERGED, not even a patch: Keycloak is the IdP that gates SSO for every
    // app AND kubectl OIDC, and platform/keycloak/passkey-reconciler.yaml encodes Keycloak 26.0.6
@@ -370,6 +393,45 @@
      "automerge": false,
      "addLabels": ["needs-review", "range-widen"]
    },
    // ADDED 2026-11-08 — KEEP THE LOWER BOUND WHEN A MINOR-LOCKED RANGE MOVES. MEASURED PROBLEM:
    // PRs #131 (longhorn), #130 (headlamp) and #115 (kube-prometheus-stack) turned
    // `">=1.12.0 <1.13.0"` into `"<1.14.0"` — an UNBOUNDED-below range that also admits every
    // older chart (and #130 merged: platform/headlamp/helmrelease.yaml carried `<0.47.0`). WHY,
    // from the 44.167.0 source this file is validated against: no rangeStrategy is set, so it is
    // `auto`; the flux manager has no getRangeStrategy and no updateLockedDependency, so
    // lib/modules/manager/index.ts getRangeStrategy resolves `auto` -> `replace`. The helm
    // datasource's defaultVersioning is `helm`, which is the npm versioning api verbatim
    // (lib/modules/versioning/helm/index.ts: `api = { ...npm }`), and npm getNewValue under
    // `replace` rewrites `>=A <B` to just `<C`. Calling that getNewValue in the 44.167.0 image for
    // currentValue `>=1.12.0 <1.13.0`, newVersion 1.13.0:
    //   replace -> `<1.14.0`   widen -> `>=1.12.0 <1.14.0`   bump -> `>=1.13.0 <1.14.0`
    // Only `bump` keeps a one-minor window. (widen keeps the bound but grows the window by a minor
    // on every PR — the opposite of minor-locking.) Under bump the new lower bound is the release
    // proposed (e.g. 1.13.2 -> `>=1.13.2 <1.14.0`); Flux still picks the newest that satisfies it.
    // Scoped to the bounded `>=A <B` shape only: on a ceiling-only range like `<0.47.0` bump
    // returns the value unchanged, so Renovate would silently propose NOTHING for it, ever.
    {
      "description": "ADDED 2026-11-08. HelmRelease `version: \">=A <B\"` ranges move as a whole window (rangeStrategy bump), so a range-widen PR yields `>=1.13.0 <1.14.0`, not `<1.14.0`. See the comment above for the measured cause.",
      "matchManagers": ["flux"],
      "matchDatasources": ["helm"],
      "matchCurrentValue": "/^>=\\S+ <\\S+$/",
      "rangeStrategy": "bump"
    },
    // ...AND DO NOT LET bump ADD NOISE. Under bump Renovate also raises the lower bound for a
    // release ALREADY INSIDE the range (lib/workers/repository/process/lookup/generate.ts sets
    // isBump; e.g. 1.12.5 against `>=1.12.0 <1.13.0` -> `>=1.12.5 <1.13.0`), which `replace` never
    // proposed. Flux already installs that release with no PR, and the range-widen rule above
    // would hold every one for review. util/package-rules/update-types.ts matches isBump as
    // updateType `bump`, so exactly those updates are switched off here; out-of-range moves are
    // ordinary minor/major updates and are unaffected.
    {
      "description": "ADDED 2026-11-08. No PR for a chart release the HelmRelease range already admits — Flux adopts it on its own.",
      "matchManagers": ["flux"],
      "matchDatasources": ["helm"],
      "matchCurrentValue": "/^>=\\S+ <\\S+$/",
      "matchUpdateTypes": ["bump"],
      "enabled": false
    },
    {
      "description": "ADDED 2026-10-30. velero-plugin-for-aws is pinned to a RELEASE CANDIDATE (v1.14.3-rc.2) and Renovate proposed NOTHING for it, although v1.14.3 and v1.14.4 are stable on Docker Hub. Docker versioning treats everything after the first `-` as a compatibility SUFFIX, so only other `*-rc.2` tags qualify and the RC is a fossil forever. Semver versioning reads it as a prerelease, and a prerelease upgrades to the stable release.",
      "matchPackageNames": ["velero/velero-plugin-for-aws"],
@@ -441,14 +503,5 @@
    "conformance/tests/**",
    "platform/image-automation/per-service/**",
    "platform/kyverno/tests/**"
-  ],
+  ]
  // The `docker` manager (Dockerfiles, docker-compose) and the `kubernetes` manager above were
  // BOTH previously `enabled: false`, solely so Renovate would not fight Flux image-automation
  // over a running tag. That controller was never enabled and never will be here (see _comment),
  // so the only thing those two flags were achieving was leaving every plain `image:` tag in the
  // repo unmanaged. `docker` is left out of enabledManagers because this repo contains no
  // Dockerfile that ships to the cluster; `kubernetes` is now ON and is the one that matters.
  "docker": {
    "enabled": true
  }
 }
@@ -123,6 +123,9 @@ Each affected runbook says so on its own page and gives the by-hand equivalent.
 - **`vpn-remote-access`** — the INBOUND remote-access VPN (WireGuard, udp/51820): standing it up,
  enrolling and revoking devices, and the off-LAN checks that are the only honest proof it works.
  The opposite direction to the two pages above — do not confuse them.
 - **`cloudflare-tunnel`** — the Cloudflare Tunnel connector (inbound HTTPS from the Cloudflare edge to
  the shared Gateway): enrolling it, the canary, `CloudflaredTunnelDown` / `CloudflaredReplicasDegraded`,
  and rollback. Never `cloudflared tunnel route dns` — external-dns owns the records.
 ### Backups and DR
 - **`backup-failure`** — a Velero backup failed. The R2 object-tagging incompatibility is documented here.
@@ -0,0 +1,337 @@
 # cloudflare-tunnel — enrolling the Cloudflare Tunnel, proving it, and the two alerts
 Alerts that link here: **`CloudflaredTunnelDown`** (no replica has a live edge connection, or none
 is being scraped) and **`CloudflaredReplicasDegraded`** (fewer than 2 replicas Available).
 Component: `platform/cloudflared/` (Flux Kustomization `cloudflared`, `clusters/base/cloudflared.yaml`).
 Design record: `decisions/2026-11-08-cloudflare-tunnel.md`.
 Every step below is an **operator** step. An agent must not mint the tunnel credential, run
 `flux reconcile`, or mutate the cluster (CLAUDE.md §5, §6).
 ## What this means here
 cloudflared is an **outbound-only** connector: two replicas dial the Cloudflare edge on 7844
 (QUIC/udp, falling back to HTTP/2 over tcp) and carry requests back to the shared Gateway
 (`https://cilium-gateway-shared.gateway.svc.cluster.local:443`, SNI = the request's Host, so the
 Gateway's existing `wildcard-home-tls` Let's Encrypt certificate is what cloudflared validates).
 The tunnel is **locally managed**: its ingress rules are `platform/cloudflared/config.yaml`, in git,
 not in the Cloudflare dashboard.
 The operator decisions it encodes (2026-11-08):
 1. Public hostnames stay `*.home.keithjackson.dev` and the apex; edge certificates come from
   Cloudflare **ACM + Total TLS**.
 2. **Plex is not on the tunnel** (`plex`, `plex-vip` → 404).
 3. **jackett, pihole, semaphore, qbittorrent, radarr, sonarr** and **headlamp** stay LAN/VPN-only,
   as do the L4 VIPs **unifi** and **crafty-mc** (all → 404 in cloudflared, and the LAN-only
   Gateway backends are also refused by the egress policy). Headlamp is an operator decision of
   2026-11-08; the other six are the catalog's `exposure: internal` services.
 4. **The Kubernetes API is never reachable through the tunnel.** No rule targets it and the egress
   policy admits neither the `kube-apiserver` nor the `host` entity. The L0 check
   `cloudflared-ingress-rules` fails the gate if that ever changes.
 **Nothing goes through the tunnel until DNS points at it.** Cloudflare routes a hostname through a
 tunnel only when its record is a CNAME to `<UUID>.cfargotunnel.com`. Phase 1 (this component)
 creates no such record; moving public DNS is a separate, later change. So in Phase 1 the alerts
 firing means "the tunnel is not ready", not "users are down".
 Before the steps below are done, the Deployment CrashLoops (placeholder UUID and credentials) and
 `CloudflaredTunnelDown` fires. That is expected, and it is contained: the `cloudflared` Kustomization
 is `wait: false` and nothing depends on it.
 ## First commands
 ```
 # Are the pods running, and on different nodes?
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared get deploy,pods -o wide'
 # What does cloudflared say? Healthy: four "Registered tunnel connection" lines per replica.
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared logs deploy/cloudflared --tail=50'
 # Did Flux apply the current revision, and is the Secret decrypted (key NAMES only)?
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n flux-system get kustomization cloudflared;
   kubectl --context k3s-prod -n cloudflared get secret cloudflared-credentials \
     -o go-template="{{range \$k, \$v := .data}}{{\$k}}{{\"\n\"}}{{end}}"'
 ```
 Never print the Secret's values. `ENC[` in the decrypted Secret means the overlay's decryption
 patch is missing; see causes below.
 ## Likely causes, most likely first
 **Read the FIRST error line, not the last.** When cloudflared cannot use the credentials file it
 falls back to looking for an origin certificate, and the error it ends on —
 `Cannot determine default origin certificate path` / `error parsing tunnel ID: Error locating
 origin cert` — is a **red herring** whatever the real cause was. The credentials file is the
 whole of what this Deployment needs; no `cert.pem` ever belongs in the pod.
 1. **`permission denied` on `credentials.json` → the "origin cert" error is a red herring.** Logs:
   `couldn't read tunnel credentials from /etc/cloudflared/creds/credentials.json: permission
   denied`, then the origin-cert lines. The container (uid 65532) cannot read the Secret file:
   the pod `securityContext.fsGroup: 65532` is missing, so the 0440 file is `root:root`. Fix it in
   git (`platform/cloudflared/deployment.yaml`); the L0 check `cloudflared-ingress-rules`,
   assertion (7), should already have refused that. Do NOT loosen `defaultMode` to 0444. (This is
   what CrashLooped every replica on 2026-11-08.)
 2. **The tunnel has not been enrolled yet** (Phase 1, steps 1-5 below not done). Logs show an
   error about the tunnel `REPLACE-ME-TUNNEL-UUID` (and the origin-cert red herring). Do steps 1-5.
 3. **UUID and credentials disagree** — `config.yaml` `tunnel:` is not the `TunnelID` in the
   credentials JSON. Logs: authentication / "tunnel not found". Check with step 4's verify command.
 4. **Credentials still encrypted in the cluster** — the `cloudflared` decryption patch is missing
   from `clusters/prod/kustomization.yaml`. The L0 check guards this, so this means someone bypassed
   it.
 5. **Egress blocked** — a policy change, or the edge moved off 7844. Look for drops:
   ```
   infra/devcontainer/dev.sh run bash -c \
     'kubectl --context k3s-prod -n kube-system exec ds/cilium -c cilium-agent -- \
        hubble observe --verdict DROPPED --from-namespace cloudflared --last 100 -o compact'
   ```
   (`ds/cilium` picks one node's agent; repeat per node if empty.)
 6. **The tunnel was deleted or its secret rotated in Cloudflare** — `cloudflared tunnel list`
   (step 2's podman form) no longer shows it. Re-enrol with steps 2-5.
 7. **`CloudflaredReplicasDegraded` only**: one replica cannot schedule (the spread is hard, one per
   node — a cordoned node during a kured reboot is normal and clears) or an image pull failed.
 ## When to escalate or stop
 - **Stop** if anything suggests a rule reaching the Kubernetes API or a LAN-only service. Do not
  hot-fix through the dashboard: the tunnel is locally managed, and a dashboard edit is silently
  overridden by `config.yaml`. Use the emergency stop under *Rollback*, then fix in git.
 - **Never run `cloudflared tunnel route dns`** — not for a real hostname, not for a test one.
  **external-dns owns the `home.keithjackson.dev` records** (`platform/externaldns`, `policy: sync`);
  a record created behind its back either fights it or is deleted by it, and a real hostname
  routed that way is an unreviewed DNS flip. Moving records onto the tunnel is the separate Phase 2
  change. The only record this runbook creates is the hand-made canary in step 6, and you delete it.
 - Escalate to Cloudflare status (https://www.cloudflarestatus.com) if every replica is
  Running, egress shows no drops, and logs show edge connection errors on both QUIC and HTTP/2.
 ---
 ## Enrolling the tunnel (once)
 cloudflared is **not** in the pinned toolbox (checked 2026-11-08). Run it from the same pinned image
 the Deployment uses, under podman — no host install. The working directory is
 `~/.config/sops/cloudflared`: it is private to you, and it is the one host directory `dev.sh`
 already mounts into the toolbox (as `/root/.config/sops`), so the toolbox's `sops` can read the
 credentials JSON without it ever entering the repository tree.
 ```
 CF_IMG='docker.io/cloudflare/cloudflared:2026.10.7@sha256:4955524aeaa379e154023d5a027c250f7b35041a2e7e950cf13f144bc9b9c5ba'
 CF_DIR="$HOME/.config/sops/cloudflared"
 mkdir -p "$CF_DIR" && chmod 700 "$CF_DIR"
 cf() { podman run --rm -it --user 0 --security-opt label=disable \
         -e HOME=/root -v "$CF_DIR:/root/.cloudflared" "$CF_IMG" "$@"; }
 ```
 (Keep `CF_IMG` in step with `platform/cloudflared/deployment.yaml` if Renovate has bumped it.)
 ### Step 1 — edge certificate (ACM) and edge settings (Cloudflare dashboard)
 Universal SSL covers only `keithjackson.dev` and `*.keithjackson.dev`; it does **not** cover the
 second-level `*.home.keithjackson.dev`. In the `keithjackson.dev` zone:
 1. **SSL/TLS → Edge Certificates → Advanced Certificate Manager**: subscribe.
 2. Then EITHER **Order an advanced certificate** for exactly `home.keithjackson.dev,
   *.home.keithjackson.dev` (covers every name at once, independent of proxy state), OR turn on
   **Total TLS** (issues one certificate per *proxied* hostname as each one appears). The ordered
   certificate is the more predictable of the two for the later DNS flip; either satisfies the
   operator decision.
 3. **Wait until the certificate shows `Active`** in the Edge Certificates list. Do not do step 6's
   canary before then — a TLS error at the edge would be the certificate, not the tunnel.
 4. Leave **SSL/TLS encryption mode** at *Full (strict)*. (It does not govern tunnel origins —
   cloudflared verifies the Gateway itself — but it is the safe default for anything else.)
 Edge settings that would break non-browser clients once traffic moves (check them now, in the zone):
 - **Security → Bots → Bot Fight Mode: OFF.** It cannot be scoped by hostname or skipped by a WAF
  rule, and it challenges `git`, `docker`/registry pushes, Nextcloud sync clients and API calls.
 - **Browser Integrity Check** and the **WAF / managed rules**: make sure neither challenges or
  blocks non-browser user agents on `gitea.home.keithjackson.dev` (git over HTTPS and the container
  registry under `/v2/`) or on API paths. If either does, add a WAF **Skip** custom rule for those
  hostnames/paths rather than turning protection off zone-wide.
 - **Network → WebSockets: ON** (Grafana live, Headlamp-style consoles, Nextcloud, KubeVirt VNC).
 ### Step 2 — log in and create the tunnel
 ```
 cf tunnel login
 # Open the printed URL, pick the keithjackson.dev zone. Writes $CF_DIR/cert.pem — an ACCOUNT-level
 # origin certificate. Treat it as a credential.
 cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s
 # Prints "Created tunnel home-k3s with id <UUID>". The UUID is not secret; note it.
 cf tunnel list
 ```
 ### Step 3 — put the UUID into the config
 Do this in your own worktree on a new branch, as CLAUDE.md §0 requires:
 ```
 UUID=<the id from step 2>
 sed -i "s/^tunnel: REPLACE-ME-TUNNEL-UUID$/tunnel: ${UUID}/" platform/cloudflared/config.yaml
 grep '^tunnel:' platform/cloudflared/config.yaml
 ```
 ### Step 4 — encrypt the credentials into the Secret, without printing them
 Run from that worktree. `sops set` edits the already-encrypted document in place; the JSON goes
 from the mounted file to `sops` inside the container and never reaches the terminal:
 ```
 ./infra/devcontainer/dev.sh run bash -c '
  set -euo pipefail
  cd /work
  f=/root/.config/sops/cloudflared/tunnel.json
  v="$(jq -c . "$f" | tr -d "\n" | jq -Rs .)"
  sops set platform/cloudflared/cloudflared-credentials.sops.yaml \
    "[\"stringData\"][\"credentials.json\"]" "$v"
  echo "credentials written"
 '
 ```
 (From a worktree, `dev.sh` needs `DEV_KUBECONFIG=/dev/null` and the `.git` mount shim — CLAUDE.md §3.
 Or run this step from the main checkout and copy the encrypted file across.)
 Verify — prints only `ok` or fails, never a value:
 ```
 ./infra/devcontainer/dev.sh run bash -c '
  cd /work
  UUID=$(sed -n "s/^tunnel: //p" platform/cloudflared/config.yaml)
  sops -d --extract "[\"stringData\"][\"credentials.json\"]" \
      platform/cloudflared/cloudflared-credentials.sops.yaml \
    | jq -e --arg u "$UUID" ".TunnelID == \$u and (.TunnelSecret | length) > 20 and (.AccountTag | length) > 0" \
      >/dev/null && echo ok'
 ```
 Then store `tunnel.json` **and** `cert.pem` in your password manager, and delete them from disk:
 ```
 rm -f "$CF_DIR/tunnel.json" "$CF_DIR/cert.pem"
 ```
 (`cert.pem` is only needed again to inspect (`tunnel info`), create or delete a tunnel. DNS records are
 external-dns's, never `tunnel route dns`'s.)
 ### Step 5 — PR it
 Commit `config.yaml` and the re-encrypted `cloudflared-credentials.sops.yaml`, gate, open a PR. No
 other wiring is needed: the resource line and both overlay decryption patches already exist. On
 merge Flux rolls the Deployment (the ConfigMap is content-hashed).
 That PR touches `**/*.sops.yaml`, a sensitive path in `infra/gitea/approval-policy.json`, so
 `approval-policy` will block it until you post `/approve-override <reason>` as a PR **review** of
 type Comment (CLAUDE.md §1).
 ### Step 6 — verify
 ```
 # Both replicas Ready, on different nodes:
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared get pods -o wide'
 # Logs: "Registered tunnel connection connIndex=0..3" on each replica.
 infra/devcontainer/dev.sh run bash -c \
  'kubectl --context k3s-prod -n cloudflared logs -l app.kubernetes.io/name=cloudflared --tail=40'
 ```
 From the workstation: `cf tunnel info home-k3s` (needs `cert.pem` back in `$CF_DIR` temporarily)
 lists **two connectors**. In Grafana/Prometheus, `sum(cloudflared_tunnel_ha_connections)` is ~8 and
 both alerts are clear.
 **Canary — end to end, without touching any real record.** Hand-make ONE proxied CNAME for a
 throwaway name (Cloudflare dashboard → DNS → Records → Add record):
    Type CNAME   Name tunnel-canary.home   Target <UUID>.cfargotunnel.com   Proxy status: Proxied
 Do NOT use `cloudflared tunnel route dns` for it (see *When to escalate or stop*). The name hits the
 wildcard rule, so it travels edge → cloudflared → Gateway, where no HTTPRoute matches it:
 ```
 curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://tunnel-canary.home.keithjackson.dev/
 # expected: "404 0" — a 404 from the Gateway's Envoy (no route for that host) over a VALID edge
 # certificate (ssl_verify_result 0). A TLS error = step 1 is not Active yet. 502 = cloudflared
 # could not reach or verify the Gateway; check the logs and `hubble observe` above.
 ```
 **Then prove a REAL backend answers the tunnel's identity without a 403.** The canary cannot show
 this: Envoy 404s an unknown host before it resolves any backend, and the `403 Access denied`
 failure (Cilium re-checking the source pod's egress against the backend, CVE-2025-30162) only
 happens once a backend is chosen. An ephemeral container in a cloudflared pod shares that pod's
 network identity, so it takes exactly the tunnel's path to the Gateway Service:
 ```
 infra/devcontainer/dev.sh run bash -c '
  POD=$(kubectl --context k3s-prod -n cloudflared get pod -l app.kubernetes.io/name=cloudflared \
          -o jsonpath="{.items[0].metadata.name}")
  kubectl --context k3s-prod -n cloudflared debug "$POD" -q -i --image=docker.io/curlimages/curl:8.11.1 \
    --target=cloudflared -- sh -c "for h in grafana homepage-unused gitea; do
      printf \"%s \" \$h; curl -sS -o /dev/null -w \"%{http_code}\n\" \
        --connect-to ::cilium-gateway-shared.gateway.svc.cluster.local:443 https://\$h.home.keithjackson.dev/; done"'
 # expected: grafana and gitea answer 200/302 (anything but 403); a 403 there means the egress
 # policy's backend selector does not cover that namespace. "homepage-unused" is a control: 404.
 # A LAN-only host (e.g. jackett) is EXPECTED to 403 here — that is the second layer working.
 ```
 (`kubectl debug` adds an ephemeral container: a cluster mutation, so an operator action. It
 disappears with the pod on the next rollout.)
 Finally **delete the `tunnel-canary` CNAME** in the dashboard and confirm it is gone:
 `dig +short tunnel-canary.home.keithjackson.dev @1.1.1.1` returns nothing.
 ## Rotating the tunnel credential
 Do this whenever the credentials JSON (`TunnelSecret`) or `cert.pem` may have been exposed — a
 transcript, a log, a screen share. A tunnel's secret cannot be changed in place from this
 workflow, so rotation is **a new tunnel**: the old credential stays valid until the old tunnel
 is deleted, which is why the last step is not optional.
 1. **Create the new tunnel** (*Enrolling*, step 2's `cf` helper; `cf tunnel login` again if
   `cert.pem` is not to hand). Use a NEW name — the old one is still taken:
   ```
   cf tunnel list                       # note the OLD tunnel's id and name
   cf tunnel create --credentials-file /root/.cloudflared/tunnel.json home-k3s-2
   ```
 2. **Swap the UUID** in your own worktree, on a new branch (CLAUDE.md §0):
   ```
   OLD=$(sed -n 's/^tunnel: //p' platform/cloudflared/config.yaml)
   sed -i "s/^tunnel: ${OLD}$/tunnel: <NEW UUID>/" platform/cloudflared/config.yaml
   ```
 3. **Swap the encrypted credentials** with *Enrolling* step 4 — the same `sops set`, then the
   same `ok`-only verify, which now checks the JSON's `TunnelID` against the NEW UUID. Never
   `sops -d` to the terminal.
 4. **Merge** — gate, PR, `/approve-override` by the operator (it touches `**/*.sops.yaml`), merge.
   Flux rolls the Deployment because the content-hashed ConfigMap changed.
 5. **Verify** with *Enrolling* step 6: both replicas Ready, `Registered tunnel connection` in
   each replica's log, `cf tunnel info home-k3s-2` lists two connectors, and
   `cf tunnel info <old name>` lists **none**.
 6. **Only then delete the old tunnel**, which revokes the exposed credential:
   ```
   cf tunnel cleanup <old name>   # drops any stale connector records first
   cf tunnel delete <old name>
   cf tunnel list                 # the old tunnel is gone
   ```
   After the Phase 2 DNS flip, records point at `<UUID>.cfargotunnel.com`: confirm external-dns
   has repointed every record to the NEW UUID before deleting, or those hostnames go dark.
   Then replace `tunnel.json`/`cert.pem` in the password manager and delete them from disk.
 ## Rollback
 - **Phase 1 (no DNS on the tunnel):** revert the PR. Flux prunes the `cloudflared` namespace and
  everything in it. No user traffic is affected either way. Optionally `cf tunnel delete home-k3s`
  (needs `cert.pem`) to retire the credential.
 - **Emergency stop (operator, cluster mutation):** stop the connectors without waiting for a PR:
  ```
  infra/devcontainer/dev.sh run bash -c \
    'flux --context k3s-prod suspend kustomization cloudflared &&
     kubectl --context k3s-prod -n cloudflared scale deploy/cloudflared --replicas=0'
  ```
  Cloudflare then answers tunnel hostnames with a 530/1033 error. Follow up in git, then
  `flux resume kustomization cloudflared`.
 - **After the DNS flip:** the fast path back is DNS, not the tunnel — restore the previous records
  (that change's own runbook). Revoking the tunnel credential (`cf tunnel delete`, or rotating it
  in the Zero Trust dashboard) cuts every connector at once if the credential is believed leaked.
@@ -58,6 +58,7 @@ configMapGenerator:
      # improvised live, and the one hazard it exposed -- a StatefulSet with
      # persistentVolumeClaimRetentionPolicy whenScaled: Delete loses its PVC when scaled to 0 --
      # destroyed monitoring/loki's volume before anyone noticed.
      - cloudflare-tunnel=cloudflare-tunnel.md
      - cluster-maintenance-shutdown=cluster-maintenance-shutdown.md
      - cdi-import-stuck=cdi-import-stuck.md
      - cert-expiry=cert-expiry.md
Author	SHA1	Message	Date
jakman85andClaude Opus 5.5	d9ab57f2d1	fix(cloudflared): fsGroup so uid 65532 can read the 0440 credentials Secret gate / gate (pull_request) Successful in 42s Details gate / conformance (pull_request) Skipped Details The pod ran as 65532:65532 with no fsGroup, so the kubelet laid the Secret's credentials.json down root:root 0440 — unreadable to the container. cloudflared logged "permission denied" and then the red-herring "Error locating origin cert". - deployment.yaml: securityContext.fsGroup 65532 (mode stays 0440) - L0 cloudflared-ingress-rules (7): Secret volumes must be readable by the container's run-as uid/groups; mutation self-tested - runbook: permission-denied troubleshooting entry, credential rotation section - decisions/2026-11-08-cloudflared-creds-permission.md Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 15:49:56 -06:00
jakman85	9724445a08	Merge pull request 'Added cloudflare creds and tunnel config' (#153 ) from feature/cloudflare-tunnel-config into main gate / gate (push) Successful in 40s Details gate / conformance (push) Skipped Details Reviewed-on: #153	2026-11-08 21:34:24 +00:00
jakman85	32b7d5bac1	Added cloudflare creds and tunnel config gate / gate (pull_request) Successful in 42s Details gate / conformance (pull_request) Skipped Details	2026-11-08 15:30:19 -06:00
jakman85	83e0ac194f	Merge pull request 'feat(cloudflared): Cloudflare Tunnel connector, Phase 1 (isolated, no traffic change)' (#152 ) from feat/cloudflare-tunnel into main gate / gate (push) Successful in 40s Details gate / conformance (push) Skipped Details Reviewed-on: #152 Reviewed-by: Keith Jackson <[email protected]>	2026-11-08 18:43:42 +00:00
jakman85andClaude Opus 5.5	09f219fe7f	fix(cloudflared): auditable Service+ServiceMonitor scrape; placeholder-only template value gate / gate (pull_request) Successful in 42s Details gate / conformance (pull_request) Skipped Details servicemonitor-ports-resolve could not audit a PodMonitor with no in-repo Service, and no-plaintext-secrets requires a template's secret value to be a bare placeholder. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 12:39:56 -06:00
jakman85andClaude Opus 5.5	aabc8c60b6	feat(cloudflared): Cloudflare Tunnel connector, Phase 1 (isolated, no traffic change) Adds platform/cloudflared: a locally-managed tunnel whose ingress rules live in git. LAN/VPN-only hosts (plex, plex-vip, jackett, pihole, pihole-dns, semaphore, qbittorrent, radarr, sonarr, headlamp, unifi, crafty-mc) map to 404. The apex and *.home go to cilium-gateway-shared.gateway over TLS with matchSNItoHost. Last comes a catch-all 404. Nothing routes to the Kubernetes API. - 2-replica Deployment, hard node spread, non-root, pinned digest, PDB - default-deny plus an egress CNP: Cloudflare edge 7844/443, the Gateway, and Gateway backends by gateway-access=shared minus the LAN-only namespaces (needed because of the CVE-2025-30162 L7LB egress re-check) - PodMonitor plus two alerts that fire on absent series, with promtool tests - its own Flux Kustomization (dependsOn platform-config, wait:false, nothing depends on it), plus SOPS decryption patches in both overlays - placeholder credentials (SOPS-encrypted) so it can merge before enrolment - L0 cloudflared-ingress-rules, and a cloudflared_ scrape owner - runbooks/cloudflare-tunnel.md and decisions/2026-11-08-cloudflare-tunnel.md Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 12:38:22 -06:00
renovate-bot	eb62c13c6b	Merge pull request 'chore(deps): Update gitea.home.keithjackson.dev/home-network/conformance-runner:main Docker digest to 63e1aa7' (#151 ) from renovate/gitea.home.keithjackson.dev-home-network-conformance-runner-main into main gate / gate (push) Successful in 41s Details gate / conformance (push) Skipped Details	2026-11-08 14:15:18 +00:00
renovate-bot	4226292dd6	Merge pull request 'chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to 89ce7ee' (#150 ) from renovate/gitea.home.keithjackson.dev-home-network-agent-runner-main into main gate / gate (push) Successful in 40s Details gate / conformance (push) Skipped Details	2026-11-08 08:17:22 +00:00
renovate-bot	213429a41f	chore(deps): Update gitea.home.keithjackson.dev/home-network/conformance-runner:main Docker digest to 63e1aa7 gate / gate (pull_request) Successful in 41s Details gate / conformance (pull_request) Skipped Details	2026-11-08 08:16:28 +00:00
renovate-bot	0120bc10d4	chore(deps): Update gitea.home.keithjackson.dev/home-network/agent-runner:main Docker digest to 89ce7ee gate / gate (pull_request) Successful in 52s Details gate / conformance (pull_request) Skipped Details	2026-11-08 08:16:27 +00:00
jakman85	d1963bd000	Merge pull request 'chore(deps): Update Helm release longhorn to <1.14.0' (#131 ) from renovate/longhorn-1.x into main gate / gate (push) Failing after 40s Details gate / conformance (push) Skipped Details Reviewed-on: #131	2026-11-08 06:55:55 +00:00
jakman85	0b0b5abc8e	Merge pull request 'feat(ci): run L0/L1 offline checks, mutation suites and yamllint in bounded parallel pools' (#149 ) from feat/parallel-l0-checks into main gate / gate (push) Successful in 39s Details gate / conformance (push) Skipped Details Reviewed-on: #149 Reviewed-by: Keith Jackson <[email protected]>	2026-11-08 06:53:42 +00:00
jakman85	6e2e353bcc	Merge pull request 'feat(ci): run the merge gate inside the CI-built toolbox image, pinned by digest' (#147 ) from feat/gate-toolbox-container into main gate / gate (push) Successful in 2m33s Details gate / conformance (push) Skipped Details Reviewed-on: #147 Reviewed-by: Keith Jackson <[email protected]>	2026-11-08 06:51:00 +00:00
jakman85	045d81c7a2	Merge pull request 'feat(gitea-actions): act_runner capacity 2, dind sized for two concurrent jobs' (#146 ) from feat/runner-capacity-2 into main gate / gate (push) Canceled after 1m15s Details gate / conformance (push) Canceled after 0s Details Reviewed-on: #146	2026-11-08 06:47:05 +00:00
jakman85	7be693bf37	Merge pull request 'chore(deps): Update quay.io/oauth2-proxy/oauth2-proxy Docker tag to v7.15.6' (#136 ) from renovate/quay.io-oauth2-proxy-oauth2-proxy-7.x into main gate / gate (push) Canceled after 1m32s Details gate / conformance (push) Canceled after 0s Details Reviewed-on: #136 Reviewed-by: Keith Jackson <[email protected]>	2026-11-08 06:45:31 +00:00
jakman85	9e9a3ea8cc	Merge pull request 'fix(keycloak): one owner for the keycloak ServiceMonitor (26.8 operator 409 loop blocks Flux)' (#148 ) from fix/keycloak-servicemonitor-owner into main gate / gate (push) Successful in 3m0s Details gate / conformance (push) Skipped Details Reviewed-on: #148	2026-11-08 06:37:13 +00:00
jakman85andClaude Opus 5.5	89f48e4a5f	docs(decisions): handoff for parallel L0 checks; runner usage names --jobs gate / gate (pull_request) Successful in 59s Details gate / conformance (pull_request) Skipped Details Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 00:34:48 -06:00
jakman85andClaude Opus 5.5	35ba875380	fix(keycloak): one owner for the keycloak ServiceMonitor (26.8 operator 409 loop blocks Flux) gate / gate (pull_request) Successful in 3m2s Details gate / conformance (pull_request) Skipped Details The 26.x Keycloak operator creates its own ServiceMonitor named after the CR (keycloak/keycloak) when metrics are enabled; spec.serviceMonitor.enabled defaults true. That collides with the Flux-owned platform/keycloak/ servicemonitor.yaml: the operator's PATCH 409s, it exhausts retries, the CR stays Ready=False/HasErrors=True, and platform-config (wait: true) stalls every downstream Kustomization. Keep the repo's ServiceMonitor (conformance L0/L2 and the dashboard are written against it; the operator's scrapes the same port 9000 /metrics) and set spec.serviceMonitor.enabled: false. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 00:25:45 -06:00
jakman85andClaude Opus 5.5	a775082ae1	feat(ci): run L0/L1 offline checks, mutation suites and yamllint in bounded parallel pools - conformance/lib/run_l1_offline.py: checks run in a thread pool (--jobs / CONF_OFFLINE_JOBS, default min(8, usable CPUs)); results consumed in submission order, so log lines, JUnit, step summary and the 0/1/3 aggregation are unchanged. Exports a smaller MUTATION_JOBS to the children when parallel, so nested pools do not oversubscribe. - keycloak-{passkey,client-secret}-reconciler/mutations.py: anchors checked first (serial), then the --selftest subprocesses run in a pool (MUTATION_JOBS, default min(8, CPUs)); verdicts reported in MUTATIONS order. Output byte-identical to the serial loop. - ci/lint.sh: yamllint over LINT_JOBS contiguous slices concurrently, per-slice output buffered and printed in order; stage rc = highest slice rc. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 00:18:57 -06:00
jakman85	1a33814522	Merge pull request 'chore(renovate): hand-migrate docker key, track gitea/runner, keep range lower bounds' (#145 ) from chore/renovate-config-cleanup into main gate / gate (push) Successful in 2m59s Details gate / conformance (push) Skipped Details Reviewed-on: #145	2026-11-08 06:17:59 +00:00
jakman85	4a3f6dee59	Merge pull request 'fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop)' (#144 ) from fix/keycloak-operator-crds into main gate / gate (push) Canceled after 0s Details gate / conformance (push) Canceled after 0s Details Reviewed-on: #144	2026-11-08 06:16:56 +00:00
jakman85andClaude Opus 5.5	b548224db8	feat(gitea-actions): act_runner capacity 2, dind sized for two concurrent jobs gate / gate (pull_request) Successful in 2m59s Details gate / conformance (pull_request) Skipped Details The in-cluster runner ran at the default capacity 1 with no config file, so gate / images / renovate runs on different refs queued behind each other (29-759s waits; run 472 waited 759s behind a renovate burst). - ConfigMap act-runner-config with ONLY runner.capacity: 2, selected via CONFIG_FILE (honoured by the v0.6.1 image's run.sh). v0.6.1 LoadDefault() applies the same defaults to a partial file as to no file; the generate-config example is deliberately not used (its labels would override the registered ones, force_pull differs from the default). - Pod-template config-revision annotation so ConfigMap-only edits can roll it. - dind: cpu request 100m -> 2 (no limit), memory 256Mi/2Gi -> 1Gi/4Gi. Measured at capacity 1: peak 0.69 core, 447Mi working set (30d), 0 OOMs. - runner: memory 128Mi/512Mi -> 256Mi/1Gi (peak 314Mi at capacity 1). apps/wave-1/gitea-actions.yaml is a hand-owned EXTRA_UNITS file (the renderer only places it in a wave), so it is the source, not generated output. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 00:09:08 -06:00
jakman85andClaude Opus 5.5	759a0e51c8	chore(renovate): hand-migrate docker key, track gitea/runner, keep range lower bounds gate / gate (pull_request) Successful in 2m58s Details gate / conformance (pull_request) Skipped Details - Replace the deprecated top-level `docker` key with the packageRule Renovate's own migration produced (matchCategories docker, enabled true), keeping every comment. Supersedes #139, which dropped all comment lines. - Add gitea/runner (act_runner's upstream rename) to the blast-radius rule. - rangeStrategy bump for flux/helm `>=A <B` ranges: the default `auto` resolves to `replace`, which turned `>=1.12.0 <1.13.0` into `<1.14.0` (#131, #130, #115). In-range `bump` updates are disabled; Flux adopts those unaided. - Restore headlamp's lower bound: `<0.47.0` -> `>=0.46.0 <0.47.0` (live 0.46.0). Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 00:05:49 -06:00
jakman85andClaude Opus 5.5	ea30ba876c	fix(keycloak): install the 26.8.2 operator's new CRDs (operator CrashLoop) gate / gate (pull_request) Successful in 3m5s Details gate / conformance (pull_request) Skipped Details PR #121 bumped the three keycloak-k8s-resources URLs to 26.8.2, but 26.8.2 ships two more CRDs (keycloakoidcclients, keycloaksamlclients). The new operator fails to start its keycloaksamlclients informer (Not Found) and CrashLoops, holding Flux `platform` in health checks. - add both CRD URLs with the same renovate marker/depName - patch operator memory 450Mi -> limit 768Mi / request 512Mi (26.0.6 was OOMKilled at 421/450Mi, 154 restarts) - README/renovate comments: Renovate cannot add CRDs a release introduces Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-08 00:00:01 -06:00
jakman85andClaude Opus 5.5	3d7ecbcfe9	fix(conformance): regenerate L1 render goldens for oauth2-proxy v7.15.6 gate / gate (pull_request) Successful in 3m0s Details gate / conformance (pull_request) Skipped Details PR #136 bumps the renderer's OAUTH2_PROXY_IMAGE; the container-workload and egress-class (vpn + internet) goldens still pinned v7.8.1. Regenerated with the check scripts' own invocation; each diff is the image tag only. Also updates the crafty README's current-pin row. Co-Authored-By: Claude Opus 5.5 <[email protected]>	2026-11-07 23:58:29 -06:00
renovate-bot	893c9dee4f	chore(deps): Update quay.io/oauth2-proxy/oauth2-proxy Docker tag to v7.15.6 gate / gate (pull_request) Failing after 2m57s Details gate / conformance (pull_request) Has been skipped Details	2026-11-07 08:18:51 +00:00
renovate-bot	619c2943da	chore(deps): Update Helm release longhorn to <1.14.0 gate / gate (pull_request) Successful in 2m54s Details gate / conformance (pull_request) Has been skipped Details	2026-11-07 08:17:40 +00:00