fix(cilium): point cgroup.hostRoot at the host's real cgroup2 root

hostRoot was /run/cilium/cgroupv2 with autoMount disabled, but nothing mounts cgroup2 there. The agent mounted one from inside its own private cgroup namespace, so socket-LB attached to the agent's own cgroup and host processes got no Service/LB translation. k3s-prod-1 (Gateway L2 lease holder + kube-vip VIP) therefore could not reach the Gateway VIP, and its apiserver's OIDC authenticator has failed since 2026-09-12. Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-03 18:07:18 -05:00
15 changed files with 110 additions and 802 deletions
@@ -14,12 +14,10 @@ resources:
 - nextcloud-db-backup-credentials-prod.sops.yaml
 - nextcloud-db-backup-credentials.sops.yaml
 - nextcloud-db.yaml
 - pihole-bootstrap-secret.sops.yaml
 - pihole-dns.yaml
 - pihole-oauth2-proxy-credentials.sops.yaml
 - pihole.yaml
 - plex-claim-secret.sops.yaml
 - plex-remote.yaml
 - plex.yaml
 - qbittorrent-bootstrap-secret.sops.yaml
 - qbittorrent-oauth2-proxy-credentials.sops.yaml
@@ -1,72 +0,0 @@
 # =================================================================================================
 # PiHole WEB/API ADMIN PASSWORD — the one piece of PiHole state that was NOT declared anywhere.
 # =================================================================================================
 # WHY THIS FILE EXISTS. Measured on live prod 2026-10-04, `pihole.toml` marks every non-default
 # setting with `### CHANGED`, and names the ones that came from the environment `### CHANGED (env)`.
 # The complete list was:
 #
 #     dns.upstreams          ### CHANGED (env)   <- declared, catalog/pihole.yaml
 #     dns.listeningMode      ### CHANGED (env)   <- declared, catalog/pihole.yaml
 #     dns.rateLimit.count    ### CHANGED (env)   <- declared, catalog/pihole.yaml
 #     misc.dnsmasq_lines     ### CHANGED (env)   <- declared, catalog/pihole.yaml
 #     files.macvendor        ### CHANGED         <- set by the image itself, container path
 #     webserver.api.pwhash   ### CHANGED         <- SET OUT OF BAND. Declared NOWHERE. This file.
 #
 # So exactly ONE operator-set value lived only on the Longhorn PVC: the admin password, as a
 # one-way BALLOON-SHA256 hash. Nobody could recover it (a hash does not reverse), nothing in git
 # produced it, and a rebuild from git would have come up with a DIFFERENT password — or none.
 #
 # THE REASON THIS BLOCKS HA, which is why it is fixed first. The next step is running more than one
 # PiHole replica, and a replica with a fresh PVC would come up with its own unrelated password. The
 # admin UI would then accept or reject the same credential depending on which pod answered. Declaring
 # it here makes every replica, and every rebuild, identical by construction.
 #
 # MECHANISM: `webserver.api.password` is a real, settable FTL config key — verified in the running
 # image, where `pihole setpassword` is itself just `setFTLConfigValue "webserver.api.password"` with
 # the comment "pihole-FTL will automatically hash the password". So FTL takes CLEARTEXT on this key
 # and writes the hash into `webserver.api.pwhash` itself. catalog/pihole.yaml therefore passes this
 # value as `FTLCONF_webserver_api_password`, and the hash on the PVC becomes a derived artefact
 # rather than the source of truth.
 #
 # DEFENCE IN DEPTH, NOT THE ONLY GATE. The admin UI is already behind the oauth2-proxy sidecar
 # (`--allowed-group=platform-admins`), so this password is a SECOND gate. It is kept rather than set
 # empty because the `pihole` Service still publishes port 80 (PiHole direct, bypassing the proxy on
 # :4180) and only the CiliumNetworkPolicy closes it — and the default-deny policy in this repo has a
 # documented drift history. An unauthenticated admin UI that can rewrite LAN DNS is not a good thing
 # to leave resting on one policy staying correct. Narrowing that Service port is tracked follow-up.
 #
 # ROTATION: `sops` this file in your own terminal, change the value, commit. Flux re-applies the
 # Secret, the pod restarts, FTL re-hashes. Do NOT `sops -d` it into a shared terminal or a log
 # (CLAUDE.md section 6); read it in an editor. The value below was generated randomly at authoring
 # time and has never been printed to a transcript.
 apiVersion: v1
 kind: Secret
 metadata:
    name: pihole-bootstrap
    namespace: pihole
    labels:
        app.kubernetes.io/name: pihole
 type: Opaque
 stringData:
    #ENC[AES256_GCM,data:TMOuRyJeb5IVn3lZPXY7lgh1KGxu7GY9+WIOLHpMB50JqzmM2tHIaWUWKTt1JtZoX/E5/Y7WCUTnfkJLhRBN2zV2MwWRBSLYVy5CMg7xV8PTLRZJYAVUYS5Trg==,iv:q+DpfT6PQfY5ZuNsmP7Ddl+PDLjSm89fJICAhYoW64k=,tag:axijaO3DaxIZgZ3LeK2hXA==,type:comment]
    webui-password: ENC[AES256_GCM,data:7FcK/bUW79M46f/MlouAmATKbMnoUTNuMUjtqJ5vpU4=,iv:I8DBFB39uXZV42Ie2IPwSefmMgdKEtiDknfc9Tu7h5E=,tag:L423784UrDU3TvylpHc6KQ==,type:str]
 sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
        - recipient: age175l8mdpdfu7hq0aml3exs3gykarwjmuvr23sq56s3wjejnyk44wsp4mfzg
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0MkRSNjZLZlpXYkcvaVJL
            cjlqb2wrcWREZnZLdGpsTHNPWEpseHlRZnpjCnJVM1FDQ2M2dXlaS1JBZ2k2V1BP
            Yi9QUDZSMzZsL09pTi93RTdjdGxtY2MKLS0tIDIrQTZIVWdhMGIxSCtlUFN2Mlho
            SWVybld4VjJyaHRSKzZZU0xsUUV2SEEKbDYHV/qBZVspK7DpBW3vYjRMJ/HmAN+9
            G9bOlNEp0jAtcNZEn6lpQ9ORgvmIrhJHoBQRLabdMWzNWSZsNn56tA==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2026-10-04T06:01:27Z"
    mac: ENC[AES256_GCM,data:rhTMTOwLOtHXHgqI5M2ZW/6w+UpLpxBlTk7NoN0wmGSqOcE+OGrXA1DhkZrJweyQT0RSNSSBm/6QMztLP81eepOzdp2rEkwvbQPK/l6K9g16Ww9c1IatS3NQsKEB8AoMDx7jBQ6exXYgggPx3MJsNYbE+WyloQ96U9KPdrecAM8=,iv:n34v2zX9D5xsaR1PFCH94mNMiuRfhUyI2lseUCoqiTA=,tag:9YmN9nIVLXfsGYIb0bUl2w==,type:str]
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.9.4
@@ -90,21 +90,9 @@ spec:
          value: '0'
        - name: FTLCONF_misc_dnsmasq_lines
          value: address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}
        - name: FTLCONF_webserver_api_password
          valueFrom:
            secretKeyRef:
              name: pihole-bootstrap
              key: webui-password
        volumeMounts:
        - name: config
          mountPath: /data/config
        lifecycle:
          postStart:
            exec:
              command:
              - /bin/bash
              - -c
              - "# Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and\n# does NOT run a gravity rebuild.\nset -u\nGRAVITY=/etc/pihole/gravity.db\n\n# THE DECLARED SET \u2014 one URL per line, the source of truth for what PiHole blocks.\n# To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.\nDECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'\n\n# On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,\n# and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.\ni=0\nwhile [ ! -s \"$GRAVITY\" ] && [ \"$i\" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done\nif [ ! -s \"$GRAVITY\" ]; then\n  echo \"pihole-adlists: gravity.db absent after 180s - leaving adlists untouched\" >&2\n  exit 0\nfi\n\nsql() { pihole-FTL sqlite3 \"$GRAVITY\" \"$1\"; }\nCHANGED=0\n\n# 1) Every declared URL must be present AND enabled.\necho \"$DECLARED\" | while IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  printf '%s\\n' \"$url\"\ndone > /tmp/declared.txt\nwhile IFS= read -r url; do\n  [ -n \"$url\" ] || continue\n  # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.\n  have=$(sql \"select count(*) from adlist where address='$url';\")\n  if [ \"$have\" = \"0\" ]; then\n    echo \"pihole-adlists: adding $url\"\n    sql \"insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');\"\n    CHANGED=1\n  else\n    en=$(sql \"select enabled from adlist where address='$url';\")\n    if [ \"$en\" != \"1\" ]; then\n      echo \"pihole-adlists: re-enabling $url\"\n      sql \"update adlist set enabled=1 where address='$url';\"\n      CHANGED=1\n    fi\n  fi\ndone < /tmp/declared.txt\n\n# 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).\nsql \"select address from adlist where enabled=1;\" | while IFS= read -r addr; do\n  [ -n \"$addr\" ] || continue\n  if ! grep -qxF \"$addr\" /tmp/declared.txt; then\n    echo \"pihole-adlists: disabling undeclared $addr\"\n    sql \"update adlist set enabled=0 where address='$addr';\"\n    echo changed >> /tmp/adlist-changed\n  fi\ndone\n[ -s /tmp/adlist-changed ] && CHANGED=1\nrm -f /tmp/adlist-changed /tmp/declared.txt\n\n# 3) Rebuild gravity ONLY if the declared set actually moved.\nif [ \"$CHANGED\" = \"1\" ]; then\n  echo \"pihole-adlists: declared set changed - running gravity update\"\n  pihole -g || echo \"pihole-adlists: gravity update failed; existing gravity.db retained\" >&2\nelse\n  echo \"pihole-adlists: already matches the declared set - nothing to do\"\nfi\nexit 0\n"
      - name: oauth2-proxy
        image: quay.io/oauth2-proxy/oauth2-proxy:v7.8.1
        imagePullPolicy: IfNotPresent
@@ -1,126 +0,0 @@
 ---
 # =================================================================================================
 # PLEX REMOTE ACCESS — raw tcp/32400 north-south exposure, and the policy that admits it.
 # =================================================================================================
 # HAND-WRITTEN, NOT RENDERED. A non-catalog app-tier unit anchored to the `plex` ServiceDefinition
 # and placed into wave 0 by `catalog/render.py --emit-waves` via its EXTRA_UNITS table — the same
 # mechanism gitea-actions and pihole-dns use. It is hand-written for the SAME reason those are:
 # schemas/servicedefinition.schema.json is `additionalProperties: false` and has NO field for a
 # type=LoadBalancer Service, for a pinned lbipam VIP, or for raw-TCP north-south exposure at all.
 # `ingressRoutes` renders HTTPRoutes on the shared Gateway, which is an L7 HTTPS terminator.
 #
 # WHY THE L7 GATEWAY IS NOT ENOUGH HERE — and it is NOT because the Gateway fails to serve Plex.
 # Measured 2026-10-03 through the Gateway VIP: `GET https://plex.home.keithjackson.dev/identity`
 # returned HTTP 200 with `claimed="1"` and the real machineIdentifier, and the protected endpoints
 # (/library/sections, /status/sessions, /:/prefs) all returned 401. The L7 path WORKS.
 #
 # The problem is DISCOVERY, not proxying. Plex apps (phone, TV, Roku) do not resolve
 # `plex.${BASE_DOMAIN}`; they ask plex.tv for the server's connection list. A server gets onto that
 # list one of two ways: it advertises a custom URL (`customConnections`, the ADVERTISE_IP env), or it
 # publishes a reachable public IP:port for its own tcp/32400. Measured in the live Preferences.xml on
 # 2026-10-03: `customConnections` ABSENT, `ManualPortMappingMode` ABSENT, while
 # `PublishServerOnPlexOnlineKey="1"` — the server was trying to publish and had nothing reachable to
 # publish, so remote clients saw NO connection at all. The operator chose the NATIVE path (this
 # file) over advertising the Gateway URL: Plex then terminates its own TLS with its
 # `*.plex.direct` certificate and video streams bypass the Envoy L7 hop entirely.
 #
 # VIP: ${PLEX_L4_VIP} (cluster-vars, R-ENV-02 — dev 192.168.1.57, prod 192.168.2.24), PINNED via
 # lbipam.cilium.io/ips out of the CiliumLoadBalancerIPPool `default-pool`
 # (${CILIUM_LB_POOL_START}..${CILIUM_LB_POOL_STOP}) and ARP-announced on the flat LAN by the existing
 # CiliumL2AnnouncementPolicy `default-l2` (platform-config/cilium/*). No new Cilium config is needed;
 # this Service just consumes the pool. PINNING IS NOT OPTIONAL: the address is typed into a WAN
 # port-forward on the SOHO router, which is hand-maintained and off-repo, so it can never move.
 # 192.168.2.24 verified FREE live 2026-10-03 — `default-pool` reported IPsTotal 11 / IPsUsed 4 /
 # IPsAvailable 7, and the only four allocations were .20 (cilium-gateway-shared), .21 (unifi-l4-*),
 # .22 (crafty-l4-*) and .23 (pihole-dns-*).
 #
 # ONE SERVICE, TCP ONLY. Unlike unifi/crafty/pihole-dns this needs no second Service and no
 # lbipam.cilium.io/sharing-key: Plex's remote access is tcp/32400 and nothing else. (Plex's
 # DLNA/GDM discovery uses UDP, but that is LAN-only multicast and is not part of remote access.)
 #
 # NO `loadBalancerSourceRanges`, DELIBERATELY — and this is the one place this file diverges from
 # its closest sibling. apps/wave-0/pihole-dns.yaml pins its source range to ${LAN_CIDR} because a
 # resolver must never answer the internet. This Service is the OPPOSITE case: admitting remote
 # clients from arbitrary internet addresses is its entire purpose, so there is no range to narrow
 # to. Plex authenticates every caller itself (measured above: 401 on every protected endpoint
 # without a token), which is what makes an unfiltered port defensible here.
 #
 # NO external-dns ANNOTATION, DELIBERATELY. unifi and crafty both carry
 # `external-dns.alpha.kubernetes.io/hostname` so external-dns publishes their VIP as a public
 # Cloudflare record. This Service must not, for pihole-dns's reason: ${PLEX_L4_VIP} is an RFC1918
 # address that is useless in public DNS. Remote clients get this server's address from plex.tv, not
 # from the zone. The existing public `plex.${BASE_DOMAIN}` record is published from the rendered
 # HTTPRoute via the gateway source and is NOT affected by this file — the web UI keeps working
 # through the Gateway exactly as before.
 #
 # -------------------------------------------------------------------------------------------------
 # externalTrafficPolicy: Cluster — inherited from crafty's 8-DAY LAN OUTAGE, not re-derived.
 # -------------------------------------------------------------------------------------------------
 # `Local` is the tempting choice because it preserves the client source IP, which this service
 # actually has a use for (see the allowedNetworks note below). It is still WRONG here, for the
 # reason platform/crafty/service-l4.yaml documents at length: Cilium's L2 announcer elects ONE lease
 # holder per Service and does NOT follow the backend pod, so with `Local` the VIP can be announced
 # by a node that has no Plex pod, which black-holes the port while every object still reports
 # healthy. Plex is a single-replica Deployment with `strategy: Recreate` pinned to wherever the
 # scheduler put it, so the mismatch is the normal case, not a rare one. Take the SNAT.
 #
 # KNOWN CONSEQUENCE OF THAT CHOICE, stated rather than discovered later: `Cluster` SNATs the client
 # to the ingress NODE's address (192.168.2.1x), which falls inside the `192.168.0.0/16` entry of
 # catalog/plex.yaml's ALLOWED_NETWORKS. Plex will therefore classify every REMOTE client as LAN.
 # Measured 2026-10-03 on the Gateway path, which SNATs the same way: Plex logged the request as
 # `[10.0.2.214:36040 (Allowed Network (WAN))]`. This is a METERING effect, not a security one —
 # authorization is unchanged (401s above), but Plex's remote-quality/bandwidth limits will not bind
 # on these streams because it believes they are local. Narrowing ALLOWED_NETWORKS cannot fix it
 # without also de-classifying genuine LAN clients, so it is accepted.
 apiVersion: v1
 kind: Service
 metadata:
  name: plex-l4-tcp
  namespace: plex
  labels:
    app.kubernetes.io/name: plex
  annotations:
    lbipam.cilium.io/ips: "${PLEX_L4_VIP}"
 spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  selector:
    app.kubernetes.io/name: plex
  ports:
    - name: plex-remote
      port: 32400
      targetPort: 32400
      protocol: TCP
 ---
 # INGRESS allow for the raw port (R-NET-01/02).
 #
 # The rendered plex.yaml already carries `plex-allow-gateway-ingress` (fromEntities: ingress ->
 # 32400) and three reciprocal allows for sonarr/radarr/seerr. CiliumNetworkPolicies are ADDITIVE, so
 # this file adds the off-cluster case without touching those.
 #
 # `world` is Cilium's entity for anything outside the cluster, which is what a remote Plex client is.
 # `host` and `cluster` mirror platform/crafty/networkpolicy.yaml's game-port rule and are what make
 # this work under `externalTrafficPolicy: Cluster`: after SNAT the packet's source is the ingress
 # NODE, which Cilium classifies as `host` (same node) or `remote-node` (a different one) — NOT
 # `world`. Listing `world` alone would compile cleanly and then drop every remote stream, so all
 # four entities are named.
 apiVersion: cilium.io/v2
 kind: CiliumNetworkPolicy
 metadata:
  name: plex-allow-remote-32400
  namespace: plex
  labels:
    app.kubernetes.io/name: plex
 spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: plex
  ingress:
    - fromEntities:
        - world
        - cluster
        - host
        - remote-node
      toPorts:
        - ports:
            - { port: "32400", protocol: TCP }
@@ -100,20 +100,6 @@ env:
  # while resolution for the zone does not work. Confirmed both ways against the live image.
  - name: FTLCONF_misc_dnsmasq_lines
    value: "address=/${BASE_DOMAIN}/${GATEWAY_VIP};address=/unifi.${BASE_DOMAIN}/${UNIFI_L4_VIP};address=/crafty-mc.${BASE_DOMAIN}/${CRAFTY_L4_VIP};address=/pihole-dns.${BASE_DOMAIN}/${PIHOLE_DNS_VIP}"
  # THE ADMIN PASSWORD, previously the ONE operator-set value declared nowhere. Sourced from
  # apps/wave-0/pihole-bootstrap-secret.sops.yaml (SOPS/age, same wave => same reconcile), which
  # carries the full rationale and the measurement that found it.
  #
  # `webserver.api.password` takes CLEARTEXT and FTL hashes it itself into `webserver.api.pwhash`.
  # Verified in the running image, not assumed: `pihole setpassword` is itself
  # `setFTLConfigValue "webserver.api.password" "$PASSWORD"`, commented upstream as "pihole-FTL will
  # automatically hash the password". So the BALLOON-SHA256 hash on the PVC is now a DERIVED artefact
  # and this is its source of truth. Rotating the Secret rotates the password.
  - name: FTLCONF_webserver_api_password
    valueFrom:
      secretKeyRef:
        name: pihole-bootstrap
        key: webui-password
 ingressRoutes:
  - host: pihole.${BASE_DOMAIN}
    port: 80                  # admin web UI, protected at the Gateway by forward-auth (docs/09)
@@ -128,110 +114,6 @@ auth:
  # rather than a renderer constant: the measurement that justifies the flag on two apps is
  # evidence about those two apps. Its oauth2-proxy log holds 0 unauthenticated requests across
  # the current pod's whole 23h lifetime (measured 2026-10-01), so there is nothing to act on.
 lifecycle:
  # LEVEL-TRIGGERED ADLIST ASSERTION, run on EVERY container start. Same pattern, and the same
  # reasoning, as catalog/qbittorrent.yaml's postStart: desired state lives in git and is
  # RE-ASSERTED unconditionally rather than set once by hand.
  #
  # WHY THIS IS NEEDED AT ALL — the blocklists were NOT reproducible from git. Measured on live prod
  # 2026-10-04: `gravity.db` held exactly ONE adlist (StevenBlack's hosts, enabled) and 72,234
  # gravity domains, and NOTHING in this repo named it. Worse, a FRESH database does not seed it:
  # the image's own schema template `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the
  # `adlist` table and inserts NO rows into it (grepped in the running image — there is no
  # `INSERT INTO adlist` at all). So a rebuild from git, or a replica with a new PVC, would come up
  # SERVING DNS AND BLOCKING NOTHING, silently and with every object reporting healthy.
  #
  # WHY A postStart HOOK AND NOT A FILE. PiHole v6 keeps adlists as DB rows, not config. The
  # `/etc/pihole/adlists.list` file still exists in `gravity.sh` but is MIGRATION-ONLY: its import
  # sits inside `migrate_to_database()`, guarded by `if [ ! -e "$gravityDBfile" ]`, so it fires only
  # while CREATING a database and never again on an existing one. A ConfigMap mounted there would
  # therefore be silently ignored on every pod this cluster currently runs. There is also no
  # `pihole adlist` CLI subcommand (only allow/deny for domains), so SQL against the DB is the
  # actual interface. FTL ships an embedded shell (`pihole-FTL sqlite3`, SQLite 3.53.1) so no extra
  # tooling is needed in the image.
  #
  # IT RECONCILES, IT DOES NOT JUST ADD: adlists present in the DB but absent from DECLARED below are
  # DISABLED (not deleted — deleting cascades to `gravity` rows and would force a full re-download of
  # everything on a typo). So this list is the source of truth in both directions, and `pihole -g`
  # runs ONLY when something actually changed, which keeps the steady-state start free of a 70k-domain
  # rebuild.
  #
  # NO BRACED DOLLAR EXPANSION ANYWHERE IN THIS SCRIPT, DELIBERATELY — the same contract
  # catalog/qbittorrent.yaml documents at length. apps-wave-0 declares postBuild.substituteFrom
  # cluster-vars, so Flux runs envsubst over this whole manifest at reconcile time; a `${VAR}` form
  # would behave differently on the cluster than under a plain `kubectl apply` of the golden. Bare
  # `$VAR` and `$(cmd)` are untouched by envsubst and so behave identically however this is run.
  # `render.py`'s assert_substitution_safe refuses to emit a manifest that breaks this.
  postStart:
    exec:
      command:
        - /bin/bash
        - -c
        - |
          # Assert the declared adlists in gravity.db. Idempotent: steady state writes nothing and
          # does NOT run a gravity rebuild.
          set -u
          GRAVITY=/etc/pihole/gravity.db
          # THE DECLARED SET — one URL per line, the source of truth for what PiHole blocks.
          # To add a list: add a line here, commit, let Flux reconcile. Nothing else to do.
          DECLARED='https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts'
          # On a fresh PVC, FTL creates gravity.db a moment after the container starts. Wait, bounded,
          # and NEVER block or fail the start: a missing DB here must not crashloop DNS for the LAN.
          i=0
          while [ ! -s "$GRAVITY" ] && [ "$i" -lt 90 ]; do sleep 2; i=$(expr $i + 1); done
          if [ ! -s "$GRAVITY" ]; then
            echo "pihole-adlists: gravity.db absent after 180s - leaving adlists untouched" >&2
            exit 0
          fi
          sql() { pihole-FTL sqlite3 "$GRAVITY" "$1"; }
          CHANGED=0
          # 1) Every declared URL must be present AND enabled.
          echo "$DECLARED" | while IFS= read -r url; do
            [ -n "$url" ] || continue
            printf '%s\n' "$url"
          done > /tmp/declared.txt
          while IFS= read -r url; do
            [ -n "$url" ] || continue
            # Single quotes inside a URL would break this SQL; adlist URLs do not contain them.
            have=$(sql "select count(*) from adlist where address='$url';")
            if [ "$have" = "0" ]; then
              echo "pihole-adlists: adding $url"
              sql "insert into adlist (address,enabled,comment) values ('$url',1,'declared in catalog/pihole.yaml');"
              CHANGED=1
            else
              en=$(sql "select enabled from adlist where address='$url';")
              if [ "$en" != "1" ]; then
                echo "pihole-adlists: re-enabling $url"
                sql "update adlist set enabled=1 where address='$url';"
                CHANGED=1
              fi
            fi
          done < /tmp/declared.txt
          # 2) Anything enabled in the DB but NOT declared gets disabled (never deleted - see above).
          sql "select address from adlist where enabled=1;" | while IFS= read -r addr; do
            [ -n "$addr" ] || continue
            if ! grep -qxF "$addr" /tmp/declared.txt; then
              echo "pihole-adlists: disabling undeclared $addr"
              sql "update adlist set enabled=0 where address='$addr';"
              echo changed >> /tmp/adlist-changed
            fi
          done
          [ -s /tmp/adlist-changed ] && CHANGED=1
          rm -f /tmp/adlist-changed /tmp/declared.txt
          # 3) Rebuild gravity ONLY if the declared set actually moved.
          if [ "$CHANGED" = "1" ]; then
            echo "pihole-adlists: declared set changed - running gravity update"
            pihole -g || echo "pihole-adlists: gravity update failed; existing gravity.db retained" >&2
          else
            echo "pihole-adlists: already matches the declared set - nothing to do"
          fi
          exit 0
 update:
  channel: stable             # R-UPD-01
  range: ">=2024.0.0"
@@ -2002,24 +2002,6 @@ EXTRA_UNITS: dict = {
            "pihole-dns.yaml",
        ],
    },
    # Plex's remote-access exposure: the raw tcp/32400 LoadBalancer Service on a pinned VIP, plus the
    # CiliumNetworkPolicy that admits off-cluster clients to it (apps/wave-0/plex-remote.yaml).
    # `above: False` co-locates it in plex's OWN wave — these are the plex pod's own Service and its
    # own policy, applied in the same wave as the Deployment they select, exactly as the rendered
    # Service and CNPs in plex.yaml are. Nothing in a higher wave gates on it.
    #
    # It is a non-catalog unit for the same reason pihole-dns is: the ServiceDefinition schema cannot
    # express a LoadBalancer Service or a pinned lbipam VIP (additionalProperties is false, and
    # `ingressRoutes` only renders HTTPRoutes on the L7 Gateway, which cannot carry Plex's raw port).
    # Plex's own settings stay in the catalog as env on catalog/plex.yaml; the only thing living
    # outside it is the exposure the schema has no word for.
    "plex-remote": {
        "anchor": "plex",
        "above": False,
        "files": [
            "plex-remote.yaml",
        ],
    },
 }
@@ -100,8 +100,7 @@ data:
  # platform/keycloak/postgres-cluster.yaml but is rendered by platform-config, which is the ONLY
  # tier that lists it), crafty (CRAFTY_L4_VIP), apps-wave-0 (nextcloud-db + semaphore-db:
  # S3_BACKUP_ENDPOINT, CNPG_BACKUP_PATH; pihole + pihole-dns: PIHOLE_DNS_VIP, LAN_CIDR, plus
-  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers;
+  # GATEWAY_VIP/UNIFI_L4_VIP/CRAFTY_L4_VIP read BY pihole to author its LAN split-horizon answers),
  # plex-remote: PLEX_L4_VIP),
  # velero (S3_BACKUP_ENDPOINT, S3_BACKUP_REGION,
  # VELERO_BACKUP_BUCKET, VELERO_BACKUP_PREFIX), apps-vms (UNIFI_L4_VIP). A ${TOKEN} in an
  # UNSUBSTITUTED path would ship the literal string into the cluster.
@@ -176,19 +175,6 @@ data:
  # dev  -> 192.168.1.56  (base default; next free after crafty .55 in dev's .53-.60 pool)
  # prod -> 192.168.2.23  (next free after crafty .22 in prod's .20-.30 pool; see clusters/prod)
  PIHOLE_DNS_VIP: "192.168.1.56"             # PiHole: authoritative LAN DNS for ${BASE_DOMAIN}
  # Plex's remote-access listener: raw tcp/32400 for REMOTE Plex clients (apps/wave-0/plex-remote.yaml).
  # A FOURTH pinned L4 VIP, for the usual reason — the `shared` Gateway is an L7 HTTPS terminator and
  # cannot carry Plex's own TLS. Plex clients do not resolve `plex.${BASE_DOMAIN}`; they fetch this
  # server's address from plex.tv, so the Gateway's L7 path (which still serves the web UI, and was
  # measured working) cannot put the server on a client's connection list at all.
  #
  # PINNED, AND THE PIN IS LOAD-BEARING OFF-REPO: the SOHO router forwards WAN tcp/32400 to this
  # address by hand. A VIP taken from the pool head would silently move on a reschedule and break
  # remote access with nothing in-cluster looking wrong. See runbooks/plex-remote-access.md.
  #
  # dev  -> 192.168.1.57  (base default; next free after pihole-dns .56 in dev's .53-.60 pool)
  # prod -> 192.168.2.24  (next free after pihole-dns .23 in prod's .20-.30 pool; see clusters/prod)
  PLEX_L4_VIP: "192.168.1.57"                # Plex: tcp/32400 remote access (NOT the web UI, which is L7)
  # The FLAT LAN this cluster's nodes and LB VIPs live on, in CIDR form. Consumed as the ingress
  # peer of the PiHole DNS CiliumNetworkPolicy (apps/wave-0/pihole-dns.yaml): a resolver reachable
@@ -476,26 +476,6 @@ patches:
      - op: replace
        path: /data/PIHOLE_DNS_VIP
        value: "192.168.2.23"
      # Plex's remote-access listener VIP — raw tcp/32400 for REMOTE Plex clients
      # (apps/wave-0/plex-remote.yaml). 192.168.2.24 is the NEXT FREE address in the prod pool
      # 192.168.2.20-.30, verified live 2026-10-03: the CiliumLoadBalancerIPPool `default-pool`
      # reported IPsTotal 11 / IPsUsed 4 / IPsAvailable 7, and the only four allocations are .20
      # (cilium-gateway-shared), .21 (unifi-l4-tcp/-udp), .22 (crafty-l4-tcp/-udp) and .23
      # (pihole-dns-tcp/-udp). It is inside the pool block the operator already asserted sits outside
      # the SOHO router's DHCP scope, and it is not a node address (.10/.11/.12) or the
      # control-plane VIP (.9).
      #
      # UNLIKE .23, THIS ONE IS ALSO A WAN PORT-FORWARD TARGET. The router forwards WAN tcp/32400
      # here, by hand, off-repo — so changing this value is a two-place edit and the router is the
      # half that CI cannot check. runbooks/plex-remote-access.md is the procedure.
      #
      # NOT ARP-PROBED. .23 was verified free with `arping` from a LAN host before it was claimed;
      # for .24 only the in-cluster pool accounting above was checked, so an off-cluster squatter on
      # this address would not have been caught. Probe before trusting it if remote access comes up
      # with a conflict.
      - op: replace
        path: /data/PLEX_L4_VIP
        value: "192.168.2.24"
      # The prod flat LAN. This is the 192.168.2.0/24 segment the SOHO ROUTER at 192.168.2.1 owns —
      # NOT pfSense's 192.168.1.0/24, which is a different network and not in the LAN-DNS path.
      # It scopes the ingress allow for PiHole's DNS listener to LAN clients only.
@@ -0,0 +1,76 @@
 ---
 date: 2026-10-03
 branch: fix/cilium-cgroup-hostroot
 title: Cilium socket-LB never covered host processes — cgroup.hostRoot pointed at an unmounted path
 ---
 ## What changed
 `cgroup.hostRoot` changes from `/run/cilium/cgroupv2` to `/sys/fs/cgroup` in all three parity
 copies: `platform/cilium/helmrelease.yaml` (authoritative), `platform/cilium/values.yaml`
 (mirror) and `infra/ansible/roles/cilium/templates/cilium-values.yaml.j2` (bootstrap).
 `autoMount` stays `false`.
 **The defect.** With `autoMount: false`, the Cilium chart only bind-mounts whatever exists at
 `hostRoot`, and expects the operator to have mounted cgroup2 there (cilium 1.20.1 values.yaml).
 The repo's rationale was "k3s provides the cgroupv2 mount". That is true only for
 `/sys/fs/cgroup`. Nothing mounts cgroup2 at `/run/cilium/cgroupv2` on these nodes; it is an
 empty `DirectoryOrCreate` hostPath. So the agent mounted cgroup2 there itself, from inside its
 own private cgroup namespace, and that mount's root is the agent container's own leaf cgroup.
 The socket-LB programs (`cil_sock4_connect` etc.) were attached to it and covered only the
 agent.
 **The consequence.** Every other host-netns process got no Service/LB translation:
 - Service ClusterIPs were unreachable from every node's host.
 - The node holding the Gateway's L2 lease could not reach the Gateway VIP (`no route to host`).
  Other nodes reached it only because their packets crossed the wire to the lease holder, where
  the tc datapath handles them.
 - k3s-prod-1 holds both the Gateway lease and the kube-vip API VIP. Its apiserver could not reach
  the OIDC issuer, logging `oidc: authenticator not initialized` since at least 2026-09-12
  (899 occurrences by 2026-10-03). kubectl SSO through the VIP therefore failed, and in-cluster
  OIDC clients failed on whichever calls landed on that apiserver.
 - Webhooks and aggregated APIs were unaffected only because k3s runs
  `--enable-aggregator-routing=true` (endpoint IPs, not ClusterIPs).
 Cilium's own source (v1.20.1 `bpf/bpf_sock.c`, `__sock4_xlate_fwd`) handles exactly this case
 when the hook does run: a host-netns connect to an L7-LB Service (the Gateway) is rewritten to
 `127.0.0.1:<l7_lb_proxy_port>`, the local Envoy listener. The fix is to make the hook cover the
 host.
 ## What I verified
 - `cilium-dbg status --verbose` on k3s-prod-1: `Socket LB: Enabled`, `Coverage: Hostns-only`.
  The live cilium-config shows `bpf-lb-sock=false`, but socket LB is implied by kube-proxy
  replacement. The Gateway Service frontends `192.168.2.20:{80,443}` and
  `10.43.225.79:{80,443}` map to `127.0.0.1:14337`.
 - Host: `findmnt /run/cilium/cgroupv2` is empty on all three nodes, and `/sys/fs/cgroup` is
  `cgroup2`.
 - In the agent: `/run/cilium/cgroupv2` is a `cgroup2 none` mount, the agent's own cgroup reads
  `0::/`, the mount has 0 child cgroups, and `bpftool cgroup show /run/cilium/cgroupv2` lists all
  `cil_sock*` programs. k3s runs in `0::/system.slice/k3s.service`, and the agent container in
  `/kubepods.slice/…/cri-containerd-3ad04cde….scope`.
 - **The decisive test**, run on k3s-prod-1 in the same netns: raw TCP connect to
  `192.168.2.20:443`, `10.43.225.79:443` and `10.43.59.141:8080` CONNECTED from inside the agent
  container (covered cgroup) and FAILED from the host (`No route to host` / timeout).
 - Blast radius: host-network pods are cilium, cilium-envoy, cilium-operator (API via
  `192.168.2.9:6443`, not a Service), kube-vip (`127.0.0.1:6443`, `svc_enable=false`) and
  node-exporter (no outbound). Nothing depends on host Service translation being absent; the
  change only makes Service/LB addresses reachable from the host where they currently fail.
 - `helm template cilium/cilium --version 1.20.1` with the new values, in the toolbox: the
  DaemonSet bind-mounts hostPath `/sys/fs/cgroup` into `cilium-agent` and `clean-cilium-state`,
  the cilium-config shows `cgroup-root: "/sys/fs/cgroup"`, and the init containers are unchanged
  (no `mount-cgroup`).
 - Toolbox gate: see the PR description.
 ## What I did NOT verify
 - **The rollout itself.** After merge, the agent DaemonSet rolls one node at a time
  (`maxUnavailable: 1`). On each node, afterwards: host TCP connects to the three targets above
  must succeed. On k3s-prod-1, the apiserver's background OIDC init retry must stop logging
  `authenticator not initialized` with no k3s restart.
 - That the old programs, attached to the previous agent container's cgroup, disappear with it.
  Expected, because cgroup BPF attachments die with the cgroup, but not observed.
 - An end-to-end kubectl OIDC login (needs a browser).
 - No regression test was added. A natural one is an L2 conformance check that a host-netns
  connect to the Gateway VIP succeeds on the lease holder, but `conformance/**` is a sensitive
  path. Left as a follow-up.
@@ -1,129 +0,0 @@
 ---
 date: 2026-10-03
 branch: feat/plex-remote-access-l4
 title: Plex remote access — raw tcp/32400 via a pinned LoadBalancer VIP, not the L7 Gateway
 ---
 ## What changed
 Reported symptom: users cannot reach Plex from outside the LAN. **Two independent causes**, measured
 rather than inferred:
 1. **Plex advertised no remote connection at all.** Live `Preferences.xml` on prod:
   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
   `PublishServerOnPlexOnlineKey="1"` — publishing was on and there was nothing reachable to publish.
   Plex's automatic UPnP/NAT-PMP path cannot work from a pod on the Cilium overlay.
 2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` refused;
   `136.51.115.252:443` answered with a **self-signed router cert** (`C=ZZ, CN=%D`), not the
   Gateway's LE wildcard. The Plex Service was `ClusterIP` only — no off-cluster path to 32400 by
   any route.
 This branch fixes the in-cluster half. **Remote access does not work until the operator also does
 two manual steps** (`runbooks/plex-remote-access.md`): the WAN tcp/32400 port-forward, and ticking
 "manually specify public port" in the Plex UI.
 - **`apps/wave-0/plex-remote.yaml` (new, hand-written).** A `type=LoadBalancer` Service on tcp/32400
  pinned to `${PLEX_L4_VIP}` via `lbipam.cilium.io/ips`, plus a `CiliumNetworkPolicy`
  (`plex-allow-remote-32400`) admitting `world`/`cluster`/`host`/`remote-node` to 32400. Non-catalog
  because `schemas/servicedefinition.schema.json` is `additionalProperties: false` and has no field
  for a LoadBalancer Service or a pinned VIP — the same reason `apps/wave-0/pihole-dns.yaml`,
  `platform/crafty/service-l4.yaml` and `apps/vms/unifi/50-service-l4.yaml` are hand-written.
 - **`catalog/render.py`:** an `EXTRA_UNITS` entry (`plex-remote`, anchored to `plex`, `above: False`)
  so `--emit-waves` places the unit in plex's own wave.
 - **`clusters/base/cluster-vars.yaml`:** new `PLEX_L4_VIP` (dev default `192.168.1.57`), plus the
  substitution inventory comment updated to name `plex-remote: PLEX_L4_VIP`.
 - **`clusters/prod/kustomization.yaml`:** prod override `PLEX_L4_VIP: 192.168.2.24`.
 - **`apps/wave-0/kustomization.yaml`:** `plex-remote.yaml` added to `resources`.
 - **`runbooks/plex-remote-access.md` (new):** the two manual steps, the four-step verification, why
  no alert is possible, and the two documented warts below.
 **`ADVERTISE_IP` was deliberately NOT set.** Setting it to `https://plex.home.keithjackson.dev:443`
 was the alternative — it needs no new WAN hole and the mechanism was verified working on an
 already-claimed server. **The operator chose the native path (2026-10-03)** so Plex terminates its own
 TLS (`*.plex.direct`) and streams bypass the Envoy L7 hop. `catalog/plex.yaml`'s deferral note for
 `ADVERTISE_IP` remains accurate and untouched.
 **`externalTrafficPolicy: Cluster`, with a known cost.** `Local` would preserve the client IP, but
 Cilium's L2 announcer does not follow the backend pod — the wart that cost this LAN an 8-day
 Minecraft outage (see `platform/crafty/service-l4.yaml`). Consequence accepted and documented: SNAT
 puts the client in `192.168.0.0/16`, inside `ALLOWED_NETWORKS`, so **Plex classifies remote streams
 as LAN**. That is a metering effect (remote bandwidth limits will not bind), not an authorization one.
 **SENSITIVE PATHS — this PR touches `clusters/**` (two files), so `ci/approval-policy.sh` will BLOCK
 it.** It needs the operator's `/approve-override <reason>` posted as a PR **review** of type
 "Comment". I have deliberately not posted it: CLAUDE.md §1 forbids an agent clearing this on its own
 PR, and the agent account holds `owner`, so the prohibition is a rule, not a limit.
 ## What I verified
 All measurements against live prod, 2026-10-03, read-only.
 - **Plex's advertised state**, the finding this branch rests on: `customConnections=<ABSENT>`,
  `ManualPortMappingMode=<ABSENT>`, `ManualPortMappingPort=<ABSENT>`,
  `PublishServerOnPlexOnlineKey="1"`, `allowedNetworks="10.0.0.0/8,192.168.0.0/16"`. Read by
  attribute grep, never by dumping `Preferences.xml` (it holds `PlexOnlineToken`).
 - **The L7 Gateway path works and is NOT the problem.** Via the Gateway VIP,
  `GET https://plex.home.keithjackson.dev/identity` → `HTTP 200`, `claimed="1"`, real
  `machineIdentifier`. HTTPRoute `plex-32400`: `Accepted=True`, `ResolvedRefs=True`.
 - **Plex enforces its own auth**, which is what makes an unfiltered WAN port defensible:
  `/library/sections`, `/status/sessions`, `/:/prefs` all → **401** unauthenticated through the
  Gateway.
 - **The SNAT/allowedNetworks wart is real, not theoretical.** A tagged probe through the Gateway was
  logged by Plex as `[10.0.2.214:36040 (Allowed Network (WAN))]` — the Envoy pod IP, classified as an
  allowed network.
 - **WAN state:** `136.51.115.252:32400` → connection refused. `136.51.115.252:443` → self-signed
  cert `C=ZZ, ST=Somewhere, O=%D342e7e6b, CN=%D`. Public DNS resolves correctly:
  `plex.home.keithjackson.dev` → CNAME `wan.home.keithjackson.dev` → `136.51.115.252`.
 - **`192.168.2.24` is free in-cluster.** `CiliumLoadBalancerIPPool default-pool`: IPsTotal 11 /
  IPsUsed 4 / IPsAvailable 7; the only allocations are `.20` (cilium-gateway-shared), `.21`
  (unifi-l4-*), `.22` (crafty-l4-*), `.23` (pihole-dns-*). Not a node address (`.10/.11/.12`) and not
  the kube-vip control-plane VIP (`.9`).
 - **The manual-port-mapping step genuinely cannot be automated.** Grep of the running image's whole
  init surface (`/etc/cont-init.d`, `/plex-common.sh`, `/usr/local/bin`) for `ManualPortMapping` and
  `PLEX_PREFERENCE` → **NONE**.
 - **`ADVERTISE_IP` would have worked on a claimed server** (recorded because the runbook's
  "why not" rests on it): `setPref "customConnections"` at line 126 of `40-plex-first-run` sits
  **outside** the `PLEX_CLAIM`/`PlexOnlineToken` guard, and the `/.firstRunComplete` early-exit
  marker is on the **overlay** FS, not the `/config` PVC, so the script re-runs every pod start.
 - **Toolbox gate** (`./infra/devcontainer/dev.sh run bash -c './ci/gate.sh'`): **rc 3**, with
  `spec-immutability`, `decisions-log`, `lint`, `l0`, `render-golden`, `l1` and `approval-policy` all
  **pass**. The rc 3 is two environmental skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`)
  and `conformance` L2/L3 (`CLUSTER_AVAILABLE` unset). **Not green. Reported as rc 3, not as pass.**
 - **`render-golden` passing is the specific proof** that the `render.py` `EXTRA_UNITS` change, the new
  hand-written unit and the edited wave kustomization are mutually consistent.
 ## What I did NOT verify
 - **THE ACTUAL FIX. Nothing in this branch has been proven to make remote access work** — it cannot
  be, because both remaining steps are off-repo and manual (router forward, Plex UI toggle). This
  branch removes two blockers and adds none; it does not demonstrate an outcome.
 - **Outside-in reachability, at all.** Every probe above was taken from inside the LAN, so the
  `:443` self-signed cert strictly proves only that there is **no NAT hairpin** — it does **not**
  prove outside-in failure. A router cert on WAN:443 is suggestive (its management UI would shadow a
  forward), but unproven. **This repo has no off-LAN vantage point.** Test 3 in the runbook
  (app.plex.tv from a cellular connection) is the only honest check.
 - **`192.168.2.24` was never ARP-probed off-cluster.** Only in-cluster pool accounting was checked.
  `.23` was `arping`-probed before it was claimed; `.24` was not, so an off-cluster squatter would
  not have been caught.
 - **The LoadBalancer has never been reconciled.** `plex-l4-tcp` does not exist in prod yet, so
  LB-IPAM has not actually assigned the pinned VIP, L2 announcement has not been observed for it, and
  the new CiliumNetworkPolicy has never admitted a real packet. Whether `world`/`host`/`cluster`/
  `remote-node` is the correct entity set **under this SNAT path** is reasoned from
  `platform/crafty/networkpolicy.yaml`'s working game-port rule, **not measured**.
 - **Whether Plex's remote path needs anything beyond the manual port tick.** If plex.tv still lists
  no remote connection after both manual steps, this branch is necessary-but-insufficient.
 - **The CI approval-policy outcome.** The local stage reported green, but that is an offline N/A, not
  a verdict: this PR touches `clusters/**`, so CI will block it pending an override.
 - **PRE-EXISTING DRIFT I FOUND AND DELIBERATELY LEFT ALONE — worth its own follow-up.** Running the
  documented `python3 catalog/render.py --emit-waves` regenerates the wave kustomizations and wants to
  add **`vm-demo.yaml`** to wave 0 and **`keycloak.yaml`** to wave 1. Both are listed as
  intentionally not onboarded in `ci/render-golden.sh`
  (`RENDER_GOLDEN_NOT_ONBOARDED="keycloak vm-demo"`); their manifests are tracked in `apps/` but
  deliberately absent from the `resources` lists so Flux does not apply them. `--emit-waves` places
  any file present on disk and has **no awareness of that exclusion**, while `render-golden.sh` uses
  `--wave-layout` and compares rendered manifests only — it never checks kustomization `resources`,
  which is why this has survived unnoticed. **So the header "DO NOT hand-edit: re-run
  `render.py --emit-waves`" is currently unsafe advice**: following it onboards an IdP and a KubeVirt
  VM. I therefore restored both kustomizations and hand-added only my one line. I did not fix
  `--emit-waves`, and I have not determined how long the drift has existed or whether either service
  is *meant* to be onboarded.
@@ -1,133 +0,0 @@
 ---
 date: 2026-10-04
 branch: feat/pihole-declare-state
 title: PiHole's operator-set state is now declared in git — the prerequisite for running more than one replica
 ---
 ## What changed
 Phase 1 of making LAN DNS survive a node failure. The operator asked for multi-replica PiHole;
 measurement showed that cannot be done safely first, because PiHole's important state was not
 reproducible from git. A second replica with a fresh PVC would have come up **serving DNS and
 blocking nothing, with a different admin password** — answers varying by which pod replied. This
 branch closes that; the replica work is phase 2 and is deliberately NOT here.
 **The undeclared-state inventory, measured not guessed.** `pihole.toml` marks every non-default
 setting `### CHANGED`, and those that came from the environment `### CHANGED (env)`. The complete
 live list on 2026-10-04:
 | Setting | Status |
 |---|---|
 | `dns.upstreams`, `dns.listeningMode`, `dns.rateLimit.count`, `misc.dnsmasq_lines` | `### CHANGED (env)` — already declared in `catalog/pihole.yaml` |
 | `files.macvendor` | `### CHANGED` — set by the image itself (container path), not operator state |
 | `webserver.api.pwhash` | **`### CHANGED` — set out of band. Declared NOWHERE.** |
 So exactly **one** operator-set value lived only on the PVC. Separately, the **adlists** were also
 undeclared: `gravity.db` held one list (StevenBlack's hosts) and 72,234 gravity domains, with nothing
 in this repo naming it.
 - **`apps/wave-0/pihole-bootstrap-secret.sops.yaml` (new, SOPS/age).** Secret `pihole-bootstrap`,
  key `webui-password`. Named to the `<svc>-bootstrap-secret.sops.yaml` convention so
  `render.py --emit-waves` places it with no renderer change. The value is a 32-char random password
  generated at authoring time and **never printed to a transcript**.
 - **`catalog/pihole.yaml`:** `FTLCONF_webserver_api_password` sourced from that Secret.
  `webserver.api.password` takes CLEARTEXT and FTL hashes it itself — verified in the running image,
  where `pihole setpassword` *is* `setFTLConfigValue "webserver.api.password"`, commented upstream
  "pihole-FTL will automatically hash the password". The PVC's `pwhash` becomes a derived artefact.
 - **`catalog/pihole.yaml`:** a `lifecycle.postStart` hook that **reconciles** the declared adlist set
  into `gravity.db` on every container start — adds missing, re-enables disabled, and **disables**
  (never deletes) anything enabled but undeclared, running `pihole -g` only when something moved.
  Same level-triggered idiom, and the same reasoning, as `catalog/qbittorrent.yaml`'s postStart.
 - **`apps/wave-0/pihole.yaml`, `apps/wave-0/kustomization.yaml`:** re-rendered / secret registered.
 **Why a postStart hook and not a file.** PiHole v6 keeps adlists as DB rows. `/etc/pihole/adlists.list`
 still exists in `gravity.sh` but is **migration-only**: its import sits inside `migrate_to_database()`
 behind `if [ ! -e "$gravityDBfile" ]`, so it fires only while *creating* a database. A ConfigMap
 mounted there would be silently ignored on every pod this cluster runs today. There is also no
 `pihole adlist` CLI subcommand, so SQL is the actual interface; FTL ships an embedded shell
 (`pihole-FTL sqlite3`) so nothing had to be added to the image.
 **The password is kept rather than set empty, deliberately.** The admin UI is already behind the
 oauth2-proxy sidecar (`--allowed-group=platform-admins`), so an empty password would have been
 simpler. But the `pihole` Service still publishes **port 80** (PiHole direct, bypassing the proxy on
 `:4180`) and only the CiliumNetworkPolicy closes it — and this repo has a documented default-deny
 drift history. An unauthenticated admin UI that can rewrite LAN DNS should not rest on one policy
 staying correct. Narrowing that Service port is follow-up, not done here.
 **SENSITIVE PATHS — this PR adds a `**/*.sops.yaml`, so `ci/approval-policy.sh` will BLOCK it.** It
 needs the operator's `/approve-override <reason>` as a PR **review** of type "Comment". Not posted by
 me: CLAUDE.md section 1 forbids an agent clearing this on its own PR.
 ## What I verified
 Live prod, 2026-10-04, read-only except where noted.
 - **The undeclared-state inventory above** — read from `pihole.toml`'s own `### CHANGED` markers, with
  the password value masked. This is the measurement the whole branch rests on.
 - **A fresh `gravity.db` seeds NO adlists.** The image's own schema template
  `/etc/.pihole/advanced/Templates/gravity.db.sql` creates the `adlist` table and contains no
  `INSERT INTO adlist` at all — grepped in the running image. Confirmed empirically below (0 rows).
  This is why the adlist declaration is load-bearing and not cosmetic.
 - **The adlists.list path really is migration-only** — read the guard in `/opt/pihole/gravity.sh`
  (`migrate_to_database()`, `if [ ! -e "$gravityDBfile" ]`).
 - **`webserver.api.password` is a real settable key** — `pihole-FTL --config webserver.api.password`
  returns a masked value, and `/usr/local/bin/pihole`'s `setpassword` sets exactly that key.
 - **THE RENDERED HOOK ACTUALLY WORKS — all four paths, extracted from `apps/wave-0/pihole.yaml` and
  run in the live pod against THROWAWAY COPIES of `gravity.db` in `/tmp`** (production `gravity.db`
  was never written, and `pihole -g` was stubbed so no 70k-domain download could start):
  1. *Steady state* → `already matches the declared set - nothing to do`. No writes, no rebuild.
  2. *True fresh DB built from the image's schema template* → 0 adlists before; hook added the
     declared list and triggered the rebuild. **This is the fresh-PVC/new-replica case.**
  3. *An undeclared list enabled* → disabled it (`enabled=0`), left the declared one alone, and did
     not delete the row.
  4. *Declared list present but disabled* → re-enabled it.
 - **Required tools exist in the image:** `bash`, `expr`, `grep`, `printf`, `sleep`, `pihole-FTL`.
 - **The Secret is genuinely encrypted** to the repo's age recipient; `stringData.webui-password`
  starts `ENC[`. The gate's `no-plaintext-secrets (R-SEC-01)` L0 check passes.
 - **Schema:** `check-jsonschema` against `schemas/servicedefinition.schema.json` passes.
 - **Render determinism:** the canonical
  `render.py --all catalog -o apps --wave-layout --no-image-automation` changed **only**
  `apps/wave-0/pihole.yaml`. `render-golden` passes, so the hook's text also satisfies
  `assert_substitution_safe` (no braced dollar expansion — the Flux envsubst contract).
 - **Toolbox gate:** rc **3**, with `spec-immutability`, `decisions-log`, `lint`, `l0`,
  `render-golden`, `l1`, `approval-policy` all **pass**. The rc 3 is the two usual environmental
  skips — `gitea-settings` (no local `SETTINGS_API_TOKEN`) and `conformance` L2/L3
  (`CLUSTER_AVAILABLE` unset). **Not green; reported as rc 3.**
 ## What I did NOT verify
 - **Nothing here has been reconciled.** Flux has not applied any of it, so the password has never
  been set by this mechanism and the hook has never run during a real container start. Everything
  above is the rendered artifact tested out-of-band, plus the image's own source.
 - **THE CURRENT ADMIN PASSWORD WILL STOP WORKING, and that is the intent.** FTL re-hashes
  `webserver.api.password` on start, overwriting the existing `pwhash`. Whoever knows the present
  password (nobody, which is why this started) loses it; the declared one replaces it. I have not
  observed that replacement happen.
 - **Cold-start cost is unmeasured and could matter.** `postStart` must complete before the container
  is reported Running. On a fresh PVC the hook waits for `gravity.db` (bounded, up to 180s) and then
  runs a real `pihole -g` (72k domains, network downloads). `apps-wave-0` is `wait: true` and
  `apps-wave-1` `dependsOn` it, so a slow cold start **delays the app tier**. The steady-state path
  exits in about a second (scenario 1), so this is a rebuild/first-boot cost only — but it is not
  measured, and the stub in testing means `pihole -g`'s real duration here is unknown.
 - **Whether `pihole -g` succeeds from inside the hook's environment.** Its failure is caught and the
  existing `gravity.db` retained, so a failure degrades rather than breaks — untested live.
 - **Concurrency.** Nothing serialises the hook against FTL's own scheduled gravity update. SQLite
  handles the locking, and the hook's writes are three small statements, but no interleaving was
  exercised.
 - **MULTI-REPLICA — PHASE 2, NOT STARTED.** This branch only removes the blocker. The remaining work,
  and it is substantial: a `replicas`/placement field on the ServiceDefinition schema (currently
  `additionalProperties: false` with no such field), `render.py` emitting a StatefulSet with
  `volumeClaimTemplates` (the config PVC is **ReadWriteOnce**, so a 2-replica Deployment cannot
  work), pod anti-affinity, a PodDisruptionBudget (there is none today), re-rendered goldens, and
  `conformance/**` updates — the last making phase 2 sensitive-path as well. The PVC rename a
  StatefulSet forces will discard the existing `gravity.db`, which is only safe **because** of this
  branch, and should be done only after this one is confirmed working in-cluster.
 - **The real risk the operator named is NOT fixed by phase 2 either.** Replicas address *node*
  failure. LAN DNS still depends on the cluster as a whole — etcd quorum, Cilium, Longhorn, a bad
  reconcile. Quantified on 2026-10-04: with `tolerationSeconds=300` plus node-NotReady detection
  plus a Longhorn RWO force-detach, a failure of the node running the pod is roughly **6-10 minutes
  of no LAN DNS**; a failure of the node holding the `pihole-dns-*` L2 lease (currently a
  *different* node — lease on k3s-prod-2, pod on k3s-prod-1) is tens of seconds. An out-of-cluster
  secondary resolver carrying the same `address=` lines is the only option that removes the
  dependency, and it was considered and not chosen.
 - **Port 80 on the `pihole` Service is still published** and still closed only by policy. Not narrowed.
@@ -51,11 +51,14 @@ hubble:
 l2announcements:
  enabled: true
-# k3s ships its own cgroup mount; point Cilium at it instead of auto-mounting.
+# Point Cilium at the host's real cgroup2 root (/sys/fs/cgroup, mounted by systemd) instead of
 # auto-mounting. NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so the agent mounted its
 # OWN namespaced cgroup and socket-LB covered only itself — host processes (the apiserver) lost
 # Service/LB translation. [parity] platform/cilium/helmrelease.yaml explains (2026-10-03).
 cgroup:
  autoMount:
    enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup
 # Roll the cilium agent DaemonSet ONE node at a time. [parity] must match the
 # platform/cilium HelmRelease. The cilium 1.20.1 chart defaults this to 2, which on a
@@ -106,14 +106,33 @@ spec:
    # CiliumL2AnnouncementPolicy (reserved 10.73.0.240-.250) authored by WS-NET in Phase 2.
    l2announcements:
      enabled: true
-    # --- cgroup: k3s ships its own cgroup mount; point Cilium at it, don't auto-mount ---
+    # --- cgroup: point Cilium at the HOST's cgroup2 root, which systemd mounts at /sys/fs/cgroup ---
-    # [parity] MUST equal Ansible cgroup block. Not in the deliverable's headline list, but it IS
+    # [parity] MUST equal Ansible cgroup block (roles/cilium/templates/cilium-values.yaml.j2).
-    # part of the working CNI config on k3s: reconciling without it lets Cilium try to auto-mount
+    #
-    # cgroupv2 and break on adopt. Kept in parity for a safe Phase-2 adoption.
+    # FIXED 2026-10-03 — hostRoot WAS /run/cilium/cgroupv2, AND THAT SILENTLY BROKE SOCKET-LB FOR
    # EVERY HOST PROCESS. With autoMount disabled, the chart only bind-mounts whatever already exists
    # at hostRoot ("it's expected that users have mounted cgroup2 filesystem at the specified
    # cgroup.hostRoot" — cilium 1.20.1 values.yaml). Nothing ever mounted cgroup2 at
    # /run/cilium/cgroupv2 on these nodes (it is an empty DirectoryOrCreate hostPath), so the agent
    # mounted one ITSELF from inside its own private cgroup namespace. That mount's root is the
    # agent container's OWN leaf cgroup, so cil_sock4_connect & co. were attached there and covered
    # only the agent. Every other host-netns process (the k3s apiserver in
    # /system.slice/k3s.service, curl over SSH) was untranslated: Service ClusterIPs timed out from
    # every node, and the node holding the Gateway's L2 lease could not reach the Gateway VIP at all
    # ("no route to host"). That is why k3s-prod-1's apiserver logged `oidc: authenticator not
    # initialized` from 2026-09-12 on — kubectl SSO through the kube-vip VIP failed outright.
    # Measured on k3s-prod-1 before this change: TCP connect to 192.168.2.20:443, 10.43.225.79:443
    # and 10.43.59.141:8080 SUCCEEDED from inside the agent container (the covered cgroup) and
    # FAILED from the host, same node, same netns. Full write-up:
    # decisions/2026-10-03-cilium-cgroup-hostroot.md.
    #
    # /sys/fs/cgroup IS a cgroup2 mount in the host's root cgroup namespace on every node (verified
    # with findmnt on all three), so bind-mounting it gives the agent the real host root. autoMount
    # stays off because the mount it would create already exists.
    cgroup:
      autoMount:
        enabled: false
-      hostRoot: /run/cilium/cgroupv2
+      hostRoot: /sys/fs/cgroup
    # --- Agent rollout: ONE node at a time (blast-radius bound) — 2026-09-01 incident ---
    # [parity] MUST equal Ansible: updateStrategy.rollingUpdate.maxUnavailable: 1.
    # The chart's own default is 2 (cilium 1.20.1 values.yaml: updateStrategy.rollingUpdate.
@@ -41,11 +41,13 @@ hubble:
 # [parity] ingress VIP announcement on the flat LAN (docs/09, OQ-1)
 l2announcements:
  enabled: true
-# [parity] k3s provides the cgroupv2 mount; do not auto-mount
+# [parity] the host's cgroup2 root is /sys/fs/cgroup (systemd mounts it); do not auto-mount.
 # NOT /run/cilium/cgroupv2: nothing mounts cgroup2 there, so socket-LB attached to the agent's own
 # cgroup and host processes lost Service/LB translation (2026-10-03 — see helmrelease.yaml).
 cgroup:
  autoMount:
    enabled: false
-  hostRoot: /run/cilium/cgroupv2
+  hostRoot: /sys/fs/cgroup
 # [parity] roll the agent DaemonSet ONE node at a time. Overrides the cilium 1.20.1 chart
 # default of 2, which on 3 nodes lets a single bad render break 2/3 agents at once — as the
 # 2026-09-01 incident did (k8sServiceHost: "" → the `config` init container fell back to the
@@ -1,148 +0,0 @@
 # plex-remote-access — getting Plex reachable from outside the LAN, and proving it
 **Status: the in-cluster half is shipped; the last mile is TWO MANUAL STEPS and remote access does
 NOT work until both are done.** Neither can live in this repo.
 **Alert:** none, and none is possible from in-cluster signals — see "Why no alert" below. This is an
 **operator procedure**, not an incident page.
 **Files:** `apps/wave-0/plex-remote.yaml` (the tcp/32400 LoadBalancer + its CiliumNetworkPolicy),
 `clusters/base/cluster-vars.yaml` + `clusters/prod/kustomization.yaml` (`PLEX_L4_VIP`),
 `catalog/render.py` (the `EXTRA_UNITS` entry that places the unit), `catalog/plex.yaml` (Plex's own
 env — note `ADVERTISE_IP` is deliberately NOT set; see "Why not the Gateway URL").
 **Related:** `runbooks/plex-quicksync.md` and `runbooks/plex-claim.md` — the other two Plex
 procedures with a manual last mile.
 ## What was actually broken (measured 2026-10-03)
 Plex was serving correctly and was reachable on the LAN. Two independent things stopped remote
 clients:
 1. **Plex advertised no remote connection at all.** In the live `Preferences.xml`:
   `customConnections` **ABSENT**, `ManualPortMappingMode` **ABSENT**, while
   `PublishServerOnPlexOnlineKey="1"`. The server was trying to publish itself to plex.tv and had
   nothing reachable to publish. Plex's automatic path (UPnP/NAT-PMP) cannot work here: the process
   lives in a pod on the Cilium overlay (`10.0.2.136`), it cannot speak UPnP to the SOHO router, and
   a mapping to the pod IP would be meaningless if it could.
 2. **Nothing from the WAN reached the cluster.** `136.51.115.252:32400` → connection refused.
   `136.51.115.252:443` → a **self-signed router certificate** (`C=ZZ, CN=%D`), not the Gateway's
   Let's Encrypt wildcard.
 The Service was `ClusterIP` only, so port 32400 had no path off-cluster by any route.
 **Caveat on that second measurement, because it is easy to over-read:** it was taken from *inside*
 the LAN, so strictly it proves there is no NAT hairpin (reflection). It does not prove outside-in
 failure. But a router certificate answering 443 means the router's own management UI is bound to the
 WAN port, and that *would* shadow a port-forward for genuine outside traffic too. **Verify from a
 real outside vantage point** (phone on cellular, or an external port checker) before concluding
 either way — this repo has no such vantage point and must not pretend otherwise.
 ## Step 1 — forward WAN tcp/32400 on the router (operator, off-repo)
    WAN tcp/32400  ->  192.168.2.24  (${PLEX_L4_VIP})  port 32400
 `192.168.2.24` is a pinned Cilium LB VIP, ARP-announced on the flat LAN by the existing
 `CiliumL2AnnouncementPolicy default-l2`. It is **pinned on purpose**: this address is typed into the
 router by hand, so it must never move on a reschedule.
 While in the router, **check what else is bound to the WAN interface.** If remote management is
 listening on 443 (see the certificate measurement above), move it or disable it — otherwise the
 Gateway's 80/443 forward is shadowed and the Plex *web UI* over the internet stays broken even
 though remote *streaming* works. These are two separate paths and they fail independently.
 Confirm the VIP is actually answering on the LAN first, so a failure later is unambiguous:
    ./infra/devcontainer/dev.sh run bash -c \
      'kubectl --context k3s-prod -n plex get svc plex-l4-tcp -o wide'
 `EXTERNAL-IP` must read `192.168.2.24`. If it is `<pending>`, LB-IPAM could not assign — the address
 is outside the pool or already taken; see `clusters/prod/kustomization.yaml` for the pool accounting.
 **`192.168.2.24` was never ARP-probed off-cluster.** Only the in-cluster pool accounting was checked
 (IPsTotal 11 / IPsUsed 4 / IPsAvailable 7 on 2026-10-03). If the VIP behaves oddly, probe it from a
 LAN host — `arping 192.168.2.24` — the way `.23` was probed before it was claimed.
 ## Step 2 — tick "manually specify public port" in Plex (operator, UI only)
 **There is no GitOps path to this and no environment variable for it.** Measured in the running
 image on 2026-10-03: a grep of the entire init surface (`/etc/cont-init.d`, `/plex-common.sh`,
 `/usr/local/bin`) for `ManualPortMapping` and `PLEX_PREFERENCE` returned **NONE**. The image writes
 only six preferences, and this is not one of them. It is the same class of manual last mile as the
 `HardwareAcceleratedCodecs` toggle in `runbooks/plex-quicksync.md`.
 In the Plex web UI (`https://plex.home.keithjackson.dev`, on the LAN):
 **Settings → Remote Access**
 - Ensure **Enable Remote Access** is on. (`PublishServerOnPlexOnlineKey` was already `1`.)
 - Tick **Manually specify public port** and set it to **32400**.
 - **Apply**, then let the panel settle. It should report *"Fully accessible outside your network"*.
 Without this tick Plex waits for a NAT mapping it can never obtain and publishes no remote
 connection, so the port-forward in step 1 sits there unused.
 ## How to verify
 Do these in order; each one isolates a different half.
 **1. The port is open from outside.** From a phone on cellular, or any off-LAN host:
    curl -sk -o /dev/null -w '%{http_code}\n' https://136.51.115.252:32400/identity
 `200` means the forward and the policy are both good. A hang or refusal means step 1.
 **2. Plex itself believes it is reachable.** Settings → Remote Access must say *"Fully accessible
 outside your network"*. If it says *"not available outside your network"* while test 1 passes, step 2
 was not applied.
 **3. plex.tv is handing out a remote connection.** The decisive test, because it is what clients
 actually consume: sign in to [app.plex.tv](https://app.plex.tv) from an off-LAN network and confirm
 the server appears and plays. A client finds this server through plex.tv's connection list, never
 through DNS.
 **4. The policy is admitting the traffic.** If test 1 fails *after* the forward is confirmed, the
 CiliumNetworkPolicy is the next suspect:
    ./infra/devcontainer/dev.sh run bash -c \
      'kubectl --context k3s-prod -n plex get cnp plex-allow-remote-32400'
 See `runbooks/denied-flows.md` for reading actual drops.
 ## Why no alert
 Every in-cluster signal is green whether or not remote access works. The Service has endpoints, the
 pod is `Running`, the policy is applied, and `plex-down` watches
 `kube_deployment_status_replicas_available` — none of which can see a router with a missing
 port-forward or an unticked checkbox. An alert would need an off-LAN prober, which this cluster does
 not have. **Test 3 above is the only honest check, and it is manual.**
 ## Why not the Gateway URL (`ADVERTISE_IP`)
 The alternative was to leave Plex on `ClusterIP` and set
 `ADVERTISE_IP: https://plex.home.keithjackson.dev:443`, which writes Plex's `customConnections`
 preference and would have put the server on plex.tv's list via the existing L7 Gateway — no new WAN
 hole. That mechanism was verified to work on an already-claimed server (the `ADVERTISE_IP` block at
 line 126 of the image's `40-plex-first-run` sits **outside** the claim guard, and `/.firstRunComplete`
 lives on the overlay FS, not the `/config` PVC, so it re-runs every pod start), and the Gateway path
 itself was measured serving Plex correctly: `HTTP 200`, `claimed="1"`, with `401` on every protected
 endpoint.
 **The operator chose the native path anyway (2026-10-03), and the reasons are real:** Plex terminates
 its own TLS with its `*.plex.direct` certificate, and video streams bypass the Envoy L7 hop instead
 of being proxied through it. The cost is a new unfiltered WAN port — acceptable because Plex
 authenticates every caller itself. `ADVERTISE_IP` therefore stays unset in `catalog/plex.yaml`, and
 the deferral note there is still accurate.
 ## Known wart: remote clients are classified as LAN
 `apps/wave-0/plex-remote.yaml` uses `externalTrafficPolicy: Cluster`, which SNATs the client to the
 ingress node's address (`192.168.2.1x`). That falls inside the `192.168.0.0/16` entry of
 `ALLOWED_NETWORKS`, so **Plex treats every remote stream as local.** Measured on the Gateway path,
 which SNATs the same way: Plex logged `[10.0.2.214:36040 (Allowed Network (WAN))]`.
 This is a **metering** effect, not a security one — authorization is unchanged, as the 401s above
 show — but Plex's remote-quality and bandwidth limits will not bind on these streams.
 `Local` would preserve the client IP and fix it. **Do not switch to it.** Cilium's L2 announcer
 elects one lease holder per Service and does *not* follow the backend pod, so the VIP can be
 announced by a node with no Plex pod, black-holing the port while everything reports healthy. That
 exact change cost this LAN an 8-day Minecraft outage in September 2026 — the full write-up is at the
 top of `platform/crafty/service-l4.yaml`. Plex is a single-replica `Recreate` Deployment, so the
 node mismatch is the normal case here, not a rare one.