--- # Gitea Actions runner (act_runner) — Phase 6 CI/CD (WS-GITOSS). # # WHY A DEDICATED, NON-APP NAMESPACE: # The runner's job executor is a Docker-in-Docker (DinD) sidecar that MUST run # `privileged: true` (dockerd needs it). The Kyverno `restrict-host-namespaces` # policy denies privileged containers, but it (and the egress-class / auth-mode / # default-deny-baseline policies) scope ONLY to namespaces labeled # homelab.keithjackson.dev/app-namespace: "true". This namespace deliberately does # NOT carry that label — it is CI plumbing, not a user-facing app — so the privileged # DinD sidecar is admitted WITHOUT a policy exception. (Verified: a privileged Pod in # a non-app namespace passes server-side admission; Kyverno test fixture # `platform-workload` is likewise Excluded from restrict-host-namespaces.) # # EXECUTOR MODEL — Docker executor + DinD sidecar (privileged): # act_runner runs each job step inside a container it launches via a Docker daemon. # That daemon is the co-located `dind` sidecar (shared pod netns → tcp://localhost:2376, # TLS via a shared emptyDir of certs). Tradeoff vs the host/self-hosted executor: DinD # needs `privileged`, but it gives clean per-job isolation and lets the workflow pick its # own job image (the pinned toolchain), which the host executor cannot. Privileged is # contained to this CI-only namespace and the runner mounts NO ServiceAccount token # (automountServiceAccountToken: false) — CI holds no cluster credentials (R-SEC-02). apiVersion: v1 kind: Namespace metadata: name: gitea-actions labels: # NOTE: intentionally NO homelab.keithjackson.dev/app-namespace label (see header). homelab.keithjackson.dev/ci: act-runner --- # Default-deny ingress; broad egress (a CI runner legitimately needs the internet: # ghcr.io / github.com / docker hub / dl.k8s.io / pypi for job images + pinned tools, # and the Gitea ingress VIP for registration + repo clone). Nothing connects INTO the # runner, so ingress is denied entirely. apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-ingress-allow-egress namespace: gitea-actions labels: homelab.keithjackson.dev/baseline: default-deny spec: podSelector: {} policyTypes: - Ingress - Egress ingress: [] egress: # DNS (cluster + split-horizon resolution of gitea.home.keithjackson.dev). - to: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: kube-system ports: - protocol: UDP port: 53 - protocol: TCP port: 53 # Everything else the runner + DinD need: Gitea VIP + internet registries/tool hosts. - {} --- # Runner state (config.yaml + the `.runner` registration file) persists here so the # runner registers exactly ONCE and reuses its identity across restarts (no duplicate # offline rows in Gitea). RWO → Deployment strategy must be Recreate (fix/recreate-strategy-rwo). apiVersion: v1 kind: PersistentVolumeClaim metadata: name: act-runner-data namespace: gitea-actions labels: app.kubernetes.io/name: act-runner spec: accessModes: - ReadWriteOnce storageClassName: longhorn resources: requests: storage: 1Gi --- apiVersion: apps/v1 kind: Deployment metadata: name: act-runner namespace: gitea-actions labels: app.kubernetes.io/name: act-runner spec: replicas: 1 strategy: type: Recreate selector: matchLabels: app.kubernetes.io/name: act-runner template: metadata: labels: app.kubernetes.io/name: act-runner spec: # CI holds no cluster credentials — do not mount a SA token into the runner or # (via the shared netns) the job containers (R-SEC-02, docs/14). automountServiceAccountToken: false containers: # ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ---- - name: runner image: gitea/act_runner:0.2.11 # TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal # entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first # boot and runs `act_runner daemon`. # # STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup # and hard-exits if it is missing (a startup race against the sidecar). # STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is # necessary but NOT sufficient. dockerd writes the certs and only THEN begins # listening on :2376, so a runner that proceeds on the file alone still hard-exits # with "cannot ping the docker daemon at tcp://localhost:2376" — observed # crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce. # Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar # surfaces as a CrashLoopBackOff with a clear message instead of hanging forever. # # ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the # braced form or the double-dollar form; both are wrong here, for different reasons. # clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so # Flux envsubst runs over this ENTIRE manifest — YAML comments very much included. # Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml, # and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is: # - A bare $name is NOT a reference to envsubst: it passes through untouched, and # that holds even when the name IS set in cluster-vars. So $i and $j reach the # cluster verbatim and both 60s bounds below are real, enforced bounds. # - $((i + 1)) is likewise untouched: "(" cannot begin a variable name. # - A BRACED dollar-name IS a reference and gets replaced — with the empty string # when the name is unset. One of those here would make a bound read [ "" -ge 60 ], # which errors, short-circuits the &&, and silently turns the wait UNBOUNDED. # - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced # form — still wrong for us, because on any path where envsubst does NOT run # (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell # reads that leading double-dollar as its own PID, so a bound would compare a # string like 1234{i} against 60. The bare form is the only spelling that is # correct on BOTH paths. # - A braced name containing punctuation is a HARD ERROR ("unable to parse variable # name") that fails this whole Kustomization — and this Kustomization gates every # merge in the repo. So never write one anywhere in this file, INCLUDING in a # comment. That is why this note names the three forms in words rather than # showing them: an earlier draft of it spelled them out literally and broke # `flux envsubst` on the file. (Same trap the indexed-expansion warning in # clusters/base/apps-vms.yaml describes.) command: - /bin/sh - -c - | # Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir. i=0 until [ -f /certs/client/ca.pem ]; do i=$((i + 1)) [ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; } echo "waiting for dind TLS certs ($i)"; sleep 1 done # Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this # Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout). j=0 until nc -z -w1 127.0.0.1 2376; do j=$((j + 1)) [ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; } echo "waiting for dind daemon on :2376 ($j)"; sleep 1 done exec /sbin/tini -- /opt/act/run.sh env: - name: GITEA_INSTANCE_URL value: https://gitea.home.keithjackson.dev - name: GITEA_RUNNER_REGISTRATION_TOKEN valueFrom: secretKeyRef: name: gitea-actions-runner key: token - name: GITEA_RUNNER_NAME value: incluster-dind # runs-on label -> job image. Both map to the catthehacker "act" image, which # carries bash/git/curl/tar/python3+pip/node so .gitea/workflows scripts can # install the pinned CI toolchain (ci/versions.env) and actions/checkout runs. - name: GITEA_RUNNER_LABELS value: "ubuntu-latest:docker://ghcr.io/catthehacker/ubuntu:act-22.04,homelab:docker://ghcr.io/catthehacker/ubuntu:act-22.04" # Reach the DinD daemon over the shared pod network namespace, TLS-verified # against the certs the sidecar drops into the shared /certs emptyDir. - name: DOCKER_HOST value: tcp://localhost:2376 - name: DOCKER_CERT_PATH value: /certs/client - name: DOCKER_TLS_VERIFY value: "1" volumeMounts: - name: data mountPath: /data - name: certs mountPath: /certs readOnly: true resources: requests: cpu: 100m memory: 128Mi limits: memory: 512Mi # ---- Docker-in-Docker: the job executor daemon (privileged) ---- # # FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change): # Convert this container into a NATIVE SIDECAR and delete both shell wait-loops. # k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the # ordering that the loops above hand-roll — which also removes the envsubst hazard # from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape). # Shape to build: # 1. Move this whole container into `initContainers:` as the FIRST entry. # 2. Add `restartPolicy: Always` to it. That single field is what makes an # initContainer a sidecar: it starts before the regular containers, keeps # running alongside them, and is terminated after them. # 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS # connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify # --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem # --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe: # it would re-open the same "port accepting but TLS/certs not ready" gap that # Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60` # preserves today's 60s cap. # 4. kubelet then holds the `runner` container until that startupProbe passes, so # `runner` can drop its `command:` override completely and run the image's # default entrypoint. # Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this # entire repository. If the conversion is wrong, nothing can merge — so it must land # alone, where the failure is unambiguous and not entangled with anything else, and # only after a real workflow run has been observed green on the converted pod. - name: dind image: docker:27-dind securityContext: privileged: true env: # Generate + publish TLS certs into the shared emptyDir; dockerd then listens # on tcp://0.0.0.0:2376 (TLS) which the runner reaches at localhost:2376. - name: DOCKER_TLS_CERTDIR value: /certs volumeMounts: - name: certs mountPath: /certs - name: dind-storage mountPath: /var/lib/docker resources: requests: cpu: 100m memory: 256Mi limits: memory: 2Gi volumes: - name: data persistentVolumeClaim: claimName: act-runner-data - name: certs emptyDir: {} - name: dind-storage emptyDir: {}