diff --git a/apps/wave-1/gitea-actions.yaml b/apps/wave-1/gitea-actions.yaml index 4447d82..198edb5 100644 --- a/apps/wave-1/gitea-actions.yaml +++ b/apps/wave-1/gitea-actions.yaml @@ -103,21 +103,65 @@ spec: # ---- act_runner: registers with Gitea, dispatches jobs to the DinD daemon ---- - name: runner image: gitea/act_runner:0.2.11 - # Wait for the DinD sidecar to publish its TLS client certs into the shared - # emptyDir before starting the daemon — act_runner reads /certs/client/ca.pem at - # startup and hard-exits if it is missing (a startup race against the sidecar). - # Then exec the image's normal entrypoint (/sbin/tini -- /opt/act/run.sh), which - # registers-from-env on first boot and runs `act_runner daemon`. + # TWO-STAGE BOUNDED WAIT on the DinD sidecar, then exec the image's normal + # entrypoint (/sbin/tini -- /opt/act/run.sh), which registers-from-env on first + # boot and runs `act_runner daemon`. + # + # STAGE 1 — the certs must EXIST: act_runner reads /certs/client/ca.pem at startup + # and hard-exits if it is missing (a startup race against the sidecar). + # STAGE 2 — the daemon must actually ACCEPT a connection: the cert FILE appearing is + # necessary but NOT sufficient. dockerd writes the certs and only THEN begins + # listening on :2376, so a runner that proceeds on the file alone still hard-exits + # with "cannot ping the docker daemon at tcp://localhost:2376" — observed + # crashlooping 11 times in ~8 minutes on a fresh pod after a node bounce. + # Both loops are bounded (60 x 1s) and exit 1 on timeout, so a genuinely dead sidecar + # surfaces as a CrashLoopBackOff with a clear message instead of hanging forever. + # + # ENVSUBST SAFETY — the BARE $i / $j below are CORRECT. Do not "tidy" them into the + # braced form or the double-dollar form; both are wrong here, for different reasons. + # clusters/base/apps-wave-1.yaml carries postBuild.substituteFrom cluster-vars, so + # Flux envsubst runs over this ENTIRE manifest — YAML comments very much included. + # Its contract (drone/envsubst; the same one spelled out in clusters/base/apps-vms.yaml, + # and re-verified against this exact file with `flux envsubst`, flux v2.4.0) is: + # - A bare $name is NOT a reference to envsubst: it passes through untouched, and + # that holds even when the name IS set in cluster-vars. So $i and $j reach the + # cluster verbatim and both 60s bounds below are real, enforced bounds. + # - $((i + 1)) is likewise untouched: "(" cannot begin a variable name. + # - A BRACED dollar-name IS a reference and gets replaced — with the empty string + # when the name is unset. One of those here would make a bound read [ "" -ge 60 ], + # which errors, short-circuits the &&, and silently turns the wait UNBOUNDED. + # - A DOUBLE-dollar braced name is envsubst's escape and survives as the braced + # form — still wrong for us, because on any path where envsubst does NOT run + # (a plain `kustomize build` or `kubectl apply -k` of this directory) the shell + # reads that leading double-dollar as its own PID, so a bound would compare a + # string like 1234{i} against 60. The bare form is the only spelling that is + # correct on BOTH paths. + # - A braced name containing punctuation is a HARD ERROR ("unable to parse variable + # name") that fails this whole Kustomization — and this Kustomization gates every + # merge in the repo. So never write one anywhere in this file, INCLUDING in a + # comment. That is why this note names the three forms in words rather than + # showing them: an earlier draft of it spelled them out literally and broke + # `flux envsubst` on the file. (Same trap the indexed-expansion warning in + # clusters/base/apps-vms.yaml describes.) command: - /bin/sh - -c - | + # Stage 1: the sidecar's TLS client certs must appear in the shared emptyDir. i=0 until [ -f /certs/client/ca.pem ]; do i=$((i + 1)) [ "$i" -ge 60 ] && { echo "dind certs never appeared" >&2; exit 1; } echo "waiting for dind TLS certs ($i)"; sleep 1 done + # Stage 2: dockerd must actually accept a connection on :2376. busybox nc in this + # Alpine-based image (gitea/act_runner:0.2.11) supports -z (scan) and -w (timeout). + j=0 + until nc -z -w1 127.0.0.1 2376; do + j=$((j + 1)) + [ "$j" -ge 60 ] && { echo "dind daemon never accepted a connection on 2376" >&2; exit 1; } + echo "waiting for dind daemon on :2376 ($j)"; sleep 1 + done exec /sbin/tini -- /opt/act/run.sh env: - name: GITEA_INSTANCE_URL @@ -155,6 +199,31 @@ spec: limits: memory: 512Mi # ---- Docker-in-Docker: the job executor daemon (privileged) ---- + # + # FOLLOW-UP (deliberately NOT bundled into the readiness fix above — own change): + # Convert this container into a NATIVE SIDECAR and delete both shell wait-loops. + # k3s is now v1.36, where sidecar containers are GA, so kubelet can enforce the + # ordering that the loops above hand-roll — which also removes the envsubst hazard + # from this file entirely (no shell, no $i/$j, nothing to escape or mis-escape). + # Shape to build: + # 1. Move this whole container into `initContainers:` as the FIRST entry. + # 2. Add `restartPolicy: Always` to it. That single field is what makes an + # initContainer a sidecar: it starts before the regular containers, keeps + # running alongside them, and is terminated after them. + # 3. Give it a `startupProbe` that succeeds only when dockerd accepts a TLS + # connection — e.g. `exec` a `docker -H tcp://127.0.0.1:2376 --tlsverify + # --tlscacert /certs/client/ca.pem --tlscert /certs/client/cert.pem + # --tlskey /certs/client/key.pem info`. Do NOT use a bare `tcpSocket` probe: + # it would re-open the same "port accepting but TLS/certs not ready" gap that + # Stage 1 above exists to close. `periodSeconds: 1` + `failureThreshold: 60` + # preserves today's 60s cap. + # 4. kubelet then holds the `runner` container until that startupProbe passes, so + # `runner` can drop its `command:` override completely and run the image's + # default entrypoint. + # Why it is a separate change: this Deployment is the ENFORCED MERGE GATE for this + # entire repository. If the conversion is wrong, nothing can merge — so it must land + # alone, where the failure is unambiguous and not entangled with anything else, and + # only after a real workflow run has been observed green on the converted pod. - name: dind image: docker:27-dind securityContext: