jakman85andClaude Opus 5
1157c1b071
Re-authors the fix stranded on origin/fix/act-runner-dind-readiness (242951a), which
applied cleanly but had never been merged.
The act_runner startup waited only for the dind TLS CERT FILE (/certs/client/ca.pem),
but dockerd writes the certs before it begins accepting TLS connections on :2376. A
runner proceeding on the file alone still hard-exits with "cannot ping the docker daemon
at tcp://localhost:2376" — observed crashlooping 11 times in ~8 minutes on a fresh pod
after a node bounce. Adds a second bounded wait (nc -z -w1 127.0.0.1 2376, 60s cap)
mirroring the existing cert-file loop, so the runner starts only once the daemon
actually answers. busybox nc in the Alpine-based gitea/act_runner:0.2.11 supports -z/-w.
This Deployment is the ENFORCED MERGE GATE for the whole repo, so it ships alone.
A REPORTED "latent envsubst bug" in this file was INVESTIGATED AND DISPROVEN. The claim
was that Flux envsubst expands bare `$i`, making the existing `[ "$i" -ge 60 ]` bound a
silent no-op. Running the real `flux envsubst` (v2.4.0) shows bare `$i` is NOT
substituted — even with i set in the environment, `$i` passes through literally while
`${i}` expands. This matches the repo's own contract in clusters/base/apps-vms.yaml,
which was itself written after empirical probing: "`$VAR` -> left alone by envsubst =>
bare shell vars need NO escaping." The existing 60s bound was always real. NO escaping
was applied: `$${i}` would be an ACTIVE REGRESSION, correct only on the envsubst path —
on a plain `kustomize build`/`kubectl apply -k`, sh reads `$$` as its own PID and the
bound compares garbage against 60. Bare is correct on both paths. A comment now records
the real contract so nobody "fixes" the bare form into escaping.
NEAR MISS worth recording: the first draft of that comment spelled the dangerous forms
out literally, and `flux envsubst` HARD-FAILED on it — "unable to parse variable name".
Flux's envsubst runs over the entire manifest INCLUDING YAML COMMENTS, so a braced name
containing punctuation is a fatal parse error that would have failed the whole
apps-wave-1 Kustomization — breaking every merge in the repo. From a comment. The
shipped comment names the forms in words; the file now contains no braced-dollar
sequence at all.
Verified without a cluster: flux envsubst over the file is byte-identical (exit 0, and
--strict 0); post-envsubst YAML parses with containers unchanged; `sh -n` clean; a
dry-run with a stubbed failing nc fires the stage-2 bound and exits 1; render==golden
green (gitea-actions.yaml is an EXTRA_UNITS entry — render.py places the filename into a
wave and never emits its bytes, so it is not among the 10 compared goldens).
Follow-up documented in-file, deliberately NOT done here: convert dind to an
initContainers sidecar (restartPolicy: Always) with a startupProbe that execs `docker
info` over TLS — NOT a bare tcpSocket probe, which would reopen the same "port open but
certs not ready" gap stage 1 closes — periodSeconds 1 / failureThreshold 60 to preserve
the cap, after which runner drops its command: override. k3s v1.36 has GA sidecars, so
this is the correct long-term shape; it ships separately because a failure in the merge
gate must be unambiguous.
Serves: R-GITOPS-01, R-TEST-03.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
home-network-k3s-impl
Implementation of the home k3s + Longhorn platform. This repo holds the reconciled artifacts
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the read-only
specification at ../home-network-k3s.
Rules (from the spec — non-negotiable)
- The spec is read-only. Never modify anything under
../home-network-k3s. Implement against it; escalate spec problems, don't edit them (specAGENTS.md§11, R-TEST-06). - Done = a green, reproducible, requirement-tagged conformance test — never "merged," never a
manual demo (spec
docs/14,agents/definition-of-done.md). - No work on
main. Every change is a feature branch merged intomain(specCONTRIBUTING.md). - No plaintext secrets — SOPS-encrypted only (spec
docs/07). - Phase discipline — only build what the active roadmap phase authorizes (spec
roadmap/phases.md).
Layout
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
pinned dev container that carries the whole toolchain
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
Split out because CRs and the CRDs they use cannot apply in one pass.
apps/ application tier, reconciled as dependency-ordered WAVES:
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
vms/ KubeVirt VirtualMachines (non-gating tier)
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
schemas/ ServiceDefinition JSON Schema + renderer contract
clusters/ Flux entrypoints + dev/ and prod/ overlays
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
Status
Roadmap phases 0–6 are delivered and deployed on the dev cluster; Phase 7 (prod) is not
open — clusters/prod is a design-only overlay.
| Phase | Delivered |
|---|---|
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
| 1 Cluster bring-up | 3-node k3s HA on Proxmox VE (192.168.1.50/.51/.52), Cilium, Flux, SOPS |
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. Velero still blocked on OQ-BKP; image-automation authored but disabled |
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but UniFi OS Server is not yet installed in the guest |
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · kubectl OIDC SSO with group→RBAC · in-cluster Gitea Actions merge gate |
Conformance: 36 / 51 requirements covered (70.6%), 15 gaps, 1 safety-critical gap
(R-STATE-02, Longhorn node-loss). Regenerate with ./conformance/run.sh --coverage-only;
the full report is conformance/_artifacts/coverage/coverage.md.
The running narrative — decisions, root causes, incidents, follow-ups — is
DECISIONS-LOG.md. It, not this table, is the authoritative record.
Working on this repo
Everything the gate needs is baked into the pinned dev container — do not install the toolchain on the host:
infra/devcontainer/dev.sh build # once
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
The container mounts the repo at /work, your age keyring, and the sibling spec repo read-only
so the spec-immutability check can actually run. See
infra/devcontainer/README.md.
A green run prints MERGE GATE: PASS. The L2/L3 conformance layers stay SKIPPED unless
CLUSTER_AVAILABLE=true — the static gate never pretends to have run them.