Commit Graph
7 Commits
Author SHA1 Message Date
jakman85andClaude Opus 5.5 e539109830 feat(images): build, verify and publish the three images in CI; Renovate carries the digests
Operator rule 2026-10-29: image creation, hosting and lifecycle belong to CI/CD, never to a
workstation build/push. Nothing had ever been pushed to the Gitea registry.

- .gitea/workflows/images.yaml + scripts/images.sh: PRs build+verify (no credential); main
  builds, verifies and pushes toolbox, agent-runner (FROM toolbox) and conformance-runner as
  sha-<commit> + main, reading the digest back from the registry. Only changed images build.
- Each image's own build-image.sh does the build and in-image verification, now engine-neutral
  (DinD: no podman-only calls, probe over stdin not a bind mount). New toolbox build-image.sh.
- Digest references carry `# renovate-digest:` markers; renovate.json5 gains a custom.regex
  manager for them. The worker Job template is deliberately untracked (it would loop).
- renovate.yaml pins renovate-version 44.167.0: the action's default (39.264.0) rejects the
  current config's managerFilePatterns, so every run would have died at CONFIG_VALIDATION.
- Push credential: REGISTRY_PUSH_TOKEN secret; the Actions token has no package scope in
  Gitea 1.24.3 (container.go Authenticate). Retention: infra/gitea/registry-retention.sh.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-29 23:59:55 -05:00
jakman85andClaude Opus 5 8059297036 fix(devcontainer): bake wg + qrencode into the image instead of splitting the VPN procedure across host and container
gate / gate (pull_request) Successful in 1m53s
gate / conformance (pull_request) Has been skipped
Operator correction: "sop, wg, and any other tool requirement is supposed to be
baked into the devcontainer."

My previous commit discovered that `wg` existed only on the host, `sops` only in
the container, and `qrencode` on neither -- and fixed it the wrong way, by
splitting every step of the key procedure across the boundary and telling the
operator to `dnf install qrencode` on their workstation. That breaks CLAUDE.md §5
and the devcontainer README's "Don't install the toolchain on the host", produces
a procedure that cannot be reproduced in CI, and is miserable to follow.

wireguard-tools + qrencode are now distro packages in the Containerfile. Both are
in Fedora 44's repos (wireguard-tools 1.0.20260223 -- the same upstream version as
the image this component runs -- and qrencode 4.1.1), unpinned like every other
dnf package there: the pinned thing is the Fedora base, and neither is a gate tool
whose version changes what green means.

Three things got SIMPLER rather than merely relocated, which is the tell that the
rule was right:

  * Step 2 is one pipeline in one shell -- `sops set ... "$(wg genkey)"` writes the
    key straight into the already-encrypted document. No staged copy in .cache, no
    `*.key`-suffix warning, no sha256 comparison to prove the right bytes landed.
  * new-peer.sh derives the server public key from wg-server-key.sops.yaml itself
    (`sops -d --extract | wg pubkey`), so there is no local key copy to keep in
    step with the encrypted one, and no magic host path it must live at.
  * `.cache/` is gitignored at the repo root (mirroring infra/ansible/.gitignore),
    giving the container a persistent non-committable scratch dir. new-peer.sh
    writes profiles to .cache/wg-peers/ -- necessary because the repo bind-mount is
    the container's only persistent read-write mount, so a profile in /tmp there is
    gone before it can reach a desktop. Its blanket "refuse to write inside the
    checkout" guard is now a `git check-ignore` test of the real property.

Verified by execution, not reasoning: the image rebuilds; wg/qrencode/sops all
resolve in it; `sops -d | wg pubkey` runs in one shell; new-peer.sh refuses the
REPLACE-ME placeholder with an actionable message; and a full run with a throwaway
server key renders a correct profile at 0600 under a 0700 gitignored dir, prints
the QR, and refuses a non-ignored destination inside the repo.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-10-06 00:48:32 -05:00
jakman85andClaude Opus 5 708a05dfc1 fix(devcontainer): restore multi-cluster kubeconfig selection
infra/devcontainer/kubeconfig.sh was never staged into the delivery-A commit, so
the branch built on main carried main's single-cluster tooling and `--context
k3s-prod` failed with 'context does not exist' mid-incident. Not Flux-reconciled,
so it never affected the cluster — but it breaks prod diagnosis from a checkout of
this branch, which is when you need it most.

Carries the concurrency fix: the merged kubeconfig holds both contexts and NO
current-context, with each invocation getting a per-invocation selector file
prepended to KUBECONFIG. Two concurrent dev.sh runs with different CLUSTER values
previously overwrote each other's current-context mid-command — reproduced live,
a CLUSTER=dev reader returned k3s-prod nodes on 3 of 7 iterations.

Co-Authored-By: Claude Opus 5 <[email protected]>
2026-09-07 01:55:14 -05:00
jakman85andClaude Opus 5 f7d3884ad2 fix(ci): scope the requirement matcher to the diff and to non-doc files
`ci/touched-requirements.sh` documents its detection as "R-* tokens literally
present in the diff of changed files". It did not do that — it grepped the ENTIRE
CONTENT of every changed file. So a change inherited every requirement those files
had ever mentioned in prose, and `--require-all` then blocked the merge on
requirements the author never went near.

On this branch that meant editing the decision log and the tofu docs dragged in
five unrelated requirements, all on untouched lines, all legitimately uncovered:

  [coverage] REQUIRE: FAILED — required requirement(s) uncovered: [5 IDs]

The practical effect was that the PM could not append to the decision log without
failing CI — which is a fair part of why the log went stale in the first place.
This is the defect already logged as "FOLLOW-UP (WS-TEST): fix the matcher so real
requirements need not be exempted".

Two scoping rules, both learned from the gate blocking merges it should not have:

  1. Scan the ADDED/REMOVED lines of the diff, as the header always claimed.
     Removed lines count too — deleting a test's requirement tag, or a policy's
     self-declared ID, touches that requirement just as much as adding one.

  2. Skip `*.md`. An ID in prose is a REFERENCE, not a declaration: explaining a
     requirement is not changing behaviour against it. Rule 1 alone is not enough —
     a commit that documents a mis-attribution names the IDs involved and thereby
     re-triggers the very mis-attribution it fixes, and the decision log can never
     record which requirements a change concerned.

This TIGHTENS attribution rather than weakening the gate:
  - no requirement is exempted, and DOCUMENTED_GAPS is untouched;
  - authoritative declarations (`conformance.home.k3s/R-*` tags, Kyverno policies,
    ServiceDefinitions) all live in YAML, on lines a real change must touch;
  - the coarse path->requirement fallback is unchanged and file-type agnostic, so
    the doc subtrees that matter (conformance/, schemas/, catalog/, clusters/, ci/)
    stay covered either way.

Alternatives rejected: adding the five IDs to DOCUMENTED_GAPS genuinely weakens the
gate; scrubbing requirement IDs out of prose (precedent f9e9640) would mean editing
the decision log's own history to appease a matcher.

Gate semantics change — flagged in DECISIONS-LOG.md for review.

Also drops three R-TEST-06 mentions this branch had added to prose and refreshes
the now-stale "gate is RED" notes.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-27 00:06:28 -05:00
jakman85andClaude Opus 5 12c40d8c9c docs: reconcile the documentation with the deployed reality
The docs had drifted badly behind ~85 commits of delivery. The root README still
said "Phase 0 — Foundations. No cluster exists yet"; the infra docs still
described a libvirt/KVM cluster that has been retired for two days; and
DECISIONS-LOG.md — the PM's authoritative record — stopped at 2026-08-25 AM.
Anyone following these docs would have been working from a false map.

Verified against the live cluster (3/3 nodes, 7/7 Flux Kustomizations Ready,
10/10 HelmReleases Ready) rather than inferred from commit messages.

README.md
  - Phase 0 -> a per-phase status table: 0-6 delivered and deployed, 7 not open.
  - Layout now shows the wave split, platform-config, apps/vms, infra, bootstrap.
  - Real conformance number (36/51, 70.6%) and how to regenerate it.
  - Devcontainer workflow; note that the gate was red and why.

infra/PHASE1-STATUS.md — rewritten. Was a `virsh`/libvirt resume guide for a
  cluster that no longer exists. Now: the Proxmox substrate, the pfSense NAT
  paths, both kubectl access paths (Keycloak OIDC normal / cert break-glass),
  the toolchain, off-repo state that must not be lost, and the fact that Flux's
  git source is an app inside the cluster (`git push origin main` IS the deploy).

infra/PHASE1-SCOPE.md — historical banner. The exit criteria and sizing
  rationale are still worth keeping; the libvirt execution steps are not.

infra/README.md — Proxmox, plus a map of the subtree and where the cluster
  actually lives (it is not on this workstation).

infra/devcontainer/README.md — the "run tofu and ansible on the HOST" limitation
  is OBSOLETE: it was a dmacvicar/libvirt polkit/PID problem, and bpg/proxmox
  talks to an HTTP API. Everything runs in the container now. Documents the full
  CI-gate toolchain and what each bind mount is load-bearing for (the age keyring
  and the read-only spec repo especially — without the latter, spec-immutability
  silently skips instead of verifying).

infra/tofu/README.md — was "Authoring only — nothing here has been applied". It
  is applied and live. Disk sizes corrected 40/100 -> 64/128 GiB (confirmed
  against the running nodes), pfSense NAT marked done, and a warning that
  terraform.tfstate is absent on a fresh workstation — an apply with empty state
  would try to recreate the three live nodes.

infra/tofu/variables.tf — section comment disagreed with its own defaults.

infra/cluster.sh — retired-with-instructions header. It silently drives libvirt
  domains that do not exist; behaviour unchanged, but it no longer lies.

apps/vms/unifi/README.md — was "NOT WIRED IN", 60Gi, and listed a file that has
  since been dropped. Now: live status, 40Gi, resolved open questions marked
  resolved, and the two constraints the next person will otherwise rediscover —
  KubeVirt's 2048B inline userData cap, and that apps-vms must never carry
  postBuild.substituteFrom (envsubst corrupts the cloud-init).

DECISIONS-LOG.md — backfilled the gap as one structured entry: the Proxmox
  migration, NAT + kube-vip, the twice-proven oauth2-proxy cold-start deadlock,
  the Flux source cutover, CI going live in-cluster, ten app onboardings and the
  renderer bugs they exposed, the wave split (verified: DBs adopted, not
  recreated — original ages intact, prune restored), KubeVirt storage, the UniFi
  VM, and kubectl SSO. Plus verified live state and the carried-forward open list.

Two corrections to the record while verifying:
  - kubectl OIDC is LIVE (`kubectl auth whoami` -> oidc:<user>,
    [oidc:platform-admins]), so the rolling restart 3472f1d deferred has
    happened.
  - UniFi OS Server is NOT installed in the guest. The VM is healthy (Ubuntu
    24.04.4, guest agent connected, live-migratable) but :11443 answers nothing,
    and the CNP explicitly permits cluster -> :11443, so it is genuinely not
    listening rather than a policy artifact.

Docs and comments only; no manifest semantics change.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-08-27 00:00:05 -05:00
jakman85andClaude Opus 4.8 017ed0fb9c Phase 1 checkpoint: status/resume doc + cluster helper + toolchain note
Cluster is UP (3/3 Ready, Cilium OK); Phase 1 paused before Flux bootstrap
(Option A: suspend platform/apps until Phase 2). Add infra/PHASE1-STATUS.md
(what's done, state file locations, reconnect steps, remaining T-1.4/1.5,
Phase-2 platform prereq), infra/cluster.sh (status/start/stop/kubeconfig), and
document that tofu+ansible run on the host (libvirt provider limitation).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 02:18:39 -05:00
jakman85andClaude Opus 4.8 5d2b248b68 Phase 1 toolchain: pinned dev container (prereqs not all Python)
Per operator rule: OpenTofu/age/sops/flux are Go binaries, so bake the toolchain
into a pinned Fedora-44 dev container (matches host libvirt 12) instead of a venv.
infra/devcontainer/: Containerfile + versions.env (tofu 1.9.0, age 1.2.1, sops
3.9.4, flux 2.4.0, kubectl 1.32.0, helm 3.16.0) + dev.sh (runs with host libvirt
socket + repo + ssh mounted) + README. Built & validated: all tools resolve;
virsh reaches host libvirt from inside the container.

Also record resolved Phase 1 decisions in PHASE1-SCOPE.md (6GiB/4vCPU x3;
k3s-dev NAT 10.73.0.0/24; container toolchain). Gate green.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
2026-08-16 00:59:47 -05:00