jakman85andClaude Opus 5
21e111b08c
Operator report: dev.sh only reached the dev cluster. Line 72 hardcoded
`KUBECONFIG=${DEV_KUBECONFIG:-…/dev.kubeconfig}`, and the only escape was an
undocumented override whose NAME misleads — you set `DEV_KUBECONFIG` to a *prod*
path. infra/PHASE1-STATUS.md had recorded this as a "Known gap" for months.
WHY THIS WAS NOT A ONE-LINER. Both kubeconfigs name their context, cluster AND
user `default`. kubectl's merge keeps the FIRST occurrence of a duplicated name,
so the obvious `KUBECONFIG=dev.kubeconfig:prod.kubeconfig` resolves `default` to
DEV and `--context default` can never select prod — an operator or agent would
believe they were on prod and be issuing commands against the LIVE dev cluster
that runs the operator's platform. Same collision class as ~/.kube/config an hour
earlier, and worse here because agents run in this container.
Interface: `CLUSTER=dev|prod` as an env prefix, default **dev** (back-compat: every
existing invocation, CI job and habit keeps meaning dev).
infra/devcontainer/dev.sh run kubectl get nodes # dev
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # prod
Its one weakness — `CLUSTER` can be inherited from an exported shell var — is
covered by a banner printed on EVERY invocation before the container starts:
`==> dev.sh: cluster=prod (kube context k3s-prod)`.
New infra/devcontainer/kubeconfig.sh generates a merged config with DISTINCT names
(contexts k3s-dev/k3s-prod, users k3s-*-admin), renaming structurally with `yq`
on .clusters[].name / .users[].name / .contexts[].{name,context.cluster,
context.user} before letting `kubectl config view --flatten` merge them. NOT sed —
sed renaming corrupted a config earlier today because cluster and user entries are
both `- name: default` at identical indent. Output is 0600 in the gitignored
.cache/, regenerated every run, and post-conditions assert the current context is
`k3s-<target>` and that NO context named `default` survived.
FAILS LOUDLY AT BOTH LAYERS, and never falls back to the other cluster — that
fallback is exactly how a prod command ends up hitting dev. Host-side refusal if
the requested cluster's kubeconfig is missing; in-container refusal likewise;
unknown CLUSTER exits 2. A missing source for the cluster you did NOT ask for is
tolerated (merges the other one only).
VERIFIED BY ME, independently of the agent — the two BASE_DOMAIN reads are the
unambiguous proof of which cluster was reached, since a node list is not:
default (no selector) -> home.keithjackson.dev (DEV, unchanged)
CLUSTER=prod -> prod.home.keithjackson.dev (PROD)
contexts -> k3s-dev, k3s-prod; NO `default` context survives
exit codes -> invalid CLUSTER=2, valid dev/prod=0
standup.sh -> `--kubeconfig` still wins over the new default
(preflight against prod: target confirmed 192.168.2.9)
credentials -> dev.kubeconfig and prod.kubeconfig byte-intact and each
still reaching 3 nodes after the agent's move-aside test
git -> no kubeconfig committable; clusters.kubeconfig ignored
Full CI gate PASS.
NOT VERIFIED: `dev.sh build` and the interactive shell path (existing image reused,
no TTY available). The shell path shares run()'s code; only the TTY differs.
DEV_KUBECONFIG still wins verbatim, now with a deprecation warning. All existing
mounts are untouched: SOPS age keyring rw, ~/.ssh ro, ansible-vault-pass ro, spec
repo ro + SPEC_REPO, --network host, label=disable.
Closes the PHASE1-STATUS "Known gap".
Refs R-ENV-02, R-TEST-02.
Co-Authored-By: Claude Opus 5 <[email protected]>
89 lines
5.2 KiB
Markdown
89 lines
5.2 KiB
Markdown
# home-network-k3s-impl
|
||
|
||
Implementation of the home k3s + Longhorn platform. This repo holds the **reconciled artifacts**
|
||
(Flux config, Kustomize overlays, KubeVirt/Helm manifests, ServiceDefinitions, Kyverno policies,
|
||
Chainsaw conformance tests, CI). It is driven by — and subordinate to — the **read-only
|
||
specification** at [`../home-network-k3s`](../home-network-k3s).
|
||
|
||
## Rules (from the spec — non-negotiable)
|
||
- **The spec is read-only.** Never modify anything under `../home-network-k3s`. Implement against it;
|
||
escalate spec problems, don't edit them (spec `AGENTS.md` §11, R-TEST-06).
|
||
- **Done = a green, reproducible, requirement-tagged conformance test** — never "merged," never a
|
||
manual demo (spec `docs/14`, `agents/definition-of-done.md`).
|
||
- **No work on `main`.** Every change is a feature branch merged into `main` (spec `CONTRIBUTING.md`).
|
||
- **No plaintext secrets** — SOPS-encrypted only (spec `docs/07`).
|
||
- **Phase discipline** — only build what the active roadmap phase authorizes (spec `roadmap/phases.md`).
|
||
|
||
## Layout
|
||
```
|
||
infra/ node provisioning: OpenTofu (Proxmox VE) + cloud-init + Ansible, and the
|
||
pinned dev container that carries the whole toolchain
|
||
bootstrap/ one-time cluster bootstrap: Flux install, SOPS/age key, host Gitea
|
||
platform/ platform CONTROLLERS (Flux HelmReleases / operator bases): kyverno, cilium,
|
||
longhorn, external-snapshotter, cert-manager, externaldns, gateway, coredns,
|
||
vpn-egress, monitoring, cnpg, keycloak, oauth2-proxy, kubevirt
|
||
platform-config/ platform CONFIGS — the custom resources of the CRDs installed above
|
||
(ClusterIssuers, Gateway, Kyverno policies, KubeVirt/CDI CRs, Keycloak CRs).
|
||
Split out because CRs and the CRDs they use cannot apply in one pass.
|
||
apps/ application tier, reconciled as dependency-ordered WAVES:
|
||
wave-0/ wave-1/ wave-2/ rendered container apps (see DESIGN-waves.md)
|
||
vms/ KubeVirt VirtualMachines (non-gating tier)
|
||
catalog/ ServiceDefinition entries (the onboarding surface) + render.py
|
||
schemas/ ServiceDefinition JSON Schema + renderer contract
|
||
clusters/ Flux entrypoints + dev/ and prod/ overlays
|
||
conformance/ Chainsaw behavior/conformance suite + harness + coverage report
|
||
ci/ the gate as scripts (lint · schema · Kyverno L0 · L1 render · decision)
|
||
.gitea/ Gitea Actions CI running that gate on an in-cluster runner
|
||
```
|
||
|
||
## Status
|
||
|
||
Roadmap phases **0–6 are delivered and deployed** on the dev cluster; **Phase 7 (prod) is not
|
||
open** — `clusters/prod` is a design-only overlay.
|
||
|
||
| Phase | Delivered |
|
||
|---|---|
|
||
| 0 Foundations | ServiceDefinition schema + renderer, Kyverno L0 gate, conformance harness, CI |
|
||
| 1 Cluster bring-up | 3-node k3s HA on **Proxmox VE** (`192.168.1.50/.51/.52`), Cilium, Flux, SOPS |
|
||
| 2 Platform core | Cilium · Longhorn · cert-manager (LE-prod wildcard) · ExternalDNS · Gateway API · split-horizon CoreDNS |
|
||
| 3 Egress / VPN | ADR-0015 node-level OpenVPN split-tunnel + Cilium egress-gateway steering + host kill-switch |
|
||
| 4 Observability | kube-prometheus-stack · Loki · Alloy · Alertmanager. **Velero still blocked on OQ-BKP**; image-automation authored but disabled |
|
||
| 5 IaaS | KubeVirt + CDI on real nested KVM; live-migratable RWX VM storage. The UniFi VM runs, but **UniFi OS Server is not yet installed in the guest** |
|
||
| 6 Identity + CI | Keycloak (Operator + CNPG) · oauth2-proxy forward-auth · Grafana OIDC · **kubectl OIDC SSO with group→RBAC** · in-cluster Gitea Actions merge gate |
|
||
|
||
**Conformance: 36 / 51 requirements covered (70.6%)**, 15 gaps, 1 safety-critical gap
|
||
(`R-STATE-02`, Longhorn node-loss). Regenerate with `./conformance/run.sh --coverage-only`;
|
||
the full report is `conformance/_artifacts/coverage/coverage.md`.
|
||
|
||
The running narrative — decisions, root causes, incidents, follow-ups — is
|
||
[`DECISIONS-LOG.md`](DECISIONS-LOG.md). It, not this table, is the authoritative record.
|
||
|
||
## Working on this repo
|
||
|
||
Everything the gate needs is baked into the pinned dev container — **do not install the
|
||
toolchain on the host**:
|
||
|
||
```bash
|
||
infra/devcontainer/dev.sh build # once
|
||
infra/devcontainer/dev.sh run make -C ci gate # the same gate CI runs
|
||
infra/devcontainer/dev.sh # interactive shell (sops, flux, kubectl, tofu, ansible)
|
||
```
|
||
|
||
`kubectl`/`flux`/`helm` in there talk to the **dev** cluster by default. Prefix `CLUSTER=prod` to
|
||
target prod instead:
|
||
|
||
```bash
|
||
infra/devcontainer/dev.sh run kubectl get nodes # dev (default)
|
||
CLUSTER=prod infra/devcontainer/dev.sh run kubectl get nodes # prod
|
||
```
|
||
|
||
Both clusters are always present in the container's `KUBECONFIG` as the contexts `k3s-dev` and
|
||
`k3s-prod`; `CLUSTER` picks which one is current, and each run prints the cluster it is targeting.
|
||
|
||
The container mounts the repo at `/work`, your age keyring, and the sibling spec repo read-only
|
||
so the spec-immutability check can actually run. See
|
||
[`infra/devcontainer/README.md`](infra/devcontainer/README.md).
|
||
|
||
A green run prints `MERGE GATE: PASS`. The L2/L3 conformance layers stay `SKIPPED` unless
|
||
`CLUSTER_AVAILABLE=true` — the static gate never pretends to have run them.
|