a984442cae1884bfc84faa47c4272e6dd918fcaf
59
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  jakman85andClaude Opus 5
|
a984442cae |
harden: pin node timezone to UTC, and TEST the guard/value coupling that bit us
Two debt items, both about making a latent problem impossible rather than fixed. === 1. Node timezone pinned to Etc/UTC (node_baseline, `system` tag) === The prod nodes disagreed, measured from k3s's own etcd snapshot annotations: prod-1/2 at -05:00 (Central), prod-3 at -04:00 (Eastern). Functionally harmless — Kubernetes and etcd are UTC internally, and in UTC those snapshots are ~4 minutes apart, which is correct staggering — but it makes cross-node log correlation actively misleading, and this cluster has had two unplanned reboots in two days. `Etc/UTC` chosen on repo convention, not just principle: catalog/plex.yaml records "no other service in the repo sets a timezone ... so use the safe default" with TZ: Etc/UTC, and apps/wave-0/plex.yaml matches (platform/crafty is America/Chicago DELIBERATELY — a user-facing game schedule, not a log clock). Decisive extra fact: THE THREE DEV NODES ARE ALREADY Etc/UTC, so this makes prod match dev rather than inventing a convention. UTC also removes the class — no DST discontinuity, node time == platform time. New knob `node_baseline_timezone`, overridable per environment via group_vars (the same seam as node_baseline_backports_kernel) but deliberately NOT restated there: no environment needs to differ, and a second copy is a second thing to disagree. Guarded by an assert in the role's existing BLOCKED style before the module runs. `--check --diff --tags system -e k3s_hosts=k3s_prod` INDEPENDENTLY CONFIRMS the offset evidence, and is the whole verification — the play was never run for real: k3s-prod-1: -"America/Chicago" +"Etc/UTC" changed k3s-prod-2: -"America/Chicago" +"Etc/UTC" changed k3s-prod-3: -"America/New_York" +"Etc/UTC" changed The same --check against DEV reports the task `ok` on all three — idempotence demonstrated against live hosts, in check mode only. `--list-hosts` confirms `-e k3s_hosts=k3s_prod` selects the three prod nodes (the diagnostic plays take their OWN var; the wrong flag silently targets dev). *** AUTHORED, NOT APPLIED. *** Applying it is a live-cluster action and was deliberately left for the operator to schedule: prod's Longhorn had only just finished resyncing after a node reboot, and changing a node's clock config during storage recovery is not an unattended move. `ansible-inventory --list` was NOT used anywhere — it prints DECRYPTED vault values (the passphrase is mounted into the dev container) and has leaked the prod become-password to a transcript once already. === 2. New L0 check `placeholder-guard-coupling` (R-ENV-02, R-STATE-03) === On 2026-09-05 the PM changed a placeholder in clusters/base/cluster-vars.yaml from `TBD-PROD-S3-BACKUP-ENDPOINT` to `https://TBD-R2-ACCOUNT-ID...`. The guards in bootstrap/standup.sh matched the glob `TBD-*` — a PREFIX. All three silently died, and `resume` would have enabled velero on prod against a nonexistent hostname: installs clean, reports healthy, no backups exist. Found by accident. A guard in one file keyed on the SHAPE of a value in another file, with nothing tying them. The check exercises the REAL predicate — it does not reimplement it. standup.sh is not sourceable (top-level argv parsing hard-exits, cd, export, EXIT trap), so the check extracts the predicate's own source lines and sources only those. Faithfulness is pinned by: discovery of every `*_is_placeholder` (not a hardcoded name), exactly-one-definition, "extracted block is a verbatim substring", a self-containment gate, a canary (must refuse "", must accept a real URL), and a scan for a divergent second copy. If the predicate grows a dependency the check FAILS LOUDLY and names the remedy rather than degrading to testing a copy — the failure mode it exists for. Both sides derive from real files: guarded knobs from a taint trace in standup.sh (`x="$(cvar KEY)"` -> alias hop -> `*_is_placeholder "$x"`); values from base cluster-vars plus each overlay's JSON6902 /data/KEY patches. The independent detector is deliberately NOT the guard: seed idioms UNION the guard's own words with glob anchors STRIPPED — so a prefix-anchored guard cannot pass, and deleting an arm cannot make a value stop being placeholder-shaped. Real tree: 32 assertions, 18 values through the extracted predicate, PASS. NON-VACUITY VERIFIED BY ME, twice, on throwaway copies (the real repo and the read-only spec repo untouched). My first perturbation tripped the self-containment rail instead of the anchoring property, so I redid it shape-preserving — substring globs -> prefix globs, nothing else — and it failed on exactly the right property, naming mid-string and tail acceptances with the diagnosis "This is the 2026-09-05 failure exactly". Control: unperturbed copy exits 0. === 3. ci/touched-requirements.sh: bootstrap/ was UNMAPPED (found by the agent) === The path->requirement map had no `bootstrap/*` entry, so a change touching standup.sh ALONGSIDE any already-mapped path scoped the L0 run to that other path's requirements and silently excluded the new check — the one that exists to catch exactly the 2026-09-05 class. A safety net with a hole where the incident was. Mapped to R-STATE-03 R-GITOPS-01; verified the map now resolves bootstrap paths. VERIFIED: full CI gate PASS. `run.sh --layer l0` 8/8 PASS including the new check. Coverage unchanged at 45/51 (the new check tags existing requirements). Refs R-ENV-02, R-STATE-03, R-TEST-01, R-PLAT-01. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
fe45d0f5c5 |
feat(conformance): cover R-TLS-02, R-ENV-01, R-ENV-03 — 42/51 -> 45/51, and find a real gap
DECISIONS-LOG records Phase 7 being opened knowingly while the uncovered set included R-ENV-01/02/03 — the very requirements prod provisioning serves. Two of those three are now covered, plus R-TLS-02. tls-lifecycle-automated R-TLS-02 L0 (in the merge gate) cert-lifecycle-automated R-TLS-02 L2 (authored, NOT executed) dev-nodes-provisioned-as-code R-ENV-01 L0 prod-baremetal-not-dev-surface R-ENV-03 L0 Coverage 42/51 (82.4%) -> 45/51 (88.2%), 6 gaps, 0 safety-critical. Remaining: R-C2-01, R-C2-02, R-NET-04, R-PLAT-01, R-SEC-03, R-STATE-04. Verified through the real runner: --layer l0 7/7 PASS, the three new ones 3/3 PASS, --strict-coverage exit 0 with 0 label/annotation mismatches. *** R-TLS-02 "REVOKED" IS NOT SATISFIED — A REAL GAP, NOT A TEST DEFECT. *** The spec requires certificates be "issued, renewed, AND REVOKED without manual steps", and docs/08 reads revocation as "removing/rotating a certificate is a declarative change". That needs enableCertificateOwnerRef: true; platform/cert-manager/helmrelease.yaml does not set it (chart default false). Verified read-only on BOTH live clusters: gateway/wildcard-home-tls has ZERO ownerReferences, so deleting the Certificate would leave the private key serving indefinitely. The L2 test asserts the requirement AS WRITTEN and will fail until this is decided. It was deliberately NOT softened — weakening a test to match the implementation is how a suite stops meaning anything. NOT FLIPPED UNATTENDED, and the reason is specific rather than caution: the wildcard Certificate lives under ./platform, which runs prune: true. With owner refs ON, any transient prune of that Certificate — exactly what happened to the CDI StorageProfiles hours ago — would delete the SERVING TLS Secret for every host, and reissuance is bounded by Let's Encrypt's duplicate-certificate limit (5/week per identical name set), which could leave a cluster with no trusted cert for a week. Security gap vs a plausible week-long TLS outage is an operator call. NON-VACUITY DEMONSTRATED, which is the part that makes these worth having: 7/7 perturbations fail for R-TLS-02, 8/8 for R-ENV-01, 13/13 for R-ENV-03, and each restored copy passes. Perturbations were applied to a throwaway copy; neither this repo nor the read-only spec repo was modified. TWO CHECKS INITIALLY PASSED WHEN PERTURBED and were rewritten — the exact degrade-to-no-op failure this repo keeps hitting. The first draft asked "is this file's DIRECTORY referenced by a kustomization", which a directory reference always satisfies, and keyed Secrets by name alone, so external-dns's copy of `cloudflare-api-token` satisfied cert-manager's reference. Now a recursive kustomize resource-graph walk from each Flux path, and Secrets keyed by (namespace, name). Both bite. SECURITY NOTE, recorded because this project has already leaked a password once: `ansible-inventory --list` PRINTS DECRYPTED VAULT VALUES — it emitted the prod become-password to stdout during exploration, since the vault passphrase file is mounted into the dev container. Both new checks use only --syntax-check and --list-hosts, and say so in a comment. Confirmed no plaintext value reached disk: vault.yml is still AES256-encrypted and the tree has no assigned value. WHAT IS NOT PROVEN, per requirement: R-TLS-02 "revoked" (above), and issuance + observed renewal await the unexecuted L2. R-ENV-01 cannot prove the running VMs came from this module (needs OpenTofu state) — it proves the four lifecycle stages exist as code and cover every dev node, and that the chain RESOLVES under real ansible-playbook. R-ENV-03 cannot prove the HARDWARE (the K8s API exposes no "am I virtual" fact; mini-PC/bare-metal is operator-attested) — it proves prod is AUTHORED as 3 directly-reached nodes with no IaC creating them, sharing dev's playbooks and roles, with no prod-only manifest, branch or role anywhere. VERIFIED BY ME: coverage delta and the 3/3 L0 pass through the real runner; the zero-ownerReferences finding on both clusters; that the new checks avoid ansible-inventory; that no credential material reached disk. Full CI gate PASS. Refs R-TLS-02, R-ENV-01, R-ENV-03, R-TEST-01..05. Co-Authored-By: Claude Opus 5 <[email protected]> |
||
|
jakman85andClaude Opus 5
|
59ffbaf11a |
feat(conformance): L2 tests derive their environment from the cluster, not a literal
Six L2 behavioural tests embedded `home.keithjackson.dev`. They are test INPUTS,
not reconciled manifests, so postBuild substitution never touched them — correct
while dev held the apex, wrong the moment prod came up on
prod.home.keithjackson.dev, and blocking §9's "suite passes green against prod".
Each test now asks the cluster what it is, via a test-level chainsaw binding over
flux-system/cluster-vars — the SAME ConfigMap Flux renders the platform from, so
a test cannot disagree with the cluster it runs on:
bindings:
- name: baseDomain
value: (x_k8s_get($client,'v1','ConfigMap','flux-system','cluster-vars').data.BASE_DOMAIN)
Chosen over a flag or env var because there is nothing to forget at run time, and
test-level scope covers every step (step outputs/bindings are step-scoped).
THE APEX WAS NOT THE ONLY LEAK. dns-exposure and dns-internal-resolution also
hardcoded DEV'S Cilium LB range 192.168.1.53-.60; prod's is 192.168.2.20-.30, so
both were wrong for prod regardless of hostname. Replaced with assertions on the
OBSERVED value — the DNS answer must equal that cluster's GATEWAY_VIP, and the
public record must equal the LB IP the Service was actually assigned — which is
strictly STRONGER than the old range check as well as environment-independent.
sso-forward-auth held the literal only in prose (it talks over Service DNS).
PROVEN, NOT ASSERTED — and the positive control is the important half:
* expressions extracted verbatim from the tracked files resolve to the correct,
DIFFERENT values per cluster (dev home.keithjackson.dev / 192.168.1.53,
prod prod.home.keithjackson.dev / 192.168.2.20);
* the same assert with the HARDCODED apex passes on dev and FAILS on prod
("Invalid value: *.prod.home... Expected: *.home..."), so the harness is
provably not blind — that is exactly the drift removed;
* chainsaw lint VALID on all six; run.sh --lint clean over the whole tree.
END-TO-END L2 RUN AGAINST LIVE DEV: `dns-exposure` PASSED (42.6s) — the binding
produced conformance-dns.home.keithjackson.dev, external-dns published it, and the
probe Job asserted the record against the assigned LB IP. This is the first time
one of these tests has been executed since parameterization, and it exercises the
full path rather than just the binding.
FINDING FROM THAT RUN, worth recording: the test's teardown is ASYNCHRONOUS. It
reports PASS as soon as chainsaw deletes the namespace, but the Cloudflare record
is reaped later by external-dns's own --interval=1m sync. A zone query immediately
after PASS still showed conformance-dns.home.keithjackson.dev plus its owner=default
TXT registry entry. Harmless here, but it means a green dns-exposure does NOT prove
cleanup happened, and if external-dns were suspended the record would persist
silently. Verified separately that it clears.
conformance/incluster/rbac.yaml: its cluster-wide `get configmaps` is now
LOAD-BEARING for these bindings — narrowing it to the test namespace breaks five
tests. Commented so it is not "tidied" later.
l3-fault-injection/dr-restore: its KNOWN-GAPS block said off-host is unproven
because the MinIO shares a Proxmox host with dev's VMs. Still true as run, so the
text now says both — that the R2 move (
|
||
|
jakman85andClaude Opus 5
|
c8c1d2935d |
feat(dns): per-environment BASE_DOMAIN — prod moves to its own subdomain
INCIDENT FIX. Prod's external-dns DELETED EVERY ONE OF DEV'S PUBLIC DNS RECORDS. Both
clusters ran policy: sync against the SAME Cloudflare zone with the SAME implicit
txtOwnerId (`default`), so prod believed it owned dev's records, found no matching sources,
and removed them plus their TXT registry entries. Dev recovered ~30s after prod was scaled
to 0; verified restored against Cloudflare directly (my first check went through a pfSense
override and falsely showed healthy — local resolvers are not evidence about public DNS).
THE MISS WAS MINE, and it was a failure of imagination in how I briefed every agent. I put
"never touch dev" in brief after brief and enforced it — separate kubeconfigs, verified
--limit, sha-checked dev's kubeconfig before and after. Every one of those guards held. NONE
of them covered SHARED EXTERNAL RESOURCES. Prod reconciling its OWN correctly-authored
manifests reached across a third party and destroyed dev's state. Cluster-to-cluster
isolation was verified; the dependency that mattered was cluster-to-Cloudflare.
Operator's architecture: dev -> *.dev.home.keithjackson.dev, prod ->
*.prod.home.keithjackson.dev, apex CNAME <svc>.home -> <svc>.prod.home.
PHASE 1 ONLY (this commit): prod moves to its own subdomain. DEV IS NOT MIGRATED and its
effective hostname stays home.keithjackson.dev, so dev renders byte-identically and nothing
live moves. That alone ends the collision and unblocks prod. Phase 2 (dev -> dev.home. +
apex CNAMEs) is deferred to when the operator is present: it relocates Keycloak's issuer and
12 clients' redirect URIs, forces a rolling k3s restart and kubectl SSO re-issue for every
user, and moves gitea.home. — which is BOTH the GitOps source and the current push target.
BASE_DOMAIN WAS A DEAD KNOB — it existed in cluster-vars but the status report flagged it as
referenced by nothing, because hostnames were hardcoded throughout. Now real: 141 refs.
Searched the AUTHORITATIVE reconciled set by resolving each Flux spec.path transitively via
conformance/lib/status.py — the same resolver the L0 gate uses, so "what Flux reconciles"
has one definition. 185 occurrences across 47 files in 154 reconciled files. Tokenized:
Gateway listeners, the wildcard Certificate, coredns-custom, external-dns, Keycloak
instance/realm (26 redirect URIs), oauth2-proxy, Grafana OIDC, all HTTPRoutes, Homepage's 15
links, 30 runbook_urls, unifi cloud-init, render.py's ZONE constant (which drove
--oidc-issuer-url/--cookie-domain/--whitelist-domain on every forward-auth sidecar) and 11
catalog/ ServiceDefinitions re-rendered canonically. Left alone: 43 comment/prose
occurrences and non-reconciled paths.
TWO THAT CANNOT BE TOKENIZED, flagged not forced:
- headlamp's OIDC URLs live inside a SOPS-encrypted Secret; re-encrypting rewrites the MAC
over every value and the OFFLINE harness cannot verify it (kustomize does not decrypt),
so an error would break live-dev SSO with no pre-flight proof. Prod needs its own copy
anyway — client secrets are realm-scoped.
- kube-apiserver --oidc-issuer-url is a k3s SERVER FLAG, not a manifest, so Flux can never
reach it. Done on the inventory seam instead (k3s_base_domain -> k3s_oidc_issuer_url,
overridden in group_vars/k3s_prod). Verified per host: dev keeps the default, prod gets
prod.home.
EXTERNAL-DNS OWNER, the actual incident cause: txtOwnerId is now explicit per environment —
dev renders exactly `default` (the chart default it was already using, so its existing
records stay recognised) and prod gets `prod`. The BASE default is a NON-COLLIDING SENTINEL,
deliberately not `default`, so a future environment that forgets the knob owns nothing and
can delete nothing rather than silently inheriting dev's identity.
I OVERRODE ONE AGENT DECISION against our own prior finding: it narrowed prod's
domainFilters to prod.home.keithjackson.dev as defence in depth. FINDING F-DNS1
(DECISIONS-LOG 2026-08-17) already proved a subdomain filter DROPS THE PARENT ZONE —
ExternalDNS generates the record then skips it with "no hosted zone matching record DNS
Name". Prod would have published NOTHING. Set to the ZONE (keithjackson.dev) like dev.
Isolation comes from the subdomain split plus txtOwnerId, which is what the incident
actually needed and what a domain filter cannot provide.
DEV PROVEN UNCHANGED: 11 of 12 substituting paths byte-identical; the 12th differs by
exactly one line, `+ txtOwnerId: default` — the value dev was already using implicitly.
POSITIVE CONTROLS confirm the harness is not blind: perturbing BASE_DOMAIN moves 10 of 12
hashes (the 2 that don't provably contain no hostname), perturbing the owner id moves
exactly 1. Prod's rendered output has ZERO home.keithjackson.dev references outside prod.,
and its cert renders *.prod.home.keithjackson.dev — no overlap with dev's.
L0 substitution assertions 106 -> 254; BASE_DOMAIN is no longer flagged dead. Gate PASS.
PROD REMAINS SUSPENDED (platform + 10 downstream tiers) pending review; nothing applied.
Serves: R-ENV-02, R-DNS-01, R-DNS-02, R-DNS-03, R-TLS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
380ea98a38 |
feat(conformance): R-STATE-03 PROVEN — DR restore from off-cluster backup, executed
The second requirement ever proven on this project, and the one the operator had attested was working but the suite could not attest to. Test authored 2026-08-31 carrying a `conformance.home.k3s/never-executed` annotation; executed for the first time tonight. test dr-restore run 20260902T001846Z-fdc3265 req R-STATE-03 (safety-critical, DR) junit passed / 0 failed / 0 skipped sentinel DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK proof proven sha |
||
|
jakman85andClaude Opus 5
|
fdc3265585 |
fix(conformance): the ledger counted as evidence FOR the requirements it proves
Committing the first PROVEN row immediately degraded it to `stale-proof`. Cause: a self-defeating loop. `IMPL_EXCLUDE_PREFIXES` excluded `conformance/tests/` and `ci/tests/` but NOT `conformance/runs/`. The ledger and the preserved run logs CONTAIN the requirement IDs they are evidence FOR — `ledger.jsonl` literally holds `"requirements":["R-STATE-02"]` — so they were scanned as IMPLEMENTING evidence for R-STATE-02. That made `last_change` for the requirement move to the ledger commit itself, which the row's `git_sha` necessarily predates, so the freshness check failed. The consequence was total, not marginal: **recording a proof destroyed it, so NO ledger row could ever have been non-stale.** The ledger would have reported `PROVEN: 0 (stale N)` forever while looking like it was working — the same shape of silent-uselessness as the COMMAND-echo bug in the previous commit, and found the same way: by actually running the thing and disbelieving a green-looking result. Fix: exclude `conformance/runs/`. The ledger is the proof RECORD; it must not participate in the freshness computation about itself. Verified: `PROVEN: 1 (stale 0, self-skipped 0, failed 0, unknown 0) from 1 ledger row(s)`; R-STATE-02 reads **proven**. status.py --selftest PASS. The staleness machinery is otherwise untouched — a genuinely stale row (impl changed after the run) still degrades, which the selftest continues to assert. Serves: R-TEST-01..04. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85
|
4cb5e7a8d4 |
feat(conformance): first PROVEN row — R-STATE-02 node-loss
Records the first requirement this project has ever PROVEN, in the only sense its own
definition of done accepts: a green, requirement-tagged, reproducible behaviour test,
executed against a real cluster, with the evidence committed.
test node-loss run 20260901T235117Z-e9a4b66
req R-STATE-02 (safety-critical) junit passed
sentinel NODE_LOSS_VOLUME_AVAILABLE_OK proof proven
sha
|
||
|
jakman85andClaude Opus 5
|
57d4df8c4f |
fix(conformance): ledger read tokens from Chainsaw's COMMAND echo, not real output
The run ledger mis-classified the FIRST real L3 execution — a genuine PASS recorded as
`skipped`. Diagnosis, from the preserved log, section-classified:
line 51 COMMAND echo NODE_LOSS_SKIPPED_OK <- script SOURCE
line 190 STDOUT NODE_LOSS_VOLUME_AVAILABLE_OK <- the only REAL emission
Chainsaw echoes each script into a `=== COMMAND` block before running it, and every
skip-gated test contains `echo <TEST>_SKIPPED_OK` as source text. record_run.py scanned the
raw log without distinguishing sections, so it matched the skip token in the echo.
NODE_LOSS_SKIPPED_OK never appeared in output at all.
SEVERITY: since every skip-gated test embeds its own skip token as source, and skip outranks
pass, THE LEDGER COULD NEVER HAVE RECORDED `proven` FOR ANY TEST WITH A SKIP GATE. It would
have silently recorded nothing forever while appearing to work. It failed CLOSED — the right
direction — but a ledger that always fails closed is an expensive way to write "unknown".
FIX: classify_sections() splits the log on indented `=== NAME` markers (a section runs to the
first non-blank line indented LESS than its marker). Verified against the real log's actual
markers, including that Go's column-0 `=== RUN/PAUSE/CONT` lines carry a test name and must
never open a section. Then:
PASS tokens are read from STDOUT ONLY — exactly where Chainsaw's own contains($stdout,…)
looks, so a source echo or stderr can never manufacture a proof.
SKIP tokens are read from every section that is NOT COMMAND, so an unknown future section
name fails closed in BOTH directions.
Lines outside any section prove nothing — this also closes a second false-positive vector:
a FAILING Chainsaw run prints the assertion expression, which contains both tokens.
A log with no markers is scanned whole exactly as before (such logs have no COMMAND echo).
SECOND BUG, same root: _LOG_HEADER_RE never matched a real Chainsaw log — the lines are
` l.go:52: <BS>| 23:51:18 | <ANSI>node-loss<ANSI> | …`, not `| … |`. EVERY genuine run was
silently falling into DEGRADED whole-log mode. Now strips ANSI/backspace noise first and
accepts the Go source prefix, so runs attribute per-testcase as designed.
Safety properties re-asserted, not assumed: skip still beats pass WITHIN real output; unknown
layers still require a sentinel; JUnit failed/error/skipped still outrank everything; missing
JUnit or log still exits 3. Negative control: with scan_texts forced section-blind in a
throwaway copy, exactly the new cases fail — they do catch this bug.
Includes the preserved run directory 20260901T235117Z-e9a4b66 (report.junit.xml + run.log)
for R-STATE-02 / node-loss — 131s, cordon -> evict -> volume stayed available -> reader
re-attached -> clean teardown, the first L3 fault-injection test ever executed on this
cluster. The ledger row is recorded in the FOLLOWING commit, deliberately: git_sha is sampled
at record time and a dirty tree records `-dirty`, which is unresolvable and correctly degrades
to stale-proof. Recording against this clean commit is the honest way to get a resolvable sha
— NOT a --git-sha override, which would be a lever for recording a clean sha over a dirty
tree and was deliberately not added.
Serves: R-TEST-01..04, R-STATE-02.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
e9a4b669d4 |
feat(conformance): committed run ledger — the only thing that can move PROVEN off zero
status.py has always reported PROVEN: 0 for all 51 requirements, and named the fix in its
own header: "a committed, machine-readable run ledger". There was none — _artifacts/ is
gitignored, the root .gitignore kills *.junit.xml, and a stray JUnit was surfaced only as
a quarantined mtime advisory. Built BEFORE the first real L3 execution so that run is
captured rather than reconstructed from memory.
THE TRAP THIS EXISTS TO AVOID. These tests exit 0 and Chainsaw records `passed` on a
SELF-SKIP exactly as on a real run, and Chainsaw's JUnit carries no stdout — only
properties. So committing the JUnit as proof would let DR_RESTORE_SKIPPED_OK be recorded
as proof of R-STATE-03: authoritative-looking and wrong, which is worse than no ledger.
The sentinel token is the ONLY thing distinguishing proven from skipped, and it lives
solely in the run log. The ledger therefore records the LOG alongside the JUnit and
derives proof from the sentinel, never from the JUnit result.
Sentinel tokens are DERIVED, not hand-listed: record_run.py parses every
contains($stdout,'TOKEN') out of each test's own chainsaw-test.yaml, so the list cannot
drift from the tests. 23 of 35 tests declare no sentinel at all.
decide() is the single place `proven` is emitted, ordered WORST-FIRST:
JUnit failure/error -> failed
JUnit <skipped/> -> junit-skipped
ANY declared skip token in the log -> skipped (beats a green JUnit AND beats a pass
token present in the same log)
no sentinel declared -> proven ONLY for L0/L1 (whose JUnit is truthful on its own, since
run_l1_offline.py maps exit 3 to <skipped/>); written as an
explicit exempt-set so an unknown/blank layer fails CLOSED
EVERY declared pass token present -> proven
otherwise -> unknown-sentinel-absent
Log attribution is a state machine over Chainsaw's per-test headers; a log with no
recognisable header falls back to whole-log scanning restricted to GLOBALLY-UNIQUE tokens,
so DNS_OK (owned by two tests) can never prove either. record_run.py exits 3 if the JUnit
or the log is missing — nothing recorded, never green.
STALENESS: a proven row is honoured only while `git merge-base --is-ancestor` holds
between the requirement's last implementing/test change and the row's git_sha; otherwise
it degrades to stale-proof. A -dirty sha, an unknown rev, or an uncommitted test path all
fail ancestry the same way. A requirement reads proven only when EVERY verifying test
does (worst-of), and the latest row per test wins — not best-ever.
REFUSAL POSTURE PRESERVED: with an empty ledger the report is unchanged in intent — every
requirement not-derivable, headline "PROVEN: 0 — NOT DERIVABLE". This replaces "not
derivable" with real evidence only where evidence exists, never with optimism.
--record is explicit, never implicit (a run that silently records itself is how a bad run
becomes permanent false proof), is rejected for --status/--coverage-only/--lint, and
re-execs under tee returning ${PIPESTATUS[0]} so recording can never soften a gate. It
rm -f's a stale _artifacts/report.junit.xml first — a hazard hit live during development,
where a stale JUnit got paired with a fresh log.
Selftests: record_run.py 23 checks (incl. end-to-end Chainsaw-passed + NODE_LOSS_SKIPPED_OK
-> proof "skipped"), status.py +16 against a throwaway git repo (self-skip -> skipped;
proven-at-older-sha -> stale-proof; dirty/unresolvable -> stale-proof; worst-of;
latest-row-wins; malformed lines dropped). No regressions: --coverage-only, --status,
--lint, --strict-coverage and the merge-gate shape all rc=0 and unchanged; nothing in ci/
references status.py.
Root .gitignore needed no change — conformance/runs/ is not itself excluded, so the deeper
negations override the root's global *.junit.xml. Verified with git check-ignore.
Serves: R-TEST-01, R-TEST-02, R-TEST-03, R-TEST-04.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
8a2be824f6 |
feat(ci): substitution-safety gate + escape 6 comment hazards; strict mode is now SAFE
Adds conformance/tests/l0-static/flux-postbuild-substitution/ — an offline, stdlib-only
L0 check tagged R-ENV-02 + R-GITOPS-01 (deliberately NOT R-TEST-01, which would
overclaim). It reuses lib/status.py's `resources:` walk rather than forking it, so "what a
path renders" has exactly one definition. It asserts: every ${...} on a substituted path
is defined-or-escaped (comments included); no default-operator form on an undefined name;
no unparseable expansion; the rendering path actually substitutes cluster-vars; every
overlay-patched /data/KEY has a base default; and that the gate itself is load-bearing.
WIRED AS AN UNCONDITIONAL STAGE (3c) in ci/l0.sh, and that matters: stage 3b scopes to
touched requirements, and ci/touched-requirements.sh maps only clusters/* to R-ENV-02 —
so an edit to a substituted manifest under apps/ or platform/ would have filtered this
check out of exactly the change that introduces the bug. l0 fail -> BLOCK=1 in gate.sh.
Mutation-tested all five assertion classes (undefined token, indexed expansion, :- form,
missing substituteFrom, un-wired gate); each fails as intended, baseline green.
STRICT MODE IS PROVEN SAFE TO ENABLE — measured, not inspected. For all 11 substituted
paths, `kubectl kustomize --load-restrictor LoadRestrictionsNone <path> | flux envsubst
--strict` exits 0 against cluster-vars base defaults. Negative control confirms the
incident: with K3S_API_HOST unset, non-strict renders k8sServiceHost empty while strict
exits 1 with `variable not set (strict mode)`. The feature gate itself is NOT flipped
here — that is a live-cluster change and the operator's call.
Escapes the six comment hazards that stood in the way (apps/vms/unifi x4,
platform/longhorn x2). All were prose ABOUT the escaping convention; since the repo's own
contract is that $${...} emits a literal ${...}, escaping them is both precise and
self-consistent. kustomize strips these particular comments today, but comment survival is
an implementation detail — a body comment carrying $ WAS observed surviving into built
apps/wave-1 — and a survivor would kill the whole Kustomization.
DOCTRINE CORRECTED (measured against flux 2.4.0): clusters/base/apps-vms.yaml claimed an
indexed expansion is fatal "escaped or not". Escaping is TOTAL — $${ARR[0]} emits the
literal text safely; only the UNESCAPED indexed form is fatal. Corrected in place.
Also empties DOCUMENTED_GAPS in conformance/run.sh with rationale: R-ENV-02 and
R-GITOPS-01 now have a real verifying test, which run.sh's own comment named as the
trigger for removing them.
Advisory output (not failures): 6 dead cluster-vars knobs — BASE_DOMAIN, CLUSTER_ENV,
CLUSTER_POD_CIDR, PIA_REGION, PLATFORM_NODE_SELECTOR_{KEY,VALUE}. The last four are
documented reference-only; BASE_DOMAIN and CLUSTER_ENV look like genuine rot (hostnames
are hardcoded elsewhere) and are worth a follow-up.
Gate: MERGE GATE: PASS. 102 assertions, 0 hazards.
Serves: R-ENV-02, R-GITOPS-01.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
14be9b68f6 |
test(conformance): dr-restore actually asserts off-cluster restore (R-STATE-03)
Closes the LAST scaffold-only safety-critical requirement. The operator has attested DR works (Velero -> Proxmox versitygw S3, restore demonstrated), but the L3 test carried the right tag and layer label over a 5-line TODO body — so coverage counted it and reported "0 safety-critical gaps" while the suite asserted nothing. That was test debt, not capability debt. This makes the suite assert what the operator already demonstrated. Three steps: 1. assert-backup-target-is-off-cluster (read-only): BSL must be phase Available (Velero itself reached the store), s3Url host is not localhost/*.svc/*.cluster.local, and the resolved IPv4 is none of — loopback, any Node Internal/ExternalIP, any Service ClusterIP, any Pod IP, inside any node podCIDR, inside any ServiceCIDR. A BSL that exists but never reaches Available FAILS (a live DR defect), it does not skip. 2. assert-backups-are-scheduled: R-STATE-03 says SCHEDULED off-cluster backups — Velero + BSL but zero Schedule in phase Enabled is a fail. 3. The drill: seed a nanosecond-unique canary onto a longhorn PVC in its own scratch namespace; Backup to Completed; assert a PodVolumeBackup for that volume is Completed WITH a non-empty status.snapshotID (the kopia snapshot of volume DATA reached the remote repo, not just resource JSON); ANNIHILATE the source (delete ns, poll until gone, force-delete a lingering PV, FAIL if the original PV survived — a survivor makes the restore prove nothing); Restore with namespaceMapping into a clean namespace; assert the restored PVC bound a DIFFERENT PV and that the canary bytes are identical. Off-cluster proof = endpoint outside every cluster address + BSL Available + kopia snapshotID present + every in-cluster copy destroyed + restore landing on a new PV. The bytes could only have come from outside. Skips GREEN on: no longhorn SC, Velero API absent, no BSL, Velero/node-agent not ready — all three steps re-derive the same gates so it can never half-run. An RBAC denial is deliberately NOT a skip: it fails loudly as a harness defect. Teardown is a step-level `finally` that ALWAYS runs — deletes the Restore, issues a DeleteBackupRequest so objects leave the bucket (2h ttl as backstop), deletes both scratch namespaces by name and by label, releases leftover PVs whose claimRef is one of its own. Everything it destroys is something it created. conformance/incluster/rbac.yaml: authored the R-STATE-03 grants (velero.io backups/ restores/deletebackuprequests create+delete+read; BSL/schedules/podvolume* read-only; persistentvolumes get/list/DELETE — the off-cluster proof requires destroying the PV). Also two pre-existing gaps other tests already depended on: storageclasses get/list (the skip gate of five tests) and apps deployments/daemonsets read. No write verbs on platform controllers. NOT ASSERTED, stated in the file header rather than implied: off-HOST (on dev the nodes are VMs on the same Proxmox host as the S3 target, so this does not survive hypervisor loss); no direct S3 client read (that would hand a privileged test identity the backup credential for weaker evidence); real service data (Nextcloud/Gitea/CNPG drills stay operator exercises); bare-metal rebuild DR; encryption-at-rest; retention/GC. status: scaffold-only 5 -> 4; safety-critical w/o executable L3 1 -> 0 (of 6); executable 35 -> 36. coverage unchanged at 40/51 (correct — coverage only asks whether a tagged test exists). PROVEN stayed 0 / not-derivable: it did NOT flip to proven just because the body is non-empty. AUTHORED, NEVER EXECUTED — carries a conformance.home.k3s/never-executed annotation. No cluster access from this workstation. To convert authored -> proven the operator must, IN A MAINTENANCE WINDOW (it deletes namespaces and PVs and writes/deletes bucket objects): apply the updated rbac.yaml, run `./conformance/run.sh --requirements R-STATE-03 --layer l3`, confirm the log shows DR_RESTORED_FROM_OFF_CLUSTER_BACKUP_OK and NOT DR_RESTORE_SKIPPED_OK (green but proves nothing), confirm no leaked *-dr-src/*-dr-dst namespaces or orphan PVs or *-dr-backup objects, and record the green run. Serves: R-STATE-03, R-TEST-05. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
6e77150ae1 |
fix(ci): kill the false-green base resolution; cover R-SEC-01/02 + R-TEST-06 (L0)
Unblocks the enforced Gitea gate, which correctly BLOCKED this branch on ['R-SEC-01','R-SEC-02','R-TEST-06'] being touched but uncovered. Tests were authored; the coverage rule was NOT narrowed. === THE FALSE GREEN (root cause) === `make -C ci gate` locally reported PASS having checked NOTHING. resolve_base() validated nothing, and EVERY way of failing to find a base degraded silently to an empty touched set + exit 0: an `HEAD~1` last resort, a two-dot diff fallback masking a failing three-dot diff, and no check that HEAD is even AHEAD of the base. Run while sitting on `main` (main == origin/main), `origin/main...HEAD` is empty -> "no changed files vs base" -> the empty set means run.sh is invoked with NO --requirements -> coverage.py's --require gate never runs at all. Green by vacuum. Now: candidates are BASE_REF > GITHUB_BASE_REF > origin/main > main, each verified to resolve. HEAD~1 and the two-dot fallback are gone. Unresolvable base / no merge base / failed diff / HEAD-not-ahead -> loud banner, EMPTY stdout, exit 3, and a machine-readable reason (on-base | no-base | diff-failed). gate.sh turns exit 3 into an UNDECIDED decision: `on-base` -> "NOT APPLICABLE — no branch under gate" (exit 0 so push-to-main runs don't break, but it never reads as PASS); `no-base`/`diff-failed` -> BLOCKED. It also now prints the diff base it used. A stale local main warns (widens, never falsely greens). Also fixed `make -C ci test`, which was SWALLOWING failing suites. Regression: ci/tests/touched-requirements-base.test.sh — 31 assertions over 8 throwaway-repo scenarios including the exact false-green case, an already-merged branch, no base ref, a single-commit repo, bogus BASE_REF, a shallow GITHUB_BASE_REF, and origin/main-preferred-over-stale-main. === NEW L0 STATIC LAYER (conformance/tests/l0-static/) === Where docs/14 actually puts these invariants. Each check.sh really executes offline via `run.sh --layer l0` and is wired into ci/l0.sh, so it runs on every PR. No scaffolds. R-SEC-01 no-plaintext-secrets (165 assertions / 445 tracked files): .sops.yaml holds only PUBLIC age recipients; every committed Secret is SOPS-wrapped with a mac and ENC[...] values; ansible vault.yml files are $ANSIBLE_VAULT; no secret-valued ansible key carries a literal outside a vault; no PEM key / age secret key / kubeconfig client-key-data is tracked; Flux declares sops decryption; the disallow-plaintext-secrets policy Enforces and has a proving `result: fail` row. Exemptions are SELF-POLICING, not path allowlists — a .example must be referenced by no kustomization and hold only placeholders, and the Kyverno fixture is exempt ONLY while a test asserts the policy rejects it. Proven against 7 injected violations, including deleting the negative row and weakening Enforce->Audit. R-SEC-02 least-privilege-rbac (41 assertions): no wildcard apiGroup/resource/verb and no escalate|bind|impersonate in any in-repo Role/ClusterRole; no ServiceAccount bound to cluster-admin or a wildcard role; the conformance runner's own test identity is scoped and holds no rights over RBAC objects; the default-deny/host-namespace/egress-class policies Enforce and each carry a negative row. Proven against 6 injected violations. R-TEST-06 spec-immutability-enforced (16 assertions): drives the REAL ci/spec-immutability.sh against HERMETIC throwaway spec repos and asserts its exit code in every state — clean-at-pin 0, edits under docs/ or decisions/ 1, untracked addition 1, unadopted drift 1, explicit pin bump 0, ABSENT spec repo 3 (honest skip, never green), unpinned SPEC_PIN 1 — plus that gate.sh sets BLOCK=1 on its failure, so the mechanism is load-bearing. The real spec repo is never written to. Each check documents what it does NOT cover (runtime decryption, git history, runtime least-privilege, branch protection) rather than overclaiming. status.py now reads check.sh too, so a placeholder L0 body still outranks an `executable: true` annotation. Coverage 37/51 -> 40/51 (11 gaps); executable tests 32 -> 35; scaffold-only stays 5 — no sixth scaffold was added. Serves: R-SEC-01, R-SEC-02, R-TEST-01..06. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 5
|
1252823d80 |
feat(conformance): repo-derived spec status report (conformance/run.sh --status)
Operator ruling: "I care about the state of the repository and what the status of each spec is based on that state" — not a hand-maintained task board. This is that report. conformance/lib/status.py (stdlib-only, --selftest) sits beside lib/coverage.py and IMPORTS it rather than forking the registry/test parsers. Driven by a new `conformance/run.sh --status`; artifacts land in _artifacts/status/ alongside coverage and are git-ignored by the existing convention. Exit codes follow ci/lib.sh (0/1/3); --strict-coverage also applies the status hygiene gate. Signals derived per requirement: is there an implementing artifact; is it RECONCILED (walk resources: from clusters/dev, expand each Flux Kustomization's spec.path, recurse — so a commented-out resource correctly does not count); is there a tagged test and is its body executable or a TODO scaffold (TODO markers OUTRANK an executable: "true" annotation — ambiguity never upgrades a row). DELIBERATELY NOT DERIVED, and reported as such rather than defaulting to green: proof-of-pass (nothing tracked in git records a run result — _artifacts/ and results/ are git-ignored) and cluster liveness. "Reconciled" means Flux WOULD apply it, not that it is healthy. PROVEN is reported as "not derivable" for all 51. WHAT IT SHOWS THAT coverage.md DOES NOT. Coverage reports 37/51 covered, 0 safety-critical gaps. Three of those claims do not survive repo state: 1. 5 of the 37 "covered" requirements have only a TODO-body scaffold (R-STATE-03, R-OBS-02, R-OBS-04, R-GITOPS-03, R-GITOPS-02). Coverage counts a tagged file; it never looks inside it. 2. R-STATE-03 is SAFETY-CRITICAL and its L3 fault test asserts nothing. Coverage reports "0 safety-critical gaps" only because dr-restore/chainsaw-test.yaml exists and carries an L3 label. --strict now fails on this. 3. R-UPD-01/02/03 are NOT reconciled: platform/image-automation/ is commented out of platform/kustomization.yaml. Coverage shows them covered with executable L3 tests, which reads as delivered; automated updates are actually 0/3 wired+testable. Headline: 51 requirements | reconciled 20 (+12 partial) | executable test 32 | wired+testable 22 (+8 partial) | scaffold-only 5 | PROVEN 0 (not derivable). Also surfaces 18 requirements with no reconciled manifest citing their R-ID — listed as a traceability gap (R-TEST-04) rather than guessed either way. Annotating those manifests makes the report self-correct, so it acts as a forcing function instead of another hand-maintained map. Serves: R-TEST-01..06. Gate: MERGE GATE: PASS. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
72d4720d58 |
feat(auth): gate pihole UI with per-app oauth2-proxy sidecar (canary)
Close the un-gated-UI security finding for forward-auth apps (R-IDP-02) by rendering the Option-B per-app oauth2-proxy reverse-proxy sidecar that crafty proves working. catalog/render.py now emits, for any forward-auth app with an exposed UI: the oauth2-proxy sidecar (upstream http://127.0.0.1:<uiport>, :4180, --allowed-group per auth.group), an authproxy :4180 Service port, an HTTPRoute backendRef -> :4180, and a gateway-ingress CiliumNetworkPolicy that allows the Gateway to reach ONLY :4180 (fail-closed: the bare UI is never served un-authenticated). oidc/native/none apps render byte-identical. This is the CANARY: only pihole's golden carries the sidecar and is verified fail-closed in-cluster first. jackett/qbittorrent/radarr/sonarr are deferred to the rollout commit (temporarily listed in ci/render-golden.sh) so their UIs keep serving exactly as today until their clients+secrets land. - keycloak: confidential pihole client (realm-import + realm-clients mirror), provisioned live via kcadm (groups membership mapper, bare names). - secret: SOPS/age pihole-oauth2-proxy-credentials (client-id/secret/cookie), wired into apps/wave-0 via the renderer's new sidecar-creds wave rule. - L1 render goldens regenerated for the sidecar (forward-auth fixtures). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d60e5531b1 |
test(conformance): correct stale ImageVolume-gate rationale for k3s v1.36.4
The three KubeVirt VM conformance tests (vm-lifecycle, vm-default-deny,
vm-data-survival) carried comments asserting the ALPHA `ImageVolume` feature
gate is DISABLED "on this k3s v1.31 cluster" and that the apiserver therefore
prunes the container-disk `image` volume, breaking the containerDisk boot path.
The cluster was upgraded v1.31 -> v1.36.4. Revalidated the gate live:
- kubernetes_feature_enabled{name="ImageVolume"} = 1 (enabled by default;
stage="" — no longer ALPHA)
- a server-side `kubectl apply --dry-run=server` of a Pod with an `image`
volume RETAINS the volume + mount instead of pruning them.
So the old premise is stale: containerDisk roots would boot on 1.36.4. The
tests never asserted the gate state and never used containerDisk — they boot
from CDI DataVolume roots, which remains the correct choice because it is the
spec-required, rebuild-surviving disk model (docs/06, R-STATE-01); an ephemeral
containerDisk could never satisfy data-survival. Test logic is therefore
unchanged; only the explanatory comments are corrected (v1.31 -> v1.36.4, and
the "containerDisk is broken here" rationale marked stale with the live
evidence).
Merge gate: PASS (lint, l0, render-golden, l1 green).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
8a424bcba3 |
test(l1): regenerate render goldens for the keycloak-egress renderer change
The container-workload and egress-class L1 fixtures use auth.mode=forward-auth, so the renderer now emits the keycloak backend egress allow for them too. Regenerate the committed goldens to match the canonical render. MERGE GATE: PASS. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
22db5fdb63 |
kyverno: 3.3/v1.13 -> 3.9/v1.19 (stay-current) + CLI lockstep
- HR chart 3.3.x -> 3.9.x (Kyverno v1.13.6 -> v1.19.0). - crds.install:false + CRDs pre-applied server-side (5.7 MB bundle exceeds kubectl's 256 KB annotation limit; Helm client-side apply would stall the platform Kustomization, kps-style). - CLI lockstep (ci/conformance/devcontainer versions.env + run-l0.sh default) -> 1.19.0. Pre-verified: v1.19.0 kyverno CLI passes the L0 suite 17/17 with the existing test format. - Policies are kyverno.io/v1 ClusterPolicy (stable API, unchanged in v1.19). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
cbc2301b30 |
docs: reconcile docs/comments with post-upgrade cluster state
Text-only cleanup after the upgrade campaign. Updates stale version strings, prose, and comments to the verified current facts; no Kubernetes behavior or version pins changed except the kubectl toolchain bump (below). Verified facts applied: k3s v1.36.4+k3s1, Debian 13 trixie, Cilium 1.20.1 / Gateway API v1.6.1, Longhorn 1.12.1, node RAM 24 GiB, dedicated virtio2 SSD etcd disk per node, Velero chart 12.x / v1.18, Grafana 13.2.0. - Version strings across infra/ and platform/ READMEs + comments - kubectl toolchain 1.31.4 -> 1.36.4 (devcontainer/ci/conformance versions.env; 1.31 vs a 1.36 cluster exceeded the +-1 skew) - Retired-single-apps-Kustomization refs -> apps-wave-0/1/2 + apps-vms with per-wave SOPS decryption (flux/clusters docs + bootstrap.sh) - COMPLETED/historical banners on the three executed plan docs - Resolved TODO(WS-SEC) markers -> Renovate-maintained ranges - Appended dated DECISIONS-LOG entry (2026-08-30) superseding the 2026-08-26 LIVE STATE line Left as revalidation items (not blind-edited): external-snapshotter v8.2.0 vs 1.36, Cilium ext_authz/forward-auth relevance under 1.20.1, conformance ImageVolume feature-gate assertions under 1.36. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d24a443c7d |
chore(flux): migrate image-automation API v1beta2 -> v1
Migrate image.toolkit.fluxcd.io/v1beta2 -> /v1 for ImageRepository,
ImagePolicy, and ImageUpdateAutomation ahead of the Flux v2.7+ upgrade
(cluster-upgrade-plan §3.2). Flux v2.4 (current) already serves /v1, so
migrating the manifests now is a safe no-op at runtime.
- catalog/render.py: r_image_automation emits v1 for ImageRepository/ImagePolicy
- platform/image-automation/{image-update-automation,per-service/pihole.example}
- conformance L3 hands-off-update chainsaw applies v1 CRs
- conformance L1 goldens re-rendered to match (apiVersion-only diff)
No spec-field changes required: all spec fields used (image, interval,
imageRepositoryRef, filterTags.pattern/extract, policy.semver.range,
sourceRef, git.*, update.strategy/path) are unchanged in v1; the v1
messageTemplate already uses .Changed (not the removed .Updated).
apps/ golden unaffected (canonical render uses --no-image-automation).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
fdf5c69adc |
test(conformance): add L3 node-loss test for R-STATE-02 (Longhorn node-loss resilience)
R-STATE-02 was the last safety-critical requirement with no verifying test,
exempted via DOCUMENTED_GAPS. Author the L3 fault-injection test that proves the
guarantee and remove the exemption.
tests/l3-fault-injection/node-loss/chainsaw-test.yaml:
GIVEN a 3-replica Longhorn volume with written data attached on node N,
WHEN N is removed from service (cordon + evict the attached workload, forcing
the volume to detach from N),
THEN a fresh workload on a SURVIVING node re-attaches the same volume and reads
the canary intact — volume stays available, cluster reconverges.
Safe-by-construction for the 3-node embedded-etcd dev cluster:
- Disruption scoped to EXACTLY ONE node, at the scheduling layer only (cordon +
evict our own pod). kubelet/etcd stay up — etcd quorum is never touched.
- Never `kubectl drain`; only our own writer pod is deleted, so no unrelated or
system pod is disturbed.
- Victim node labelled conformance.home.k3s/cordoned-by=<test-ns>; `finally`
ALWAYS uncordons every node with THAT label and removes it — restored even on
failure/timeout, and never touches an operator-cordoned node.
- Self-skips green until a longhorn StorageClass AND >= 3 Ready nodes exist.
Bookkeeping:
- run.sh: remove R-STATE-02 from DOCUMENTED_GAPS; add node-loss to CONF_HEAVY_TESTS
(serial pass, matches the other Longhorn/KubeVirt L3 tests).
- incluster/rbac.yaml: enable the scoped nodes get/list/patch rule the test needs
(cordon/uncordon/label); no eviction rights (no drain).
- README.md / coverage.py docstring: refresh the documented-gap example.
Coverage: 36/51 -> 37/51 (70.6% -> 72.5%); safety-critical gaps 1 -> 0. Only
R-STATE-02 changed ([] -> node-loss); no other requirement or golden touched.
NOTE: LIVE execution is DEFERRED to an operator-chosen maintenance window. This
change AUTHORS + statically validates the test (renders, chainsaw-lints, harness
selects it, coverage updates). The static gate runs L2/L3 as SKIP without a
cluster (expected). The node-disruption test was NOT run live.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 5
|
9e8b694ba9 |
docs(conformance): record that the matcher debt is half paid
The content-scan half of the 'stop keying on doc-header tokens' follow-up is done (ci/touched-requirements.sh). The path-map half remains: clusters/* and ci/* are still keyed wholesale to R-ENV-02 + R-GITOPS-01, which is why those two stay in DOCUMENTED_GAPS. Notes the accepted consequence for header-declared manifests. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
948c390272 |
conformance: serialize heavy L2/L3 tests + author alerting SMTP sink
TASK 1 — reliably single-pass green L2/L3: Split run_chainsaw into a LIGHT pass (concurrent, config parallelism) and a HEAVY pass (serial, parallel=1). The KubeVirt-CDI/Longhorn/DR tests (vm-lifecycle, vm-data-survival, vm-default-deny, data-survival, dr-restore) spike embedded-etcd on the small dev VMs; running them serially in their own pass keeps their etcd spikes from ever coinciding with another test's setup — the coincidence that let one heavy test's apiserver blip fail-close Kyverno admission and abort every in-flight test at 0.01s. Heavy set is keyed by test dir basename and overridable via CONF_HEAVY_TESTS. Each pass writes its own JUnit; lib/merge_junit.py stitches them into the one stable report.junit.xml CI collects, so the split is invisible downstream. Overall rc is the union of both passes (no masking — real failures still surface). Proof: full --layer l2,l3 run now completes single-pass green: 24/24 passed, 0 failures/errors/skipped. vpn-killswitch, which previously failed with an apiserver EOF (etcd blip), passed; no mass environmental setup-aborts. TASK 2 (T-4.1) — author conformance/tests/l3-fault-injection/alerting/ fixtures/smtp-sink.yaml: a hermetic Mailpit SMTP capture sink (Service smtp-sink, SMTP 1025, HTTP API 8025) the alerting test applies as its mail sink. restricted-PodSecurity compliant, in-memory (no PVC). Unblocks the previously-aborting alerting scaffold; it now passes. Touches conformance/ + ci-read content only — no apps/ or platform/ workload. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
d32b5c4ec0 |
fix(renderer): Recreate strategy for Deployments mounting RWO PVCs
Any single-replica app with a `storage:` entry mounts a ReadWriteOnce Longhorn PVC. The default RollingUpdate strategy starts the new pod before terminating the old one, but the old pod still holds the RWO volume -> the new pod hangs in ContainerCreating and the rollout deadlocks (observed on gitea's update). Emit spec.strategy.type=Recreate whenever storage is present so the old pod releases the PVC first. Non-storage apps (and VMs) keep the default (no strategy emitted). Re-rendered apps: gitea, pihole, sonarr, radarr, plex, qbittorrent, jackett now carry strategy: Recreate. Updated the egress-class L1 render goldens (storage fixture) accordingly; all L1 render checks pass. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
9f52f195a7 |
test(conformance): lower chainsaw parallelism 4->2 to avoid dev etcd overload cascades
On the small dev VMs, 4 concurrent heavy tests (KubeVirt CDI + Longhorn) spiked embedded-etcd into leader-changes/apiserver-ServiceUnavailable; with fail-closed Kyverno admission this cascaded into mass test aborts. 2 lets the full L2/L3 suite complete in a single clean pass. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
63b87ffe46 |
test(conformance): update DNS tests to Proxmox Gateway VIP 192.168.1.53 (was 10.73.0.24x)
Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
bb48f675ee |
ci: bake pinned test toolchain into devcontainer + fix conformance layer/spec-immutability seams
Bake the conformance harness toolchain into infra/devcontainer so `make -C ci gate` runs identically here and in CI: - Containerfile: install kyverno CLI (1.13.4), kyverno-json, chainsaw (0.2.12), yamllint (1.35.1), check-jsonschema (0.29.4), yq (4.44.6) via new pinned RUN block; add diffutils + python3-pip; drop stale LIBVIRT_DEFAULT_URI; refresh header for the Proxmox(bpg)+conformance toolbox; extend /etc/dev-toolchain-versions provenance. - versions.env: add KYVERNO/KYVERNO_JSON/CHAINSAW/YAMLLINT/YQ/CHECK_JSONSCHEMA; align KUBECTL 1.32.0 -> 1.31.4. dev.sh build(): pass the new --build-args. - ci/versions.env: pin CHECK_JSONSCHEMA_VERSION=0.29.4 (single source of truth). Fix harness seams so the gate can run: - dev.sh run(): mount the sibling spec repo (home-network-k3s) read-only at /home-network-k3s for ci/spec-immutability.sh (R-TEST-06); conditional so the container still runs without it. - conformance/lib/common.sh conf_dirs_for_layer(): accept a comma/space list of layers and union the dirs, so ci/conformance.sh `run.sh --layer l2,l3` selects both layers in one pass (backward compatible for single layers). NOTE: ci/versions.env pins KYVERNO_JSON_VERSION=0.0.5, which has no upstream release (max tag is v0.0.3; platform/kyverno/scripts/run-l0.sh already defaults to v0.0.3). Baked v0.0.3 and flagged for PM to correct ci/versions.env. kyverno-json is not used by any static gate stage. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85
|
7e8ae472c9 |
fix(net): renderer emits world/internet egress allow per egress class + fix jackett/plex tags
Root cause the media stack surfaced: the default-deny baseline is DNS-only egress (correct for R-NET-01 east-west), but the egress CLASS (internet/vpn) had no policy ALLOWING north-south egress — so qbittorrent/sonarr/radarr couldn't reach the internet at all (VPN downloaders broken). Add an additive per-app CiliumNetworkPolicy '<app>-allow-internet-egress' with toEntities:[world] (non-cluster); the egress-class steering + kill-switch (ADR-0015) govern the PATH (internet=WAN, vpn=tun0). East-west stays denied except declared dependencies; R-NET-01 baseline unchanged. Re-rendered all apps + regenerated L1 goldens. Also fixed jackett + plex image tags (bad pins -> :latest; Renovate/update policy pins later). |
||
|
jakman85
|
5ba3f9d46d |
fix(conformance): trusted-tls verifies LE-prod cert via system trust + split-horizon DNS (R-TLS-01/03)
Rewrote the probe for the LE-production switch: no embedded staging root, no --cacert (system CA bundle validates ISRG Root X1), resolves the hostname via in-cluster split-horizon DNS to the Gateway VIP (the real-client path). PASSES live (9s). Removed the le-staging-root-ca fixture. |
||
|
jakman85andClaude Opus 4.8
|
c6a480eed1 |
test(net): L2 conformance — exposed app reachable through Gateway under default-deny (R-NET-03)
New runnable chainsaw test gateway-ingress-reachable (R-NET-01/02/03): stands up an
exposed-app fixture (namespace default-deny + the renderer's additive gateway-ingress
CiliumNetworkPolicy allowing fromEntities:[ingress] to the declared port 8080 only, a
backend listening on 8080 + an undeclared 9091, and declared/undeclared HTTPRoutes on
the shared Gateway) and proves BEHAVIOR through the REAL Gateway VIP:
* then-reachable-through-gateway -> the Gateway (ingress identity) reaches backend:8080
through default-deny (backend marker returned) [R-NET-02/03]
* then-undeclared-port-denied -> port 9091 (no allow) is NOT reachable through the
Gateway (marker never returned; envoy 503) [R-NET-02]
* then-east-west-still-denied -> a non-ingress in-namespace peer is DROPPED on 8080;
the allow is least-privilege, east-west default-deny intact (hermetic) [R-NET-01]
Verified live: reachable=REACHED (200, marker), undeclared=DENIED, east-west=DENIED.
Self-skipping: the gateway probes classify each attempt REACHED / DENIED / NO-GATEWAY
(curl 000). NO-GATEWAY self-skips (green + note) because a node-local Cilium L7LB/envoy
desync can make pods on an affected node unable to reach the Gateway VIP at all — a
SEPARATE platform fault, not default-deny — so this test never false-fails on it. The
east-west denial is hermetic and always asserts. VIP discovered per-step via a kubectl
output binding (env-agnostic; chainsaw bindings are step-scoped). L2 layer stays gated by
CLUSTER_AVAILABLE in run.sh. chainsaw v0.2.12 quirks honored (scalar/JMESPath asserts,
no readinessProbe under default-deny, step-level finally delete).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
c13ef69e71 |
feat(net): renderer emits gateway-ingress allow for exposed services (R-NET-01/02)
Exposed apps got a default-deny NetworkPolicy with ingress: [] (deny all), so the
Cilium Gateway (Envoy, reserved identity `ingress`) could not reach the backend — an
Accepted HTTPRoute was still unreachable. render.py now emits an ADDITIVE, least-
privilege CiliumNetworkPolicy for any ServiceDefinition that renders an HTTPRoute:
ingress:
- fromEntities: [ingress] # the shared Cilium Gateway/Envoy proxy identity
toPorts: [<exposed service port(s)> TCP]
Verified against the live cluster (Cilium 1.16.5): with default-deny in place, Hubble
shows the Gateway proxy reaching pihole as source identity `(ingress)` and the flow
`Policy denied DROPPED` on :80 — so `fromEntities: [ingress]` is exactly the source the
Gateway uses, scoped to the service targetPort. Default-deny stays intact (ingress: []
unchanged); the allow only opens the north-south path (docs/09), east-west stays denied.
- render.py: add r_gateway_ingress_policy(); emit it after dependency policies; selftest
asserts the allow is additive (default-deny ingress stays []), least-privilege (only the
exposed port, only the ingress identity), and absent for non-exposed services.
- L1 goldens updated for the two exposed fixtures (container-workload, egress-class);
default-deny-baseline golden unchanged (no exposure). --selftest + run.sh --layer l1 green.
- apps/pihole.yaml re-rendered (--no-image-automation) to carry the allow.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85
|
c05fd6702a | Merge fix/kubevirt-vm-boot: real-KVM DataVolume VMs (R-PLAT-03/04 green; useEmulation:false, EFI, DataVolume roots) | ||
|
jakman85
|
a177d62925 |
fix(conformance): busybox-compatible wget in oidc-sso-grafana probe (R-IDP-01)
busybox wget has no --max-redirect flag → it printed usage, so the probe never saw the 302 and R-IDP-01 falsely failed. Grafana verified live: /login/generic_oauth 302s to the Keycloak homelab authorize endpoint. Use 'wget -S' (dumps the 302 Location before follow). |
||
|
jakman85andClaude Opus 4.8
|
3a59ef1918 |
fix(kubevirt): boot VMs via CDI DataVolume + EFI + real KVM (Phase 5)
VMs did not boot on the dev cluster. Three distinct live-verified causes, all fixed GitOps-side: 1. containerDisk root failed at OCI init: exec "/container-disk-binary/usr/bin/container-disk": no such file. KubeVirt v1.9 supplies that binary to the volumerootdisk init container via a k8s-native `image` volume (VolumeSource.Image), gated by the ALPHA `ImageVolume` feature gate. k3s v1.31 has it DISABLED, so the apiserver silently prunes the image volume + its mount -> binary missing -> crash. (Proved: a server-side dry-run of a pod with an `image` volume drops it.) Fix: convert the VM roots in all three tests to CDI DataVolumes imported onto Longhorn PVCs (mounted disks bypass the container-disk/image-volume path). This is also the spec-required disk model (docs/06, R-STATE-01). vm-lifecycle now PASSES green live (VMI Running on start + restart). 2. Guest dropped to emergency mode: the containerdisks/fedora:41 image is UEFI-native (ESP + /boot mounted via systemd GPT auto-discovery); under KubeVirt's SeaBIOS default the ESP+/boot by-uuid mounts time out -> local-fs.target fails -> cloud-init runcmd/poweroff never runs. Fix: boot the VMs with EFI (OVMF, secureBoot:false) in all three tests and fixtures/vm.yaml. Confirmed live: emergency mode gone with EFI. 3. VMs ran under TCG software emulation (guests ~10x slow, saturating nodes until one went NotReady and VMs were evicted). Cause: useEmulation:true in the KubeVirt CR makes virt-controller NOT request devices.kubevirt.io/ kvm, so /dev/kvm is never mounted and qemu falls back to TCG even though all nodes advertise kvm=1k / hypervisor=true. (Proved: `ls /dev/kvm` in a virt-launcher compute container = "No such file"; qemu accel = tcg.) Fix: useEmulation:false so the kvm device is requested and VMs run on real (nested) KVM. Requires Flux reconcile to take effect live (PM action). The virt-handler clientcertificates tls.crt error is UNRELATED: node-labeller (294 labels, hypervisor=true), device plugins, and VMI scheduling all work. Also raised test exec timeouts to 30m (CDI import + boot) and, in vm-default-deny, import the root BEFORE applying the default-deny NetworkPolicy so the CDI importer's egress to quay.io is not severed by the policy. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85
|
3b97f5f058 | Merge fix/split-horizon-dns: in-cluster split-horizon DNS (R-DNS-01/03) — unblocks SSO | ||
|
jakman85andClaude Opus 4.8
|
8b199a6780 |
test(dns): internal split-horizon resolution conformance (R-DNS-01/R-DNS-03)
Add dns-internal-resolution (L2 behavioral): an in-cluster Job resolves keycloak.home.keithjackson.dev via the default cluster resolver and asserts the answer is the in-cluster Gateway VIP (10.73.0.24x), not a public/off-cluster hairpin. Complements dns-exposure (R-DNS-02, external record). coverage.py now maps R-DNS-01/R-DNS-03 to this test (covered); chainsaw v0.2.12 lint clean; strict hygiene gate green. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85
|
186ff23098 | Merge feature/servicedefinition-renderer: ServiceDefinition renderer T-0.3 (R-GITOPS-02) + L1 render tests green | ||
|
jakman85andClaude Opus 4.8
|
a79f24f75e |
Add container-workload-render L1 test (R-PLAT-02)
The renderer's PaaS path (kind: container) had no verifying conformance test — the R-PLAT-02 token appears only in renderer comments, so a change touching catalog/ surfaced it as an uncovered touched-requirement (blocking the merge gate). Adds the container render analog of vm-lifecycle (R-PLAT-03): asserts a kind: container ServiceDefinition renders a Deployment (running the declared image) + a Service, with the egress-class/auth-mode labels admission requires. Offline render-check.sh, wrapped by a chainsaw Test for in-cluster runs. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
4a017c590e |
Wire L1 render conformance tests to the renderer (R-NET-01, R-NET-05/06/07)
Implements the two L1 rendering tests that were failing scaffolds: - egress-class-render (R-NET-05/06/07): renders vpn + internet fixtures, diffs each against a committed golden, asserts the vpn steering label is present on vpn and absent on internet (fail-closed), and that flipping the class changes ONLY the egress path (R-NET-07). This fixes the --requirements R-NET-05 merge gate. - default-deny-baseline-render (R-NET-01): asserts every render carries the namespaced default-deny baseline, shaped deny-all + both directions. Each test carries a pure-offline render-check.sh (renderer + golden diff, no cluster). run.sh runs L1 render checks offline and emits a REAL pass/fail JUnit (docs/14: L1 rendering assertions run in CI without a cluster) instead of skipping L1 with the cluster-gated layers; the chainsaw-test.yaml wrappers invoke the same render-check.sh so an in-cluster run executes the identical assertion. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
cbe5fab040 |
test(updates): complete hands-off-update L3 conformance (R-UPD-01/02/03)
Runnable, self-skipping (green until image.toolkit CRDs exist). Proves: an ImagePolicy auto-advances to the newest STABLE tag on schedule with no edit and never a pre-release (R-UPD-01/02, via podinfo ImageRepository/ImagePolicy status); a deliberately bad update is health-gated and AUTO-ROLLS-BACK to the last-good revision while the old ReplicaSet keeps serving, and the UpdateHealthGateFailed alert is wired (R-UPD-02/03); and an exact-range (pinned) policy does not advance past its pin (R-UPD-03). Hermetic to the chainsaw per-test namespace; block-style contains() checks per chainsaw v0.2.12. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85
|
b95b540468 |
Merge feature/keycloak-auth-stack: Phase 6 Keycloak IdP + oauth2-proxy + CNPG (R-IDP-01..06, R-OBS)
# Conflicts: # platform-config/kustomization.yaml # platform/kustomization.yaml |
||
|
jakman85
|
4bebd4b6d2 | keycloak: coverage tests + dashboard + reword cross-cutting tags | ||
|
jakman85
|
9da209c71e |
Merge branch 'main' into feature/kubevirt-iaas
# Conflicts: # conformance/run.sh |
||
|
jakman85
|
0cb25c12e3 | Merge branch 'main' into feature/keycloak-auth-stack | ||
|
jakman85andClaude Opus 4.8
|
f5f52b0b8c |
chore(harness): adopt spec pin b633b7f + exempt R-ENV-02/R-GITOPS-01 coverage gaps
Adopt the architect's spec advance (ADR-0013 amendment + ADR-0015 egress steering/invariants) into ci/SPEC_PIN; the impl main pin was stale (a7fc19f) so the R-TEST-06 spec-immutability gate was red on main. Verified the a7fc19f..b633b7f range is spec-only architect commits. Add R-ENV-02 and R-GITOPS-01 to conformance/run.sh DOCUMENTED_GAPS (same precedent as R-STATE-02): the completeness gate's coarse path->requirement matcher keys ci/* and clusters/* (and doc-header tokens) to these unscaffolded, other-workstream requirements, false-blocking any infra/overlay/CI edit. Documented + reviewable, not silent. Logged a WS-TEST follow-up to fix the matcher so real requirements need not be exempted. Serves: R-TEST-06. PM/Integration-Release governance change. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85
|
ce83c6af3b | WIP keycloak stack (gate probe) | ||
|
jakman85andClaude Opus 4.8
|
dbffb8fab8 |
Phase 5: KubeVirt + CDI IaaS proof (R-PLAT-03/04, R-STATE-01, R-NET-01)
- platform/kubevirt: pinned KubeVirt v1.9.0 + CDI v1.66.0 operators (remote bases); KubeVirt/CDI CRs tuned for Longhorn; useEmulation=true (prefer real KVM, emulate fallback). - Controllers/configs split (F-MON1): CRs + ServiceMonitors + alerts via platform-config. - Observability: ServiceMonitors, KubeVirt platform-health alerts (R-OBS-04), VM dashboard. - catalog/vm-demo.yaml (kind: vm) + rendered VirtualMachine/DataVolume fixture. - Conformance: vm-lifecycle (R-PLAT-03/04), vm-data-survival (R-STATE-01, L3), vm-default-deny (R-NET-01) — self-skipping until KubeVirt+longhorn exist. - ci/SPEC_PIN: adopt accepted spec b633b7f (ADR-0015 egress work already merged). - conformance/run.sh: document R-ENV-02/R-GITOPS-01 coverage gaps (WS-GITOPS/INFRA-owned). Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
4c21f9f6e3 |
conformance: fix invalid test-level spec.finally for chainsaw v0.2.12
chainsaw v0.2.12 has no test-level `spec.finally` (finally exists only on a
step), and a step-level `finally` accepts only catch-style ops (script/command/
delete/wait), NOT `apply`/`assert`. Both tests failed to parse ("unknown field").
- egress-class: drop the redundant test-level `spec.finally` delete-pods block —
chainsaw auto-deletes applied fixtures on cleanup (skipDelete: false). The
valid step-level `finally` that restores the egress-class label is kept.
- vpn-killswitch: move the SAFETY-CRITICAL self-healing tunnel restart out of the
invalid `spec.finally` and into the STEP-level `finally` of the fault step
`when-force-tunnel-down-then-no-leak`, implemented as a `script` (finally can't
use apply/assert) that `kubectl apply`s the privileged node-pinned restart Job
(name `tunnel-restart`, distinct from `tunnel-stop`) and `kubectl wait`s for it
to complete — the equivalent of `assert succeeded:1`, blocking teardown until
the host confirms the tunnel is up. This finally runs even if the no-leak /
internet-unaffected asserts in the same step's `try` fail, so no path leaves the
gateway tunnel down. The recovery step is now poll-only (restart already
happened in the prior step's finally).
Validation with the pinned binary (~/.local/bin/chainsaw v0.2.12):
`chainsaw lint test --file ...` => "The document is valid" for both; `chainsaw
test ... --no-cluster` loads both past the parse stage (no "unknown field"), then
fails only at runtime for lack of a cluster (expected). yamllint + run.sh
--strict-coverage clean (RC=0). No live apply / fault injection, no spec edits.
Co-Authored-By: Claude Opus 4.8 <[email protected]>
|
||
|
jakman85andClaude Opus 4.8
|
0e69c63b55 |
conformance: complete vpn-killswitch L3 test + ADR-0015 RBAC/fixtures
Turn the vpn-killswitch scaffold into a runnable SAFETY-CRITICAL L3 test for R-NET-06 under ADR-0015 (node-level host OpenVPN tunnel steered by Cilium Egress Gateway with a host nftables kill-switch), replacing the obsolete in-cluster gluetun design. - fixtures/vpn-and-internet-pods.yaml: two egress-class probe pods + documented RBAC/PodSecurity/namespace requirements for node-level fault injection. - vpn-killswitch/chainsaw-test.yaml: GIVEN healthy tunnel (vpn egress != WAN); WHEN the host `vpn-egress-tunnel` systemd unit is STOPPED via a privileged, hostPID, node-pinned nsenter Job THEN vpn-class egress is blackholed (curl fails, never returns the WAN IP = no leak) while internet-class is unaffected; WHEN restarted THEN vpn egress recovers. spec-level `finally` ALWAYS runs an idempotent start Job (self-healing) so a mid-test failure never leaves the gateway tunnel down. - incluster/rbac.yaml: grant the runner least-privilege for the delivered tests (namespaces create/delete/patch for the PSA relabel, pods/services/configmaps/ pods-exec/pods-log, batch/jobs) and replace the gluetun-scale rule with the ADR-0015 host-systemd mechanism note (privilege via PodSecurity, not RBAC). Offline validation: python yaml parse (all docs), `run.sh --lint` (structural), `run.sh --coverage-only --strict-coverage` (no tag/annotation drift; R-NET-05/06/ 07 covered, R-NET-06 has its L3 test), yamllint. No live apply, no cluster run. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
e7a567db0b |
WIP conformance(egress-class): ADR-0015 assertions (partial — killswitch test TODO)
Session-limit interrupted the WS-TEST agent. egress-class test updated; vpn-killswitch test + host fault-injection fixture still outstanding. Preserved for review/continuation. Co-Authored-By: Claude Opus 4.8 <[email protected]> |
||
|
jakman85andClaude Opus 4.8
|
34f41a0fba |
phase3(vpn): egress-class conformance fixtures + steering design finding
Author the internet/vpn egress-class fixture pods for the egress-class + vpn-killswitch conformance tests. Record the datapath finding: ADR-0013's preferred steering (Cilium egress gateway -> gluetun tun0) is NOT viable because gluetun must run hostNetwork to put tun0 on the node, and gluetun then redirects the NODE's default route through the VPN (breaks k3s/etcd). Escalated: remaining viable path is option 2 (gateway-pod routing). No steering datapath committed until the mechanism is chosen + validated. Co-Authored-By: Claude Opus 4.8 <[email protected]> |