Enable monitoring in platform/kustomization.yaml (Phase 4). Grafana admin +
Alertmanager SMTP secrets are provisioned (SOPS) and wired; Longhorn SC is up for
Prometheus/Loki PVCs. Set the Alertmanager receiver 'to:' to a valid inbox
([email protected]) so the config loads (REPLACE-ME is not a valid address);
operator can change it. NOTE: smtp_from ([email protected]) differs from
the Gmail auth user — Gmail may reject/rewrite unless a send-as alias is configured;
that only affects email delivery at alert time, not stack health.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Was a scaffold (fixture TODO, fault-injection a no-op echo). Implemented against the
live datapath: fixtures/allowed-pair.yaml = default-deny baseline + source + dest(:8080)
+ the sole explicit allow. The test verifies source->dest works (ALLOWED_OK), then
REVOKES both halves of the allow (real fault: kubectl delete networkpolicy), and asserts
connectivity fails CLOSED (FAIL_CLOSED_OK) while DNS still resolves (DNS_OK). Proves
default-deny is datapath-enforced and fails closed on revocation. DNS check asserts the
resolved answer, since busybox nslookup exits non-zero on a successful forward lookup.
PASSES (74.8s).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
My previous commit converted =~ to contains() but kept flow style
'check: { (contains($stdout, 'X')): true }' — the comma inside contains() is a
YAML flow-map separator, so 9 tests failed yamllint ('expected , or }'). Convert
those checks to block style (matching data-survival), where a comma in the key
scalar is fine. Gate PASS; data-survival still green.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Every chainsaw `check` used the regex operator `=~` (`($stdout =~ 'X'): true`),
which chainsaw's expression engine rejects ('SyntaxError: Unknown char: ~') — so
EVERY such test failed on the assertion even when the behavior passed. Proven with
data-survival (R-STATE-01): the Longhorn canary genuinely SURVIVED pod destruction
(DATA_SURVIVED_OK) but the test FAILED on the `=~` syntax. Converted all 10 tests to
JMESPath `contains($stdout, 'X')` (data-survival's alternation → contains||contains).
data-survival now PASSES; the other 9 (later-phase components) are syntactically fixed
for when their phases deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The on-cluster cluster-vars ConfigMap wasn't reconciled by Flux — this bootstrap
model had no Kustomization watching clusters/dev, so cluster-vars/overlay changes
needed a manual 'kubectl apply -k clusters/dev' (e.g. DNS_ZONE substituted empty
until re-applied). Add clusters/dev/sync.yaml: a self-managing root Kustomization
(flux2 gotk-sync pattern, name flux-system) that reconciles ./clusters/dev from git
— cluster-vars, the GitRepository, platform/platform-config/apps, and itself. A git
push now suffices. Bootstrap unchanged ('kubectl apply -k clusters/dev' creates it
once). Verified: dev build contains all 6 objects (root + existing), prod unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The kyverno chart's cleanup jobs (policyReportsCleanup, webhooksCleanup) default to
bitnami/kubectl:1.30.2, which Bitnami removed from Docker Hub (2024 catalog purge) ->
ImagePullBackOff. policyReportsCleanup is a Helm post-upgrade HOOK, so its failure
stalls every helm upgrade (5m timeout -> HelmRelease not-Ready -> platform Kustomization
blocked -> platform-config/apps cascade). Override both to Bitnami's relocated
bitnamilegacy/kubectl (identical image: /bin/bash + awk/wc/xargs + kubectl). Verified
via helm template that no bitnami/kubectl reference remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Phase-2 exit tests (were Phase-0 scaffolds), authored against the live platform:
- trusted-tls (R-TLS-01/03): in-cluster probe dials the gateway Service (env-agnostic
via --connect-to) with wildcard SNI and verifies the served cert CHAINS TO THE LE
STAGING ROOT (operator decision A; dev uses LE-staging per ADR-0006). Fixture embeds
the '(STAGING) Pretend Pear X1' root. Probes public + internal-only hostnames (R-TLS-03).
PASSES.
- dns-exposure (R-DNS-02): a LoadBalancer Service with an external-dns hostname; polls
public DNS until ExternalDNS auto-publishes the record; teardown + policy=sync removes it.
ExternalDNS fix (root cause of the DNS test): domain-filter was the platform SUBDOMAIN
home.keithjackson.dev, which filters OUT the actual Cloudflare zone keithjackson.dev
('no hosted zone matching record DNS Name' — record generated but skipped). Add a
DNS_ZONE cluster-var (keithjackson.dev) and set domainFilters to it. policy=sync +
txt-registry still only manage ExternalDNS-owned records (the pre-existing wildcard is
untouched). Resolves the file's WS-NET R-DNS-02 TODO.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
run.sh exited non-zero (breaking the L2 gate) when invoked from the repo root:
chainsaw's config report.path is relative (_artifacts), so it resolved against
$PWD and failed to write ('open _artifacts/chainsaw-report.xml: no such file')
even though all tests passed. Pass --report-path ${ARTIFACTS} (absolute) +
mkdir -p first. Verified: R-NET-01 from repo root now exits 0 with report written.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The deployed cluster has TWO default StorageClasses — k3s's built-in local-path
AND longhorn — which is ambiguous for PVCs without an explicit class. Longhorn
(ADR-0003) is the cluster storage, so add local-storage to k3s --disable. Effective
on fresh installs (the k3s install task is idempotent → no restart of the running
cluster); the running dev cluster keeps both defaults harmlessly (all platform PVCs
pin storageClassName: longhorn). Rebuilds get a single default.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The wildcard cert stalled: challenge Presented=true (Cloudflare TXT created OK —
token works) but propagation self-check looped 'DNS record not yet propagated'.
Verified the record IS publicly resolvable (1.1.1.1/8.8.8.8); the NAT'd k3s-dev
cluster's own DNS just can't see it. Set dns01RecursiveNameservers=1.1.1.1:53,
8.8.8.8:53 + dns01RecursiveNameserversOnly=true so the self-check bypasses cluster
DNS. Realises the pre-existing TODO in this file.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Kyverno HelmRelease install failed at Helm templating: config.webhooks was a
LIST [{failurePolicy, namespaceSelector}], but the kyverno 3.3.9 chart expects
config.webhooks to be a MAP (webhooks.namespaceSelector) — the chart helper reads
.namespaceSelector off the value directly ('can't evaluate field namespaceSelector
in type []interface {}'). Also, failurePolicy is not a per-webhook field here; the
chart's global switch is config.forceFailurePolicyIgnore.enabled (bool).
Fix: config.webhooks -> map with namespaceSelector (system+platform excludes);
add config.forceFailurePolicyIgnore.enabled = ${KYVERNO_FORCE_FAILURE_POLICY_IGNORE}.
Renamed the cluster-var KYVERNO_WEBHOOK_FAILURE_POLICY (Fail/Ignore) ->
KYVERNO_FORCE_FAILURE_POLICY_IGNORE (false base / true dev). Verified with
helm template (renders 73 objects, no error); gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Un-suspend deadlocked: the single platform Kustomization applies HelmReleases
(which register CRDs) AND custom resources of those CRDs (ClusterIssuer,
Certificate, Kyverno ClusterPolicies) in ONE pass. Flux's server-side apply
fails atomically ('no matches for kind ClusterIssuer') → nothing installs, so
the CRD-registering HelmReleases never run either. Classic ordering deadlock.
Fix (flux2 controllers/configs pattern, applies to dev+prod equally — no
R-ENV-02 divergence): add a platform-config Flux Kustomization (dependsOn
platform, wait) carrying the CRs; platform keeps only the controllers.
- New clusters/base/platform-config.yaml (path ./platform-config, substituteFrom
cluster-vars, no decryption — no SOPS content); wired into base + apps now
dependsOn platform-config (transitively waits for platform).
- platform-config/ references the CR files IN PLACE via ../platform/... (Flux
builds LoadRestrictionsNone; Kyverno policies must stay under platform/kyverno/
for the CI L0 gate). CR resources dropped from the platform
component kustomizations (cert-manager, gateway, kyverno).
- Authored the missing Cilium L2 resources (were only a TODO): CiliumLoadBalancerIPPool
(dev NAT 10.73.0.240-.250) + CiliumL2AnnouncementPolicy (ens*), so Gateway
LoadBalancer listeners get a VIP. Prod range must be parameterized later.
Verified: platform build has zero CRs; platform-config build has 2 ClusterIssuers,
6 ClusterPolicies, Certificate, Gateway, both Cilium L2 CRs; dev/prod build; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The un-suspend reconcile stalled: platform Kustomization Ready=False with
'StorageClass/longhorn ... .parameters.numberOfReplicas: expected string, got 3'.
StorageClass.parameters is map[string]string, but kustomize strips the quotes
around the bare "${LONGHORN_DEFAULT_REPLICA_COUNT}" placeholder, so Flux postBuild
substitution produced an unquoted numeric that server-side apply rejects as int.
Hardcode the quoted numeric literal "3" (kustomize preserves those, as "30" shows);
topology is locked at 3 nodes. The replica knob still drives the global default via
defaultSettings.defaultReplicaCount in the helmrelease (a Helm int value, unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Rename the committed encrypted secrets to canonical cloudflare-api-token.sops.yaml
(cert-manager + external-dns namespaces) and uncomment the resource lines in both
kustomizations, so cert-manager DNS-01 + ExternalDNS get the token at reconcile.
- Exempt **/*.sops.yaml from ci/yamllint.yaml: SOPS re-emits encrypted files with its
own indentation + long ENC[...] lines (never hand-edited), which was failing the lint
gate on main after the secrets were committed. Same treatment as **/vault.yml.
Both Secrets confirmed present + encrypted in the platform build; gate PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The terse 'Zone->DNS->Edit + Zone->Read' sent the operator to the Account
permission scope. Spell out switching the scope dropdown to Zone + the exact
two Zone-scoped rows + Specific-zone resource.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Two dev.sh gaps surfaced while running bootstrap/sops/install-age-key.sh:
- kubectl defaulted to localhost:8080 (connection refused) because KUBECONFIG
was never set in the toolbox. Set KUBECONFIG=/work/infra/ansible/.cache/dev.kubeconfig
(override via DEV_KUBECONFIG); --network host already reaches 10.73.0.11:6443.
- age-keygen.sh writes the PRIVATE key to ~/.config/sops/age/keys.txt, but that
path was not mounted in the --rm container, so the key was lost on exit. Mount
${HOME}/.config/sops -> /root/.config/sops (created first) so the key persists
on the host and SOPS_AGE_KEY_FILE resolves to it.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
jakman85
