phase2-prep: per-component no-secret rollout + component hardening (author-first)

Prep only — nothing deployed; running cluster untouched.

Reconciliation graph (operator-approved Option A): author clusters/dev/rollout/
= 4 per-component Flux Kustomizations (kyverno->cilium->longhorn->monitoring,
monitoring dependsOn longhorn). Dev overlay $patch:delete's the monolithic base
platform/apps and adds ./rollout, so only the no-secret tier reconciles; the
secret-blocked components are simply not created until hand-offs arrive. The
no-secret tier carries no SOPS content, so it needs no sops-age key. prod is
untouched (still builds the monolith). Convergence path coded in (delete rollout/
+ the two deletes) — R-ENV-02 divergence flagged for architect review.

Component hardening via cluster-vars (prod keeps strict posture):
- KYVERNO_VALIDATION_ACTION Enforce->Audit(dev), _WEBHOOK_FAILURE_POLICY
  Fail->Ignore(dev), _ADMISSION_REPLICAS 3->1(dev); webhook namespaceSelector
  excludes system+platform namespaces.
- MONITORING_STORAGE_CLASS var; GRAFANA_ADMIN_SECRET var (dev ''=chart default);
  Alertmanager SMTP/secret deferred to Phase-4 with verbatim re-enable markers.

Cilium parity verified against the live CNI (no-op adopt). Validated:
kubectl kustomize dev/prod/all-components, envsubst->YAML parse, make -C ci gate PASS.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
jakman85andClaude Opus 4.8 committed 2026-08-16 21:34:57 -05:00
1 parent 99eef2fe4c
commit d9a80e5350
16 files changed
+278 -53

No files matched your search

+61
View File
@@ -91,6 +91,67 @@ primary play runs, so a limit that excludes the primary yields
self-contained. `ansible-playbook --syntax-check` OK on all playbooks; role lint clean under the CI
gate (`make -C ci gate` PASS). No cluster change; the running cluster is untouched.
## Phase 2 un-suspend PREP — no-secret rollout (author-first, 2026-08-16)
Operator directive: "Phase 2 un-suspend prep … verify cilium parity + fill REPLACE-MEs for the
no-secret components … author-first. Does NOT un-suspend/deploy yet." **Nothing was deployed;** the
running cluster is untouched (still 0 Flux Kustomizations / 0 HelmReleases — only controllers + the
GitRepository).
### Findings from three parallel readiness audits (Longhorn / monitoring / Kyverno)
- **Cilium parity VERIFIED (safety-critical):** every `[parity]` field in `platform/cilium/helmrelease.yaml`
equals the live CNI — `k8sServiceHost=10.73.0.11` (rendered by the chart as the pod env
`KUBERNETES_SERVICE_HOST`, **not** a configmap key — the empty `k8s-service-host` cm value was a red
herring), kube-proxy-replacement, gateway-api, egress+bpf masquerade, hubble, l2-announcements,
cgroup, v1.16.5. **Flux adoption is a no-op reconcile.**
- **Topology:** "suspended" = Option A (**no root Kustomization was ever created**), and the single base
`platform` Kustomization renders **all 11 components at once** — so a naive un-suspend would drag in
the secret-blocked components (cert-manager/externaldns/velero/keycloak/oauth2-proxy/image-automation)
and never go healthy (`apps` dependsOn platform, wait:true).
- **Longhorn:** manifests clean; node prereqs (open-iscsi, /var/lib/longhorn mount, iscsid) are in
cloud-init IaC; the REPLACE-ME bulk-NFS SC is isolated/harmless. Ready.
- **Monitoring:** right-sized (~1.25 GiB) but blocked by (a) needing the `longhorn` SC first and (b) two
**by-name secrets** (`grafana-admin-credentials`, `alertmanager-smtp`) that don't exist — the missing
mounts (not the REPLACE-ME text) are what wedge Grafana/Alertmanager.
- **Kyverno:** zero policy violations today (policies scope to `app-namespace=true`, none exist), but the
fail-closed webhook + 3 HA replicas + Enforce are unsafe/oversized for a cold cluster.
### DECISION D-P2A — per-component Flux Kustomizations for the dev no-secret rollout (operator-approved)
Chose **Option A (per-component Kustomizations)**. Authored `clusters/dev/rollout/` = 4 Flux
Kustomizations (`kyverno` → `cilium` → `longhorn` → `monitoring`, monitoring `dependsOn: longhorn`),
and the dev overlay now `$patch: delete`s the monolithic base `platform`/`apps` and adds `./rollout`.
The no-secret tier carries **no SOPS content**, so the missing `sops-age` key does not block it
(decryption dropped from these Kustomizations). Secret-blocked components are simply **not created**
until their hand-offs land.
- **cilium** uses `prune: false` (adopt the out-of-band CNI; never own-delete it).
- **R-ENV-02 divergence — REVIEW FLAG (architect):** the dev overlay now *adds resources* and diverges
its reconciliation graph from prod (base doc says "dev adds NO resources … graph reused verbatim").
This is a **transitional bootstrap scaffold**, consistent with the already-inlined dev bootstrap
patches (SOPS/URL). **Convergence path is coded in:** delete `clusters/dev/rollout/` + the two
`$patch: delete`s to re-adopt the unified base graph once all hand-offs arrive. **prod is untouched**
(still builds the monolith; verified). Flag for architect ratification.
### Component hardening (env-gated via cluster-vars so prod keeps the strict posture)
- New base knobs (strict/prod defaults) + dev overrides: `KYVERNO_VALIDATION_ACTION` (Enforce→**Audit**),
`KYVERNO_WEBHOOK_FAILURE_POLICY` (Fail→**Ignore**), `KYVERNO_ADMISSION_REPLICAS` (3→**1**),
`MONITORING_STORAGE_CLASS` (longhorn), `GRAFANA_ADMIN_SECRET` (grafana-admin-credentials→**""** in dev
→ chart-default password, dev-only/NAT-isolated).
- Kyverno webhook also gains an explicit `namespaceSelector` excluding kube-system/flux-system/kyverno/
longhorn-system/monitoring (belt-and-suspenders vs a fail-closed brick).
- Monitoring Alertmanager: the `alertmanager-smtp` mount + smtp_*/email_configs are **deferred** (commented
with verbatim re-enable markers) so Alertmanager starts with NULL receivers — alerts fire/route/show in
UI+Grafana, just no email until the Phase-4 SMTP hand-off.
- Validated: `kubectl kustomize` clean for clusters/dev (exactly the 4 Kustomizations, monolith removed),
clusters/prod (monolith intact), and all 4 components; envsubst→YAML-parse of kyverno+monitoring valid;
`make -C ci gate` PASS.
### Still required BEFORE an actual deploy (not part of this author-first prep)
1. **Push current `main` to the host Gitea** — Flux source is pinned at `6c3f9ed` (pre-dates these merges).
2. **Runtime node check:** `systemctl is-active iscsid` + `mountpoint /var/lib/longhorn` on all 3 nodes.
3. **Cilium adoption mechanism:** confirm helm-controller can adopt the Ansible-installed release (was it
`helm`-installed with a release secret, or CLI/kubectl?) — verify before creating the cilium Kustomization.
4. Create the dev root Kustomization (the actual un-suspend) — deliberately NOT done here.
## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed)
| # | Issue | Impact | Fix |
|---|---|---|---|