phase2-prep: per-component no-secret rollout + component hardening (author-first)
Prep only — nothing deployed; running cluster untouched. Reconciliation graph (operator-approved Option A): author clusters/dev/rollout/ = 4 per-component Flux Kustomizations (kyverno->cilium->longhorn->monitoring, monitoring dependsOn longhorn). Dev overlay $patch:delete's the monolithic base platform/apps and adds ./rollout, so only the no-secret tier reconciles; the secret-blocked components are simply not created until hand-offs arrive. The no-secret tier carries no SOPS content, so it needs no sops-age key. prod is untouched (still builds the monolith). Convergence path coded in (delete rollout/ + the two deletes) — R-ENV-02 divergence flagged for architect review. Component hardening via cluster-vars (prod keeps strict posture): - KYVERNO_VALIDATION_ACTION Enforce->Audit(dev), _WEBHOOK_FAILURE_POLICY Fail->Ignore(dev), _ADMISSION_REPLICAS 3->1(dev); webhook namespaceSelector excludes system+platform namespaces. - MONITORING_STORAGE_CLASS var; GRAFANA_ADMIN_SECRET var (dev ''=chart default); Alertmanager SMTP/secret deferred to Phase-4 with verbatim re-enable markers. Cilium parity verified against the live CNI (no-op adopt). Validated: kubectl kustomize dev/prod/all-components, envsubst->YAML parse, make -C ci gate PASS. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This commit is contained in:
1 parent
99eef2fe4c
commit
d9a80e5350
16 files changed
+278
-53
No files matched your search
@@ -91,6 +91,67 @@ primary play runs, so a limit that excludes the primary yields
|
||||
self-contained. `ansible-playbook --syntax-check` OK on all playbooks; role lint clean under the CI
|
||||
gate (`make -C ci gate` PASS). No cluster change; the running cluster is untouched.
|
||||
|
||||
## Phase 2 un-suspend PREP — no-secret rollout (author-first, 2026-08-16)
|
||||
|
||||
Operator directive: "Phase 2 un-suspend prep … verify cilium parity + fill REPLACE-MEs for the
|
||||
no-secret components … author-first. Does NOT un-suspend/deploy yet." **Nothing was deployed;** the
|
||||
running cluster is untouched (still 0 Flux Kustomizations / 0 HelmReleases — only controllers + the
|
||||
GitRepository).
|
||||
|
||||
### Findings from three parallel readiness audits (Longhorn / monitoring / Kyverno)
|
||||
- **Cilium parity VERIFIED (safety-critical):** every `[parity]` field in `platform/cilium/helmrelease.yaml`
|
||||
equals the live CNI — `k8sServiceHost=10.73.0.11` (rendered by the chart as the pod env
|
||||
`KUBERNETES_SERVICE_HOST`, **not** a configmap key — the empty `k8s-service-host` cm value was a red
|
||||
herring), kube-proxy-replacement, gateway-api, egress+bpf masquerade, hubble, l2-announcements,
|
||||
cgroup, v1.16.5. **Flux adoption is a no-op reconcile.**
|
||||
- **Topology:** "suspended" = Option A (**no root Kustomization was ever created**), and the single base
|
||||
`platform` Kustomization renders **all 11 components at once** — so a naive un-suspend would drag in
|
||||
the secret-blocked components (cert-manager/externaldns/velero/keycloak/oauth2-proxy/image-automation)
|
||||
and never go healthy (`apps` dependsOn platform, wait:true).
|
||||
- **Longhorn:** manifests clean; node prereqs (open-iscsi, /var/lib/longhorn mount, iscsid) are in
|
||||
cloud-init IaC; the REPLACE-ME bulk-NFS SC is isolated/harmless. Ready.
|
||||
- **Monitoring:** right-sized (~1.25 GiB) but blocked by (a) needing the `longhorn` SC first and (b) two
|
||||
**by-name secrets** (`grafana-admin-credentials`, `alertmanager-smtp`) that don't exist — the missing
|
||||
mounts (not the REPLACE-ME text) are what wedge Grafana/Alertmanager.
|
||||
- **Kyverno:** zero policy violations today (policies scope to `app-namespace=true`, none exist), but the
|
||||
fail-closed webhook + 3 HA replicas + Enforce are unsafe/oversized for a cold cluster.
|
||||
|
||||
### DECISION D-P2A — per-component Flux Kustomizations for the dev no-secret rollout (operator-approved)
|
||||
Chose **Option A (per-component Kustomizations)**. Authored `clusters/dev/rollout/` = 4 Flux
|
||||
Kustomizations (`kyverno` → `cilium` → `longhorn` → `monitoring`, monitoring `dependsOn: longhorn`),
|
||||
and the dev overlay now `$patch: delete`s the monolithic base `platform`/`apps` and adds `./rollout`.
|
||||
The no-secret tier carries **no SOPS content**, so the missing `sops-age` key does not block it
|
||||
(decryption dropped from these Kustomizations). Secret-blocked components are simply **not created**
|
||||
until their hand-offs land.
|
||||
- **cilium** uses `prune: false` (adopt the out-of-band CNI; never own-delete it).
|
||||
- **R-ENV-02 divergence — REVIEW FLAG (architect):** the dev overlay now *adds resources* and diverges
|
||||
its reconciliation graph from prod (base doc says "dev adds NO resources … graph reused verbatim").
|
||||
This is a **transitional bootstrap scaffold**, consistent with the already-inlined dev bootstrap
|
||||
patches (SOPS/URL). **Convergence path is coded in:** delete `clusters/dev/rollout/` + the two
|
||||
`$patch: delete`s to re-adopt the unified base graph once all hand-offs arrive. **prod is untouched**
|
||||
(still builds the monolith; verified). Flag for architect ratification.
|
||||
|
||||
### Component hardening (env-gated via cluster-vars so prod keeps the strict posture)
|
||||
- New base knobs (strict/prod defaults) + dev overrides: `KYVERNO_VALIDATION_ACTION` (Enforce→**Audit**),
|
||||
`KYVERNO_WEBHOOK_FAILURE_POLICY` (Fail→**Ignore**), `KYVERNO_ADMISSION_REPLICAS` (3→**1**),
|
||||
`MONITORING_STORAGE_CLASS` (longhorn), `GRAFANA_ADMIN_SECRET` (grafana-admin-credentials→**""** in dev
|
||||
→ chart-default password, dev-only/NAT-isolated).
|
||||
- Kyverno webhook also gains an explicit `namespaceSelector` excluding kube-system/flux-system/kyverno/
|
||||
longhorn-system/monitoring (belt-and-suspenders vs a fail-closed brick).
|
||||
- Monitoring Alertmanager: the `alertmanager-smtp` mount + smtp_*/email_configs are **deferred** (commented
|
||||
with verbatim re-enable markers) so Alertmanager starts with NULL receivers — alerts fire/route/show in
|
||||
UI+Grafana, just no email until the Phase-4 SMTP hand-off.
|
||||
- Validated: `kubectl kustomize` clean for clusters/dev (exactly the 4 Kustomizations, monolith removed),
|
||||
clusters/prod (monolith intact), and all 4 components; envsubst→YAML-parse of kyverno+monitoring valid;
|
||||
`make -C ci gate` PASS.
|
||||
|
||||
### Still required BEFORE an actual deploy (not part of this author-first prep)
|
||||
1. **Push current `main` to the host Gitea** — Flux source is pinned at `6c3f9ed` (pre-dates these merges).
|
||||
2. **Runtime node check:** `systemctl is-active iscsid` + `mountpoint /var/lib/longhorn` on all 3 nodes.
|
||||
3. **Cilium adoption mechanism:** confirm helm-controller can adopt the Ansible-installed release (was it
|
||||
`helm`-installed with a release secret, or CLI/kubectl?) — verify before creating the cilium Kustomization.
|
||||
4. Create the dev root Kustomization (the actual un-suspend) — deliberately NOT done here.
|
||||
|
||||
## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed)
|
||||
| # | Issue | Impact | Fix |
|
||||
|---|---|---|---|
|
||||
|
||||
Reference in new issue
Block a user