Merge Phase-2 no-secret rollout prep into main

This commit is contained in:
jakman85 committed 2026-08-16 21:34:58 -05:00
commit e1c738fa63
16 files changed
+278 -53

No files matched your search

+61
View File
@@ -91,6 +91,67 @@ primary play runs, so a limit that excludes the primary yields
self-contained. `ansible-playbook --syntax-check` OK on all playbooks; role lint clean under the CI
gate (`make -C ci gate` PASS). No cluster change; the running cluster is untouched.
## Phase 2 un-suspend PREP — no-secret rollout (author-first, 2026-08-16)
Operator directive: "Phase 2 un-suspend prep … verify cilium parity + fill REPLACE-MEs for the
no-secret components … author-first. Does NOT un-suspend/deploy yet." **Nothing was deployed;** the
running cluster is untouched (still 0 Flux Kustomizations / 0 HelmReleases — only controllers + the
GitRepository).
### Findings from three parallel readiness audits (Longhorn / monitoring / Kyverno)
- **Cilium parity VERIFIED (safety-critical):** every `[parity]` field in `platform/cilium/helmrelease.yaml`
equals the live CNI — `k8sServiceHost=10.73.0.11` (rendered by the chart as the pod env
`KUBERNETES_SERVICE_HOST`, **not** a configmap key — the empty `k8s-service-host` cm value was a red
herring), kube-proxy-replacement, gateway-api, egress+bpf masquerade, hubble, l2-announcements,
cgroup, v1.16.5. **Flux adoption is a no-op reconcile.**
- **Topology:** "suspended" = Option A (**no root Kustomization was ever created**), and the single base
`platform` Kustomization renders **all 11 components at once** — so a naive un-suspend would drag in
the secret-blocked components (cert-manager/externaldns/velero/keycloak/oauth2-proxy/image-automation)
and never go healthy (`apps` dependsOn platform, wait:true).
- **Longhorn:** manifests clean; node prereqs (open-iscsi, /var/lib/longhorn mount, iscsid) are in
cloud-init IaC; the REPLACE-ME bulk-NFS SC is isolated/harmless. Ready.
- **Monitoring:** right-sized (~1.25 GiB) but blocked by (a) needing the `longhorn` SC first and (b) two
**by-name secrets** (`grafana-admin-credentials`, `alertmanager-smtp`) that don't exist — the missing
mounts (not the REPLACE-ME text) are what wedge Grafana/Alertmanager.
- **Kyverno:** zero policy violations today (policies scope to `app-namespace=true`, none exist), but the
fail-closed webhook + 3 HA replicas + Enforce are unsafe/oversized for a cold cluster.
### DECISION D-P2A — per-component Flux Kustomizations for the dev no-secret rollout (operator-approved)
Chose **Option A (per-component Kustomizations)**. Authored `clusters/dev/rollout/` = 4 Flux
Kustomizations (`kyverno` → `cilium` → `longhorn` → `monitoring`, monitoring `dependsOn: longhorn`),
and the dev overlay now `$patch: delete`s the monolithic base `platform`/`apps` and adds `./rollout`.
The no-secret tier carries **no SOPS content**, so the missing `sops-age` key does not block it
(decryption dropped from these Kustomizations). Secret-blocked components are simply **not created**
until their hand-offs land.
- **cilium** uses `prune: false` (adopt the out-of-band CNI; never own-delete it).
- **R-ENV-02 divergence — REVIEW FLAG (architect):** the dev overlay now *adds resources* and diverges
its reconciliation graph from prod (base doc says "dev adds NO resources … graph reused verbatim").
This is a **transitional bootstrap scaffold**, consistent with the already-inlined dev bootstrap
patches (SOPS/URL). **Convergence path is coded in:** delete `clusters/dev/rollout/` + the two
`$patch: delete`s to re-adopt the unified base graph once all hand-offs arrive. **prod is untouched**
(still builds the monolith; verified). Flag for architect ratification.
### Component hardening (env-gated via cluster-vars so prod keeps the strict posture)
- New base knobs (strict/prod defaults) + dev overrides: `KYVERNO_VALIDATION_ACTION` (Enforce→**Audit**),
`KYVERNO_WEBHOOK_FAILURE_POLICY` (Fail→**Ignore**), `KYVERNO_ADMISSION_REPLICAS` (3→**1**),
`MONITORING_STORAGE_CLASS` (longhorn), `GRAFANA_ADMIN_SECRET` (grafana-admin-credentials→**""** in dev
→ chart-default password, dev-only/NAT-isolated).
- Kyverno webhook also gains an explicit `namespaceSelector` excluding kube-system/flux-system/kyverno/
longhorn-system/monitoring (belt-and-suspenders vs a fail-closed brick).
- Monitoring Alertmanager: the `alertmanager-smtp` mount + smtp_*/email_configs are **deferred** (commented
with verbatim re-enable markers) so Alertmanager starts with NULL receivers — alerts fire/route/show in
UI+Grafana, just no email until the Phase-4 SMTP hand-off.
- Validated: `kubectl kustomize` clean for clusters/dev (exactly the 4 Kustomizations, monolith removed),
clusters/prod (monolith intact), and all 4 components; envsubst→YAML-parse of kyverno+monitoring valid;
`make -C ci gate` PASS.
### Still required BEFORE an actual deploy (not part of this author-first prep)
1. **Push current `main` to the host Gitea** — Flux source is pinned at `6c3f9ed` (pre-dates these merges).
2. **Runtime node check:** `systemctl is-active iscsid` + `mountpoint /var/lib/longhorn` on all 3 nodes.
3. **Cilium adoption mechanism:** confirm helm-controller can adopt the Ansible-installed release (was it
`helm`-installed with a release secret, or CLI/kubectl?) — verify before creating the cilium Kustomization.
4. Create the dev root Kustomization (the actual un-suspend) — deliberately NOT done here.
## KNOWN ISSUES to fix before Phase-2 un-suspend (logged, not yet fixed)
| # | Issue | Impact | Fix |
|---|---|---|---|
+9
View File
@@ -27,6 +27,15 @@ data:
MONITORING_PROMETHEUS_RETENTION: "7d" # dev short | prod long
MONITORING_PROMETHEUS_PVC_SIZE: "10Gi"
MONITORING_LOKI_PVC_SIZE: "10Gi"
MONITORING_STORAGE_CLASS: "longhorn" # PVC tier for Prometheus/Loki (dev may override)
GRAFANA_ADMIN_SECRET: "grafana-admin-credentials" # by-name SOPS secret (prod); dev="" → chart default
# --- admission enforcement (Kyverno, ADR-0011). Base defaults are the STRICT/prod posture;
# the dev overlay softens them for bootstrap (Audit / Ignore / 1) so a cold cluster can
# come up before policies are proven and admission is HA. ---
KYVERNO_VALIDATION_ACTION: "Enforce" # dev: Audit (observe before enforcing)
KYVERNO_WEBHOOK_FAILURE_POLICY: "Fail" # dev: Ignore (fail-open during bootstrap)
KYVERNO_ADMISSION_REPLICAS: "3" # dev: 1 (no HA needed to iterate)
# --- identity / auth (ADR-0014) ---
KEYCLOAK_REPLICAS: "1" # dev 1 | prod 2 (HA)
+30 -16
View File
@@ -2,39 +2,38 @@
# clusters/dev — the ACTIVE build target overlay.
#
# `flux bootstrap ... --path=clusters/dev` points the dev cluster's Flux here.
# This overlay adds NO resources of its own. Its ONLY job is to patch the shared cluster-vars
# ConfigMap with dev-scoped values (R-ENV-02). The base reconciliation graph is reused verbatim.
# TRANSITIONAL (Phase-2 bootstrap): this overlay reuses the base source + cluster-vars, but
# REPLACES the monolithic base `platform`/`apps` Kustomizations with a per-component NO-SECRET
# rollout (./rollout) — so dev brings up cilium/longhorn/monitoring/kyverno in order WITHOUT
# dragging in the secret-blocked components. It patches cluster-vars with dev values (R-ENV-02)
# and softens Kyverno for bootstrap. CONVERGENCE: delete ./rollout + the two $patch:delete
# below to return to the unified base graph once all operator hand-offs land. See DECISIONS-LOG.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../base
- rollout
patches:
# --- SOPS decryption + bootstrap source (PM-inlined from bootstrap/flux/patches/, 2026-08-16) ---
# Inlined (not cross-dir file refs) to stay within kustomize's default load restrictions.
# Realises the commented decryption TODOs in clusters/base and sets the real source URL.
# The `sops-age` Secret is created out-of-band (bootstrap/sops/install-age-key.sh) — name only.
# --- Drop the monolithic base platform/apps Kustomizations. They reconcile ALL 11 components
# as one unit (incl. secret-blocked cert-manager/externaldns/velero/keycloak/oauth2-proxy/
# image-automation/gateway/kubevirt); ./rollout replaces them with the per-component
# no-secret set for dev bootstrap. Remove these two deletes to re-adopt the unified graph.
# (No decryption patch is needed here: the rollout components carry no SOPS content, so
# the `sops-age` Secret is not required for the dev bootstrap tier.) ---
- patch: |-
$patch: delete
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: platform
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
$patch: delete
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: apps
namespace: flux-system
spec:
decryption:
provider: sops
secretRef:
name: sops-age
- patch: |-
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
@@ -84,3 +83,18 @@ patches:
- op: replace
path: /data/VELERO_RETENTION
value: "72h"
# dev-softened Kyverno for cold-start (base defaults Enforce/Fail/3 = prod posture).
- op: replace
path: /data/KYVERNO_VALIDATION_ACTION
value: "Audit"
- op: replace
path: /data/KYVERNO_WEBHOOK_FAILURE_POLICY
value: "Ignore"
- op: replace
path: /data/KYVERNO_ADMISSION_REPLICAS
value: "1"
# dev: no Grafana admin Secret yet → empty makes the chart use its default admin password
# (dev-only, NAT-isolated). Prod keeps the by-name SOPS secret from the base default.
- op: replace
path: /data/GRAFANA_ADMIN_SECRET
value: ""
+25
View File
@@ -0,0 +1,25 @@
---
# Cilium — ADOPT the running Ansible-installed CNI. Parity between platform/cilium and the
# live config was verified (k8sServiceHost/port via pod env, kube-proxy-replacement, gateway-api,
# egress+bpf masquerade, hubble, l2, cgroup, v1.16.5) → this reconcile is a NO-OP.
# prune: false — Cilium was bootstrapped out-of-band; adopt its config, never own-delete the CNI.
# No decryption (no SOPS content). No dependsOn (it is the datapath everything else rides on).
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cilium
namespace: flux-system
spec:
interval: 30m
retryInterval: 2m
timeout: 10m
prune: false
wait: true
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform/cilium
postBuild:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
+20
View File
@@ -0,0 +1,20 @@
---
# Phase-2 NO-SECRET incremental rollout — DEV bootstrap scaffold.
#
# Per-component Flux Kustomizations for the components that need NO operator secret:
# kyverno (admission, dev-softened) · cilium (adopt running CNI) · longhorn (storage)
# · monitoring (dependsOn longhorn).
# Secret-blocked components (cert-manager, externaldns, gateway, velero, keycloak,
# oauth2-proxy, image-automation, kubevirt) are intentionally NOT created until their
# operator hand-offs arrive (DECISIONS-LOG A2 / hand-off table).
#
# TRANSITIONAL: this is the dev bootstrap graph. Once every hand-off lands, dev drops
# ./rollout and re-adopts the unified base monolith (clusters/base/platform.yaml) so dev
# and prod share one reconciliation graph again (R-ENV-02 convergence). See DECISIONS-LOG.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- kyverno.yaml # first: admission present (Audit/Ignore in dev — non-blocking) before workloads
- cilium.yaml # adopt the running CNI (parity verified — no-op reconcile)
- longhorn.yaml # storage; monitoring depends on its StorageClass
- monitoring.yaml # dependsOn longhorn
+25
View File
@@ -0,0 +1,25 @@
---
# Kyverno — in-cluster admission enforcer. DEV-SOFTENED via cluster-vars:
# KYVERNO_VALIDATION_ACTION=Audit, KYVERNO_WEBHOOK_FAILURE_POLICY=Ignore,
# KYVERNO_ADMISSION_REPLICAS=1 (base defaults Enforce/Fail/3 for prod).
# Non-blocking on today's cluster (policies match only `app-namespace=true` namespaces,
# none exist yet). No SOPS content. Listed first so admission is present before workloads.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: kyverno
namespace: flux-system
spec:
interval: 1h
retryInterval: 2m
timeout: 10m
prune: true
wait: true
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform/kyverno
postBuild:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
+25
View File
@@ -0,0 +1,25 @@
---
# Longhorn — block storage / CSI (default `longhorn` StorageClass). No SOPS content.
# No dependsOn. The bulk-NFS StorageClass in this component carries REPLACE-ME (OQ-2) but is
# an isolated, unreferenced object — it does NOT block Longhorn's HelmRelease reconcile.
# PRE-DEPLOY CHECK (runtime, out of git): confirm `iscsid` active + /var/lib/longhorn mounted
# on all 3 nodes (cloud-init provides both; verify after the T-1.5 rebuild).
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: longhorn
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 10m
prune: true
wait: true
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform/longhorn
postBuild:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
+26
View File
@@ -0,0 +1,26 @@
---
# Monitoring — kube-prometheus-stack + Loki + Alloy. Prometheus/Loki PVCs bind the
# `longhorn` StorageClass, so this dependsOn longhorn (Flux waits for it Ready first).
# Grafana admin password (chart default in dev) and Alertmanager SMTP (deferred to the
# Phase-4 hand-off) are handled WITHOUT an operator secret in dev → NO decryption block.
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: monitoring
namespace: flux-system
spec:
interval: 10m
retryInterval: 2m
timeout: 15m
prune: true
wait: true
dependsOn:
- name: longhorn
sourceRef:
kind: GitRepository
name: flux-system
path: ./platform/monitoring
postBuild:
substituteFrom:
- kind: ConfigMap
name: cluster-vars
+20 -6
View File
@@ -34,18 +34,32 @@ spec:
remediation:
retries: 3
values:
# HA admission controller so a single pod restart cannot open an enforcement gap.
# Admission controller replicas — cluster-vars knob (prod 3 / dev 1). HA so a single pod
# restart cannot open an enforcement gap; dev drops to 1 to save RAM during bootstrap.
admissionController:
replicas: 3
replicas: ${KYVERNO_ADMISSION_REPLICAS}
backgroundController:
enabled: true # needed for generate (default-deny baseline) + sync
reportsController:
enabled: true
cleanupController:
enabled: true
# Fail-CLOSED: if the webhook is unreachable, reject rather than silently admit
# non-conforming resources (coordination-protocol.md #8). Tune per-resource in
# the policies' webhook config if a bootstrap deadlock appears.
# Webhook failure policy — cluster-vars knob (prod Fail / dev Ignore). Fail is fail-CLOSED
# (reject rather than silently admit when the webhook is unreachable, coordination-protocol.md
# #8); dev uses Ignore (fail-open) so a cold/unhealthy Kyverno cannot brick admission during
# bootstrap. The namespaceSelector ALSO excludes system + platform namespaces at the webhook
# layer — belt-and-suspenders: policies already scope to app-namespace=true, but this
# guarantees kube-system/flux-system/CNI/storage/monitoring admission never routes to Kyverno.
config:
webhooks:
- failurePolicy: Fail
- failurePolicy: ${KYVERNO_WEBHOOK_FAILURE_POLICY}
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values:
- kube-system
- flux-system
- kyverno
- longhorn-system
- monitoring
@@ -24,7 +24,7 @@ metadata:
homelab.keithjackson.dev/requirements: R-SEC-01
homelab.keithjackson.dev/scope: admission-only
spec:
validationFailureAction: Enforce
validationFailureAction: ${KYVERNO_VALIDATION_ACTION}
background: false
rules:
- name: secrets-must-be-flux-managed
@@ -25,7 +25,7 @@ metadata:
Pods in application namespaces must declare
homelab.keithjackson.dev/auth-mode = oidc | forward-auth | native | none.
spec:
validationFailureAction: Enforce
validationFailureAction: ${KYVERNO_VALIDATION_ACTION}
background: true
rules:
- name: require-auth-mode
@@ -25,7 +25,7 @@ metadata:
Pods in application namespaces must declare
homelab.keithjackson.dev/egress-class = internet | vpn.
spec:
validationFailureAction: Enforce
validationFailureAction: ${KYVERNO_VALIDATION_ACTION}
background: true
rules:
- name: require-egress-class
@@ -24,7 +24,7 @@ metadata:
Application-namespace pods must not use hostNetwork/hostPID/hostIPC (which
bypass default-deny) or run privileged containers.
spec:
validationFailureAction: Enforce
validationFailureAction: ${KYVERNO_VALIDATION_ACTION}
background: true
rules:
- name: disallow-host-namespaces
@@ -20,7 +20,7 @@ metadata:
to include both Ingress and Egress. Rejects a baseline that silently leaves
a direction open.
spec:
validationFailureAction: Enforce
validationFailureAction: ${KYVERNO_VALIDATION_ACTION}
background: true
rules:
- name: baseline-must-deny-both-directions
+31 -25
View File
@@ -62,7 +62,7 @@ spec:
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: longhorn # replicated tier (ADR-0003)
storageClassName: ${MONITORING_STORAGE_CLASS} # replicated tier (ADR-0003); cluster-vars knob
accessModes: ["ReadWriteOnce"]
resources:
requests:
@@ -80,22 +80,24 @@ spec:
memory: 32Mi
limits:
memory: 96Mi
# SMTP credentials are an OPERATOR HAND-OFF: a SOPS-encrypted Secret, referenced BY NAME
# only (docs/07). Mounted at /etc/alertmanager/secrets/<name>/ so the config below can read
# the password from a file (never inline plaintext).
secrets:
- alertmanager-smtp
# Routing/receivers. Non-secret SMTP endpoint fields (smarthost/from/to/username) are
# operator-supplied placeholders (REPLACE-ME, same convention as velero); the PASSWORD is
# read from the mounted SOPS secret via *_file. Both severities email; subject/route differ.
# SMTP is an OPERATOR HAND-OFF DEFERRED to Phase 4 — no creds yet, and the by-name
# `alertmanager-smtp` Secret does NOT exist. Mounting a missing Secret wedges Alertmanager
# in ContainerCreating, so the mount is DISABLED until the hand-off. Re-enable verbatim
# (uncomment) when the SOPS `alertmanager-smtp` Secret is delivered. TODO(Phase-4 SMTP).
# secrets:
# - alertmanager-smtp
# Routing stays live; RECEIVERS ARE NULL until the SMTP hand-off (alerts still fire, route,
# and show in the Alertmanager UI + Grafana — they just aren't emailed). The smtp_* globals
# and email_configs are retained below, commented, so re-enabling is a pure uncomment.
config:
global:
resolve_timeout: 5m
smtp_smarthost: "REPLACE-ME:587" # operator hand-off (e.g. smtp.gmail.com:587)
smtp_from: "REPLACE-ME" # operator hand-off (From: address)
smtp_auth_username: "REPLACE-ME" # operator hand-off (SMTP login)
smtp_auth_password_file: /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
smtp_require_tls: true
# TODO(Phase-4 SMTP hand-off): uncomment together with the alertmanager-smtp Secret mount.
# smtp_smarthost: "smtp.example.com:587"
# smtp_from: "[email protected]"
# smtp_auth_username: "<smtp-login>"
# smtp_auth_password_file: /etc/alertmanager/secrets/alertmanager-smtp/smtp-password
# smtp_require_tls: true
route:
group_by: ["alertname", "namespace"]
group_wait: 30s
@@ -109,25 +111,29 @@ spec:
group_wait: 10s
repeat_interval: 1h
receivers:
# NULL receivers until Phase-4 SMTP. email_configs retained (commented) for verbatim re-enable.
- name: email-warning
email_configs:
- to: "REPLACE-ME" # operator hand-off (alert inbox)
send_resolved: true
headers:
subject: '[WARNING] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
# email_configs:
# - to: "<alert-inbox>"
# send_resolved: true
# headers:
# subject: '[WARNING] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
- name: email-critical
email_configs:
- to: "REPLACE-ME" # operator hand-off (alert inbox)
send_resolved: true
headers:
subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
# email_configs:
# - to: "<alert-inbox>"
# send_resolved: true
# headers:
# subject: '[CRITICAL] {{ .CommonLabels.alertname }} ({{ .CommonLabels.namespace }})'
# --- Grafana: dashboards (R-OBS-03), surfaced in the C2 UI (docs/12) behind Keycloak SSO ---
grafana:
enabled: true
# Admin bootstrap + (later) OIDC client are SOPS secrets referenced BY NAME (WS-SEC/ADR-0014).
# cluster-vars knob: prod = the SOPS `grafana-admin-credentials` Secret; dev = "" (empty),
# which makes the chart fall back to its default admin password (dev-only, NAT-isolated) so
# Grafana starts without an operator secret. TODO: supply a dev Secret if wider dev access.
admin:
existingSecret: grafana-admin-credentials
existingSecret: ${GRAFANA_ADMIN_SECRET}
userKey: admin-user
passwordKey: admin-password
# DEV TRIM: emptyDir (default). Dashboards are provisioned from ConfigMaps (sidecar), so no
+1 -1
View File
@@ -60,7 +60,7 @@ spec:
replicas: 1
persistence:
enabled: true
storageClass: longhorn # replicated tier (ADR-0003)
storageClass: ${MONITORING_STORAGE_CLASS} # replicated tier (ADR-0003); cluster-vars knob
size: ${MONITORING_LOKI_PVC_SIZE} # overlay knob (dev 5Gi / prod large)
resources:
requests: